LLM Assistant: Cost & Compliance Driven

Context

Developers had individually adopted a mix of AI coding assistants and LLM providers, with no shared standard for how usage was tracked, logged, or paid for.

Problem

There was no centralized logging or cost visibility into LLM usage across the team, and no consistent policy on data sovereignty — GDPR-covered usage needed to stay on EU-hosted infrastructure, and some data needed to remain fully in-house rather than reach any third-party provider.

Approach

• Standardized all LLM usage — self-hosted models and EU-hosted providers alike — behind a single LiteLLM proxy, replacing a patchwork of individual subscriptions and personal API keys
• Routed requests to self-hosted or EU-hosted models based on data sensitivity, keeping GDPR-covered and in-house-only data inside the approved sovereignty boundary instead of reaching third-party APIs
• Centralized logging, usage metering, and cost tracking at the proxy layer, giving per-team/per-project spend visibility the same way cloud costs are tagged and attributed
• Layered in a RAG pipeline so developers got answers grounded in internal knowledge without that data leaving the infrastructure boundary
• Applied normal production discipline — containerized deployment, secrets management, monitoring/logging

Tools

LiteLLM Self-hosted LLM EU-hosted LLM RAG Usage metering Access control

Outcome

Consolidated all LLM usage — self-hosted and EU-hosted — behind a single LiteLLM proxy, replacing scattered subscriptions and personal API keys with one governed, centrally logged layer. Every request became visible for cost and usage tracking per team/project, and GDPR-covered or sensitive data stopped leaving the approved EU/self-hosted boundary entirely.