LLM Token Gateway + AI FinOps

Context

Companies adopted LLMs across dozens of use cases — and lost control of the spend. Without observability, nobody knows which team consumes the most tokens, which model has the best cost/quality trade-off, or when a bug started silently burning budget. The invoice arrives at the end of the month and the postmortem starts there, which is the worst possible place to start one.

This project implements the answer: a gateway that intercepts every call, and a lakehouse that turns those calls into cost intelligence.

It is also the flagship of this portfolio, for a reason worth stating plainly: most data portfolios use AI, and this one measures and governs it. And it is grounded in real work rather than a tutorial — it builds on an existing FinOps system I maintain. Concretely, that means the parts that are easy to get wrong (modeling cost as of the time it was incurred, versioning price history, choosing a threshold that alerts without crying wolf) come from having already run into them, not from reading about them. What lives in this repository is a self-contained build focused specifically on LLM spend.

Business Problem

How do we get visibility and control over LLM API consumption — cost per team, per model, per use case — with anomaly detection before the invoice arrives?

"Before the invoice arrives" is the requirement that shapes everything else. A monthly bill tells you what happened; it does not tell you which team, which model, or which deploy caused it. Attribution has to be captured at the moment of the call or it cannot be reconstructed later.

Architecture

Request path

A transparent proxy rather than a custom SDK: applications change exactly one thing, the base URL. The gateway exposes an endpoint compatible with the original API, forwards the request, returns the provider's response untouched, and captures the metadata it needs on the way through — model, tokens in and out, latency measured at the gateway, status, and the calling team resolved from the gateway's own API key. Nothing about the client's code has to know it exists.

Data path — medallion lakehouse

• Bronze: raw log of every request (JSON) in Delta/Parquet on S3 — timestamp, model, tokens in/out, latency, team/user, status. Append-only, partitioned by date.
• Silver: parsed and typed events, enriched with a per-model price table → cost computed per request.
• Gold (dbt marts): cost per team/day, tokens per use case, p95 latency per model, consumption ranking.
• Consumption: FinOps dashboard + spend anomaly alerts.

A lakehouse (Delta on S3) instead of a plain relational database: the raw log is cheap, append-only and never mutated, time travel gives an audit trail, and the trustworthy marts sit on top of it rather than replacing it. If the cost model turns out to be wrong, bronze is still the truth and silver can be rebuilt.

Data Source

There is no public dataset here — the system produces its own data. Every call that passes through the gateway becomes one usage event, which is precisely why attribution works: the team, the model and the token counts are recorded at the moment the cost is incurred, not inferred afterward from an aggregate bill.

• Events: timestamp, model, tokens in/out, latency, team/user, status — one row per request.
• Monitored APIs: Anthropic, OpenAI.
• Caller identity: resolved from the gateway's own API key, mapped to a team/project. Applications authenticate to the gateway; the gateway holds the provider credentials.
• Price reference: a per-model price table maintained as a versioned dbt seed, joined at the price in effect at request time.
• Storage: Delta Lake / Parquet on AWS S3, append-only, partitioned by date.

Methodology

Medallion architecture (bronze → silver → gold) applied to a FinOps use case:

1. Gateway (FastAPI): an endpoint compatible with the original API that forwards the request and captures response metadata — usage, latency, model.
2. Caller identity via the gateway's own API key, mapped to team/project.
3. Bronze ingestion: each event appended to S3, append-only, partitioned by date.
4. Silver/Gold with dbt: cost calculation against the versioned price seed, aggregations, p95 latency.
5. Consumption & alerting: dashboard plus spend anomaly detection — daily spend beyond N deviations from the moving average.

FinOps marts

• fct_llm_requests — grain = 1 request (tokens, cost, latency, team, model).
• cost_per_team_daily — budget tracking.
• model_efficiency — average cost per 1k tokens vs. p95 latency; the evidence you need to justify switching models.
• spend_anomalies — days with spend above N deviations from the moving average.

Latency is measured at the gateway, deliberately. That captures the real experience of the calling application — network included — rather than only what the provider reports about its own processing.

AI and responsible use

AI is the subject of this project, used with governance rather than hype. The gateway makes AI usage observable and accountable: who spends what, on which model, for what. Where AI is used to classify requests — for example, tagging use cases from prompt metadata — the classification is validated against a manually labelled sample and the precision is reported. AI assists, the human verifies. The design goal is to demonstrate good use of AI (cost governance, evaluation, guardrails), not dependency on it.

Challenges

1. Logging without taxing the request path

The gateway sits in the hot path of every LLM call, which makes any latency it adds a tax paid by every application on every request. Writing to S3 synchronously would do exactly that — object storage latency on top of an already slow call, to serve an analytics need the caller does not care about. The event is therefore captured in-process and written outside the response path, so the client waits for the provider and nothing else. The trade-off is explicit rather than hidden: a crash can lose a small window of events. That is acceptable for cost analytics and would not be acceptable for billing, and the difference is worth being honest about.

2. Never leaking API keys

The gateway is a natural place to concentrate secrets: it holds the provider keys (Anthropic, OpenAI), the gateway keys that identify each team, and AWS credentials for the lake. All of them live only in .env or environment variables, never committed — .gitignore from commit #1, no secret in the git history — and IAM follows least privilege. The subtler risk is the logging itself. A proxy that logs "the request" for debugging convenience is one careless line away from persisting an Authorization header into an append-only lake that is designed never to be mutated. Bronze stores usage metadata — model, tokens, latency, team, status — and credentials are scrubbed before anything is written, not after.

3. Pricing history: the numbers move under you

Model prices change, and they mostly go down. A cost model that joins to a single current price table silently rewrites history every time a provider cuts prices — last quarter's spend report stops matching last quarter's invoice, and the dashboard quietly becomes a liar. Prices live in a versioned dbt seed, and silver joins on the price that was in effect at the request timestamp. Historical cost has to reflect the price of the time, or it is not a cost, just a recalculation.

4. Anomaly detection needs a baseline first

"Spend anomaly" is meaningless on day one — a spike is only a spike relative to something. The mart flags days beyond N deviations from a moving average, which means the detector is unreliable until enough history accumulates, and it is sensitive to the choice of N and window: too tight and it cries wolf until people mute the channel, too loose and the runaway retry loop goes unnoticed until the invoice. It also cannot distinguish a legitimate launch from a bug. It flags for review; it does not judge, and it should not be sold as if it does.

Results

Planned outputs:

• A FinOps dashboard: cost per team/day, tokens per use case, p95 latency per model, consumption ranking.
• model_efficiency — cost per 1k tokens against p95 latency, the trade-off made visible.
• Spend anomaly alerts firing on daily spend beyond N deviations from the moving average.
• Reported precision of the AI use-case classification against a manually labelled sample.
• Measured gateway overhead, so the "no meaningful latency added" claim above is a number rather than an intention.

Next steps once the base is working: rate limiting and per-team budgets enforced in the gateway (moving it from observer to controller), response caching for repeated prompts (real cost reduction), and streaming (SSE) support with real-time token counting.

Tech Stack