Where your LLM bill actually goes

An LLM bill is a token bill. You pay for what you send and you pay more for what comes back. Almost every method below is a way of sending fewer tokens, receiving fewer tokens, or not making the call twice. None of them require a rewrite, and most can be done in an afternoon.

Most of your bill is re-reading

A chat has no memory of its own. Every turn resends the entire conversation so far, so a session does not cost one unit per turn. It costs roughly the sum of every turn before it.

Say each turn adds 500 tokens. Turn 2 sends 500 tokens of history. Turn 20 sends about 9,500 tokens of history to add 500 new ones, so 95 percent of that request is text the model has already read. Doubling the length of a session roughly quadruples what it costs.

The lesson is not to talk less. It is that long sessions are where the money goes, so starting a fresh session when the subject changes is worth more than any wording change you could make.

Ask for less back

Output tokens cost several times what input tokens cost, which makes the cheapest sentence the one you never asked for. Set a maximum output length. Say "answer in one paragraph" or "return only the function body" and mean it.

This is the most commonly skipped step. Teams will spend an afternoon tightening a system prompt and never once cap the response, and the response was always the expensive half. A model asked for a thorough explanation will write a thorough explanation, and you will pay for every word of it.

Send the payload, not the packaging

Pretty-printed JSON spends real money on indentation. A table sent as one record per row repeats every column name on every row. A base64 image pasted into a prompt is enormous and is rarely what anyone meant to send.

Before attaching a file, ask whether the model needs the whole file or one function out of it. You would not fax somebody a filing cabinet to ask a question about one folder.

Let a smaller model do the boring work

Not every call needs your best model. Classification, extraction, reformatting, routing, and yes-or-no checks usually run fine on a cheaper one. A practical pattern is to let the small model answer first and escalate only when it reports that it is unsure.

Most production traffic is boring. Boring is cheap, if you let it be.

Answer the same question once

Support assistants, documentation search, and internal tools get asked the same handful of questions all day. Keep a cache keyed on the question and check it before you check the model.

An exact-match cache is a dictionary with a hash for a key. It takes an afternoon, costs nothing to run, and on repetitive traffic it is close to free money. The interesting versions of this idea get complicated quickly, but the boring version already pays.

Cut words, not meaning

Prompt engineering saves money mainly by deleting instructions the model was already going to follow. Long preambles about being helpful, formatting rules repeated three times, and few-shot examples that stopped earning their keep are all common and all billable on every single call.

Remove one block at a time and look at the output. If nothing changed, keep the cut. If something changed, put it back. This is dull work and it is the only version of prompt engineering that reliably shows up on an invoice.

One caution: judge the change on the total, input plus output together. A prompt that got shorter but produced a longer answer did not save you anything.

Know which prompt is expensive

Most teams cannot name their most expensive prompt. Log tokens per request with a tag for the feature that made the call, then leave it running for a week.

That week almost always turns up one runaway prompt costing more than everything else combined, and it is almost never the one people expected. Optimizing before you have this number is guessing, and guessing tends to tune the prompt someone happens to remember rather than the one that is actually billing.

Where Tidbit fits

Everything above is yours to do, and worth doing whether or not you ever use a gateway. Tidbit exists for the part that is tedious to do by hand, call after call.

It sits in front of your existing setup as a gateway, so pointing your client at it is the only change, and pointing the client back is the only removal. Your Console shows the savings measured and attributed to Tidbit’s own work, and savings produced by your own client-side caching are never counted as ours.

The published benchmark numbers each carry an evidence ID, a fingerprint of the source data, and a checked-in derivation, so any figure can be traced back to the run that produced it. Read those before assuming a result transfers to your workload, because what you save depends on request size, session length, and how much stable context you repeat.

Tidbit is free on a Claude subscription, and free on the API for the duration of early access.