Batching and prompt engineering, in that order

Two cost levers get mentioned in the same breath and they are not the same size. Batching is a discount you claim by agreeing to wait, and claiming it is mostly configuration. Prompt engineering is real work with a real return, and it is also where teams spend a week to save a rounding error. Do them in the order the title suggests.

Batching is a discount for patience

Most major providers, Anthropic included, run an asynchronous batch endpoint. You submit a set of calls, results come back within a stated window rather than immediately, and the rate is roughly half the standard one. The model and the prompt are unchanged. What you give up is a synchronous reply.

Rates and windows move, so read the current price sheet rather than trusting this paragraph. The shape of the deal has been stable: patience is the cheapest thing you can trade.

Most work does not need an answer now

The question to ask about a call is who is waiting for it. Nightly enrichment jobs, backfills over historical records, evaluation and regression suites, classification sweeps over a document store, morning digest emails, synthetic data generation, and first-pass moderation queues all answer it the same way, which is that a cron job is waiting.

If a person will read the result tomorrow morning, paying the synchronous premium bought nothing. This is the rare optimization with no quality argument attached to it, which is exactly why it tends to get skipped in favor of something that feels more like engineering.

Many items, one call

Grouping several items into a single call is a different lever, and it works by amortizing the instructions. If your system prompt is 800 tokens and each item is 50, then one call per item spends 850 tokens per item. Twenty items in one call spends about 1,800 total, or 90 per item.

Four cautions. Output grows in proportion, so cap it. Quality falls off past some group size, and the size is workload-specific, so find it with a small evaluation set rather than guessing. One malformed item can derail the ones after it, so ask for a structured result keyed by item id and re-run the misses individually. And position matters: items late in a long list tend to get less careful treatment than the first few.

Concurrency is not batching

Firing fifty calls in parallel shortens the wall clock and leaves the bill identical. It is worth doing for latency and worth not confusing with a cost lever. It is also the quickest route to a rate limit, which is how a latency optimization turns into a reliability incident.

The prompt changes that actually bill

Ask for a schema instead of prose. "Return JSON matching this shape" produces a shorter answer and removes the parsing step, where a request for an explanation produces an explanation and you pay for the connective tissue around the part you wanted.

Stop paying for deliberation the task does not need. Step-by-step reasoning earns its keep on genuinely hard problems and is pure cost on classification, extraction, and formatting. On models with a thinking budget, that budget bills like output. Set it to what the work needs and no higher.

Spend few-shot examples where they change behavior. Examples are billed call after call, forever. Two well-chosen ones usually beat six near-duplicates. Remove one, run your evaluation set, and keep the removal if the score held.

Keep the stable part of the prompt stable. Providers document a prompt cache that rewards a prompt whose opening does not change, and a timestamp or session id injected at the top of a long system prompt is a common way to give that up for nothing. Volatile text belongs near the end.

Do the cheap thing first

A workable order of operations: move deferrable work to the batch endpoint, group items where the instructions dominate the payload, cap output and drop deliberation the task does not need, and only then start trimming wording with an evaluation set open beside you.

Most teams run that list backwards. Wording is the most visible lever and the smallest, and it is the only one of the four where you can convince yourself something improved without measuring. Judge any prompt change on the total, input plus output together, against a fixed set of real inputs. Ten saved prompts in a file is enough to start.

Where Tidbit fits

Scheduling and prompt discipline are yours, and they pay whether or not a gateway is involved. Tidbit works on the per-call waste that is tedious to chase by hand once the obvious passes are done.

It sits in front of your existing setup, so adopting it is one configuration change and removing it is the same change in reverse. The Console reports savings attributed to Tidbit’s own work, and reductions your own client-side caching produced are not counted as ours.

Published benchmark figures each carry an evidence ID, a fingerprint of the source data, and a checked-in derivation, so any number can be traced to the run behind it. Read those before assuming a result transfers, because outcomes depend on request size, session length, and how much stable context repeats.

Tidbit is free on a Claude subscription, and free on the API for the duration of early access.