Batch API vs Real-Time: When 50% Off Is Worth the Wait

Last updated: October 2026

Every major LLM provider sells the same model at two prices: full price for answers in seconds, roughly half price for answers in hours. The Batch API is the most underused discount in the industry — same model, same quality, ~50% off, in exchange for patience. Here's when it's worth it and when it isn't.

What Batch Mode Actually Is

A batch request is exactly what it sounds like: you submit a batch of requests as a file, the provider processes them when it has spare capacity, and you collect the results later — typically within 24 hours. Same weights, same capabilities, same output quality. The discount exists because you're letting the provider schedule your work during off-peak capacity instead of demanding instant compute.

On GPT-5.6 Sol, batch processing cuts the promo rates in half — $2.00 input / $10.00 output per 1M tokens instead of $4.00/$20.00. The mechanics are simple: upload a JSONL file of requests, poll for completion, download the results.

The Worked Example: What 50% Off Looks Like

Take a concrete workload: a nightly eval run — 500 test cases, each with 8K input tokens and 2K output tokens, on GPT-5.6 Sol's promo pricing.

Real-time: 500 x 8,000 = 4M input tokens x $4.00/1M = $16.00; 500 x 2,000 = 1M output tokens x $20.00/1M = $20.00. Total: $36.00 per nightly run → ~$1,080/month

Batch (50% off): 4M x $2.00/1M = $8.00 input; 1M x $10.00/1M = $10.00 output. Total: $18.00 per run → ~$540/month

Same model, same evals, $540/month saved — for changing when the work runs, not what the work is. Note this stacks with other optimizations: batch requests with well-structured cached prefixes get both discounts.

The Latency Tradeoff, Honestly

The SLA is typically 24 hours. In practice providers often return results sooner, but architect for the SLA, not the typical.

Fits batch: nightly eval suites, dataset labeling and backfills, bulk document processing, embedding generation, offline classification, regression tests, report generation. Anything where the consumer is a pipeline, a dashboard, or tomorrow's standup.

Doesn't fit: chat, interactive agents, anything with a human waiting, real-time moderation, request-response APIs, multi-step agent loops.

Many "real-time" workloads aren't: audit your async jobs — teams routinely find 30–50% of their API spend is on work that could wait hours with zero user impact.

The Migration Is Smaller Than You Think

1. Collect requests instead of sending them — write them to a JSONL file. 2. Submit and poll — one submit call, one status poll. 3. Process results the same way — the response format matches the real-time API. Most teams do this in a day for a single pipeline.

Stacking Discounts: Batch Plus Everything Else

  • Batch + caching: both the ~50% batch discount and the ~90% cached-input discount apply.
  • Batch + smaller models: batching GPT-6.1 Sol ($2/$10 → $1/$5 batched) for bulk work is routinely 70–80% cheaper than the naive real-time flagship setup.
  • Batch + off-peak scheduling: submit in the evening, collect by morning standup.

Finding Your Batch Candidates

List every automated caller — cron jobs, pipelines, background workers. For each, ask: who reads the output, and when? If the answer is "a dashboard tomorrow" or "a data warehouse," it's a batch candidate. Rank by spend, migrate the top two first.

The Bottom Line

The Batch API is a 50% discount for the cost of patience, and most teams are leaving it on the table because their async work is accidentally running on the real-time API. Audit your jobs, find everything that doesn't need an answer in seconds, and move it.

Popular posts from this blog

Argon vs Wonder: Which AI Should Developers Build On?

Image
Last updated: October 2026 Two flagship models, two very different price tags — and one of the prices is about to double. If you're choosing a foundation model to build on right now, the decision isn't just about capability. It's about timing. Here's the honest comparison: what each model costs, what the benchmarks actually say, and which one to pick depending on what you're optimizing for. The Pricing: Read the Dates Gemini 4 Argon is on introductory pricing: $2.00 per 1M input tokens / $10.00 per 1M output tokens . That intro rate runs through November 30, 2026 . From December 1 , the price doubles to $4.00 / $20.00 . Muse Wonder is priced at $4.00 / $20.00 per 1M tokens — no intro period, no scheduled change. So today, Argon costs half what Wonder does. On December 1, they cost exactly the same. That convergence matters more than it looks. Every cost comparison you run today has a 6-week expiry date. If you're doing the math — and you should be — do...
Read more

GPT-5.6 API Pricing: The November 21 Deadline

Image
Last updated: October 2026 If you're running GPT-5.6 through the API right now, you're on a promotional price. And that promotion has an expiry date: November 21, 2026 . OpenAI's pricing page states the current rate is "available at least through November 21, 2026." Read that carefully. "At least through" — not "until." The promo could extend. It could also end exactly on that date. OpenAI has published no pricing for what comes after. This post lays out exactly what the pricing is today, what it costs on a realistic workload, and how to budget for the unknown that follows November 21. The Current Pricing, Exactly GPT-5.6 is available in several variants. The one most developers are using — the flagship reasoning model, referred to here as GPT-5.6 Sol — is currently priced at: Input: $4.00 per 1M tokens Cached input: $0.40 per 1M tokens Output: $20.00 per 1M tokens That is promotional pricing. Before the promo, announced August 21, ...
Read more

Prompt Caching: The 90% Discount You're Missing

Last updated: October 2026 There is a 90% discount sitting in your API bill, and most developers are either not using it or using it in a way that doesn't work. Prompt caching — paying a tenth of the input price for tokens the provider has already seen — is the highest-leverage cost optimization available on modern LLM APIs. Here's how it actually works and how to get the savings. How Prompt Caching Actually Works The idea is simple. When you send a request, the provider processes your input tokens through the model — that's the expensive computation you're paying for. But if your next request starts with the exact same tokens — the same system prompt, the same repo context, the same few-shot examples — the provider doesn't need to recompute all of that. It kept the intermediate results cached. So it charges you a fraction of the price for those repeated tokens. The key phrase is stable prefix . Caching works from the start of your prompt forward: the provider ...
Read more