I recently built a side project called Ask Mandi which lets you ask questions about mandi (Indian agricultural market) prices in plain English. So you can ask stuff like "where can I find some potatoes in Mumbai?" and it’ll write the SQL, query the database and give you an answer.

Everything felt great until I added a tiny token usage badge under each answer that shows how much the query cost.

That's when I realised... I fucked up!

How much could an apple cost?

I asked "where are apples cheapest today?" and the token badge very casually showed me I'd just burned through 3000 tokens.

Three. Thousand. Tokens.

To find "cheap" apples. The irony wasn't lost on me.

My first thought was that vibe coding had finally caught up with me. Maybe the coding agent had created a loop somewhere or done something stupid.

I knew what would fix this.

I got cheap

With the models. I got cheap with the models.

My theory was I could use a dumber model with better prompts and get basically the same result. Like maybe gpt-5-mini isn't as smart but if I'm super specific about what I want surely it'll manage some SQL?

So I switched from gpt-5.1 (which... is not cheap) to gpt-5-mini for SQL and gpt-5-nano for summaries.

The cost went down but the latency... boy was it high.

Like Seth Rogen in Pineapple Express high.

At first I blamed Supabase. Then MCP. Then the network. Eventually I figured out it was just the gpt-5 models being unusably slow for anything real-time, even with reasoning_effort: "low".

So I switched again, this time to models I thought I'd never have to touch again:

gpt-4.1-mini for SQL

gpt-4.1-nano for summaries

And to my surprise the app felt fast for the first time. 5–7 seconds TTFT instead of the 30s+ I was getting earlier. Much better.

But deep down I knew this was just a band aid fix. I was procrastinating on the real problem since I knew it’d need a lot of trial and error to get right.

What the hell was happening?

So here's what I had built:

question → LLM writes SQL → database → LLM summarises results → answer

Pretty simple. Except my SQL builder had a massive 1500+ token prompt so even for the simplest questions I'd already burned through 1500 tokens before it had generated a single one.

Then there was the SQL itself. My generator had developed a particular fondness for SELECT * so it would fetch a bunch of rows and columns it didn't actually need and dump all of it into the summariser.

So even when the final answer was just:

"Apples are cheapest at Patti APMC in Punjab at ₹45/kg."

One sentence. One row would've been enough.

Instead I was fetching a bunch of unnecessary data just to have another model throw most of it away.

I wasn't paying too much per token. I was using way too many tokens.

Time to get shredded!

I finally accepted that prompt bulking season was over. It was time to get shredded.

So I got aggressive with the SQL prompt rules. Like actually aggressive 😡:

Default to latest date only

Return 1–3 rows unless the user specifically asks for more

For trends, return daily aggregates (avg/min/max) instead of raw rows

Never SELECT *

Only fetch columns that actually matter for the answer

The goal was simple: return the smallest possible result set needed to answer the question.

If a question can be answered by fetching 3 rows, don't fetch 300.

JSON? In this economy?

JSON is great until you're feeding a lot of it to an LLM and paying for every token. All those keys, quotes, braces and repeated structure add up fast.

One thing I actually did right from the start was use TOON. I'd heard a lot about it on Twitter and finally had a project where I could use it. It's basically a much more compact way of representing the kind of structured data I'd otherwise be sending as JSON.

I converted my query results to TOON before sending them to the summariser and was blown away to see 50–55% fewer tokens with literally zero difference in answer quality.

Turns out all those curly brackets aren't free.

Without TOON my summarisation costs would've been even more insane, so shoutout to Johann Schopplich for creating it.

The best model is no model

There was one more obvious optimization. Ask Mandi's underlying data only refreshes once a day at 3:30pm IST.

So if you ask "where are apples cheapest today?" at 10am and someone else asks the exact same thing at 11am, why the hell am I paying a model to figure it out again? The data hasn't changed.

So I cache answers until the next data refresh, at which point the cache expires naturally.

Repeat questions now cost basically nothing and they're instant.

Turns out the fastest, cheapest LLM call is the one you don't make.

TL;DR

I was spending ~3000 tokens to answer "where are apples cheapest today?"

Which is absolutely insane.

My first instinct was to use cheaper models. That helped but it turned out I was optimizing the wrong thing. The bigger problem was that I was making the models process way more data than they actually needed.

The fix:

Use cheaper, faster models where you don't need that much intelligence

Make SQL return the smallest possible result set

Use TOON instead of dumping raw JSON into the summariser

Don't pay twice for the same question

The biggest lesson for me was that optimizing an LLM app isn't always about finding a cheaper LLM.

If your token bill looks insane, before spending three hours comparing model pricing tables, check how much shit you're feeding the model.

Edit (Aug ‘26): I got even cheaper

I recently switched Ask Mandi from OpenAI to Groq using GPT-OSS 20B and it works beautifully. It's stupid fast and Groq gives you a pretty generous free tier.

Which means Ask Mandi now costs me exactly $0 in LLM usage.

So after writing an entire post about making LLM calls cheaper I somehow ended up not paying for them at all.