Udayan Kumar My public notes

Do I even need a big LLM for this?

18 Mar 2026

The question that started this

I was looking at my LLM calls and had a thought: most of these calls are doing simple stuff. Classify this. Tag that. Extract a few fields. Return a label.

Do I really need a frontier model for that?

So I did the math. A sentiment classification pipeline running 500k calls per month on Claude Sonnet 4 with 100-token inputs and 20-token outputs costs roughly $900/mo. A fine-tuned 4B model on a single rented A100? About $90/mo — after a one-time training cost under $5.

That’s 10x. It got me curious enough to dig in.

This post is a record of that rabbit hole: when small language models actually make sense, when they don’t, and a calculator I built so you can test it with your own numbers.

A note on scope: Everything here is about text-only NLU/NLP tasks — classification, extraction, tagging, sentiment, routing. Not image generation, not vision, not open-ended reasoning.

First check: does this even apply to me?

Before I went further, I asked myself four questions. If you’re considering the same thing, these are worth running through:

  • Is the task narrow? Classification, extraction, structured output — not open-ended generation or multi-step reasoning.
  • Is the output well-defined? A label, a short phrase, a JSON blob, a structured extraction — not free-form prose.
  • Is the volume high enough? The calculator below will tell you exactly where the break-even is, but generally: the more calls you make, the faster it pays off.
  • Is 90–95% accuracy good enough? Even though well trained task-specific SLM often beat the LLMs, lets not bank on it for our project.

If any of those are a no, this probably isn’t the right path. A cheaper LLM tier might be the better move (more on that below).

The calculator I built

I have built a calculator that compares LLM API costs against running a fine-tuned SLM on rented GPU infrastructure. You plug in your workload, it shows the break-even.

Where to get your numbers:

  • Token counts: Check your LLM provider’s usage dashboard, or log a sample of 1,000 calls and average them.
  • Monthly volume: Your billing page or API call logs for the last 3 months — use the median.
  • Training samples: 1,000–10,000 labeled examples is a good starting range. Diminishing returns kick in fast for narrow tasks. You have the option to put the cost of generating labeled training data.

Brean Even Calls

This number tell you how many LLM calls can cover the cost of training.

The kinds of tasks where this makes sense

As I explored this, a pattern kept showing up. These are the workloads where a fine-tuned SLM seems to hold its own:

Customer support & operations

  • Per-message sentiment analysis in chat or support systems
  • Support ticket routing from short problem descriptions
  • Call center transcript snippets: intent or urgency detection

Content & media

  • Short-form content labeling for headlines, tweets, or posts
  • Highlightable features from real-estate listings
  • Feedback survey classification into themes

Data pipelines & CRM

  • E-commerce review tagging: shipping issues, quality complaints, feature mentions
  • Entity extraction for CRM from brief sales notes
  • PII detection and redaction flagging

The common thread: narrow task, well-defined output, repetitive, and high throughput.

The costs the calculator doesn’t show

The calculator only covers infrastructure and API fees. But that’s not the full picture — here’s what I had to think about on top of the numbers:

Upfront effort

  • Data preparation: 1–3 days. Labeling, schema design, edge-case collection. Goes much faster if you bootstrap labels with a big LLM first.
  • Training and evaluation: 1–2 days of compute; about a week of my time for experimentation and quality checks.
  • Deployment: 1–2 weeks for serving, monitoring, and CI/CD. Faster if you already use vLLM, TGI, or a managed inference platform.

Ongoing effort

  • Monitoring: Tracking accuracy drift, latency, edge-case failures. A few hours per week.
  • Retraining: Roughly quarterly as data distribution shifts. A few days each cycle.
  • Fallback: I’d keep the LLM API key active for the first few months and route low-confidence predictions there as a safety net.

Skills needed

  • Comfort with Python, HuggingFace Transformers, and LoRA/PEFT fine-tuning.
  • Familiarity with GPU instances and basic model serving.
  • No ML PhD required — LoRA fine-tuning for classification is well-documented and approachable.

But wait — do I even need to fine-tune anything?

Before going down the SLM path, I made myself check the simpler options first:

Alternative When it works When it doesn’t
Cheaper LLM tier (e.g., Claude Haiku, GPT-5 mini) Quality holds for your task; volume is moderate Still expensive at 1M+ calls/mo
Prompt caching Many requests share the same system prompt or context prefix High input diversity; cache hit rate is low
Batch API Latency is not critical (async processing, nightly jobs) Real-time or user-facing latency requirements
Negotiated enterprise pricing You spend >$10k/mo with one provider You are multi-provider or pre-revenue

If one of those closes the gap, great — less work. If not, that’s when fine-tuning starts to look worth it.

My go/no-go checklist

Signal Go No-go
Break-even in the calculator Under 3 months Over 12 months
Task complexity Single-step classification or extraction Multi-turn reasoning, open-ended generation
Output structure Constrained: labels, JSON, short extractions Free-form prose or creative text
Monthly volume Break-even under 3 months at your volume Break-even over 12 months
Accuracy tolerance 90–95% is fine, or human review exists Every error is a P0
ML comfort I’ve done some fine-tuning or am willing to learn No interest in touching model weights

Mostly “Go”? Worth a proof-of-concept. Mixed? Start with a cheaper LLM tier and revisit when volume grows.

What I’d do if the numbers check out

Here’s the rough playbook I’d follow:

1. Build a training dataset (Day 1–3)

If you’re already running this task in production with an LLM, you have real inputs and real outputs sitting in your logs. That’s most of the work done.

  • Sample from production logs. Pull 1,000–10,000 real input/output pairs. This is your training set.
  • Define a strict schema. Standardize the output format (labels, JSON keys, short phrases) if it isn’t already.
  • Spot-check labels. Audit a few hundred examples by hand. Fix obvious errors from the LLM. If you’re bootstrapping from scratch, use a strong LLM to draft labels and then correct a subset.
  • Grab edge cases. Pull any examples your current pipeline has flagged, retried, or gotten wrong.
  • Hold out 10%. Reserve it for evaluation.

2. Fine-tune and evaluate (Day 3–5)

  • Pick a model in the 1B–4B range for most classification tasks; go up to 8B if extraction complexity demands it.
  • Use LoRA (rank 16–64 is a reasonable start) to keep training fast and memory-light.
  • Evaluate on the holdout set. Compare accuracy, latency, and edge-case performance against the LLM baseline.

3. Deploy and monitor (Week 2–3)

  • Easiest path: Server less Upload your adapter to a hosted platform (Fireworks, Together, Modal, AWS Bedrock custom models) and get an API endpoint back. No GPUs to manage, and the cost is still a fraction of a frontier LLM.
  • Self-host/Rent if you need to. If data can’t leave your network, serve with vLLM, TGI, or similar on a single GPU.
  • Shadow-run alongside the LLM API for 1–2 weeks. Route a percentage of traffic to the SLM and compare.
  • Set up alerts for accuracy drops, latency spikes, error rate increases.
  • Keep the LLM API as a fallback. Route low-confidence predictions there until trust is built.

Timeline

If you already have production data, you can go from logs to a fine-tuned model in under a week. Shadow mode runs for 1–2 weeks to build confidence. Full cutover in 3–4 weeks total.

Where I landed

SLMs aren’t worse models — they’re right-sized ones. For narrow, high-volume NLP work, the cost gap is too big to ignore, and the engineering lift is surprisingly manageable if you already work with APIs and cloud infrastructure.

I’m still exploring this, and I’ll update the post as I learn more. If you try the calculator and the break-even looks compelling — say under 3 months — it’s probably worth a few days of effort: pull 1,000 examples from your production logs, fine-tune, and compare.

Calculator limitations

  • Estimates are directional; real costs depend on batching, caching, latency targets, and GPU utilization.
  • GPU pricing and availability are volatile; the calculator assumes steady hourly rates.
  • Token counts are averages; long-tail inputs can skew costs upward.
  • Quality, safety, and error-handling costs (human review, retries, fallbacks) are not included.
  • Engineering time for data prep, fine-tuning, deployment, and monitoring is excluded.
  • Vendor pricing and model availability can change; live rate refreshes may lag official updates.
  • Costs for renting GPUs and self hosting inference have a wide range. Really depends on your situation. We assume you have several such models and no GPU cycles go unused.

References

Related Posts