Skip to main content

Few-Shot and Chain-of-Thought, When They Earn Their Tokens

Few-shot and CoT are the two most over-applied prompting techniques of the last three years. Here is when each one pays for itself, when it does not, and how to tell the difference before you ship.

Updated 11 min readMarkdown
Part of seriesAI for Builders
Few-Shot and Chain-of-Thought, When They Earn Their Tokens

Half the "advanced prompting techniques" the internet tells you to use don't do what they say they do on the current generation of frontier models.

Few-shot prompting and chain-of-thought (CoT) are the two biggest examples. They were both genuine breakthroughs when they were published — few-shot in 2020 with GPT-3, CoT in 2022 with "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." The techniques worked, the papers were cited thousands of times, and a generation of prompt engineers absorbed them as gospel. Then the models got bigger, got post-trained with RLHF, and got reasoning modes, and a lot of the gains from these techniques quietly evaporated or moved around.

Today, both techniques still pay for themselves — but only for specific kinds of problems, and only if you apply them with a cost-per-token mindset. This post is the honest accounting of when they work, when they don't, and how to decide before you ship. We'll keep it concrete: real token costs, real failure cases, real before/after snippets.


Few-shot: the shape

"Few-shot prompting" means including a handful of input-output examples in your prompt, so the model can see the pattern you want before it attempts its own answer. The canonical shape:

# pip install anthropic
import os
from anthropic import Anthropic

client = Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])

EXAMPLES = [
    {
        "input": "My card was charged twice this month.",
        "output": "billing",
    },
    {
        "input": "The app crashes when I tap settings.",
        "output": "technical",
    },
    {
        "input": "How do I change my password?",
        "output": "account",
    },
]

def build_prompt(new_input: str) -> str:
    parts = ["Classify each message into one of: billing, technical, account."]
    for ex in EXAMPLES:
        parts.append(f"\nInput: {ex['input']}\nOutput: {ex['output']}")
    parts.append(f"\nInput: {new_input}\nOutput:")
    return "\n".join(parts)

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=20,
    temperature=0,
    messages=[{"role": "user", "content": build_prompt("I forgot my 2FA device.")}],
)
print(response.content[0].text)

Three examples, then the real query. The pattern is: "here's what I mean, now do one for me." Before 2023, this was a large gain over a plain instruction. Today, on a frontier instruction-tuned model with a clear schema, the gain is much smaller — sometimes zero.

Here's why. Post-trained models already know what "classify into one of three categories" means. You gave them structured output in B1.3, and constrained decoding makes sure the output shape is right without any examples. The model's prior on the task is already good.

So when does few-shot still pay?


When few-shot earns its tokens

1 · The task is non-obvious from the instruction alone

If the thing you want is hard to describe in one sentence but easy to show in three examples, few-shot wins. Classic cases:

  • Domain-specific tagging. "Tag the legal argument type" — the model knows general legal concepts but not your taxonomy. Three examples of "statute-based," "precedent-based," and "equity-based" reasoning will produce better tagging than a definition-heavy instruction.
  • House style. "Match the tone of our product's error messages" — the tone is specific and only visible in examples. Show three, the model matches. Describe it in words, you get generic polite text.
  • Specific output format. "Produce a bullet list in our unusual markdown dialect." Show two examples, done. Describing the dialect in text takes twice as many tokens and gets it wrong.

2 · You're on a smaller, weaker model

The smaller the model, the more few-shot helps. On a 7B local model, a well-chosen example is worth a page of instruction. On a frontier model, the gap closes. Rule of thumb: if you're running anything under 30B parameters, budget for few-shot in every non-trivial call. If you're running a frontier model, try without examples first and measure.

3 · The cost of examples is low relative to the call

If your call is already 4,000 tokens of context, adding 200 tokens of examples is a 5% cost bump. If your call is a 40-token classification, adding 200 tokens of examples is a 5x cost bump. The decision is about proportion, not absolute. Few-shot is cheap on long-context calls and expensive on short ones.

4 · You have failures you can learn from

Here's the best kind of few-shot: examples that are just past the edge of what the model gets wrong by default. You run the eval set without examples, find three cases where the model fails, and add those three failure-mode-turned-success cases to the prompt. These targeted examples are worth 10x a generic "here's what a good answer looks like" example, because they're teaching the model the specific mistake not to make.


When few-shot is expensive cargo-culting

1 · The instruction is already clear

"Translate English to Spanish." You don't need examples. The model was trained on billions of translation pairs. Your three examples are noise.

2 · The examples are off-distribution from the real query

You're running a customer-support classifier, so you grab the first three tickets from your test set and put them in the prompt. Problem: those three tickets are much easier than the average real ticket. The model happily matches the easy pattern and fails on harder ones. Off-distribution examples can be worse than no examples, because they create a misleading prior.

3 · You're doing schema enforcement

If you're already using structured output (which you should be — see B1.3), the schema enforces the output format. Few-shot examples that were there purely to teach "output as JSON" are now redundant. Delete them.

4 · Your examples are stale

Six months in, your examples are about features that no longer exist, or use phrasing that's no longer how users talk. The prompt still "works" on the eval set because the eval set is also stale. Production quality quietly degrades. Evals from B2.2 catch this. Without them, you won't know.


Chain-of-thought: the shape

"Chain-of-thought" means asking the model to produce its reasoning before its answer, on problems that benefit from multi-step thinking. The canonical trigger is a phrase like "Let's think step by step" or an explicit instruction to show the reasoning. The canonical payoff is on multi-step math or logic:

Question: A train leaves city A at 2pm traveling at 60 mph. Another train leaves city B,
which is 200 miles from A, at 3pm traveling at 40 mph towards A. At what time do they meet?

Without CoT: "4:30pm" (sometimes right, sometimes wrong)

With CoT: "At 3pm, train 1 has traveled 60 miles, so the remaining distance is 140 miles.
The combined closing speed is 60+40=100 mph. 140/100 = 1.4 hours. 1.4 hours after 3pm
is 4:24pm." (reliably right)

The 2022 paper showed that adding "Let's think step by step" alone gave large gains on arithmetic, commonsense, and symbolic reasoning. For a while, the phrase itself became the meme.

What actually happened in the intervening years is more interesting than the original result.


CoT in 2026: the three things you need to know

1 · Frontier models now do CoT implicitly

Modern frontier models are post-trained to reason step-by-step when the task warrants it, even without being asked. You write "solve this math problem" and the model naturally produces working before the answer. The "Let's think step by step" trigger is mostly redundant on top-tier models in 2026. Try without first; if the answer is right, save the tokens.

2 · Reasoning models do CoT invisibly

The new generation of reasoning models (Claude's extended thinking, OpenAI's o-series, Gemini's deep thinking) run internal chain-of-thought before producing an answer, and you don't see the reasoning — you just see the final output. You're paying for those reasoning tokens (sometimes a lot of them) but they don't show up in the response text. On these models, asking for CoT in the prompt is actively counter-productive: you're duplicating work the model is already doing.

3 · CoT still helps when you want to audit the reasoning

Here's where CoT still pays. If your feature needs to show the user why the model picked an answer — e.g. "here's our recommendation, here's the three reasons" — then you want visible reasoning in the output. In that case, ask for it explicitly and parse it out of the response. The "let's think step by step" part is for the product, not for the model.

# Useful CoT: you want the reasoning in the output
SYSTEM = (
    "You recommend meal plans. For each recommendation, first list "
    "the three factors that drove the choice, then give the recommendation. "
    "Use this format:\n"
    "Factors:\n- ...\n- ...\n- ...\nRecommendation: ..."
)

That's a CoT prompt with a purpose. The model writes the factors. The user sees them. The product is more trustworthy because the reasoning is visible. Worth every token.

Versus the useless version:

# Useless CoT: you're asking for reasoning you then throw away
SYSTEM = "You classify support tickets. Think step by step before answering."

You don't need the reasoning. You need the category. You're paying for tokens you're going to discard. Delete "think step by step" and save the money.


A practical decision tree

The default branch is "start plain": no few-shot, no CoT, write the instruction clearly, see what the model produces. If quality is bad, then decide which technique your failure mode asks for. Don't pre-load both techniques out of superstition — you'll pay for tokens you don't need and you won't know which technique did the work.


The one habit that beats both techniques

If you have to pick one thing to add to every prompt beyond a clear instruction, it is not few-shot and it is not CoT. It is a specific definition of "done". Example:

  • Bad: "Summarise the ticket."
  • Better: "Summarise the ticket in one sentence under 120 characters. Focus on the user's problem, not their tone. Do not include pleasantries or apologies."

That second version is five sentences longer and gives you a 3x quality improvement on most models, no few-shot required, no chain-of-thought required. Clarity about the output beats both techniques combined. If you're tempted to add few-shot because the output isn't what you want, first ask: could I have been more specific about the output? Usually the answer is yes.


Admit what breaks

  • Few-shot examples leak into the output. The model sometimes quotes phrases from your examples in its answer, especially at low temperature. If your examples contain product names, internal jargon, or placeholder text, it may show up in user-facing output. Fix: use examples that don't look like "bait" for mimicry, and check the output before shipping.
  • CoT reasoning is not audit-quality. A visible "chain of thought" looks like a real explanation, but it's still generated by the same next-token prediction. The model sometimes reasons plausibly and arrives at the wrong answer, or (worse) reasons wrong and arrives at the right answer by coincidence. Showing reasoning to users doesn't make the answer more correct — it makes the errors more legible.
  • "Let's think step by step" doesn't guarantee step-by-step thinking. The model may still shortcut. Measure, don't trust.
  • Few-shot examples age. They drift out of sync with your product, your users, and your eval set. Review quarterly.
  • CoT balloons token usage in loops. If you have an agent loop running CoT at every step, your bill is 5x what you'd pay for terse output. Use reasoning modes sparingly in inner loops.

What just changed in your code

  • Start plain. No few-shot, no CoT, just a clear instruction with a specific "done" definition. Measure quality with your eval set.
  • Add few-shot only when the task is hard to describe, when you're on a weak model, or when you have specific failure cases to teach against. Three targeted examples beat ten generic ones.
  • Delete "think step by step" from every prompt where you're throwing away the reasoning. It's a token cost without benefit on frontier models.
  • Ask for reasoning only when the product needs to show it. Reasoning for the model is already happening — you don't need to ask.
  • When output is bad, first improve the instruction, then reach for techniques. "Be more specific about what good looks like" is the free move.

Next post, B2.4, is the one I most wish every LLM app engineer read before shipping: prompt injection is your SQL injection. The security hole every team ignores until the first one on Twitter, with working exploits and defences that actually hold up.


Course navigation

⬅️ Previous📍 You are hereNext ➡️
⬅️ Previous
B2.2 · Prompts Are Code, Not Config
B2.3 of B6.4Next ➡️
B2.4 · Prompt Injection Is Your SQL Injection

📚 AI for Builders · Course Home — 28 posts, six modules.


Cover photo via Unsplash. This post is part of the AI for Builders series.

No comments yet. Be the first to comment.

More from this blog