Today's AI Tip
Sorting customer emails with AI? Show it three labeled examples first.
The obvious first try
Tested in GPT-5.5 (OpenAI API), Oct 2026
The obvious first try: name the labels and paste in the emails. We ran exactly that on 12 fictional support emails.
Labels don't say your rules
Our fictional team's rule: if the app caused it, it's a bug, even with money involved. Email 4: a crash during an upgrade, now $40 instead of $20. Email 9: constant logouts, so the trial renewed.
Without examples, it broke our rule
Real output · GPT-5.5 (OpenAI API) · Oct 2026
It scored 10 of 12 by our rule. It sent emails 4 and 9 to billing. That's a fair everyday reading, but the model had no way to know our rule.
The move: add three examples
Tested in GPT-5.5 (OpenAI API), Oct 2026
Same request plus three labeled examples, one per label. One is an edge case: a freeze that double-charged is a bug.
Source: Anthropic, Prompting best practices, Oct 2026
Better, but not perfect
Real output · GPT-5.5 (OpenAI API) · Oct 2026
Emails 4 and 9 became bug, and the score rose to 11 of 12. But email 11 flipped the other way: it was right without examples, and now the word "broken" pulled a select-all request to bug.
Emails that fit nowhere
Real output · GPT-5.5 (OpenAI API) · Oct 2026
In this run, a thank-you and a job inquiry got forced into bug and feature request. With an 'other' option added, both came back 'other', in 2 of 2 runs.
Try it: Add it right after the first sentence: If none fits, write other.
Pick examples that cause arguments
Choose examples from the emails your team argues about, not the easy ones, and add an other bucket. Spot-check a few each time, since results vary run to run.
Try it: paste three emails you've already sorted, each with its label, above today's batch.
Sources and assumptions
- Anthropic, Prompting best practices (Claude Platform Docs), section 'Use examples effectively': Verbatim: 'A few well-crafted examples (known as few-shot or multishot prompting) improve accuracy and consistency.' Examples should be Relevant ('Mirror your actual use case closely'), Diverse ('Cover edge cases and vary enough that Claude doesn't pick up unintended patterns'), Structured ('Wrap examples in <example> tags (multiple examples in <examples> tags)'). Tip: 'Include 3–5 examples for best results.' (checked 2026-10-12)
- OpenAI API docs, Prompt engineering guide, section 'Few-shot learning': Verbatim: 'Few-shot learning lets you steer a large language model toward a new task by including a handful of input/output examples in the prompt, rather than fine-tuning the model.' The guide recommends diverse examples. (checked 2026-10-12)
Assumptions:
- All 14 emails are fictional and were written for this test. No real customer data was used.
- The answer key follows a fictional team rule: if the software caused the problem, it's a bug, even when money is involved. A missing capability is a feature request even when the customer calls it 'broken'. Problems with charges, invoices or cancellations not caused by the app are billing. Key: 1 billing, 2 bug, 3 feature request, 4 bug, 5 feature request, 6 bug, 7 billing, 8 bug, 9 bug, 10 feature request, 11 feature request, 12 bug. By everyday logic, 4 and 9 could fairly be called billing. The slides must say the no-examples run broke the team's rule, not that it was 'wrong' in general.
- All quoted runs used GPT-5.5 through the OpenAI API (label 'GPT-5.5 (OpenAI API)', Oct 2026), not the ChatGPT app. The Claude runs failed to execute in this research session (two entries in tests.json with output null: ids 'zero-shot' and the first 'three-examples'). Claude was not tested, so the post must not claim a Claude result. Anthropic's docs are cited as general guidance on examples only.
- tests.json has two entries with id 'three-examples'. The first is the failed Claude run (output null). The second is the GPT-5.5 run to quote. Use the one with an output.
- Results vary from run to run. Earlier GPT-5.5 runs of the same prompts during research (scratch file tests-gpt.json, not attached) scored 10/12 with no examples twice, 12/12 with three examples twice, and labeled the off-topic emails 'other' even without being told to. Slides should quote only the attached runs (no-examples 10/12, three-examples 11/12, off-topic forced into bug/feature request). They may say results changed between runs, which is why spot-checks matter.
- This is one small test of 12 emails, not a benchmark. The slides must not claim a general accuracy figure.
The short version
- Labels alone don't teach your team's rules
- Three examples fixed two misses but broke one
- Add an 'other' option for emails that fit nowhere
- Spot-check, because results vary run to run
