Today's AI Tip
I hid 16 edits in two contract drafts. GPT-5.5 found all 16.
Small words are easy to miss
When a contract comes back "with a few edits," most of us skim it, or ask "What changed?" and get a summary we can't check word for word. Small words like "and" and "or" are easy to miss.
Ask for old and new wording, quoted exactly
Tested in GPT-5.5 (OpenAI API), Oct 2026
This prompt asks for every change, with old and new quotes and nothing guessed. Each version goes in its own labeled tag, which Anthropic's and OpenAI's prompting guides both suggest.
Source: Anthropic and OpenAI prompting guides, checked Oct 2026
16 planted edits, two rounds
The test was a fictional 9-clause agreement with two rounds of 8 planted edits: one round obvious, one round subtle, like "and" turned into "or."
It quoted the sneaky ones
Real output · GPT-5.5 (OpenAI API) · Oct 2026
In the subtle round, it caught "and" changed to "or" and a termination right narrowed from either party to Studio, each with exact old and new wording. (Showing 2 of 8 changes.)
The plain question caught them too
Both prompts found every edit, but only the structured prompt (tags plus "quote exactly") gave exact quotes you can check. In our test, all 6 runs caught every planted edit and invented none.
Where the answers got noisy
The plain answer paraphrased: it wrote "30 days’ written notice," but the contract says "30 days of written notice." In the obvious round, the tagged runs also listed clauses that were only renumbered after the deleted one.
Try it: Short, clean test; on long or messy files, also run Word's Review > Compare.
Source: Microsoft Support, Compare document differences, Oct 2026
Check it, then check privacy
AI can explain what an edit might mean, but it isn't legal review. Before uploading a real contract, remove names, account numbers and other personal details first, and check the tool's data policy or your employer's approved tools.
Try it: Ctrl+F (Cmd+F on Mac) three "new wording" quotes in the returned file.
Sources and assumptions
- Anthropic, Prompting best practices (Claude Platform Docs): Verbatim: 'When using multiple documents, wrap each document in <document> tags with <document_content> and <source> (and other metadata) subtags for clarity.' Also: 'XML tags help Claude parse complex prompts unambiguously, especially when your prompt mixes instructions, context, examples, and variable inputs.' (checked 2026-10-12)
- OpenAI, Prompt engineering guide (API docs): Verbatim: 'XML tags can help delineate where one piece of content (like a supporting document used for reference) begins and ends.' This supports using tags with GPT-5.5, the model actually tested. (checked 2026-10-12)
- Microsoft Support, Compare document differences using the legal blackline option: Word for Microsoft 365, Word 2024 and Word 2021: Review tab > Compare > Compare two versions of a document (legal blackline), pick the Original and Revised documents, OK. Word creates a new document showing the differences and leaves both originals unchanged. (checked 2026-10-12)
- Microsoft Support, Compare and merge two versions of a document: Word 2019 for Mac: Review tab > Compare > Compare Documents, pick the Original and Revised documents, OK. (checked 2026-10-12)
Assumptions:
- All runs used GPT-5.5 through the OpenAI API (engine/ai-test.js, labeled 'GPT-5.5 (OpenAI API)'), not the ChatGPT app. Claude runs were attempted for the same tests but failed in the research environment (the claude CLI errored even on 'Say ok.'), so the post makes no claim about Claude's output. The Claude docs are cited only for the tag structure.
- The agreement, both parties (Brightline Studio, Harbor Bakery Co.) and every edit are fictional and were written for this test.
- Scoring is against the 16 known planted edits: round 1 (obvious) = start date Mar 3→Mar 10 2027; hours 20→18; fee $2,400→$2,650; due 15→30 days; late fee 'Late payments incur a 1.5% monthly fee.' added; revisions two→one; 'Unused hours' clause deleted; 'will respond'→'will aim to respond'. Round 2 (subtle) = six→twelve months; 'and'→'or' copywriting; invoiced 1st→15th; 'Fees are non-refundable.' added; Payment and Revisions clauses swapped; 'business day'→'working day'; 'once paid in full'→'once all invoices are paid in full'; 'Either party'→'Studio' may terminate.
- 6 runs in total: per round, the tagged prompt twice and the plain 'What changed between these two?' (no tags) once. Every run caught all 8 edits of its round, and none reported a change that wasn't made. Renumbering rows (round 1 tagged runs) are true side effects of the deleted clause, not inventions. Some runs merged two edits in one row (e.g. invoice date + non-refundable), still naming both.
- The result comes from one short (9-clause), cleanly formatted document. It does not show that the prompt catches everything in long contracts, scanned PDFs, or files with tracked changes.
- The plain-prompt runs wrote '30 days’ written notice', but both versions say '30 days of written notice'. That is a paraphrase, not a changed fact.
- Run times: 2026-10-13T02:11–02:13 UTC (2026-10-12 local), so label outputs 'Oct 2026'.
The short version
- Give each version its own labeled tag.
- Ask for old and new wording quoted exactly.
- Spot-check a few quotes with Ctrl+F.
- On long contracts, also run Word's Compare.
