Today's AI Tip
I asked AI for 5 survey questions. One asked two things.
The request most of us start with
Tested in GPT-5.5 (OpenAI API), Oct 2026
Maple and Rye, a fictional café, launched a new lunch menu three weeks ago and raised lunch prices by about $1.50. We gave the AI that context plus this request.
What the plain prompt wrote
Real output · GPT-5.5 (OpenAI API) · Oct 2026
In our test, both runs asked about taste and quality in one question. The satisfaction question also had no answer for someone who hasn't tried the menu.
Three problems, four rules
A "Poor" on a two-part question doesn't say which part. Two of three price answers (good value, slightly high) accept the new price. Some customers have no answer. A fourth rule adds an open question.
Source: Pew Research Center; Case Western Reserve University, Office of Institutional Research
Give it the rules up front
Tested in GPT-5.5 (OpenAI API), Oct 2026
Four rules in plain words, plus a request to check its own work. We gave it the same café context first; add two lines about your own business.
What the rules changed
Real output · GPT-5.5 (OpenAI API) · Oct 2026
In one run, the taste question stands alone, the prices run from too low to too high, and a not tried answer is offered.
Its check passed a mismatched question
Real output · GPT-5.5 (OpenAI API) · Oct 2026
In the other run, the question asks about taste but the answers measure satisfaction. None of the four rules covered this, so its own rule check said Yes to all five questions.
Source: Case Western Reserve University, Office of Institutional Research
Then read every question yourself
Ask of each question: does it ask one thing, do the answers match it, and could every customer answer it honestly?
Try it: Run the four-rule prompt for your next survey, then ask the three questions above of every item.
Sources and assumptions
- Pew Research Center, Writing Survey Questions: Questions that ask about more than one concept ('double-barreled questions') 'are difficult for respondents to answer and often lead to responses that are difficult to interpret'; closed-ended questions 'should include all reasonable responses (i.e., the list of options is exhaustive)'; word choice can bias answers. (checked 2026-10-22)
- Case Western Reserve University, Office of Institutional Research, Basic Tips for Choosing Scales: 'Agreement, quality and satisfaction scales should be balanced with equally positive and negative options'; 'The stems of questions and the scales should match.' (checked 2026-10-22)
- Digital.gov, Plain language guide: Organize (used only to reject idea i-2026-10-18-12): 'Start by stating your purpose and the bottom line.' / 'Put the most important information at the beginning.' (checked 2026-10-22)
Assumptions:
- Maple and Rye, its menu and its $1.50 price rise are fictional sample material written for the test.
- All outputs are from GPT-5.5 via the OpenAI API (label 'GPT-5.5 (OpenAI API)', Oct 2026), not the ChatGPT app. Claude could not be tested this session: the claude -p test runner failed, so Claude must not be named as tested anywhere.
- Results come from 2 runs of the plain prompt, 2 runs of the 3-rule prompt and 2 runs of the 4-rule prompt. Other runs and models may differ, so slides should say 'in our test', not 'always'.
- tests.json also holds unrelated entries (failed Claude 'before'/'after', before-gpt/after-gpt, before-gpt-2/after-gpt-2, plain-1..3, fixed-1..2) from rejected angles. The draft uses only the survey-* entries.
- 'Two of the three price opinions accept the new price' counts 'Still a good value' and 'Slightly high but reasonable' as accepting and 'Too expensive' as the one negative; 'Not sure' is not an opinion.
The short version
- Give the AI four rules first.
- One thing per question.
- Answers must fit everyone.
- The AI's check only checks your rules.
- Read every question yourself.
