The Test Kitchen · Field notes
Five ways a prompt goes off
Each failure below has a cause, a question to ask yourself, a repair, and before-and-after recipes from the pantry that demonstrate it.
01. Missing context
The model fills gaps with the statistically typical case, which is rarely yours. A request with no audience, purpose or constraints produces a fluent average.
Ask yourself: Who is this for, what will they do with it, and what do I know that the model cannot?
Repair: Add two or three sentences of situation. Paste the actual error, document or data instead of describing it.
Recipes that show it (12)
- Compare two policy frameworks (Research) — before: “Which is better, the EU AI Act or NIST AI RMF?”
- Interview question bank (Research) — before: “Give me interview questions about remote work.”
- Fix a bug with the actual error (Coding) — before: “My code doesn't work, fix it.”
- Code review against a checklist (Coding) — before: “Review my pull request.”
- Refactor without behaviour change (Coding) — before: “Make this code cleaner.”
- Prepare for a supplier negotiation (Negotiation) — before: “How do I negotiate with my supplier?”
- Mediating a team disagreement (Negotiation) — before: “How do I settle an argument between two colleagues?”
- Long document, specific question (Summarising) — before: “TL;DR this 80-page report.”
- Thread summary that keeps disagreement (Summarising) — before: “Summarise the discussion.”
- Pre-mortem a plan (Analysis) — before: “Is my plan good?”
- Compare options with criteria (Analysis) — before: “Should I use Postgres or MongoDB?”
- Iterating a draft (Creative writing) — before: “Make this better.”
02. Invented sources (confabulation)
Language models can produce plausible citations, statistics and quotes that do not exist. NIST AI 600-1 lists confabulation among its generative-AI risks; OWASP's 2025 list includes misinformation (LLM09).
Ask yourself: Could I check every factual claim in the answer, and did I give the model a way to say “I don't know”?
Repair: Ground answers in material you supply, require checkable references, and give an explicit UNVERIFIED or 'not stated' escape hatch. Then actually check.
Recipes that show it (7)
- Literature scan with honest gaps (Research) — before: “Tell me everything about microplastics in the brain.”
- Fact-check a viral claim (Research) — before: “Is it true that we only use 10% of our brain?”
- Salary counter-offer (Negotiation) — before: “Write an email asking for more money.”
- Decline politely and keep the door open (Negotiation) — before: “Say no to this client.”
- Meeting notes to decisions (Summarising) — before: “Summarise this meeting.”
- Interpret a small dataset (Analysis) — before: “What does this data say?”
- Root cause from logs (Analysis) — before: “Why did the server crash?”
03. Over-constraint
Piling on absolute rules (always, never, exactly) creates contradictions and crowds out the goal. The model satisfies the easiest rules and quietly drops the important one.
Ask yourself: If two of my rules conflicted, which would I want to win?
Repair: Keep rules that protect the outcome; convert style wishes into one example. Rank priorities explicitly.
Recipes that show it (6)
- Write a function with tests (Coding) — before: “Write a function to validate emails.”
- Role-play the other side (Negotiation) — before: “Pretend to be my landlord.”
- Paper abstract for a lay reader (Summarising) — before: “Explain this paper simply.”
- Short story with real constraints (Creative writing) — before: “Write a story about a robot.”
- Brand voice with examples (Creative writing) — before: “Write an Instagram caption for my perfume.”
- Worldbuilding consistency check (Creative writing) — before: “Is my fantasy world consistent?”
04. Leakage
Prompts get logged, shared, pasted into tools and sometimes reflected back. OWASP's 2025 list names sensitive information disclosure (LLM02) and system prompt leakage (LLM07).
Ask yourself: Would I be comfortable if this whole prompt appeared in a screenshot?
Repair: Redact before sending. Never put keys or passwords in prompts. Do not rely on a system prompt staying secret.
Recipes that show it (2)
- Summarise for leakage-safe sharing (Summarising) — before: “Summarise this customer complaint for the team.”
- Poem in a form, not a copy (Creative writing) — before: “Write a poem like a famous poet.”
05. Prompt injection
Text the model reads (a web page, email, PDF) can contain instructions that override yours. OWASP ranks prompt injection first (LLM01:2025) and notes there may be no fool-proof prevention, only mitigations.
Ask yourself: Does my prompt mix my instructions with content written by someone else?
Repair: Delimit untrusted content, state that it is data, limit what tools can do, and require human approval for consequential actions. Wording alone is not a security control.
Recipes that show it (3)
- Source-grounded answer from a pasted document (Research) — before: “Summarise this and tell me what experts think.”
- Agent with tool access (Coding) — before: “Read my inbox and handle anything urgent.”
- Risk register for an AI feature (Analysis) — before: “What could go wrong with our chatbot?”