Prompt Sensitivity Protocol

Question to Test

The prompt-sensitivity question is narrow: if every task input stays the same, does changing the instruction wording, order, or markup change the answer in a way your evaluator would care about? This is not a general prompt-engineering tutorial; broad advice belongs on Claude Helps. Curious Claude keeps the scope to experiment design.

Begin with one production-like task and one measurable output standard. Examples: extraction must return every named party from a contract; summarization must include all material risks; classification must use exactly one allowed label. The standard is written before any prompt variants are tried.

Variables and Controls

Hold the model, temperature or sampling settings, tool availability, source document, and output schema constant. Change one prompt property at a time: heading names, XML tags, example count, instruction order, or the presence of a brief rubric. If two properties move together, the result can show that a bundle helped, but it cannot say which property mattered.

For each variant, record the exact prompt text, model identifier, date, account surface, and any platform feature used. The September 2026 generally available models are Claude Fable 5.1 (claude-fable-5-1), Claude Opus 5 (claude-opus-5), Claude Sonnet 5 (claude-sonnet-5), and Claude Haiku 4.5 (claude-haiku-4-5). Anthropic documents current model families and platform surfaces in the models overview; experiments should cite that page or the relevant release note when model behavior is part of the claim.

Effort is a control, and it is the one most often left uncontrolled. Fable 5.1 runs adaptive thinking by default, and Anthropic documents its default effort as high in Claude Code but medium on Claude.ai and Cowork. A wording change tested in Claude Code and then confirmed in the web app has changed two things. Fix the effort level explicitly, or record the surface and treat each surface as a separate condition. The model-under-test fields list what the card needs.

• Control prompt: the existing production or baseline instruction.
• Variant A: a minimal wording change.
• Variant B: a structural change, such as XML sections or a numbered rubric.
• Fixed model condition: one model ID, one effort setting, one surface across all variants.
• Blind scoring: hide variant names from the scorer when human review is involved.

Runbook

Create at least ten fixtures before looking at outputs. The fixtures should represent normal cases, edge cases, and one intentionally boring case. A boring case matters because a prompt that only shines on adversarial examples may add cost or verbosity to routine work.

Run each fixture against every prompt variant in the same session style. If the surface has conversation memory or hidden state, reset it between runs. Store raw outputs exactly as returned. A clean result table has rows for fixture, prompt variant, model, timestamp, pass/fail, rubric notes, and reviewer initials or automated score version.

Scoring

A variant wins only if it improves the predeclared metric without failing a guardrail. Guardrails can include output length, citation presence, JSON validity, refusal behavior, or latency budget. Record ties honestly; a non-result is valuable because it prevents unnecessary prompt churn.

If a model upgrade changes the result, do not backfill a conclusion. Create a new dated row in the experiment ledger. Two releases landed inside a single quarter this year — Opus 5 on 2026-07-24 and Fable 5.1 on 2026-09-01 — so a prompt-sensitivity result from the spring describes a model that is no longer the one your users reach. A result from one model generation should not be silently treated as a property of all Claude models.

Common Failure Modes

The most common false positive is judging a single eloquent answer as proof. The second is using the same examples in the prompt and the test fixture, which rewards memorization of the example pattern. The third is changing the model, prompt, and scoring standard together, then calling the whole bundle a prompt improvement.

When the question turns into broader prompt-writing advice, link out. When it turns into benchmark reporting, link to Claude Reports. This page owns the repeatable method, not the league table.