Why Is My AI Assistant Getting Worse? How to Diagnose a Sudden AI Model Quality Drop

In April 2026, Anthropic published a postmortem on Claude Code quality reports. For weeks, users had been saying the assistant felt worse: shallower reasoning, forgotten context, sloppier code. Many assumed the model had been quietly downgraded. The investigation found three separate changes, each in a different part of the product: the default reasoning effort, how session state was cached, and the system instructions the tool sent along with every request. They hit different users at different times and overlapped enough to feel like one general decline.

That is the most useful thing to know about "my AI tool got worse." What you use is a stack: a model, a serving layer, a client application, a system prompt, default settings, and the session you have been building up for the last two hours. Any of those layers can regress, and most of them can change without anyone telling you. This guide gives you a way to find which one moved.

What can actually change when an AI tool gets worse

SEO teams learned this discipline long ago. Seer Interactive's guide to diagnosing organic traffic declines works through seasonality, technical errors, ranking shifts, and competition one at a time, because "Google changed something" covers too much to be a diagnosis. AI tools need the same treatment.

Here are the layers, ordered roughly by how often they change without notice:

Most people check this table from the bottom up and blame the model first. The diagnostic order should run top-down, because the top rows are cheaper to test and more likely to be the cause.

Your own changes count too. If your codebase doubled, your CLAUDE.md grew by 400 lines, or you moved from greenfield features to debugging a legacy service, the task changed. We covered why those files deserve version control in Agent Instruction Files Are Source Code.

What the Claude Code postmortem teaches about product-layer regressions

The postmortem is worth reading in full. It is one of the few public, detailed accounts of how an AI product got worse while the model underneath stayed the same. Its three failure classes map onto problems any AI tool can have.

Reasoning effort defaults. A lower default effort means less thinking per request. That trade is reasonable for latency and cost on simple tasks, and it hurts on hard ones. Users who never touched the setting got a different product. Those who set effort explicitly may have noticed nothing. We argued in Thinking Budgets Are a Product Decision that effort should be tuned per task, and this incident shows the other side: a default someone else picks is a product decision made on your behalf.

Caching and session state. The postmortem describes a change in how earlier reasoning was kept across a session that ended up dropping context the model needed. From the outside it looks like forgetfulness: the assistant loses track of decisions it made ten minutes ago, or redoes work. Users experience it as "the model got dumber," but the model is reasoning normally over a degraded history. If you want the mechanics of how cached prefixes and session state work, see What Is Prompt Caching?.

System instructions. A change to the hidden system prompt, meant to adjust one behavior such as verbosity, can shift others in ways nobody intended. Telling a model to be brief can also make it reason less on the page, check less, and explain less.

Diagnosis was hard because the three causes overlapped. Each affected a different slice of users, models, or session types. Any single user's experience mixed several of them, and generic complaints like "it's worse lately" could not separate them. That is why the checklist below tests one variable at a time.

A 15-minute diagnostic checklist

Work through these in order. Each step either clears a layer or points at it.

1. Start a fresh session with a clean task. Close the long conversation. Open a new one, state the task in a few sentences, and include only the files that matter. If quality comes back, you had session decay, not a regression. Long context degrades performance on its own. The Lost in the Middle study showed models use information placed in the middle of long inputs less reliably, and Chroma's context rot research found performance dropping as input length grew, even on simple tasks. Compaction helps but loses detail. Context Compaction for Agents covers what survives a summary.

2. Check the effort or reasoning setting. Find where your tool exposes it (a flag, a settings file, a model picker) and set it explicitly. Rerun the failing task at your old level. If an explicit high setting fixes the problem, the default changed or never fit your work.

3. Check the client version and read the changelog. Run the version command (for Claude Code, claude --version) and compare it with the last version you remember working well. Read the release notes between the two. Look for mentions of context handling, caching, tools, prompts, or defaults. If your tool can install a specific older version, run the same task on both.

4. Diff your own instructions. Run git log -p on your instruction files, your MCP server config, and any custom commands. A teammate's new rule ("always write tests first," "never modify files in /legacy") can change behavior in ways that look like a smarter or dumber model.

5. Compare against a pinned model version. Call the API directly with a dated model snapshot rather than an alias, using a minimal system prompt. If the pinned call does well and the product does badly on the same prompt, the problem is in the product layer. If both do badly, it is the model or your task. Pinning matters because a named model can drift: a 2023 study from Stanford and UC Berkeley researchers documented large behavior shifts between dated versions of the same named model. For planning model transitions, see Model Deprecation Migration.

6. Run each test several times. One bad output proves nothing. Thinking Machines Lab showed that inference can be nondeterministic even at temperature 0, because server load changes batch composition and therefore the numerics. Run each test five times and compare pass rates. Our post on LLM nondeterminism at temperature 0 explains why evals move between runs.

7. Ask whether the task got harder. Be honest about this one. Last month's success may have been a simpler version of today's request.

If steps 1 through 4 all come back clean and step 5 shows the pinned model failing too, you have a real model-level change. That is rare, and now you can show it.

Keep a small personal regression set

The checklist works better when you have a baseline. Without one, "it used to handle this" is a memory, and memory is a weak instrument. A regression set fixes that, and it doesn't need an eval platform.

Pick 10 to 20 tasks the tool has done well, drawn from your real work:

• A refactor in a module with tricky dependencies
• A bug fix where the right answer needs reading three files
• A task that tests whether it follows your instruction file (naming, test style, forbidden directories)
• A long-session task that requires remembering an early decision
• A task where the correct move is to ask a clarifying question or refuse

For each one, write down the input and a pass/fail check you can apply quickly. Hamel Husain's case for product-specific evals recommends starting with simple assertions before anything fancier, and that advice holds here. A test that passes, a file left untouched, or a function signature kept intact are all checkable.

A plain file is enough:

- id: refactor-payment-retry
  prompt_file: cases/refactor-payment-retry.md
  fresh_session: true
  effort: high
  checks:
    - "pytest tests/payments -q exits 0"
    - "no changes outside src/payments/"
  known_good:
    client_version: "recorded at baseline"
    pass_rate: "5/5"
- id: remembers-early-decision
  prompt_file: cases/long-session-db-choice.md
  fresh_session: false
  checks:
    - "final migration uses the schema agreed in turn 2"

Rerun the set after client updates, after changing instruction files, and whenever someone says the tool feels off. Record the client version, model, and effort with each run. After a few cycles you will have a history that shows exactly when a pass rate dropped and what changed that week.

Every failure you hit in daily work is a candidate for the set. That loop is the subject of Every Production Failure Should Become an Eval Case, and it applies to an individual developer as much as to a production agent. If you are comparing two configurations of your own setup, such as a new instruction file or a different tool list, Harness A/B Testing for Agents covers how to tell whether the change helped.

How to file a report a vendor can act on

Vendors get thousands of "it's worse now" messages. Those reports matter in aggregate, and the postmortem started from user complaints, but a reproducible report gets triaged faster and helps more. Include:

The "what you ruled out" row matters most. It tells the vendor's engineers which layers you have already cleared, which is the same narrowing they would otherwise do themselves. A report that says "effort set explicitly, fresh session, pinned API call passes, product fails" points almost straight at the system prompt or client.

Keep the tone factual. Engineers on the receiving end want to fix the bug, and a clean repro is the fastest route.

For team leads: make quality drops boring

When one developer's tool gets worse, they lose an afternoon. When a whole team's tool gets worse, you get weeks of slower merges, quiet workarounds, and an argument about whether the AI rollout is working. You can make these incidents routine:

• Pin what you can. Use dated model versions in API integrations. For client tools, decide whether everyone auto-updates or updates on a schedule after a quick regression run.
• Set effort explicitly. Put effort settings in shared config so a vendor default change doesn't reach your team silently.
• Version instruction files. Treat shared instruction files like code, with review and history, so "what changed" is one git log away.
• Own a team regression set. Twenty tasks from your real codebase, run by one person after each client update, takes less than an hour.
• Track outcomes over time. Revert rate and review rework catch slow drift that no single session reveals. Measuring AI Coding Assistant Productivity covers which metrics hold up.

A composite example shows why this pays off. A team of 30 engineers sees complaints spike after a client update. Without a baseline, they spend a week debating whether to switch vendors. With a regression set, one engineer reruns it the next morning, sees long-session tasks failing while fresh sessions pass, links it to the update, files a report, and tells the team to start fresh sessions more often until a fix ships. The whole thing takes one morning.

Regression checks for agents you run in production

These failure classes don't stop at coding assistants. A support agent, a research agent, or a document review workflow sits on the same stack: a model, a vendor's defaults, a system prompt, and session state. When one of those layers moves, customers notice before your team does.

OpenNash builds production agents with this layer in place from day one: pinned model versions, explicit effort settings per task, versioned prompts, and a regression suite built from real workflow cases, all handed over to the client. If your team keeps arguing about whether the AI got worse and nobody can prove it either way, book a call and we'll help you pick the first ten regression cases from your own workflow and set up the baseline run. Start by writing down the three tasks your team relies on most, before the next update.