Do Coding Agents Still Need Skills? Which Agent Skills to Cut, Keep, and Write

Coding agents still need skills, but fewer of them and of a different type. In a 22-run eval by dbt Labs, every Claude 5.5 model scored at least 92% with no dbt skills installed. When the author sorted the failures by cause, none of them came from missing dbt knowledge. Cut the skills that teach a well-known tool. Keep the skills for new or niche features, and write skills for how your team tests, verifies, and defines done.

An agent skill is a packaged set of instructions that an agent loads only when a task needs it. Our guide Agent Skills Over Prompts explains the format. This post is about which skills earn their place after the model already knows your tools.

The key numbers

All figures come from dbt Labs' eval write-up unless the row names another source.

Why every team with a skill library has this question

Skills arrived for Claude in October 2025 and became an open standard that December, according to the dbt write-up. Since then, vendors have published skill packs, and teams have written their own. Microsoft now supports skills in Copilot Studio agents that run on the GitHub Copilot harness. Microsoft describes the use case as "the step-by-step way your organization does something". It also gives this warning: "It is not a guarantee, so evaluate it rather than assume it".

Skills also bring risk and maintenance work. A systematization-of-knowledge paper on agentic skills reports that "nearly 1,200 malicious skills infiltrated a major agent marketplace" in the ClawHavoc campaign. Each skill that you keep is text that you must review, version, and update. Agent Instruction Files Are Source Code covers that maintenance work.

Few organizations measure whether their skills help. In the sources we reviewed, only dbt Labs and Trail of Bits publish measured results. The other sources describe skills or recommend them, but they give no measurements.

How dbt Labs tested its own skills

dbt Labs released agent skills for four areas: analytics engineering, the Semantic Layer, platform operations, and migration. Its launch post made a claim about these skills. It said that skills "can transform generalist coding agents into highly capable data agents". The later eval tested that claim against dbt's own project.

The test project used the Formula 1 World Championship dataset: 14 relational tables, about 700,000 rows, and 589,000 lap times. Round 2 added a 3-table vendor feed and 20 million telemetry rows. The models were Opus 5.5, Sonnet 5.5, Haiku 5.5, and the older Sonnet 4.5.

The agents ran in Claude Code's Auto permission mode with a canned "use your best judgment" reply. In Round 1, every run but one passed all 43 answer-key checks.

Finding 1: the model generation moved scores more than skills

The chart shows the five per-run scores that the write-up states as numbers. Use them to compare the two effects in four steps:

1. Take the older model without skills. Sonnet 4.5 scored 38%.
2. Take the newer models without skills. Every 5.5 model scored at least 92%.
3. Subtract. The generation change is worth at least 92 - 38 = 54 points.
4. Compare the skills effect on the older model. Forcing skills moved Sonnet 4.5 from 38% to 17%, a change of -21 points.

Forced skills did not rescue the older model. The 17% run "had read the skills before it wrote its first model," so the skills were in its context. The two Sonnet 4.5 runs were the two lowest scores of any run.

Code review scores showed the same pattern. Sonnet 4.5 scored 2.67 without skills and 2.22 with skills forced. Round 2 runs from the 5.5 models scored from 3.9 to 4.7.

Finding 2: run-to-run variance hid any skills effect

The same arm, Sonnet 5.5 with skills available, scored 61%, 100%, and 67% on three runs. That is a 39-point spread (100 - 61) with no change to the setup. The author writes: "Run-to-run variance is bigger than any skills effect."

Variance also limits the code review scores. In Round 1, every skills arm scored above its model's no-skills run, by about 0.1 for Opus and 0.1 to 0.5 for Sonnet and Haiku. But one project reviewed twice scored 4.00 and 4.22. That 0.22 gap is pure reviewer noise, and it is larger than some of the skills gains. Skills may help code quality slightly. This eval cannot show it with confidence.

Most arms had one run each. The author says so directly: "With one run per arm you can't say whether skills help". Our guides on agent run variance and harness A/B testing give more detail. They explain how many runs you need before a difference means something.

Finding 3: agents seldom open skills, and forcing them costs tokens

Agents opened between zero and five skill files across an entire suite. Sonnet 4.5 with skills available opened none. When agents did open a skill, it was usually the Semantic Layer skill or the dbt Mesh skill. Those features are less common in public code than core dbt.

Forcing skills changed how the agent worked, but it did not change the results. Haiku with skills forced used about 1.6x the output tokens and 1.5x the dbt calls of Haiku without skills in Round 1. The write-up says that "the extra reading didn't change the scores."

Finding 4: every failure was a process failure

The author sorted every failure by cause: "none were dbt knowledge related." The failures were in how the agent worked:

• Frozen tests. Nine runs, three in each arm, wrote a test that froze today's answers. The test passed on the current data and broke when correct new data arrived.
• Unchecked restores. The 61% run restored its tables from a backup and silently reverted 215 laps.
• Skimmed definitions. The 67% run got is_podium off by one and missed a 215-row lap-time correction.
• Short lookback windows. The same 67% run used an incremental lookback window that was too short.
• Costly guardrails. Fifteen of 17 projects added vendor-drift checks. Each build then took about 13 minutes instead of 15 seconds, 40 to 57x slower.

The engineering basics were strong. All 9 Round 2 runs built exactly the edited model in Slim CI. All 9 matched every one of 41 drivers and 167 sessions in the vendor feed. Eight of 9 keyed their incremental telemetry models on load time.

A dbt skill cannot fix these failures, because the failures are not about dbt. They are about what "done" means, which checks are required, and what a check may cost. That information is specific to your team.

What Trail of Bits built around process

Trail of Bits worked with OpenAI on Patch the Planet. It built two Codex plugins, patching-bugs and regression-hunt. It based them on "our analysis of merged patches and thousands of our engineers' Codex sessions". Its write-up reports the review results. The review found 21 regressions in over 130 merged pull requests. Of these, 16 were functional, one was performance, and four were build or test.

The plugins follow four principles:

1. Opinionated guidance in as few words as possible.
2. Modular steps.
3. Strict input and output schemas.
4. A live proof of concept for every claim.

Trail of Bits gives a strict definition of a regression. It is "a new defect introduced by a change that breaks previously correct and supported behavior". The behavior must be "along a path a real user or attacker can reach". For a fix, patching-bugs requires a three-arm proof. The original code must fail, the patched code must pass, and a complete revert must fail again. It also requires a run of regression-hunt.

The plugins do not teach Codex how to write code. They tell it what to check, in what order, and what evidence to return. Projects add their own process through THREAT_MODEL.md and PATCHING_GUIDELINES.md (or AGENTS.md). GPAC, Liboqs, and validator.js wrote threat models or patching skills.

Trail of Bits measured the result against a baseline Codex agent. The baseline was prompted to "review the PR for regressions". The plugin produced 25% fewer false-positive findings. The 95% confidence interval was 14% to 38%. There was a cost. Trail of Bits reports that the fewer false positives "came with a tradeoff in how many known regressions it could confidently report."

Trail of Bits did not test skills against model upgrades. Its baseline is a one-line prompt, and it used Codex instead of Claude. The link between its design and dbt's findings is our synthesis. Both point to the same thing: the durable content in a skill is process and evidence.

How the sources compare

The SoK paper reports a split result. In its words, "curated skills can substantially improve agent success rates while self-generated skills may degrade them". That does not conflict with dbt's result. A curated skill can still help when the model lacks the knowledge. dbt is close to a best case for knowledge "already in the weights", as its author says.

Cut, keep, or write: sort your skill library

For rules that a script can enforce, write the script. Keep Deterministic Steps Deterministic and Output Contracts explain when to do this.

Worked example: a 12-skill library (hypothetical)

This composite example shows the sort. The counts are illustrative.

1. List every skill. The data team has 12.
2. Mark tool tutorials. Five explain dbt models, tests, Git branching, and SQL style for a public dialect. Cut them.
3. Mark new or niche knowledge. Three cover an in-house orchestrator, a vendor SDK from last year, and a Semantic Layer setup. Keep them.
4. Mark process content. Four skills mix tool tutorial with team rules. Remove the tutorial text and keep the rules.
5. Add the missing process skills. Use the dbt failure list as a guide: a definition of done, a rule against tests that freeze current values, a required check after each restore, and a lookback-window rule.
6. Add an evidence schema to each process skill. Use Trail of Bits' fields: claim, code revision, required check, raw evidence, open questions.
7. Move hard rules into CI checks. Set a time budget for each check, so that a drift check cannot make builds 40x slower.

The result is 3 kept skills plus about 8 process skills, all shorter than the originals.

Which approach fits your team

When the cut advice is wrong

The dbt eval covers one project, one tool, and one model family. The author calls it a "proof of concept" and says "I wouldn't call that conclusive." Eight runs read another run's notes. The code reviews were not fully blind, because run IDs survived in schema names. The result "is about dbt, not about frameworks in general."

Keep tool skills when the tool is new, niche, or internal. Keep them when you run a model that has not seen the tool. And do not assume that process skills always help. Trail of Bits' plugin reduced false positives, but it also reported fewer known regressions with confidence. Test each skill against the failures that matter to you.

How to test your own skills

1. Pick 5 to 12 real tasks from your backlog, each with checks that a script can run.
2. Set up three arms: no skills, skills available, and skills forced.
3. Run each arm at least three times. If you can afford more runs, add them to the arm with the widest spread.
4. Record the score, output tokens, and tool calls for each run.
5. Compare the gap between arms with the spread inside each arm. If the spread is larger, do not claim an effect.
6. Sort each failure by cause: missing knowledge or process.
7. If most failures are process failures, write process skills and CI checks, then run the eval again.

Why AI Data Agents Still Fail in BigQuery, dbt, and Airbyte and The Bitter Lesson for AI Agents give more detail. They explain what to build into the harness and what to leave to the model.

Turn your skill library into reliable agent output

A smaller skill library that encodes your process gets more correct work per token from each model upgrade. OpenNash puts forward-deployed engineers on your team, and we work with Fortune 2000 companies. A typical engagement:

1. Audit your current skills, instruction files, and agent failures, and sort each one into cut, keep, or write.
2. Build an eval with enough runs per arm to separate a real effect from variance.
3. Write process skills with evidence schemas, and move hard rules into deterministic CI checks.
4. Pick the right model for each task, and test each upgrade against your own eval.
5. Train your team to maintain skills as versioned source code.

At the end, you own the skills, the eval suite, the CI checks, and the documentation.

Book a call with an OpenNash forward-deployed engineer or email us at hello@opennash.com. In the first session, we review your top AI use cases and data. Then we show which ones give the highest business value and the fastest ROI.