Downloaded Agent Skills Are Untrusted Operational Code

An agent skill can look like documentation and behave like code.

That is enough reason to treat every downloaded skill as untrusted operational code until it has been pinned, reviewed, and tested.

The risk is not limited to a Python or shell file tucked into scripts/. The instructions are part of the execution path. They can influence what the agent reads, which tool it calls, what it sends to an external service, and whether an ordinary approval is reused for a harmful action. A supporting reference can carry more instructions. A dependency can change after installation. A skill does not need to exploit the host in the traditional sense if it can persuade an already-authorized agent to misuse its access.

Two recent preprints make this concern concrete. Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt Injections demonstrates targeted attacks through a modified skill and its supporting script. Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale measures risky patterns across a large early-marketplace snapshot.

The papers are not the final word. One is a proof of concept. The other relies mainly on automated detection over a December 2025 dataset. Neither establishes that every community skill is dangerous or that repository pinning makes a skill safe.

They do support a practical default: do not install first and inspect later.

What The Papers Actually Establish#

The first paper's presentation-editing scenario shows the failure path clearly. The researchers modified a public presentation skill so that an ordinary edit also invoked a helper described as a backup script. The helper uploaded the presentation to an external endpoint. During the legitimate editing work, the user approved Python commands with a persistent "do not ask again" choice. The later upload then ran without a new approval.

The web-interface result also shows why the claim needs boundaries. That environment blocked the helper's outbound request. The researchers adapted by placing sensitive slide data inside a link in the final response, which would exfiltrate the data only if the user opened the link. One control worked. A different path remained. Defense in depth reduced the attack but did not justify trusting the skill.

The 26.1% number needs special care. The study divides it into 5.2% high-severity patterns, 8.1% medium-severity patterns, and 12.8% low-severity practices such as unpinned dependencies or excessive permissions. Its detector reached 86.7% precision and 82.5% recall on a 200-skill validation set. A pilot dynamic check of 25 high-confidence skills found exploitable behavior in 72%, but full runtime analysis of the dataset remains future work.

That is serious evidence for review. It is not evidence that one in four skills is malware.

The snapshot also covers two public marketplaces during an early period of the ecosystem. It excludes non-English skills, short placeholders, duplicates, and repositories that were already returning 404. Enterprise skills, private exchanges, other marketplaces, and later platform controls fall outside the measured population.

The responsible conclusion is narrower and more useful: enough risky behavior exists, and enough of it can hide in ordinary-looking instructions or scripts, that blind installation is not a reasonable default.

Why The Package Boundary Matters#

The Agent Skills specification makes the package shape explicit. A skill has a required SKILL.md and may include executable scripts/, on-demand references/, assets/, and other files. The Markdown body contains instructions that load when the skill activates. Referenced resources can load later as the task calls for them.

That creates three related review surfaces:

• The instruction surface: metadata, SKILL.md, references, examples, and hidden or encoded text that can steer the model.
• The execution surface: scripts, package manifests, install steps, generated files, and code downloaded at runtime.
• The authority surface: the files, credentials, network destinations, shells, applications, and write actions the host already lets the agent use.

This is why "it is only Markdown" is the wrong security model. The Markdown is not machine code, but it participates in deciding which machine code or tool call runs.

It is also why a permission prompt is not a complete defense. A human may approve a broad command family for a legitimate part of the task without seeing that a later instruction will reuse it. The first paper's presentation example is useful precisely because the malicious step looked like an ordinary backup inside a real editing workflow.

Line-By-Line Review Is Necessary, Not Sufficient#

"Review every line" is a good minimum. It should mean more than reading the rendered SKILL.md on a marketplace page.

Review the raw file. Rendered Markdown can hide HTML comments. Follow every local reference. Read every script. Inspect package and lock files. Look for generated or encoded content. Search for external URLs, dynamic downloads, environment-variable access, credential paths, shell invocation, permission changes, install hooks, and instructions to conceal actions or bypass higher-priority policy.

Then review the relationship between the pieces.

A line that says "run the backup helper" may look harmless until the helper is opened. A script that performs an HTTP request may be reasonable until the skill instructions tell the agent to send the current workspace. An unpinned package may be clean today and different tomorrow. A reference file may change the workflow even if the main prompt did not move.

Static search helps find review leads:

git ls-tree -r --name-only <full-commit-sha>
 
git grep -n -I -E \
  'curl|wget|https?://|eval\(|exec\(|subprocess|os\.system|sudo|chmod|\.ssh|credentials|token|password|api[_-]?key' \
  <full-commit-sha> --

Those matches are not a verdict. A documentation skill may contain many URLs. A security skill may legitimately discuss credentials. A malicious instruction may avoid every obvious keyword. Search reduces the surface; judgment still has to connect the requested capability to the actual behavior.

Automated skill scanners belong in the same supporting role. The marketplace study documents both false positives and false negatives. The prompt-injection paper warns that an LLM-based scanner has its own instruction-following and evasion surface. Use scanners to prioritize attention, not to outsource the trust decision.

A Repository-Pinned Intake Contract#

The safest intake flow starts before the skill enters an auto-discovered directory.

1. Define The Capability Before Looking At The Package#

Write down what the skill needs to do and what it must not do. Name the expected inputs, outputs, tools, network access, filesystem scope, credentials, and side effects.

This gives the reviewer a basis for rejecting unrelated behavior. Without that contract, almost any suspicious step can be explained as "helpful."

2. Quarantine Before Discovery#

Do not place an unreviewed skill in .agents/skills, .claude/skills, a Codex skill directory, or any other location the client scans automatically. Keep it in a disposable review directory with no secrets and no live project data.

Do not run its installer, build, hooks, scripts, examples, or tests during the first inspection pass. A test command is still code execution.

3. Pin Provenance#

Record the original repository URL, owner, license, retrieval date, full commit SHA, and the hash of the reviewed artifact. A branch name, tag, release label, marketplace listing, or latest is not an immutable review receipt.

For consequential workflows, vendor the reviewed snapshot into a controlled repository and preserve its provenance record. That keeps the accepted bytes available if the upstream repository disappears and makes local review history independent of a marketplace listing.

Pinning prevents silent drift. It does not make bad code good.

4. Inventory The Entire Tree#

List every file at the pinned revision before opening the most obvious one. Identify executable scripts, package manifests, lockfiles, binary assets, nested archives, references, templates, configuration, and unexpected paths.

The review scope is the package, not the front page.

5. Review Behavior And Authority Together#

For each instruction or executable path, ask:

• Is this necessary for the named capability?
• What data can it read?
• Where can it send data?
• What can it write, delete, install, or execute?
• Does it ask the agent to hide an action, weaken review, or ignore a higher-priority rule?
• Does it fetch code or configuration after the reviewed revision?
• Does it depend on unpinned packages, floating container tags, branches, or mutable URLs?
• Can a narrow approval for the legitimate task authorize a broader later action?

Reject obfuscation, unexplained credential access, self-modification, arbitrary external code loading, broad shell grants, and unrelated data collection by default. An exception should have a named owner, a narrow business need, an enforced sandbox, and an expiration or review date.

6. Test Without Valuable Data#

Use a fresh sandbox, synthetic fixtures, no personal files, no production credentials, and no network access unless the capability explicitly requires a named destination. Observe file reads, writes, commands, and network attempts. Confirm that denial paths fail closed.

The test should prove the skill stays inside its contract, not only that it produces an attractive answer.

7. Promote And Update Deliberately#

Only after review and isolation tests pass should the exact snapshot move into an active skill location. Keep the source receipt beside it or in a repository registry.

Treat every update as a new dependency decision:

git diff --no-ext-diff --text --no-renames \
  <previous-reviewed-sha> <candidate-sha> -- .

Review the complete diff, rerun the safety tests, update the receipt, and promote the new snapshot explicitly. Do not auto-sync a trusted local skill from a moving upstream branch.

A Simple Default Policy#

This is a routing policy, not a universal compliance standard. The stronger the data and authority, the stronger the evidence should be.

The Tradeoff Is Slower Adoption#

This intake contract adds friction. A useful skill may take hours to review instead of seconds to install. Vendoring snapshots creates maintenance work. Sandboxed tests can be awkward. Exact pins mean upstream fixes do not arrive automatically.

That cost should change the selection strategy. Do not collect skills because they might be useful someday. Review the few capabilities tied to recurring work. Prefer instruction-only packages for low-risk tasks. Independently author a small local skill when reviewing a broad third-party package would cost more than rebuilding the required behavior. Reserve network, shell, credential, and write-capable skills for cases where their operational value earns the review and containment work.

Automatic updates trade review latency for silent behavior drift. For software that can steer an agent near private files or write-capable tools, I prefer the latency. A visible dependency update is a manageable cost. An invisible authority change is not.

The Control Is The Reviewable Decision#

There is no magic file format that makes third-party instructions trustworthy.

Repository pinning helps because it turns "I downloaded a useful skill" into a claim about exact bytes. Line-by-line review helps because the behavior is visible before activation. Isolation helps because missed behavior has less authority. Diff-based updates help because trust does not silently carry from one revision to the next.

None of those controls is complete alone. Together they make the decision reproducible, reviewable, and reversible.

That is the standard I would apply before a downloaded skill can touch source code, personal files, credentials, customer data, or a write-capable tool. It is also the operational side of the broader argument in Treat Agent Skills Like Supply-Chain Dependencies and the installation warning in Your Repo Needs an Agent Harness, Not More Prompt Paste.

The useful default is simple:

Download into quarantine. Pin the revision. Review the whole package. Test it without secrets. Grant only the authority the capability earns.