What belongs in the AI supply chain?
Models and datasets are only part of it. Agent skills, IDE rules, tool descriptions, MCP servers, hooks, packages, and remotely fetched instructions can influence an agent’s actions. Treat instructions as security-relevant input, not harmless documentation.
OWASP’s supply-chain guidance covers third-party components, models, and data. Agents of Chaos, which I co-authored, examines attacks across tools, memory, external services, compromised skills, and other agent capabilities. The workflow below is a practical synthesis, not a certification standard.
1. Establish provenance before installation
Record the publisher, repository, exact version or commit, download source, and intended purpose. Check that the package name matches the repository and that a setup command uses the expected registry. Inspect dependencies and installation hooks before executing them.
Prefer an immutable release you can review over a moving branch or an unpinned launcher. A hash identifies the bytes you checked; it does not prove those bytes are safe. If you cannot establish where a component came from, defer installation.
2. Read the instructions and code together
Read raw Markdown, configuration, scripts, and tool metadata—not just the rendered preview. Look for hidden text, encoded commands, unexpected download destinations, requests for credentials, approval bypasses, and instructions to conceal actions or override the user.
A suspicious string is a reason to investigate, not proof of malware. Connect each finding to the proposed task: what data is accessed, what leaves the machine, what executes, and whether the requested permission is necessary.
The unpick documentation describes offline checks for these patterns. Use scanner output as evidence to review, not an automatic approval.
3. Test behavior in a disposable environment
Only run an untrusted component in an appropriately isolated environment. Use synthetic files and throwaway credentials, with no production tokens, host credential directories, privileged access, or sensitive mounts. Deny network access by default; allow only what the test genuinely requires.
Observe file changes, subprocesses, attempted connections, and access to data. Compare those actions with the component’s stated purpose. A container alone is not a complete security boundary, and a quiet test does not rule out delayed or environment-specific behavior.
The Devil Wears Markdown combines static findings, safety-constrained sandbox evidence, and model-based reasoning. Its results support combining evidence; they do not establish that every component with a clean result is safe.
4. Limit identity, permissions, and autonomy
Even a reviewed component should receive only the tools and data needed for its task. Separate test and production identities. Prefer short-lived, narrowly scoped credentials, read-only access where possible, and explicit human approval for consequential actions such as publishing, deleting, or sending sensitive data.
OWASP’s excessive-agency guidance distinguishes excessive functionality, permissions, and autonomy. Enforce authorization outside the model: a prompt telling an agent to behave is not an access-control boundary.
5. Preserve the review and reassess changes
Keep a small review record: source, version, purpose, findings, observed behavior, permitted access, unresolved concerns, and your decision. Re-review changed instructions, dependencies, permissions, or remote services. Stop use and revoke access when behavior no longer matches the approved task.
Review is a lifecycle activity. An approved version does not approve future releases, and a trusted publisher does not make every dependency trustworthy.
A before-install checklist
- Can I identify the publisher and exact version?
- Have I read the raw instructions, code, configuration, and install hooks?
- Can I explain every requested permission and network destination?
- Have I reviewed alerts instead of treating a scanner verdict as proof?
- Can I test without exposing real credentials or sensitive files?
- Are consequential actions gated by authorization and approval?
- Have I saved the evidence and decided what changes trigger another review?
If a critical question remains unanswered, pause. With no sandbox or specialist tooling, provenance checks, raw-file review, and refusing unnecessary access are still useful—but they are not substitutes for behavior testing.
Research and further reading
- Agents of Chaos: A Systematization of Knowledge (SoK) on Malicious Attacks on Agentic AI Systems — research paper available on arXiv.
- The Devil Wears Markdown: Detecting Malicious Intent Hidden in AI Agent Skill Instructions through Hybrid Static, Dynamic, and Agentic Analysis — research paper available on arXiv.
- OWASP LLM03:2025 — Supply Chain.
- OWASP LLM06:2025 — Excessive Agency.
- unpick source and documentation — check the current documentation for commands and limitations.
The Devil Wears Markdown — presentation resources & slides →Get in touch about AI security strategy, speaking, or consulting.