Skip to content

Mylonite

Model robustness is not the same as application security. A frontier model can resist every generic prompt-injection and still hand an attacker a win, because the hole is in your app's design, not the model's alignment. Mylonite tests whether your app-layer controls are stopping the attack, writes a validated regression test for each weakness, and gates CI so a model upgrade can't silently strip the protection away.

Mylonite is an open-source framework for AI-layer security testing. It targets the AI/agentic part of an application — system prompt, tools, RAG pipeline, agent memory — and emits validated regression tests that gate CI. It deliberately does not test the surrounding traditional code; that work belongs to SAST/DAST tools.

The phased build plan lives in ROADMAP.md.

Status: the full pipeline works

scan → generate → validate → gate runs end to end against the bundled reference app and your own MCP (Model Context Protocol) app (--target-file). Findings are proven by the control-efficacy check (the two-build differential on the reference app), emitted as committed pytest regression tests, and surfaced as a gating PR, SARIF, or a JSON bundle. Every claim is checked by an independent verification harness. The surface is deliberately narrow — every shipped feature runs on an MCP app you didn't author and is on a path to third-party proof. See the changelog.

Example: same model, two app versions

The same model, two versions of the bundled app: against the vulnerable version Mylonite catches weaknesses including a send_email dispatched with no approval step (a pure app-design flaw); against the guarded version it reports no findings at all. The app's design decides — see the full independent scorecard, negatives included.

Coverage by app shape

What Mylonite can test depends on how your app is reachable — not every shape gets the same classes, and some get a weaker claim than you might expect:

App shape How Mylonite connects What's testable What reads NOT TESTED
MCP server, local process transport: stdio — Mylonite drives your real tools with its own planner W1–W4 against the server's declared tools and prompt; server-side guards Guardrails that live in your agent or host, not the server — Mylonite's planner replaces your agent loop, so an approval UI, an input filter, or a model-side guardrail call in your own agent code is never exercised
MCP server, remote sse / http, static-token auth Same as local Same as local; no OAuth sign-in yet
Plain HTTP agent, no tool surface transport: rest — one endpoint, a prompt in, a reply out W2 only (direct prompt injection, judged on the reply text) W1 (tool-description poisoning), W3 (egress), W4 (consequential action) — a black box has no tool surface and no side-effect probe by default to test them against
Agent built on a framework (in-process tools, no MCP, no HTTP endpoint) No adapter Nothing, as-is Everything — wrap the agent in an HTTP endpoint for W2, or expose its tools over MCP for the full W1–W4 surface
Agent-side guardrails (input filters, tool-call policies, host approval steps) Not reached by any transport above — Always, today — these guardrails sit in your agent process, which Mylonite's planner replaces rather than drives

A kept W1–W4 finding also only earns the strongest claim — "your own control is load-bearing" — once you declare control_env so Mylonite can switch your real guard off; see Add a kill switch to your own guard. Without it, a KEPT verdict proves the attack is real and that a canonical control of that class closes it, not that your specific implementation does.

Where to go next