Benchmarks

Basilisk uses an isolated, deterministic ground-truth lab to qualify the installed CLI and Electron desktop application. Both surfaces execute the same pinned probes and are scored against the same expected outcomes.

Latest Verified Results

PASS: the latest checked-in artifacts record 46/46 passing scenarios in both the CLI and desktop harnesses, with detection and scenario parity between the two surfaces.

Measured result	CLI	Electron desktop
Scenario coverage	46/46 passed	46/46 passed
Vulnerable HTTP controls	6 TP, 0 FP	6 TP, 0 FP
Hardened HTTP controls	6 TN, 0 FN	6 TN, 0 FN
Precision / recall / accuracy	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
Protocol controls	20/20 passed	20/20 passed
Runtime readiness	No scanner errors	Backend and renderer ready

The authorization matrix made 28 requests to each local HTTP target. The hardened target produced 0 authorization violations; the deliberately vulnerable target produced the 2 expected violations, and both outcomes were classified correctly.

Authorization target	Requests	Violations found	Classification	Duration
Hardened HTTP	28	0	Correct	7.025 s
Vulnerable HTTP	28	2	Correct	7.109 s

Recorded Performance

These timings come from a single repetition of the checked-in local run. They describe that run only and are not hardware-independent performance claims.

Surface and target	Wall-clock time	Peak RSS
CLI — vulnerable HTTP	6.316 s	4.17 MiB
CLI — hardened HTTP	3.879 s	3.90 MiB
Desktop — vulnerable HTTP	10.887 s	Not recorded
Desktop — hardened HTTP	6.674 s	Not recorded

The ground-truth manifest was current and CLI-to-desktop detection parity and scenario parity both passed. Its SHA-256 fingerprint is 2e1c088d51659a0d1d1fcc2a802c2bb34913224d3435a0ded8319aea9035b42c.

Inspect the machine-readable evidence in the repository:

• Parity result
• CLI summary
• Desktop summary

Scenario Inventory

Each surface must cover exactly 46 duplicate-free scenarios:

Scenario group	Count
HTTP finding controls	12
Authorization-persona controls	14
WebSocket controls	10
Hostile HTTP controls	10
Total	46

The target is local and synthetic. It does not fetch external URLs, execute shell commands, or query a real database.

What the Harness Measures

For CLI and desktop runs, the harness records:

• true positives, false positives, true negatives, and false negatives;
• precision, recall, specificity, F1, and accuracy;
• authorization-control and protocol outcomes;
• wall-clock timing and peak memory where supported;
• report SHA-256 values and scanner errors;
• a 95% timing confidence interval for repeated runs.

Release Gates

The deterministic lab requires precision and recall of 1.0, correct authorization classification, no scanner errors, all protocol controls passing, and identical scenario outcomes between CLI and desktop.

These figures apply only to the checked-in synthetic lab. They are regression gates, not a claim that Basilisk has perfect accuracy against hosted models. Live LLM behavior is nondeterministic and can change with model, provider, configuration, and time.

Reproduce the Run

Run the installed CLI benchmark:

python benchmarks/cli/run.py --repetitions 3

Run the Electron benchmark:

cd desktop
npm ci
npm run benchmark

Compare fresh CLI and desktop results without rerunning them:

python benchmarks/run_all.py --compare-only

Reports are written below benchmarks/results/. The parity result demonstrates that both supported interfaces produce the same outcomes for the controlled scenarios; it does not compare commercial LLM vendors or publish a universal jailbreak success rate.