Engineering Notes: Basilisk

These notes describe the v2.0.2 implementation and its validation boundaries. Deterministic local regression results are kept separate from live-model observations, which can change by provider, model, configuration, and date.

Note 1: Deterministic Labs Before Live Claims

Hosted LLMs are nondeterministic and frequently updated. Basilisk therefore qualifies its CLI and Electron surfaces against a synthetic local lab with fixed inputs and expected outcomes before interpreting any live-model run.

The lab contains 46 scenarios across HTTP findings, authorization personas, WebSockets, and hostile HTTP behavior. Perfect precision and recall are release gates for this lab only, not claims about all models.

Note 2: Smart Prompt Evolution

SPE-NL mutates, crosses over, and scores natural-language attack candidates. Operator selection can adapt to previous outcomes so the search spends more time on strategies that remain useful for the current target context.

Fitness represents search guidance. It is not equivalent to proof that a safety boundary was crossed, and it must be interpreted alongside the response and evidence policy.

Note 3: Intent Drift

Repeated mutation can produce a prompt that earns a high score while no longer testing the original objective. Basilisk tracks intent and diversity to discourage a population from converging on irrelevant but superficially interesting text.

Lexical similarity is only an approximation. Optional semantic models can improve comparisons, while the zero-dependency fallback remains less capable of recognizing paraphrases.

Note 4: Versioned Probes and Attack Modules

Basilisk v2.0.2 ships 263 YAML probes, 33 attack modules across 9 categories, and 5 reconnaissance modules. Probes carry mappings and optional metadata; modules have production, beta, or research trust tiers.

This structure lets operators reproduce a probe, select a bounded module set, and distinguish stable checks from exploratory research behavior.

Note 5: Evidence Policy

Keyword matches can confuse ordinary model discussion with genuine leakage or tool abuse. Basilisk stores typed evidence signals and applies module-specific proof requirements. Weak evidence can downgrade a high-severity candidate while retaining the original assessment and downgrade reason.

Evidence gates reduce noisy reporting but do not remove the need for human judgment. The target’s intended behavior and system context still determine impact.

Note 6: Audit and Native Integrity

Audit events are hash-chained and signed with Ed25519. Persistent identity requires a configured key; otherwise a session can use an ephemeral key that cannot establish continuity across runs.

Native libraries are checked against a signed manifest before loading. A verified manifest proves that bytes match the signed release metadata; it does not prove that the native code is free from vulnerabilities.

Note 7: Stagnation, Budgets, and Cancellation

Evolutionary searches can stop improving while continuing to consume tokens and time. Basilisk exposes generation limits, request controls, cancellation, and stagnation handling so an operator can bound a campaign.

Provider billing and latency vary. Reproducible reports should record the configuration, model identifier, timestamps, and any provider-side changes known during the run.

Note 8: Multi-Objective Scoring

A single refusal keyword is too weak for many LLM security questions. Basilisk can combine multiple signals such as goal progress, evidence strength, novelty, and intent preservation.

Weights express an evaluation policy. They should be reviewed for the target instead of treated as a universal definition of jailbreak success.

Note 9: Response Caching and Sensitive Data

Caching can avoid repeating identical requests during development and controlled regression tests. Cache keys and stored responses may contain prompts, system text, or private model output, so retention and secret handling matter as much as performance.

Production engagements should use the encrypted desktop secret store where available, redact exported artifacts, and avoid sharing raw project databases.

Note 10: Differential Testing

Differential scans compare how multiple configured targets respond to the same probe set. A behavior difference is useful evidence of policy drift, configuration variation, or a regression; it is not automatically proof that one model is vulnerable.

The ground-truth parity harness checks that CLI and desktop produce identical outcomes for the 46 controlled scenarios. It does not rank OpenAI, Anthropic, Google, or other vendors.

Reproducibility

python benchmarks/cli/run.py --repetitions 3

cd desktop
npm ci
npm run benchmark

python benchmarks/run_all.py --compare-only

Use Basilisk only against systems you own or are explicitly authorized to test.