Introduction

Rothalyx RE Framework is a native reverse engineering framework and desktop workstation for binary analysis, disassembly, graph reconstruction, decompilation, debugging, scripting, and AI-assisted analyst workflows. The repository holds the C++ core, the native Qt desktop application, the CLI, the public C SDK, fuzzing harnesses, and the release packaging that ships desktop builds for Linux, macOS, and Windows. This case study documents the framework as it exists in the source tree at the v1.0.3 line: how it is layered, why the technology choices were made, how the analysis pipeline is composed, and which properties are designed versus which still need measurement before I would present them as results.

The organizing premise is simple and it drives almost every decision below: the analysis engine should be a single native core, and every other surface (the desktop app, the CLI, the SDK, plugins, and the optional AI layer) should consume that core rather than reimplement it. The architecture doc states the rule directly, and the repository layout is the proof of it. Most of what follows is an explanation of what that premise costs and what it buys.

Problem

Reverse engineering rarely happens inside a single program. An analyst disassembles in one tool, decompiles in another, attaches a debugger from a third, and glues the pieces together with scripts. Each tool models the binary differently, so moving from static structure to runtime context and back means re-establishing state by hand. Automation layered on top of that stack tends to be fragile, because there is no shared representation to automate against: the disassembler’s notion of a function is not the debugger’s, and neither matches whatever the scripting layer assumes.

The problem Rothalyx targets is not “produce a disassembler” or “produce a decompiler” in isolation. Both of those exist in mature form elsewhere. The problem is the fragmentation between them, and specifically the loss of context every time an analyst crosses a tool boundary. When the disassembly view, the decompiler view, the CFG, the call graph, the debugger, and the coverage overlay are all clients of one model, an annotation made in one view is immediately meaningful in every other. When they are separate tools, that shared meaning has to be manufactured, and it usually is not.

Background

The framework’s primary language is C++ (roughly 95% of the tree), with CMake for builds, Python for scripting and plugins, and Shell and PowerShell for packaging. The desktop UI is Qt 6 Widgets. Persistence is SQLite. Disassembly depends on Capstone, and the full toolchain pulls in cURL. The project is AGPL-3.0 and, at the time of writing, has a single contributor and one release line, 1.x, with v1.0.3 as the latest release. v1.0.3 itself was a branding and packaging refresh that completed a rename from the project’s earlier name, Zara, across source, resources, package metadata, SDK version strings, and release automation.

Two facts about that background matter for the rest of the study. First, this is a solo project, which means there is no external reviewer enforcing the architecture; the discipline has to be structural rather than social. Second, the release history is short and deliberate rather than sprawling, which tells you the design was mostly decided up front and then hardened, rather than discovered incrementally through churn.

Threat Model

Rothalyx is a tool that ingests hostile input by design. Its threat surface is not the analyst; it is the binaries and traces the analyst loads. The security policy in the repository names the categories that matter most: parser bugs triggered by hostile binaries, sandbox escapes, debugger privilege or attachment issues, distributed-worker authentication or transport flaws, and credential leakage or unsafe model-backend behavior.

Two threat classes deserve emphasis because they shaped concrete parts of the implementation. The first is the parser surface. The loader accepts PE, ELF, and Mach-O, and the trace subsystem accepts trace inputs, and both are exposed to adversarial files. A malformed section header, an out-of-range symbol offset, a truncated trace, or an oversized label must fail cleanly rather than crash, corrupt memory, or execute attacker-influenced paths. Because the loader is the very first stage of the pipeline, a bug there is maximally dangerous: it runs before any other validation.

The second is the AI layer, which introduces a data-egress boundary that does not exist in a purely local tool. When a hosted provider is configured, function context leaves the machine. That turns two ordinary concerns into security concerns: how much data leaves (request bounding) and where the provider key lives (secret storage). The design treats both parser robustness and AI egress as first-class, and I will return to how each is addressed.

Why These Libraries

A native reverse engineering framework is a series of technology commitments, and each of the major ones in Rothalyx is defensible on grounds beyond familiarity.

Qt 6 Widgets for the desktop. A reverse engineering workstation is a dense, multi-view, keyboard-driven application: disassembly, decompiler, CFG, call graph, hex, debugger, coverage, and annotation views coexisting and cross-navigating. Qt Widgets is built for exactly that class of native desktop application, with mature support for complex custom views, docking layouts, and cross-platform native rendering on Linux, macOS, and Windows from one codebase. A web-stack UI would have forced either an embedded browser (heavy, and awkward for the tight coupling between views and the C++ core) or a client-server split that contradicts the single-process, single-core premise. Widgets over QML is the right call for an information-dense analyst tool where the interface is tables, trees, and graphs rather than animated surfaces.

SQLite for persistence. Project state in Rothalyx is inherently relational and versioned: functions, basic blocks, edges, xrefs, comments, type annotations, AI insights, and multiple analysis runs per project. SQLite gives a single-file, serviceless, transactional, inspectable store that the desktop app, CLI, and SDK can all open without a running database process. For a tool whose projects are meant to be portable artifacts an analyst can copy, share, or archive, a single .sqlite file is close to ideal. The alternative, a bespoke binary project format, would have meant hand-writing serialization, migration, and crash-safety that SQLite already provides.

Capstone for disassembly. Instruction decoding is a problem where correctness across many instruction encodings is the entire game, and it is not where a framework should spend its novelty budget. Capstone is a well-tested multi-architecture disassembly engine, so leaning on it for the decode step lets core/disasm focus on architecture metadata and integration rather than reimplementing decoders. This is the same logic that makes ProtoCrash lean on established parsers: reuse the boring, correctness-critical component and spend effort on the parts that differentiate the tool.

cURL for the full toolchain. The AI layer speaks to hosted providers over HTTP (OpenAI, Anthropic, Gemini, OpenAI-compatible gateways). cURL is the obvious, portable, battle-tested HTTP client for a C/C++ codebase, and pulling it in only for the full toolchain keeps the core buildable without networking when AI is not needed.

Embedded Python for extensions. Plugins and automation are where users need to move fast and iterate, and Python is the lingua franca of the security research community. Embedding Python as an extension surface (rather than exposing the core only through C++) lowers the barrier for analysts to script the tool, while the architecture keeps Python off the main execution path so the heavy analysis stays native. This is a deliberate split: native where performance and correctness matter, Python where flexibility matters.

A C ABI for the public SDK. The SDK is exported as a C ABI rather than a C++ interface specifically so it can be consumed from Python, Rust, Go, or any FFI target without leaking internal C++ types (name mangling, template instantiations, STL types across the boundary) into the public surface. C is the stable common denominator of foreign-function interfaces, so a C-shaped boundary is what makes the engine genuinely polyglot.

CMake and Ninja for builds. A cross-platform native project shipping five package formats needs a build system that all three desktop platforms and CI understand. CMake with presets (dev, asan-fuzz) plus Ninja is the standard modern answer, and the presets encode the important build configurations so that a contributor or a CI job gets the intended flags without memorizing them.

Architecture

The system is layered, and the layering is the architecture. The pipeline is: loader, memory model, disassembly, function discovery and CFG recovery, IR and SSA, analysis and type recovery, decompiler, debugger integration, and finally persistence, SDK, plugins, and optional AI. Each stage consumes the previous stage’s output.

Concretely, the core modules map onto those stages. core/loader parses supported formats into a normalized BinaryImage. core/memory owns mapped regions, permissions, rebasing, and symbol lookup. core/disasm decodes instructions and carries architecture metadata. core/cfg recovers functions, basic blocks, successors, loops, switch edges, and call relationships. core/ir, core/ssa, core/analysis, and core/type lift instructions into analysis-friendly form and run recovery and simplification passes. core/decompiler emits structured C-like output. core/debugger provides runtime control and static/runtime integration. core/database persists projects, annotations, AI output, and artifacts. core/sdk publishes the stable C ABI.

The applications, apps/desktop_qt and apps/cli, are clients. The extension surfaces, Python plugins and embedded scripting, sit at the edge rather than on the main execution path. The desktop split is explicit: a Qt Widgets GUI over a C++ application layer over the C++ core, with embedded Python at the edge. That ordering is not incidental. It keeps heavy UI and state management in the native application layer, keeps the analysis engine independent of any UI, and keeps the scripting surface from being able to destabilize the core’s hot path.

Parser Development

The loader is the sharpest edge of the whole system, because it runs first and it runs on hostile input. core/loader parses PE, ELF, and Mach-O and produces a normalized BinaryImage; core/memory then maps sections into a virtual address space, handles rebasing, and resolves symbols. Normalizing three very different container formats into one image model is the design move that lets every downstream stage be format-agnostic: disassembly, CFG recovery, and the decompiler never need to know whether the bytes came from an ELF or a Mach-O.

That normalization is also where the threat model bites. Each of the three formats has its own ways of being malformed: overlapping or out-of-range sections, symbol tables that point outside the file, load commands with inconsistent sizes, headers whose declared lengths do not match the actual data. The design intent, stated in the security policy and reinforced by the fuzzing harness, is that all of these are rejected cleanly rather than trusted. The rothalyx_loader_corpus_runner exists precisely to exercise binary loading, section mapping, symbol population, and rebasing against a corpus of valid and malformed binaries, and the pass criterion is strict: malformed inputs must be rejected cleanly, and a crash or sanitizer finding is a failure.

The trace subsystem is the second parser surface, exercised by rothalyx_trace_corpus_runner against valid, truncated, oversized, and malformed traces. The concrete behavior of each parser on each malformed-input class (what is rejected, how, and with what diagnostics) is Additional validation required: the harness and the pass criterion are documented, the per-case behavior is not published.

Implementation

Walking the pipeline stage by stage shows how the shared-core premise plays out in code.

The loader and memory model produce and own the normalized image and its address space. Everything downstream addresses code and data through that space, which is what makes rebasing a memory-model concern rather than something each analysis has to redo.

Disassembly (core/disasm) decodes instructions through the architecture layer (Capstone underneath) and attaches architecture metadata. Keeping architecture knowledge concentrated here means the CFG and IR stages can work against a uniform decoded-instruction representation.

Function discovery and CFG recovery (core/cfg) is where raw instructions become program structure: functions, basic blocks, successors, loops, switch edges, cross-references, and call relationships. This is the stage that turns a flat instruction stream into the graph an analyst actually reasons about, and it is the input to both the desktop CFG/call-graph views and the decompiler.

IR, SSA, analysis, and type recovery (core/ir, core/ssa, core/analysis, core/type) lift instructions into an analysis-friendly form, transform into SSA, and run recovery and simplification passes. Routing through SSA before decompilation is the conventional and defensible choice, because most modern data-flow and type-recovery passes assume SSA form; doing it any other way would mean fighting the analyses.

The decompiler (core/decompiler) emits structured C-like output from the recovered, typed, simplified program state. Its output quality depends entirely on the stages before it, which is the practical argument for the pipeline ordering.

The debugger (core/debugger) provides runtime execution control and, critically, static/runtime integration: breakpoints, thread-aware state inspection, runtime patching, and pivots back into the static picture. Because it is part of the same core, a runtime observation can be correlated to the same function and basic-block model the static views use.

Persistence (core/database) writes all of this to SQLite, scoped by analysis run, so a project can hold multiple runs and support comparison over time.

The SDK (core/sdk) publishes the C ABI over the whole thing.

The desktop application is a client of all of this. It provides a startup launcher for new or existing projects; navigation across functions, imports, exports, strings, and xrefs; and the full set of views. It persists comments, type annotations, version history, and workspace state. A binary or a saved .sqlite project can be opened directly from the command line by passing the path to the desktop executable. The CLI (rothalyx_cli) runs the same pipeline headless; an AI-assisted run selects a backend through environment variables (ROTHALYX_AI_BACKEND, provider key, model) and invokes an ai-model subcommand.

Algorithms

The recovery path is a conventional static-analysis lifting chain, implemented natively: decode instructions, discover functions and basic blocks, recover control flow (successors, loops, switch edges, call relationships), lift into IR, transform into SSA, run analysis and type-recovery passes, and emit structured pseudocode. The AI layer adds a separate, bounded algorithm on top: run normal static analysis, choose a bounded set of candidate functions, build a compact function-context payload, submit the request to the selected provider, normalize the response into Rothalyx insight records, and persist those results with the analysis run.

The specific heuristics inside function discovery, type recovery, and decompiler structuring are Additional validation required: the repository documents the stages and their ordering, but the internal algorithm details are not published in the docs surface reviewed here. The full set of supported architectures beyond the documented layer is likewise Additional validation required.

Design Decisions

The central design decision is the shared core. I made the rule explicit: if a capability must exist in more than one interface, it belongs in the core first, and the desktop app, CLI, and SDK all consume the same implementation. This is the decision that most shapes the codebase, because it forbids the common shortcut of adding a feature directly to the GUI. In a solo project, that structural rule is doing the job that code review would do in a team: it prevents the GUI from quietly accumulating logic that the CLI and SDK then lack.

A second decision is that AI is optional and additive. The pipeline runs without a hosted model, and the desktop app can stay in heuristic-only mode. The AI response parser is deliberately conservative: if a provider response is malformed or unusable, Rothalyx falls back to the heuristic path instead of blocking analysis. This mirrors the loader’s posture toward hostile binaries. In both cases untrusted input (a malformed binary, an unusable model response) degrades to a safe default rather than breaking the run.

A third decision is that secrets are stored through the host operating system rather than in normal app settings: Windows Credential Manager, macOS Keychain, Linux Secret Service via secret-tool. If secure storage is unavailable, Rothalyx does not silently write keys to plaintext settings; it leaves the user to rely on environment variables or install host keyring tooling. This is a fail-closed choice: the inconvenient path (refuse to store) is chosen over the dangerous path (store in the clear).

A fourth decision is the C-shaped public boundary, discussed above under library choices: the SDK is a C ABI so it can be consumed from Python, Rust, Go, or other FFI targets without leaking internal C++ types.

Architecture Evolution

The architecture did not sprawl; it was decided and then hardened, and the release history shows that shape. The earliest visible commit finalizes the release packaging workflow, which says the project treated cross-platform delivery as a first-class concern from early on rather than as an afterthought. The v1.0.1 release is explicitly a security-hardening release, which fits a project whose threat model puts hostile input first: harden the parsers and the execution surface before adding breadth.

The scalability of the design is framed the same way it is in the fuzzing and architecture docs: a single core that the desktop app, CLI, and SDK consume, with the distributed-analysis infrastructure named as a capability and as a threat-model surface (distributed-worker authentication and transport). The evolution is therefore less about restructuring the core and more about widening the surfaces that consume it. The distributed-analysis design details are Additional validation required.

Refactoring Lessons

The one large, documented refactoring in the project’s history is the Zara→Rothalyx rename delivered as v1.0.3, and it is instructive precisely because it was done as its own release rather than folded into a feature change. The rename touched the source tree, the desktop resources (PNG, SVG, ICO, ICNS logo assets), the CMake packaging metadata, the SDK version strings, and the Linux, macOS, Windows, and Arch packaging scripts, standardizing on the rothalyx-re-framework slug.

The lesson encoded in that choice is that an identity change which reaches into packaging metadata and SDK version strings is not a cosmetic edit; it is a cross-cutting refactor with real regression surface. Doing it as a dedicated release isolates the risk: if a package target or an SDK version string breaks, the cause is unambiguous because nothing else changed in that release. A team that folds a rename into a feature release loses that clean attribution. The historical zara repository alias still resolving is a second small lesson: a rename should preserve inbound references rather than orphan them. A published verification that no legacy zara identifiers survive in built artifacts is Additional validation required.

Research Process

The development record visible in the repository is a small number of deliberate releases along a single 1.x line rather than a large branching history. Notable commits include the early “Finalize release packaging workflow,” the “v1.0.1: Security hardening release,” and the v1.0.3 “Rename Zara to Rothalyx and prepare v1.0.3 release.” The trajectory reads as: establish the pipeline and packaging, harden, then stabilize identity and release automation. A detailed reconstruction of the research and debugging process beyond what the commits and docs state is Additional validation required.

Experiments

The repository does not publish experiment logs, evaluation datasets, or measured comparisons. Any specific experiment (for example, decompiler output quality against a labeled corpus, or CFG recovery accuracy on stripped binaries) is Additional validation required.

Benchmarks

No benchmark results are published in the repository. The project site displays figures such as 847 functions, 62 imports, and 4.1K cross-references, but these are illustrative interface values in a product mockup, not measured benchmarks, and I do not present them as performance data. Real throughput, memory, and accuracy benchmarks are Additional validation required.

The table below is the structure to fill in once measurements exist. Replace each Additional validation required with Observed during testing and the measured value, and record the environment (CPU, RAM, OS, build preset, and the specific target binary and its size) alongside it.

Metric	How to measure	Value
Loader latency	Time from open to normalized BinaryImage, per format (PE/ELF/Mach-O), across small/medium/large binaries	Additional validation required
Disassembly throughput	Instructions decoded per second on a representative target	Additional validation required
CFG recovery time	Time to recover functions/blocks/edges for a target of known function count	Additional validation required
Decompiler latency	Time to emit structured output per function, and full-binary total	Additional validation required
Analysis throughput	End-to-end pipeline time (loader→decompiler) per MB of target	Additional validation required
Memory usage	Peak RSS during full analysis, excluding the target under debug	Additional validation required
Database size	On-disk .sqlite project size relative to target size and run count	Additional validation required
Debugger overhead	Added latency per breakpoint hit / single-step	Additional validation required
AI run cost	Requests per run and wall-clock added, at a given max_model_functions	Additional validation required

Performance Analysis

No performance measurements (analysis time per binary, memory footprint, decompiler latency, debugger overhead) are published. Performance analysis is Additional validation required.

Security Analysis

The security posture is defined by the threat model and the fuzzing harnesses. Two sanitizer-backed corpus runners exist: rothalyx_loader_corpus_runner, which exercises binary loading, section mapping, symbol population, and rebasing against valid and malformed binaries, and rothalyx_trace_corpus_runner, which exercises trace parsing against valid, truncated, oversized, and malformed inputs. They build under an asan-fuzz preset. The stated pass criterion is that malformed inputs are rejected cleanly; a crash, sanitizer finding, or unexpected termination is a failure, and sanitizer findings are treated as release blockers until triaged.

The security policy asks reporters to use private vulnerability reporting, keep exploit detail out of public issues, and includes credential leakage and unsafe model-backend behavior in scope. Concrete security findings, fixed vulnerabilities, or CVEs are Additional validation required: none are published, and the v1.0.1 “security hardening” commit is not accompanied by a public advisory in the material reviewed.

Tradeoffs

The shared-core rule trades short-term convenience for long-term consistency: a feature cannot be bolted onto the GUI quickly, because it must land in the core first. The optional-AI decision trades some out-of-the-box “smart” behavior for privacy and determinism, since heuristic-only mode is the default and hosted inference is opt-in. Storing secrets in the OS keyring trades setup friction (a missing keyring blocks secure key storage) for the guarantee that keys are never silently written to plaintext. The C ABI trades expressiveness at the boundary for FFI portability and ABI stability. Embedding Python trades a heavier build and a larger attack surface for a scripting experience the research community already knows. Each of these is a case where the less convenient option was chosen to protect a longer-term property (consistency, privacy, safety, portability).

Limitations

The framework is single-contributor and early in its release history, with one 1.x line. Architecture-layer breadth (which instruction sets are supported beyond the documented pipeline) is not enumerated in the reviewed docs and is Additional validation required. Decompiler output quality, type-recovery accuracy, and debugger platform coverage are not quantified. The distributed analysis infrastructure is mentioned as a capability and named in the threat model, but its design details are Additional validation required. Per-case parser behavior on malformed inputs is documented in intent but not in published results.

Future Work

The repository does not publish a roadmap. Grounded, near-term directions implied by the current tree include broadening the architecture layer, expanding the fuzzing corpus categories beyond loader and trace, quantifying decompiler and recovery quality, and publishing the benchmark set above. Any specific committed roadmap is Additional validation required.

Lessons Learned

The clearest lesson encoded in the codebase is that the “single core, many clients” rule pays off only if it is enforced from the start; retrofitting it after the GUI grows its own logic is far harder, and in a solo project the rule is what substitutes for code review. A second lesson is that treating hostile input as the primary threat surface, and wiring sanitizer-backed corpus runners into the release gate, turns parser robustness into a testable release criterion rather than an aspiration. A third is that making AI optional and its response parsing conservative keeps a model failure from becoming an analysis failure. Additional first-hand lessons from debugging specific subsystems are Additional validation required.

Conclusion

Rothalyx is a native reverse engineering framework whose defining choice is architectural: one analysis core, consumed by the desktop app, CLI, SDK, plugins, and an optional AI layer. That choice, together with an adversarial-input threat model backed by sanitizer fuzzing and an opt-in, OS-keyring-backed AI layer, describes a coherent system as it stands at v1.0.3. What it does not yet publish is measurement: benchmarks, experiments, and security findings remain to be documented, and this case study marks each of those gaps rather than filling them with invented numbers.