The lexer (tamper_framework/lexer.py) is the foundation the whole framework stands on, and it solves two of the three critical bugs by itself. First, it recognizes multi-character operators (>=, <=, <>, !=) as single tokens by checking them before single-character operators, so encoding produces %3E%3D rather than the broken %3E=. This longest-match-first ordering is the difference between valid and invalid output. Second, it assigns every token a UUID at lex time that never changes, even when a transformation rewrites the token’s value (for example, wrapping SELECT into /*!50000SELECT*/).

That UUID identity is what makes everything downstream reliable. Position-based tracking desynchronizes the moment any transformation changes a token’s length, because every subsequent position shifts. By referring to tokens by UUID instead, the transformer can apply multiple rules in sequence, protect against reapplication, and reassemble the query without ever losing track of which token is which. The lexer also preserves string literals and comments as their own token types, so transformations do not accidentally mangle string contents. The lexer test suite (10 tests) specifically covers multi-character operators, string literals, comments, and UUID tracking, which are precisely the correctness-critical behaviors.