REGAAN RBenchmarks & Fine-Tuning Performance
All training performance, loss metrics, and hardware requirements for PoCSmith are measured directly from the training campaign run on consumer-grade hardware.
Hardware Performance & VRAM Budget
The entire fine-tuning campaign was completed on a single consumer laptop GPU under strict memory bounds.
| Parameter | Measured Value | Notes |
|---|---|---|
| GPU | NVIDIA GeForce RTX 4050 Laptop GPU | 6GB VRAM total |
| Peak VRAM Usage | 5.9 GB / 6.0 GB | 96% memory utilization |
| VRAM Headroom | ~228 MB | Thin margin, successfully absorbed evaluation spikes |
| GPU Temperature | ~73°C | Stable thermal envelope during sustained load |
| Training Duration | 3 hours 17 minutes | Complete run over 3 epochs |
| Step Latency | ~12 seconds/step | Batch size 1, gradient accumulation 4 |
| Total Steps | 885 steps | 295 steps per epoch |
Training Curve & Token Accuracy
The QLoRA fine-tune achieved a 30% reduction in training loss and a 5.8-point improvement in next-token accuracy on the exploit code structure dataset.
Epoch Progress Metrics
| Training Milestone | Training Loss | Token-Level Accuracy | Evaluation Loss |
|---|---|---|---|
| Start | 1.20 | 72.6% | — |
| Epoch 0.34 | 1.03 | 75.4% | — |
| Epoch 1.00 | 0.89 | 78.0% | — |
| Epoch 2.00 | 0.85 | 78.2% | — |
| Epoch 3.00 | 0.84 | 78.4% | 0.926 (stable) |
Learning Curve Analysis
- Epoch 1 Dominance: Approximately 93% of the total next-token accuracy improvement (5.4 out of 5.8 points) was achieved within the first epoch.
- Plateau Behavior: Epochs 2 and 3 yielded minimal accuracy improvements (+0.4 points total), indicating that the LoRA adapter capacity and dataset bounds had reached their optimal transfer limit.
- Generalization Stability: The training loss (0.84) and validation evaluation loss (0.926) maintain a healthy, narrow margin, confirming that QLoRA regularization (dropout 0.05, 4-bit frozen base) successfully guarded against severe overfitting.
Dataset Partitioning
The model was specialized on a curated instruction-tuning dataset containing 1,472 samples distributed as follows:
- Train Set (80%): 1,177 samples
- Validation Set (10%): 147 samples
- Test Set (10%): 148 samples
Dataset Split by Feature Type
- Shellcode Payloads: 1,065 samples (72%) — covers Linux, Windows, and ARM architectures.
- CVE-to-Exploit Pairs: 407 samples (28%) — maps CVE descriptions directly to python/C exploits.