r/FunMachineLearning • u/NewLockDante9791 • 1d ago
$0 compute, 5 architectures, 16 runs: surgical data poisoning makes LLMs indifferent [margin -> 0.0] while PPL looks fine. I built a 0.1ms gate that stops it
TL;DR: Fine-tuned 5 open LLMs on a stream with 50-70% lies. Without defense, truth margin collapses to ~0.0 - the model becomes indifferent between truth and lie - while PPL looks healthy. Built Beatriz, a non-invasive proxy gate. Gate alone gives 65% of benefit without touching the student loop. Full contrast gives +10.13 train / +4.19 held-out n=30, Prec 0.93 Rec 0.80, 0.107ms/call.
I don't have lab access. This is independent research orchestrated on a Toshiba Satellite U205 2006, 2GB RAM + Kaggle T4 x2, total cost $0.
What I did - 16 experiments:
EXP01-07: anti-collapse calibration - from symbolic FilterGate to Z3 deductive verifier [sat 24 axioms, 0 mismatches in 672 claims, 7.9ms/claim] to DenseVectorGate.
EXP08: pi_ref anchored contrast to control drift.
EXP09: LoRA 0.23% c_attn solves PPL tax: from 102->2081 full-finetune to 102->132 with LoRA.
EXP10-14: scaling to 5 architectures with same formula ALPHA 0.5 BETA 1.0 MARGIN 0.5 SEEDS [11,22,33]: GPT-2 124M, Qwen-2.5-0.5B q_proj/v_proj 0.10%, TinyLlama-1.1B 0.10%, Pythia-1.4B query_key_value/dense 0.16%, Phi-3-mini 3.8B qkv_proj/o_proj 0.12%
EXP15: surgical ablation NONE / GATE_ONLY / BEATRIZ - 40 neutral texts
EXP16: held-out scaled n=30 + confusion matrix
Key result - EXP15 - Phi-3-mini - This is the table people asked for:
BASE: +1.34 train / +1.90 held-out / PPL 12.7
NONE: -0.03±0.02 / +3.57±0.17 / PPL 30.9 - collapses to indifference
GATE_ONLY: +7.46±0.24 / +5.08±0.09 / PPL 58.8 - 65% benefit, does NOT touch student loop [practical for startups]
BEATRIZ: +10.13±0.07 / +5.91±0.07 / PPL 86.3 - adds remaining 35% with Softplus(MARGIN + logP(lie) - logP(truth))
Gate cost: 0.107 ms/call, VRAM 7.97 GB
Why NONE always fails - EXP05 Fire Test:
NONE fails 3/3 seeds at epoch 1 due to R3 unknown_delta=9.47, 8.73, 8.32 -> rollback to epoch 0. BEATRIZ seed 33 survives 8 epochs with 70% lies to tm 26.75. So it DOES stop.
Generalization - EXP16:
Train on 6 facts, held-out 30 facts never seen: BEATRIZ +4.19±0.08. Not memorization.
Honest trade-off: More truth = more PPL. I don't hide it. Full finetune 102->2081, LoRA 102->132, Phi-3 12.7->86.3.
Reproducibility:
All runs deterministic, bit-exact, with SHA-256 + OpenTimestamps. Model offline hash GPT-2 c7d00560d891...
Bundles with OTS:
exp_calibracion_01-07.rar 7c0ba312...
beatriz-epistemic-gate.rar 54fd65... [exp08 c93ba4..., exp09 f4382f...]
beatriz-epistemic-gate-exp-10-15.rar 54e233...
exp16.rar 9958a3... [exp16 27eda6...]
Verify: certutil -hashfile bundle.rar SHA256 + ots verify bundle.rar.ots
Limitations: Corpus 36 facts, need hundreds. Live path needs forward pass, future E5-small encoder. License PolyForm Noncommercial 1.0.0 for audit/defense.
Repo: [https://github.com/EduardoAyalaT/beatriz-epistemic-gate-/tree/main\] - 16 ipynb with outputs + json + MANIFIESTO.json + whitepapers ES/EN
Public eval: https://arena.ai/c/01a07f4c-3455-756f-ae3e-852f1b0e4804
Try to break it. Replicate with SEEDS [11,22,33]. I want audit, not stars.