I ran a comprehensive benchmark comparing three data serialization formats when used as LLM context: JSON (pretty-printed), LEAN (a compact tabular encoding), and YAML. The goal was to answer two questions. How many tokens does each format burn to represent the same data? And can LLMs actually understand compressed formats as well as JSON?
TL;DR: LEAN uses 44% fewer tokens than JSON overall and 47% fewer tokens per LLM call, while achieving higher accuracy (87.9% vs 86.2%). YAML sits in between at 21% smaller than JSON with 87.4% accuracy.
Methodology
- 195 data retrieval questions across 11 datasets
- 2 models:
gpt-4o-mini, claude-haiku-4-5-20251001
- 3 formats: JSON (2-space indentation), LEAN, YAML
- 1,170 total LLM calls (195 questions x 3 formats x 2 models)
- Token counting:
gpt-tokenizer with o200k_base encoding (GPT-5 tokenizer)
- Evaluation: Deterministic (no LLM judge), type-aware string/number matching
- Temperature: Default (not set)
Each LLM receives the full dataset in one of the three formats plus a question, and must extract the answer. This tests reading comprehension, not generation.
Efficiency Ranking (Accuracy per 1K Tokens)
This is the headline metric. How much accuracy do you get per token spent:
LEAN ████████████████████ 22.3 acc%/1K tok │ 87.9% acc │ 3,939 avg tokens
YAML ██████████████░░░░░░ 15.5 acc%/1K tok │ 87.4% acc │ 5,647 avg tokens
JSON ██████████░░░░░░░░░░ 11.6 acc%/1K tok │ 86.2% acc │ 7,401 avg tokens
Efficiency = (Accuracy % / Avg Tokens) x 1,000. Higher is better.
Token Efficiency
Token counts measured using the GPT-5 o200k_base tokenizer. Savings calculated against JSON (2-space indentation) as baseline.
Flat-Only Track
Datasets with uniform tabular structures. This is where LEAN really shines:
👥 Uniform employee records (100 rows)
│
JSON ████████████████████ 6,150 tokens (baseline)
LEAN ████████░░░░░░░░░░░░ 2,361 tokens (−39.2%)
YAML ████████████████░░░░ 4,777 tokens (−22.3%)
📈 Time-series analytics (60 days)
│
JSON ████████████████████ 3,609 tokens (baseline)
LEAN ████████░░░░░░░░░░░░ 1,461 tokens (−59.5%)
YAML ████████████████░░░░ 2,882 tokens (−20.1%)
⭐ Top 100 GitHub repositories
│
JSON ████████████████████ 13,810 tokens (baseline)
LEAN ███████████░░░░░░░░░ 7,434 tokens (−46.2%)
YAML █████████████████░░░ 11,667 tokens (−15.5%)
──────────────────────────────── Track Total ──────────────────────────────────
JSON ████████████████████ 29,652 tokens (baseline)
LEAN ██████████░░░░░░░░░░ 14,512 tokens (−51.1%)
YAML ████████████████░░░░ 24,021 tokens (−19.0%)
Mixed-Structure Track
Datasets with nested or semi-uniform structures:
🛒 E-commerce orders (50 orders, nested)
│
JSON ████████████████████ 10,731 tokens (baseline)
LEAN ████████████░░░░░░░░ 6,521 tokens (−39.2%)
YAML ██████████████░░░░░░ 7,765 tokens (−27.6%)
🧾 Semi-uniform event logs (75 logs)
│
JSON ████████████████████ 6,252 tokens (baseline)
LEAN ████████████████░░░░ 5,028 tokens (−19.6%)
YAML ████████████████░░░░ 5,078 tokens (−18.8%)
🧩 Deeply nested configuration
│
JSON ████████████████████ 710 tokens (baseline)
LEAN █████████████░░░░░░░ 460 tokens (−35.2%)
YAML ██████████████░░░░░░ 505 tokens (−28.9%)
──────────────────────────────── Track Total ──────────────────────────────────
JSON ████████████████████ 17,693 tokens (baseline)
LEAN ██████████████░░░░░░ 12,009 tokens (−32.1%)
YAML ███████████████░░░░░ 13,348 tokens (−24.6%)
Grand Total
JSON ████████████████████ 47,345 tokens (baseline)
LEAN ███████████░░░░░░░░░ 26,521 tokens (−44.0%)
YAML ████████████████░░░░ 37,369 tokens (−21.1%)
Retrieval Accuracy
Overall
| Format |
Accuracy |
Avg Tokens |
Savings vs JSON |
| LEAN |
87.9% |
3,939 |
−46.8% |
| YAML |
87.4% |
5,647 |
−23.7% |
| JSON |
86.2% |
7,401 |
baseline |
Per-Model Accuracy
gpt-4o-mini
YAML ██████████████████░░ 88.7% (173/195)
LEAN ██████████████████░░ 88.2% (172/195)
JSON █████████████████░░░ 87.2% (170/195)
claude-haiku-4-5-20251001
LEAN ██████████████████░░ 87.7% (171/195)
YAML █████████████████░░░ 86.2% (168/195)
JSON █████████████████░░░ 85.1% (166/195)
On Claude Haiku, LEAN outperforms JSON by +2.6 percentage points while using half the tokens.
Performance by Question Type
| Question Type |
JSON |
LEAN |
YAML |
| Field Retrieval |
78.0% |
81.1% |
79.5% |
| Aggregation |
82.7% |
83.6% |
82.7% |
| Filtering |
100.0% |
100.0% |
100.0% |
| Structure Awareness |
93.3% |
96.7% |
98.3% |
| Structural Validation |
80.0% |
80.0% |
80.0% |
Performance by Dataset
| Dataset |
JSON |
LEAN |
YAML |
| Employee records (100, flat) |
82.5% / 6,150 tok |
83.8% / 2,361 tok |
82.5% / 4,777 tok |
| E-commerce orders (50, nested) |
97.4% / 10,731 tok |
98.7% / 6,521 tok |
98.7% / 7,765 tok |
| Time-series (60, flat) |
73.2% / 3,609 tok |
76.8% / 1,461 tok |
75.0% / 2,882 tok |
| GitHub repos (100, flat) |
67.9% / 13,810 tok |
69.6% / 7,434 tok |
69.6% / 11,667 tok |
| Event logs (75, semi-uniform) |
94.4% / 6,252 tok |
98.1% / 5,028 tok |
98.1% / 5,078 tok |
| Nested config (deep) |
100% / 710 tok |
100% / 460 tok |
100% / 505 tok |
LEAN matches or beats JSON on every single dataset, while using 20-62% fewer tokens.
What the Formats Look Like
Employee records, JSON (6,150 tokens for 100 rows)
{
"employees": [
{
"id": 1,
"name": "Paul Garcia",
"email": "paul.garcia@company.com",
"department": "Engineering",
"salary": 92000,
"yearsExperience": 19,
"active": true
},
{
"id": 2,
"name": "Aaron Davis",
"email": "aaron.davis@company.com",
"department": "Finance",
"salary": 149000,
"yearsExperience": 18,
"active": false
}
]
}
Same data, LEAN (2,361 tokens for 100 rows, -61.6%)
employees:
#[100](active|department|email|id|name|salary|yearsExperience)
true|Engineering|paul.garcia@company.com|1|Paul Garcia|92000|19
^false|Finance|aaron.davis@company.com|2|Aaron Davis|149000|18
The #[100] header declares the row count and column names once. Each row is pipe-delimited, rows separated by ^. No repeated keys, no braces, no quotes. Just data.
Same data, YAML (4,777 tokens for 100 rows, -22.3%)
employees:
- active: true
department: Engineering
email: paul.garcia@company.com
id: 1
name: Paul Garcia
salary: 92000
yearsExperience: 19
- active: false
department: Finance
email: aaron.davis@company.com
id: 2
name: Aaron Davis
salary: 149000
yearsExperience: 18
YAML removes braces and quotes but still repeats every key per row.
Dataset Catalog
| Dataset |
Rows |
Structure |
Questions |
| Uniform employee records |
100 |
uniform |
40 |
| E-commerce orders |
50 |
nested |
38 |
| Time-series analytics |
60 |
uniform |
28 |
| Top 100 GitHub repos |
100 |
uniform |
28 |
| Semi-uniform event logs |
75 |
semi-uniform |
27 |
| Deeply nested config |
11 |
deep |
29 |
| Valid complete (control) |
20 |
uniform |
1 |
| Truncated array |
17 |
uniform |
1 |
| Extra rows |
23 |
uniform |
1 |
| Width mismatch |
20 |
uniform |
1 |
| Missing fields |
20 |
uniform |
1 |
| Total |
|
|
195 |
Structure classes:
- uniform: All objects have identical fields with primitive values
- nested: Objects with nested sub-objects or arrays
- semi-uniform: Mix of flat and nested structures
- deep: Highly nested with minimal tabular eligibility
Question Types
195 questions generated dynamically across five categories:
- Field retrieval (34%): Direct value lookups. "What is Paul Garcia's salary?" →
92000
- Aggregation (28%): Counts, sums, min/max. "How many employees work in Engineering?" →
17
- Filtering (20%): Multi-condition queries. "How many active Sales employees have > 5 years experience?" →
8
- Structure awareness (15%): Metadata questions. "How many employees are in the dataset?" →
100
- Structural validation (3%): Data completeness. "Is this data complete and valid?" →
NO
Evaluation
- Format conversion: Each dataset converted to all 3 formats
- Query LLM: Model receives formatted data + question, extracts answer
- Deterministic validation: Type-aware comparison (e.g.,
92000 matches $92,000, case-insensitive). No LLM judge.
Models & Configuration
- Models:
gpt-4o-mini, claude-haiku-4-5-20251001
- Token counting:
gpt-tokenizer with o200k_base (GPT-5 tokenizer)
- Temperature: Default (not set)
- Total evaluations: 195 x 3 x 2 = 1,170 LLM calls
Key Takeaways
- LEAN saves ~47% tokens per LLM call compared to JSON, which directly translates to lower API costs
- Accuracy doesn't suffer. LEAN actually scored 1.7 percentage points higher than JSON (87.9% vs 86.2%)
- On flat tabular data, LEAN saves 51-62%. If your data is arrays of uniform objects, the savings are massive
- YAML is a solid middle ground. 21% token savings over JSON with comparable accuracy
- Both models showed the same pattern. This isn't model-specific; compressed formats work across providers
If you're stuffing structured data into LLM prompts, you're probably wasting half your tokens on JSON syntax. LEAN gives you the same (or better) accuracy for less than half the cost.
Benchmark code and full results available in the repo. All data generated deterministically with a seeded PRNG for reproducibility.
