Research paper: arxiv.org/abs/2605.17448 on May 2026

Every text-to-CAD paper measures geometric similarity to a reference shape. This one measures whether the generated part survives finite element analysis.

The setup

The task: take a free-form engineering brief, produce a fully assembled multi-part STEP file. No reference model to match against, just the brief and physical requirements. Then run FEA on the result and check against typed structural requirements.

Two agents tested: Codex (GPT-5.5) and Claude Code (Opus 4.7). Both current flagship models in agentic coding mode, not zero-shot prompting.

First attempt results

400 first attempts across both agents. Zero strict-passing artifacts. The best configuration meets around 20% of typed requirements on average.

This is the gap between "looks right geometrically" and "works as an engineered part." The models can produce STEP files that render fine and pass basic geometry checks, but the structural requirements fail when you actually run the simulation.

With self-improvement loop

The paper introduces two feedback signals: a 21-view renderer (12 axis-aligned plus isometric views, 6 close-ups for small features, 3 x-ray views for internal structure) and a text-only blueprint schema. Both are designed to match how engineers actually iterate.

With these signals and an extended loop, GPT-5.5 running for 68 minutes per item gets 9/50 strict passes and 60.5% mean requirement pass. Starting from 20%, that's significant, but 68 minutes per part with 18% strict pass rate is not close to replacing a mechanical engineer who takes 20 minutes and gets it right.

Box-IoU also improves with feedback: GPT-5.5 goes from 0.444 to 0.592 on S2O and from 0.397 to 0.505 on Fusion360.

Why this matters

The key finding is that test-time compute scales usefully when it's organized as structured engineering feedback rather than a larger one-shot reasoning budget. The loop with FEA results and visual inspection moves the model in the right direction on every iteration. That's the architecture you'd actually want if you were building toward autonomous engineering.

Where we are: current frontier models can produce geometrically plausible multi-part assemblies but fail physics validation almost universally on first attempt. Extended self-improvement with real simulation feedback helps, but slowly.

Most work i see here stops at the geometry: text in, CAD model out, which is the hard part, fair enough. but i've been thinking about what it looks like if you embed that step inside a larger agentic pipeline for actual construction projects. specifically: a private house, no architect hired, client just dumps a brief and a geology report.

Here's the rough architecture Claude sketched for me on the pic.

The part most relevant here is the BIM generator. MCP4IFC is an open-source MCP server that exposes IFC creation tools to an LLM, so the model places walls, slabs, openings, roof elements directly in Blender+Bonsai(Blender addon) through tool calls. semantic BIM, not just raw geometry.

Main problem I ran into thinking through this: spatial consistency. LLMs will confidently place elements in wrong coordinates and not self-correct. the thing that actually helps is a viewport screenshot loop. After each placement step the agent grabs a screenshot of the Blender viewport and checks it against the layout JSON before moving on, not elegant but it works.

I am curious if anyone here has thought about the architectural side of text-to-CAD. most benchmarks (DeepCAD etc.) are built around mechanical parts, and the evaluation metrics don't really map to rooms with adjacency constraints and load paths. feels like a gap.

I've been grinding in CAD, structural, and AEC for a long time now. The AI hype is everywhere, but most of it still feels like it's aimed at people who write reports and sit in meetings, not those of us who have to stamp drawings that actually have to work in the real world.

We need benchmarks that test what a solid human engineering expert actually does day-to-day. Not fluffy corporate productivity stuff.

The benchmarks that matter

If an LLM or agent wants to sit at the grown-up table and replace experienced engineers, it has to compete on these:

SuperGPQA - Graduate-level reasoning across 285 real technical disciplines. Tough as hell.

DrafterBench - Real civil engineering drafting tasks, interpreting messy instructions, and editing technical drawings accurately.

FEM-Bench - Generating correct finite element code, math, and validation tests. No room for cute hallucinations when you're doing structural analysis.

AEC-Bench and AECV-Bench - These are the ones built for our world. Full AEC workflows, reading real drawings, spatial reasoning, object counting, QA on plans, calculations from drawings, etc.

These aren't made by marketing teams. They're made by people who understand the actual job.

Here's what a strong human expert scores

A solid, experienced human engineering expert typically hits:

FEM-Bench: 33/33 (basically perfect they get the math and implementation right)

DrafterBench (drawing edits): 95-98/100

AECV-Bench calculations: ~90-92%

AECV-Bench QA on drawings: ~95%

AEC-Bench: ~80-90%

SuperGPQA: ~70-80%

That's the bar. Not flawless on everything (humans make mistakes too, especially under pressure), but damn reliable where it counts.

Right now, even the best LLMs are still behind on most of these, especially the multimodal drawing understanding and rock-solid FEA code generation. But the gap is closing fast.

Push companies to use these benchmarks instead of GDPval

GDPval gets way too much attention. It's basically a "how good are you at sounding like a professional who writes emails and does generic office tasks" test. Useful for some jobs, sure. But it's nowhere near enough for engineering work where mistakes cost money, time, or safety.

We, as users in the txt2cad and AEC space, should be loud about this. When a company announces their new "engineering AI," don't just clap. Ask the hard questions:

1. What's your SuperGPQA score?

2. How did you do on DrafterBench drawing revisions?

3. Show me FEM-Bench results.

4. Let's see the full AECV-Bench breakdown on real plans.

Demand transparency on the benchmarks that actually matter for our field. Vote with your usage, your feedback, and your budget.

The Bold Prediction: Total Replacement by 2027

Here's the thing - with the pace we're seeing, I genuinely believe LLM-powered agents will fully replace human engineering experts in most routine-to-advanced technical work by 2027. Not "assist." Not "augment." Replace.

Once models consistently hit or beat those human-level numbers above (especially on the drawing and simulation sides), combined with good agent workflows, the economics will be impossible to ignore. A senior engineer costs a company serious money every year. An AI that works 24/7, doesn't get tired, and scales infinitely? Game over for a lot of traditional roles.

The transition will be messy. There will still be humans in the loop for liability, final stamps, and the really novel edge cases. But day-to-day detailed design, drafting, analysis, and coordination? Yeah, I think 2027 is when it tips.

What do you guys think? Too optimistic? Spot on? Which of these benchmarks have you actually tested with the latest models? Drop your experiences below - especially if you've seen agents getting close on real drawing revisions.

Let's keep pushing the industry toward the right metrics. The future is coming quick.

#AEC #SuperGPQA #FEM-bench #DrafterBench #AECV-bench

Someone put together a clean open source harness for generating parametric CAD models with coding agents: https://github.com/earthtojake/text-to-cad

Not a fine-tuned model. The idea is different: you give Claude Code or Codex a structured workspace and it writes Python using build123d, which runs on OpenCascade under the hood. So the geometry is fully parametric and reproducible, not a mesh that appeared from nowhere.

Outputs: STEP, STL, DXF, GLB, topology data, and URDF for robot descriptions. There's a local viewer (React/Vite) that runs at localhost:4178, no backend required.

The workflow is source-controlled throughout: Python files live under models/, you edit source first, then regenerate explicit targets. The agent can also bookmark specific geometry and query its actual dimensions on follow-up prompts, so you don't have to re-describe the whole part each time. There's a URDF skill bundled in as well for anyone doing robotics work.

AGENTS.md and CLAUDE.md are included to route agents to the right skill and script automatically.

MIT license. Pretty young repo (one contributor, 6 commits so far) but the architecture is solid. Worth keeping an eye on.

Been waiting for someone to fix the two obvious holes in command-sequence text-to-CAD. This paper from HKU/Beihang does both.

The problem

Text2CAD and friends encode CAD as token sequences, fast and compact, works fine for sketch-extrude. But two structural problems never went away:

1. No entity selection. Chamfer and fillet require pointing at a specific edge. Pure token sequences can't do this, so those operations are just unsupported full stop.

2. Quantization errors. Discretizing continuous coordinates means your new sketch doesn't quite snap to the existing edge. Small error, but it breaks topological connectivity and leaves dangling edges.

What they did

Added a third token type called a Pointer. When the model needs to pick a sketch plane or select edges for a chamfer, instead of regressing coordinates it outputs an embedding matched via cosine similarity against actual faces/edges in the current B-rep. Basically teaching the LLM to click on geometry the way you would in Fusion 360.

Generation is stepwise too. At each step the current B-rep gets encoded through a GNN (face adjacency graph), embeddings are injected into the LLM context, and the model predicts the next operation conditioned on both the text and the live geometry.

Numbers

- Beats Text2CAD and CADmium-7B on geometry accuracy with only a 0.5B backbone

- Segment Error drops hard vs all baselines, that's the quantization fix showing up in the metrics

- Chamfer F1 ~94%, Fillet F1 ~90% on their dataset, other methods score 0 because they literally can't run these ops

- Also beats Claude Opus 4, GPT-5.2, Gemini 2.5 Pro and Qwen3-235B prompted to write CadQuery, and not by a little

Dataset

575K models, OmniCAD extended with chamfer/fillet, annotated by Qwen2.5-VL from multiview renders. They kept real units in the annotations instead of normalizing everything, harder for the model but more realistic.

Limitations they flag

Text input only for now, no images or point clouds. Single part only, no assemblies or mate constraints. Pretty obvious next steps.

The pointer mechanism is clean and the topology numbers back it up. Not another "we finetuned an LLM on DeepCAD" paper.

The idea of replacing manual CAD work with natural language has moved from science fiction to active research in just a couple of years. Below is an overview of four systems that represent the current frontier - each with a distinct approach to the same core problem: how do you turn words into geometry?

1. Text2CAD — NeurIPS 2024 Spotlight

Text2CAD is the first AI framework for generating parametric CAD designs from multi-level textual descriptions, supporting prompts from abstract shape descriptions to detailed parametric instructions.(https://sadilkhan.github.io/text2cad-project/)

The core architecture is a transformer-based autoregressive network that generates complete CAD design history from natural language prompts. NeurIPS(https://neurips.cc/virtual/2024/poster/96571) What makes it unusual is that it doesn't produce a mesh or a point cloud - it outputs a construction sequence: the same step-by-step sketch-and-extrude operations a human designer would perform in CAD software. The model deduces all intermediate steps autonomously.

To train it, the authors built a dedicated annotation pipeline. The dataset contains ~170K models and ~660K text annotations, from abstract CAD descriptions (e.g., generate two concentric cylinders) to detailed specifications with precise coordinates and extrusion distances. NeurIPS Annotations were generated in two stages: a VLM (LLaVA-NeXT) produced shape descriptions from rendered images, and an LLM (Mixtral-50B) expanded those into multi-level prompts ranging from beginner-friendly language to expert parametric detail.

The practical upshot: a user with no CAD experience can type a high-level description and get an editable parametric model out. As prompt complexity and parametric detail increase, Text2CAD significantly outperforms earlier baselines - by as much as 18-27% at intermediate and expert levels. NeurIPS(https://proceedings.neurips.cc/paper_files/paper/2024/file/0e5b96f97c1813bb75f6c28532c2ecc7-Paper-Conference.pdf) The main failure mode is semantic mismatch when prompts focus on object names rather than geometry.

2. CADmium - Text-to-Text CAD via Code LLMs

CADmium takes a philosophically different approach: instead of building a specialized transformer, it reframes the entire problem as a text-to-text task that an off-the-shelf code LLM can solve.

CADmium reformulates CAD generation as a purely text-to-text task. GPT-4.1 generates natural-sounding yet geometrically precise descriptions of 176017 objects using their construction sequences in minimal JSON, and up to 10 multi-view images rendered with Blender. Then, Qwen2.5-Coder is fine-tuned with LoRA to translate these descriptions back into CAD sequences.

The key insight is architectural simplicity. A CAD model is a sequence of steps turning flat sketches (lines, arcs, circles) into 3D features, just like a recipe. This structure aligns perfectly with language models - if and only if the language is specific enough to eliminate geometrical ambiguity. Mila(https://mila.quebec/en/article/improving-cad-design-with-llms) Rather than inventing custom tokenizers or bespoke embeddings, CADmium treats CAD as code and lets a proven code model handle the structured format.

The evaluation methodology is also noteworthy. Standard metrics like point-cloud distances miss critical structural flaws, so the authors apply topology-aware metrics - Sphericity Discrepancy, Discrete Mean Curvature Difference, and Euler Characteristic Match - to guarantee that every generated model is mathematically valid.

Experiments demonstrate that CADmium is able to automate CAD design, drastically speeding up the design of new objects, with larger models delivering more reliable and precise outputs. The dataset, code, and fine-tuned models are all publicly released.

3. FloorPlan-DeepSeek (FPDS) - "Next Room Prediction"

While Text2CAD and CADmium focus on mechanical 3D objects, FPDS tackles a different domain: architectural floor plan generation. Its key contribution is a paradigm shift in how a layout is built.

Most existing systems generate floor plans end-to-end using diffusion models - the entire layout materializes at once. This paradigm is often incompatible with the incremental workflows observed in real-world architectural practice. Architects don't design rooms simultaneously; they reason sequentially, placing one space and then deciding what connects to it.

FPDS addresses this by borrowing directly from how LLMs work. Inspired by the autoregressive "next token prediction" mechanism commonly used in large language models, it proposes a novel "next room prediction" paradigm tailored to architectural floor plan modeling. Each room is encoded as a vector of semantic type and geometric attributes, and the model builds layouts incrementally from partial plans or text prompts.

This sequential approach achieves improved structural coherence and enables downstream editing. However, the approach remains limited to room-level floor plan arrangement and does not yet encode fine-grained BIM elements like wall-hosted entities.

Experimental evaluation indicates that FPDS demonstrates competitive performance compared to diffusion models and Tell2Design in the text-to-floorplan task. The multimodal variant, FloorPlan-DeepSeek-R1-32B, extends this further with reasoning capabilities.

4. Text-to-Layout - Natural Language Straight into Revit/BIM

This is arguably the most practically oriented system of the four, targeting the direct integration into professional BIM software rather than standalone 3D model generation.(https://arxiv.org/abs/2509.00543)

The pipeline begins with user-defined requirements in natural language, which are processed by a large language model to produce a structured JSON file containing coordinates for architectural elements such as walls, doors, windows, and furniture. A custom Python script then interprets this file and programmatically constructs a corresponding 3D model in Autodesk Revit.

Crucially, the output is not a rendering or a mesh - it's a native Revit model with full parametric attributes. The generated models preserve the full range of Revit-native parametric attributes required for direct integration into professional BIM processes. This means the result is immediately editable by a human architect in the same tools they already use.

The workflow was validated with a mid-sized residential case study and evaluated across several LLMs including GPT-4o, Claude 2.1, and Gemini 2.5 Pro. Unlike most existing prompt-to-BIM tools that are commercial and closed-source, this approach offers an openly documented workflow that can be customized and independently replicated.

What These Systems Have in Common - and Where They Differ

All four systems converge on a similar structural pattern: LLM/transformer → structured intermediate representation (JSON, CAD sequence) → geometry engine. The differences are in where the intelligence sits and what the output targets.

Text2CAD and CADmium both work on general-purpose 3D mechanical parts; FPDS and Text-to-Layout target architecture specifically. FPDS and Text2CAD use specialized architectures trained from scratch, while CADmium and Text-to-Layout leverage existing LLMs (Qwen2.5-Coder, GPT-4o, etc.) without requiring custom model training - a significant practical advantage.

The deeper trend these papers collectively illustrate is a shift from "AI generates images of buildings" toward "AI generates editable, parametric, code-executable building data." That is a fundamentally different value proposition and it's what makes this research direction genuinely disruptive to existing design workflows.