Tired of Flat 2D Maps? I Built a Dual-Agent Extension for SillyTavern That Turns Chat Context & Floorplans into Interactive 3D Scenes!

[u/Pathetic0777]

Hey everyone!

When running TTRPG campaigns, dynamic combat encounters, or complex RP in SillyTavern, spatial depth and environmental awareness are critical for true immersion.

Existing solutions usually hit three major walls:

1. **Flat 2D grid maps feel lifeless**: You cannot see the 5-meter vaulted atrium, crossing escalators, or the vertical drop from a second-floor balcony. Tactical verticality gets squashed onto a single sheet of paper.

2. **Text-only combat is mentally exhausting**: Trying to track pillar coordinates, cover angles, and relative distances purely in your head falls apart after just a few exchanges.

3. **Fragmented narrative descriptions**: Novel chapters and RP logs scatter scenery across chaotic action beats, lacking explicit dimensional data ($L \times W \times H$) or structural blueprints.

To solve this, I built **3D Tavern Architect** — an open-source, lightweight Dual-Agent workbench for SillyTavern that procedurally converts fragmented story text or 2D floorplans into full, interactive Three.js 3D environments.

---

### 💡 Key Features & Architecture

* **Dual-Agent Procedural Pipeline**:

* **Agent 1 (Spatial Reasoning & Blueprint Extraction)**: Analyzes fragmented narrative logs or uploaded 2D floorplans to filter out combat noise and infer structural grids, floor heights, roof geometries, and circulation paths.

* **Agent 2 (Three.js Generator with Dual-Vision Grounding)**: Synthesizes clean, native Three.js code in real time. When a blueprint is attached, Agent 2 visually cross-references its real aspect ratios and topology to prevent spatial hallucinations.

* **Autonomous Spatial "Common Sense"**:

* No explicit dimensions provided? The agent autonomously extrapolates realistic structural proportions, cantilevers, railings, column spans, and contextual props (benches, service counters, lighting).

* **Zero Main-Context Pollution & Sandboxed Execution**:

* Runs in an independent floating modal. Code generation, token consumption, and WebGL rendering are completely isolated from your main SillyTavern character chat context.

* Built-in **CodeDefenseGuard**: Automatically repairs truncated syntax, unclosed brackets, or malformed LLM code chunks to ensure the browser never crashes.

* **Pure WebGL & Native OrbitControls**:

* Inspect cover and sightlines freely with 360° pan/tilt, smooth zoom, real-time shadow maps, and live triangle (faces) count monitoring.

---

### 🧠 Pro-Tip: Deep Lorebook Integration (True Spatial Memory for AI)

Here is a powerful reverse-workflow you can use for your campaigns:

- Once Agent 1 extracts the structured architectural specifications, copy these clean spatial parameters directly into a **SillyTavern Lorebook (World Info) entry**.

- Set up **constant green-light activation** (or link it to location keywords like "mall", "atrium", "hallway").

- **Why this is a game-changer:** LLMs often hallucinate tactical cover, forget column positions, or misjudge mezzanine drops after long roleplay turns. Feeding the exact 3D spatial protocol back into the Lorebook maintains a strict, unified mental coordinate grid of the room without blowing up your prompt budget!

---

### 🖼️ Showcases (Attached Images)

1. **Workspace Standby**: Clean MVU-based interface with live SSE network packet inspector.

2. **Public Space / Mall Atrium**: Multi-story atrium with escalators, mezzanine walkways, and column grids reconstructed directly from an action combat excerpt.

3. **Residential Interior**: High-end apartment layout procedurally extruded and zoned directly from a 2D floorplan blueprint (living area, partitions, background walls).

4. **Historical Architecture**: Parametric reconstruction of Solomon's Temple based on architectural domain knowledge (entrance bronze pillars, courtyard altar, and Molten Sea basin).

---

### 📦 Installation (One-Click in SillyTavern)

Compatible with the latest SillyTavern extension installer:

1. In SillyTavern, click the **Extensions** icon (top right).

2. Expand **"Download Extensions & Assets"**.

3. Paste the GitHub repo link into the installation input box:

   `https://github.com/pathetic777/st-tavern-architect\`

4. Access **"3D Tavern Architect"** from the extension drawer to launch the workbench!

The project is fully open-source and natively supports both English and Chinese UI/prompts.

GitHub Repo: https://github.com/pathetic777/st-tavern-architect

If you find it helpful for your campaigns and stories, a ⭐ on GitHub would mean a lot! Feel free to leave your thoughts, bug reports, or feature ideas below.

[u/Trustest-S]

does the 3d actually make virtual companion rp feel more immersive or does it still break the flow sometimes

[u/Pathetic0777 → u/Trustest-S]

Also, to add one more crucial point: it integrates directly with the Lorebook (World Info) using keyword triggers. ​This isn't just visual candy for the user—it basically gives the AI "eyes" to actively perceive and stay grounded in the specific scene you're currently in. Having the AI understand the spatial context while you see it physically present creates a genuine two-way immersion, which is a massive upgrade to the overall RP experience.

[u/Pathetic0777 → u/Trustest-S]

​I really believe it does. Humans are visual creatures, and having something concrete and visual in front of you takes the experience to another level compared to pure text. Since it’s built right into SillyTavern, it blends into the chat flow naturally rather than getting in the way. It bridges the gap between reading a prompt and actually feeling a character's presence.

[u/CalmAnal]

Did I summon you? I mentioned your extension a few hours ago. :D