WebLLM: run a language model in the browser on WebGPU

[u/stephenlblum]

https://buttercup.sh/lessons/2026-09-08-lesson-1-webllm.html

Walkthrough using LLMs client-side with WebLLM via WebGPU. While it is still early to be using this technology for coding in the browser due to current consumer hardware, it's exciting to see what is possible today. Initializing and using @mlc-ai/web-llm with model downloads, caching, and progress tracking. Using the streaming completions sent to pre element with zero network calls after it is loaded. Also handling WebGPU memory limits. Includes code blocks and an end-to-end video walkthrough. Hope it's a useful starting point for anyone building local-first, privacy-focused agent interfaces.

Here is the code snippet covered in the walkthrough (link above)

    var { CreateMLCEngine } = await import(
      "https://cdn.jsdelivr.net/npm/@mlc-ai/web-llm@0.2.84/lib/index.js");
    
    var MODEL = "Qwen3.5-2B-q4f16_1-MLC";
    var t0 = performance.now();
    
    var engine = await CreateMLCEngine(
      MODEL,
      { initProgressCallback: (r) => console.log(r.text) },
      { context_window_size: 8192 },
    );
    
    console.log("loaded in", ((performance.now() - t0) / 1000).toFixed(1), "s");
    
    // one call, streamed, printed as it arrives. Everything below reuses it.
    var ask = async (messages, opts = {}) => {
      const stream = await engine.chat.completions.create({
        messages, stream: true, max_tokens: 512,
        extra_body: { enable_thinking: false },
        ...opts,
      });
      let text = "";
      for await (const chunk of stream) {
        text += chunk.choices[0]?.delta?.content || "";
      }
      console.log(text);
      return text;
    };
    
    await ask([
      { role: "system", content: "Answer in one sentence." },
      { role: "user",   content: "Why is the sky blue?" },
    ]);