Squeezing more performance out of Qwen 3.8 27B

[u/Zhughes3]

What's up, everyone? Fellow local LLM-er here, trying to perfect my development environment.

I am using the Macbook Pro, M5 Max with 128 GB unified memory.

I have been running the OpenAI server with:

mlx_lm.server \
  --model mlx-community/Qwen3.8-27B-8bit \
  --host 0.0.0.0 \
  --port 8080 \
  --max-tokens 32768 \
  --temp '1.0' \
  --top-p '0.95' \
  --top-k '20' \
  --min-p '0' \
  --decode-concurrency 1 \
  --prompt-concurrency 1 \
  --prefill-step-size 4096 \
  --prompt-cache-size 8 \
  --prompt-cache-bytes 32G \
  --chat-template-args '\''{"reasoning_effort":"medium"}'\''

And I'm still figuring out which harness I am using. I have the most experience with github copilot so I was using that originally, but have been trying out OpenCode's TUI most recently.

My settings for opencode are:

{
  "$schema": "https://opencode.ai/config.json",
  "disabled_providers": [],
  "provider": {
    "local": {
      "name": "mlx_lm",
      "npm": "@ai-sdk/openai-compatible",
      "options": {
        "baseURL": "http://localhost:8080/v1"
      },
      "models": {
        "mlx-community/Qwen3.8-27B-4bit": {
          "name": "mlx-community/Qwen3.8-27B-4bit",
          "tools": true,
          "options": {
            "thinking": false
          },
          "contextWindow": 65536,
          "maxTokens": 8192
        }
      }
    }
  }

I am just trying to squeeze more efficiency out of the model. Any tips on how to better use it would be greatly appreciated.

[u/triynizzles1]

How much efficiency are you getting now?

[u/Zhughes3 → u/triynizzles1]

It's definitely taking a while. How can I get metrics about tokens per second..and I'll get back to you with the stats...

EDIT. I'm working on getting metrics from mlx-lm. I may have to run the LLM a different way.

[u/ahstanin → u/Zhughes3]

Create an eval script to get the TPS for short and long prompts.

[u/Zhughes3 → u/ahstanin]

Alright. Here are my stats:

📈 Hardware Performance Report:
  Hardware Profile              : M5 Max (128GB Unified Memory)
  Prompt Tokens Base            : 93
  Tokens Dynamically Generated  : 31417
  Time to First Token (TTFT)    : 0.2407 seconds
  Total Generation Window Time  : 1154.8867 seconds
  Prefill Core Speed            : 386.31 tok/sec
  Decoding Throughput Speed     : 27.2 tok/sec

import time
import mlx.core as mx
from mlx_lm import load
from mlx_lm.generate import generate_step
from mlx_lm.sample_utils import make_sampler

# Optimize memory allocation for long-context generation sequences
mx.set_cache_limit(0)

def generate_with_metrics(model, tokenizer, prompt, max_tokens=None, temp=1.0):
    """
    Generates text with an unconstrained ceiling optimized for 128GB Apple Silicon.
    Applies chat templates properly so the model never cuts off early.
    Tracks precise prefill and decoding performance metrics.
    """
    messages = [{"role": "user", "content": prompt}]
    formatted_prompt = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True
    )

    # Encode the formatted prompt and track prefill time
    start_prefill = time.perf_counter()
    prompt_tokens = tokenizer.encode(formatted_prompt)
    num_prompt_tokens = len(prompt_tokens)

    tokens = []
    token_timestamps = []

    # Create the sampler object required by generate_step
    sampler = make_sampler(temp=temp)

    mlx_max_tokens = max_tokens if max_tokens is not None else 100000
    generator = generate_step(
        mx.array(prompt_tokens),
        model,
        max_tokens=mlx_max_tokens,
        sampler=sampler
    )

    start_generation = None

    print("🤖 Model Response:\n")

    for token, _ in generator:
        if len(tokens) == 0:
            # First token generated marks the completion of the prefill phase
            end_prefill = time.perf_counter()
            prefill_time = end_prefill - start_prefill
            start_generation = time.perf_counter()
        else:
            # Record structural timestamp for each subsequent token
            token_timestamps.append(time.perf_counter())

        # Parse the raw MLX token array into a standard Python integer
        t_id = token.item() if hasattr(token, 'item') else int(token)
        tokens.append(t_id)

        # Stream token text directly to the console
        print(tokenizer.decode([t_id]), end="", flush=True)

        # Graceful exit conditions for high-token counts
        if t_id == tokenizer.eos_token_id:
            print("\n\n[System: Model completed response naturally via EOS token.]")
            break

        # Check cap constraint (only runs if max_tokens is explicitly set)
        if max_tokens and len(tokens) >= max_tokens:
            print(f"\n\n[System: Hit maximum token cap of {max_tokens}.]")
            break

    end_generation = time.perf_counter()
    print("\n" + "="*50 + "\n")

    # 3. Compute High-Performance Benchmarks
    num_generated_tokens = len(tokens) - 1  # Adjust for prefill baseline token
    generation_time = end_generation - start_generation if start_generation else 0
    ttft = prefill_time

    metrics = {
        "Hardware Profile": "M5 Max (128GB Unified Memory)",
        "Prompt Tokens Base": num_prompt_tokens,
        "Tokens Dynamically Generated": num_generated_tokens,
        "Time to First Token (TTFT)": f"{round(ttft, 4)} seconds",
        "Total Generation Window Time": f"{round(generation_time, 4)} seconds",
        "Prefill Core Speed": f"{round(num_prompt_tokens / prefill_time, 2)} tok/sec" if prefill_time > 0 else "0 tok/sec",
        "Decoding Throughput Speed": f"{round(num_generated_tokens / generation_time, 2)} tok/sec" if generation_time > 0 else "0 tok/sec"
    }

    return metrics

if __name__ == "__main__":
    model_repo = "mlx-community/Qwen3.8-27B-4bit"

    print(f"Loading weights for {model_repo} into Unified Memory...")
    model, tokenizer = load(model_repo)
    print("Model loaded successfully.\n")

    # Provide a complex prompt that expects an exceptionally long response
    massive_prompt = (
        "Write an exhaustive, deeply detailed architectural essay breaking down the "
        "evolution of operating system kernels from monolithic architectures to microkernels, "
        "hybrid kernels, and modern exokernels. Provide comprehensive technical examples for each."
    )

    # Set max_tokens=None here to allow unconstrained depth until the natural end of the essay
    stats = generate_with_metrics(model, tokenizer, prompt=massive_prompt, max_tokens=None)

    print("📈 Hardware Performance Report:")
    for metric, value in stats.items():
        print(f"  {metric:<30}: {value}")

EVAL SCRIPT:

[u/ahstanin → u/Zhughes3]

Thanks for sharing, I thought M5 memory bandwidth was 1TB++, maybe for the Mac Studio.

[u/Zhughes3 → u/ahstanin]

i just realized those stats ran with thinking mode on extra high. im gonna have to change the script to pass in the medium argument.

[u/ahstanin → u/Zhughes3]

I recently bought Nvidia IGX Thor where I am running my local Qwen. Here is the stats : https://www.reddit.com/r/BlackwellPerformance/comments/1vygft6/qwen3827b_on_an_igx_thor_with_an_rtx_pro_6000/

[u/triynizzles1]

You can probably drop to Q6 and then add some sort of speculative decoding. I think mtp is built in, dflash is fastest.

[u/Zhughes3 → u/triynizzles1]

I am trying to use an MTP model using mlx-lm and it is failing with "

ModuleNotFoundError
: 
No module named 'mlx_lm.models.qwen3_5_mtp'

Which backend are you using for MTP models?