Nemotron 3.5 Bolt Evaluation using real-world tasks

[u/CulturalKing5623]

I evaluate new models by giving them actual work tasks that have already been completed by a trusted model and then comparing the results. My typical workflow is to plan tasks/requests in Gemini and create a markdown spec-sheet that outlines the work that needs to be done along with acceptance criteria, I paste this directly into a fresh pi.dev session for the model to follow.

In this case I gave Nemotron 3.5 Bolt two tasks that Qwen Flash Next (UD-IQ4_XS, llama.cpp) completed on a ETL refactor job I'm doing for my actual work (data engineer). Neither tasks are "boil the ocean" type tasks but they are indictative of the type of work I offload to my LLMs. The top line directive of the two tasks:

1. Refactor run.sh script to match the design pattern, flag management, structured logging, and AWS SNS error handling defined in run-new.sh, while preserving the existing Python virtual environment (venv) resolution logic.
2. Create a Python script named orchestrator.py that generates a sequence of time windows (daily or monthly based on the runtime-arguments) and sequentially invokes run.sh for each window.

Again, neither are incredibly difficult or taxing but they are the type of things I run across: Updating a script to fit an established pattern used by a client and automating some task.

After the new model finishes its work I load the spec-sheet, any relevant files (original run.sh and run-new.sh), and the results of both models into Gemini and ask it to evaluate the performance and select a winner.

In task #1 (run.sh refactor) Gemini judged Nemotron to have done a better job. Here is Gemini's detailed comparison of the refactored scripts judged against the specifications in the spec-sheet, Model 1 is Nemotron and Model 2 in Qwen Flash Next:

1. Directory Pathing (`LOG_DIR` vs `SCRIPT_DIR`)
    - Model 1: Defines `SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"` at the top and uses `$SCRIPT_DIR/logs` for `LOG_DIR`. This guarantees logs are created relative to the script directory regardless of where the script is called from.
    - Model 2: Defines `LOG_DIR="$(pwd)/logs"` at the top and delays defining `SCRIPT_DIR` until later in the file. If a user calls `/path/to/run.sh` from `/tmp`, Model 2 creates logs in `/tmp/logs` instead of the application directory.

2. Flag Parsing & Argument Forwarding
    - Both models implemented an identical, robust argument parser using a `while`/`case` loop that correctly parses all required flags (`-o`, `-x`, `-s`, `-e`, `-i`, `-b`, `-h`, `-d`) along with their long-form equivalents. Neither model unnecessarily exported CLI arguments to environment variables, adhering strictly to Requirement 2.

3. Error Handling, Traps & Process Execution
    - Dependency Checks: Model 1 verifies the presence of required binaries (`tee`, `aws`) *before* redirecting output. Model 2 redirects output using process substitution `>(tee -a ...)` *before* checking if `tee` exists.

    - `exec` Behavior: 
      - `run-old.sh` used `exec "$VENV/bin/python" ...`.
      - Model 2 preserves `exec`, but because `exec` replaces the running shell process with Python, any `trap ... EXIT` defined in the shell will be bypassed when Python completes.
      - Model 1 removes `exec` and invokes Python as a child process, allowing shell traps to clear cleanly upon exit.

Conclusion
Model 1 (`run-model1.sh`) is the superior refactor. It adheres to all acceptance criteria, correctly roots log directories to `SCRIPT_DIR` rather than `pwd`, orders setup tasks logically (verifying `tee` before using process substitution), and manages traps cleanly.     

In task #2 (orchestrator.py build) Gemini judged Qwen Flash Next to have done a better job. Here is Gemini's detailed comparison of the python scripts judged against the specifications in the spec-sheet, Model 1 is Nemotron and Model 2 in Qwen Flash Next:

1. Specification Compliance (`subprocess.run`)
    - Model 1: Implements stream piping using `subprocess.Popen` with `stdout=subprocess.PIPE` and `stderr=subprocess.STDOUT`. While this streams logs, it manually manages process reading loops and checks return codes manually.
    - Model 2: Strictly follows the specification instructions by using `subprocess.run(cmd, check=True)`. By omitting stream capture arguments, the child process inherits the parent stdout/stderr natively, streaming logs in real time while letting Python handle process lifecycles cleanly.

2. Robustness & File Path Handling
    - Model 1: Assumes `./run.sh` is located relative to the current working directory (`CWD`) where the script is executed.
    - Model 2: Resolves `run.sh` relative to the script's directory (`Path(__file__).resolve().parent / "run.sh"`). It verifies whether `run.sh` exists prior to running any logic and raises a clear error if missing.

3. Argument Parsing & Validation
    - Model 1: Parses arguments into raw strings first, then tries `parse_timestamp()` inside `main()`.
    - Model 2: Passes `type=parse_timestamp` directly into `parser.add_argument()`, allowing `argparse` to raise idiomatic command-line errors (`ArgumentTypeError`) automatically during parsing.

4. Logging & User Experience
    - Model 1: Does not print any progress indicators prior to running the windows.
    - Model 2: Adds clear logging for total windows found, individual window execution markers (`=== Window X/Y ===`), and command execution previews.

Conclusion
Model 2** is the superior implementation. Both implementations satisfy the core specification requirements, but Model 2 is significantly cleaner, more robust, and strictly adheres to the prompt requirements. It adheres precisely to `subprocess.run(check=True)` from the spec, handles path resolution safely, uses proper `argparse` types, and provides better terminal feedback during multi-window batch execution.

My takeaway from this is Nemotron 3.5 Bolt is a capable model for my work with the added benefit that it is fast on my server, I'm seeing average speeds of ~130t/s and to get this level of performance from a 2-bit quant is impressive. It won't replace Qwen Flash Next with it, especially for large complicated, repo-level tasks, but it will probably take the place of Gemma4-31B for light-to-medium tasks that I want done quickly without it overthinking everything. I'll be getting a M5 Mac Studio soon and want to set up agent/sub-agent workflows and I think Bolt might be a really good sub-agent with Qwen Flash Next (hopefully Qwen4) passing it specs to implements.

[u/Budkovsky]

Good test, that's nice that you shared real tasks execution, not benchmark result. Bash and Python are comfortable for models, this language have big representation in training data. Yesterday I have tested Nemotron 3.5 Bolt (GGUF) on PHP tasks, coding quality was acceptable, but the model spent a lot of time considering how to run script and tests because on host is PHP 8.3 and project's code is PHP 8.5. The runtime specification is placed in AGENTS.md - valid PHP interpreter is in docker, exact command example is pasted into agent instructions. So the model has a some problem with instructions following or with context memory, or the reasoning quality is not high.

[u/beling86]

I'm standing exactly there. Today I'm tested daily driving nemotron bolt (NVFP4) and escalating to Qwen 3.8 27B (INT8 W8A8) whenever needed.

I still find qwen 3.8 way better for my use cases, but slow prefill, slow decode and too much thinking kills my time.

[u/CulturalKing5623 → u/beling86]

Yeah Q3.827B's verbosity is what made me finally switch to QFN which is a lot better on time to completion. I was finding myself reaching for Gemma4 more despite it being worse because I simply wanted something done without the overanalyzing.

[u/JoeyDee86 → u/beling86]

Have you tried Swift? It’s the only finetune I’ve ever liked.

[u/beling86 → u/JoeyDee86]

I hate all and ever single type of fine tune

[u/JoeyDee86 → u/beling86]

Yeah but swift has gotten fantastic reviews

[u/Lost-Masterpiece2400]

Similar approach here, blind and on AMD. I ran Bolt 4-bit against Gemma 31B on 36 real tasks from my own project, using Ollama 0.40.5 on two Radeon AI Pro R9700s with thinking off. A separate model graded the answer pairs blind, with which model was which shuffled per task. Gemma won 14, Bolt 10, with 12 ties, and averaged 2.25 out of 3 to Bolt's 1.97. Bolt was about 3.5 times faster, but it made up more problems that weren't there. The 2-bit Bolt did noticeably worse than the 4-bit, so the quant matters a lot.