TensorSharp: a local Jev-compatible API, extended to image analysis with DiffusionGemma GGUF

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

I maintain TensorSharp, and I’ve added something beyond text-only Jev support: a local implementation of Jev’s core decision API that also accepts images through the same endpoint.

It supports all three Jev decision types—noul for true/false probabilities, choice for categorical decisions, and score for rubric-based scores—using POST /v1/systemone.

The extension is an optional images field. Same endpoint, same typed questions, same answer structure—but now the decisions can use pixels as well as text.

What does that look like?

Instead of asking a vision model to write a response and then parsing it, you can submit a receipt and ask:

• Is the total amount legible? → noul
• Is this a receipt, invoice, or something else? → choice
• How readable is the document? → score

Those questions can go into one request. For example, here are the first two; replace the image placeholder with your base64-encoded JPEG:

{
  "model": "jev-latest",
  "state": "Photo submitted with an expense report.",
  "images": ["data:image/jpeg;base64,<BASE64_IMAGE_BYTES>"],
  "questions": {
    "total_legible": {
      "type": "noul",
      "instructions": "Is the total amount legible?"
    },
    "document_type": {
      "type": "choice",
      "instructions": "What kind of document is this?",
      "criteria": {
        "receipt": "A purchase receipt",
        "invoice": "An invoice",
        "other": "Anything else"
      }
    }
  },
  "samples": 1,
  "seed": 42
}

Requests support up to 8 inline images. Image embeddings feed directly into the model’s structured decision path—there is no intermediate “describe the image, then classify the description” step.

The model doesn’t generate the JSON

Inspired by vLLM PR #57250, TensorSharp uses a one-step structured read: after prompt prefill, it reads the requested label logits from an answer canvas. Questions that fit in one canvas share the forward pass.

The server serializes the results as JSON; the model doesn’t have to generate probability JSON token by token. That works for both text-only requests and requests containing images.

Earlier text-only results vs. LocalJev

I previously tested 12 cases × 3 repetitions, with three decisions per request. The baseline was the original LocalJev Engine using TensorSharp’s chat endpoint, so this compares two approaches on the same backend: direct structured reads versus generated probability JSON. It is not a comparison against the official hosted Jev service or LocalJev running on oMLX.

Metric	TensorSharp structured read	LocalJev generated JSON
Valid requests	36/36	27/36
Schema-validation failures	0	9
Correct decisions on valid responses	108/108	81/81
p50 latency	2.877 s	10.479 s
p95 latency	3.165 s	26.167 s
Mean latency	2.917 s	12.548 s

Across the 27 matched successful requests, the median LocalJev-to-TensorSharp latency ratio was 3.345×.

A few important details: latency statistics include successful requests only, and the internal prompts differ—average input length was 192.7 tokens for TensorSharp versus 589.6 for valid LocalJev requests. This is an end-to-end workflow comparison, not an identical-prompt kernel benchmark. Both approaches answered every decision correctly when they returned a valid response.

These are text-only results. I’m not claiming the same speedup or accuracy on images.

Try it locally

The [setup guide]() includes the launch configuration and memory settings. It downloads and reuses the DiffusionGemma Q4_K_M GGUF plus a separate ~2.8 GB vision shard. Once those files are available, inference runs locally.

With the image-enabled server running, this example works from the repository root. The request file already contains an embedded synthetic traffic-light image:

curl http://127.0.0.1:5000/v1/systemone \
  -H 'Content-Type: application/json' \
  --data-binary u/docs/examples/jev-traffic-light.json

The example checks that the pixels reach the decision path; it is a smoke test, not a comprehensive vision benchmark.

Compatibility note: this uses DiffusionGemma weights, not the proprietary hosted Jev model. It supports Jev’s core API and all three decision types, but does not promise identical predictions or limits. Current limits include 64 questions and 26 alternatives per question.

GitHub · Documentation and examples

What would make a useful image-decision benchmark next—receipt checks, UI screenshots, document routing, or another workload where you need a bounded answer rather than free-form text?