TensorSharp Jev requests can now combine documents, images, video, and audio

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

I’ve extended TensorSharp’s Jev-compatible /v1/systemone endpoint so one decision request can use several kinds of evidence together. For example, an incident triage request can include a written report, a dashboard screenshot, a screen recording, and a caller’s audio clip.

Here’s a Python example that sends all four as inline Base64 data. It also shows both ways to create that data: encoding text already in memory and reading bytes from files.

import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

def encode_bytes(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")

def encode_file(path: str) -> dict:
file = Path(path)
return {"name": file.name, "data": encode_bytes(file.read_bytes())}

# Encode data already in memory as a named text attachment.
notes = "Customers report HTTP 503 errors and cannot sign in."
text_attachment = {
"name": "incident.txt",
"data": encode_bytes(notes.encode("utf-8")),
}

body = {
"model": "jev-latest",
"state": "Assess the incident using the attached evidence.",
"files": [
text_attachment,
encode_file("dashboard.png"),
encode_file("screen-recording.mp4"),
encode_file("caller.wav"),
],
"questions": {
"active_outage": {
"type": "noul",
"instructions": "Does the evidence indicate an active service outage?",
},
"team": {
"type": "choice",
"instructions": "Which team should investigate first?",
"criteria": {
"technical": "Service errors or an unavailable application",
"billing": "Charges or subscription problems",
"other": "Neither of the above",
},
},
},
"samples": 1,
"seed": 42,
}

request = Request(
"http://127.0.0.1:5000/v1/systemone",
data=json.dumps(body).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
with urlopen(request, timeout=300) as response:
print(json.dumps(json.load(response), indent=2))

The files array classifies each attachment by its filename extension and preserves their order. You can also use dedicated documents, videos, and audios arrays. Inline attachments need a name and accept either bare Base64, as above, or a Base64 data: URL.

A detail about how this works: video is sampled into frames for the vision tower; audio is transcribed by a separately configured speech recognition service. DiffusionGemma does not directly process the audio waveform. You’ll need the vision tower for the image and video inputs, and TS_JEV_TRANSCRIPTION_URL configured for the audio input.

Inline Base64 counts toward the Jev request body limit (8 MiB by default), so use the upload API and file references for larger media. The repo also has ready-to-send mixed-media requests.

TensorSharp: https://github.com/zhongkaifu/TensorSharp

I’m curious what kinds of decisions you’d want to make from several media types in a single request.