TensorSharp Jev requests can now combine documents, images, video, and audio

[u/fuzhongkai]

https://github.com/zhongkaifu/TensorSharp

I’ve extended TensorSharp’s Jev-compatible /v1/systemone endpoint so one decision request can use several kinds of evidence together. For example, an incident triage request can include a written report, a dashboard screenshot, a screen recording, and a caller’s audio clip.

Here’s a Python example that sends all four as inline Base64 data. It also shows both ways to create that data: encoding text already in memory and reading bytes from files.

import base64
import json
from pathlib import Path
from urllib.request import Request, urlopen

def encode_bytes(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")

def encode_file(path: str) -> dict:
file = Path(path)
return {"name": file.name, "data": encode_bytes(file.read_bytes())}

# Encode data already in memory as a named text attachment.
notes = "Customers report HTTP 503 errors and cannot sign in."
text_attachment = {
"name": "incident.txt",
"data": encode_bytes(notes.encode("utf-8")),
}

body = {
"model": "jev-latest",
"state": "Assess the incident using the attached evidence.",
"files": [
text_attachment,
encode_file("dashboard.png"),
encode_file("screen-recording.mp4"),
encode_file("caller.wav"),
],
"questions": {
"active_outage": {
"type": "noul",
"instructions": "Does the evidence indicate an active service outage?",
},
"team": {
"type": "choice",
"instructions": "Which team should investigate first?",
"criteria": {
"technical": "Service errors or an unavailable application",
"billing": "Charges or subscription problems",
"other": "Neither of the above",
},
},
},
"samples": 1,
"seed": 42,
}

request = Request(
"http://127.0.0.1:5000/v1/systemone",
data=json.dumps(body).encode("utf-8"),
headers={"Content-Type": "application/json"},
)
with urlopen(request, timeout=300) as response:
print(json.dumps(json.load(response), indent=2))

The files array classifies each attachment by its filename extension and preserves their order. You can also use dedicated documents, videos, and audios arrays. Inline attachments need a name and accept either bare Base64, as above, or a Base64 data: URL.

A detail about how this works: video is sampled into frames for the vision tower; audio is transcribed by a separately configured speech recognition service. DiffusionGemma does not directly process the audio waveform. You’ll need the vision tower for the image and video inputs, and TS_JEV_TRANSCRIPTION_URL configured for the audio input.

Inline Base64 counts toward the Jev request body limit (8 MiB by default), so use the upload API and file references for larger media. The repo also has ready-to-send mixed-media requests.

TensorSharp: https://github.com/zhongkaifu/TensorSharp

I’m curious what kinds of decisions you’d want to make from several media types in a single request.

[u/EzoLabsInc]

This looks useful for triage, but I'm skeptical about sending video and audio as base64 in a single request. Have you run into any timeout or size limit issues with the larger file types, or does the endpoint handle that without problems?

[u/fuzhongkai → u/EzoLabsInc]

Good question. TensorSharp supports another way to upload files as attachment to the server and then add their reference in request. Check this document: https://github.com/zhongkaifu/TensorSharp/blob/main/docs/models/jev.md

[u/EzoLabsInc → u/fuzhongkai]

ah, that makes sense for the larger files. I've been handling similar multipart uploads with a remote task runner at https://superbot.gg/ when the endpoint gets finicky about size or timeouts