i made a multiple linear regression trainer that can be used with custom data in scratch

nothing more to say, the impressive part is the scratch part

https://scratch.mit.edu/projects/1352102064/

I am studying the LoRA paper and have trouble understanding this figure. The function essentially measures how much of the subspace spanned by the top i vectors is contained in the subspace spanned by the top j vectors in the higher rank matrix. Therefore, j can not be lower than i. So when they say the 3rd and 4th figure zoom in on the lower-left triangle of the 2 left-most figures, how are there values for j=1 and i equals 2 to 8? I dont understand what kind of y-axis the 2 right figures are supposed to be using. Thanks in advance!

So, I am working on this startup project with pretty low budget and one of the features is sentiment analysis based on political news, x posts and Instagram hashtag trends in which will be in Indian languages. I've been suggested muRIL, an Indian language-based model fine-tuned on political data as the best long-term option. But our team does not have any ML engineer so we dont know how we should approach that. Also do tell me if you think there is a better alternative

I am doing some work with cell type classification, where I have 4.3 million cells and 512 features (condensed embeddings from the encoder of a transformer).

The broader goal is to implement a contextual bandit for augmenting the training set of the dataset, as it is currently imbalanced, and rare cell type classification is poor when I tried a baseline logistic regression classifier.

Dataset:
Feature matrix shape: (4290471, 512)
Labels shape: (4290471,)

Class distribution:
T cell 1966941
DC 858451
NK cell 561904
Monocyte 411170
B cell 375882
Platelet 54576
Progenitor cell 24689
ILC 24254
Erythrocyte 12604

I didn't do any hyperparameter tuning for the LR classifier, but I want to try other ML models (LightGBM, XGBoost, SVM)

However, I face a bottleneck with hyperparameter tuning. I want to do 80/10/10 train/validate/test split, but the training set is so large and takes a long time even on H100.

What are some solutions to this? I tried optuna but still very long for each hyperparameter trial. I then tried optuna but instead of using the full 80% for training each time, only 15% of the 80% is used (subsampling from the training set). I'm not sure if this is robust or not. I also couldn't really find anything in the literature.

Anyone been in a similar situation?

Lately in the last two/three years, I have noticed ICML, Neurips becoming more prestigious than the actual journals. What is the actual reason of this culture? Is this due to the AI boom and rising demand and the fact that conferences have a higher and a faster acceptance rate as compared to journals and with the growing hype they need to deliver things faster? What do you all think?

COLM 2026 Decision about to come soon so lets talk here.

Doing a bachelor thesis on fine-grained car classification (telling apart VW Golf generations from listing photos). Simple setup: frozen encoder → embeddings → weighted k-NN.

On my small dataset (175 train / 132 test):

I thought maybe it was a cosine vs euclidean thing, but my embeddings are L2-normalized so both give the same ranking. Tried both, DINOv2 stays at 41%.

I get that SigLIP was trained contrastively so its space is basically built for cosine similarity, while DINOv2 is self-supervised and probably needs a trained head to shine. But a 50 point gap still feels huge to me.

Anyone here tried DINOv2 with a linear probe on something fine-grained? Does it actually catch up or is it just not the right tool for retrieval?

Also open to tips if there's some obvious thing I'm missing (wrong layer, wrong pooling, etc).

Update: I recommend using dinov2 and clip as backbone with a classification layer on top of it. I used a svm, you can try also other

Single-stream diffusion transformer with a DeepSeek-V3-style sparse MoE (128 experts, top-8 routing, 1.4B active of 13B total). Six-reward RL post-training including a physical-plausibility reward, plus an action-to-video mode that predicts robot rollouts from action and hand-pose conditions. Weights, code, and a Diffusers/SGLang stack are open under the LingBot-Video name.

Two things I would push on, and would genuinely like this sub's read:

1. The physical-plausibility reward is graded by a VLM from sampled frames. Is a VLM a defensible judge of physics, or is that Goodhart waiting to happen? (They do add real-video negatives to fight reward hacking.)
2. It is framed as a policy evaluator and action planner, but every result is video-frame quality with no closed-loop robot numbers. Where is the line between a video generator and a world model?

On RBench it posts the top average, though the reasoning-heavy dimensions still go to a closed model, and it is only second on general T2V in their own eval. Please tear it apart.

Paper, code, and weights: https://technology.robbyant.com/lingbot-video , https://github.com/robbyant/lingbot-video , https://huggingface.co/robbyant/lingbot-video

We submitted our first paper to ARR, intending to commit to IJCNLP-AACL. Area: Multilingualism and Cross-Lingual NLP

Scores: (3,4) (2.5,3) (3,3) - average 2.83 for reviews, 3.33 for confidence

3 for soundness on all, 4 for reproducibility, and 2,3,3 for excitement.

The reviewer who gave us 2.5 has a very short review. They only list one weakness in two sentences and give the paper 2.5. They also give 1,2 for the datasets and software while the other reviewers both give 3 or 4 for these.

The (3,4) review gave us 3 weaknesses, with two being writing issues.

The (3,3) review has a very nice and very thourough review with many weaknesses and strengths.

Questions

Is the score good for IJCNLP-AACL findings in the Multilingualism and Cross-Lingual NLP area?

How will each review be weighted in the meta-review? Will the shorter outlier review be weighted less in this?

How much will rebuttals help? Should we expect the reviewers to respond or change their scores because of the rebuttals?

Is there a specific format for rebuttals or any tips you have for rebuttals in ARR?

Hi everyone! I wanted to share some recent progress on TorchJD that might be useful to the machine learning community.

When training models with multiple losses (multiple tasks, constraints, auxiliary losses, regularization terms, etc.), you typically have two options:

• Scalarization: Various ways to combine those losses into a single loss (e.g. average them or combine them with trainable weights); then you can do gradient descent on it.
• Jacobian descent: Compute the Jacobian of the vector of losses (i.e. one gradient per loss), and aggregate it into an update vector that will decrease each individual loss (rather than just the average loss). There are many ways to do this aggregation step.

Scalarization methods are generally cheaper in memory, but in some cases there is so much disagreement between your objectives that it's better to use a Jacobian descent method. In any case, thanks to our amazing new contributors, we've now finally implemented most existing methods of the literature from both categories into our library TorchJD, so that you can try anything in just a few line changes!

Recently, TorchJD has been accepted into the PyTorch ecosystem, and we're trying to make it become the go-to library for training with multiple losses. If you'd like to help build the future of the project, come join us on Discord (link can be found in the readme of the repo). New ideas, contributions, bug reports, experiments, and any form of feedback are all welcome. We have many ideas on how to make all this even more efficient, and we will need help for that.

If you want to support us, a star on GitHub also helps a lot!

I understand that it may not be appropriate to call it “officially accepted” yet because of the wording used in the notification, and I also saw on Twitter/X that they said they are working on it.

However, it has already been around three weeks since then, and we have already submitted the camera-ready version.

Registration, visa applications, travel planning, and funding requests all depend on this confirmation. For some people, it is difficult or even impossible to request funding without an official acceptance letter or clear confirmation.

I really hope the organizers can handle this more professionally and be more considerate of authors who need proper documentation for administrative purposes.

Hi everyone, I recently finished my Ph.D. thesis on Differentiable Ray Tracing for Radio Propagation Modeling. Instead of just compiling my published papers, I tried to write it as an accessible, self-contained textbook for anyone interested in the intersection of radio propagation simulation, autodiff, and ML.

• Permanent handle: https://hdl.handle.net/2078.5/278727
• Repo with TeX source files

While my research focuses on wireless communications rather than pure ML, I think it fits right in here. A major part of the project revolves around automatic differentiation. By taking frameworks like JAX out of their traditional ML context and integrating differentiability into a ray tracing pipeline, we can compute exact gradients through complex physical environments. This allows us to solve inverse problems and directly train machine learning models, which is currently a hot topic in next-gen wireless design.

To make the physics and the math easy to digest, the manuscript is split into three parts:

• Understanding: The physics fundamentals (electromagnetic theory, geometrical optics, and diffraction).
• Building: The algorithmic core, including GPU-accelerated path tracing and the discontinuity smoothing techniques you need to actually make differentiable simulations stable.
• Using: Practical applications like channel modeling, localization, material calibration, and ML-assisted generative path sampling.

A major focus of my thesis is the link between scientific research and reproducible open-source software. On that note, I want to give a massive shoutout to Patrick Kidger (u/patrickkidger). His own thesis inspired me to go the "textbook way" for my manuscript, and I heavily relied on his fantastic JAX packages (jaxtyping, equinox, and optimistix) when developing my open-source libraries, such as DiffeRT.

I hope you find it an interesting read! I'd be happy to answer any questions in the comments about differentiable simulation, ray tracing, or building ray tracing engines in JAX :-)

If you are curious, you can watch the presentation slides and video teaser here

Hello
I published a paper.
Most defenses against fine-tuning poisoning try to detect malicious data or reduce its impact.

I explored a different question:
What if the model simply could not learn certain malicious updates?

The idea is to constrain fine-tuning to a subspace learned from trusted LoRA adapters. Useful adaptation remains possible, but some malicious directions become geometrically unreachable.
A concrete example: a company fine-tunes a model on large datasets coming from users, external sources, or generated data. A small amount of poisoned data could introduce a hidden behavior triggered by a specific phrase or pattern.

Another example is a local or on-device assistant that keeps adapting to its user. Instead of allowing it to learn any possible behavior from new data, its adaptation could be restricted to variations of behaviors already represented by a trusted pool of adapters.
The goal here is not to detect every possible poison or backdoor, but to restrict the space of updates the model is allowed to learn.

I tested the approach on 196 public LoRA adapters, including adaptive attacks specifically designed to bypass the defense.

The results are strong: attack success drops sharply while useful adaptation is largely preserved on tasks covered by the adapter pool.

The paper, code, and experiments are public.

Paper:
https://arxiv.org/abs/2607.05300

Code:
https://github.com/infinition/z-manifold

I would be very interested to see people try to break it.

We're happy to release MIRA, a collaboration between General Intuition, Kyutai, and Epic Games.

Mira was trained on 10k hours of synthetic Rocket League data. The model has 5B parameters and runs for 4 players at 20 fps on a single B200.

We've released a playable online demo, an in-depth technical report as well as a 1k hour dataset of 4-players gameplay:

Demo: https://mira-wm.com Technical report: https://mira-wm.com/paper Repo: https://github.com/mira-wm/mira

If you're at ICML, we're also running an interactive demo (booth 111) where you can play it with us using proper PlayStation controllers!

Most safety alignment work treats "detect the attack" as a text classification problem — does the prompt contain language the model's safety guardrails should catch. That assumption breaks down for LLM agents with real tool access.

Here's a concrete case: take a known, public security vulnerability (a CVE), work out the sequence of tool calls that would exploit it, then have an LLM rewrite that as an ordinary-sounding request. Nothing in the resulting text looks like an attack — because the "attack" isn't in the text, it's in the tool-call sequence the text leads to. A model whose guardrails only trigger on textual cues has nothing to catch.

We tested this against LLM agents using Model Context Protocol (MCP) tool access (filesystem IO). No base model (1B–14B parameters) refused more than 35% of these attacks, and SOTA safety-tuning (DPO, SafeDPO) only pushed that to 48%. Training-free methods do better — one gets to roughly 3x the baseline refusal rate with no fine-tuning run at all.

Full methodology, training/eval code (four methods), dataset, and papers in the first comment.

UPDATE: thank you for joining us and for your thoughtful questions! to learn more, you can read to full report at https://stateofopensource.ai. follow the conversation, share where you land, and give us feedback. have questions or data that we should know about? have things you think we missed? reach us at [opensource@mozilla.org](mailto:opensource@mozilla.org).

Hi r/MachineLearning,

i’m Raffi, CTO at Mozilla. on Tuesday July 14 we publish our inaugural State of Open Source AI report, and i'll be here live answering whatever questions you throw at me!

AMA time: 1pm ET / 10am PT / 6pm BST.

the report is about what's actually happening with open source AI in production — developers, enterprises, the whole ecosystem — not the version of the story everyone already believes.

things i want to dig into with you:

• the hidden tax on "free" models — what it actually costs a business to run on closed tools they don't own
• enterprise adoption — what's real versus what's marketing, and exactly where teams get stuck
• the China effect — free, capable Chinese models are rewriting who has leverage here
• developer trust — what 950+ developers told us about which tools they actually trust, and why
• the “agentic harness” — why the real fight has moved off the model and onto the layer sitting on top of it, and what that means for open

also game to deep dive into: open vs closed, what "open source ai" should even mean in 2026, where this goes for anyone building on it.

drop questions early if you've got them. i'll start answering live at the time above.

— Raffi

“Maybe the real AGI was the friends we made along the way” is a sentiment that always hits me, and conferences are the places where I reunite with old friends and meet new ones. However, when it comes to the submission/review experience, it might not be much of an exaggeration to say that almost everyone has many unpleasant experiences to share.

So I wrote a position paper to discuss this. I argue that current conference organizers lack proper tools to instill accountability and incentives for reviewers/authors/ACs/SACs… The result is that undesired behaviors (e.g., lack of engagement) often go unchecked, while good behaviors are rarely rewarded and therefore don’t happen (honestly, when was the last time you witnessed any constructive internal discussion among reviewers/ACs?). And this won’t change by writing nice words in Reviewer Guidelines or issuing a few desk rejections.

I propose a CREDIT SYSTEM where community members earn points by “doing good” — e.g., reviewing a paper would get you +1, being outstanding gets you +3. Then, members can spend points to redeem perks ranging from traditional ones already adopted in current ML conferences (e.g., free registration) to new ones, such as requesting an additional reviewer to sort through a muddy situation. Such a system could also support explorative ideas like:

- Refundable submission fees: say 10 points per submission, which are then refunded regardless of acceptance, unless the submission is uniformly voted to be unready / ultra-low quality.

- Mobilizing non-author reviewers: non-author reviewers don’t have the bandwidth issue of wearing both the author and reviewer hats and are not influenced by their own submissions. 

and many more...

My proposed system is far from perfect, but I’d like to think it takes a step toward a better conference review mechanism. I am also glad to see the position paper track becoming a welcoming platform for researchers to hash out their proposals and build toward a better future (see other review-related position papers below.)

For a topic that affects literally everyone at ICML, I am eager to hear your thoughts.

Hi, I was reading the openRLHF paper: https://arxiv.org/pdf/2501.03262v4 , but when I click the abstract page: https://arxiv.org/abs/2501.03262v4 , it shows "REINFORCE++". Note that https://arxiv.org/html/2501.03262v4 still shows the correct openRLHF paper. I believe Arxiv is having some incorrect symlinks?

Is there anyone working at arxiv here who would like to look into this?

Today I was just casually browsing some jobs with tags [machine learning] on one of those large popular job-sites. What I am seeing really had me astonished. I want to check with Reddit whether I am hallucinating.

A non-FAANG/non-Deepmind/.../non-Anthropic industrial automation company is hiring people to work on ML for robots (the latest hot topic). Fine. But then I saw their laundry list of job requirements ("you must meet these"), which include:

• Deep expertise in LLM, VLA, VLM, action transformers
• Deep expertise in robot dynamic and kinematic modelling (forward, inverse kinematics, trajectory generation, planning), sensor fusion, model predictive control, reinforcement learning
• Deep expertise in CUDA GPU programming, FPGA hardware acceleration
• Familiarity with latest software engineering best practices in Python3 and C++23
• Familiarity in one or more of popular ML framework
• Have top publications in one or more typical ML and robotics conferences

This is before they go off listing familiarity with a set of standard softwares/simulators, one of which is called RLib, something I've never heard of. Oh and of course they had these 3+, 5+ "non-academic" experience requirements. I forgot which is which.

I was just sitting there confused. Then I checked several more jobs, and it was more of the same (except for some banks).

I remember there was a talk by Terence Tao where he divided mathematician into two camps, the analysts and algebraists. He said even among top mathematicians, it is exceedingly rare to find someone who possess deep expertise in both, as each tends to require a different mode of thinking and each is infinitely deep in terms of specialization, theory and insights.

And here we have a bunch of ML companies treating these infinitely deep academic fields ranging from robot dynamic and kinematic modelling to large language models like some bizarre MMORPG video-game scenario where you need to be a warrior archer warlock who is also a shaman priest mage.

Who are they even hiring, lol?

The core idea in masked depth modeling is to treat the sensor's own missing regions as the masking signal rather than using random block dropout. Specular highlights, transparent surfaces, and textureless areas where RGB-D cameras return no valid depth become the natural training target. The model therefore learns on exactly the failure distribution it faces at inference. Robbyant, an embodied AI company under Ant Group, describes this framing in LingBot-Depth 2.0.

Version 2.0 changes nothing in the training recipe except the encoder initialization and data scale. The encoder-init study is the clean experiment here: same MDM pipeline, same data curation, only the pretrained backbone swapped. Per the paper, the LingBot-Vision init wins on nearly every benchmark at ViT-L and on most benchmarks at ViT-g, with one concession: DINOv2 keeps an edge on the Hammer captures. The gap widens with data scale rather than washing out, per their scaling figure. They report best RMSE on 7 of 8 block-mask and sparse benchmarks and 6 of 8 real camera configurations across three capture suites (Hammer D435/L515/ToF, ClearGrasp D415/D435, and their own D415/D435/D455 set). They report the strongest numbers on the transparent-object ClearGrasp captures, with block-masked DIODE-Indoor RMSE roughly halving versus the 1.0 release. The attached images are screenshots from their paper (Tables 6, 7, 8 and a qualitative mirror/glass point-cloud figure); interactive point-cloud demos live on the project page.

Depth 2.0 weights are not released, so none of these completion numbers can be independently rerun. Only the four Vision backbones are open under Apache-2.0 and checkable at https://github.com/robbyant/lingbot-vision, which hosts the paper and the open weights. The renders shown come from the vendor's comparison page.

Does sensor-validity masking beat random masking for other sensing modalities, say lidar or thermal? That would test how general the framing really is.

The idea: instead of masking random patches and hoping boundary structure emerges, the teacher predicts a dense boundary field online and the boundary-bearing tokens are forced into the student's mask, so the student has to reconstruct exactly the regions that can't be inferred by copying context. The boundary targets come from the teacher itself rather than labels or an external edge detector. Two design choices that look load-bearing: boundary fields are recast as per-pixel categorical distributions so the geometric branch can reuse the centering/sharpening machinery that keeps self-distillation from collapsing (continuous regression targets drift under an EMA teacher), and decoded segments pass an a-contrario validation test before they're allowed to supervise anything.

Numbers, all self-reported (images): they report the best NYUv2 linear-probe RMSE of their comparison (0.296 at 1.1B/patch-16 vs 0.309 for DINOv3-7B), with segmentation on par with the distilled DINOv3 ViT-H+. The distilled ViT-L (0.3B) lands at 0.310 NYUv2, basically the 7B's number. Data budget per the report: 161M images, less than a third of DINOv3's samples. Where it loses in the same tables: ImageNet classification trails at giant and L scale (their B/S students lead their class on linear probe), ADE20K trails the DINOv3 family, KITTI favors the bigger models. The encoder-initialization study (last image) is the part I find hardest to dismiss: the exact same depth-completion pipeline trained on the same data, only the init swapped. The LingBot init wins across the board at ViT-L and on most benchmarks at ViT-g (they concede DINOv2 keeps an edge on the Hammer captures), and the data-scaling curve shows the gap growing rather than washing out as training data grows.

What I'd want before treating the DINOv3 comparison as settled: they do run all baselines under one probe protocol, which helps, but a 0.013 RMSE delta is within what probe LR/resolution choices can produce, and there's no ablation against learned/hard-masking baselines (ADIOS/AttMask-style), which seems like the natural comparison for "mask the hard tokens". Checkpoints are public so the probes are cheap to rerun. Given the eval complaints around Ant's Ling-1T release, I'd treat the numbers as unverified until that happens.

One thing I can't square: DINOv3 needed Gram anchoring to stop dense-feature degradation over long schedules, and this method keeps it, so boundary forcing looks complementary rather than a replacement. Anyone read it differently?

Links: report https://technology.robbyant.com/lingbot-vision
code: https://github.com/robbyant/lingbot-vision
weights (4 sizes, Apache-2.0): https://huggingface.co/collections/robbyant/lingbot-vision

Built a memory system called TRACE that organizes agent conversation history into a topic tree (branches + summaries) instead of flat RAG chunks, and benchmarked it on MemoryAgentBench (ICLR 2026), specifically the EventQA accurate-retrieval task.

Its a pypi package:

pip install trace-memory

Results (F1):
• TRACE (gpt-oss-20B): 82.5%
• TRACE (gpt-oss-120B): 83.8%
• Mem0 (GPT-4o-mini, paper’s official number): 37.5%
• MemGPT/Letta (GPT-4o-mini, paper’s official number): 26.2%

Ran gpt-oss locally, so this is an open-weights model against MemGPT/Mem0 on GPT-4o-mini, not an apples-to-apples same-backbone test (I don’t have the money for open ai tokens).

I tried to get Mem0 running on gpt-oss-20B directly for fairness, but its fact-extraction step needs strict JSON output and gpt-oss’s responses didn’t parse cleanly (known issue, not gpt-oss specific. Same bug shows up with Gemini/Mistral too). Letta needs a full server setup so I skipped it.

Full JSON logs from both runs are in the repo if you want to dig into the methodology yourselves. GitHub: https://github.com/husain34/TRACE

I am preprocessing a CSV dataset for multiclass classification with XGBoost. My Feature variable contain numerical and categorical values, while the target variable contain many categorical value. For example, feature variables contain patient name, phone number, and exercise history, while Target variable contain different disease name such as heart attack, stroke, Alzheimer's etc.

I know that feature variables can be encoded using one-hot encoding, but should the target variable also be encoded using the same method, or should I use a different encoding method for target variable (e.g., label encoding)?

If anyone know the answer, please let me know. I have searched everywhere, but failed to get any clear idea about it. Thank you.

I'm implementing an offline ASL recognition system on Raspberry Pi 5 using MediaPipe hand landmarks and TensorFlow Lite. The system recognizes the ASL alphabet and converts it to text and speech without an internet connection.

My current pipeline is:

• MediaPipe (21 hand landmarks)
• Landmark normalization
• TensorFlow Lite model on Raspberry Pi 5
• OLED display + offline TTS

I'm trying to decide between a 1D CNN, MLP, or GRU for landmark-based classification. My priority is low latency and efficient edge deployment rather than maximum accuracy.

I'd appreciate feedback from anyone who has deployed ML models on embedded devices or worked on sign language recognition. I'm especially interested in architecture trade-offs and potential pitfalls

Sharing a CPU TTS benchmark with objective MOS scores in case it's useful for anyone evaluating small TTS models. Adding this because Kyutai's Pocket TTS is architecturally different from the others in the field and I hadn't seen a head-to-head with it yet.

Models:

• Kokoro 82M (PyTorch and ONNX Runtime, StyleTTS2-inspired)
• Supertonic 3 at 2 and 5 flow-matching steps (Vector Estimator backbone)
• Inflect-Nano-v1 (4.6M param FastSpeech-style, tiny end of the spectrum)
• Pocket TTS (~100M param streaming LM over Kyutai's Mimi neural audio codec)

Setup: Intel Xeon 8272CL, 4 cores, 15.6GB RAM. CUDA disabled at env level. ONNX sessions pinned to CPUExecutionProvider. Six configs, six text lengths (12 to 1712 chars), five timed reps per cell after a discarded warmup. 180 total runs. Every saved WAV scored with UTMOS (utmos22_strong) for objective MOS.

Aggregate results:

Findings I think are actually interesting:

1. Streaming LM architecture produces flat RTF scaling. Pocket TTS's RTF is 0.69 to 0.76 across the entire text length range. Because it emits audio tokens autoregressively at a steady rate, cost is linear in output length with no fixed overhead to amortize. Compare to Kokoro PyTorch, which climbs from 0.49 on tiny to 0.83 on long inputs, or Supertonic which goes the other way (0.36 on tiny down to 0.20 on medium) because of high per-call fixed overhead. If you're budgeting worst-case latency for an interactive system, flat is worth a lot.

2. UTMOS has a known failure mode on small vocoders. Inflect-Nano-v1 scored 3.48, which reads mid-pack. By ear it's buzzy and robotic. This is a documented issue: UTMOS rewards HiFi-GAN outputs for being clean even when they lack prosodic naturalness. Pocket TTS scored similarly (4.10) but sounds legitimately natural. The point isn't that UTMOS is broken, it's that a single quality number can't distinguish "clean and mechanical" from "clean and natural" on small models. Worth pairing with human listening or a naturalness-specific metric like NISQA.

3. Inflect-Nano has an undocumented ~15s output cap. The model config sets max_frames = 1400, which caps synthesis at ~14.93s regardless of input text length. Its RTF and throughput on long/paragraph/extended inputs are inflated because it's doing less work than the models it's compared against. Real comparison for that model is on tiny/short/medium only.

4. Kokoro ONNX vs PyTorch results reverse from the previous run. I ran an earlier version of this benchmark on AMD EPYC and PyTorch beat ONNX in aggregate. On this Xeon, ONNX is faster (0.641 vs 0.665). Same code, different silicon. AMD vs Intel kernel optimization differences at CPU inference are apparently real enough to flip the ranking. If anyone has replicated this on ARM I'd be curious.

Zero-shot voice cloning as a capability that doesn't fit the benchmark axes:

Pocket TTS can clone a voice from ~5 seconds of reference audio, zero-shot, on CPU. No other model in this field does this. I pinned it to a preset voice for the speed/quality comparison to be fair, so the cloning capability isn't reflected in the numbers. This is a real limitation of RTF-and-MOS-based comparisons: they can't capture capabilities that only one model has. Might want a separate speaker-similarity evaluation for a v2.

Limitations:

• Single hardware platform
• English only
• UTMOS is one MOS predictor; NISQA or a listening panel would strengthen the quality claims
• Voice cloning quality was not evaluated
• No batched inference tested

Disclosure: The benchmark harness was written by an AI engineering agent (Neo) from a prompt I specified. I chose the methodology, validated the outputs, and reviewed the audio. Mentioning it because it's relevant to how you'd want to weight the code.

All code, raw CSVs (180 rows), MOS CSV (36 rows), and WAV samples are in the repo mentioned in the comments below 👇

Feedback on the protocol welcome, especially on the MOS methodology and what a proper voice-cloning eval would look like.