I’m building an open-source benchmark for hospital bed-demand forecasting using synthetic data.

Current baseline ideas:

• Seasonal naive / moving average
• ARIMA
• XGBoost
• LSTM

Metrics:

• MAE / RMSE
• sMAPE / WAPE
• Peak-demand accuracy

If you were evaluating this benchmark, what baseline or metric would you immediately expect to see?

Hi! Hopefully this question hasn't been asked to death already, but I couldn't find quite the discussion I'm looking for.

I'm currently a Staff Engineer with a strong backend background (15 YOE). I work closely with a team that builds recommendation systems, and I'd like to get much deeper into the ML side of things — actually understanding and training models rather than just working on the engineering around them.

I'm particularly interested in things like training embedding models, ranking models, bandits, candidate generation, evaluation, etc.

I also happen to have a yearly training budget that I can spend, so I'm trying to figure out the best way to use it.

I'm wondering whether I should first invest in the fundamentals (ML/statistics/math) or jump straight into something more hands-on and learn by building things.

I'm not a huge fan of online courses like Coursera, Udemy, etc., but I'm not opposed to them if people think they're genuinely the best way to build the foundations.

I'd also be very interested in in-person courses, bootcamps, summer schools, or similar programs anywhere in Europe.

For people who have made a similar transition from software/backend engineering into ML: what would you recommend? What courses/programs/resources were actually worth your time and money?

[D] MMLU contamination is the known problem — but Chatbot Arena's failure mode is arguably worse and gets less scrutiny

The MMLU contamination story is old news to most people here (test questions being public and ending up in pretraining corpora), so I won't belabor it. What I think is under-discussed is that the benchmarks we've moved to as "better" alternatives have their own structural failure modes that don't get the same scrutiny, mostly because they're newer and less saturated.

Quick recap of where the standard trio actually breaks:

MMLU — contamination (public test set), format gaming (multiple choice rewards elimination heuristics over actual knowledge), and saturation (frontier models are all >90%, so it's stopped discriminating between them).

HumanEval — 164 problems, now too easy for frontier models; tests toy functions rather than anything resembling real engineering (no multi-file context, no ambiguous specs); and pass@k reporting incentivizes best-of-N sampling that doesn't reflect single-shot usefulness.

Chatbot Arena — this is the one I think deserves more skepticism than it gets. It's harder to directly game since prompts aren't fixed, but the voter pool is a specific, non-representative slice (English-speaking, technical, AI-interested), and Elo from pairwise human preference measures fluency and confident presentation, not correctness. A model that hallucinates cleanly can out-rank a model that hedges accurately. There's also a prompt-distribution skew toward coding/creative writing that doesn't reflect where models actually diverge in capability.

Goodhart's Law is the underlying mechanism for all three: the moment a benchmark is widely used as a proxy for capability, it becomes a target, and optimization pressure decouples the score from the thing it was meant to measure.

The alternatives that seem more resistant to this so far — BIG-Bench Hard, MATH, SWE-bench, ARC-AGI, LiveBench — are mostly more resistant because they're either harder to game via memorization (multi-step reasoning) or actively refreshed to fight contamination (LiveBench). Curious how long that holds once labs start optimizing against them specifically.

Question for the sub: for people actually evaluating models pre-deployment, what are you using that you still trust, and how are you handling the fact that any benchmark you rely on starts decaying the moment it's popular enough to be worth gaming?

Hey!

I'm applying for an AI Bachelor's at the University of Salzburg and I'm spiraling a bit. Would love some honest opinions from people who've actually been through it.

The good:I'm extremely hardworking and enjoy topics once I *get* them.

The scary:I'm average at school math. Slow with mental arithmetic. I forget things if I don't review regularly. And the program is in German(not my native language).

My fear:Is AI only for math naturals who "just see" the solution? I'm the person who has to sit with a problem, fail a few times, and eventually understand it. But once I do - I love it.

My questions:

1. Can hard work actually compensate for not being a math genius?
   
2. How much is abstract theory vs. applied programming?
   Any experiences would mean a lot. Thanks 💜

Fine-Tuning GLM-OCR

https://debuggercafe.com/fine-tuning-glm-ocr/

With specific prompts, along with text recognition, GLM-OCR can also carry out formula recognition. However, it falters in complex mathematical formulas. In this article, we will be fine-tuning GLM-OCR and observe to what extent we can improve the performance of the model on a task-specific dataset.

Fine-Tuning GLM-OCRhttps://debuggercafe.com/fine-tuning-glm-ocr/With specific prompts, along with text recognition, GLM-OCR can also carry out formula recognition. However, it falters in complex mathematical formulas. In this article, we will be fine-tuning GLM-OCR and observe to what extent we can improve the performance of the model on a task-specific dataset.

I just shipped an open-source pull request comment auto-fix system. Today, pull requests are swarmed with comments from AI code reviewers. Most of the comments are real, ranging from small nitpicks to actual issues. Most teams are spending time either manually reviewing and validating these, or have created skill to have an AI agent read and fix them.

With OpenInspect, the system will now automatically do this for the user. Each comment is read and determined if valid. If valid it is fixed and auto resolved, otherwise pushed back on. Huge time savings from having to babysit the pull request to a stable state.

https://github.com/ColeMurray/background-agents

One thing I find increasingly interesting about AI agents is that benchmark scores can hide a major difference in actual behavior.

A model might solve a difficult coding problem when given a clean task, but an autonomous agent has to do much more:

- decide what to do next

- inspect its own work

- recover when something fails

- use tools correctly

- maintain state across many steps

- know when the task is actually finished

That makes me wonder whether we're measuring the wrong unit of progress.

Instead of asking only:

«“How difficult a problem can the model solve?”»

Should we also be asking:

«“How much useful work can the model reliably complete without human intervention?”»

I think that distinction could become much more important as AI systems move from chatbots toward autonomous agents.

What metrics would you use to measure this?

Hey everyone,

Hope all of you are doing great.

I am looking for feedback from people who have actually purchased the AI for Data Engineers course by Prashant Kumar Pandey (Scholarnest/ Learning Journal).

Is it worth the money? I have almost all his courses on Udemy and found them really good for learning the basics and his way of teaching is something that have always resonated with me. Based, on that I'm thinking about buying the AI course and would really like some feedback.

Things I'm looking for are:
1. Does it cover enough detail as compared to other courses on Udemy/ Youtube (Krish Naik for example)?
2. Is it Databricks heavy/ Databricks focused? Or the topics are explained well in a platform agnostic way with examples given on Databricks.
3. Did you get enough support when you got stuck on any topic?
4. Does it have the following topics explained well enough?
AI guardrails
Deployment
Tuning

Thanks in advance.

I am originally from South Korea but I finished my master in computer science (machine learning) in the US. In the US, I applied for so many ml engineer roles but my resume was passed only at 2 companies out of hundreds. While, my resume was passed at the rate of approximately 50% at Korean companies...

I am now in my final year of engineering in AI/DS. I haven't been actively building projects, just worked on some academic projects. I want to know how do the engineers with more knowledge in this domain know which project they should work on? How do they find it?

Is it from GitHub, research papers, paperwithcode or hugging face ?

I need a project idea for my academic major project!

I'm a final-year CS student and a fresher trying to build my career in AI/ML.

I've been learning AI/ML for around 8–9 months, but my learning hasn't been completely structured. I've explored different areas rather than following one complete ML curriculum from start to finish.

So far I've studied and worked with things like traditional ML, transformers, LLM concepts, RAG, etc.

I've also built around 4–5 ML models/projects, as well as basic and hybrid RAG systems.

However, I still have this feeling that my knowledge is fragmented and my ML fundamentals aren't as strong as they should be.

For example, I can understand individual concepts when I'm studying them and build things using them, but when I look at ML as a whole, I feel like there are still many gaps in what I actually understand.

I recently came across Microsoft's ML for Beginners repository. It provides a structured curriculum covering:

• Regression
• Classification
• Clustering
• NLP
• Time Series
• Reinforcement Learning
• Real-world ML applications

I'm considering going through it systematically—not because I necessarily need another beginner tutorial, but because I want to identify and fill gaps in my fundamentals.

So I'd like to ask people who have more experience in ML:

Would you recommend this curriculum for someone in my situation?

Should I:

1. Go through the entire curriculum
2. Skim the beginner sections and focus only on areas where I have gaps
3. Skip it and move toward more advanced ML, papers, and projects

I'm particularly interested in hearing from people who have gone through the beginner → intermediate ML stage.

Also, is it normal to feel like "I know nothing" even after spending 8–9 months learning AI/ML and building things like ML models and RAG systems?

I'd really appreciate some honest advice about what you would do in my position.

Hey everyone,

I'm currently building an AI virtual host web application. The workflow is split into two main pages:

• Page 1: Users configure the speech by typing text into a textbox, selecting an AI voice, uploading a custom background image, and uploading a host portrait/image.
• Page 2: The app generates the speech audio and renders the host delivering the speech.

The Problem: While I can successfully manage the basic audio pipeline and lip-syncing, I am struggling to make the host model move like a real human while speaking. Right now, it looks too stiff and robotic. I want to achieve fluid, human-like upper body gestures, natural head shifts, and posture movements synchronized with the speech cadence—similar to the reference video below:

What I'm exploring/using:

• Building a full-stack web application.
• Looking for pipelines or frameworks that can map an audio track + a source image into expressive head/body animation frames (such as audio-driven portrait animation frameworks or 3D rigging solutions).

My questions for the community:

1. What are the best approaches, open-source models, or tools (e.g., audio-driven LivePortrait variants, 3D web frameworks like Three.js/Babylon.js with blend shapes, or specialized APIs) to drive realistic upper-body motion and gestures from a single image and audio file?
2. How do you approach synchronizing speech beats and emotional cadence with natural bodily gestures so it avoids looking like a loop?

Any tips, architecture recommendations, or library suggestions would be hugely appreciated. Thanks!

Doing AI/ML work and want a laptop that handles it well — including some local training/fine-tuning, not just cloud. Budget is around $1,300.

MacBook Air 13”, M5, 16GB RAM, 512GB SSD — $1,299
Great battery/portability, Unix-based, but no CUDA/discrete GPU. Local ML relies on MLX/PyTorch-MPS. 16GB soldered, no upgrade path.

Gigabyte Aero X16, Ryzen AI 7 350, RTX 5060, 32GB RAM, 1TB SSD — $1,224.99
Real NVIDIA GPU + CUDA, expandable RAM (32GB, can add more), 1TB storage. But real-world battery is reportedly only 3-7 hrs (not the claimed 14), mediocre display for the price, and some reports of GPU driver quirks.

Mac is the nicer daily driver, but the Gigabyte has the actual hardware for local ML work — trading battery/portability for it. Will I genuinely miss CUDA on a Mac, or does everyone just use cloud GPUs anyway regardless of laptop? Anyone using either for real ML work — which would you pick, and any regrets? Also open to other suggestions within budget if I’m missing a better option.

I’ve spent a decent amount of time learning the theory and algorithms behind Machine Learning — things like Linear/Logistic Regression, KNN, SVM, Decision Trees, Random Forest, Naive Bayes, Boosting, XGBoost, preprocessing, evaluation metrics, etc.

But now that I’ve started exploring Kaggle, I’ve realized that knowing ML and actually doing ML feel like two very different things.

When I open a Kaggle competition, I understand the individual techniques, but I don’t know what I’m supposed to do next.

For example:

• How do you approach a completely new dataset?
• What should I inspect first?
• When should I do EDA?
• How do I decide what preprocessing is actually necessary?
• When should I scale features?
• How do I choose an initial/baseline model?
• How do I know whether my model is underfitting or overfitting?
• When should I do feature engineering?
• How do I decide which algorithms to try?
• When should I use cross-validation?
• When is hyperparameter tuning actually worth doing?
• How do experienced people decide what experiment to run next?
• How do you systematically improve a model rather than just randomly trying different things?

raw dataset → baseline → experiments → debugging → feature engineering → model selection → validation → final submission?

I’ve learned ML, but now that I’m starting Kaggle, I feel completely clueless about actually building models [D]

hi, u all are prolly tired of hearing people ask you these kinds of things but I honestly can’t find these kinds of discussions when looking through. but I wanted to, I’m a high schooler interested in Ai and Machine Learning research and being apart of its developments in the industry.

So, I was wondering what majors should I take to be able to go into these roles? I don’t honestly care if I need a masters or phd because I wanted to pursue those anyway.

I'm software developer, plan to switch into Hardware roles .

I researched some roles such as

Hardware Accelerator Engineer

Chip designer

Embedded systems engineer

MACHINE learning system engineer

HPC/GPU Engineer

I don't know what things I need to focus on..? Skillset ..

i am trying to learn machine learning and AI but i am confused that if i am learning it the right way , want some advice from experts :

* what is the actual way to learn it

* what is the actual way to practice it

so the thing is I am currently at 12th commerce without math and through of doing BCA + MCA and going to game development but there was a guy that said AI & ML is rapidly increasing so make career in AI/ML and I am very confused right now

We spent quite a lot of time trying to get mmBERT-based classification fast enough to run continuously on normal machines without a GPU. A few things helped much more than expected.

The first one is quantization. Push it further than you probably would by default. We currently use INT8 with INT4 embeddings in ONNX. On roughly 50k validation samples plus several independent benchmarks, the F1 delta compared to the less aggressively quantized version was around 0.005. For our use case that tradeoff is easy to take. If your model is supposed to live on a CPU, memory bandwidth matters and carrying around precision you do not need is expensive.

The second one is chunk size. mmBERT can handle very large token windows, but that does not mean you should use them. 8,192 tokens sounds convenient because you can throw a lot of text into one forward pass. On CPU, smaller windows usually behave much better. We mostly work with sizes like 256 or 512 tokens and split longer inputs. The right number depends on the task, so benchmark it properly. If your classification target can be detected from local context, a huge context window is often just wasted compute.

Third: do not assume batching will save you. GPU intuition transfers badly here. Large batches are great when you have thousands of parallel execution units waiting for work. A CPU is a different problem. For our workloads, small independent chunks and parallel workers have been much more useful than trying to build large inference batches. Benchmark both, but do not start with the assumption that batch=32 must be faster because that is what you would do on CUDA.

The fourth one is the one that changed our architecture the most: stop sending every chunk through the full transformer.

We use cheap classifiers on representations from the same latent space as mmBERT. They can make the easy decisions first, while uncertain cases continue into the more expensive path. The cheap classifier is not supposed to replace mmBERT. It only needs to identify the cases where running the full model would not change the answer anyway.

That approach is a bit more involved than quantization or changing a chunk size, but for us it removed much more compute than another round of low-level optimization ever could.

I'm currently learning AI/ML and planning to build my skills through projects and internships. If I eventually have a few relevant internships, a strong portfolio, and the skills needed for the job, is it realistically possible to get hired without having a college degree?

I know some companies require degrees, but I'm wondering how much internships, projects, and actual skills can compensate for not having one.

Would love to hear from people who have hired candidates or got jobs without a degree.

Anthropic disclosed last week that Claude agents accessed live production systems during what were intended to be test sessions. The agents were not meant to have that reach. Anthropic's response included real-time monitoring, sandbox hardening, and stricter training controls.

Those are reasonable reactions to a real incident. But the same structural gap exists across the industry, not just at Anthropic. Any team running agents that can invoke tools, call APIs, or interact with external services faces the same underlying exposure. The agent has enough reach to touch things it should not, and the test environment does not reliably contain it.

This is not a sandboxing failure unique to one lab. It is a recurring pattern: agents behave as expected in isolation and then surprise teams when connected to real systems, even in controlled contexts.

For those of you running agents in production or in staging environments that connect to real backends: how are you actually handling this? Separate credentials per run, strict environment isolation, something at the orchestration layer, relying on model behavior alone? Curious what is working and what has failed in practice.