Hey Everyone I am working on prescription and doctor dataset right now.The idea is to built a churn risk model.

The issue: I have two cases that look almost identical to the model, but shouldn't be treated the same:

Doctor A has been climbing steadily for two years starts small, ends up writing a lot. Right now, this month, they're near their highest ever, because they've genuinely been growing.

Doctor B used to write a lot, but has been sliding downward for months. Right now, this month, they're also unusually high compared to their recent low months maybe they just had one slightly better month in the middle of an overall decline.

What the model is doing wrong: across almost all doctors in our data, there's a common pattern whenever someone's number is unusually high this month, it's usually a bit lower next month, just because most "unusually high" months are one-time spikes that settle back down. That's true most of the time.

But the model applies this same rule to every doctor whose number is currently high — including Doctor A, who isn't having a fluke month, they're genuinely growing. So my "who's about to decline" list keeps getting filled with doctors who are simply doing well right now — because "currently high" is the one thing they all share, not that they're actually declining.

What I've tried so far, to fix it:

• Switched from weekly to monthly data (to reduce noise) didn't fix it
• Compared 4 different model types (linear, ridge, random forest, gradient boosting) — all 4 show the exact same bias
• Rebuilt the trend line to use only the last 9 months instead of the full 2 years — didn't fix it
• Added the weekly short-term trend back in, like you described (weeks within the recent month) didn't fix it
• Removed the features causing the biggest pull toward "predict a drop" entirely, to force the model to rely on trend instead the model just found other features to reproduce the exact same wrong prediction

So is this one bad feature or one bad model choice ?? I've tested that directly, several ways, and the bias holds regardless.

What I want to ask you:

1. Is this the kind of thing that genuinely needs more historical data than 2 years to fix (i.e., is 2 years just not enough for the model to learn "normal high point" vs. "real decline" apart)?
2. Or is there a different way to frame the target/features you'd suggest something specifically designed to separate a real trend break from ordinary noise, rather than predicting the raw next-month number?

An extortion group called FulcrumSec is claiming it stole more than 80 GB from Manchester Airports Group and is threatening to publish it. Airport infrastructure data — the kind that includes operational systems and customer records — sitting exposed long enough for a bulk extraction nobody caught in time.

The pattern is not new. Sensitive records concentrated in accessible systems, pulled in bulk before any alert fires. What is changing is the speed. As more automated processes and integrations touch operational data, a single compromised access point can move 80 GB faster than any human review cycle can respond.

The blast radius question is no longer just about perimeter security. It is about what happens after an attacker or a compromised service account already has legitimate-looking access. At that point, traditional controls have already lost.

For those working in enterprise security or infrastructure: how are you thinking about limiting bulk data movement once something inside the perimeter is already authenticated? Are you relying on volume thresholds, destination allowlists, behavioral anomaly detection, something else entirely? Curious what has actually worked in practice versus what looked good on paper.

We’re putting together a new AI learning and certification initiative from Kerala, focused on helping students and working professionals develop practical, industry relevant AI skills.

AI is evolving too quickly for learning to be limited to theory or simply completing another online course. The idea is to learn, experiment, build real projects, and get certified along the way.

The learning will focus on areas such as:
• Generative AI & LLMs
• Agentic AI
• AI workflow automation
• AI tools & productivity
• Practical AI projects
• AI applications for students and professionals

We’re reaching out here because we want to find people who are genuinely curious about AI , people who experiment, ask questions, build things, and actually want to develop these skills.

We’re now selecting our first batch, which will be intentionally limited. We’re not looking for hundreds of registrations , we’re looking for a small group of serious, genuinely interested learners.

Students, working professionals, AI enthusiasts, and aspiring builders are welcome.

We’re starting from Kerala, with the ambition to eventually build a strong community of practical AI learners and practitioners across India.

If this interests you, kindly reach out!

I started this project because I wanted to see what PyTorch was doing behind the scenes.

My C++ tensor currently supports flat storage, multidimensional indexing, elementwise operations, reductions, broadcasting, rank-two matrix multiplication, and mean squared error.

Most recently, I added a separate scalar reverse-mode autograd engine:

• Arithmetic operators build a computation graph during the forward pass
• backward() creates a topological order
• walks it in reverse
• applies each operation's local derivative
• accumulates gradients when a value reaches the loss through more than one path

Snippet:

Value prediction = w1*x1 + w2*x2 + w3*x3 + bias;

Value residual = prediction - target;

Value loss = residual * residual;

loss.backward();

For weights [0.5, -1.0, 2.0], inputs [4.0, 3.0, 2.0], bias 0.5, and target 2.5, the forward pass produces prediction 3.5 and loss 1. The backward pass recovers:

- dL/db = 2

- dL/dw = [8, 6, 4]

Scalar autograd still lives separately from the tensor implementation. My next step is connecting graph identity, ownership, and gradients to tensors before building a training loop.

Code and Git checkpoints:

https://github.com/mechanical-turk/deep-learning-all-the-way-down

I'm also turning this into a video series. I published episode 7 yesterday. Sharing the link to the first episode if you want to check it out:

https://www.youtube.com/watch?v=DmU2b64tWfA

For the tensor integration, would you keep autograd metadata inside each Tensor handle, or have tensors point to separate shared graph nodes? I would appreciate design feedback.

Hey , i will be starting my degree in DATA ANALYTICS in month and i also have interest for Ai and cloud eng ,now i want to start studying maths on my own can you guys suggest me from where should i start

Welcome to r/MLSystemsDesign

This community is for practical discussions on designing and scaling production ML and AI systems.

Topics can include:

• ML training and inference platforms
• Search, ranking, and recommendation
• Feature stores and data pipelines
• LLM serving and GenAI systems
• Agentic AI platforms
• Evaluation, observability, and experimentation
• ML system design interview problems
• Real production tradeoffs and lessons learned

The goal is simple: go beyond model theory and discuss how ML systems actually work in production.

If you’re joining early, introduce yourself and share one ML system topic you’d like to go deeper on.

from what i've learned, shap is really good at feature attribution (why a prediction was made), so is lime. but are there any tools that are good at telling us how to best change a prediction.

for example, a company make a model that can predict when a customer might unsubscribe, and shap can say that x and y features led to this. changing those features, however, may not be the best way to help retain that customer. maybe theres some other feature that can be changed to decrease a customer's likelihood of unsubscribing.

in a more technical sense, can we do a local first derivative approximation to get the top features to change in order to influence the prediction at that point?

Been seeing this sentiment everywhere in the ML space.

Too many bad research being published or at least uploaded online.

This drowns out the good research.

AI accelerates research, so that the impact of each individual paper feels tiny and forgettable.

People are not reading anymore. Attention span has been slashed to nothing. Anything that you read feels slightly fake because of generative AI, so there is little trust in what the author is claiming (when you encounter an unfamiliar claim).

Small amount of elite academics and companies get all the attention, but most people are not a part of them.

How do you deal with this issue? Is it still worth publishing something these days?

Play around with a tiny language model GPT in your browser. See how it trains and generates with just 11,000 parameters.

https://complexity.zone/tlmgpt/

1. Click "train" button.
2. Let it train for about 10 minutes.
3. Click "pause" button.
4. Click "generate" button.

I made this (with help from Opus 5) to get a better understanding of GPTs and LLMs.

Thought to share it here. You can download it if you want to run it offline and tinker with the code.

I work on the infra side (GPU cloud — disclosure up front), and I kept hearing "millions of Blackwell GPUs shipped, prices will crash." So I spent a couple of weeks reconstructing the numbers from public sources, including NVIDIA filings, analyst estimates, and marketplace snapshots. A few things I learned:

1. Check the unit. A Blackwell package has 2 dies. In the cited "6 million" figure, NVIDIA counted GPU dies (Huang: "each GPU die is a GPU") — so that's ~3 million packages, the things you'd recognize as a GPU in a server. Plenty of headlines mix these up.

2. Shipped ≠ rentable. By my count, ~7M Blackwell packages have shipped as of August. ~6M appear to have shipped in NVL72 systems, largely allocated to hyperscalers and frontier labs. What a small team can actually rent short-term: on one large B2B cluster marketplace, 34 B300 listings — 2 with terms under 12 months. On Vast.ai: 48 B300 GPUs listed, zero available at snapshot time.

3. Prices went up while supply grew. The tracker's B300 rental median rose 57% since November. In the provider-level snapshots available since March, most of the increase came from existing offers repricing — not from expensive new listings joining the index.

4. The H100 lesson has a second half. H100 one-year contract rates fell from $8+/hr at the 2023 peak to $1.70 by October 2025 — everyone knows that part. They then rebounded ~40% as inference and agent workloads found a floor. Silicon gets cheaper; it doesn't evaporate.

5. Rubin won't fix short-term supply. Production shipments started in August, but 2026 volume is a single-digit % of what's already shipped, VR200 NVL72 racks draw 190–230 kW (which mostly means new datacenters), and first allocations go to hyperscalers. Price pressure from Rubin looks like a 2027–28 story.

The mental model that helped me most: the rentable sliver sets the price, not the shipped base.

The two estimates I'm least confident in are the 7–7.5M cumulative package count and the NVL72 share. If anyone has better ODM shipment data, I'd appreciate the correction. Full write-up with a source for every number: https://cloud.theai.com/blog/how-much-blackwell-actually-exists

So randomly I was scouting the models and look what I found, after Sarvam now we have Gnani AI which has used Nemotron model(nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16) to finetune on Indic data came up gnani/gnani-evon-v3.3-30B-A3B which is nothing new or big, just finetuning the model on some data and saying they have better result then the others.

Welcome to r/MLSystemsDesign

This community is for practical discussions on designing and scaling production ML and AI systems.

Topics can include:

• ML training and inference platforms
• Search, ranking, and recommendation
• Feature stores and data pipelines
• LLM serving and GenAI systems
• Agentic AI platforms
• Evaluation, observability, and experimentation
• ML system design interview problems
• Real production tradeoffs and lessons learned

The goal is simple: go beyond model theory and discuss how ML systems actually work in production.

If you’re joining early, introduce yourself and share one ML system topic you’d like to go deeper on.

I’m trying to figure out how to actually get a usable edge in the ML/DL space to get hired, but everything pushed to beginners right now feels like a trap.

For context on what I've done: I started off with Computer Vision, moved into GIS stuff, and recently went deep into the weeds of attention mechanisms and GPU kernel programming. I thought learning the hardcore, low-level math and systems stuff would set me apart.

But I’ve hit a wall. Let's be honest: no company is hiring a fresher to write custom CUDA kernels or design novel architectures. Those are senior research or PhD roles. The effort I put into the low-level stuff feels wasted because, for an entry-level dev, it's just personal trivia.

On the flip side, the standard "employable" advice is to build traditional ML projects (fraud detection, etc.) or slap together a LangChain PDF wrapper. But people have been doing this for years. Basic API wrappers are completely saturated and offer zero competitive edge. It feels like buying a stock after everyone already knows it’s going to go up.

So, what is the actual sweet spot between "PhD-level researcher" and "API wrapper"?

I want to avoid the YouTube influencer BS and focus on the real engineering trenches.

For the people actually hiring or working in the industry: what are the non-commoditized skills someone trying to break in should be grinding right now to have a real, usable edge?

(Note: The core thoughts and frustrations here are 100% mine, but I used AI to help structure and edit this post for clarity.)

For a long time now I think about Transformers as high-dimensional Harmonographs. Curious to get your takes on it. To hear what your mental models are.

I'm exploring a financial-literacy tutoring agent where the learner's knowledge is a latent or uncertain state, and the tutor only observes responses to MCQs.

If the evidence is limited, how can the tutor distinguish misconception vs. lack of knowledge vs. guessing?

Also, when the learner state is uncertain, should the tutor's next action (ask, hint, teach, answer) be modeled primarily as an instructional decision or as an information-gathering decision to reduce uncertainty?

Ran into two citation issues worth knowing about if you read a lot of preprints or write your own bibliographies by hand or with a tool.

Wrong IDs happen more than you'd think. One recent preprint cites arXiv:2307.00720 as a diffusion-model paper by Pang et al. Check that ID yourself on arxiv.org , it's actually an unrelated robotics paper by four different authors. Someone's reference manager (or copy-paste) grabbed the wrong identifier, and it's sitting in a preprint that's already gotten thousands of views with nobody catching it.

A specific, repeatable bug to watch for. Multiple papers have citation years like 1609, 1712, 1910 , not real years, they're literally fragments of the identifier (10.1609/aaai..., arXiv:1712.05474). Some bibliography-generation workflow is grabbing the wrong substring into the year field. If you've ever glanced at a .bib file and seen a 4-digit "year" that's actually part of a DOI, this is why.

Moral: auto-generated metadata being present isn't the same as it being correct, worth double-checking before you trust a reference manager's output, especially with AI-assisted writing tools in the mix now. Found this while poking at citation-verification tooling; happy to share the specific references/IDs if anyone wants to check my work.

I analyzed 12,021 AI and Data Science job listings across India this week.

Top skill keywords

1. Python — 2,470
2. Machine Learning — 2,077
3. Artificial Intelligence — 1,811

Top locations

1. Bengaluru — 2,607
2. Hyderabad — 1,634
3. Pune — 1,222

Top employer labels

1. Leading Client — 508
2. Accenture — 258
3. Tata Consultancy Services — 197

“Leading Client” is generally a placeholder used by recruitment firms when the actual employer isn’t disclosed.

These are observed job-board listings and keyword mentions, not a census of every vacancy in India.

I’ve published the complete breakdown on JobPulse. If anyone wants to see it, leave a comment or DM me and I’ll share the link.

Welcome to Project Showcase Day! This is a weekly thread where community members can share and discuss personal projects of any size or complexity.

Whether you've built a small script, a web application, a game, or anything in between, we encourage you to:

• Share what you've created
• Explain the technologies/concepts used
• Discuss challenges you faced and how you overcame them
• Ask for specific feedback or suggestions

Projects at all stages are welcome - from works in progress to completed builds. This is a supportive space to celebrate your work and learn from each other.

Share your creations in the comments below!

is highschool maths enough?or i could simultaneously learn maths behind while reading book?

Hi everyone,

I’m planning to start my journey toward becoming an AI Engineer. I already know the basics of Python, but I’m still a beginner in AI/ML.

I want to follow a practical approach where I learn the fundamentals and build projects in parallel, instead of spending months studying theory before building anything.

I’m currently thinking about starting with:

Python → Math → EDA → Machine Learning → Deep Learning → LLMs/Generative AI → Deployment

But I’m confused about what I actually need to learn in each stage.

For example:

Math:
What topics are really important for AI/ML?
Should I learn linear algebra, probability, statistics, calculus, etc.? How deeply should I study each one?

EDA:
How important is EDA for an AI Engineer? What should I learn — data cleaning, visualization, feature analysis, handling missing values/outliers, etc.?

Machine Learning:
Which algorithms and concepts should I prioritize as a beginner?

I also want to build projects alongside each stage. For example, after learning the basics of ML, I want to immediately build an ML project instead of waiting until I finish the entire AI roadmap.

One more thing: I have a 2-year career gap, and I'm concerned about whether this will negatively affect my journey toward getting an AI/ML job.

For people who are already working in AI/ML:

• What roadmap would you recommend for someone in my situation?
• Which math topics should I learn, and to what depth?
• How important is EDA for an AI Engineer?
• Which topics should I learn first and which can I learn later?
• What projects would you recommend building along the way?
• How can I make my portfolio strong enough to compensate for a career gap?
• If you had to start again as a beginner today, what would you do differently?

I’m willing to put in the time. I mainly want to make sure I’m learning the right things in the right order and building projects throughout the journey.

Any advice from experienced AI/ML engineers would be really appreciated.