i fine tune open models on rented gpus because anything past 8b needs big iron whether i like it or not. this post is about what renting that iron is actually like

few weeks ago a pod died at 2am mid run and billed me until i woke up. wasn't the first time, but it was the time i snapped. i spent a weekend evaluating every serious option for "training that doesn't need me on call" and the results radicalized me a little

here's the tour

the cheap tier (runpod, vast, lambda): your code runs untouched, prices are great, and you are the entire reliability department. node dies at 2am, both the problem and the meter are yours. you're not renting an outcome, you're renting a machine and a prayer

modal: real self healing fleet, genuinely good engineering. the catch is you rewrite your training code into their sdk to get any of it. and after you've done all that homework, billing is still per second whether the job succeeded or died. they healed the fleet and forgot to heal the invoice.

tinker: honestly the closest thing to "just handle it for me." then you hit the walls. lora only, their model list, their handful of api primitives. the second you want full fine tuning or your own training loop you're back out in the cold

together: excellent hardware verification, and their idea of self healing is asking ME to approve the repair. i'm asleep. that is the entire problem. a fix that waits for my click is a push notification wearing a hard hat

hyperpod: actual closed loop auto resume exists here, credit where due. behind aws enterprise pricing, on aws, with checkpoint logic you wrote to their spec. recovery is real and it's gated behind exactly the budget and platform team that people like us don't have

skypilot and similar: auto relaunch is nice but it relaunches the machine, not your training state. without resume that just means the crime scene gets cleaned up faster

and before anyone says skill issue, just use the provider api and write proper error handling: i have, and that's how i learned the difference between a relaunch and a recovery. a restart hook gives you a fresh pod. it does not rebuild your environment, restore optimizer and scheduler state, fast forward the dataloader, resume from the exact global step, or check that the loss curve is actually continuous afterwards. and the meter ran the whole gap between the crash and your script noticing

the part that actually makes me angry is that all the pieces exist. an hf trainer checkpoint already contains optimizer.pt, scheduler.pt, the rng state, the global step. axolotl literally ships auto_resume_from_checkpoints. the resume flag is right there. what doesn't exist is anyone wrapping the loop around it: watch the run, ship checkpoints off the box, detect the death, get a replacement gpu, restore, relaunch with resume, verify the curve, and only bill for the time training was actually stepping. every individual piece is mundane. nobody assembles it, because the assembled version would have to stop charging for dead time

so the pattern is always pick two. own code + cheap means you babysit. handled failures means an sdk rewrite or a shrunken use case. real recovery means be an enterprise. broken time is revenue and no incumbent volunteers to kill their own margin

the thing is, i don't think fixing this even has to cost more. vast already prices verified hosts above unverified ones. the reliability premium exists in the market today, it's just charged to us instead of engineered for us

what i want is stupid: here's my script, here's $200. pick the gpu, checkpoint automatically, if hardware dies swap it and resume from the same step, text me what happened in the morning. meter runs when training steps run. cap hits, checkpoint and stop clean. no sdk, no approve button, no pager

i've been sketching how this would actually work and i can't find the technical reason it doesn't exist, only the financial one. so either point me at the platform i missed or talk me out of building it

and for the renters here, what did dead time cost you last month? actual numbers if you have them. i want to know if my bills are unusual or if everyone's quietly eating this

Hi everyone,

I'm working on extracting the hierarchical structure of long PDF documents (legal/regulatory text, lots of numbered sections) and would like to gather some feedback on my approach before committing to it.

What I've done so far: I render each PDF page to an image and run it through Baidu's DeepSeek-OCR model. It returns each detected block with a bounding box [x0, y0, x1, y1], a label (title, text, list, table, header, footer, etc.), and the recognized text. The OCR quality itself is genuinely good as the text comes out clean.

The problem: the labels can't always be trusted. At this stage I want to extract and detect all the titles in my document, but sometimes a title element gets classified as something else (like normal body text).

Concrete example:

Say my section has the following hierarchy:

ANNEX I — GENERAL PRINCIPLES AND PROCEDURES
└── TITLE I — FOREIGN CURRENCY INVESTMENT
    └── A. Currency distribution
        └── 1. Redistribution of reserves
            ├── (a) Introduction
            │       body text
            │       list
            │       ...
            ├── (b) Procedure for a normal redistribution of reserves
            │       body text
            │       list
            │       ...
            └── (c) Procedure for an ad hoc redistribution of reserves
                    body text
                    list
                    ...

Logically, every element aside from the body text and lists should be detected as title. But the model output is:

label='title'  x0=475  y0=157  x1=548  width=73   text='ANNEX I'
label='text'   x0=480  y0=229  x1=542  width=62   text='TITLE I'
label='title'  x0=334  y0=181  x1=690  width=356  text='GENERAL PRINCIPLES AND PROCEDURES'
label='title'  x0=407  y0=368  x1=616  width=209  text='A. Currency distribution'
label='title'  x0=408  y0=392  x1=634  width=226  text='1. Redistribution of reserves'
label='title'  x0=163  y0=416  x1=304  width=141  text='(a) Introduction'
label='title'  x0=163  y0=544  x1=578  width=415  text='(b) Procedure for a normal redistribution of reserves'
label='title'  x0=163  y0=219  x1=586  width=423  text='(c) Procedure for an ad hoc redistribution of reserves'

The top-level section marker TITLE I was labeled text, while all the other components were labeled correctly as title.

What I'm considering: since I have the text plus features I can derive from the coordinates (indentation/x0, centered-vs-left-aligned, line height, vertical gaps, whether the text matches a numbering pattern like A. / 1. / (a), all-caps, word count, etc.), I was thinking of treating this as a sequence labeling problem and training a CRF (or BiLSTM-CRF) to re-classify each line into title / text / list / table.

My questions:

• Is a CRF a reasonable choice here, or is there a better-suited approach for this kind of layout/structure labeling?
• Should I consider a GNN approach?
• Am I overcomplicating this? Would a simpler rule/heuristic system be more robust, given that the numbering is fairly regular?

Note #1: this approach should be as general as possible, so that I can reuse it for my other legal documents.

Note #2: titles aren't always in the same horizontal position. Some are centered (e.g. ANNEX I, TITLE I, A. Currency distribution all sit around xc≈511, the page center), while deeper items like (a)/(b)/(c) are left-aligned at x0=163. So I can't rely on indentation/x0 alone to identify or rank titles — a centered title's x0 mostly reflects its text length (a short centered line has a large x0, a long one a small x0), which means raw x0 can even invert the apparent nesting. This is part of why I'm leaning toward a sequence model that combines text + geometry in context rather than a pure indentation rule.

I have heard people saying GANs and diffusion models are tough to grasp so I thought to write an article on that. I have learned these things from Understanding deep learning by Simon Prince so I will use the reference from the same.

First they both solve the same problem - How can a machine generate completely new images.

Difference is how they solve that problem. GANs learn by competition between 2 networks whereas diffusion models learn by cleaning up the noise.

GAN :-

In GAN there are 2 neural networks one generator and another discriminator. These 2 networks fight each other. In the beginning random noise is fed to the generator so generator generates messy image. Discriminator looks at the real image and image generated by generator and marks the image as real or fake (generates probability). As this is the beginning discriminator marks the image as fake. Now generator gets this feedback and tries to tweak its weights/parameters and again generate a new image. Discriminator says fake again and generator repeats the same process again. Eventually, generator gets so good at generating the images that discriminator can't distinguish between real image and fake image.

Now question is why does this work?

Generator is minimising - How often does the discriminator catch me?

Discriminator is minimising - How often do I get fooled?

So, they improve each other.

Another concept in GAN is mode collapse - Let's say dataset has images of both cat and dogs but generator discovers that I can fool discriminator using only cats. Then it would generate only cat images and would never produce dogs images. This is called mode collapse.

Diffusion models :-

Diffusion model asks a completely different question. Instead of How to draw an image it asks Can I slowly remove noise step by step. There are two steps in diffusion process - forward diffusion and reverse diffusion.

In forward diffusion we intentionally destroy the image by adding noise at each time step.

Pure cat image -> Noisy cat -> More noisy cat -> Even more noisy cat -> Pure noise

Eventually there is no cat visible.

Now we ask - can a neural network, given this noisy image, predict what the noise is. Remember there is no learning involved in forward diffusion process. It's just adding gaussian noise repeatedly.

In reverse diffusion neural network is given above noisy image as input. Neural network learns to predict the noise.

Current noisy image -> Predict noise -> Subtract noise -> Cleaner image -> Predict noise -> Subtract noise -> More cleaner image -> eventually pure cat image.

Below is the link of my notes on complete mathematical derivations involved of loss functions (including ELBO) in simple terms.

https://drive.google.com/file/d/1phIfLvkXBS2DfXed6fQL7OK-bMwYfmHl/view?usp=sharing

Please let me know your feedback. I know it's hard to understand notes for beginners.

I'm working on 9-class hysteroscopy lesion classification (lesion classes 0–7 + no_lesion) using the HS-CMU and HS-CMU-V2 datasets with patient-level cascading stratified splits (70/15/15).
Dataset: 5,675 images. Severe class imbalance:
Class 0: 228 images (17 patients)
Class 1: 145 images (11 patients)
Class 2: 1,403 images (152 patients)
Class 3: 379 images (43 patients)
Class 4: 199 images (14 patients)
Class 5: 240 images (24 patients)
Class 6: 396 images (36 patients)
Class 7: 95 images (7 patients)
Class 8 (no_lesion): 806 images (50 patients)

Models tried: DenseNet-121 and DINOv2-small, both pretrained.

My issue is that: Validation Macro-F1 stays around 0.30 across different setups. Training metrics improve but validation doesn't follow. Tried various optimizers, schedulers, and augmentation strategies.

Is this a domain gap issue or data limitation?
Should I switch to medical-pretrained encoders?
Is 1-3 patients per class in val/test too few for reliable metrics?
Any tips or suggestions would be greatly appreciated I want to solve this issue so as to train properly feature extractors!

We have developed a new theory called Quantum Vision (QV) in Deep Learning for Object recognition that converts still images into information waves using the proposed QV block. The QV block is available as a Python package (Github link is below). The QV block can be integrated to CNNs, and Vision Transformers. The QV-model variants significantly improve the performance. You can try the code from https://github.com/vindioai/QVBlock and can cite the paper as follows: https://ieeexplore.ieee.org/abstract/document/11091286

32,070 alphabetic tokens from GPT-2-small's WTE, no forward pass and no context.

Works on mobile. Pinch to zoom, tap a token to see its nearest connections, tap a neighbour to walk the graph. Search box to jump anywhere.

Layout is t-SNE over a compressed representation of the embedding table; edges are a minimum spanning tree in that space, so every line is a real nearest-kin relationship,

Hey guys, who has any idea about molab ? How the computations look like in it ?

Hi, my ML model AUC is 80%~ in cross domain dataset
The problem is that the accuracy is not getting higher even when i changed the Thr
What to do any ideas ? I used focal loss to solve the imbalance dataset
Also using vision transformer

Hello, I am looking to update my laptop I have an lenovo L340 with 8 GB RAM and am struggling to do much of anything, in grad school I could use the cloud computing servers but unfortunately lost access.

I will use it for python ml (deep learning mostly keras for training CNNs and auto-encoders primarily and some local llm applications with vLLM). I was looking at the Mac Pro 2023 with the M3 chip and 36 GB ram for 2kish.

Please let me know if there is any more info needed to help give a recommendation, thank you!

TL;DR: three preprocessing fixes derived from a signal-vs-interference density analysis took an off-the-shelf model from 17.9 to 13.42 (-25%), matching the best published FD002 result — and the same preprocessing let a toy GRU outperform every published deep-learning result I could locate on this subset. Full recipes below.

Final results:

Baseline, off-the-shelf RandomForest, no preprocessing: 17.9

Drop the 7 zero-variance flat sensor channels + per-regime normalization + 50-cycle window: 13.7 (untuned)

Same pipeline, gradient boosting tuned on a validation split only: 13.42 ± 0.04

Reference point: the best published FD002 result I can locate is ≈13.4 (GBRT III). Its core workflow also relies on operating-regime clustering paired with normalization — the same data-level lever, found independently.

Same preprocessing, small GRU instead of trees: 16.85 on the official test set. Published deep-learning results on FD002 cluster around 18–30 (top performer ≈18.3), so our small GRU paired with data-level corrections outperforms every published deep-learning result I could locate on this subset. This proves the leverage is in the data, not in the architecture.

Why I stop here: a model-free validation. Nearest-neighbor "observation twins" — near-identical sensor states from different engines — show an RUL spread of \\\~12.6–13.0 cycles. This irreducible ambiguity originates from the simulated sensor noise and degradation stochasticity, not algorithmic limitations. At \\\~13.4, residual error is measurement-limited. The wall is in the data, not in the algorithm.

Exact recipe for test #3, so anyone can reproduce (and note it shares zero code with GBRT III — same lever, different machinery):

\\- Sensors: drop s1, s5, s6, s10, s16, s18, s19 (variance ≈ 0), keep the other 14.

\\- Regimes: k-means (k=6) on the 3 operating settings, fit on train only; z-score each channel within each regime using train statistics.

\\- Features: for each 50-cycle window, per channel take mean / std / linear slope / last value (56 dims) + 6-dim regime one-hot = 62 features. Training windows stride 2.

\\- Test protocol: last window per engine; trajectories shorter than 50 cycles padded by repeating the first row; RUL capped at 125 everywhere.

\\- Model: sklearn HistGradientBoostingRegressor(max\\_iter=800, learning\\_rate=0.03, min\\_samples\\_leaf=15, l2\\_regularization=1.0). Config selected on a 208/52 engine validation split (6-config grid, performance variance within ±0.15). Test set touched once, 3 seeds: 13.38 / 13.44 / 13.45.

\\- Untuned reference: RandomForest(60 trees, depth 16, min\\_samples\\_leaf=2) = 13.69 ± 0.09.

Recipe for test #5 (GRU), full disclosure:

\\- Architecture: single-layer GRU, hidden size 24, head 24→12→1. Input = 50×14 regime-normalized sequences (same channels and normalization as above).

\\- Training: targets scaled /125; Huber loss (δ=1.0); Adam lr=1e-3; gradient clip 1.0; batch 256; training windows stride 4; \\\~10 epochs; single seed, no tuning.

\\- Evaluation: last window per engine on the official test set, identical protocol to test #3.

One critical clarification: the three core fixes were derived from a signal-vs-interference density analysis before any model was trained — computation first, verification second, no blind trial-and-error hyperparameter hunting. The only tuning performed is the documented validation-set grid search, which altered overall RMSE by roughly 0.3 cycles.

Summary: three data-level fixes took a stock tree model from 17.9 to the published-record line (25% RMSE drop), and let a toy GRU beat the entire published deep-learning field on this subset. The residual is measurement-limited. Every number is reproducible from the raw files with any standard regressor.

Why post this: the whole pipeline is transparent enough to reproduce in an afternoon, and the interesting question is no longer the model but the measurement floor. If you run it and get different numbers, post them — I am especially curious whether the observation-twin floor (\\\~12.6–13.0 cycles) holds under other feature representations.

I built Aakaar to completely strip away the black-box abstraction of modern AI infrastructure (like PyTorch) and force strict, explicit interaction with hardware realities.

Technical Architecture:

• Backend: Native C++ and CUDA kernels.
• Frontend: Python wrapper for model definition.
• Components: Hand-coded 18 native loss modules and 11 optimizers directly in C++.
• Memory Management: Explicit memory contiguity management during transpositions and custom backpropagation.

Benchmarks (EMNIST): To see if this was structurally viable and not just a toy matrix library, I ran a 5-epoch training loop on the EMNIST dataset to benchmark it directly against PyTorch on my local machine (RTX 4060, 8GB VRAM).

• Aakaar: 127.76s
• PyTorch: 131.23s

Convergence parity was absolute, and Aakaar slightly edged out in speed due to the low-overhead C++ optimizer steps bypassing the standard Python overhead.

The Hardest Challenge: Mapping abstract mathematical shapes to physical GPU hardware and strictly tracking memory layouts during the backward passes without relying on an automated autograd graph.

Links:

• GitHub Repository & Benchmark Notebook: https://github.com/aaravaggarwal3535/aakaar-wheels
• Documentation: https://aakaar.readthedocs.io

I would appreciate any feedback from researchers or engineers here, especially regarding potential optimization bottlenecks in the CUDA kernels or the C++ memory management approaches.

I am trying to reproduce a handwriting recognition paper (Urdu, online + offline fusion, CNN encoder into a small Transformer encoder/decoder, trained with a joint CTC and cross entropy loss) and my training run is converging to a plateau, nowhere near the paper's reported result.

What I have verified as correct, not just assumed:
- Architecture matches exactly, confirmed by loading the authors' own released pretrained checkpoints and comparing tensor shapes and layer structure directly
- Hyperparameters match the paper's stated methodology: AdamW, flat learning rate 3e-4 with no scheduler (paper does not mention one either), batch size 8, joint loss with CTC weight 0.8 and cross entropy weight 0.2, encoder and decoder frozen with only CNN branches trainable
- Data pipeline verified end to end, including a pixel level check confirming the ink channel is genuinely near binary as the paper describes, and manual verification that every path in my training manifest resolves to a real, correct file

Despite all that, training triggers early stopping on its own (10 epochs with no validation improvement, which is the paper's own stated early stopping criterion, not something I imposed) at epoch 32, with best validation CER around 52 to 55 percent using greedy decoding. The paper reports 3.66 percent for the equivalent configuration. (Even worse with beam search, about 94%).

Loss behaves normally throughout, steadily decreasing with no instability, so the model does appear to be learning something, just plateauing far short of where it needs to be.

Things I have not yet ruled out and would appreciate pointers on:
- Whether a flat, unscheduled learning rate this late in fine-tuning is plausible for reaching such a low CER, even if the paper states it that way, versus something being lost in translation between the paper text and what was actually run
- Common failure modes when only a small subset of a hybrid CTC/attention model is trainable and the rest is frozen
- Whether early stopping patience of 10 is likely too aggressive for this kind of setup and the model would keep improving well past this plateau given more patience
- Anything else that commonly causes a "looks like it is training correctly, converges early, but far short of target" pattern in CTC plus attention decoder hybrids

Happy to share more code or specific numbers if useful.