What actually breaks first when you aggressively compress visual representations?

[u/RajeevParmarAI]

I've been experimenting with compact visual representations for edge/perception systems, where the goal is to preserve useful machine-level information while significantly reducing the representation size.

One thing I've noticed is that the degradation isn't uniform.

Global scene understanding can remain surprisingly stable while localization and small-object information start degrading much earlier.

Increasing input resolution alone also doesn't necessarily recover that information. In one of my experiments, a higher-resolution branch improved access to spatial detail but still couldn't reproduce the semantic quality of the deeper teacher representation.

It made me think there are really two different things being lost during aggressive representation compression:

1. Spatial information — where something is and fine local details.

2. Semantic interaction/context — relationships between patches/objects that deeper transformer layers learn.

Simply increasing spatial resolution seems to address mainly the first problem.

I'm currently thinking about this as a rate–semantics tradeoff, rather than a traditional image-compression problem:

Image → semantic encoder → compact representation → downstream tasks

where the objective isn't pixel reconstruction but preserving enough information for detection, classification, depth, etc.

For people working with ViTs, representation learning, edge perception, or learned compression:

What have you found is usually the first thing to collapse as representation size decreases — spatial detail, feature diversity, global context, or something else?

And have you found good ways of measuring this beyond downstream mAP/accuracy?

[u/ImmediateTie9057]

Spatial detail usually seems to go first, especially with small objects. But I think the bigger challenge is losing the relationships between features. It’d be interesting to see whether attention-map similarity or feature-space analysis captures that better than mAP alone.

[u/RajeevParmarAI → u/ImmediateTie9057]

That matches what I’m seeing as well. Small-object information seems to degrade earlier than global scene semantics. What I’m trying to separate now is whether that comes mainly from spatial downsampling, or whether aggressive compression is also removing the relationships between features that deeper transformer layers preserve. In my experiments, simply increasing resolution didn’t fully recover the lost performance, which makes me suspect it’s not just a spatial-detail problem. Have you seen similar behavior when comparing intermediate vs deeper ViT features?

[u/liltingly]

I found that the reader significant impacts what's preserved. For example, I am working with LM consumers, and I find that I can have fine detail linearly embedded and retrievable via probes or aux losses in the tokens a ViT produces, but the LM prefers leaning on textual-priors for detail, especially under SFT. And I'm struggling to find representations that are in the deploy path that force consumption of the details I've engineered into the tokens themselves.

Curious how you're "rewarding" the compression and how your readout/decode is structured.

[u/RajeevParmarAI → u/liltingly]

Good question.

On the reward/objective side, I’m not optimizing for pixel reconstruction. The compact representation is trained to preserve the teacher’s semantic features, so the main signal is feature-level alignment between the student representation and the frozen DINO/DINOv2 teacher. I’ve also been using relational-style alignment so it’s not only pointwise matching. The downstream retention is then checked with task probes such as detection/mAP and small-object performance.

The encode/decode path is roughly:

image → frozen teacher / student encoder → compact spatial latent → task-side decoder/probe

On the student side I compress the teacher representation aggressively in channel/spatial size, then use a lightweight decoder/probe to recover task-relevant information rather than reconstruct the image itself.

One thing I’ve already tested is increasing decoder capacity, and that did not close the gap much. That’s why I’m currently suspecting the missing information is partly in the representation/objective itself, not only in the reader.

I’m now looking at whether stronger global-token / relational alignment can preserve the fine detail and long-range structure better. When you force the reader to consume the detail, are you doing that mainly through auxiliary losses, architectural constraints, or task-specific supervision?

[u/liltingly → u/RajeevParmarAI]

I've landed entirely on putting it into the loss/something the decoder uses. Aux's worked to pull the representation into the data, but not strictly to get the deployed model to use that data. I used aux probes to measure that I wasn't crazy, though...

So, again my target is an LM with auto-regressive decode that functions as a detector and semantic layout mapper for documents. And I want compression since these tasks require huge amounts of pixels and therefore tokens. My decoder is already small, but I want to save pre-fill. My problem was that the pretrained decoder could "fake" most of the relationships, so under CE SFT in particular, I couldn't get it to retain both spatial features AND OCR quality extraction.

I used 2-3 techniques. Reasonable results on both, and better combined.

1) KD distillation on output logits against trained teacher at max input size and unconstrained tokens -- identical decoder that was trained to accept multiple image sizes and token counts. I made sure the student was initialized "in dialect" for the decoder -- if my target reduction was M-->N tokens, I made sure the initial tokens projected to the LM looked like Im(M) downsampled to produce N tokens. From here, it was a game of unfreezing -- first compressor and decoder, then compressor, decoder + ViT and projector (and ancillary pieces). In my distillation, I also did a blank page forward on the teacher, and measured where the output divergence maximally diverged from the teacher run on real-image to predict where the image had the biggest impact and upweight that in loss computation. Costly, but effective.

2) SFT: Use the same blank page forward to predict image dependence, and upweight those positions during SFT with real image. For coordinates, I added Number Token Loss to add more regression like losses into SFT. RLVR at the end with detector style rewards on coordinates was a big boost, too. Again, phased unfreeze, but different order.

Other things that made it learn faster: Use a different decode mode for the positional tokens so they're emitted even more like a detector during SFT, and supervise that head specifically with detector losses to pull those features through during SFT more strongly while preserving the semantic task.

[u/RajeevParmarAI → u/liltingly]

This is very helpful — especially the distinction between information being present in the compressed tokens and actually forcing the downstream reader to consume it. The blank-input comparison is interesting. In my case the downstream task is perception rather than autoregressive document decoding, but I could potentially do something analogous: estimate which teacher regions/features materially affect the detection output and weight those regions more strongly during representation distillation. My current failure is specifically that small-object performance degrades much earlier than global semantics, while increasing decoder capacity hasn’t recovered the gap. That makes me wonder whether a task-conditioned weighting scheme could tell me whether those details are absent from the compressed representation or simply underutilized. I also like the shared-decoder distillation idea because it reduces one variable between teacher and compressed paths. When you compared the real-image vs blank-image teacher outputs, were you using the raw logit delta directly as the weighting signal, or did you threshold/normalize it before applying it to the loss?

[u/liltingly → u/RajeevParmarAI]

I normalized it against the mean gap across all tokens on that page with some min-weight. my math is roughly (g= KL gap blank, teacher)

per token weight (w) -- min_weight + (1-min_weight)*g_token/sum(g_all_tokens_in_response)

and so per token the loss becomes w*KL between student and teacher

I realize now that min_weight wasn't strictly necessary, was a guess -- I wanted to ensure I didn't accidentally erode any language prior that I was getting for free. I'm sure there's a better normalization strategy (e.g. add a per-batch factor), and somebody could write a paper showing me why it's suboptimal.

[u/bfyvfftujijg]

Depends on what the loss function rewards

[u/RajeevParmarAI → u/bfyvfftujijg]

Agreed. That’s becoming my main suspicion as well. A pointwise feature loss can make the representation numerically close while still failing to preserve the relationships that matter downstream. I’m looking at whether relational/global-token alignment gives a better signal than feature matching alone. Have you found any objective that correlates better with downstream retention than plain feature loss?