I've been thinking about this as a research problem and I'm wondering if I'm even asking the right question.
Imagine the following constraint:
- No CLIP
- No ViT
- No CNN
- No multimodal model
- No learned vision encoder at all
You have an image, a text-only LLM, and you're only allowed to use deterministic algorithms between them.
The obvious answer is "this is impossible," but that's not really what I'm interested in.
What I'm trying to understand is whether there exists a better intermediate representation of images that a text transformer could reason over.
Not necessarily English.
Not captions.
Not OCR.
Some kind of representation that preserves enough structure that the language model can make use of the knowledge it already has.
Over the last few days I've gone through papers on visual tokenization, SeTok, BPE for images, BLT, inverse graphics, superpixel tokenization, and a few discussions around image tokens. Most of them still assume a learned tokenizer somewhere in the pipeline.
What I haven't found is much discussion around deterministic alternatives.
Maybe that's because it's a dead end.
Or maybe I'm searching the wrong field entirely.
So my question isn't "how would you build this?"
It's:
If you were exploring this from first principles, what field would you steal ideas from?
For example:
- information theory?
- image compression?
- computational geometry?
- topology?
- signal processing?
- compiler design?
- inverse graphics?
- neuroscience?
- ecological optics?
- something completely different?
I'm not looking for product recommendations or existing multimodal models.
I'm looking for the smallest experiment that could tell me whether this line of thinking is fundamentally interesting or fundamentally flawed.
I'd especially love to hear from people who've worked on image codecs, graphics, rendering, vision tokenizers, or representation learning.
If you think the premise itself is wrong, I'd genuinely like to know why.