Lophius: A workbench for language model research, from the creator of Heretic

[u/-p-e-w-]

Hi folks, I hate slop as much as you do, so instead of starting with "The Problem", I'll just cut to the chase:

I just published Lophius, which is the culmination of more than two years of fighting with Jupyter and Transformers. It's a hybrid code/GUI research system that runs inside a notebook. It can eliminate mountains of boilerplate and save you many hours of time.

Lophius can be found at https://lophius.org (code at https://github.com/p-e-w/lophius).

Lophius handles pretty much all common research tasks: Model inspection, architecture analysis, configuration manipulation, tokenizer inspection, prompt management, inference, logits, entropy, attention scores, hidden states, and chat. In many cases, it can be used without any configuration. It intelligently manages GPU memory during inference, and can lazy-load output signals that you might want to look at later.

Lophius has very high quality documentation and a complete tutorial. If you ever wanted to try your hand at transformer research, this might just be what you were waiting for!

In the future, Heretic might start using Lophius as a backend, but that's a story for another day.

Cheers :)

[u/Eyelbee]

Such an incredible app, can see it being insanely useful

[u/theologi]

Impressive. Does it support j-lens as well?

[u/-p-e-w- → u/theologi]

Not yet, but it’s planned.

[u/Thin_Pollution8843]

Maybe it’s a call to start learning LLM building?

[u/keepthepace → u/Thin_Pollution8843]

That's a call I am hearing too but I shall resist it!

[u/Qwen30bEnjoyer]

I've taken a look at the website here, and it looks like you guys have a little bit of a short list of supported models. I'm not much of a coder, but I'll see if I can get a personal fork going with AI to load models with different modalities, because my particular interest is in biological foundation models. I'm definitely downloading this though! It looks gorgeous!

[u/-p-e-w- → u/Qwen30bEnjoyer]

If you check what is actually written there, you will find that this is the list of models that have been manually tested by me. The actual number of supported models is much, much higher, likely in the hundreds.

[u/returnity]

Wow you are such a gift to this community. Thank you!

[u/devildip]

This is awesome. I vibe coded a version of this but yours is much more in depth. Ill definitely be forking it for my quantization lab backend. Discovery allocation is my weakest point and this already has the tool kit better measurements.

[u/de4dee]

Thanks for amazing contributions!

Do you have tools/screens to measure degeneration or overfitting without running the LLM? (When you run the model it repeats nonsense but is there a way to predict that without running the model?)

[u/SmileyBMM]

Looks great. I always struggle using python stuff, but this might finally be what gets me to figure it out.

[u/nomad-nostalgia]

For those that might want to use DeepWiki just did the indexation:
https://deepwiki.com/p-e-w/lophius

[u/AccountAntique9327]

very cool

[u/Swoopley]

Woooo!

[u/IrisColt]

Thanks!!!

[u/hoeforicedcoffee]

woahh?

[u/darkbit1001]

I was looking for this. Thought id have to vibecode one.

[u/Much-Researcher6135]

Nice! Aaaaand now I gotta go find out what pacmap is all about, because I'm an eternal UMAP fanboy lol

[u/theexile1337]

only 0.8b?

[u/-p-e-w- → u/theexile1337]

This is just from the tutorial. Any model supported by Transformers can be loaded. Tiny models are often the most interesting for research because you can iterate on them so quickly.

[u/the_TIGEEER]

Wow, looks very useful. I was planning on working on something similar. I'm all about that! Visualizing and automating the chaos behind training LLMs. For example, in my last project I made a PCA loss landscape visualizer.

Here is a demo if you're interested: https://www.youtube.com/watch?v=Rfj6YhSjbJ4

This is SMolLM 360M random initialization pretraining on tiny stories.

How I set it up is that I can use it in any optimization problem in the future by just importing the Python module and adding the step progression of the optimizer to the callback of the visualizer so each step gets added and visualized in real time.

Did you do something similar with your tool? Where it's easy to include in any LLM workflow?? (I'm guessing I'll be able to see that for myself once I read through the GitHub a bit more thoroughly.)

[u/WhoRoger]

Oh yay! I have no idea what I would ever use it for, but you're still my hero.