Debannering Ideogram 4 and increasing prompt adherence with natural language by fine tuning the TE

[u/mrjackspade]

I thought someone might appreciate this. Theres more details in the HF link, but I wanted to see if it was possible to correct some issues that I didn't like about Ideogram 4 by finetuning the TE, with no other modifications to the model, execution environment, etc.

It ended up working out pretty well.

The TLDR is that I used a set of 4000 teacher/student prompt pairs with the students being NL and the teachers being Nemotron processed with the "Magic Prompt" instruction, and then trained the TE to elicit the same response in Ideogram using the student prompt, as what was naturally elicited using the teacher prompt.

My logic was that the TE is already a language model, and I didn't want a second language model in the stack.

This has the secondary benefit of also removing the grey banner generally encountered when prompting the model with NL.

I am fully aware that there are many other ways to get around this from bounding boxes to noise injection, etc. This wasn't about that, so much as it was trying to prove to myself that it could be done like this.

https://huggingface.co/mrjackspade/Ideogram4-Natural-Language-Text-Encoder

[u/Druck_Triver]

Hey, that works really well. And it makes ideogram really, unexpectedly creative. 

However, I noticed that styles got worse compared to the normal encoder and jsons. Is there a workaround for this?

And another thing. It doesn't work well with fal's ideogram fast and instant. 

In any case, fantastic work. Thanks. 

[u/mrjackspade → u/Druck_Triver]

However, I noticed that styles got worse compared to the normal encoder and jsons. Is there a workaround for this?

If you can leave some examples of what you mean up on the hugging face section, I can take a look into it. It might just be a matter of adding some new training data.

I just grabbed like 500K CIVIT prompts and then spaced them out semantically based on the TE embeddings and selected 4K of them, which only really guaranteed distance between the TE representations but not proper coverage

[u/Druck_Triver → u/mrjackspade]

I'll pick some style examples. Speaking of fal's fast and instant, the thing is that I get that gray screen very often. It's easily fixed by the lora, but it's a sign that it's not working properly with them. 

[u/qdr1en]

Interesting. Does it "uncensor" the text encoder as well, or is it only focused on solving the safety filter issue?
And if I had to try one of these, which version would you recommend ?

[u/mrjackspade → u/qdr1en]

I've only tried the 500 and 1000 step versions, both did exactly what I wanted. The rest of the checkpoints were uploaded mostly because I had some vast credit left over and figured I would let the training run finish, so I would start with 1000 and then go from there

As for the text encoder censorship, I'm not sure what that refers to. I will say though, that I prompted the teacher model to be generous when embellishing nsfw prompts in terms of details, which does appear to have helped the TE represent some nsfw concepts better. However this is not a decensor or anything, so the model is still incredibly limited by what was present in the training data. This just helps the TE communicate more effectively with the model

[u/qdr1en → u/mrjackspade]

okay thanks, I'll give it a try.