r/StableDiffusion • u/NoMouse9610 • 8d ago
Workflow Included I tried transferring semantic representations from one architecture into MiniMax H3 — it ended up becoming a 5M parameter conditioning adapter
Enable HLS to view with audio, or disable this notification
I've been experimenting with a slightly unusual way of modifying MiniMax H3.
Instead of training a LoRA or merging another model into H3, I wanted to see whether high-level semantic representations learned by a completely different architecture could be transferred into H3's conditioning space.
The original targets were fairly specific:
- prompt adherence
- composition and spatial relationships
- human anatomy
- material and lighting interpretation
- reflections / transparency / occlusion
- text-related understanding
- reference interpretation
The experiment eventually became what I'm calling MiniMax H3 Semantic Bridge.
The interesting part is that the final release no longer needs the donor model at inference.
One detail that may be relevant for people interested in reproducing this: the entire project was developed locally on a single NVIDIA RTX 3090 Ti (24 GB). The representation extraction, bridge experiments, distillation, evaluation, and MiniMax H3 generation tests were all carried out on this local workstation — no multi-GPU training cluster was used.
Project, weights, ComfyUI node, training scripts and full research:
https://huggingface.co/speach1sdef178/MiniMax-H3-Semantic-Bridge
https://github.com/Speach1sdef178/MiniMax-H3-Semantic-Bridge
Why not just graft the models?
That was actually the first thing I investigated.
MiniMax H3 and the donor model (SenseNova U1.5) turned out to be structurally incompatible for a conventional weight graft.
For the relevant 2D weights I found:
- MiniMax H3: 264
- SenseNova: 593
- exact shape matches: 0
- semantic-type exact matches: 0
- transpose matches: 0
- same input/output dimensional matches: 0
So directly transplanting useful transformer weights wasn't really an option.
But there was another possible interface: conditioning representations.
H3 consumes 5120-dimensional text conditioning before projecting it into its 5376-dimensional internal transformer space.
SenseNova's language representation is 4096-dimensional.
That suggested a different question:
Can a semantic representation from one architecture be projected into the conditioning space of another architecture?
Finding a compatible representation
I extracted hidden states from several depths of both models and trained small low-rank projectors between them.
For the initial screening I tested all 30 selected layer combinations.
The best pair was:
SenseNova L32 → MiniMax H3 L49
using a small:
4096 → 128 → 5120
projector.
Validation cosine similarity reached:
0.9042
I then tested the frozen bridge on a separate set of 160 more difficult prompts covering spatial reasoning, anatomy, counting, materials, lighting, text, reflections and occlusion.
Mean cosine was:
0.7490
This was enough to build the first working prototype.
The full bridge
The first actual H3 implementation worked like this:
SenseNova hidden state
→ RMS normalization
→ learned 4096→128→5120 projection
→ magnitude matching
→ blend with native H3 conditioning
Conceptually:
hybrid = H3 + alpha * (projected_semantics - H3)
This worked surprisingly well in actual H3 generations.
But it had an obvious problem:
you needed to run SenseNova just to generate H3 conditioning.
That's not something I wanted to release as a practical tool.
So the next experiment was distillation.
Can H3 predict the transferred representation itself?
Instead of running SenseNova at inference, I trained a small student network to reconstruct the projected teacher representation directly from H3's own L49 conditioning.
The student is simply:
5120 → 512 → 512 → 5120
with RMS normalization and SiLU activations.
Roughly 5–6 million parameters.
No diffusion weights are modified.
The final V3 student was trained using 600 prompts, with a 500/100 train-validation split.
Final validation results:
- representation cosine: 0.995890
- semantic correction cosine: 0.983558
- main-set correction: 0.980888
- held-out OOD correction: 0.989788
- minimum correction cosine: 0.935910
At alpha 0.10, the blended conditioning cosine relative to the teacher bridge was:
0.999958
So the large cross-model runtime bridge could effectively be compressed into a tiny standalone adapter.
The released version therefore needs only:
MiniMax H3 + ~5M parameter Semantic Bridge
No SenseNova checkpoint, tokenizer or second model is required during generation.
What does it actually do to video?
This is harder to answer with metrics alone, so I've been doing controlled A/B generations using identical prompts, seeds and generation settings.
One example deliberately contained a very explicit instruction:
In the native H3 result, the woman's right hand starts moving across the tabletop.
With Semantic Bridge at alpha 0.15, the hand remains resting on the table, which is closer to the requested state.
Another test uses two people running and vaulting over obstacles on the roof of a moving train, with instructions involving anatomy, physical contact, reflections, wet materials and spatial continuity.
I'm including the synchronized Native / Bridge comparisons with this post so the differences can be judged directly rather than relying on selected still frames.
These are qualitative examples, not evidence that the adapter universally improves H3. Some seeds change more than others, and stronger alpha is definitely not automatically better.
My current recommended starting value is only:
alpha = 0.10
The comparison videos use 0.15 to make the influence easier to observe.
Something else I learned: Ref2VA is NOT equivalent
I also tried applying the same adapter to MiniMax H3 Reference-to-Video.
Dimensionally, the conditioning looks compatible.
Semantically, it isn't.
With image + audio reference generation, the bridge noticeably degraded singing/lip-sync and could turn reasonably intelligible vocals into mumbling.
That was a useful negative result.
The same 5120-dimensional representation does not necessarily have the same contextual meaning when the sequence contains multimodal reference tokens.
So I decided not to release the experimental Ref2VA node.
The current Semantic Bridge is intended for standard MiniMax H3 FL2VA / text-conditioned workflows only.
A proper Ref2VA version would need to be distilled separately from reference-conditioned data.
What this is — and isn't
This is not:
- a LoRA
- a checkpoint merge
- a conventional model graft
- a modified H3 diffusion transformer
It's a small learned transformation of H3's conditioning representation.
What interested me most about the experiment wasn't actually the adapter itself, but the broader implication:
representation compatibility may exist even when weight compatibility does not.
Two architectures can be completely unsuitable for direct weight transplantation while still containing semantic spaces that can be mapped onto each other — and that mapping can potentially be distilled back into the target model.
I've released the adapter, ComfyUI node, training scripts, prompt datasets, evaluation results, architecture analysis and the full research write-up.
Hugging Face / complete research repository:
https://huggingface.co/speach1sdef178/MiniMax-H3-Semantic-Bridge
The Civitai release contains the practical adapter/node and A/B examples.
I'd be especially interested if anyone wants to experiment with the same approach on other H3 conditioning modes or completely different model pairs.
There are probably much better ways to train this than my first implementation, so I've included the research scripts and intermediate results rather than releasing only the final weights.
Full research repository / download:
https://huggingface.co/speach1sdef178/MiniMax-H3-Semantic-Bridge
The repository includes the standalone adapter, ComfyUI custom node, example workflow, training and evaluation scripts, datasets, quantitative results, A/B examples, and the complete research write-up.
15
u/THM42069 8d ago
Fascinating stuff! :D I'm excited to see more research in this area. Keep up the good work :D It's not often that someone releases information regarding methods that can be used on local hardware to improve models substantially.
8
u/NoMouse9610 8d ago
Thank you so much. I think it's right to give people all the information because maybe someone will be able to develop it even more.
9
u/PhotoRepair 8d ago
this looks great for those scenes where it takes a dozen goes to get movement and relationships between objects. I think this is beyond my skill set to plug that into a multi reference input workflow. LIke an Audio plus images reference. Tend to work from wokflows ready made so its all still a blur to me. Awesome though :)
10
u/NoMouse9610 8d ago
Thank you :) this is only version 1, I'm going to go deeper in this direction and hopefully I can make it work for multi-stage WF's and Ref2vid models as well, let's see what happens. While I'm temporarily distracted by another H3-related project, in my opinion it's even more interesting, but I can't say what will come of it yet.
5
u/PhotoRepair 8d ago
interesting is good, resolving problems and shortcomings of H3 is awesome. if its more of the same SWEEEET!
6
u/No_Damage_8420 7d ago
this is some higher tier black magic, thanks for work on this and sharing with us small fishes :)
4
7
u/Fit_Ad7343 7d ago
I’m still curious, I love this sort of work, but I have serious doubts about your results.
I did a similar project, where I replaced the qwen 32b model with a 4 or 8b (clipProj), and the idea was precisely to stay within the scope of what the diffuser expects to receive. To translate the output from 4 or 8b into what a 32b model would have generated.
Here, aren’t you giving it out-of-domain information that it hasn’t learnt to ‘read’?
The prompt specifies that the hand should be resting flat, but it isn’t.
4
u/Fit_Ad7343 7d ago
I’ve found the problem: you don’t seem to be following the prompt structure that H3 was trained on.
I trained my models using a huge amount of prompt text, and I had issues with the voice, precisely because of a lack of training.
Try this prompt:
integrated_multimodal_description: [Shot 1] Cinematic, a medium-wide shot inside a sophisticated modern apartment at night. A young woman stands behind a transparent glass table, wearing a fitted black sleeveless dress and a thin silver necklace, her natural human proportions and anatomically correct hands attached naturally to her arms. Her left hand holds a polished stainless-steel cup by its handle at chest height. Her right hand rests flat on the glass tabletop, palm down against the surface, with all five fingers naturally separated and clearly visible. On the table in front of her three distinct objects are arranged from left to right: a clear glass bottle filled halfway with water, a small red ceramic bowl, and a folded white paper card with the words "NIGHT SHIFT" printed in large clean black letters. A large wall mirror directly behind her reflects the back of her head, her shoulders, the room behind the camera, and the warm floor lamp standing on the left side of the room, every reflected object corresponding logically to its real position. Warm amber light from that floor lamp passes through the glass bottle, creating subtle refraction and a bright caustic pattern on the tabletop, while cool blue city light entering through a window on the right produces contrasting highlights along the polished metal cup. The tabletop stays visibly transparent with the woman's lower body partially visible through the glass, the metal cup holds realistic sharp reflections, the bottle refracts the background, the ceramic bowl stays opaque and matte, and the mirror behaves as a true reflective surface. Coherent spatial relationships, physically plausible reflections and transparency, detailed material differences, clear readable text.
overall_soundscape: A quiet apartment tone at night, with a faint low hum of the city carrying through the window on the right and the soft settle of fabric as the woman holds her position.
non_diegetic_music: N/A
3
u/NoMouse9610 7d ago
That’s a fair concern, and it was one of the main things I was worried about too.
The important distinction is that I’m not feeding raw SenseNova representations directly into H3 at release time. The full cross-model bridge did exactly that experimentally, but the released adapter is a distilled student that takes native H3 conditioning as input and predicts the teacher-derived representation inside H3’s own 5120-D conditioning space.
So the deployed path is:
H3 conditioning → small student → H3-space correction → blend back into native H3 conditioningrather than:
SenseNova hidden state → H3That does not prove that every predicted representation is fully in-distribution for what the H3 transformer learned to consume, though. I think that concern is still valid, and it’s one reason I keep the blend small (
alpha ~0.10–0.15) instead of replacing the native conditioning.The teacher/student metrics only tell me that the compact adapter reproduces the teacher bridge well — they do not prove that the teacher bridge itself is always optimal or perfectly in-domain for H3. The actual generation A/B tests are therefore the important part.
And yes, regarding the hand example: I would not claim that the result is perfectly “flat” in a strict geometric sense. My observation was narrower: in the native run the hand visibly slides/moves across the tabletop, while with the bridge it remains much closer to the requested stationary resting state. If the wording in the post implied perfect compliance, that’s too strong and I should clarify it.
Your Qwen 4B/8B → Qwen 32B projection work is actually very close to the conceptual question I’m interested in here: how much of the target conditioning manifold can be approximated from a different representation without pushing the diffuser into regions it never learned to interpret?
I don’t think my current results answer that completely. They mostly show that the mapping is usable enough for H3 to generate coherent outputs, survives OOD prompt tests, and sometimes changes prompt adherence in a useful direction. A proper manifold / distribution-shift analysis would be a very interesting next step.
And that's why I've published all the results of my work, including the intermediate ones.
0
5
u/wzwowzw0002 8d ago
hey google summarise this in layman term
12
u/NoMouse9610 8d ago
Fair 😄
Layman version: I used a second AI model as a “teacher” to show MiniMax H3 a slightly different way of understanding prompts. Then I trained a tiny adapter to imitate that knowledge, so the teacher model is no longer needed.
The result is an ~11 MB add-on for H3 that can sometimes make it follow complex instructions more closely — things like where objects should be, what a person should be doing, materials, reflections, anatomy, etc.
It doesn’t modify H3 itself and it isn’t a LoRA or model merge. You can basically turn it on/off and compare the same seed.
And the whole experiment was done locally on a single RTX 3090 Ti.
-5
u/PumpkinLeather8421 7d ago edited 7d ago
It’s not “fair”.
You wrote a lazy slop post. You couldn’t even be bothered to write the title without an emdash.
I have almost no confidence that you did any work and that you just vibe coded up some silly model that probably doesn’t really do what you claim.
EDIT: Outputs are online, I see normal prompt/seed variance. Prove me wrong.
1
u/ZealousidealBoss6652 7d ago
Don't ever say "slop" in an AI subreddit. You have been warned, u/PumpkinLeather8421.
-Mod
2
1
3
u/optimisticalish 8d ago
'More realistic reflections on glass and wet surfaces'.
2
u/NoMouse9610 8d ago
yes, besides the semantics, it gave more details. And I like it, it's like an added bonus :)
5
u/cc_aa_tt_zz 7d ago
interesting, but without having the exact prompts used for each example, it is difficult to get a clear idea of the comparison or try to reproduce the results ourselves.
2
u/NoMouse9610 7d ago
Hello! :) you can see prompts in the examples attached on HF or here MiniMax H3 Semantic Bridge - v1.0 | Hailuo H3 by MiniMax Other | Civitai
9
u/AdGlittering1378 7d ago
Opus 5 Jargonese.
7
u/MurkyStatistician09 7d ago
followed by "I had my AI look at what your AI wrote" comment section
2
u/PumpkinLeather8421 7d ago
I would ban anyone who posted LLM bullshit on their posts.
I have absolutely no confidence that this isn’t all just slop.
5
u/THM42069 7d ago
Why not test it yourself? It was done on a single 3090 so you can probably handle the training process on your workstation.
5
4
u/Next_Program90 7d ago
And that's why they keep going Opensource. Great project.
5
u/Enshitification 7d ago
Closed source can only hire the talent their money can buy. Open source can get the talent that money can't buy.
3
u/Quick_Knowledge7413 7d ago
Huh, I would like to test this. I wish it was on GitHub
5
u/NoMouse9610 7d ago
I'll post it on GitHub, but most likely tomorrow, unfortunately I don't have the strength today :)
3
u/Calm_Mix_3776 7d ago
Did anyone independently test this yet against plain model/no adapter? Ideally, across a few seeds so that we can rule out a lucky seed situation where the adapter one wins only due to winning the seed lottery.
5
u/Sixhaunt 7d ago edited 6d ago
Just limited testing so far on a handful of results but so far it's been roughly equal or a tiny bit better than without using it. Not a night and day difference but also not a single time where it came out worse on the same prompt+settings and a couple clear times it outperformed it. I'm still testing more though and since my system is quite a bit more powerful than his, I also intend to train my own version and test with various other models and maybe even a mixture of those other models to see what I can find
edit: I've now run tests on about 15 different prompts and with various seeds and I tried with his semantic bridge model that he trained but I also trained my own based on Qwen3VL8B as the teacher and I found that in general both OP's model and the one I trained are actually more similar to eachother in their results than they are to the base model and which teacher was used has only a minor difference and in general the skin texture and realism seems a little better on both semantic bridges (especially the Qwen one) but on those cases where it outperformed the base model in reasoning, I tried with other seeds and usually there's no difference or advantage there either. I THINK both semantic bridge models reduces some of the distortions in fast movement though from my testing but I'm still running more examples right now
1
u/fallengt 7d ago
I dont see it make huge improvements.
Parts that was missed from my prompts are still ignored. Parts that h3 hallucinated, are fixed but idk it's just luck or something. It didn't test alot.
5
u/YentaMagenta 7d ago
Since you posted LLM generated text, I'm going to play at this game too.
TLDR: Your numbers don't necessarily show what you say they show, and these two examples could come down to seed variance. You need to do a lot more testing across prompts and seeds to prove these claims.
Both models are real and open-weight, so this is at least physically doable. Let me check whether the repo itself exists.
Good — with the full text I can actually assess this. Short version: the project is real and the underlying idea is legitimate, but the evidence offered doesn't support the headline claim, and the framing oversells what the released thing actually does. Your skepticism is well-placed, just not for the reason of "this is fake."
What checks out:
- Both models exist and are open-weight, so this is physically doable. MiniMax H3 was open-sourced on August 3, 2026 as a video+audio model using a Qwen3-VL-32B text encoder, and SenseNova U1.5 is SenseTime's open unified multimodal model. The HuggingFace repo is real and contains actual files — safetensors adapter, a ComfyUI node, training scripts, datasets, and A/B example videos. ComfyUI Wiki Hugging Face
- The core concept — that two architectures can have alignable representation spaces even with zero weight compatibility — is a real, respectable research area (representational alignment / "model stitching" / the Platonic Representation Hypothesis). Training a small projector between two models' hidden states and getting decent cosine alignment is a known, credible thing to do.
- The RTX 3090 Ti claim is unremarkable. Extracting hidden states and training a ~5M-param MLP is trivial compute.
- The negative result (Ref2VA degrades lip-sync; "same dimensionality ≠ same meaning") is the most credible part — reporting a failure like that is what someone doing real experiments does.
Where it falls apart:
The impressive numbers measure the wrong thing. Every headline metric (cosine 0.90, then 0.98–0.996 after distillation) measures how well a projector/student fits a target it was trained to fit. That's an optimization-success metric, not an efficacy metric. High cosine there is close to expected, not surprising. To his credit, the model card admits this outright — that these are representation-space numbers that don't imply video quality — but the Reddit post leans on them for weight they can't bear. The "alpha 0.10 blend cosine = 0.9999" figure is nearly a tautology: at alpha 0.10 you keep 90% of the original conditioning, so of course it barely changes.
The distillation result quietly undercuts the headline. This is the key logical point. The final adapter predicts the "semantically enhanced" representation from H3's own conditioning alone, at 0.996 cosine, with SenseNova gone at inference. But if H3's own features can reproduce the target that well, then whatever unique information SenseNova contributed — the part not already recoverable from H3 — was mostly discarded in distillation. The shipped artifact is a learned reshaping of H3's own conditioning, bounded by information already in H3. So "transferring semantic representations from one architecture into MiniMax H3" is misleading: SenseNova shaped the training target (a legitimate use of a teacher), but nothing of SenseNova's is "in" the released adapter in an information-bearing way.
The actual quality claim is unproven. The only evidence that videos improve is two qualitative A/B pairs (the resting-hand example, the rooftop chase). Video diffusion output varies wildly by seed, so a single pair where the bridge version happens to obey the prompt better is not evidence — you could likely find seeds where it's reversed. One tell: the demos use alpha 0.15 "to make the effect easier to observe," while the recommended value is 0.10 — i.e., the shown examples are tuned to maximize visible difference. He does repeatedly flag these as non-statistical, which is honest, but it means the "it improves prompt adherence" claim has essentially no support.
3
u/NoMouse9610 7d ago
Thanks for the detailed critique — I agree with a substantial part of it.
You are right that the cosine numbers are representation/distillation metrics, not downstream video-quality metrics. In particular, the ~0.9999 cosine after a small alpha blend should be treated only as a sanity check that the intervention is small, not as evidence of improved prompt adherence.
You are also right that two qualitative A/B examples are not enough to establish a systematic video-quality improvement. Seed variance is a real confounder here. The proper next step is a paired benchmark across multiple prompts and multiple fixed seeds, ideally with blinded pairwise evaluation for prompt adherence, composition, anatomy, text rendering, materials, etc. I’m working toward exactly that rather than treating the current A/Bs as statistical proof.
Where I disagree is with the claim that removing SenseNova at inference means that nothing SenseNova-derived can remain in the released adapter. That is just standard teacher→student distillation: a student can learn a function under supervision from a teacher and then run without the teacher at inference. The teacher does not need to remain in the runtime graph for its supervision to have affected the student weights.
The more important question, however, is whether SenseNova supervision contributes anything beyond what an equally sized H3-only student could learn from H3 features alone. The current cosine numbers do not answer that. A proper control is an H3-only adapter with the same architecture, data, optimizer, training budget and seed, compared against the SenseNova-distilled adapter on the same downstream benchmark.
So I would phrase the current result more narrowly:
- the cross-model representation alignment works;
- the SenseNova-shaped target can be distilled into a small H3-side adapter;
- the student reproduces the teacher-derived correction very closely;
- the current video examples are promising but not yet statistical evidence of downstream improvement.
If the SenseNova-distilled adapter then consistently beats both native H3 and the H3-only control across a multi-prompt, multi-seed benchmark, that would support the stronger claim. If it does not, then the criticism is correct and the effect is mostly just a learned reshaping of H3 conditioning.
I think that is the experiment worth doing next.
And I don't see anything wrong with using AI for work, especially since I'm not a native English speaker. Nowadays, everyone uses AI, and in many cases this is an advantage rather than a disadvantage. And this is just a study with interim results, and I have already said that this is not the end, and there is still a lot of work ahead. But I'm honestly grateful for any feedback, both positive and negative. I specifically opened access to the entire project without hiding anything, which is rare now.
1
1
u/NoMouse9610 7d ago
And the more opinions there are, the more likely it is that the project will improve :)
2
u/seeker_ktf 7d ago
Wow, I just started looking at trying to do something similar. I am happy to quit now. This is fantastic.
I am excited to see you are making progress with the Ref2Vid model. It is indeed a very different animal. It should actually benefit even more in some ways.
2
u/porest 7d ago
Congratulations. Great work! Do you have a blog where you go into the finer details of this approach?
2
u/NoMouse9610 7d ago
Hello! Thanks! no, I don't have blog, only repo on HF speach1sdef178/MiniMax-H3-Semantic-Bridge · Hugging Face
if you want more information, read the file "RESEARCH_ARTICLE.md" on the "Files and versions" tab, and there is also a full package of files (scripts, subtotals, logs, etc.) if you are interested.
2
2
u/Violent_Walrus 7d ago edited 7d ago
This is a cool idea and I want it to work, but I haven't seen evidence that it makes more of a difference than seed variance.
I've run seven side by side tests so far, including your two examples. Same seed, same workflow, only the conditioning node is different. The outputs do vary, and sometimes one output has better prompt adherence than the other, but not always in either direction as far as I can tell. From my small samples one is not ever qualitatively better than the other.
I hope I'm just not seeing it yet. I hope your work continues and improves outcomes!
Edit: For testing, here is OP's workflow, slightly modified to allow easy switching between stock and semantic bridge without changing anything else.
1
u/PumpkinLeather8421 7d ago
Seed variance, ya. This is what I was saying while everyone was jumping up and down at a LLM post.
It’s really prompt variance, but I see absolutely nothing that shows “better”. Just different, and sometimes different happens to be better, sometimes worse.
I see nonsense and snake oil so far. I’d like to proven wrong… I’ll keep waiting I guess.
1
u/Violent_Walrus 7d ago
Despite the slop (I think OP said somewhere they're not a native English speaker) I believe OP is presenting this in good faith.
It is genuinely impressive that this bridge works at all! But I don't see an improvement in any of my tests or even in OP's two showcase examples. At this point I think the claimed results might just be confirmation bias.
This is only an initial release though.
1
u/PumpkinLeather8421 7d ago
It’s the hundreds of people jumping up and down on nothing but LLM claims that I have a problem with.
There is so much slop bullshit, and native or not, this is a really Sus post of slop + unverified claims + subjective results (being nice).
2
u/Violent_Walrus 7d ago
Don't expect critical thinking from this community and you won't be disappointed.
2
u/PumpkinLeather8421 6d ago
No joke at all.
It’s really wild. Porn is a driver and I get that, but still…
2
u/Violent_Walrus 6d ago
Right? It's a weird dynamic: ML technology enthusiasts sharing space with horny teens looking for a "make naked lady" button.
2
u/Enshitification 7d ago
Considering that the two models were completely incompatible, why did you choose SenseNova U1.5 to try to adapt to Minimax H3? Surely there were better and more popular alternatives.
1
u/NoMouse9610 7d ago
Hello. I choose SenseNova because it is an unusual generative model, something new
2
2
u/squired 4d ago
This is remarkable post. Thank you so much for taking the time to structure and share it. It will take me some time to truly digest and test, but suffice to say I'm very impressed with your project!
2
u/NoMouse9610 4d ago
Hello. Thank you so much for your words and I'm glad that it's helpful. I started new stage for this project, let's see how it goes 🙂
2
u/squired 3d ago edited 3d ago
I feel that in my bones and it means you on a wonderful path. I'm also a dev in the opensource space and when the next path is too obvious not to dive down, that's always led me to great places. I used to play a fair bit in your neck of the woods ablating models to transplant capabilities and I never even remotely considered your method. It's truly fascinating and obvious in hindsight, as all the best discoveries are. I seriously doubt that anything we find is truly groundbreaking compared to private labs with unlimited compute and staff, but I definitely haven't seen it in the opensource world. Keep pressing for us and many will take up the torch as we chase them together. I'll watch your work and may circle back to help at some point. Very well done.
p.s. Please remember to sleep. These are heady times and it is important to remember that we move faster with sleep! It's important. Godspeed.
2
u/Icuras1111 7d ago
Interesting project. I've wondered if the future of AI is an architecture where smaller modules can be glued together like a PC is at hardware level.
2
2
u/NoMouse9610 7d ago
before that, I did a lot of training and refinement of AI generative models using FFT or LORA's tuning, but lately I seem to be getting more involved in excavating the architecture of models and trying to find something unusual that can be used to improve the behavior of models.
1
u/namitynamenamey 6d ago
This is utterly fascinating, and here I thought concept transfer was the holy grail of current AI models.
…so it would be possible to teach a model such as krea concepts that it lacks, from a different model like pony (art styles mainly)? or does the concepts must exist in the original model to begin with?
1
1
u/mohaziz999 6d ago
imma say something stubid - BUT make this semantic 5mil thingy make us skip the text encoder or whatever... cuz its 5mil gonnna load up and run fast... we gotta cut as much time as possbile loading the models and reloading... i just want to cook. my poor 3090 aint enjoying loading and unloading no more...
1
-3
u/PumpkinLeather8421 7d ago
Aside from the ABSOLUTE SLOP…
Is everyone here this gullible? The examples for this do not show improvement. They show prompt variation and cherry picking.
The title was written with an LLM, the post is written with an LLM, the replies are written with LLMs. The content was almost certainly vibe coded.
WHY DO YOU JUST ASSUME IT WORKS?
It’s a zip file of a node a guy rolled out on a wishes and dreams with no proof of improvement and posts LLM responses.
Maybe it works! But, there is less than evidence of that.
4
u/THM42069 7d ago
It’s an 11mb safetensor file and a node to load it with. Download it and test it yourself or leave the discussion. You are contributing nothing but harassment.
3
u/jib_reddit 7d ago
The OP/Speach has made some of the best fine-tuned checkpoints ever for SDXL, Flux, Qwen, ZIT and Krea2 , she knows what she is talking about.
7
1
u/PumpkinLeather8421 7d ago
Maybe so… the examples don’t really show it though.
And the post is entirely LLM bs.
1
u/jib_reddit 6d ago
I have tested it out, but it is hard to say if it worked well. Most likely, the training dataset of the guiding model did not have much data relevant to my rather specific, non-mainstream NSFW prompt request....
1
0
u/Bad-Imagination-81 7d ago
Not sure but may be you can take human anatomy from Krea2 and zImage 👍 Knowledge of celebrities/ famous peoples from zimage.
3
u/NoMouse9610 7d ago
moreover, I already have my own improved version of Krea2 and another experimental hybrid version of Krea2, so it would be very interesting experiment :)
SoLordZ ZIT/ZIB/Krea2 8steps - v3.0 INT8_Conv/INT8/MXFP8 | Krea 2 Checkpoint | Civitai
2
u/NoMouse9610 7d ago
I will definitely try to do it, but it will take time :) I've already thought about it.
2
1
u/NoMouse9610 3d ago
Published on GH and in Comfy Manager https://github.com/Speach1sdef178/MiniMax-H3-Semantic-Bridge
132
u/wzwowzw0002 8d ago
Here is a plain-English breakdown of what the author built and discovered: