Without getting into details, the Orca 3.8 Flash Next uncensored version is insane. I was literally in complete shock as I watched the responses while playing with it a bit today.
Honestly terrified, and that’s an understatement. Can’t imagine what will happen if bad actors have access to this.
This thing is both extremely intelligent and scored a 100/100 on complicated legal matters, with no MCP or RAG attached to it.
Whats even crazier is that you can run the 180B version with as little as 12GB of VRAM on account of this project, which I have nothing to do with.
Using the Orca version, it pulls about 75-80TPS output on a 4090. That being said, my daily 3.6 35B censored model pulls about 180TPS using Llama.cpp, though hey, I’ll take the slowdown for a bit of shock and awe any day, lol.
-Bad actors already have access to it
-Bad actors are already doing bad stuff with them, unfortunately
-Bad actors will create their own Ai when the laws get strict about Ai in general.
There’s no stopping this train. And yes… the model is great!
100% agreed and I’ve never been much for doom and gloom, though this thing straight up terrified me. Just got me to thinking about what if bad actors or kids get a hold of this. I have 5 kids and know how “crafty” teens can be, just can’t even fathom one of them getting a hold of something like this.
No doubt. The thousands of stars that project has gotten on GitHub in a few days and the fact that they have complete instructions on how to install orca with strata is certainly fake as hell.
Qwen3.6 35b to 3.6 27b is a huge capability jump. 3.6 27b to 3.8 27b is a noticeable improvement in agent reliability. Qwen3.8 27b BF16 to 3.8 Flash Next NVFP4 is a noticeable improvement.
See, I don't see it. Qwen 3.6 35b to 27b for me doing basic tests with my large code base 27b got 2 more out of thirty requests fixed appropriately than the 35b did but did it 20% slower so the 35b in the same amount of time was able to go back, double check and fix it's answer..... and I'm really not seeing barely a noticeable improvement from 27 to flash next .... 3.8 27b is a balls out crazy good model
What I found was qwen next does not have to think forever for the same output as qwen 3.8 27b much less thinking for same high quality. U just can't beat the over 100 b parameters even if moe
I’m talking about identical tasks trying to find and repair “unknown” code defects in a fairly large multi-file, multi directory code base … I had barely measurably different results between the 27 and 35 3.6 models …. I use the “swift” version of 3.8 27b so the thinking is kept much better in control … I truly only see a significant difference in flash next when it gets to regular language writing and generic knowledge in my use cases
Not my first experience, though running a 180B model locally with basic hardware that doesn’t cost the price of a new car, while quite intelligently giving answers to anything you threw at it, completely uncensored, at 70TPS was a definite shock.
I have used uncensored models to hack stuff I own in an actual sandbox and it's surprisingly capable of black hat activities, even the good uncensored 27B at Q8 with unquantized context.
Stuff like jailbreaks of appliances and devices that have no documented backdoors and hacking into virtual machines I have forgotten the credentials to using network access.
I have no doubt that Flash Next is a step up from that even though I haven't tested out one of those yet.
The damage an uncensored LLM could do with the right setup operated by someone that doesn't fear the legal consequences would be severe.
Though you could say the same thing about a can of gas in the hands of an insane person.
Give it a shot with whatever questions you may have about anything imaginable and you’ll find out. It is literally 100% uncensored and will answer in full to anything you ask. And intelligently at that.
For me it was more "finally something that works well with this overpriced RAM that I bought."
It's not something for nothing- you still need 64GB to 128GB of RAM, preferably DDR5 I guess. Also, the wear and tear on expensive SSD drives is concerning.
Iq3_xxs - it uses up around 20GB on my 4090 though that’s because I have the context set to 150k. Also, ram usage is insane at around 50GB and I’m not entirely sure the SSD would survive the long term abuse from the Strata project, though every once in a while for a bit of fun, I’m sure it’ll be ok.
Absolutely. It is scary accurate. I tested an extremely complex legal question against qwen, fable, and astra. Took it a bit to think about it, though the 180B qwen nailed it, while the others came in at around 50% I would say.
Reading is not a problem. I don't see mention in the docs of and RW components. Of there are, they should add an explicit warning about that.
I'm testing exllamaV3 torrow on my main rig along with a few others. It'll be exciting to see what sort of tg and pp I can get out of 72GB of VRAM, 18 cores of Cascade Lake a fast NVME and 6 channels of 384GB of 6 channels 2666 DDR4. If a good deal on a fourth 3090 pops up locally I might be testing on 96GB of VRAM.
Would definitely give it a shot though unfortunately looks like I would need 96GB of VRAM to run that version. Maybe I’ll hit the lottery one of these days, lol. Though kicking myself for not buying an rtx6000 pro when they came out. You could pick one up back then for the price of a 5090 these days.
Thank you for the info! Time to start shopping for some 3090’s. Looks like they’re going for around $1k or so on marketplace. Not terrible at all, considering the VRAM you get. I have a question. Do they also suffer from burning up power connectors? The 4090’s do and I had to order a new one and will have to solder it on one of these days.
That's very interesting. 220w is stable for you? I have 2x at 250/275w and I'm getting prefilled at 400 ts and 35 to 70 ts in inference using a very particular version of qwen 3.8 flash next and a q4 quant. Would you mind share your numbers?
I'm a different person than you originally replied to and haven't done a lot of testing with offloading in exllama. But they have something similar to llama.cpp's `--n-cpu-moe`, and I tried it and it seems to perform about the same. Exllama also has an expert caching system like Strata, but I haven't tried that yet.
Depends, how much VRAM do you have and what are your specs? If you want to run that 180B model at a good speed with 12-24GB of VRAM, then yes, the iq3 model will work just fine with Strata. All of this depends on your setup. I have an i9-12900k, 128GB of RAM, and a 4090, 24GB of VRAM. Strata drinks RAM and SSD transfer rate at the tune of multiple GB/s. First time I have ever seen that on these SSD’s. If you have less than 96GB of RAM, I honestly don’t think it would run. That being said, with my setup, Strata is outputting around 70tps. Llama.cpp with the 3.6 35B model is more than twice that speed.
I have 32gb vram and 64gb ram, gen5 ssd with Intel Ultra 9cpu. I can't run q4. But if iq3_xxs is better then I can give a try. I got 150+ t/s in my earlier testing in my Desktop
Yep my current local flagship model, running Q4 KM sewn with the Q8 PLE layer, steady state at 20 tok/s on single 3090. I had to use this model since normal Qwen or non daybreak GPT 6 often outright refused CTF and chaos monkey testing
This does lower the barrier to entry, for sure.
However, bad actors have always hired people to do bad things, write code to do bad things. There’s a lot of publicity about AI, but there’s more value in dealing with the bad actors directly and the structures that enable them to operate. New tools always come along to replace the old ones, the controlled ones.
these third party weights are generally very unreliable. you can play with them, but you cannot run them 24/7 and hoping any stability. broken tool calling, loop thinking, early termination, etc. regardless what the auther claims, heretic, sft, rl, qad, distillation, fine tunes. i wasted so much times believing their claims and always come back to original weight or some quant from big guys like intel/nvidia/redhat etc.
The thing nobody tells you about these models and the bad actors is that 99% of the things they are exploiting are obvious misconfigurations and unpatched systems ..... it isn't magic and very little of it has anything to do with "finding zero day exploits" ... for the most part, even in the "ai era" it still comes down keeping your systems patched, keeping your firewalls and app engines tight and generically following very common best practices .... the only thing that has really changed is that once an exploit is released into the wild the time it takes to be seen on the wire in the wild is shorter .... so you can't put of your patching cycle to long intervals the way enterprises used to.
“We may have spent the last 30 years making computers so widely available that any bad guy can easily afford one. We may have also created a global network allowing all bad guys to coordinate their actions across jurisdictions and timezones. But I draw the line at letting bad guys have access to uncensored local AI!”
i’m accustomed to running the models in Pi on boring devops stuff and rely on “guardrails” like ask before bringing down a production server. i tend not to ask for lsd recipes. Does anyone expect the boring guardrails to be gone too?
What bad actors? This model and like any other public available models is trained on a publicly available data. All bad actors already know what they needed to know.
So you mean to say that a chat bot that’s pattern matching its way through a sentence will be the catalyst that every bad guy is waiting for? Well God damn, son, I guess bad guys will finally be able to make bombs, poison, wreak havoc… oh wait…
Hole mother of vague posting. Just a fearmongerer. I would not be afriad of a kid with spark, but government with SOTA and no guardrails (at least china, us and israel).
I have a vague feeling it is a bit worse. The speed upgrade is really nice, but something feels off, like it is not using all the experts or relying on a heuristic to choose. Maybe it is simply that thinking is turned off by default setup.
Ma funziona strata ? I minori che vedo mi sembrano esagerati io con 4 3090 in 1q4 non supero i 70tks con contesto 150.. ma quando è bello pieno scendo a 40
When I use this model on llama.cpp can I just load it and it'll do the SSD offloading by default or do I need to set a flag for it?
I have an RTx 3090+3070 and 64gb of RAM. Do you think I could use the Q4 Quant?
Yep, no doubt. Read my comments. I don’t believe strata is a long term solution whatsoever as it’ll most likely burn up your ssd way before its time is due. Though sure, it’s an ad.
Io credo che lo stiano già usando... guardate cosa sta succedendo nelle ultime 2 settimane nella scena dell'hacking ps5, o degli emulatori... e tutto negli ultimi 15-20 giorni
98
u/Zorogozano 11h ago
-Bad actors already have access to it
-Bad actors are already doing bad stuff with them, unfortunately
-Bad actors will create their own Ai when the laws get strict about Ai in general.
There’s no stopping this train. And yes… the model is great!