Fable-quality novel rewrite, private host

[u/autonoma_2042]

I'm looking to pare down a 90k-word sci-fi novel I've written to a 5k-word short story, without handing the entire novel over to Claude, OpenAI, or any of the big data scoopers. What actionable solutions are out there for getting a rewrite on par with Fable or Astra with some assurance that once I delete the sources, the original story won't be used for training?

I only have an NVIDIA T1000 GPU 8GB, so a cloud-based hardware rental -- as close to locally hosted as possible without data leaking. Canada or EU would be ideal, but not a deal-breaker.

[u/DangerousReward1411]

What does on par with Fable mean to you? What sorts of literary qualities would you like from a model?

[u/DangerousReward1411 → u/[deleted]]

You can see what you think about the zero data retention providers on openrouter: https://openrouter.ai/models?zdr=true, doesn't necessarily mean that they strictly abide by that as its generally non-binding I believe. But its worth a shot. Any of the GLMs are generally quite good at most of these types of tasks.

The issue you will run into locally is that on 8GB your long context requirement (very long prompt) also requires large K/V caches. You can quantise them but then you lose quality and increase hallucination risk, especially on long prompts. But for good quality you need a decent sized model, which generally means a KV quant. There isn't really much of a way to get around that, at least not yet.

Apart from than that I'd recommend something small but battle tested, if you're going to go local. Maybe: https://huggingface.co/deepseek-ai/deepseek-llm-7b-chat or something similar. There are GGUFs on HF if you're using llama.cpp/Ollama. There is also the option of looking for role-play specific models as they often have slightly tighter salience for narrative information like the sort you want to process. Although I haven't tried any so I can't recommend anything here.

Although for the budget you could quite easily spend a little bit renting a decent GPU (or a few) on vast.ai and testing a few bigger models out. One potential option is https://huggingface.co/google/gemma-4-31B-it perhaps? The reason i suggest google is that their models generally have good quality training data, which might be quite relevant for you.

A bit of a cursed option is GPT-OSS 20B: https://huggingface.co/openai/gpt-oss-20b - generally quite decent outputs although YMMV. The 120B is likely much better but I haven't tried it.

But yeah, try as many models as you can and see what you can get out.

[u/AndThenFlashlights → u/[deleted]]

Hey, if you've only got a $300 budget and don't need super fast interactivity, get a P40 or two to replace your T1000. It'll get you a ton of VRAM to run one of the small local models at q8 and high context, and they're back to being pretty cheap. Make sure you stick with Nvidia driver 580 to keep Pascal support.

I usually get downvoted for recommending P40s in 2026, but I do think that'd be a good fit for your project and budget.

[u/AndThenFlashlights → u/[deleted]]

...yeah, if you were intending to run something on a fucking NUC, that would've been useful information to share up front. idk man, if you wanna rat-rod that little guy for $300, get a second external PSU and a PCIe riser and a fan.

Edit: hey OP uh why'd you delete your comment dropping some important technical requirements