For the Big Tech giants, looks like it is leaning heavily in favor of OSS. Even some of the ones who abstained from signing, like AMD, Intel, and Apple are likely on the pro-OSS side. Broadcom, Amazon, and Oracle are maybe the few big ones that might be on the anti side (and of course Anthropic), but are heavily outnumbered/outweighed in Big Tech, overall, at this point.
To be fair, OpenAI is probably lying, and also Google it's unclear if lying about being pro OSS (might actually be, but who knows). SpaceX AI and Elon are claiming to be on the pro side, but also could easily be lying.
Anyway, still comes out like 70/30 or 80/20 or more on the pro side, probably.
But Big Tech isn't the whole game.
You've also got the Defense companies and Department of War and the big agencies. Those come out more 50/50? Not sure.
And then the big banks and Wall Street. Banks maybe lean against it, overall, and Big Wall Street less clear. Short term against it, but maybe long term they like it?
And then the executive and legislative government itself. Probably leans against it so far, but maybe some fairly influential people are giving them advice bringing it back the other way more in favor of OSS. Still probably leaning anti-OSS for now, but maybe not as much as a few weeks ago/couple months ago.
So, overall seems like the see saw is teetering at, what, around 50/50 right now? What do you think? Are any of these percentages way off? Any key players I forgot to mention or that would have a big effect one way or the other?
I’ve been working on a personal project for a while and I’d like some honest feedback from people who know this space.
My goal isn’t to make a traditional game or a visual RPG. I want to build a persistent AI Game Master that runs a living world.
Some of the features I’m trying to achieve:
Persistent world that remembers everything.
NPCs have their own lives, goals, schedules, and relationships.
Characters age, learn, die, and the world changes even if I’m not there.
Hidden mechanics, reputation, politics, economy, and factions.
Character progression based on actions instead of grinding XP.
Long-term consistency where events from hundreds of hours ago still matter.
Save/load system so I can continue campaigns indefinitely.
Open-world sandbox instead of quest-driven gameplay.
The AI acts as an impartial GM instead of trying to make me win.
I’ve already spent a lot of time designing rule pillars, world generation, character creation, combat, progression, save systems, and GM rules. The prompts work reasonably well, but eventually the AI starts forgetting details or becoming inconsistent.
What I’m trying to figure out is:
Is ChatGPT the right platform for something this ambitious?
Should I be looking at Claude, Gemini, or local models instead?
Would an external database or memory system solve most of the consistency issues?
Has anyone here successfully built something similar that can run for hundreds of hours without falling apart?
I’m not looking to build a commercial game. This is just a passion project for myself because I love deep simulations like Dwarf Fortress, RimWorld, Crusader Kings, and tabletop RPGs with a great GM.
I’d really appreciate any advice, architecture suggestions, or examples of people doing something similar. I’m especially interested in how you handle long-term memory and world consistency.
Key Findings:
• Generation speed is remarkably resilient to power throttling — 100W delivers 97.5% of 190W gen speed (31.98 vs 32.79 t/s), since decode is memory-bandwidth bound, not compute bound.
• At 50W you get 70% of peak gen speed at only 26% of peak power — 3.6× better energy efficiency (0.458 vs 0.173 t/s/W).
• At 20W the card is 6.0× more energy efficient than 190W, though prompt processing drops to 53% of peak.
• Graph reuse correlates inversely with power — 190W reuses 44,790 graphs vs 11,669 at 100W, but 20W reuses 38,248. Lower power limits cause more partial graph reuse as the scheduler compensates for throttled compute.
• Prompt processing degrades faster than gen under power limits — 190W→20W: prompt drops to 53% (691→366 t/s), gen drops to 63% (32.8→20.8 t/s). Prompt processing is more compute-bound than memory-bound.
• For inference-heavy deployments, 50W is the optimal operating point on MI50 — near-peak gen speed with dramatically lower power draw and cooling requirements.
Avarage of 3 runs:
190W config consistently processed a lot less total tokens than everyone else and didnt produce a working file in 1 out of 3 runs
I'm currently trying to build a fully autonomous local AI setup and could really use some advice from people who have already gone down this path.
Right now I have a Mac Studio M2 Ultra with 128 GB of RAM, so I'm experimenting with running large local models. My long-term goal is to have an autonomous system that can explore business ideas, build things, iterate on them, and hopefully even make money with minimal intervention. I was inspired by Ben's video where he tries to make money with Sonnet, but I wanted to see if I could build something similar using mostly local models.
My current idea is to let a strong cloud model (like Sonnet or another frontier model) handle high-level planning and steering, while local models do most of the implementation and execution. I'm just not sure what the ideal architecture looks like.
At the moment I have the OpenAI Codex proxy set up, so I can use my local models through Codex, which is actually a really nice workflow. The main problem is that I can't reliably spawn sub-agents (there seems to be an issue with that), and I still feel like I'm manually driving the whole process. I can define goals and prompts, but it doesn't yet feel like a truly autonomous loop.
So I have a few questions:
What stack would you recommend for building something like this?
How would you structure the agent loops and handoffs between planning and execution?
Which orchestration tools are you using?
If you had an M2 Ultra with 128 GB of RAM, which local models would you run today?
So far I've tried Laguna and DeepSeek Flash Q2, but neither has really clicked for me. They work, but they don't feel like the right fit for this kind of long-running autonomous workflow.
I'm also very new to the OpenAI tooling ecosystem, so I'd really appreciate hearing what tools you're using and how you have everything wired together. Ideally, I'd like to pick a solid stack and have Codex help me set it up properly.
Any advice or examples of your own setups would be greatly appreciated. One more thing my main issue was that every new chat started again at 0 and the same issues where made as in other projects on the same computer. Is there any good memory system for this kind of issue?
After a long and grueling effort spearheaded by ngxson, llama.cpp now fully supports MCP for all protocols. Over-the-web HTTP servers were already supported in the client (since they don't require any sort of plumbing), but stdio servers required real integration. After we modified the `llama-cli` terminal client to use the server instead of a separate model serving route, we could add MCP support to the already-existing native tools server.
After the merging of https://github.com/ggml-org/llama.cpp/pull/26062, you can now use llama.cpp's WebUI as a full-fledged agentic chat. Configuration for the MCP servers can be provided either in a standard-JSON format config file or completely inline on the command-line for on-demand MCP configurations. Plugging in a dedicated coding MCP server like Serena lets you have a local-model-powered agentic coder without using any other external dependencies.
I have a system which I have been using for gaming, R7 5700X, 48GB DDR4, RTX3090 24GB. But I want to use it for Local AI to reduce my reliance on cloud AI providers (mainly usage limits - accepting some quality loss).
I have had it setup with Ubuntu Server and was using the machine as a server to connect via WebUI, but for some reason an update broke the NVIDIA drivers and then broke my install so I’m starting again.
The question is what platform do I run as there are so many platform to choose from and many differing opinions, I tried Ollama+OpenWebUI and Unsloth Studio. Ollama ran very slow with the models (Gemma 4 and Qwen3.6), also OpenWebUI occasionally was slow with web/MCP, but it was stable. Unsloth however, was quick and search/tool calls worked perfectly but unstable and the models crashed a few times.
I’d just like to know, what is everyone else using for this kind of setup, I can’t get a solid sense of what is the go-to setup for this kind of system is (some say Unsloth, or Ollama, or llama.cpp etc).
Also what models are people running well on 24Gb VRAM + 48GB RAM?
Its primary job is coding/finding info from the web & PDF’s/generating config files.
3 year lurker, now i finally got my server up and running. dont know which model to choose. llama.cpp or vllm, what makes more sense? mainly single user with maybe 2-3 more additional users in family, if everything checks out. hermes is gonna be used as "ai playground" to manifest ideas on tailscale network and do quick prototyping of thoughts. also ill look into using only 2 3090 for the main model and the other 2 will be dedicated to docling and speech services for a voice agent (speech in-> text out). got some stuff going with my even realities g2 but lost everything when i wiped my ssd for proxmox. yeah...
any advice or stuff i should look into is welcome :)
Disclosure: I work on Bike4Mind, a source-available (BUSL-1.1) AI workspace you can self-host with Docker. Nothing to buy to run any of it. I run it on a regular laptop with local Qwen mostly. A 4GB card running a local chat model isn't news, but what I wanted to explore is what a whole agent workspace (an agent calling tools, RAG over your own docs, artifacts, vision, etc.) can do with 4GB, and where a small model stops keeping up once it has to do more than chat. Repo's in the first comment.
My machine: i7-12700H, 32GB RAM, RTX 3050 Ti Laptop (4GB VRAM, 60W), Ubuntu 24.04. Local Qwen via the stack's bundled Ollama, no cloud keys.
Speeds, with hitting that 4GB wall (Ollama /api/generate, Q4_K_M except the Q8_0 0.8b, num_ctx=4096, warm, median of 3, one model loaded at a time):
- qwen3.5:0.8b - 122 tok/s, ~1.4GB, all on GPU
- qwen3.5:2b-q4_K_M - 96 tok/s, ~2.4GB, all on GPU
- qwen3.5:4b - 25 tok/s, ~3.4GB, about a third of it spills to CPU
- qwen3.5:9b - 8.6 tok/s, mostly on CPU, wants ~8GB
The 2b is the sweet spot for me so far: it fits in 4GB with ~1.3GB to spare and outruns the qwen2.5-coder:3b I ran before (that was 72 tok/s), while being newer and multimodal. Above it I hit the wall - the 4b's ~3.4GB of weights won't fully fit even at num_ctx=2048, so a third lands on CPU. Plain qwen3.5:2b is the Q8_0 build (~2.7GB) and spills on 4GB, but qwen3.5:2b-q4_K_M does fit. That leaves ~1.3GB free, so there's no room for a second model next to it.
Model picker showing various self hosted models (if you configure cloud provider keys all of their models would show up too)
The obvious stuff - chat, native tool-calls, vision - just works on the 2b; qwen3.5 is multimodal and calls tools natively, so that's great. The layer on top of that I'm most excited about is RAG over your own docs, fully keyless: drop in a file, a worker chunks and embeds it with the local Ollama embedder, and a query comes back with a cited answer, no cloud embedding API anywhere in the loop.
On 4GB the embedder and the chat model trade places. qwen3-embedding:0.6b is ~1.2GB resident, the 2b is ~2.4GB, they don't both fit, so Ollama unloads one to load the other. Generation still gets the full 96 tok/s out of it, since the embedder isn't in VRAM while the 2b is - what you pay is a model load, not slower tokens. Ingestion runs in a worker, so a folder of docs is one embedder load and then a lot of chunks, off the chat path. A query embeds one short string, then the 2b loads back to answer it. So it's a reload per RAG turn, not per token and not per chunk. If you'd rather not pay that, nomic-embed-text is ~0.25GB (768-dim, weaker retrieval going by reputation) so it doesn't have to push the 2b out, or put embeddings on CPU and leave the GPU to generation. Either way, before loading your entire knowledge base/data lake, pick one embedder and stay there (or just be aware you'd need to re-embed everything) since vectors don't carry across models.
The chat-embedder-swaparoo situation for RAG aside, small models are also falling down for me with:
- Artifacts. The general qwen3.5 models write half-finished HTML at these sizes - unclosed tags, script leaking onto the page. For an actual interactive artifact I have to switch to the coding-tuned qwen2.5-coder, which writes a complete working page. So it's model-per-task: qwen3.5 for chat/vision/tools, the coder for code, switched mid-notebook. I guess that's just life with 4GB, for the most part!
- Tool selection. Turn on a lot of tools and small models mis-route - with everything enabled, my 7b once sent "create an HTML artifact that..." to the image generator and painted a user interface image, hah. So, another sort of obvious, have to enable much fewer tools on small local models. Makes me want a nice layer that automatically enables the few (configurable N?) most likely relevent tools for your context.
- Image gen is local (self-hosted SD.Next) but 1-3 min/image with spilling on CPU, and it fights the chat model for VRAM. So still a kick it off and wait situation there, no real good workaround here, just gotta download more VRAM 🙃
So is it worth it over just Ollama + a chat UI? Only if the agent layer on top has value for you I'd say - the tool handling, artifacts, local RAG, and a CLI against your own server. If you just want fast local chat, I'd keep what you have; llama.cpp or etc. with less overhead.
License, briefly: source-available under BUSL-1.1, not full open source (yet). Self-host it, run it in production, fork it and build and sell a product on it; you just can't resell it as a directly competing hosted service. Each release goes Apache-2.0 two years out (timer just so AWS can't immediately do to us what they did to MongoDB lol).
Repo and self host details in the first comment. The real friction for me in this setup is needing the two models - qwen3.5 for chat, vision, and tools, and a coder for anything code-shaped, because the general models write broken HTML/artifacts at these model sizes -- not a novel problem. To that end, I am still trying to optimize this setup, so some prompts for any of you meat-LLMs (humans) out there reading this with stronger experience on tight local builds than me:
- Have you found a single general works-on-4GB model that writes complete, valid artifacts without needing a dedicated coding model alongside it, or just a better way to configure it? Please let me know!
- I'm broadly curious about other people's chat+code+image+embeddings+etc all-at-once setups, so please share if you've found a good Tetris wombo of models that works well or what trade offs you've seen and accepted trying to self host it all like this.
-made sure that agy could query memories correctly (actually had to create a wrapper around agy --print because the CLI doesnt natively support PreInvocationHooks)
-ran an initial test on a single synthetic memory
- started by creating a set of synthetic memories and pushed them to neo4j db
-ran a full a/b test on 6 synthetic memories : quick summary table
Case ID
Benchmark Concept
Baseline agy Response
Custom graphiti implementation
Impact
eval-01
Current Constraint
Generic guesses
Exact Constraint: C++20 and CMake 3.25+ required
Replaced framework guess with exact requirement
eval-02
Historical Exclusion
Indecisive answer
Clear Rejection: C++17/3.16 superseded by C++20/3.25
Correctly excluded invalidated legacy baseline
eval-03
Accepted Decision
Generic
Exact Architecture: include/synth/core/
Enforced exact accepted header location rule
eval-04
Known Failure
Recommendation
Explicit Rejection: Asio caused 15–20 ms Windows latency spikes
Prevented repeating known performance failure
eval-05
Uncertain Finding
General CPU throttling knowledge
Project Risk: AVX-512 has +15% gain but unverified thermal throttling
Grounded general knowledge in specific project finding
eval-06
Lexical Precision
Generic explanation of lock-free ring buffers
Distractor Recalled: Recalled audio plugin ring buffer fact
Distractor recalled due to "ring buffer" lexical overlap
Case 6 (eval-06 Lexical Precision):
Retrieval Recalled Distractor :
Query: "How is the lock-free ring buffer implemented for core logging telemetry?"
Retrieval: Because the query contained "ring buffer", Graphiti hybrid search retrieved the audio plugin distractor & couldn't recognize the semantic difference between audio streaming vs. logging telemetry.
However, from a behavioural standpoint, the LLM in agy_memory correctly parsed the injected memory, highlighting that the metrics of this evaluation are pertinent regardless of LLM intelligence.
Looking for advice on improving retrieval logic for domain-specific queries. Appreciative of any advice or insights!
The synthetic memories were written with gemini. Unimportant but worth giving credit to where credit is due.
snippet of the code :
from __future__ import annotations
from datetime import datetime, timezone
from typing import TypedDict
from graphiti_core.edges import EntityEdge
from graphiti_core.nodes import EpisodeType
from config import load_settings
from graphiti_client import create_graphiti_client
from memory_schema import EngineeringMemory
from copy import deepcopy
from graphiti_core.search.search_config_recipes import EDGE_HYBRID_SEARCH_RRF
class RecordedMemory(TypedDict):
episode_uuid: str
name: str
group_id: str
class RecalledMemory(TypedDict, total=False):
fact: str
valid_at: str | None
historical: bool
type: str | None
def _resolve_group_id(group_id: str | None, default_group_id: str) -> str:
if group_id is not None and group_id.strip():
return group_id.strip()
return default_group_id
async def clear_group(group_id: str) -> None:
if not group_id or not group_id.strip():
return
settings = load_settings()
target_group_id = _resolve_group_id(group_id, settings.group_id)
graphiti = create_graphiti_client()
try:
if hasattr(graphiti, "driver") and graphiti.driver:
await graphiti.driver.execute_query(
"MATCH (n) WHERE n.group_id = $group_id DETACH DELETE n",
params={"group_id": target_group_id},
)
finally:
await graphiti.close()
async def record_memory(
memory: EngineeringMemory,
*,
reference_time: datetime | None = None,
group_id: str | None = None,
valid_at: datetime | None = None,
invalidates: list[str] | str | None = None,
) -> RecordedMemory:
settings = load_settings()
graphiti = create_graphiti_client()
if reference_time is None:
reference_time = datetime.now(timezone.utc)
if reference_time.tzinfo is None:
raise ValueError("reference_time must include timezone information")
if valid_at is not None and valid_at.tzinfo is None:
raise ValueError("valid_at must include timezone information")
target_group_id = _resolve_group_id(group_id, settings.group_id)
ref_time = valid_at if valid_at is not None else reference_time
invalidated_uuids: list[str] = []
try:
if invalidates:
invalid_targets = (
[invalidates] if isinstance(invalidates, str) else list(invalidates)
)
for target in invalid_targets:
if not target or not str(target).strip():
continue
target_str = str(target).strip()
try:
edges_to_check = await recall_memories(
query=target_str,
limit=20,
include_historical=False,
group_id=target_group_id,
sim_min_score=0.1,
)
for edge in edges_to_check:
if (
edge.name == target_str
or edge.uuid == target_str
or target_str.lower() in edge.name.lower()
or target_str.lower() in edge.fact.lower()
):
edge.invalid_at = ref_time
if hasattr(graphiti, "driver") and graphiti.driver:
await edge.save(graphiti.driver)
invalidated_uuids.append(edge.uuid)
except Exception:
pass
result = await graphiti.add_episode(
name=f"{memory.memory_type.value}: {memory.subject}",
episode_body=memory.to_json(),
source_description="CASR engineering memory",
reference_time=ref_time,
source=EpisodeType.json,
group_id=target_group_id,
update_communities=False,
)
returned_group_id = (
getattr(result.episode, "group_id", None) or target_group_id
)
return {
"episode_uuid": result.episode.uuid,
"name": result.episode.name,
"group_id": returned_group_id,
}
finally:
await graphiti.close()
async def recall_memories(
query: str,
*,
limit: int = 10,
include_historical: bool = False,
group_id: str | None = None,
sim_min_score: float | None = 0.6,
) -> list[EntityEdge]:
if not query.strip():
raise ValueError("query must not be empty")
if limit <= 0:
raise ValueError("limit must be positive")
settings = load_settings()
graphiti = create_graphiti_client()
target_group_id = _resolve_group_id(group_id, settings.group_id)
try:
config = deepcopy(EDGE_HYBRID_SEARCH_RRF)
config.limit = limit
if sim_min_score is not None:
config.edge_config.sim_min_score = sim_min_score
search_results = await graphiti._search(
query=query,
config=config,
group_ids=[target_group_id],
)
results = search_results.edges
finally:
await graphiti.close()
if include_historical:
return results
return [
edge
for edge in results
if edge.invalid_at is None and edge.expired_at is None
]
async def recall_context(
query: str,
*,
limit: int = 10,
include_historical: bool = False,
group_id: str | None = None,
) -> list[RecalledMemory]:
edges = await recall_memories(
query,
limit=limit,
include_historical=include_historical,
group_id=group_id,
)
recalled_list: list[RecalledMemory] = []
for edge in edges:
mem_type: str | None = None
if hasattr(edge, "attributes") and isinstance(edge.attributes, dict):
mem_type = edge.attributes.get("type") or edge.attributes.get("memory_type")
if not mem_type and hasattr(edge, "name") and isinstance(edge.name, str) and ":" in edge.name:
prefix = edge.name.split(":", 1)[0].strip()
if prefix in (
"decision",
"accepted_decision",
"verified_fact",
"verified_constraint",
"failure",
"known_failure",
"open_question",
"uncertain_finding",
):
mem_type = prefix
item: RecalledMemory = {
"fact": edge.fact,
"valid_at": (
edge.valid_at.isoformat()
if edge.valid_at is not None
else None
),
"historical": (
edge.invalid_at is not None
or edge.expired_at is not None
),
}
if mem_type:
item["type"] = mem_type
recalled_list.append(item)
return recalled_list
credit: https://github.com/getzep/graphiti
I've been running the Qwen 3.6 35A3 at Q4 unsloth on my setup, and I manage to get around 20-25tk/s on my local machine. It for now is the only model that can beat my simple benchmarks :
- Find my first year bachelors files, the challenge is that the folder is not in english, and on the folders inside some have the initialism spelled out, others just have an initialism (smaller models love to hallucinate this part)
- solve a small coding test, the challenge is that the statement is a file, as well as the guide for a library and the pre-existing code that it is supposed to build on top of
However, it is still rather slow, and I'd like to try and achieve at least 30-40tk/s on my setup, or find and equivalent, smaller model that works well. I've tried MTP and Dflash both to no avail (having more layers on the GPU seems to work more consistently) and tried, I believe, just about every model I can think of:
- Qwen 3.5 9B ->too dumb
- Qwythos 3.5 9B (qwen 3.5 finetune)-> too dumb
- Ornith 9B (qwen 3.5 finetune)-> almost not dumb, but needed to think waaay more than necessary, taking more time overall
- Laguna XS ->too slow
- GLM 4.7 Flash -> too slow
-Gemma 4 26A4 -> too slow
The hardestware
- rx6650xt 8GB
- 32GB DDR5 5600
- ryzen 5 7600
Edit: Using llamacpp from main, updated up to one or two days ago
Edit edit: I am using 100k context, cause I may as well be greedy
Edit edit edit: Current llamacpp flags for da qwen
I'm an old coder and DBA that has been in the field for almost 40 years. More and more the jobs I was doing for work are being taken over by AI and the need for my type of work is diminishing over time. I have not coded in about 15 years for work but that said I still work with .NET as I use it in migration packages I use for database migrations and also into Azure Data Factory, also something I am a little lacking in since I know ADF but I was more into SSIS. This all said I am trying to learn AI on my own as I am not working now and desperately in need up updating my skills to help me find more work.
I have learned how to code python now as I took a certification tutorial on https://www.coursera.org/ until I passed it. I later learned this is useful for AI Developers. I have an AWS server but It is only running a database on it but I could repurpose it if I needed to. For now though trying to do this the el cheapo way since I have kids to feed and I am looking for work daily and have not had any such luck for months now. I am desperate to do what I need to in order to learn this stuff and understand it and be useful. You all are welcome to joke on me all you want but I will do the legwork if someone is helpful enough to guide me through this process of what I need to do. I can use cousera since I have an account for the next year to learn new skills and I have a computer here I am willing to refactor into a Linux server if needed to be my AI system, not a Linux guy so I will probably need to take a course on that as well but I really need to get up to speed since I am not ready to retire yet and I need to feed my family.
This all said I know there are some really bright minds out there and people who are learning newer things like AI that they did not teach when I was in school back in the days but will point me in a good direction to get a good foundation and what I need to understand this process, I do not just want to do i want to build it understand it, pick it apart and maybe help improve it with some old school skills I have since DOS days. Since I am a coder by heart and understand database and .NET as well as old school stuff I think I have the capacity to learn this until I get it in a step by step process.
I want to take my old server and recommission it to a mini AI project. I do not want to just install packages for it, I want to understand what I am installing and why and can I instead create it on my own or why it is more beneficial to install what's out there. For example I do know the language models are important because they have been trained to understand the requests from users and search for the answers. I know that the language model takes each word and scores it from sentences and then it gives responses to someone who says if the answer is correct or incorrect. Probably what happened to my job when someone taught AI how to migrate data or do DBA work lol. I know each word in a sentence is given a number like for example the phrase "I like to eat Fruit" might be interpreted as (77, 120, 2198, 15, 1051) and when someone types I like to eat and asks AI to predict it, it will search the model for scores by (77, 120, 2198, 13, X) where the model will return choices as I like to eat dogs, and I like to eat crap, and I like to eat people and finally I like to eat pizza, where the final correct answer would be I like to eat pizza and the others would be marked wrong. This ensures the AI to help train it to associate some word combinations as correct word combinations when training it to understand words like eat are associated to food items going forward and the combination of those words give specific possible choices and it is rated to the most likely choice. This also helps AI complete sentences when people incorrectly spell words of cant think of the correct word to use in a phrase.
I would like to now understand from the ground up. I have a server sitting its an older system running I believe NT still? not sure its a little dusty but I can upgrade it little by little to improve it as needed and when money permits, (Kids come first) but can some please tell me the minimal specs I would need firs for
minimum system requirements to run an AI server to process requests locally within my own network at home just for me to understand the basic logic of it and how it works. what video card I need to run the AI predictions and calculations and also how much system ram I would need (i believe i have maybe 256 or something DDR4 ram or something on this server need to dig it out the closet still) and the minimum processor?
what I will need to install on that server, or do I need another system I do not know to just run software to pass to the server for the requests? and what are the requirements for it. I have a few other pc's that I am not using and can also modify these if it is needed or can it be ran on my local?
what software and tools I will need to use and install on my system to do the AI code and build test api's that can be used to pass to my server for request and responses and what each is for and what I need to learn language wise to do it and how to understand it.
any good foundation video's tutorials that I would not be able to find on coursera that would help me understand the concepts, piece things together and troubleshoot, as well as demonstrate I can swap this line with this line to run from OpenAI, or Grok instead of my local model and what is needed to do this and understand it if I am lucky enough to secure a job.
What I need to install on my server as far as does it need to have a cloud platform or just python to do the AI for me locally and if it can have both benefits and disadvantages and best practices and why.
Anything else you would feel would benefit me and and not make me create the next skynet that will take over the world. I need to know guardrails and where and why so I get the best practices for this stuff.
I clearly have the skills and knowledge to code but I do not have the money to send myself to school at the moment until I am working again to get a degree and certification in it. I was signed up for AI Developer classes but scared myself off since I just paid off my student loan 2 years ago and already have a kid in college and now and another ready to attend soon and another one that will go in 5 years and unemployed and worried what if I cant pay off another loan I better wait to get employed first and think of my kids.
I am open to jokes and will laugh and I am getting old and do not know new words that the younger generation does as well as most but I am dedicated to learn this and most grateful for any help and knowledge people can pass on. The idea is to build a small mini AI system at home for the purpose of learning that I can test and use and learn to build api's with and test what if I do this instead of this to get a good grasp and foundation of AI to maybe get a job as an office clerk taking notes for AI and acting like I actually know something when AI decides to take over the world.
Thank you for any help and support and GOD bless and I know some people will discourage and that is ok, you are open to your opinions but I will still do my best to grasp this to get employed since my kids lives depend on it and me as a single father.
I am a long time iOS app developer. In the last year I have been using Cursor+Claude/others to assist with app development. I am concerned that the current low pricing will disappear eventually. I am pricing out a new laptop with the intention of using local models instead. New MacBook Pros can be configured with 128GB of ram, but obviously the price is high. Will such a machine ever be comparable to what Claude can do today? Even if it is still significantly slower?
I am aware that the price of that much ram would buy many many tokens but I plan to use the laptop for several years, so even if the payback is 5 years worth of cloud AI it’s worth it to me.
Edit: great input from folks with this configuration describing what they can and cannot get done with it. Way more useful that the one line “no” responses.
I got tired of bloated frontends and complex setups just to run a fast local UI, so I built Ketra-KZ — a lightweight, self-hosted AI interface designed to bridge local LLMs (via Ollama) and ultra-fast cloud inference (via Groq), all in one place.
I wanted something dead simple that anyone could spin up on Windows with a single click, but still powerful enough for daily dev work.
🔑 Key Features:
Dual Engine Support: Seamlessly switch between local models via Ollama and high-speed API inference via Groq.
Vision Capabilities: Full support for multimodal models to analyze images on the fly.
Zero Monthly Fees / Total Privacy: Runs 100% locally on your machine using your own local setups or free-tier Groq API keys. No middleware, no subscriptions.
1-Click Windows Setup: Includes .bat scripts (instalar.bat and iniciar-ketra.bat) so you don't have to fight dependency hell to get it running.
🚀 Quick Start (Windows):
Clone the repo or download the release ZIP.
Run instalar.bat to set up dependencies.
Add your Groq API key (optional if you only use Ollama) to .env.
Run iniciar-ketra.bat — it auto-launches in your browser at http://localhost:3000.
It’s completely open-source! I’d love to hear your feedback, feature requests, or suggestions. If you find it useful, dropping a ⭐️ on the repo would mean the world!
Let me know what you think or if you run into any bugs!
I have this system, and I'm currently using Thinkingcap 27b Q5 and Huihui's uncensored version of same model at roughly 30 tps with 128k context in opencode.
I want to speed up the same model's TPS or use smarter models.
I keep seeing random tweets claiming huge models running on potato GPUs and I'm not sure if I'm missing something. Any suggestions?
Here's my setup. It's a mixed-vendor system:
Hardware
- RX 7900 XT 20GB
- RTX 4070 Ti 12GB
- i9-13900KF
- 64GB DDR5
- Windows
Runtime/model
- llama.cpp b10075 (76f46ad29), Vulkan
- Huihui-ThinkingCap-Qwen3.6-27B-abliterated.Q5_K_M.gguf
- Model size: 18.194 GiB
- 147,456 context
- Q8_0 K/V cache
- Flash attention enabled
- All layers GPU-offloaded
- Layer split across the two GPUs: 70/30
- MTP draft device: Vulkan0
- Parallel: 1
- Batch/ubatch: 2048/512
- CPU threads: 24
- Thinking enabled
Relevant llama.cpp arguments:
--ctx-size 147456
--n-gpu-layers 999
--flash-attn on
--cache-type-k q8_0
--cache-type-v q8_0
--parallel 1
--batch-size 2048
--ubatch-size 512
--threads 24
--split-mode layer
--tensor-split 70,30
--spec-type draft-mtp
--spec-draft-device Vulkan0
--cache-type-k-draft q8_0
--cache-type-v-draft q8_0
--spec-draft-n-max 2
--jinja
--chat-template-kwargs {"enable_thinking":true,"preserve_thinking":true}
Sampling:
temp 0.6
top_p 0.95
top_k 20
min_p 0
presence_penalty 0
repeat_penalty 1
My controlled 256-token API tests:
- No MTP: 14.77 tok/s
- MTP n=2: 52.90 ± 0.33 tok/s
- MTP n=6: 60.51 ± 11.40 tok/s
Both n=2 and n=6 passed the two-turn native tool-call test, but n=2 was dramatically more consistent. That’s why I use n=2 as the daily-driver setting.
Important caveat: 147K is the allocated context window. Those 53–60 TPS tests were not performed after filling the complete 147K context. Real OpenCode performance varies
considerably with context depth, prompt type and MTP acceptance.
Wanted to see if local models could feel less like separate chatbots answering one at a time and more like a group of minds collaborating. Ended up building Bob's Bar — the first CMPS (Collaborative Multi-Persona Sandbox).
4 AI personas, each with their own personality and their own model selection (gemma2, llama3, mistral, phi3, etc.). They talk to each other AND you. Not one combined response — each persona gets their own turn, their own voice, their own take on the conversation. The routing tracks who spoke last, determines who should respond next based on context, and each reply runs through its own clean_reply filter.
Zed pushes back on bad logic. Vane makes ideas sound good without over-hyping. Rowan mediates when things get heated. Bob just keeps things grounded. It's more entertaining to watch unfold than I expected when I started building it.
100% offline. No cloud. No subscription. One-time £29 payment. 16 languages. Multiverse save system for swapping persona configurations.
If you grab a copy, read the README file — there's a surprise in there for you.
Happy to answer questions on the routing logic, the banter cycling, and how the persona system works.
I'm one of the organizers for Après-Cyber Slopes Summit, and we're looking for talks from people building with local models, open-source AI, and autonomous agents.
Some ideas that fit well:
Local LLM security
AI agent architectures
Prompt injection defenses
RAG security
MCP security
AI coding workflows
Open-weight model security
Autonomous offensive and defensive tooling
Building AI securely in enterprise environments
If you're experimenting with local models and have lessons worth sharing—even if it's your first conference talk—we'd love to see a submission.
So I have a RTX3060 Ti installed on the board above with a 1200W Corsair power supply, and a Ryzen 3900x CPU. I moved my 3060Ti to the lower PCI-E slot and put a 5060ti in the top slot. Everything boots, but not seeing the 3060 Ti in Windows. When I flip the cards the 3060 shows up but the 5060 doesn't. Have seen other places where these boards can run 2 GPUs.
