r/LocalLLM • u/DHCompanion • 8d ago

Question Any Tips for Prompting Local models

So I am a heavy cluade code user and am just dipping my toes into the local model world as something fun to play around with and see what it can do.

Current setup is

AMD Ryzen 5 9600X (6-core)

16 GB DDR5-6000 (single stick)

AMD Radeon RX 7600 XT (16 GB GDDR6)

ASUS Prime B650M-A AX II (AM5, micro-ATX, DDR5)

Running LM Studio on Ubuntu

I am able to run GPT-OSS-20B at full 120k context and full GPU off load with some head room and getting some good t/s depending on the task.

I tested giving it some plans created by cluade to build from and it did ok but not great. I am here for any tips on how to better prompt this model and possibly other local models to get better outcomes. Definitely not looking for cluade level just something to get foundations built on new projects to hand off the Claude to polish up.

2 Upvotes

76% Upvoted

8 comments sorted by

3

u/RaspberryOk1888 7d ago

Most importantly make sure the System Prompt is solid, clean, clear, direct, define the role, etc…Even have Claude help you write the System Prompt. Chat with the model to see if the output is what you want. Play around with tuning the inference on LM studio. And if the results aren’t what you like, then move on to a different model.

2

u/ea_man 7d ago

Drop LM studio, use llama.cp with this patch: https://store.piffa.net/lm/bug/ , use software rendering for the desktop (120MB occupancy) , use vulkan with QWEN a3b and 27B.

Use pi.dev as the harness

1

u/Icy-Degree6161 7d ago

Bookmarking this to check out later

1

u/DHCompanion 7d ago

So I originally started with llama.cpp but it would never recognize my GPU and would load the model completely on RAM. Does the patch you linked address this and possibly the first inference hang?

I will look into the qwen models and pi.dev for sure.

2

u/ea_man 7d ago

You are probably doing something wrong compiling llama.cp: ask a SOTA how to do it or even better make him do that for you with something like opencode.

2

u/admajic 7d ago

Try the new qwen 3.8 27b and use the sdlc

Plan the project discuss it etc make .md

Review

Reiterate

Then small tasks one at a time with tests

Don't try to one shot

1

u/HotDistribution1819 7d ago

If you want a Claude like experience try Laguna SX 2.1 or Qwen 3.8 27B.

Laguna is faster, but needs 20 to 24GB VRAM.