How to run a Local LLM like a Boss

I have to use local LLMs now as I am out of work and I can't afford the subscription. I'll be honest: I do miss working with the large agents, they were a lot easier to work with, but I think I have found a workable local LLM process. I'm getting a lot more done than I would if I were simply writing code. I'd say I'm about five times faster using a local LLM.

I have a very lean setup. I run my LLM on a dedicated 12GB 3060 RTX card. I set up my Linux box to support VS Code and a web-based client. I use Gemma 4 & Qwen 2.5 coder; they independently take up 8GB of active memory and gives me a 32k context window. It fills up the 12GB card perfectly, and no more.

Here is how I make it work.

Don't use agents

Agents, "agent mode," or "activate tools" simply get in the way of smaller models. I've lost count of the number of times my smaller model has forgotten how to interact with tools or has lost track of my changes. It's incredibly frustrating when you're in the middle of composing a solution.

Let the chat window do the editing work, and let the model do the reasoning. I use a GitHub chat client in VS Code (Ask mode, not Agent mode); it's very stable and lets me quickly switch to a larger model in the same tool. The chat window allows me to inject code directly into the file I'm looking at.

Build it One Component at a Time

You can't turn a smaller model into a coding agent. It's too big a task; however, you can effectively pair-program with one. That means working on one component at a time. You write much better code this way, the human gains a richer understanding of the code, and can undo bad code when it happens. Most importantly the human controls the architectural decisions, not the AI. When an AI agent makes a bad design decision, it will keep going and churn out spaghetti. When I see that happening in real-time, I can prune it or write something better. The LLM, being a word-completion program, will naturally continue with my fixed code without complaint.

In all honesty, I think coding agents (vibe coding) are a major disaster waiting to happen. Ignoring the obvious and massive security issues, it completely divorces the human from the development process. That leads to more bugs for the business, and fixes are more expensive because the developer has to learn the architecture in real time when fixing the bug.

Keep your Prompts Specific to the Task

Being highly specific with your prompts helps a lot. Think about how you talk to a child compared to an adult. You can ask an adult to do anything, but when you ask a child something you need additional conditions to keep them on track.

Adult Exchange

• Adult 1: Can you go to the shops and buy me a lemonade?
• Adult 2: Sure, no problem.

Adult/Child Exchange

• Adult: Can you go to the shops and buy me a lemonade?
• Child: Sure.
• Adult: Don't talk to strangers.
• Child: Sure.
• Adult: Don't pet the neighbor's dog.
• Child: Sure.
• Adult: Don't buy sweets with the change.
• Child: (sigh) Sure.
• Adult: Shut the door after you leave.
• Child: I'm not that dumb!

Smaller LLMs are like children; they need more instruction. I find I have to remind them repeatedly within the same context about specific rules.

Open a New Chat with Every Task

Context pollution is real. If you use a single chat session for everything, after an hour or so it starts pulling from older conversations within that chat. This is very irritating and very noticeable on smaller models. So this is how I work: set a goal like "build a navigation component," open a new chat, get it done, close the chat, and start the next job.

Stuff your Context

Agents use an `Agents.md` file to get general instructions about the project. Why do that when you have a perfectly good project file? A project file contains dependencies, which is a lot of content for an LLM to consume. `Agents.md` is a solid couple of paragraphs stating clearly what to use.

`Agents.md` file is only good for agents; for a local LLM via a chat window, you can simply "stuff" the context. That may be on the server or within your client. But once you do, the LLM becomes a lot more efficient.

I also found that having multiple specialized models is a better strategy than a single "developer style" model.

Oh, and you don't have to write this yourself. You can ask a larger model to build one for you based on your specific requirements.

Jump-Start Your Local Models with a Larger Model

It isn't easy starting a project with a local model alone, mainly because it requires a lot of reasoning. When it has to reach too far to do something, it often gets it wrong. I found myself writing most of the code until the LLM learned enough to take over. I eventually realized I should work with a larger model for a few initial components, then let the smaller LLM take over. That strategy worked and saved me a ton of time.

I found testing to be a major sticking point when working with a local LLM. Simply using a larger model to create a couple of tests gave the smaller model essential knowledge regarding how to mock, which libraries to use, and proper test structure. Once it has that, it's as good as its larger parent.

Switch to a Bigger Model When You Need to

Smaller models can't do everything code-wise. I'd say most of my projects are 70% local LLM generated and 30% human/larger model generated. I know it's time to switch when the LLM can't solve a problem after three or four attempts.

Conclusion

To be completely honest, I think this is how every developer should work. I built 13 projects in three months, and I can answer technical questions about every one of them. If I had "vibed" them, I might have done it in a week, but I wouldn't know a thing. That's very dangerous for a business; developers need to understand how systems work so they can fix and improve them. They also need to keep an eye on the agents, as they can do really stupid things when left alone.

If I have one criticism about my setup, I wish it were faster. It would be nice to have a larger graphics card; the five-second wait time is a bit of a stinker, but it's a small price to pay compared to the cost of tokens!