you cant run an LLM on that
I run a local LLM at home and people are shocked that I can run it on the hardware I have. It took a lot of trial and error configuration wise, but I found a happy balance.
The hardware
I have an old gaming machine that I built in 2017. It was fairly decent Pentium 7 quad core with 32gigs of ram, but used a very unsuitable gtx 5gig card. I scoured the internet and found a second hand RTX 12gig 3060. It was probably used for crypto mining but I don't care. I could not afford a new one and I don't think you can buy one new for that hardware.
My motherboard has onboard graphics, so I configured it to use that for OS graphics, and I configured the RTX card to only the LLM. This gives the LLM a full 12gigs to work with. Without doing that the model would only have 8-10gigs to work with. That simply is not enough for a decent model to run.
The Software
I installed Debian 13, hardened its security in a multitude of ways, and then installed Ollama and WebUi for the web interface.
I tried several models and I landed on gemma4:e12b. It has the best reasoning for the hardware, and was trained within the last few months.
A big issue I discovered with AI assisted frontend development, our frontend libraries go out of date quickly and open weight models don't keep up.
The LLM
The LLM uses 8gigs of memory on the card, and I have the AI optimized so it runs on 4gigs of memory. Getting it to work in a consistent way proved to be the biggest pain. I had to configure the model in Ollama, the model settings in WebUi, and WebUI interface settings. The interface settings are buried in the webapp's admin a panel. It took several weeks to find them, AI was no help. Ended up just digging through the app until I stumbled into them.
I basically limited memory to 4gigs, and configured WebUi to aggressively compress everything. Without compression the model would answer 3 questions and then stop working, and you would have to load a new chat to get it going again. Adding compression lengthens out the question window, I have had long discussions on there now, I've not run out in a while.
I did try 4gig models but they are not as good at reasoning as the 8gig models!
How its used
I have AI models on two machines: my local mac mini and remotely on my Debian machine via WebUI.
My local Mac Mini runs gemma4:e4b model and that is just for auto complete and fix suggestions in VsCode, and I rarely use the chat window. VsCode is practically unusable without an AI which is a real shame.
I use my Debian machine for chat, code review, and code generation. I don't use agents on my projects, I believe the developer should keep creative control, and an eye on the code. Working component by component works for me. I can implement a single component, test, and storybook it in under an hour. Without AI that would have taken me a day to build and I would have been cutting corners. I really do have to pinch myself these days, I cant believe how quickly I can build an app now!
I tried running the 8gig model on my mac mini which is a M2 16gig but it was unusably slow. It would also break apps randomly because there was not enough memory. It runs happily with a 4gig model though.
The Debian machine is surprisingly quick (Once the model is loaded), it is as quick as a frontier model at generating code, class by class mind. If I ran it as an agent I suspect it would take far longer than a cloud model. I never suspected that I could have real time code generation locally and I am really happy that I do.
In Conclusion
Local open weight LLMs give you all the benefits and none of the security risks associated with cloud models. LLMs are not secure, and big tech companies have a large financial incentive to spy on your activity. You think its bad now, wait and see what comes next.
Im very happy I have a safe LLM on my network that only I can access. I hate the fact that all my local apps have AI plugins now. I am running a Mac Mini currently, its definitely getting replaced by a Debian machine before the end of the year. I've always run linux but I will be fully switching in a few months for sure.
Open Source, Open Weight for Life!