Do you host your own ML / AI / LLM? What do you use, and what do you use it for?
You must log in or # to comment.
Found vLLM to be the most efficient local runtime service. And “ray” as a good (but complicated) way to distribute the load: https://docs.ray.io/
I ran through lmstudio because it really eazy, I ran some kind of qwen 3.6 27b imatrix neo code DI, it is the best local model for coding I tried, I think it can be better than some cloud model
Why would I?
I set up ollama on our thinkstation in the lab and I use it for looking up documentation, generating readmes, searching papers, and sometimes coding when I know what to do but don’t feel it is worth it to spend time on it myself. So basically the chat with web search.
An aside for anyone reading this:
https://sleepingrobots.com/dreams/stop-using-ollama/
And that barely scratches the surface. Please.
Use anything but Ollama. Even APIs.
I agree that the concerns listed there are smells, and I wasn’t aware of some of the options listed there.
Thank you for sharing this!
looks like extreme nitpicking without any real issues beyond some VC funding a FOSS issues.
//whyre you spamming the comment to everyone? its quite alarmist actually
I completely disagree.
Frankly, I find the description “VC funding a FOSS” offensive. They aren’t funding the engine. I’ve been messing with LLM inference engines since 2022, and Ollama is the worst I’ve seen in the community.
They misname models for SEO. They leech off llama.cpp while deliberately hiding attribution yet redirecting GH support requests there. They sometimes make their own GGUFs+forked releases which are broken and incompatibile with upstream llama.cpp, just so they can get a release out a day ahead for hype, even though it doesn’t really work and they’ll never upstream one line. They set a default context size thats basically unusable, they screw up chat templates and deep internal code with no obvious indicators, they release suboptimal quants without iMatrix, they gate you into their internal quantization repo and model card format, they hide model downloads on your hard drive, they mess with standard APIs for no good reason other than to mess up other backends. I could go on and on.
And if that’s all fine, they’re enshittifying the app with closed code, and pointers to cloud models.
They GIVE LLM inference a bad name, by making it a terrible quality engine that happens to show up in search as the “default.” Hence the comments below of people being unimpressed with local inference. And they sap attention from actual llama.cpp devs, without contributing a single dime. Everyone in the localllama communtity hates their guts, and that’s not even getting into the interpersonal drama they’ve stirred.
They are a leech that’s a net drag to the whole community, that we can’t get rid of because they’re attention grifters. And they’ve gotten worse and worse over time.
It’s more morale to use any cloud API over Ollama, in my eyes. They’re a grift.
EDIT: And, to be clear, I’m not against VC funded downstream stuff.
LM Studio is good! Even though it’s closed source.
Tons of downstream projects are great.
Yes. My Actual Intelligence lives in my head, and runs mostly on coffee.
Just coffee?!? That’s cool.
Mine runs on:
- coffee
- spite
- tortilla chips
- & shame
If that’s not already on a shirt it should be
Do you get many hallucinations?
Only when I’m deprived of coffee.
Would flowers work instead?
critical security bug: if coffee is taken away my head hurts :(
As we know AI stands for “An Indian”, so if you’re not from India, its actually impossible to self host.
Well, unless you manage to trap one in your basement, but that would violate human rights and hopefully also break the laws of your country.
You may be confusing Indians with gremlins. Which might explain ChatGPTs obsession with gremlins
I’ll make sure to send you flowers, Algernon lol
That doesn’t sound artificial.
With sufficient coffee, mine shows considerable artifice.
Plastic flowers.
I recently gave it a try with qwen3.5 and deepseek coder v2. I have a RTX3090 and these are the largest models that can run comfortably on it.
Conclusion, they are both fucking useless. Free tier claude runs circles.
Yeah :(
Were not there yet on consumer rigs.
Did you serve them with ollama?
It’s basically broken, if you did. Try the same models over API, and you’ll see what I mean.
Is there an alternative to ollama? The point was to run something locally.
https://sleepingrobots.com/dreams/stop-using-ollama/
And that’s not even all of it. Basically they break models in many ways, and they’re slimey Tech Bros.
LM Studio is better, and easy.
If you’re on Nvidia, and want to run optimally, I would use the ik_llama.cpp fork. On AMD, regular llama.cpp. On a Mac, use an MLX runner (Like LM Studio) with an MLX quant (ideally an MLX-DWQ quant).
It’s all pretty technical, and… thats kinda the point. LLMs are just too performance sensitive and too finicky to not have a grasp of how they work. There is no “easy button” to run them, there can’t be.
But if you don’t have time for that and just want to see if it’s worth it, I’d suggest self hosing your own UI, and trying the dirt cheap APIs of models you can theoretically run on your setup. This will give you a “best case” taste of what they’re capable of.
Oh, and I just saw you have a 3090.
To get more specific, you can actually run way better models than Qwen 3.5 and Deepseek coder (both of which are very obsolete now). The best that’s practical depends on how much CPU RAM you have, but at the minimum you can do Qwen 3.6 27B, with a more optimal quant like ones here: https://huggingface.co/ubergarm/Qwen3.6-27B-GGUF/tree/main
Or Gemma 31B QAT: https://huggingface.co/unsloth/gemma-4-31B-it-qat-GGUF
If you have 128GB CPU RAM, I can upload my custom MiMo 2.5 quant. That should “beat” the cheapest Claude, give or take.
If you have 64GB, I’d suggest a quantization of Step 3.7.
If you have 32GB or 48, I’m not sure. I’d need to look if any “small” MoE is actually better than Qwen 27B now.
I started out playing around with code generation using Ollama/open-webui and qwen 2.5 coder 14b on a 3060 12GB, but ended up on a winding journey with an ex datacenter card called the AMD V620. Its roughly equivalent to an RX 6800XT, but with double the VRAM. At this point i’ve really done nothing productive with it but learned a lot about bios settings, GPU/ROCm drivers, and custom fan solutions/PWM controls trying to get it setup and optimized haha.
It’s pretty sick though, that amount of VRAM with 512GB/s bandwidth can run Qwen 3.6 27B dense with 100k context window at 20 tokens/sec in LM studio. Draws 300 watts at the wall on my ITX chassis (idling about 30w).
I’ve been dabbling in building an aviation weather and field condition report application using this, but my next step is to rebuild my VS Code environment into a new machine. I’m kinda enjoying just fucking around with building the hardware too though
I went down the same rabbit hole. I have a 6800xt however but have issues getting it to perform outside of llm chats into using tools like pi.dev
Is it worth getting a v620?
Yeah, I’m using qwen 31b a3b on an amd 9070xt requires a bit of cpu offloading, but still plenty fast. Using it wall llama.cpp. Combine that with some mcp’s such as ddg-search to make it truly useful by actually being able to search online.
I mostly use it for small tedious tasks with well defined inputs and outputs. For example when hyprland recently changed from their own configuration language to lua. At first I started going line by line translating my config to the new lua language until I realized oh wait this is exactly the type of thing that ML is useful for. Going from the well defined hyprland configuration language to their also well defined lua syntax. It banged it out in less than a minute with only a single mistake which I easily fixed. The mistake it made was that it forgot to translate the comments to lua. It did it in less than a minute and worked first try. Where as I had made several typos and gotten a few lines wrong when I was doing it by hand.
Not to say that I couldn’t do it. I would have gotten it done in about half an hour, but less than a minute is a lot faster.
I also used it to transform a bunch of unstructured data into json data, so that I could then use purpose built tools like jq to parse that. If I’m having trouble finding certain information. I’ll ask it to find me some resources to look at.
Basically small well defined tasks and parsing data is what I use it for and it seems to be pretty good at that.
What I don’t like is the way companies try to market it to people. I don’t believe people should be trying to summarize emails or messages from loved ones, writing essays or any other creative tasks for the most part. Translating is okay. I don’t expect a machine to be able to decide things for me or to be some filter between me and others.
Bought b70 with egpu enclosure and usb4 connection wasn’t really planning to actually run anything but now ended up with llama.cpp with openwebui - kids/parents want to/have to use chat, might as well provide local solution than them using industry options. Also started with ollama and Gemma 4 26b a4b - asked it to write script to setup llama.cpp in container.
I’ve thought about it, but I actually could never think of anything I would do with it.
Yes. Openwebui/ollama for LLM, comfyui for stable diffusion. I just dick around with it as a toy.
I was put off by ComfyUI, seems awfully complex. How is your experience?
Any suggestions to start? I have Fooocus installed now
Same. Its somewhat useful on some very small scripting or tasks…but its mostly just to try out a new model or two. Its not really useful for anything big.
I will have to say…even my tiny models are about as good as Chatgpt/Claude/etc… which makes me think about how much people are spending on tokens regularly. I was able to get the same kind of python script started with my local tiny model that was comparable to the newest Claude code offerings.
What local models have you been using? And what hardware are you running them on? I’ve been playing with local LLMs a bit for exactly your use case.
I have zero interest in vibe coding or full agentic workflows. But having a local LLM generate a Bash script to help me automate parts of my home lab infrastructure would be nice.
What are your hardware specs?
Ryzen 7 5800 X3D Radeon RX 9070XT 32GB of DDR4 system memory.
Yep.
Ollama + about 8 different models at the moment, hosted on a mac mini with open webui as a front end.
Predominantly for transcription, translation, an extra round of security checks on code, a more context friendly home assistant interface, and a daily run of context evaluation on property I’m looking for with a lot of specific needs (acreage, min elevation change, soil type, area, etc).
I have to recommend switching to llamacpp. It’s SO much faster than ollama.
On the list but haven’t gotten to it yet, but I know I should. I could probably get a bit more out of that box with it, expand the context windows a bit…
mac mini
How? What is your average response time?
Apple silicon is pretty good at it as long as you’ve got the ram for it. I wouldn’t do less than 16GB.
A few seconds for most of the tasks
What spec Mini do you use?
Just an m2 w/ 16gb I repurposed.
Can’t really do a lot at once, and the context is limited, but it does the trick. I’d buy a few more if I saw them at the right price.
Nice, I’ve got a Mac Studio M1 Max with 32GB of RAM that I use with Ollama and then I host OpenWebUI and OpenCode on my Arch Server. I use the Mac as a primary workstation, so it’s a little rough when I start running a model. I’m sure I could probably do and learn more about Ollama to improve my experience, but for now it works for certain tasks.
I got mine a few years back for some iOS builds, don’t need to do them that often so it became the model host for me
Nope.
I started running LLMs a couple months ago on my own hardware. I have a Framework Desktop that I ordered last year and also recently picked up a refurbished 24GB AMD RX 7900 XTX which I’m doing some performance testing against. The dGPU is much better for dense models, and slightly faster for MoE if I’m willing to run them at a lower quant – but uses more power and has annoying coil whine. The Framework Desktop uses ~100W under load, is quieter, and for the MoE models already runs them fast enough for most of my needs – so most of my LLM use happens on that system still.
For software: I’m using ollama on the Framework currently, but I want to replace it with just using llama.cpp directly eventually. I’ve been using llama-cli for testing the dGPU. I wrote my own chat client to interact with ollama as well as a few other programs for specific tasks.
I’ve been using the LLMs for a mix of research (both personal and professional), entertainment, practical coding tasks (mostly debugging and brainstorming, plus a bit of UI prototyping, automatic generation of sequence diagrams for documentation, and light scripting), as well as automation of tedious tasks.
As an example of the latter, people often send me requests to prepare data sets by email but don’t specify the sources they want precisely so I have to go match the name against the real name in our archives; LLMs are great for mapping the imperfect name – with typos, missing prefixes, incorrect addition of spaces, addition/removal of hyphens, etc. – to the exact name I actually need to pull the data off disk when given a lookup table to compare against.
As far as models go, I’m mostly using various Qwen 3.6 and Gemma4 variants. I have multiple versions of each for different purposes. llmfan46’s uncensored Qwen 3.6 35B-A3B @ Q6_K (from Hugging Face) is my default model currently.
