No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that?

If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?

I really like these show and tell style posts. I’m always curious how people have their setups and what tools they use. Also the blog has a nice theme and is easy to read.

I wanna get a desktop Mac for local ai so that I don’t turn my laptop into a delta 15k rpm fan when I run things.

I guess I’ll get in line for one hah.

I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc.

These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.

The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.

I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.

My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency.

I've since acquired two DGX Sparks, and it feels so much snappier.

I tried the 1 bit model of Qwen3.6 27B on my M1 pro (16G) and got 13 tok/s with only 5G of ram usage.

https://x.com/mkagenius/status/2093730391429685732

(xcancel seems to have received a cease and desist)

Apple did great work convincing people their unified memory was good at AI. Even AI says Apple is the best of all time at marketing.

Meanwhile the stock market has Nvidia at the top... Until everyone gets cuda.

> <a href="https://omlx.app">oMLX</a>

Is that supposed to be hallucination? The human or other kind. Feels like a made up URL. It's .ai, isn't it?

I have an m1 Mac 64gb and look forward to trying this out

Not many people share setup with actual setup handholding so that was very G of you

M4 Pro has been a solid performer for iterating on smaller local models. Much more convenient than spinning up cloud instances for dev.

That M4 Pro is probably a beast for quantised models. My M2 Pro handles 34B just barely; what speeds are you seeing?

yes, share performance, numbers if you can, also i wonder if you figured out a way to do a 2way audio with local models, or even explored that. I have a very similar setup but not too happy with the token speed, will try omlx though !!!

Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model

I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support system's unanswered tickets, browser X and some social media for me to filter the valued ones(though I have to say even with GPT 5.6 sol, the quality is low for the timeline X sent to me)

Btw, Tailscale is quite cool and did a good job, I was using it to serve the local LLM and connct the openclaw on a Linux Machine to it.

[flagged]

[flagged]

[flagged]

[dead]