Old news by now, but you might not be aware that llama-server can do multi-model for a while now,

Meaning that you (and by that I mean your AI agent that has read the llama.cpp code) can write an ini file pointing to your models with parameters optimized for the specific model on your specific hardware. (Optimized by you through testing. Not that AI)

Then, any api client can just select a model and the system does the right thing.

It's great software. It just works.

__

You just need to ignore the cargo culting commandline options on social media. But you should be listening to the devs.

Have you already enabled ngram-mod (or rather just spec-default)? It is practically free.

Not sure why it's on the front page now, but I highly recommend using llama.cpp for running AI model locally vs using other inference framework, unless you have a very specific requirement.

ggerganov and the team have done a stellar job maintaining the quality while still being fast to implement new models/improvements.

llama.cpp works pretty well for me on the Framework 13 laptop, but the current era of "move fast, break things, rarely fix" (sorry, that's how it feels), bites here quite a bit.

Two examples:

- https://github.com/ggml-org/llama.cpp/pull/25863 Someone's few lines change broke the native (ROCm) support for the AMD GPU inside Framework (and other integrated systems), and any rollback or proper fix is pending for almost a month. Fortunately there's workaround (switching to Vulkan rather than ROCm devices), but both the way the bug was introduced and the way it is not fixed just doesn't give much confidencen

- LM Studio is using llama.cpp internally for GGUF, they ship their own build with their closed source system as "runtimes". Their ROCm runtime does not enable the the AMD GPU inside the Framework, even thought the llama.cpp version would support it. So their runtime keeps telling me that there's no supported AMD GPU -- again, the solution is to use the GPU with the Vulkan devices. Not fixed since Jan at least https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1...

I guess overall it's the worst runtime I've seen so far, except for all the other runtimes out there... I'm a fan, though in some cases I don't have enough knowledge, or I don't have access to fix things, and that feels like a bummer...

I was a bit suspicious of the url but it is also listed on llama.cpp github

https://github.com/ggml-org/llama.cpp

Anything that suggests curl into bash just plain sketches me out. (edit: I know, this isn't totally rational, it just seems weird to me. We download and trust a lot of software and run code from a bunch of package repositories as a regular activity...).

Git clone llama.cpp and build it, it's not hard.

https://github.com/ggml-org/llama.cpp/blob/master/docs/build...

literally just a few steps for the basics:

git clone https://github.com/ggml-org/llama.cpp

cmake -B build

cmake --build build --config Release

Yesterday I installed llama.cpp to test it with local AI Data Analyst that I'm building. I was also testing other open LLM providers: Ollama, Jan, vLLM, LM Studio. I had older NVIDIA card (RTX 3070) and llama.cpp instalation was smooth, contrary to vLLM which required me to reinstall CUDA drivers because by default it installed the latest one. I'm curious if there is a speed difference between the same open LLM model served with different runners.

I tried

    curl -LsSf https://llama.app/install.sh | sh

and then

    llama serve -hf unsloth/Qwen3-4B-GGUF:Q4_0

Then I get:

    W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
    Terminated

And the web interface says

    Server unavailable

Maybe it gets killed by the OS because it uses too much RAM?

When I try

    llama serve -hf unsloth/Llama-3.2-1B-Instruct-GGUF:Q4_K_M

It seems to work. Nice.

This MacOS app used to be called LlamaBarn. Really excellent to see the fast progress being made.

Official repo, also has documentation how to configure server parameters:

https://github.com/ggml-org/Llama-macOS

Small tip, install llama.cpp with brew before llama.app, which will pick up the existing llama.cpp. That way it's easier to stay up to date with llama.cpp, since llama.app is on a slower release cadence.

Also, models installed with the hugging face CLI (hf) are picked up by llama.app automatically. The CLI will keep the model cache updated, e.g. when models get updated.

Llama.cpp became part of Huggingface recently.

Is llama.cpp (and thus llama.app) really that much better than Ollama? I've Only ever played with Ollama, so geniously curious to hear other's real-world experiences.

Llama.cpp team has failed to make their tech easy to install and use for years.

Why can’t they figure it out???

I think I can probably run Gemma 3 12B on my macbook M3 pro with 18GB. The question is, should I do it? This small model is probably not capable of doing a lot or advanced coding or reasoning. What else could it be used for, since it can run locally and privately?

There’s now a `llama serve` command? I had to do a double take in case I was reading the `ollama` website.

I tried to run in on my Arc A770, but all of the binary releases I could find were compiled without OpenVINO support enabled. I tried compiling it myself, but after two days of the compiler running it failed.

llama.cpp is like the ffmepg of AI, and one of the reasons I so greatly dislike ollama is that the latter completely obfuscates that they're a rebrand of the former. Georgi Gerganov and team did all the hard work; ollama is langchain-like VC-bait with a HF download wrapper.

Any success at transpiling it to C? Using the cfront transpiler improved with coding AI? :)

> No telemetry

Must be tough not to be able to monitor your own models!

(The odds that that tagline was AI-generated seem high.)

and? whats the point of this? Doesn't everyone already know about llama.cpp?

[flagged]

[flagged]

[flagged]

I'm confused, is this from Meta? There's no attribution anywhere. Surely releasing an AI tool called llama breaks their trademark if not

This site seems scam for not noting origins of llama.cpp and fails to quickly and clearly communicate it NOT being affiliated with GGML org.

It seems that llama.app is a direct competitor to ollama.com

I can understand the desire for the llama.cpp project to want to own the end user relationship, it is true that previous to this they were a tool provider and not really owning the end user experience.