How do you run an LLM locally with Docker Model Runner?

Docker Model Runner lets you pull a language model from Docker Hub, an OCI registry or Hugging Face, then run it on your own machine and call it through an OpenAI or Ollama compatible API. Prompts stay local. The API is not authenticated, so limit who can reach it.

What is Docker Model Runner?

Docker's documentation describes it as a way to manage, run and serve AI models using Docker, with integration into Docker Desktop and Docker Engine. Models are pulled the first time you use them and cached locally. They load into memory when a request arrives and unload when idle, which saves resources on a laptop.

It supports three inference engines. llama.cpp is the default and works on all supported platforms, using quantized GGUF models. vLLM targets higher throughput and needs NVIDIA GPUs. Diffusers handles image generation. Platform support differs: Apple Silicon on macOS, specific GPU setups on Windows, and CPU, NVIDIA, AMD and Vulkan backends on Linux with Docker Engine. Check the current requirements page before you plan hardware.

What are the basic steps?

Enable the feature in Docker Desktop settings, or enable it on Docker Engine by following the install steps in Docker's docs. Then use the docker model command family: docker model pull to fetch a model by name, docker model run to chat with it, and docker model list to see what is stored. The docs also show docker model configure for settings such as context size, which usually defaults somewhere between 2,048 and 8,192 tokens depending on the model.

For apps, point any OpenAI-compatible client at the local endpoint. That means existing code can switch between a local model and a hosted one by changing a base address, and Docker's docs list integrations with coding tools and a web chat interface.

Why run a model locally at all?

Three reasons come up. Data stays on your machine, which helps when prompts contain source code or customer text. Cost is predictable. And you can test applications offline or in CI without sending anything out.

Local does not mean safe by default. A model is code and data you downloaded from the internet. Treat it like any other dependency.

What are the security gaps?

Docker's own security notes say the Model Runner API is not authenticated. Any client that can reach it, including other containers on the same Docker network, can pull, load and run models and send inference requests. On Linux the runner and its engines run inside a container, which gives an isolation boundary. On macOS and Windows the engines run in a sandboxed environment instead.

Practical risks follow:

• Another container on a shared network can use your model, fill your disk with pulls or read what you send.
• A model file from an unknown publisher might be tampered with or simply untrustworthy. Check the publisher and the file's provenance before you load it.
• Prompts and outputs may be logged by apps you connect, so check where those logs go.

A constructed example: a developer runs the runner on a shared CI host. A neighbouring job on the same network pulls a large model every build and fills the cache disk, and the logs show prompts from another team's tests. Nothing was hacked. There was just no boundary.

How to check or apply this

1. Confirm your hardware and operating system against the current Docker requirements page.
2. Pull models only from publishers you can name. Record the model name, source and version you approved.
3. Keep the endpoint bound to the local machine. Do not publish it on a network interface.
4. On shared hosts, isolate the runner on its own Docker network and put an authenticating proxy in front if others need access.
5. Keep Docker Desktop or Engine updated, since the runner and its engines ship with it.
6. Decide what data may be sent to a model. Test with synthetic data first.
7. Log who calls the endpoint and how much disk models use.

Quick check

Sources

• Docker Docs, Docker Model Runner

Last reviewed: 18 October 2026.