Skip to content

LocalAI

Open Source

LocalAI is an open-source AI engine that runs language, voice, vision, image, video, and agent workloads on your own hardware through an OpenAI-compatible API, with no GPU required.

50

Overview

LocalAI is an open-source AI engine for running text, voice, vision, image, video, 3D, and agent workloads on hardware you already own, and it is aimed at developers and teams that want a local, OpenAI-compatible endpoint without a mandatory GPU. The project is MIT-licensed and listed at v4.8.0 on its site.

One API in Front of Swappable Engines

LocalAI ships as one binary with an OpenAI-compatible API in front of it. Point an existing client at the endpoint and the calls keep working, except the model now runs on your machine. It also speaks the Anthropic, Ollama, and ElevenLabs APIs, so many tools need only a URL change.

The engine behind that API is swappable. One model can run on llama.cpp while the next loads on vLLM, SGLang, or MLX, and the client sees the same endpoint and the same request. Switching engines is one line in the model's config. A small core pulls each engine in as a separate backend only when a model asks for it, which keeps one install from turning into a 9 GB download.

CPU First, GPU Optional

Every feature ships a CPU path first, and that path is tested in CI on hardware most people already have. It is not a degraded fallback: a GPU makes things faster but is not required. The supported targets include x86_64, ARM64, CUDA, ROCm, SYCL, Metal, and Vulkan.

That matters most for realtime work. Speech in, tool calls in the middle, and speech out over WebRTC can feel like a conversation, and transcription, diarization, and speech synthesis all run without a GPU.

Modalities Beyond Chat

A chat model can only work with what someone types at it. LocalAI's engines change what the model can access: live transcription with speaker labels and timestamps, 527 kinds of sound event, voice and face recognition, plain-language object queries that return coordinates, distance in metres from one photo, and full 3D reconstruction from a handful of images. Speech synthesis supports long-form speech in a cloned voice across dozens of languages at up to 48 kHz, and echo cancellation, noise suppression, and dereverberation help a voice loop survive a real room. One session can hear the room, work out who is talking, read what is on the desk, call a tool, and answer out loud through one API on one machine.

Distributed Operation and APEX Quantization

Adding a second machine is the runtime's problem rather than yours. Routing, VRAM-aware placement, prefix-cache affinity, autoscaling, P2P, NATS, and failover are handled by the runtime, so you add hardware and the cluster works out what to do with it.

Weights decide whether a model runs at all, so the project builds those too. APEX assigns a different precision to every tensor and layer: a 35B mixture-of-experts model goes from 64.6 GB to 12.2 GB, and from 30.4 to 74.4 tokens a second, while perplexity moves from 6.537 to 7.088. The file is an ordinary GGUF, so stock llama.cpp opens it without patches, and 201 APEX models are already in the LocalAI gallery.

The Terminal Agent

local-ai chat opens a terminal agent that runs against your LocalAI server. It is not a separate tool: the agent is compiled into the local-ai binary, so if you have LocalAI you already have it. It works in the shell you already have open, reads your files, runs commands on your machine, delegates to sub-agents, and loads MCP servers, plugins, and skills. Read-only work happens on its own; anything that can change something asks you first. The agent is nib embedded in LocalAI, which is why its plugin, skill, and MCP ecosystem works unchanged.

This is a breaking change from the old plain chat prompt. The agent now executes shell commands behind an approval gate, the old /clear command is gone, and /compact is the nearest equivalent. Read-only tools run without asking, covering reads, searches, and read-only shell commands such as ls and cat. Everything else prompts for approval, and you can approve, deny, or trust the tool for the rest of the session. --yolo, or LOCALAI_CHAT_YOLO=1, approves every tool call without asking, which is the right setting for a sandbox and the wrong one for your laptop.

Scripting and State

--cli makes the agent answer a piped question and exit. Read-only tools still run, so a piped question can inspect a repository or a log file. A tool call that is not read-only is denied because nobody is at the keyboard to approve it, and the session exits 3, deliberately not 1, so a script can tell "I refused to act" apart from "I crashed." Pipe a question in without --cli and the agent refuses, naming --cli.

Config, plugins, and skills live in ~/.config/localai/chat/, or under $XDG_CONFIG_HOME when set, and can be overridden with --config-dir or LOCALAI_CHAT_CONFIG_DIR. The directory is user-scoped rather than server-scoped because the agent is a client and may point at a remote LocalAI. Management commands are passed through, and Claude Code plugins install as they are. In scripts, pass --yes: plugin install without it in a non-interactive context installs the plugin but leaves it disabled, prints that it did, and still exits 0.

Getting Started

LocalAI runs in about a minute: a container on any platform, a DMG on macOS, a binary on Linux, or a Helm chart on Kubernetes. Backends download themselves the first time a model asks for one, so the base install stays small. The install script detects the platform, fetches the latest release binary, and local-ai run pulls the backend and starts an API on localhost:8080.

Core features

  • One binary exposes an OpenAI-compatible API, plus Anthropic, Ollama, and ElevenLabs API compatibility, so existing clients often need only a URL change.
  • Swappable backends let one model run on llama.cpp while another loads on vLLM, SGLang, or MLX, with the engine selected in the model config.
  • Backends download on demand, keeping the base install small instead of a multi-gigabyte bundle.
  • Covers text generation, tool calling, structured output, realtime voice, transcription, diarization, speech synthesis, voice cloning, vision, object detection, depth, 3D, image, video, music, and sound generation.
  • CPU paths ship first and are tested in CI; a GPU speeds things up but is not required.
  • Distributed features include routing, VRAM-aware placement, prefix-cache affinity, autoscaling, and failover.
  • A terminal agent compiled into the local-ai binary reads files, runs commands behind an approval gate, and loads MCP servers, plugins, and skills.
  • APEX quantization produces standard GGUF files that stock llama.cpp opens without patches.

Use cases

  1. 1Running local language models behind an OpenAI-compatible endpoint so existing apps keep working.
  2. 2Building realtime voice loops with speech in, tool calls in the middle, and speech out over WebRTC.
  3. 3Adding vision, detection, depth, or 3D reconstruction to a local pipeline without a GPU.
  4. 4Using the terminal agent to inspect repositories, run read-only commands, and delegate to sub-agents.
  5. 5Scaling from a laptop to multiple machines with routing, VRAM-aware placement, and failover.
  6. 6Generating images, video, music, and sound locally through the same runtime.

Strengths and things to check

Strengths

  • Open source under the MIT license, with a single binary and an OpenAI-compatible API.
  • CPU-first design means most features run without a GPU.
  • Swappable backends let you match the engine to the model and hardware.
  • Broad modality coverage, from text and voice to vision, image, video, and agents.
  • Distributed routing, autoscaling, and failover are handled by the runtime.

Things to check

  • The terminal agent executes shell commands on the machine it runs on, so the approval gate and --yolo setting deserve care.
  • local-ai chat changed from a plain chat prompt to an executing agent; the old /clear command is gone and /compact is the nearest equivalent.
  • In non-interactive scripts, plugin install without --yes installs the plugin but leaves it disabled while still exiting 0.
  • A piped tool call that is not read-only is denied and the session exits 3, which scripts must handle.

Pricing

Subscription pricing is not listed.

Basic information

Website
localai.io
Added
Oct 10, 2026
Platforms
Web
Chinese & English
Not marked
API
Marked as supported
Self-hosting
Not marked
Best for
Developers, self-hosters, and teams that want to run AI models on their own hardware with an OpenAI-compatible API and no mandatory GPU.
Information updated
Oct 16, 2026
Website status
Checking
Corrections · Privacy · Data requests?
Contact us

Features, pricing and availability may change. Always verify details on the official website. Maintained by steven · Last updated Oct 16, 2026 →

Related tools

All categories