How to Run Local LLMs With Ollama: Complete Setup Guide

Ollama makes it remarkably easy to run large language models on your own hardware — no cloud subscription, no data sent to third parties, complete control over your AI stack. This setup guide covers everything from installation to running your first model, optimized for 2025.

What Is Ollama?

Ollama is an open-source tool that packages LLM models with everything needed to run them locally: an optimized inference engine, API server, and model management CLI. It supports macOS, Linux, and Windows, with a growing library including Llama 3, Mistral, Gemma, and Phi-3. If you’re deciding between Llama 3 and Mistral, Ollama supports both and makes switching trivial.

System Requirements

• Minimum (7B models): 8GB RAM, modern CPU. GPU optional but recommended.
• Recommended (7-13B models): 16GB RAM, Apple Silicon Mac or NVIDIA GPU with 8GB+ VRAM
• High-end (70B models): 64GB+ RAM or NVIDIA GPU with 40GB+ VRAM

Installation

On macOS/Linux, run: curl -fsSL https://ollama.com/install.sh | sh. On Windows, download the installer from ollama.com. The Ollama service starts automatically and runs in the background.

Running Your First Model

Use ollama run llama3 or ollama run mistral to download and start a model. The first run downloads the model (3–20GB depending on size). Subsequent runs start instantly from the local cache. You can find all available models on the Hugging Face model hub or at ollama.com/library.

Using the API

Ollama runs a local REST API on port 11434 by default — compatible with the OpenAI API format, making it a drop-in replacement for many applications. Point your existing code at http://localhost:11434 and change the model name.

Integrating With Development Tools

Ollama integrates with Open WebUI (a ChatGPT-like browser interface), VS Code extensions, and Python libraries. For fine-tuning models, import custom GGUF-format models with ollama create. Among the top open source LLMs available in 2025, most can be downloaded and run via Ollama within minutes. Many AI coding tools now support Ollama as a backend for privacy-focused deployments.