local-llm-claude-code

Running Claude Code with Local LLMs via Ollama

Running Claude Code with Local Ollama Models on Windows

A practical, real-world guide to running Claude Code against local LLMs served by Ollama on a Windows machine — including how to pick a model that actually fits your GPU, why partial GPU offload makes everything slow, and how to fix it.

TL;DR: If ollama ps shows a split like 46%/54% CPU/GPU, your model doesn't fit in VRAM. Pick a smaller model (or quantization) that fits fully on the GPU. On a 6 GB card, llama3.1:8b (4.9 GB) runs fully on GPU while gemma4:12b (8.1 GB) does not — and the difference in speed is dramatic.

---

Table of Contents

1. Step-by-Step Installation Guide on Windows
2. Essential Ollama Commands
3. Connecting Claude Code to Ollama
4. Calling Ollama's REST API (Postman)
5. Why Is My Local Model Slow? (VRAM, sGPU vs dGPU)
6. Understanding Shared GPU vs Dedicated GPU Memory
7. Choosing a Model That Fits Your GPU
8. What Can a Local LLM Do? (Capabilities)
9. Optimizing Local Ollama with LangGraph and Efficient Models
10. Troubleshooting Notes from the Field

---

Step-by-Step Installation Guide on Windows

1. Download the Installer

• Visit the Official Ollama Download Page.
• Click Download for Windows to fetch the OllamaSetup.exe installer. [1, 2, 3]

2. Run the Installation Wizard

• Double-click the downloaded OllamaSetup.exe file.
• Click Install in the installation prompt.
• Once finished, Ollama will automatically start running in the background. You will see a small llama icon in your Windows taskbar system tray. [1, 2, 3, 4, 5, 6]

3. Run Your First Model

Ollama is controlled via the command line. [1, 2]

• Press the Windows Key, type cmd or PowerShell, and press Enter to open a terminal.
• Run the following command to download and start a model (e.g., Llama 3):

ollama run llama3

• Wait for the model files to download. Once complete, an interactive prompt will open where you can chat with the AI directly. [1, 2, 3, 4, 5]

---

Essential Ollama Commands

Use these commands in your Windows terminal to manage your local AI workspace: [1]

Task	Command
Chat with a model	ollama run <model_name>
List installed models	ollama list
See running models + CPU/GPU split	ollama ps
Delete a model	ollama rm <model_name>
Download a model without starting it	ollama pull <model_name>
Show model details (size, quantization, params)	ollama show <model_name>
Launch a coding tool (e.g., Claude Code) with a model	ollama launch claude --model <model_name>
Exit the chat interface	Type /bye and press Enter

---

Connecting Claude Code to Ollama

Since Ollama v0.15, the ollama launch command wires up coding tools like Claude Code to your local Ollama server with zero manual configuration — no environment variables or config files needed:

# Interactive picker — choose the tool and model from a menu
ollama launch

# Launch Claude Code with a specific local model
ollama launch claude --model llama3.1:8b

Ollama sets the necessary environment variables and starts Claude Code pointing at http://localhost:11434 automatically.

Claude Code's status area shows the local model name (gemma4:12b) — confirming inference is running on the local Ollama server, not Anthropic's cloud.

Important notes:

• ollama launch remembers your last selection. If you pull a new model, it won't switch automatically — pass --model <name> explicitly or use the interactive picker.
• Fully exit any running Claude Code session before relaunching with a new model. A running session keeps the model it started with.
• When running via Ollama, tokens shown in Claude Code's status bar are consumed by your local model, not by Anthropic's API. The "API Usage Billing" label just reflects the connection mode — there is no cloud billing when pointed at localhost.
• Coding tools work best with a long context window. Ollama recommends at least 64,000 tokens of context for coding agents — but note that longer context uses more VRAM (see below).

---

Calling Ollama's REST API (Postman)

Ollama isn't just a CLI — it exposes a full REST API on http://localhost:11434, which means any app, script, or backend service can use your local model like a private, free, no-API-key GenAI service.

The simplest endpoint is /api/generate:

POST http://localhost:11434/api/generate
Content-Type: application/json

{
    "model": "llama3.1:8b",
    "prompt": "what is dGPU?"
}

Testing the local model from Postman: the request body names the model and prompt; the response streams back token by token.

Understanding the streaming response

By default Ollama streams the answer as newline-delimited JSON — one small object per token chunk, each with "done": false until the final one:

{"model":"llama3.1:8b","created_at":"...","response":"A","done":false}
{"model":"llama3.1:8b","created_at":"...","response":" question","done":false}
...
{"model":"llama3.1:8b","created_at":"...","response":"","done":true, "total_duration": ...}

This is ideal for chat UIs (you can render text as it arrives), but awkward in Postman. To get one single JSON response instead, add "stream": false:

{
    "model": "llama3.1:8b",
    "prompt": "what is dGPU?",
    "stream": false
}

Other useful endpoints

Endpoint	Purpose
POST /api/generate	One-shot completion for a prompt
POST /api/chat	Multi-turn chat — send a messages array with roles (system/user/assistant)
GET /api/tags	List installed models (same as ollama list)
POST /api/embeddings	Generate embeddings for RAG / vector search

Ready-made Postman collection

A minimal Postman collection with the /api/generate request is included in this repo: GenAIService.postman_collection.json — import it into Postman (File → Import) and you're calling your local model in seconds.

Tip: Because this API is a stable local HTTP service, it's the same integration point used by LangGraph/LangChain (ChatOllama), custom backends, and tools like Claude Code — everything in this guide ultimately talks to localhost:11434.

---

Why Is My Local Model Slow?

The single most common cause: the model doesn't fit in your GPU's dedicated VRAM, so Ollama splits it between GPU and CPU.

Check it yourself:

ollama ps

Example output from a real machine (GTX 1660 Ti, 6 GB VRAM):

NAME          ID            SIZE     PROCESSOR       CONTEXT   UNTIL
gemma4:12b    4eb23ef187e2  8.1 GB   46%/54% CPU/GPU  4096     4 minutes from now

The smoking gun: ollama ps reports the 8.1 GB model split 46%/54% CPU/GPU, while Task Manager shows the GTX 1660 Ti's 6 GB of dedicated memory nearly full — the model simply doesn't fit.

That 46%/54% CPU/GPU split means roughly half the model's layers run on the CPU. Every generated token has to shuttle data between system RAM and VRAM across the PCIe bus — which is often slower than either pure-CPU or pure-GPU inference. The goal is always 100% GPU.

Other contributing factors when using Claude Code with local models:

• No prompt caching. Anthropic's cloud API caches the large system prompt between turns; Ollama reprocesses it from scratch on every message. On local hardware, re-prefilling thousands of tokens each turn adds real latency.
• Context window size. The KV cache for a large context window (e.g., 64K+) can consume gigabytes of VRAM — memory that would otherwise hold model weights. Smaller num_ctx = more room for the model on GPU.
• Other apps competing for VRAM (browsers, games, a second model still loaded in Ollama).

---

Understanding Shared GPU vs Dedicated GPU Memory

Windows Task Manager shows two GPU memory pools, and knowing the difference explains a lot of local-LLM performance behavior:

Task Manager (idle state, no model loaded): Dedicated GPU memory is the 6 GB of real VRAM on the card; Shared GPU memory is 7.9 GB borrowed from system RAM — usable, but far slower. Note also the two GPUs listed: GPU 0 (Intel integrated) and GPU 1 (NVIDIA dedicated).

Dedicated GPU Memory (dGPU memory / VRAM)

• The physical memory chips on the graphics card itself (e.g., 6 GB on a GTX 1660 Ti).
• Extremely fast, directly connected to the GPU cores.
• This is the only memory that matters for fast inference. A model runs at full speed only if its weights + KV cache fit here.

Shared GPU Memory (sGPU memory)

• A portion of your system RAM (typically up to 50% of it) that Windows lets the GPU borrow when dedicated VRAM runs out.
• Accessed over the PCIe bus — many times slower than on-card VRAM.
• When a model spills into shared memory, performance falls off a cliff. It looks like the GPU is "handling it" in Task Manager, but tokens/sec collapses.

Integrated GPU vs Dedicated GPU

On many machines you'll also see two GPUs listed (e.g., GPU 0: Intel UHD Graphics and GPU 1: NVIDIA GeForce GTX 1660 Ti):

• The integrated GPU (iGPU) shares system RAM and is not used by Ollama for CUDA inference.
• The dedicated GPU (dGPU) is the NVIDIA/AMD card with its own VRAM — this is what Ollama uses.
• Watch GPU 1 (the dedicated card) in Task Manager's Performance tab while a model is generating to see real utilization and memory usage.

How this affects performance — the practical rule

Model size (from `ollama list`) + KV cache  <  Dedicated VRAM   →  100% GPU, fast
Model size + KV cache                        >  Dedicated VRAM   →  CPU/GPU split or shared-memory spill, slow

Forcing more layers onto the GPU (e.g., OLLAMA_NUM_GPU) cannot fix a model that is physically larger than VRAM — it just spills into slow shared memory. The real fix is a smaller model or a smaller quantization.

---

Choosing a Model That Fits Your GPU

Real-world example: on a 6 GB GTX 1660 Ti —

Model	Download size	Fits in 6 GB VRAM?	Result
gemma4:12b	~7.6–8.1 GB	❌ No	46%/54% CPU/GPU split, slow
gemma4:e2b	~7.2 GB	❌ No	Still splits
llama3.1:8b (Q4_K_M)	4.9 GB	✅ Yes	100% GPU, fast
llama3.2:3b	~2 GB	✅ Yes (lots of headroom)	Very fast, lower quality

Rules of thumb:

• Leave ~1–1.5 GB of VRAM headroom beyond the model file size for the context/KV cache and display output.
• Default Ollama pulls are usually Q4_K_M quantization — a good balance of size and quality.
• A smaller model running fully on GPU almost always feels faster (and often works better as a coding agent) than a bigger model that's split with the CPU.
• Check available tags and exact sizes on the model's page at ollama.com/library/<model> — not every quantization exists for every model.

# Switch from a too-big model to one that fits
ollama pull llama3.1:8b
ollama rm gemma4:12b          # free ~8 GB of disk
ollama launch claude --model llama3.1:8b

The end result: Claude Code running llama3.1:8b — a model that fits entirely in the 6 GB of VRAM — responding quickly.

---

What Can a Local LLM Do?

A self-description generated by the local model itself:

I can be used in a variety of ways, but my primary function is to assist with tasks such as answering questions, generating text, and translating languages. I have been initialized with a large dataset of text from the internet, which I use to generate responses to user input.

I have access to a wide range of APIs and tools that allow me to perform various tasks, such as:

• Answering questions on a wide range of topics
• Generating text based on user input or prompts
• Translating languages
• Summarizing long pieces of text
• Providing definitions for words and phrases
• Offering suggestions for products and services

Some key components that I work with to answer queries include:

• Natural Language Processing (NLP) libraries, which allow me to understand and interpret user input
• Machine learning algorithms, which enable me to generate responses based on patterns in the data I have been trained on
• APIs and databases, which provide me with access to external information and resources

As for my ability to reason, think, and code — as a large language model, I don't have personal thoughts or feelings like humans do. However, I can process and analyze vast amounts of information, identify patterns, and make connections between seemingly unrelated concepts.

Regarding training data and skills, you can provide me with additional data and examples to improve my performance on specific tasks or domains. Some ways to train me include:

• Providing more high-quality training data
• Offering feedback on my responses, such as corrections or suggestions for improvement
• Engaging in conversations with me, which helps me learn from your input and adjust my responses accordingly
• Sharing relevant knowledge and expertise, either through direct input or by pointing out errors or areas where I can improve

Keep in mind that my performance is only as good as the data and resources available to me. If you provide high-quality training data and engage with me regularly, I can become more accurate and effective over time.

To further enhance my abilities, we can work together on specific tasks and projects. Some examples of how we can collaborate include:

• Improving my understanding of a particular domain or topic
• Developing new skills or capabilities through task-based training
• Enhancing my performance on specific tasks or metrics

Overall, I am designed to assist and augment human abilities, rather than replace them. By working together, we can achieve more accurate and effective results, and develop new possibilities for language understanding and generation.

Note: This is the model describing itself, and small local models tend to overstate a few things. A base Ollama model does not persistently learn from your conversations, and it only has access to external APIs/tools if you build that integration around it (which is exactly what the next section covers).

---

Optimizing Local Ollama with LangGraph and Efficient Models

Running a raw model in a chat loop is just the beginning. LangGraph — the agent-orchestration framework from the LangChain team — lets you build structured, stateful workflows around your local Ollama models, and it pairs especially well with small, efficient models on limited hardware.

Why LangGraph + local models is a great match

• Small models do better with structure. A 7–8B model may struggle with one giant open-ended prompt, but performs well when a LangGraph workflow breaks the task into small, well-defined steps (classify → retrieve → draft → verify).
• Graphs give you control over token spend. Each node sends only the context it needs, instead of resending an entire conversation every turn — a big win when your local GPU has no prompt caching.
• Route by difficulty. LangGraph makes it easy to route simple requests to a tiny fast model (llama3.2:3b) and only escalate hard steps to your larger model — keeping average latency low.
• State lives outside the model. LangGraph persists conversation/workflow state itself, so the model doesn't need a huge context window — meaning you can run a smaller num_ctx and keep everything in VRAM.

Minimal setup

pip install langgraph langchain-ollama

from langchain_ollama import ChatOllama
from langgraph.graph import StateGraph, MessagesState, START, END

# Point at your local Ollama server
llm = ChatOllama(
    model="llama3.1:8b",
    temperature=0,
    num_ctx=8192,        # keep the KV cache small enough to stay in VRAM
)

def chatbot(state: MessagesState):
    return {"messages": [llm.invoke(state["messages"])]}

graph = StateGraph(MessagesState)
graph.add_node("chatbot", chatbot)
graph.add_edge(START, "chatbot")
graph.add_edge("chatbot", END)
app = graph.compile()

result = app.invoke({"messages": [("user", "Summarize what Ollama does in 2 lines.")]})
print(result["messages"][-1].content)

Practical optimization tips for local LangGraph agents

1. Pin num_ctx per node. Give summarization nodes a bigger window and routing/classification nodes a tiny one (2048 is plenty). Smaller KV cache → more VRAM for weights → stays 100% GPU.
2. Use structured outputs. Ask the model for JSON with a strict schema at each node. Small models are far more reliable emitting a short JSON object than free-form reasoning.
3. Keep OLLAMA_KEEP_ALIVE long (e.g., 24h or -1) so the model isn't unloaded and reloaded between graph steps.
4. One model loaded at a time on small GPUs. Set OLLAMA_MAX_LOADED_MODELS=1 — two models fighting over 6 GB of VRAM guarantees a CPU spill.
5. Add a verifier node. A cheap second pass ("does this answer actually address the question? yes/no") catches many small-model mistakes for minimal extra tokens.
6. Prefer tool calls over long context. Instead of stuffing documents into the prompt, give the graph retrieval nodes/tools that fetch only the relevant chunk.

With this pattern, a modest GPU running an efficient 4–8B model can drive surprisingly capable agent workflows — the intelligence comes from the graph design as much as from the model.

---

Troubleshooting Notes from the Field

Real issues hit during this setup, and their fixes:

'OLLAMA_NUM_GPU' is not recognized as an internal or external command The VAR=value command syntax is Unix/bash-only. On Windows, set environment variables with setx OLLAMA_NUM_GPU 999 (takes effect in new terminals) or via Edit environment variables for your account. Also remember: Ollama runs as a background service — fully quit it from the system tray and relaunch for new env vars to apply. (And per the sections above — this can't rescue a model that's bigger than your VRAM anyway.)

Error: pull model manifest: file does not exist The quantization tag you asked for doesn't exist for that model. Not every model publishes q4_0 / q3_K_M variants. Check the real tag list at ollama.com/library/<model>/tags before pulling.

ollama launch still opens the old model after pulling a new one ollama launch remembers your last selection. Pass the model explicitly (ollama launch claude --model llama3.1:8b) or run bare ollama launch for the interactive picker — and make sure the previous Claude Code session is fully closed first.

Tokens/billing confusion when Claude Code shows a local model name If the status bar shows your Ollama model name (e.g., llama3.1:8b), inference is happening on localhost — the token counts are your local model's, and no Anthropic API billing is occurring.

---

Tested on: Windows 11 · NVIDIA GeForce GTX 1660 Ti (6 GB) · Ollama v0.15+ · Claude Code v2.1.x