A Practical Guide to Running Local Coding AI Models

Introduction

Running a coding model locally can provide privacy, lower recurring costs, offline access, and full control over the inference environment. However, downloading a model and starting a server is only the first step.

Reliable local inference depends on several connected decisions:

• Available VRAM and system memory
• Model architecture
• Quantization format
• CUDA version and driver
• llama.cpp version
• GPU offload settings
• Context length
• Number of parallel agents
• MTP or speculative decoding settings
• Editor and MCP integration

This guide presents a practical path for anyone who wants to run local coding models for software development, code review, security analysis, code generation, MCP tools, and agentic workflows.

The examples are based on a Windows system with an NVIDIA RTX 3070 Laptop GPU and 8GB of VRAM. The same principles can be applied to other NVIDIA GPUs and local inference systems.

1. Define the Goal Before Choosing a Model

The correct model depends on how it will be used.

A model used for autocomplete has different requirements from a model used by an agent that reads an entire repository, calls MCP tools, generates tests, and performs several reasoning steps.

Common local coding use cases include:

• Code completion
• Refactoring
• Debugging
• Test generation
• Security analysis
• Database and API design
• Repository exploration
• MCP tool usage
• Multi-agent planning
• Remote access from a phone or another computer

For short coding tasks, a 7B to 9B model can provide a very good balance between quality and speed. For complex agentic work, context capacity and reasoning quality become more important than raw generation speed.

2. Understand VRAM, Model Size, and Quantization

A common mistake is to look only at the number of active parameters. Mixture-of-experts models may activate only a small number of parameters per token, but the complete model weights still need to be stored somewhere.

For example, Ornith 1.5 35B-A3B activates approximately 3B parameters per token, but its complete 4-bit version still requires around 20GB of memory. This makes it suitable for higher-memory Apple Silicon systems or larger GPUs, but not for full GPU offload on an 8GB RTX 3070.

For an 8GB GPU, a practical range is usually:

• 7B to 9B models in Q4 or Q5 quantization
• Q5_K_M when quality is the priority and the model fits
• Q4_K_M when additional context memory is required
• Smaller models when several concurrent agents are needed

Q5 generally preserves more quality than Q4, but consumes more memory. For coding, Q5_K_M is often a good compromise when it fits comfortably.

3. Select the Correct Model File

A model repository may contain several files with similar names but different capabilities.

Before downloading a model, check:

• Whether it is an official model or a community conversion
• Whether it contains an MTP or next-token prediction head
• Whether it uses GGUF, MLX, GPTQ, or another format
• Whether it includes an importance matrix
• Whether it uses Q4, Q5, Q6, or Q8 quantization
• Whether the chat template is compatible with the runtime
• Whether the license allows your intended use

For llama.cpp on Windows, GGUF is usually the most practical format.

A stock GGUF model may not contain an MTP head. An MTP-specific GGUF is required if you want to use:

--spec-type draft-mtp

The main MTP model tested in this project was:

Ornith-1.5-9B-MTP-Q5_K_M.gguf

4. Install and Verify llama.cpp

Download a Windows CUDA build of llama.cpp and place it in a dedicated directory. Keeping different versions in separate directories makes A/B testing easier.

A practical layout can look like this:

your/path/llama_cuda_13_3/
your/path/llama_v020_b10566_cuda13_3/
your/path/models/

The first major build used in the project was an older nightly build identified as b10545. It was later compared with the stable v0.2.0 release, corresponding to build b10566.

Verify the build before testing:

.\llama-server.exe --version

Also verify that the CUDA DLL files required by the executable are available in the executable directory or in the system path.

The most important verification is the startup log. Look for evidence that the model layers were actually offloaded to the GPU. A command may contain GPU options while still leaving part of the model on the CPU.

5. Start With a Controlled Baseline

Do not begin with a 128K context, three agents, and every experimental option enabled. Start with a smaller and repeatable configuration.

A baseline server for an MTP-enabled 9B coding model can look like this:

.\llama-server.exe `
  -m 'your/path/models/Ornith-1.5-9B-MTP-Q5_K_M.gguf' `
  -c 8192 `
  -np 1 `
  --jinja `
  --flash-attn on `
  -ngl 99 `
  --fit off `
  --repack `
  --kv-offload `
  --spec-type draft-mtp `
  --spec-draft-n-max 2 `
  --threads 8 `
  --threads-batch 8 `
  --temp 0.6 `
  --top-k 20 `
  --top-p 0.95 `
  --min-p 0 `
  --alias ornith15 `
  --host 127.0.0.1 `
  --port 8080

This configuration uses one slot, a moderate context, full GPU offload, Flash Attention, and MTP.

6. Understand the Main llama.cpp Options

Full GPU offload

-ngl 99

This requests that all possible model layers be placed on the GPU. Always confirm the result in the startup log.

Flash Attention

--flash-attn on

Flash Attention can reduce memory usage and improve attention performance, especially with larger context windows.

Fit mode

--fit off

Fit mode can automatically adjust settings to fit available memory. This is convenient, but it can silently reduce GPU offload or change context behavior. For controlled benchmarking, --fit off makes the configuration easier to understand.

Repacking and KV offload

--repack
--kv-offload

Weight repacking should normally remain enabled. KV offload can improve performance, but increases VRAM pressure because the key-value cache is stored on the accelerator when possible.

CPU threads

--threads 8
--threads-batch 8

These settings affect CPU-side work, prompt processing, sampling, and scheduling. They are not normally the main factor in GPU decode speed, but explicit values make comparisons more consistent.

Options that are often misunderstood

--load-mode none primarily affects model loading and does not normally increase generation tokens per second.

--perf enables timing information. It is useful for measurement but is not itself a speed optimization.

--n-cpu-moe is intended for mixture-of-experts models and is not useful for a dense 9B model.

7. Enable MTP Carefully

MTP, or multi-token prediction, is an inference acceleration method. An additional prediction head proposes future tokens, and the main model verifies them in parallel.

Use:

--spec-type draft-mtp
--spec-draft-n-max 2

The draft depth should be benchmarked on the exact model, quantization, GPU, and workload. In this project, nmax=2 was faster than 3 and 4 on the RTX 3070.

A larger draft depth creates more speculative candidates, but the target model must verify them. If too many candidates are rejected, the verification cost can exceed the benefit.

MTP does not retrain the model or add new coding knowledge. It changes the inference path and can reduce the amount of sequential decoding work.

8. Benchmark the Real API

A low-level benchmark is useful, but it does not always represent the real experience inside a coding editor. Editors and agents add chat templates, system prompts, tool definitions, MCP payloads, sampling settings, and API overhead.

A useful benchmark should keep the following constant:

• Exact GGUF file
• Exact llama.cpp build
• Exact CUDA build
• Same prompt
• Same temperature
• Same seed
• Same context size
• Same number of slots
• At least three repetitions

Measure:

• Prompt processing speed
• Generation speed
• Total request time
• Generated token count
• MTP acceptance rate
• GPU layer offload
• VRAM usage
• Response correctness and completeness

The benchmark prompt used in this project was:

Write a concise Python function that safely joins a user path beneath a root and rejects traversal and symlink escapes. Include one important caveat.

The local tests used an RTX 3070 Laptop GPU, CUDA 13.3, context size 8192, one slot, a fixed seed, and three repetitions per configuration.

9. Local Speed Results

The following values were measured through the llama.cpp OpenAI-compatible API.

Model	b10545, nmax 2	v0.2.0/b10566, nmax 2	b10545, nmax 3	v0.2.0/b10566, nmax 3
Ornith 1.0 Heretic	59.13 tok/s	57.54 tok/s	53.79 tok/s	52.42 tok/s
Ornith 1.5	50.92 tok/s	53.23 tok/s	48.01 tok/s	48.07 tok/s

The results demonstrate that the fastest model and the best model are not always the same.

Ornith 1.0 Heretic was faster overall. Ornith 1.5 was more attractive for difficult coding tasks because it identified more hidden security and concurrency problems during qualitative testing.

The llama.cpp version change did not produce a universal speed improvement. The model, MTP acceptance, context size, and slot count had a larger practical effect.

10. Test Quality, Not Only Speed

A local coding model should be tested against tasks that expose hidden mistakes.

Useful security tasks include:

• Path traversal
• Symlink attacks
• TOCTOU races
• Command injection
• Authentication bypasses
• Tenant isolation
• Unsafe deserialization

Useful database tasks include:

• Idempotency keys
• Concurrent updates
• Transaction boundaries
• Retry behavior
• Unique constraints
• Crash recovery
• External calls inside transactions

Also check whether the model:

• Produces runnable code
• Includes imports
• Handles errors
• Writes meaningful tests
• Avoids overclaiming guarantees
• Identifies assumptions
• Explains tradeoffs

In our tests, Ornith 1.5 was better at recognizing hidden problems such as TOCTOU races, tenant-scoped idempotency, and atomic database updates. However, its output was sometimes truncated or contained invalid framework-specific code.

Ornith 1.0 was faster, but occasionally proposed solutions that looked reasonable while missing critical details.

The correct workflow is to treat the model as a development assistant, not as an automatic production engineer. Generated code must still be tested, reviewed, and executed.

11. Use a Coding Editor or Agent

The models were tested through Zcode because it was the coding environment available for this project and was suitable for connecting to a local OpenAI-compatible endpoint, MCP tools, and agent workflows.

The same approach can be used with any coding editor or coding agent that supports custom local models or OpenAI-compatible endpoints. The important requirement is that the editor allows you to configure:

• Local API base URL
• Model alias
• Context size
• Temperature
• Tool calling
• Streaming responses
• Authentication settings

Running a model inside an editor can expose problems that are not visible in a simple command-line test. During some Zcode sessions, the workflow experienced bottlenecks and stopped responding. The most common causes were likely large MCP payloads, multiple agents competing for the same GPU, long context processing, VRAM pressure, request timeouts, or editor-side limits.

This is an important difference between a clean benchmark and a real coding environment. A model may produce a good tokens-per-second number in a short isolated request but still become unstable when the editor sends a large repository context and several tool definitions.

12. Configure Long Context Correctly

Long context is one of the most common sources of confusion in local inference.

If the server is started with:

-c 128000 -np 2

llama.cpp can allocate two slots of approximately 64,000 tokens each:

128000 total context / 2 slots = 64000 tokens per slot

This does not provide 128K tokens to each agent.

During a Zcode test, the server reported:

n_ctx_slot = 64000

Zcode then sent a request containing 67,462 tokens. The request was rejected before generation began because it exceeded the 64K limit.

For one large agent request, use:

-c 131072 -np 1

For two agents with approximately 80K available per slot, the theoretical configuration is:

-c 160000 -np 2

For two agents with 128K tokens each, the total context would need to be approximately:

-c 256000 -np 2

That configuration is not realistic for an 8GB GPU. Long-context workloads require substantial KV cache memory, and multiple slots multiply the memory requirement.

13. MCP Payloads and Editor Stability

MCP can add tens of thousands of tokens to a request through tool definitions, repository context, file contents, and previous agent messages.

A practical context budget should include:

• System prompt
• Tool definitions
• MCP server descriptions
• Repository files
• Conversation history
• Model reasoning
• Expected output
• Safety margin

If MCP contributes 55K tokens, the final request can easily exceed 64K after adding system instructions and output space.

For a single large Zcode agent, the recommended starting point is:

-np 1 -c 131072

For multiple agents, reduce unnecessary MCP content before increasing the global context size. Sending the same large repository context to every agent can create unnecessary memory pressure.

14. Remote Access From a Phone

To access the local model from another device on the same network, bind the server to all network interfaces:

--host 0.0.0.0

However, this exposes the API beyond localhost. A safer configuration should use:

--api-key 'replace-with-a-long-random-key'

Also configure the operating system firewall so only trusted devices can connect. Do not expose an unauthenticated local model server directly to the public internet.

15. Recommended Profiles

Single Agent With Maximum Context

-c 131072
-np 1
--spec-type draft-mtp
--spec-draft-n-max 2
--flash-attn on
-ngl 99
--fit off

This is the best starting profile for Zcode when one active request needs a large MCP payload.

Maximum Speed for Short Coding Tasks

-c 8192
-np 1
--spec-type draft-mtp
--spec-draft-n-max 2
--flash-attn on
-ngl 99

Use Ornith 1.0 Heretic when raw generation speed is the priority.

Better Quality for Complex Coding

Use Ornith 1.5 MTP Q5_K_M with the same runtime settings. It was slower in short decode benchmarks but performed better when reasoning about hidden security and concurrency problems.

Two Concurrent Agents

-c 160000
-np 2

Use this only after confirming that the model remains fully GPU offloaded and that VRAM usage is stable. Expect lower per-request speed than the one-slot configuration.

16. Common Problems and Solutions

The request exceeds the available context

Check n_ctx_slot in the startup log. Divide the total context by the number of slots and include room for output tokens.

MTP does not improve speed

Check the draft acceptance rate. If acceptance is low, try nmax=2, verify full GPU offload, and confirm that the GGUF actually contains a compatible MTP head.

Generation speed is much lower than expected

Check:

• Whether all layers are on the GPU
• Whether the server is using multiple slots
• Whether the context size is excessively large
• Whether CUDA is using the intended build
• Whether the GPU is thermal throttling
• Whether a quantized KV cache triggered a slow attention path

The editor becomes slow or stops

Check:

• MCP payload size
• Number of active agents
• Context length
• VRAM usage
• Request timeout settings
• Editor memory usage
• Whether the server is still responding

A short command-line benchmark cannot guarantee stable behavior inside a full coding editor.

The server is reachable but unsafe

Do not leave 0.0.0.0 exposed without an API key and firewall restrictions.

Conclusion and Practical Limitations

The most reliable way to run local coding models is to follow a controlled process:

1. Choose a model that fits the available memory.
2. Select a suitable quantization.
3. Install and verify the llama.cpp CUDA build.
4. Start with one slot and a moderate context.
5. Confirm full GPU offload.
6. Enable Flash Attention.
7. Enable MTP only with a compatible model.
8. Benchmark the real API using repeatable prompts.
9. Test coding quality with adversarial tasks.
10. Increase context and parallelism gradually.
11. Calculate per-slot context instead of relying only on the global value.
12. Secure the server before exposing it to other devices.

On the RTX 3070 Laptop GPU used in this project, a 9B Q5 MTP coding model reached approximately 50 to 60 tokens per second in controlled short-context tests. This is fast enough for experimentation, learning, and personal development workflows.

However, these experiments should be treated as experimental results only. They should not be considered proof that the setup is ready for a production environment.

If you want to use a local coding model inside your preferred editor for serious daily programming, you will generally need considerably stronger hardware to obtain a reliable combination of speed, context length, memory capacity, and stability. A larger GPU with more VRAM, more system memory, stronger cooling, a faster CPU, and better storage can make a significant difference.

The RTX 3070 setup was useful as a testing platform, but it sometimes experienced bottlenecks when used through Zcode with large MCP payloads, multiple agents, or long contexts. In some sessions, the editor became slow or stopped responding. These limitations are expected when pushing a laptop GPU close to its memory and compute limits.

Zcode was used in this project because it was the available environment and provided a suitable way to test local models, MCP tools, and agent workflows. You can use any editor or coding agent you prefer, as long as it supports custom local models or an OpenAI-compatible endpoint.

For a dependable production-oriented local coding environment, prioritize stability over the highest single benchmark number. The system should maintain full GPU offload, preserve enough context for real projects, support the required number of agents, and remain responsive under sustained workloads.

The central lesson is that local AI performance is a systems problem. The best result comes from matching the model, quantization, runtime, GPU, context, editor, and workload rather than searching for one universal speed setting.

References

• llama.cpp v0.2.0 release
• llama.cpp server documentation
• llama.cpp parallel context discussion
• Ornith 1.5 9B official model card
• Ornith 1.5 9B MTP GGUF
• Ornith 1.5 35B-A3B official model card
• Ornith 1.5 35B-A3B MLX 4-bit