MCmidnight-coderlocal coding agent

Models on your infrastructure

Local LLM inference for coding

Understand the path from a coding-agent request to tokens generated by a model running on a machine you control.

Local LLM inference means executing a trained language model on local or self-hosted hardware. A coding-agent runtime sends prompts to that model provider, receives generated tokens, and combines them with workspace tools.

The local inference pipeline

  1. The agent builds a prompt from instructions, conversation state, and relevant code.
  2. The runtime sends that prompt to the configured model endpoint.
  3. The provider tokenizes the prompt and runs the model on CPU, GPU, or both.
  4. The model generates tokens, which the provider streams or returns to the agent.
  5. The agent may invoke a tool and repeat the cycle with the new result.

Hardware, quantization, and speed

Model weights and the context cache consume RAM or VRAM. Parameter count, quantization level, context length, and hardware bandwidth all affect whether a model fits and how quickly it generates tokens.

Model sizeInfluences capability and memory demand
QuantizationTrades numerical precision for lower memory use
Context lengthExpands the cache required for a request
GPU offloadCan accelerate inference when VRAM is available

Privacy boundary

When the provider runs locally and the agent is configured to use it, model prompts stay between the agent runtime and that provider host. This is narrower than sending prompts to a hosted inference service.

Local inference does not govern every process an agent can launch. Package managers, tests, web tools, SDKs, or shell commands may still access the network. Use sandbox and approval controls to define the complete boundary.

Configure local inference in Midnight Coder

/provider_conf 127.0.0.1:11434
/model
/smartcontext true

Midnight Coder probes model-list routes, stores the detected provider and model, and can align Ollama's num_ctx value with the selected model. Follow the Ollama setup guide for the complete flow.

Related definitions: AI coding and LLM glossary.

Connect local inference

Bring a working provider endpoint and select its exposed model from Midnight Coder.

Explore Midnight CoderModel providers