> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cynative.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Local Inference with Ollama, vLLM, or SGLang in Cynative

> Run Cynative against a local or self-hosted model via Ollama, vLLM, or SGLang. No API key required — point Cynative at your endpoint and pick a model.

Running Cynative against a locally hosted model means your prompts, tool outputs, and security findings never leave your environment. No data is sent to an external API, no third-party retention policies apply, and your cloud-credential context stays entirely within your control.

<Tip>
  Running Cynative on an instance in the same cloud it audits, pointed at cloud-managed inference (e.g. a vLLM deployment on the same VPC), means nothing leaves your environment — not the infrastructure data Cynative reads, and not the model calls it makes.
</Tip>

Cynative supports three local/self-hosted providers through the Bifrost SDK: **Ollama** (the simplest path for local laptops and dev machines), **vLLM** (high-throughput GPU inference), and **SGLang** (SGLang server). All three require no API key by default.

***

## Ollama

[Ollama](https://ollama.ai) is the easiest way to run open-weight models locally. It manages model downloads, quantization, and the inference server in a single binary.

### Prerequisites

1. Install Ollama: follow the instructions at [ollama.ai](https://ollama.ai).
2. Pull a model: `ollama pull llama3.3` (or any model from the Ollama library).
3. Ensure the Ollama server is running: `ollama serve` (it starts automatically on most installs).

### Quick start

```bash theme={null}
# Ollama running at http://localhost:11434 (the default)
export CYNATIVE_LLM_PROVIDER=ollama
export CYNATIVE_LLM_MODEL=llama3.3
export CYNATIVE_LLM_OLLAMA_URL=http://localhost:11434
cynative -p "list all S3 buckets in this account"
```

No `~/.cynative/config.yaml` file is required.

<Warning>
  Ollama takes its endpoint from `CYNATIVE_LLM_OLLAMA_URL` (which maps to `ollama_key_config.url`), **not** from the generic `CYNATIVE_LLM_NETWORK_CONFIG_BASE_URL`. Setting `CYNATIVE_LLM_NETWORK_CONFIG_BASE_URL` for the Ollama provider fails at startup with "provider has no keys configured: ollama".
</Warning>

### YAML configuration

```yaml theme={null}
llm:
  provider: ollama
  model: llama3.3
  keys:
    - value: ""                  # no API key
      models: ["*"]
      weight: 1.0
      ollama_key_config:
        url: http://localhost:11434
```

### Environment variable reference

| Environment variable | YAML equivalent | Description |
| - | - | - |
| `CYNATIVE_LLM_PROVIDER` | `llm.provider` | Must be `ollama`. |
| `CYNATIVE_LLM_MODEL` | `llm.model` | Model name as known to Ollama (e.g. `llama3.3`, `mistral`, `qwen2.5`). |
| `CYNATIVE_LLM_OLLAMA_URL` | `llm.keys[].ollama_key_config.url` | Base URL of the Ollama server. Default: `http://localhost:11434`. |
| `CYNATIVE_LLM_NETWORK_CONFIG_DEFAULT_REQUEST_TIMEOUT_IN_SECONDS` | `llm.network_config.default_request_timeout_in_seconds` | Per-request timeout in seconds. Cynative default: `300`. Local models can be slow — increase this for large models. |
| `CYNATIVE_LLM_NETWORK_CONFIG_MAX_RETRIES` | `llm.network_config.max_retries` | Retry attempts. Cynative default: `3`. |

***

## vLLM

[vLLM](https://docs.vllm.ai) is a high-throughput LLM inference server designed for GPU-accelerated deployments. It exposes an OpenAI-compatible API. Use it when you need to serve larger models, handle multiple concurrent requests, or run inference on a dedicated GPU host.

### Quick start

```bash theme={null}
# vLLM server running at http://localhost:8000 (the default port)
export CYNATIVE_LLM_PROVIDER=vllm
export CYNATIVE_LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct
export CYNATIVE_LLM_VLLM_URL=http://localhost:8000
cynative -p "..."
```

No `~/.cynative/config.yaml` file is required. `CYNATIVE_LLM_VLLM_URL` maps directly to `vllm_key_config.url`. `CYNATIVE_LLM_VLLM_MODEL_NAME` is also accepted for per-key model routing.

### YAML configuration

```yaml theme={null}
llm:
  provider: vllm
  model: meta-llama/Llama-3.3-70B-Instruct
  keys:
    - value: ""                       # set to your --api-key token if you launched vLLM with one
      models: ["*"]
      weight: 1.0
      vllm_key_config:
        url: http://localhost:8000
```

### Authentication

vLLM runs unauthenticated by default. If you launched it with `--api-key <token>`, supply the token in one of two ways:

* **YAML path (`keys[]`):** Put the token in the key's `value:` field as shown above.
* **Env-only path (no YAML):** Set `CYNATIVE_LLM_API_KEY` alongside `CYNATIVE_LLM_VLLM_URL`. This path uses no `keys[]`, so there is no conflict.

<Warning>
  Do not combine a top-level `api_key` / `CYNATIVE_LLM_API_KEY` with a `keys[]` block — Cynative rejects that combination at startup.
</Warning>

### Environment variable reference

| Environment variable | YAML equivalent | Description |
| - | - | - |
| `CYNATIVE_LLM_PROVIDER` | `llm.provider` | Must be `vllm`. |
| `CYNATIVE_LLM_MODEL` | `llm.model` | Model name as served by vLLM (e.g. `meta-llama/Llama-3.3-70B-Instruct`). |
| `CYNATIVE_LLM_VLLM_URL` | `llm.keys[].vllm_key_config.url` | Base URL of the vLLM server (e.g. `http://localhost:8000`). |
| `CYNATIVE_LLM_VLLM_MODEL_NAME` | `llm.keys[].vllm_key_config.model_name` | Per-key model name for routing (optional). |
| `CYNATIVE_LLM_API_KEY` | `llm.api_key` | API key if vLLM was launched with `--api-key` (env-only path, no `keys[]`). |
| `CYNATIVE_LLM_NETWORK_CONFIG_DEFAULT_REQUEST_TIMEOUT_IN_SECONDS` | `llm.network_config.default_request_timeout_in_seconds` | Per-request timeout in seconds. Cynative default: `300`. |
| `CYNATIVE_LLM_NETWORK_CONFIG_MAX_RETRIES` | `llm.network_config.max_retries` | Retry attempts. Cynative default: `3`. |

***

## SGLang

[SGLang](https://docs.sglang.ai) is a fast LLM inference and serving framework. Its Cynative provider id is `sgl`.

### Quick start

```bash theme={null}
export CYNATIVE_LLM_PROVIDER=sgl
export CYNATIVE_LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct
export CYNATIVE_LLM_SGL_URL=http://localhost:30000
cynative -p "..."
```

No `~/.cynative/config.yaml` file is required. `CYNATIVE_LLM_SGL_URL` maps directly to `sgl_key_config.url`.

### YAML configuration

```yaml theme={null}
llm:
  provider: sgl
  model: meta-llama/Llama-3.3-70B-Instruct
  keys:
    - value: ""                  # no API key by default
      models: ["*"]
      weight: 1.0
      sgl_key_config:
        url: http://localhost:30000
```

### Authentication

SGLang servers are typically unauthenticated. If you've put SGLang behind an auth proxy, supply the proxy token in one of two ways:

* **YAML path (`keys[]`):** Put the token in the key's `value:` field as shown above.
* **Env-only path:** Set `CYNATIVE_LLM_API_KEY` alongside `CYNATIVE_LLM_SGL_URL`, with no `keys[]` block.

### Environment variable reference

| Environment variable | YAML equivalent | Description |
| - | - | - |
| `CYNATIVE_LLM_PROVIDER` | `llm.provider` | Must be `sgl`. |
| `CYNATIVE_LLM_MODEL` | `llm.model` | Model name as served by SGLang. |
| `CYNATIVE_LLM_SGL_URL` | `llm.keys[].sgl_key_config.url` | Base URL of the SGLang server (e.g. `http://localhost:30000`). |
| `CYNATIVE_LLM_API_KEY` | `llm.api_key` | Auth proxy token if needed (env-only path, no `keys[]`). |
| `CYNATIVE_LLM_NETWORK_CONFIG_DEFAULT_REQUEST_TIMEOUT_IN_SECONDS` | `llm.network_config.default_request_timeout_in_seconds` | Per-request timeout in seconds. Cynative default: `300`. |

***

## Other self-hosted and OpenAI-compatible endpoints

Any inference server that speaks the OpenAI Chat Completions API can be used with the `openai` provider and a custom `base_url`. This covers LiteLLM proxies, custom gateways, and most other OpenAI-compatible servers that don't have a dedicated Bifrost provider:

```yaml theme={null}
llm:
  provider: openai
  model: my-custom-model
  api_key: env.MY_KEY               # omit or leave blank if unauthenticated
  network_config:
    base_url: https://my-inference-server.internal
```

Or via environment variables:

```bash theme={null}
export CYNATIVE_LLM_PROVIDER=openai
export CYNATIVE_LLM_MODEL=my-custom-model
export CYNATIVE_LLM_NETWORK_CONFIG_BASE_URL=https://my-inference-server.internal
```

See the [OpenAI provider guide](/providers/openai#openai-compatible-endpoints-and-proxies) for the base URL caveat (don't include a trailing `/v1`).

## Links

* [Ollama documentation](https://github.com/ollama/ollama/blob/main/docs/api.md)
* [vLLM documentation](https://docs.vllm.ai/)
* [SGLang documentation](https://docs.sglang.ai/)
* [Bifrost Ollama provider source](https://github.com/maximhq/bifrost/tree/main/core/providers/ollama)
* [Bifrost vLLM provider source](https://github.com/maximhq/bifrost/tree/main/core/providers/vllm)
* [Bifrost SGL provider source](https://github.com/maximhq/bifrost/tree/main/core/providers/sgl)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.