Local Inference with Ollama, vLLM, or SGLang in Cynative

Running Cynative against a locally hosted model means your prompts, tool outputs, and security findings never leave your environment. No data is sent to an external API, no third-party retention policies apply, and your cloud-credential context stays entirely within your control.

Running Cynative on an instance in the same cloud it audits, pointed at cloud-managed inference (e.g. a vLLM deployment on the same VPC), means nothing leaves your environment — not the infrastructure data Cynative reads, and not the model calls it makes.

Cynative supports three local/self-hosted providers through the Bifrost SDK: Ollama (the simplest path for local laptops and dev machines), vLLM (high-throughput GPU inference), and SGLang (SGLang server). All three require no API key by default.

Ollama

Ollama is the easiest way to run open-weight models locally. It manages model downloads, quantization, and the inference server in a single binary.

Prerequisites

Install Ollama: follow the instructions at ollama.ai.
Pull a model: ollama pull llama3.3 (or any model from the Ollama library).
Ensure the Ollama server is running: ollama serve (it starts automatically on most installs).

Quick start

# Ollama running at http://localhost:11434 (the default)
export CYNATIVE_LLM_PROVIDER=ollama
export CYNATIVE_LLM_MODEL=llama3.3
export CYNATIVE_LLM_OLLAMA_URL=http://localhost:11434
cynative -p "list all S3 buckets in this account"

No ~/.cynative/config.yaml file is required.

Ollama takes its endpoint from CYNATIVE_LLM_OLLAMA_URL (which maps to ollama_key_config.url), not from the generic CYNATIVE_LLM_NETWORK_CONFIG_BASE_URL. Setting CYNATIVE_LLM_NETWORK_CONFIG_BASE_URL for the Ollama provider fails at startup with “provider has no keys configured: ollama”.

YAML configuration

llm:
  provider: ollama
  model: llama3.3
  keys:
    - value: ""                  # no API key
      models: ["*"]
      weight: 1.0
      ollama_key_config:
        url: http://localhost:11434

Environment variable reference

Environment variable	YAML equivalent	Description
`CYNATIVE_LLM_PROVIDER`	`llm.provider`	Must be `ollama`.
`CYNATIVE_LLM_MODEL`	`llm.model`	Model name as known to Ollama (e.g. `llama3.3`, `mistral`, `qwen2.5`).
`CYNATIVE_LLM_OLLAMA_URL`	`llm.keys[].ollama_key_config.url`	Base URL of the Ollama server. Default: `http://localhost:11434`.
`CYNATIVE_LLM_NETWORK_CONFIG_DEFAULT_REQUEST_TIMEOUT_IN_SECONDS`	`llm.network_config.default_request_timeout_in_seconds`	Per-request timeout in seconds. Cynative default: `300`. Local models can be slow — increase this for large models.
`CYNATIVE_LLM_NETWORK_CONFIG_MAX_RETRIES`	`llm.network_config.max_retries`	Retry attempts. Cynative default: `3`.

vLLM

vLLM is a high-throughput LLM inference server designed for GPU-accelerated deployments. It exposes an OpenAI-compatible API. Use it when you need to serve larger models, handle multiple concurrent requests, or run inference on a dedicated GPU host.

Quick start

# vLLM server running at http://localhost:8000 (the default port)
export CYNATIVE_LLM_PROVIDER=vllm
export CYNATIVE_LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct
export CYNATIVE_LLM_VLLM_URL=http://localhost:8000
cynative -p "..."

No ~/.cynative/config.yaml file is required. CYNATIVE_LLM_VLLM_URL maps directly to vllm_key_config.url. CYNATIVE_LLM_VLLM_MODEL_NAME is also accepted for per-key model routing.

YAML configuration

llm:
  provider: vllm
  model: meta-llama/Llama-3.3-70B-Instruct
  keys:
    - value: ""                       # set to your --api-key token if you launched vLLM with one
      models: ["*"]
      weight: 1.0
      vllm_key_config:
        url: http://localhost:8000

Authentication

vLLM runs unauthenticated by default. If you launched it with --api-key <token>, supply the token in one of two ways:

YAML path (keys[]): Put the token in the key’s value: field as shown above.
Env-only path (no YAML): Set CYNATIVE_LLM_API_KEY alongside CYNATIVE_LLM_VLLM_URL. This path uses no keys[], so there is no conflict.

Do not combine a top-level api_key / CYNATIVE_LLM_API_KEY with a keys[] block — Cynative rejects that combination at startup.

Environment variable reference

Environment variable	YAML equivalent	Description
`CYNATIVE_LLM_PROVIDER`	`llm.provider`	Must be `vllm`.
`CYNATIVE_LLM_MODEL`	`llm.model`	Model name as served by vLLM (e.g. `meta-llama/Llama-3.3-70B-Instruct`).
`CYNATIVE_LLM_VLLM_URL`	`llm.keys[].vllm_key_config.url`	Base URL of the vLLM server (e.g. `http://localhost:8000`).
`CYNATIVE_LLM_VLLM_MODEL_NAME`	`llm.keys[].vllm_key_config.model_name`	Per-key model name for routing (optional).
`CYNATIVE_LLM_API_KEY`	`llm.api_key`	API key if vLLM was launched with `--api-key` (env-only path, no `keys[]`).
`CYNATIVE_LLM_NETWORK_CONFIG_DEFAULT_REQUEST_TIMEOUT_IN_SECONDS`	`llm.network_config.default_request_timeout_in_seconds`	Per-request timeout in seconds. Cynative default: `300`.
`CYNATIVE_LLM_NETWORK_CONFIG_MAX_RETRIES`	`llm.network_config.max_retries`	Retry attempts. Cynative default: `3`.

SGLang

SGLang is a fast LLM inference and serving framework. Its Cynative provider id is sgl.

Quick start

export CYNATIVE_LLM_PROVIDER=sgl
export CYNATIVE_LLM_MODEL=meta-llama/Llama-3.3-70B-Instruct
export CYNATIVE_LLM_SGL_URL=http://localhost:30000
cynative -p "..."

No ~/.cynative/config.yaml file is required. CYNATIVE_LLM_SGL_URL maps directly to sgl_key_config.url.

YAML configuration

llm:
  provider: sgl
  model: meta-llama/Llama-3.3-70B-Instruct
  keys:
    - value: ""                  # no API key by default
      models: ["*"]
      weight: 1.0
      sgl_key_config:
        url: http://localhost:30000

Authentication

SGLang servers are typically unauthenticated. If you’ve put SGLang behind an auth proxy, supply the proxy token in one of two ways:

YAML path (keys[]): Put the token in the key’s value: field as shown above.
Env-only path: Set CYNATIVE_LLM_API_KEY alongside CYNATIVE_LLM_SGL_URL, with no keys[] block.

Environment variable reference

Environment variable	YAML equivalent	Description
`CYNATIVE_LLM_PROVIDER`	`llm.provider`	Must be `sgl`.
`CYNATIVE_LLM_MODEL`	`llm.model`	Model name as served by SGLang.
`CYNATIVE_LLM_SGL_URL`	`llm.keys[].sgl_key_config.url`	Base URL of the SGLang server (e.g. `http://localhost:30000`).
`CYNATIVE_LLM_API_KEY`	`llm.api_key`	Auth proxy token if needed (env-only path, no `keys[]`).
`CYNATIVE_LLM_NETWORK_CONFIG_DEFAULT_REQUEST_TIMEOUT_IN_SECONDS`	`llm.network_config.default_request_timeout_in_seconds`	Per-request timeout in seconds. Cynative default: `300`.

Other self-hosted and OpenAI-compatible endpoints

Any inference server that speaks the OpenAI Chat Completions API can be used with the openai provider and a custom base_url. This covers LiteLLM proxies, custom gateways, and most other OpenAI-compatible servers that don’t have a dedicated Bifrost provider:

llm:
  provider: openai
  model: my-custom-model
  api_key: env.MY_KEY               # omit or leave blank if unauthenticated
  network_config:
    base_url: https://my-inference-server.internal

Or via environment variables:

export CYNATIVE_LLM_PROVIDER=openai
export CYNATIVE_LLM_MODEL=my-custom-model
export CYNATIVE_LLM_NETWORK_CONFIG_BASE_URL=https://my-inference-server.internal

See the OpenAI provider guide for the base URL caveat (don’t include a trailing /v1).

Get Started

Using Cynative

Agents

Connectors

LLM Providers

Configuration

Project

Local Inference with Ollama, vLLM, or SGLang in Cynative

Ollama

Prerequisites

Quick start

YAML configuration

Environment variable reference

vLLM

Quick start

YAML configuration

Authentication

Environment variable reference

SGLang

Quick start

YAML configuration

Authentication

Environment variable reference

Other self-hosted and OpenAI-compatible endpoints

Links

​Ollama

​Prerequisites

​Quick start

​YAML configuration

​Environment variable reference

​vLLM

​Quick start

​YAML configuration

​Authentication

​Environment variable reference

​SGLang

​Quick start

​YAML configuration

​Authentication

​Environment variable reference

​Other self-hosted and OpenAI-compatible endpoints

​Links

Ollama

Prerequisites

Quick start

YAML configuration

Environment variable reference

vLLM

Quick start

YAML configuration

Authentication

Environment variable reference

SGLang

Quick start

YAML configuration

Authentication

Environment variable reference

Other self-hosted and OpenAI-compatible endpoints

Links