LiteLLM

​

Overview

nOps connects to your LiteLLM proxy with a read API key to see models, keys, and spend across the gateway. You can also add an optional write API key for future management actions (keys, teams, budgets). Keys are validated before save and are never shown again after you connect. nOps shows only the last four characters of each key. You can connect more than one gateway. Each gateway is a separate connection with its own base URL and keys.

​

Prerequisites

• A LiteLLM proxy you administer, with the admin UI or the management API enabled
• The gateway reachable from the internet over public HTTPS (recommended) or HTTP (see Network requirements)
• nOps organization administrator access

​

Create the read API key

The read key needs to read model info, keys, and spend for the whole gateway. Use a key that belongs to a proxy_admin_viewer user, so it sees every team but cannot change anything, and limit it to the read routes below.

1. Create a viewer user (skip this if you already have one):
   curl -X POST https://litellm.example.com/user/new \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H "Content-Type: application/json" \
     -d '{"user_email": "nops@example.com", "user_role": "proxy_admin_viewer"}'
   
2. Create a key for that user that can call only these read routes:
   curl -X POST https://litellm.example.com/key/generate \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "user_id": "<user_id from step 1>",
       "key_alias": "nops-read",
       "allowed_routes": [
         "/model/info",
         "/key/info",
         "/key/list",
         "/user/daily/activity",
         "/spend/logs",
         "/spend/tags",
         "/global/spend/report",
         "/team/list",
         "/team/info",
         "/user/info"
       ]
     }'
   
3. Copy the key value from the response. LiteLLM shows it only once.

​

Create the write API key (optional)

Add a write key only if you want nOps to manage LiteLLM keys, teams, or budgets later. Create it for a proxy_admin user, or use "key_type": "management". Keep it separate from the read key. When you save a write key, nOps checks which management routes it can call with requests that change nothing: each one targets an id that does not exist (nops-probe-…), so LiteLLM answers “not found” instead of making a change. If the key cannot call any management route, nOps does not save it.

​

Connect in nOps

1. Go to Settings → Integrations and click the LiteLLM card.
2. Enter the Base URL of the proxy (for example https://litellm.example.com). Use the final URL; nOps does not follow redirects. If the URL starts with http://, confirm I understand API keys will be sent unencrypted to continue.
3. Paste the Read API key and, optionally, the Write API key.
4. Optionally add a label (for example Production gateway).
5. Click Connect.

nOps checks that the gateway is healthy, that each key is accepted, and which routes each key can call. Nothing is saved if a check fails.

​

Savings opportunities

Open a gateway in Settings → Integrations → LiteLLM and click Savings. nOps reads the last 14 days of spend through your read key and looks for prompt-caching opportunities: models, such as Claude, that only cache when requests carry cache markers and where little of your prompt traffic is served from the provider’s cache. Each recommendation shows:

• A monthly savings range, not a single number. It is based on your observed cache reads and the model’s own cache prices, and assumes only a modest share of the remaining prompt tokens become cache reads.
• The evidence behind it: prompt tokens, cache-read share, and spend over the period.
• How to apply: the cache_control_injection_points setting to add to that model.

Models that already cache most of their prompts, or that nOps cannot price, are left out. If nothing qualifies, nOps says so and shows your overall cache-read share.

​

Network requirements

• The base URL should use https:// with a public hostname. IP addresses, localhost, and .local / .internal hostnames are rejected.
• http:// is allowed only after an admin explicitly confirms it, because API keys and responses then travel unencrypted. nOps shows an Insecure (http) badge and a warning on the connection. Switch to https when possible. Moving a connection from https to http needs the same confirmation and the keys re-entered.
• http does not change the network rules below: private networks are still unreachable over http.
• The hostname must resolve only to public IP addresses. nOps checks DNS before every request and refuses private, loopback, link-local, and carrier-grade NAT ranges.
• Redirects are not followed. If the proxy redirects (for example from / to /ui, or from http:// to https://), enter the URL that answers directly.
• If the gateway only accepts traffic from allowlisted IP addresses, contact nOps support before connecting.

​

Manage your connection

• Re-validate re-runs the checks with the stored keys. A failure shows Action required.
• Rotate a key by entering a new value; leave a key field unchanged to keep the stored key.
• Remove the write key to go back to read-only access.
• Change the base URL on a different host only by re-entering the read key (and the write key, or removing it). nOps never sends stored keys to a new host.
• Disconnect deletes the stored keys.

​

Troubleshooting

• Could not reach the gateway: confirm the base URL is publicly reachable and that /health/liveliness returns 200.
• Host is not allowed: the hostname resolves to a private or reserved address. Expose the proxy on a public address.
• LiteLLM redirected: enter the final URL the proxy serves, without redirects. A redirect from http to https means you should enter the https:// URL.
• Confirm insecure http: the URL starts with http://. Use the https:// URL, or confirm that keys will be sent unencrypted.
• Read key rejected: confirm the key is active and was copied in full.
• Read key cannot read spend or keys: the key is missing routes, most often /user/daily/activity (spend) or /key/list (keys). Add them to the key’s allowed_routes (POST /key/update) or recreate it as described in Create the read API key, for a proxy_admin_viewer user.
• Write key cannot call any management routes: the write key’s user is not an admin, or the key’s routes exclude management routes.

​

Security

Treat both keys like passwords. Rotate them in LiteLLM if they may have been exposed, then update them in nOps.