LiteLLM

​

Overview

nOps connects to your LiteLLM proxy with a read API key to see models, keys, and spend across the gateway. You can also add an optional write API key for future management actions (keys, teams, budgets). Keys are validated before save and are never shown again after you connect. nOps shows only the last four characters of each key. You can connect more than one gateway. Each gateway is a separate connection with its own base URL and keys.

​

Prerequisites

• A LiteLLM proxy you administer, with the admin UI or the management API enabled
• The gateway reachable from the internet over public HTTPS (recommended) or HTTP (see Network requirements)
• nOps organization administrator access

​

Create the read API key

The read key needs to read model info, keys, and spend for the whole gateway. Use a key that belongs to a proxy_admin_viewer user, so it sees every team but cannot change anything, and limit it to the read routes below.

1. Create a viewer user (skip this if you already have one):
   curl -X POST https://litellm.example.com/user/new \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H "Content-Type: application/json" \
     -d '{"user_email": "nops@example.com", "user_role": "proxy_admin_viewer"}'
   
2. Create a key for that user that can call only these read routes:
   curl -X POST https://litellm.example.com/key/generate \
     -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "user_id": "<user_id from step 1>",
       "key_alias": "nops-read",
       "allowed_routes": [
         "/model/info",
         "/key/info",
         "/key/list",
         "/user/daily/activity",
         "/spend/logs",
         "/spend/tags",
         "/global/spend/report",
         "/team/list",
         "/team/info",
         "/user/info"
       ]
     }'
   
3. Copy the key value from the response. LiteLLM shows it only once.

​

Create the write API key (optional)

Add a write key only if you want nOps to manage LiteLLM keys, teams, or budgets later. Create it for a proxy_admin user, or use "key_type": "management". Keep it separate from the read key. To let nOps apply a savings change to a model (see Savings opportunities), the write key must also be allowed to call /model/*. The standard management_routes preset does not include PATCH /model/{id}/update, which is the route nOps uses to edit a model without disturbing its other settings. For example, when you create the key, send "allowed_routes": ["management_routes", "/model/*"]. Without it, nOps can still recommend changes but cannot apply them. When you save a write key, nOps checks which management routes it can call with requests that change nothing: each one targets an id that does not exist (nops-probe-…), so LiteLLM answers “not found” instead of making a change. If the key cannot call any management route, nOps does not save it.

​

Connect in nOps

1. Go to Settings → Integrations and click the LiteLLM card.
2. Enter the Base URL of the proxy (for example https://litellm.example.com). Use the final URL; nOps does not follow redirects. If the URL starts with http://, confirm I understand API keys will be sent unencrypted to continue.
3. Paste the Read API key and, optionally, the Write API key.
4. Optionally add a label (for example Production gateway).
5. Click Connect.

nOps checks that the gateway is healthy, that each key is accepted, and which routes each key can call. Nothing is saved if a check fails.

​

Savings opportunities

Open a gateway in Settings → Integrations → LiteLLM and click Savings. nOps reads the last 14 days of spend through your read key and looks for prompt-caching opportunities: models, such as Claude, that only cache when requests carry cache markers and where little of your prompt traffic is served from the provider’s cache. Each recommendation shows:

• A monthly savings range, not a single number. It is based on your observed cache reads and the model’s own cache prices, and assumes only a modest share of the remaining prompt tokens become cache reads.
• The evidence behind it: prompt tokens, cache-read share, and spend over the period.
• How to apply: the cache_control_injection_points setting to add to that model.

The same recommendations also appear on the Recommendations page, alongside the rest of your savings, under LiteLLM Prompt Caching and LiteLLM Model Routing. They are refreshed once a day from the usage nOps has already synced, so a change you make on your gateway shows up there the next day. Models that already cache most of their prompts, or that nOps cannot price, are left out. If nothing qualifies, nOps says so and shows your overall cache-read share.

​

Model routing

For an API key that sends mostly short prompts to a premium model, nOps estimates what you would save by letting a router send the simple requests to a cheaper model of the same family. The estimate is a wide range, because request size stands in for difficulty and nOps never reads your prompts. nOps does not apply routing for you yet. If you set it up yourself, test it on a sample first: on some gateway versions a request that goes through a LiteLLM complexity router ignores the caller’s max_tokens, so a short answer can come back full length and cost far more than the saving. Send a request with a very small max_tokens through the router and confirm the response honors it.

​

Prompt compression

For keys whose prompts are long (several thousand tokens on average), nOps shows how much input spend per month compression could act on. This is the spend at stake, not a savings estimate: how much a prompt shrinks depends on its content, which nOps does not read. nOps does not apply compression for you. Trial a compression guardrail on one key, then compare cost per request and answer quality before widening it.

​

Apply everything recommended

When two or more changes can be applied from nOps, Review and apply all lists each change and how far it reaches. Each is applied one at a time through the same checks as a single click, so one refusal does not stop the others, and every change can be rolled back on its own.

​

Network requirements

• The base URL should use https:// with a public hostname. IP addresses, localhost, and .local / .internal hostnames are rejected.
• http:// is allowed only after an admin explicitly confirms it, because API keys and responses then travel unencrypted. nOps shows an Insecure (http) badge and a warning on the connection. Switch to https when possible. Moving a connection from https to http needs the same confirmation and the keys re-entered.
• http does not change the network rules below: private networks are still unreachable over http.
• The hostname must resolve only to public IP addresses. nOps checks DNS before every request and refuses private, loopback, link-local, and carrier-grade NAT ranges.
• Redirects are not followed. If the proxy redirects (for example from / to /ui, or from http:// to https://), enter the URL that answers directly.
• If the gateway only accepts traffic from allowlisted IP addresses, contact nOps support before connecting.

​

Manage your connection

• Re-validate re-runs the checks with the stored keys. A failure shows Action required.
• Rotate a key by entering a new value; leave a key field unchanged to keep the stored key.
• Remove the write key to go back to read-only access.
• Change the base URL on a different host only by re-entering the read key (and the write key, or removing it). nOps never sends stored keys to a new host.
• Disconnect deletes the stored keys.

​

Troubleshooting

• Could not reach the gateway: confirm the base URL is publicly reachable and that /health/liveliness returns 200.
• Host is not allowed: the hostname resolves to a private or reserved address. Expose the proxy on a public address.
• LiteLLM redirected: enter the final URL the proxy serves, without redirects. A redirect from http to https means you should enter the https:// URL.
• Confirm insecure http: the URL starts with http://. Use the https:// URL, or confirm that keys will be sent unencrypted.
• Read key rejected: confirm the key is active and was copied in full.
• Read key cannot read spend or keys: the key is missing routes, most often /user/daily/activity (spend) or /key/list (keys). Add them to the key’s allowed_routes (POST /key/update) or recreate it as described in Create the read API key, for a proxy_admin_viewer user.
• Write key cannot call any management routes: the write key’s user is not an admin, or the key’s routes exclude management routes.

​

Security

Treat both keys like passwords. Rotate them in LiteLLM if they may have been exposed, then update them in nOps.