Skip to main content

​
Overview

nOps reads Databricks billing data from system.billing.usage and system.billing.list_prices after you export those tables to S3. Databricks is an export-only Settings integration: you choose an S3 destination in nOps, deploy a prefix-scoped read-only role, then import and schedule a generated notebook in your Databricks workspace. nOps does not create or run Databricks jobs, and does not collect a workspace URL, SQL warehouse ID, or machine credential. You can connect one AWS Databricks export per organization. Azure Databricks export is not available yet.
This flow is separate from AWS cloud onboarding and does not create a Cost and Usage Report (CUR).

​
Prerequisites

  • nOps organization administrator or owner access
  • Permission to deploy CloudFormation (including IAM roles) in the AWS account that holds the export bucket
  • Administrative access to a Databricks workspace so you can import a notebook, grant S3 write (for example a Unity Catalog external location), and schedule a daily job
  • Either permission for CloudFormation to create a new S3 bucket, or an existing bucket in the same AWS account (or separately grant that account access)
Only admins and owners can connect, view instructions, or disconnect. Members see Admin Only on connectable cards.

​
Connect in nOps

​
Where to open Databricks

  • To connect for the first time — go to Settings → Integrations and click the Databricks card.
  • To manage an existing export — open the same card or use Settings → Account Status → Connected apps and click the Databricks chip.

​
Connect steps

  1. Click the Databricks card. The AWS tab is the supported path.
  2. Choose New S3 bucket (CloudFormation creates the bucket and nOps read role) or Existing S3 bucket (CloudFormation creates a read role scoped to your prefix).
  3. Enter the S3 bucket name and a dedicated prefix (default nops/databricks). Do not reuse a shared prefix or a CUR prefix. The notebook writes usage and list-prices Parquet under that prefix and replaces only its own year/month/table/client path.
  4. Download .py notebook or Copy notebook. nOps fills in the S3 destination, nOps client ID, and organization name. The generated file exports system.billing.usage and system.billing.list_prices and is ready to import as a Databricks notebook.
  5. Click Save and open CloudFormation. Deploy the stack in the customer AWS account. It creates a prefix-scoped, read-only role that the existing nops-data-connector can assume. The stack does not modify an existing bucket policy.
  6. After the stack finishes, paste the deployed role ARN and click Save deployed role.
Use a dedicated prefix. A shared prefix can overwrite other data under that path. The export is not CUR.

​
Import and schedule the notebook in Databricks

nOps does not create, run, or manage jobs in your workspace. After you download or copy the notebook in nOps:
  1. Log in to your Databricks workspace.
  2. Import the generated .py file (or create a notebook and paste the copied script). You can name it something like NopsDatabricksBillingDataUploader.
  3. Grant this workspace permission to write to the S3 destination (for example a Unity Catalog external location). The CloudFormation role is for nOps read access only.
  4. In the notebook toolbar, click Schedule.
  5. Select a compute cluster and set the frequency to Every 1 day.
  6. Create the schedule and confirm a run completes.
Daily scheduling keeps Inform current. The first sync can take up to 24 hours.

​
Azure Databricks

Coming soon. Azure Databricks onboarding will use an Azure-native export storage and identity setup. No AWS CloudFormation step is available for Azure.

​
Troubleshooting

  • Shared or CUR prefix — use a dedicated prefix such as nops/databricks. The notebook overwrites only its own year/month/table/client path under that prefix.
  • No data after about 24 hours — confirm the scheduled notebook job is succeeding, objects exist under the prefix, and the deployed role ARN is saved in nOps.
  • Notebook cannot write to S3 — grant Unity Catalog (or equivalent) write access to the export bucket and prefix. The nOps CloudFormation role does not grant Databricks write.
  • CloudFormation / role errors — deploy in the AWS account that owns the bucket; paste the role ARN after the stack completes. Existing buckets must be in that account (or separately grant access). The stack does not update an existing bucket policy.
  • Azure tab — Azure export is not available yet. Use AWS.

​
Security

nOps stores only the non-secret AWS destination and cross-account role configuration. The CloudFormation stack creates a prefix-scoped, read-only role trusted by nops-data-connector with your external ID. nOps never stores your Databricks workspace URL, warehouse ID, machine credential, or job configuration. The customer-run notebook writes billing Parquet to your bucket; nOps reads only that prefix.