Databricks

​

Overview

​

Prerequisites

• nOps organization administrator or owner access
• Permission to deploy CloudFormation (including IAM roles) in the AWS account that holds the export bucket
• Administrative access to a Databricks workspace so you can import a notebook, grant S3 write (for example a Unity Catalog external location), and schedule a daily job
• Either permission for CloudFormation to create a new S3 bucket, or an existing bucket in the same AWS account (or separately grant that account access)

​

Connect in nOps

​

Where to open Databricks

• To connect for the first time: go to Settings → Integrations and click the Databricks card.
• To manage an existing export: open the same card or use Settings → Account Status → Connected apps and click the Databricks chip.

​

Connect steps

1. Click the Databricks card. The AWS tab is the supported path.
2. Choose New S3 bucket (CloudFormation creates the bucket and nOps read role) or Existing S3 bucket (CloudFormation creates a read role scoped to your prefix).
3. Enter the S3 bucket name and a dedicated prefix (default nops/databricks). Do not reuse a shared prefix. The notebook writes system-table Parquet under that prefix and replaces only its own year/month/table/client path.
4. Download .py notebook or Copy notebook. nOps fills in the S3 destination, nOps client ID, and organization name. The generated file exports the system tables listed under What the notebook exports and is ready to import as a Databricks notebook.
5. Click Save and open CloudFormation. Deploy the stack in your AWS account. It creates a prefix-scoped, read-only role that the existing nops-data-connector can assume. The stack does not modify an existing bucket policy.
6. After the stack finishes, paste the deployed role ARN and click Save deployed role.

​

Import and schedule the notebook in Databricks

1. Log in to your Databricks workspace.
2. Import the generated .py file (or create a notebook and paste the copied script). You can name it something like NopsDatabricksBillingDataUploader.
3. Grant this workspace permission to write to the S3 destination (for example a Unity Catalog external location). The CloudFormation role is for nOps read access only.
4. If the job will not run as a metastore admin, grant its run-as user or service principal read access to the system tables (see Grant read access to system tables).
5. In the notebook toolbar, click Schedule.
6. Select a compute cluster and set the frequency to Every 1 day.
7. Create the schedule and confirm a run completes.

​

What the notebook exports

​

Grant read access to system tables

GRANT USE CATALOG ON CATALOG system TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.billing TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.compute TO `<principal>`;
GRANT USE SCHEMA ON SCHEMA system.query TO `<principal>`;
GRANT SELECT ON TABLE system.billing.usage TO `<principal>`;
GRANT SELECT ON TABLE system.billing.list_prices TO `<principal>`;
GRANT SELECT ON TABLE system.billing.account_prices TO `<principal>`;
GRANT SELECT ON TABLE system.billing.cloud_infra_cost TO `<principal>`;
GRANT SELECT ON TABLE system.billing.attributed_usage TO `<principal>`;
GRANT SELECT ON TABLE system.compute.clusters TO `<principal>`;
GRANT SELECT ON TABLE system.compute.warehouses TO `<principal>`;
GRANT SELECT ON TABLE system.compute.node_types TO `<principal>`;
GRANT SELECT ON TABLE system.compute.instance_pools TO `<principal>`;
GRANT SELECT ON TABLE system.compute.node_timeline TO `<principal>`;
GRANT SELECT ON TABLE system.compute.warehouse_events TO `<principal>`;
GRANT SELECT ON TABLE system.compute.instance_events TO `<principal>`;
GRANT SELECT ON TABLE system.query.history TO `<principal>`;

​

Updating an existing export

​

Azure Databricks

​

Troubleshooting

• Shared prefix: use a dedicated prefix such as nops/databricks. The notebook overwrites only its own year/month/table/client path under that prefix.
• No data after about 24 hours: confirm the scheduled notebook job is succeeding, objects exist under the prefix, and the deployed role ARN is saved in nOps.
• Notebook cannot write to S3: grant Unity Catalog (or equivalent) write access to the export bucket and prefix. The nOps CloudFormation role does not grant Databricks write.
• CloudFormation / role errors: deploy in the AWS account that owns the bucket; paste the role ARN after the stack completes. Existing buckets must be in that account (or separately grant access). The stack does not update an existing bucket policy.
• Optional tables skipped: the run output lists skipped tables and the reason. Apply the grants above, or ask an account admin to enable the system.compute / system.query schemas, then re-run.
• Azure tab: Azure export is not available yet. Use AWS.

​

Security