Skip to main content

Automating GCP Setup with Terraform

The nOps GCP API Enablement Terraform Module automates the process of enabling specific Google Cloud Platform APIs across all projects in an organization. This ensures consistency and saves significant time compared to manual enablement.

Repository: github.com/nops-io/terraform-gcp-nops-integration

Overview​

This module enables the following APIs required for full nOps visibility. For a detailed explanation of why each API is needed and cost implications, see the API Requirements & Justification page.

API ServiceAPI Service IDScope
Cloud Asset APIcloudasset.googleapis.comCentral Ingestion Project
Cloud Billing APIcloudbilling.googleapis.comCentral Ingestion Project
Recommender APIrecommender.googleapis.comAll projects (scoped to billing account)
Kubernetes Engine APIcontainer.googleapis.comAll Target Customer Projects with GKE (⚠️ requires billing)
BigQuery Reservation APIbigqueryreservation.googleapis.comAll projects (configurable)
Important

The Kubernetes Engine API requires billing to be enabled on target projects, as it enables multiple paid services (Compute Engine, Container Registry, Artifact Registry, DNS, etc.).

Prerequisites​

Before using this module, ensure you have the following:

  • Terraform (>= 1.0) or OpenTofu installed.
  • Billing Enabled:
    • Central Ingestion Project: Billing must be enabled for the central ingestion project.
    • GKE Projects: Billing must be enabled for all projects where the Kubernetes Engine API will be enabled.
    • Other Projects: Billing is recommended for all projects where APIs will be enabled.
  • Google Cloud Credentials with the following permissions:
    • resourcemanager.projects.list - to list all projects in the organization.
    • serviceusage.services.enable - to enable APIs.
    • serviceusage.services.get - to check API status.
    • Organization-level or project-level admin role.

Installation​

Using Terraform / OpenTofu​

  1. Clone or download the module: 
    git clone https://github.com/nops-io/terraform-gcp-nops-integration.git
    cd terraform-gcp-nops-integration

Usage​

Basic Example​

Create a main.tf file in your working directory:

terraform {
  required_version = ">= 1.0"
  
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 4.0"
    }
  }
}

provider "google" {
  # Option 1: Use Application Default Credentials (recommended)
  # Run: gcloud auth application-default login
}

module "enable_gcp_apis" {
  source = "github.com/nops-io/terraform-gcp-nops-integration"
  
  organization_id              = "123456789012"  # Your GCP Organization ID
  central_ingestion_project_id = "my-central-project-id"
  
  # All APIs are enabled by default.
  # To customize, you can override defaults:
  # enable_cloud_asset_api = false
}

Advanced Example (Specific GKE Projects)​

module "enable_gcp_apis" {
  source = "github.com/nops-io/terraform-gcp-nops-integration"
  
  organization_id              = "123456789012"
  central_ingestion_project_id = "central-ingestion-project"
  
  # Only enable specific APIs (GKE API is always enabled in target projects)
  enable_cloud_asset_api   = true
  enable_cloud_billing_api = true
  enable_recommender_api   = true
  
  # GKE API is always enabled for specific projects listed below
  enable_gke_apis_for_all_projects = false
  target_gke_project_ids = [
    "gke-project-1",
    "gke-project-2",
    "gke-project-3"
  ]
}

Running the Module​

  1. Initialize Terraform: 
    terraform init
  2. Review the Plan: 
    terraform plan
  3. Apply Configuration: 
    terraform apply

Authentication​

gcloud auth application-default login

This uses your user credentials. Ensure you have the necessary permissions.

Option 2: Service Account Key​

  1. Create a service account with required permissions.
  2. Download the JSON key file.
  3. Set the path in your provider configuration: 
    provider "google" {
      credentials = file("path/to/service-account-key.json")
    }

Troubleshooting​

Error: "Billing account for project is not found"​

This occurs when trying to enable the Kubernetes Engine API on a project without billing enabled.

Solution:

  1. Enable billing on the affected project: 
    gcloud beta billing projects link PROJECT_ID --billing-account=BILLING_ACCOUNT_ID
  2. Or, exclude the project from the GKE API scope by setting enable_gke_apis_for_all_projects = false and specifying only valid projects in target_gke_project_ids.

Finding IDs​

Finding Your Organization ID​

gcloud organizations list

Finding Your Central Ingestion Project ID​

The central_ingestion_project_id is the project where Cloud Asset and Billing APIs will be enabled. It should be an existing project you have admin access to.

gcloud projects list

Use the PROJECT_ID column.