Skip to main content

Migrating nOps Kubernetes Agent for Container Insights

If you're migrating from a previous installation of the Container Insights Kubernetes agent (originally installed using a bash script), the new installation method involves updating the CloudFormation Stack. This update creates a Lambda function that manages the creation and update of the IAM Roles for Service Accounts (IRSA) required by the agent to push metrics to the S3 Bucket associated with your AWS account. Additionally, you'll need to run a cleanup script to remove the old Helm release and delete any unused namespaces from the previous installation.

Prerequisites​

  1. Access to the Kubernetes cluster (recommended version v1.23.6 or later) to deploy the agent.
  2. A previous version of the agent installed in your cluster. You can verify this by listing the namespaces and looking for the following: nops-prometheus-system, nops-cost, and nops-k8s-agent.
  3. AWS CLI
  4. Helm
  5. kubectl
  6. Unix-like terminal to execute the installation script.

Updating CloudFormation Stack​

Update gif

  1. Navigate to CloudFormation service

    • Go to your Container Cost CloudFormation stack (e.g. container-cost-setup-<your_Account_ID>), and ensure you're in the correct AWS region where the stack was initially created.

  2. Update CloudFormation Stack

Clean up and installation script​

  1. Save Script

    • Save the script from here.

  2. Execute Script

    • Grant execution permissions to the script.

      chmod +x nops-cleanup.sh

    • Open your Unix-like terminal.

    • Change/Switch context to your desired cluster.

    • Replace parameters with your own and run the script.

      ParameterDescription
      --datadog-api-keyDatadog API Key
      --container-insightsEnabling or not (true or false) container insights
      --eks-cluster-arnTarget EKS Cluster ARN
      --s3-bucket-nameS3 Bucket name
      --karpenops-enabledInstall KarpenOps agent (true or false)
      --api-keyAPI Key (Get it from the nOps platform onboarding process, required only if KarpenOps agent install is true).
      --cluster-idClusterID (Get it from the nOps platform onboarding process, required only if KarpenOps agent install is true)

      Example:

      ./nops-cleanup.sh  --datadog-api-key=a1234ab12a12abc1a123ab123a12a12a --container-insights=true --eks-cluster-arn=arn:aws:eks:us-east-1:123456789101:cluster/example-cluster --s3-bucket-name=nops-container-cost-123456789101 --karpenops-enabled=true --api-key=1234.a1234a1a123ab1a01234a12a1a1ab1ab --cluster-id=a+ABC1

      Wait for the script to successfully finish.

  3. Verify new agent(s)

    • Confirm new namespace and resources.

      kubectl -n nops get pods
      note

      If the script gets stuck when deleting the namespaces, you can refer to Unable to destroy namespaces.

Unable to destroy namespaces​

In some cases, when trying to delete the namespaces (nops-prometheus-system, nops-cost, nops-k8s-agent) they may end up stuck with a "Terminating" status, for such cases you can do the following:

  1. Open a new terminal

  2. Change/Switch context to your desired cluster.

  3. Run:

    kubectl proxy

  4. Open a second terminal

  5. Change/Switch context to your desired cluster.

  6. Run:

    kubectl get ns nops-prometheus-system -o json | \
    jq '.spec.finalizers=[]' | \
    curl -X PUT http://localhost:8001/api/v1/namespaces/nops-prometheus-system/finalize -H "Content-Type: application/json" --data @-
    kubectl get ns nops-cost -o json | \
    jq '.spec.finalizers=[]' | \
    curl -X PUT http://localhost:8001/api/v1/namespaces/nops-cost/finalize -H "Content-Type: application/json" --data @-
    kubectl get ns nops-k8s-agent -o json | \
    jq '.spec.finalizers=[]' | \
    curl -X PUT http://localhost:8001/api/v1/namespaces/nops-k8s-agent/finalize -H "Content-Type: application/json" --data @-