NewRelic Workflows Migration via Terraform

NewRelic Workflows Migration via Terraform

Introduction

In the NewRelic deployment guide 2021, Terraform-driven NewRelic deployment was demonstrated using legacy alert structures:

  • Alert Policies (newrelic_alert_policy)
  • Notification Channels (newrelic_alert_channel for email/Slack)

Fast-forward to 2024, NewRelic’s breaking changes have deprecated legacy alerts in favour of Workflows. Workflows offer a more flexible approach to incident management compared to legacy alerts, especially powerful for organisations with complex environments, multiple teams, or a need for richer alert context.

Here’s a detailed comparison between NewRelic’s legacy Alerts and the newer Workflows system:

Feature / Capabilitylegacy AlertsWorkflows
Event GranularityLimited to incident open/closeMultiple events: activated, acknowledged, closed, priority change
Routing FlexibilityStatic notification channels per policyDynamic routing with filters and multiple destinations
Notification DestinationsEmail, Slack, Webhook, PagerDuty (limited)Slack, Email, Webhook, PagerDuty, Jira, ServiceNow, AWS EventBridge
Message CustomizationMinimalFully customizable with templates and variables
Data EnrichmentNot supportedSupports NRQL-based enrichment and entity metadata
Visual Content in AlertsNot supportedSupports charts and screenshots in Slack/email
Workflow TestingNot availableYes – test with historical issues
ScalabilityLimited to alert policiesUp to 1000 workflows per account
Maintenance & ManagementManual updates per policyCentralized, reusable workflows

This updated guide preserves core Terraform principles from the original blog while introducing:

  • Microsoft Teams integration with AdaptiveCard JSON
  • Workflows-based alerts (replacing alert channels)
  • Custom payload engineering for actionable alerts

This post demonstrates key code snippets, guiding you on how to migrate to NewRelic Workflows via Terraform. As a prerequisite, it’s assumed that you are familiar with NewRelic and Terraform. If these tools are new to you, you can get started quickly by reading this guide, and view an example of setting up a NewRelic Provider in Terraform here.

Getting Started: What’s New

1. Provider & Authentication Updates

First, to use Terraform to deploy the Workflows, the NewRelic provider version must be upgraded from 2.x to 3.x.

required_providers {
  newrelic = {
    source  = "newrelic/newrelic"
    version = "~> 3.26.1"
  }
}

2. Replace newrelic_alert_channel with newrelic_notification_destination and newrelic_notification_channel

Next, since NewRelic does NOT support legacy alerts, the Terraform newrelic_alert_channel resource shown in the code snippet in the previous blog MUST be replaced with newrelic_notification_destination and newrelic_notification_channel, which is required in NewRelic Workflows. The newrelic_notification_destination is a reusable receiver config, e.g., Slack, Microsoft Teams, Email or AWS EventBridge, which allows multiple workflows to send alert messages to, and the newrelic_notification_channel is used to define the customised alert message, allowing users to generate alert message dynamically according to runtime alert values by simply using {{ }} placeholders. Workflow variables show the predefined variables.

Here is an example showing how to create a Microsoft Teams notification channel in NewRelic with a customized Adaptive Card payload that formats alerts for Teams. The webhook for Microsoft Teams will be created manually in advance. The payload leverages Microsoft’s Adaptive Card schema to create rich interactive messages. You can use Adaptive Card Designer to design the alert message and get feedback from the business. The key elements in the example schema include:

  1. Dynamic Alert Data: Uses NewRelic template variables like {{issueTitle}} and {{accumulations.conditionDescription}} to inject real-time alert details
  2. Structured Layout: Organises content with bold headers, descriptive text blocks, and fact sets
  3. Action Buttons: Includes two deep links:
    • NewRelic IssuePage: Direct link to the incident using {{json issuePageUrl}}
    • NewRelic Dashboard: Direct link to the dashboard using (${newrelic_one_dashboard.service_dashboard.permalink})
resource "newrelic_notification_destination" "teams_destination" {  
  name = "teams_destination"  
  type = "WEBHOOK"  

  property {  
    key   = "url"  
    value = "https://your-teams-webhook-url"  
  }   
}

resource "newrelic_notification_channel" "teams_channel" {
  depends_on = [
    newrelic_one_dashboard.service_dashboard
  ]
  name           = "teams-channel-${var.app_environment}"
  type           = "WEBHOOK"
  destination_id = newrelic_notification_destination.teams_destination.id
  product        = "IINT"

  property {
    key   = "payload"
    value = <<EOT
{
   "attachments":[
      {
         "content":{
            "$schema":"http://adaptivecards.io/schemas/adaptive-card.json",
            "body":[
               {
                  "columns":[
                     {
                        "items":[
                           {
                              "text":"Alerts DEMO",
                              "type":"TextBlock",
                              "weight":"Bolder",
                              "wrap":true
                           }
                        ],
                        "type":"Column",
                        "width":"stretch"
                     }
                  ],
                  "type":"ColumnSet"
               },
               {
                  "text":"{{issueTitle}}",
                  "type":"TextBlock",
                  "wrap":true
               },
               {
                  "facts":[
                     {
                        "title":"Details",
                        "value":"{{#each accumulations.conditionDescription}}{{this}}{{#unless @last}}, {{/unless}}{{/each}}"
                     }
                  ],
                  "type":"FactSet"
               }
            ],
            "actions": [
               {
                  "type": "Action.OpenUrl",
                  "title": "Newrelic IssuePage",
                  "url": {{json issuePageUrl}}
               },
               {
                  "type": "Action.OpenUrl",
                  "title": "Newrelic Dashboard",
                  "url": "${newrelic_one_dashboard.service_dashboard.permalink}" # Get the permalink from the dashboard resource
               }
            ],
            "type":"AdaptiveCard",
            "version":"1.0"
         },
         "contentType":"application/vnd.microsoft.card.adaptive"
      }
   ],
   "type":"message"
}
EOT
    label = "Payload Template"
  }
}

3. Workflow-Driven Alerting

Finally, alert policy and alert condition are required in legacy alert and workflows and newrelic_workflow is newly introduced. The workflow targets alert policies by conditional filters and sends alerts to alert channels created above. The following example shows how to set up a workflow to monitor Lambda invocation errors. By using a suitable operator in predicate, the workflow can capture alerts from different policies, allowing routing flexibility instead of one-to-one mapping. In addition, workflows introduce a new feature — enrichment — to ingest an NRQL query result into the alert message. In newrelic_workflow, enable the enrichment feature by setting enrichments_enabled to true and write the NRQL in Enrichments blocks with a unique name. Then an alert channel payload template can consume the query result by using {{nrql_enrichment_name}}. If the value is not working in the alert message, try to debug the variable in the variables menu. The workflow enrichment variables will appear at the top of the list after typing {{. Note that currently workflows only support up to 2 enrichments. In addition, the Test workflow button shown in the following figure helps developers and testers easily test the workflow notification with the exisiting alert data.

resource "newrelic_alert_policy" "alert_policy" {
  name                = "alert-test"
  incident_preference = "PER_CONDITION"
}

resource "newrelic_workflow" "demo_workflow" {
  name                  = "demo"
  muting_rules_handling = "NOTIFY_ALL_ISSUES"

  issues_filter {
    name = "Filter-name"
    type = "FILTER"

    predicate {
      attribute = "labels.policyIds"
      operator  = "EXACTLY_MATCHES"
      values = [
        newrelic_alert_policy.alert_policy.id
      ]
    }
  }

  destination {
    channel_id            = newrelic_notification_channel.teams_channel.id
    notification_triggers = ["ACTIVATED"] # Optional
  }
}

resource "newrelic_nrql_alert_condition" "lambda_error_rate" {
  policy_id   = newrelic_alert_policy.alert_policy.id
  type        = "static"
  name        = "Lambda ${local.lambda_name} Error Spike "
  description = "RequestId {{tags.requestId}} has errors."
  enabled     = true

  fill_option                    = "last_value"
  violation_time_limit_seconds   = 900
  aggregation_window             = 150
  aggregation_method             = "event_timer"
  aggregation_timer              = 60
  aggregation_delay              = null
  expiration_duration            = null
  open_violation_on_expiration   = null
  close_violations_on_expiration = null

  nrql {
    query = "SELECT count(*) FROM AwsLambdaInvocationError FACET aws.requestId AS requestId WHERE (`entityName` = '${local.lambda_name}') EXTRAPOLATE"
  }

  critical {
    operator              = "above"
    threshold             = 0
    threshold_duration    = 300
    threshold_occurrences = "AT_LEAST_ONCE"
  }

  lifecycle {
    ignore_changes = [enabled]
  }
}

Conclusion

In the 2021 post, I found Terraform as a game-changer for NewRelic deployments—and that still stands! With NewRelic’s mandatory shift to workflows, alerting has leveled up.

Why this matters now:

  • Legacy alerts are officially retired
  • Workflows unlock richer, context-aware notifications
  • The example smooths your migration path

Setup NewRelic Workflows migration via Terraform today and turn alert management from a chore into your superpower.

rongjun.xie@shinesolutions.com
No Comments

Leave a Reply

Discover more from Shine Solutions Group

Subscribe now to keep reading and get access to the full archive.

Continue reading