The Problem

If you’ve integrated Grafana Alerting with PagerDuty, you’ve probably noticed something frustrating: the PagerDuty incident details are cluttered with every single label and annotation from your alerts. Here’s what you typically see:

This wall of text makes it hard for your on-call engineers to quickly identify what’s wrong. And actually, this was also hard for our Managed Service DBAs. They complained about this for some while, and now I took the time to dig into this as much as I can.

Because you only need the essential information, example: alert name, service, cluster, and environment. But how do you customize this?

The Documentation Gap

The official Grafana PagerDuty integration documentation only explains how to set up the basic integration: creating a service in PagerDuty, obtaining the integration key, and connecting it to Grafana. It says nothing about customizing the alert details or the fact that you can override the default verbose output. (If someone finds that this is documented on Grafana or on PagerDuty I will stand corrected and accept my defeat.)

The Discovery Journey

Step 1: Investigating the PagerDuty Payload

The problem started with verbose PagerDuty alerts. To understand what Grafana was actually sending, I checked the Events API in PagerDuty to see the raw payload. Here’s what I found:

The issue was clear: the details.firing key contained a massive wall of text with every single label and annotation.

Step 2: Experimenting with the Details Section

Back in the Grafana UI, I noticed the PagerDuty contact point configuration had a Details section with the description: “A set of arbitrary key/value pairs that provide further detail about the incident.”

I started experimenting:

  1. Added a custom key like cluster with value {{ .CommonLabels.cluster }}
  2. Triggered an alert
  3. Checked the PagerDuty Events API – the new key appeared in the payload!

Step 3: The Breakthrough – Overriding “firing”

After a few iterations, I had a thought: What if I add a key called firing to override the default verbose one?

I added: – Key: firing – Value: {{ .CommonLabels.alertname }}

Triggered another alert, checked PagerDuty, and… it worked! The firing key now contained just the alert name instead of the entire label dump.

Step 4: Confirming with Source Code

Now knowing what to look for, I searched for documentation and found: – A GitHub discussion from 2022 where user jquick mentioned this behavior – The actual implementation in Grafana’s alerting repository at receivers/pagerduty/v1/config.go:

The comment is clear: “Default values get overwritten in case of duplicate keys.”

Next time I should go straight to the source code first! 🙂

So when you add a firing key in your PagerDuty contact point’s Details section, it replaces the default __text_alert_list template that dumps all labels and annotations.

The Solution: Override the Firing Key

Here’s how to customize your PagerDuty details to show only what matters.

Step 1: Navigate to Your PagerDuty Contact Point

In Grafana:

  1. Go to Alerting → Contact points
  2. Find your PagerDuty contact point and click Edit
  3. Scroll down to the Details section

Step 2: Add Custom Key-Value Pairs

Click + Add and create these key-value pairs:

Key Value
firing {{ .CommonLabels.alertname }}
cluster {{ .CommonLabels.cluster }}
environment {{ .CommonLabels.environment }}
node_name {{ .CommonLabels.node_name }}
service_name {{ .CommonLabels.service_name }}
status {{ .Status }}

Important: The firing key is what overrides the default verbose output!

Step 3: The Result

Now your PagerDuty incidents will have clean, readable custom details:

Much better!

You must use .CommonLabels (not .Labels) in the Details section because you’re at the root template level, not inside a loop.

What about resolved incidents?

I hope you noticed the following in the code:

When an alert resolves it will dump all the labels into the resolved key. If you want to tidy up that message as well you need to add an override for the resolved key as well.

Which Keys Can You Override?

Based on the source code, these are the default PagerDuty detail keys you can override:

  1. firing – The main incident details (what you see in PagerDuty)
  2. resolved – Details shown when alerts resolve
  3. num_firing – Count of firing alerts (usually leave as default)
  4. num_resolved – Count of resolved alerts (usually leave as default)

You can also add any custom keys you want—they’ll be added alongside the defaults.

If you feel brave enough!

Even the __text_alert_list  can be overridden just by creating a custom template and name it __text_alert_list:

But this could impact all the other contact points which might rely on this template.

Advanced: Using Notification Templates

For more complex formatting, create a notification template:

Create the Template

Go to Alerting → Contact points → Notification Templates tab → + Add notification template

Template Name: pagerduty_clean_firing

Template Content:

 

Use the Template in Details

In your PagerDuty contact point Details:

Key: firing
Value: {{ template "pagerduty_clean_firing" . }}

This gives you multi-line formatted output in PagerDuty with conditional fields.

Any workarounds?

You can also work on your alert expressions and only return the needed labels, example:

In this case Grafana will send only these labels to PagerDuty. But this may require changing some of the logic in your alert expressions.

 

Why This Matters

Clean PagerDuty details mean: 

– Faster incident response – Engineers see critical info immediately 

– Better filtering – PagerDuty can use custom details for routing rules 

– Cleaner integrations – If you’re forwarding to Slack or other tools, you get clean data 

– Professional appearance – No more walls of text in your incidents

And most importantly I hope we can finally make our DBAs happy.

Conclusion

Customizing PagerDuty details in Grafana is possible – you just need to know the secret: override the firing key in the Details section.

This isn’t a hack or workaround; it’s the intended design. The Grafana team implemented a clean merge function that prioritizes user-defined details over defaults. It just needs better documentation.

Hopefully, this guide saves you the hours of trial and error, GitHub digging and source code reading that it took to figure this out!

 

Subscribe
Notify of
guest

0 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments