Alert rule review
We go through every existing rule, ask what action the responder should take, and either rewrite it around a clear signal or delete it.
Reliability & Performance
Alerting & On-Call Tooling
Fewer pages, sent to the right person, and silence when nothing is wrong.
The work
What this actually does
An on-call rota is only sustainable when the pager means something. Teams usually arrive with hundreds of rules, most of them firing on symptoms nobody can act on, and a rotation that has learned to ignore the noise. We rebuild the alert set around user-visible conditions, then make sure each one reaches a named person who can do something about it.
The plumbing matters as much as the rules: schedules, escalation policies, notification channels and quiet hours all need to reflect how your team actually works, including holidays and handovers. We configure PagerDuty, Opsgenie or Grafana OnCall and test it with a real page during the working day, so the first time it fires is not at three in the morning.
If any of these sound familiar
- The pager fires all night for things nobody fixes
- Real incidents get lost in a stream of noise
- Nobody knows who is on call this weekend
- Every alert routes to the same shared inbox
Scope
What's included
Every engagement on this page covers the following, sized to your setup rather than delivered as a fixed package. If something here is not relevant to you, it comes off the scope and off the price.
Routing and escalation
Alerts are mapped to the owning team with escalation paths, so an unanswered page moves on after a defined interval instead of sitting unread over a weekend.
Noise reduction
Grouping, inhibition and dependency rules collapse a cascade of related failures into one page, which is usually the single largest reduction in volume.
On-call handover process
A written routine covers what the outgoing engineer hands over, how schedule swaps work, and where the current state of open incidents is recorded.
Responder feedback loop
After each shift we log which pages were useful and which were not, then adjust the rules, so the rota keeps improving rather than calcifying.
Health checks that matter
Synthetic probes from outside your network test the journeys customers actually use, catching failures that internal metrics miss entirely.
What changes
What teams typically see
60%Typical cut in page volume
1 ownerNamed per alert
24/7Escalation coverage configured
Handover
What you keep
Everything produced during the engagement is yours: the repositories, the accounts, the documentation. There is no proprietary layer and nothing to unlicense if you take the work in-house.
- Reviewed alert inventory with owners and severity
- Escalation policies mapped to owning teams
- Notification channel and quiet hours setup
- Runbook link attached to every alert page
- Alert volume baseline with monthly review notes
Tooling
Tools we use here
A starting point, not a requirement. We work in whatever you already run wherever it does the job.
PagerDuty
Prometheus
Grafana
Datadog
Kubernetes
AWS
How it runs
From first call to handover
The same four steps on every engagement. You see each one before it starts and can stop at any of them.
- 01
Measure the current noise
We pull the last month of pages and rank them by volume and by how often they were resolved with no action at all.
- 02
Rewrite around symptoms
Rules are rebuilt around user-visible conditions and clear thresholds, with the noisy infrastructure checks demoted to dashboards where they belong.
- 03
Configure routing and policy
Schedules, escalation steps and notification preferences are set up and written down, including who is accountable when a page is not acknowledged.
- 04
Test with a live page
We trigger each critical alert during working hours, watch it arrive on every channel, and fix the gaps while everyone is awake.
Questions
Asked before we start
Can you reduce noise without missing real outages?
That is the whole point, and it needs judgement rather than a blanket threshold. We look at what each page was supposed to detect, keep coverage for genuine failure, and remove only the rules that have demonstrably never led to action.
Do we have to change paging vendor?
No. We work with PagerDuty, Opsgenie, Grafana OnCall or whatever you run today. The vendor is rarely the problem; unowned alerts and missing escalation paths are, and those travel with you either way.
What if engineers disagree about an alert being useful?
That disagreement is useful evidence. We default to a written owner and an explicit action for each rule, and if neither can be produced, the alert is removed. Removing a page is reversible; burnout is not.
How do you handle alerts outside working hours?
We separate what genuinely cannot wait until morning from what can, and only the former pages anyone. Everything else waits in a queue with the same owner and severity, reviewed at the start of the next working day.
Reliability & Performance
Often needed alongside this
Reliability & Performance
SRE, On-Call & Incident Response
Error budgets, runbooks and an escalation path that keeps 3 a.m. pages rare and short.
- SLI/SLO definition and error-budget policy
- Alert tuning and on-call rotation design
- Blameless postmortems and remediation tracking
Reliability & Performance
Monitoring & Observability
Know what broke, why it broke and who it affects — before your customers have to tell you.
- Prometheus, Grafana, Datadog & CloudWatch
- OpenTelemetry tracing and structured logging
- SLO dashboards and actionable alert routing
Optimisation & Advisory
Technical Documentation & Runbooks
Diagrams, decision records and step-by-step runbooks that turn a 3 a.m. page into a checklist.
- Architecture diagrams kept in sync with reality
- Operational runbooks for every alert
- Architecture decision records and onboarding guides
Worth a conversation about Alerting & On-Call Tooling?
Bring the specific problem. We will tell you honestly whether this is the service that fixes it, and what it would take.