Game day facilitation
We plan and run a session with a stated hypothesis, a blast radius you approve, an abort condition and someone accountable for watching the system throughout.
Reliability & Performance
Chaos Engineering & Resilience Testing
Break your own system on purpose, while you are watching and in control.
The work
What this actually does
Resilience is a property you either have or you find out you lack. Chaos engineering is the practice of introducing controlled failures — killing a pod, dropping a dependency, adding network latency — to see what the system does before an unplanned event does it for you. It is an experiment with a hypothesis, not a dare.
We start small: a single dependency in a non-production environment, with a rollback understood by everyone in the room. As confidence grows you move to game days on the real system during working hours, with a stop button and a designated person watching the dashboards. Findings go into the backlog as concrete work.
If any of these sound familiar
- We only learn how things fail during real outages
- Nobody has tested what happens when a dependency dies
- Retries and timeouts make an outage worse, not better
- Resilience assumptions were written once and never checked
Scope
What's included
Every engagement on this page covers the following, sized to your setup rather than delivered as a fixed package. If something here is not relevant to you, it comes off the scope and off the price.
Failure injection
Pod kills, network partitions, latency and dependency blackholes applied in a controlled way, using tooling that respects your clusters and makes every action reversible.
Dependency failure drills
We take out the third-party API, the message broker or the identity provider and observe whether your system degrades or collapses, one dependency at a time.
Steady-state hypothesis
Before any experiment we define the normal behaviour in numbers, so we can tell the difference between an expected wobble and a real regression.
Findings into backlog
Every experiment produces notes, and every finding becomes a sized ticket with an owner, so the exercise ends in fixes rather than anecdotes.
Resilience patterns
Where drills expose missing timeouts, unbounded retries or absent circuit breakers, we implement the patterns and show your team how they behave under load.
What changes
What teams typically see
1 hourFirst game day session
0Untested critical dependencies
weeklyFindings reviewed with owners
Handover
What you keep
Everything produced during the engagement is yours: the repositories, the accounts, the documentation. There is no proprietary layer and nothing to unlicense if you take the work in-house.
- Game day plan with hypothesis and abort conditions
- Reusable experiment templates for your platform
- Written findings with evidence from each drill
- Backlog items for every resilience gap found
- Circuit breaker and timeout configuration where missing
Tooling
Tools we use here
A starting point, not a requirement. We work in whatever you already run wherever it does the job.
Kubernetes
Istio
AWS
OpenTelemetry
Prometheus
Grafana
Apache Kafka
How it runs
From first call to handover
The same four steps on every engagement. You see each one before it starts and can stop at any of them.
- 01
Map the dependencies
We inventory what each critical service depends on and rank those dependencies by how much damage their failure would cause.
- 02
Agree scope and safety
Blast radius, timing and abort conditions are agreed in writing with the service owners before any experiment touches anything.
- 03
Run the first drill
A single, reversible failure in the lowest-risk environment, watched live, with everyone clear on who can stop it and how.
- 04
Fix and widen the net
We turn findings into work, retest once fixes land, then increase the scope of later sessions as confidence and coverage grow.
Questions
Asked before we start
Is chaos engineering safe to run on production?
It can be, once you have practised elsewhere and the blast radius is small. The first experiments belong in a non-production environment. Production game days come later, during working hours, with an abort switch and the affected teams in the room.
Do we need special tools to start?
For the first session, no. A script that deletes a pod or blocks a port is enough to learn something. Dedicated platforms become worthwhile when you run experiments regularly and need scheduling and audit history.
What if an experiment causes a real incident?
That is a genuine risk, which is why scope, timing and abort conditions are agreed up front and why we start small. If it happens, the incident process runs as normal and the experiment becomes the first item in the postmortem.
How is this different from a load test?
Load testing asks how much traffic the system can take; chaos testing asks how it behaves when a component is missing or slow. They complement each other, and we often run the same journeys under both conditions.
Reliability & Performance
Often needed alongside this
Reliability & Performance
SRE, On-Call & Incident Response
Error budgets, runbooks and an escalation path that keeps 3 a.m. pages rare and short.
- SLI/SLO definition and error-budget policy
- Alert tuning and on-call rotation design
- Blameless postmortems and remediation tracking
Reliability & Performance
Backup, HA & Disaster Recovery
Tested recovery procedures with documented RTO and RPO — because an untested backup is not a backup.
- Multi-AZ, multi-region and failover design
- Automated backups with restore rehearsals
- Database HA, replication and migration safety
Reliability & Performance
Load & Performance Testing
Find the breaking point in a test environment instead of during your busiest hour.
- k6, JMeter and Locust test design
- Realistic traffic modelling and soak tests
- Bottleneck analysis across app, DB and network
Worth a conversation about Chaos Engineering & Resilience Testing?
Bring the specific problem. We will tell you honestly whether this is the service that fixes it, and what it would take.