Span instrumentation
Libraries and SDKs added at service boundaries record keys, database calls and outbound requests, so each hop contributes its own timing without rewriting business logic.
Reliability & Performance
Distributed Tracing
Follow one request through every service and see which hop added the delay.
The work
What this actually does
A request entering your gateway might cross eight services before it returns. When it takes two seconds, average latency charts tell you the system is slow but not where. Tracing records the path a single request took, timing each span, so the slow hop is named instead of hunted.
We use OpenTelemetry because it keeps instrumentation portable: the same spans can go to Jaeger, Tempo or a commercial backend, so the choice of vendor stays reversible. Context is propagated through HTTP headers, queues and scheduled jobs, which is where most tracing efforts quietly fall apart, and we make sure traces link back to the log lines from the same request.
If any of these sound familiar
- We know a request is slow but not which service
- Every team blames the service downstream of them
- Timeouts are guessed at rather than measured from data
- Async work disappears once it leaves the request thread
Scope
What's included
Every engagement on this page covers the following, sized to your setup rather than delivered as a fixed package. If something here is not relevant to you, it comes off the scope and off the price.
Context propagation
Trace identifiers travel through headers, message queues and background workers, which is what keeps one logical request connected when it is spread across asynchronous steps.
Latency breakdowns
Service maps and span waterfalls show where time is spent, separating slow databases from slow serialisation and network waits from genuine application work.
Trace and log correlation
Every trace carries an identifier that also appears in your log lines and error reports, so one click moves from a slow span to the exact failure message.
Sampling strategy
Head and tail sampling rules keep the interesting traces — errors and outliers — while dropping the routine ones, which controls both cost and storage volume.
Performance budgets
Once hops are measured, we set latency budgets per service so a regression from a new release shows up in a graph rather than in a complaint.
What changes
What teams typically see
1 traceFull path of a request
800 msTypical slow hop found
top 20Endpoints traced first
Handover
What you keep
Everything produced during the engagement is yours: the repositories, the accounts, the documentation. There is no proprietary layer and nothing to unlicense if you take the work in-house.
- OpenTelemetry instrumentation library for your main runtime
- Collector configuration with sampling rules in Git
- Service map showing call paths and dependencies
- Trace-to-log correlation setup and query examples
- Latency baseline report per critical endpoint
Tooling
Tools we use here
A starting point, not a requirement. We work in whatever you already run wherever it does the job.
OpenTelemetry
Jaeger
Grafana
Elastic
Kubernetes
AWS
Istio
How it runs
From first call to handover
The same four steps on every engagement. You see each one before it starts and can stop at any of them.
- 01
Pick the critical journeys
We start from the transactions that matter to the business and the incidents your team remembers, rather than tracing an arbitrary service because it was easy.
- 02
Instrument the first service
One service gets traced end to end and reviewed with its team, so the pattern is proven before it is rolled out across the estate.
- 03
Roll out across services
Instrumentation lands service by service through pull requests, with the collector and sampling configuration versioned centrally so consistency is not left to chance.
- 04
Use it on a real incident
We sit with your engineers during a live investigation and show how the traces shorten the search, which is what makes the tooling stick.
Questions
Asked before we start
Will tracing slow down our production traffic?
With sensible sampling the overhead is small, typically low single-digit percent. The cost is real, so we measure it in staging before enabling collection in production, and we never trace every request at full fidelity by default.
We use a vendor agent already. Is that enough?
It covers whatever the vendor instruments automatically, which is usually HTTP and a few databases. The gaps tend to be internal calls, queue consumers and custom code, and those are exactly the places incidents happen.
Do we need to standardise on one language?
No. OpenTelemetry has stable support for the mainstream runtimes, so a mixed Java, Go and Python estate is fine. The work is in the conventions for naming spans consistently across them.
How is this different from just better logging?
Logs are events from one service; a trace is the causal chain across all of them. You can reconstruct a trace from logs with correlation ids, but it is slow and manual, which is the difference between answering an incident in minutes and in hours.
Reliability & Performance
Often needed alongside this
Reliability & Performance
Centralised Logging
Every log line searchable in seconds, with retention policies that don't cost more than your compute.
- Elasticsearch, Loki and CloudWatch Logs pipelines
- Structured logging standards and parsing
- Retention, tiering and access controls
Reliability & Performance
Monitoring & Observability
Know what broke, why it broke and who it affects — before your customers have to tell you.
- Prometheus, Grafana, Datadog & CloudWatch
- OpenTelemetry tracing and structured logging
- SLO dashboards and actionable alert routing
Reliability & Performance
Load & Performance Testing
Find the breaking point in a test environment instead of during your busiest hour.
- k6, JMeter and Locust test design
- Realistic traffic modelling and soak tests
- Bottleneck analysis across app, DB and network
Worth a conversation about Distributed Tracing?
Bring the specific problem. We will tell you honestly whether this is the service that fixes it, and what it would take.