Slow or failing HTTP endpoint

Requests to a service are slow or time out, and you need to find where the time goes. All playbooks.

Split one request into DNS, connect, TLS and total time. The slowest phase tells you where to look.
Curl with timing breakdown

Shows detailed timing of DNS, connect, TLS, and transfer phases.
```
curl -o /dev/null -s -w "DNS: %{time_namelookup}s\nConnect: %{time_connect}s\nTLS: %{time_appconnect}s\nTotal: %{time_total}s\n" https://example.com
```
curl, timing, latency
If DNS is slow or wrong, follow the resolution path.
Trace DNS resolution path

Follows the DNS delegation path from root servers.
```
dig +trace <domain>
```
dns, dig
If connect is slow or fails, check the port is reachable.
Test TCP port connectivity

Uses netcat to check if a remote TCP port is reachable.
```
nc -vz <hostname> <port>
```
tcp, connectivity, nc
Check whether it affects everyone, starting with p99 latency.
PromQL: P99 latency

Calculates the 99th percentile request latency from a histogram.
```
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
```
promql, latency, p99
Then the share of requests failing with 5xx.
PromQL: HTTP error rate

Calculates the percentage of 5xx responses over the last 5 minutes.
```
sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) * 100
```
promql, errors, sre
On the server, count connections by state. A pile of TIME_WAIT or SYN_RECV points at connection handling.
Count connections by state

Groups and counts TCP connections by their state (ESTABLISHED, TIME_WAIT, etc.).
```
ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
```
connections, tcp, debug

Slow or failing HTTP endpoint

Curl with timing breakdown

Trace DNS resolution path

Test TCP port connectivity

PromQL: P99 latency

PromQL: HTTP error rate

Count connections by state