24 KiB
24 KiB
driver.py --port 8080 alerts-check (change slo-alerts, задачи 11.2 и 12.11)
- Основа: commit
ef60b47(fix: address slo-alerts verify findings), рабочее дерево чистое (git status --shortдо прогона пуст). Файлыsrc/иmigrations/(151) в контейнерахapi(2:api-13,api-14),relay,workerиcleanerпобайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемыe4b8c2d17f90./etc/prometheus/rules.ymlиprometheus.ymlв контейнере Prometheus побайтно равны файлам дереваef60b47(cmp). Код приложения с15ea156не менялся:ef60b47правит правила, их тесты, драйвер и документы. Прежний прогон этого файла — на15ea156(до правок verify). - Дата: 2026-11-14, 01:09:30Z – 01:27:20Z (~17,8 мин); хост macOS, Docker 29.9.2, Compose 5.6.1, ВМ
Docker Desktop — 12 CPU / 11,6 ГиБ. Рабочий стенд
mtusi-lab6(две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория:python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 alerts-check, код выхода 0. - Всё — через HTTP API Grafana на
127.0.0.1:3000с паролем из.env(источники-прокси uidprometheus,loki,tempo); моменты pending / firing / снятия —/api/v1/rulesраз в 5 с, время — от начала шага (уMessengerQueueLagHigh— от первой загрузки, у снятия — от восстановления); момент pending печатается перед каждой строкой firing. Пользователи, пароли, токены и файлы — синтетические, в вывод не попадают. Фоновая нагрузка драйвера через LB — 6 потоков отправки сообщения с новымIdempotency-Keyраз в 0,25 с и 1 поток чтения истории раз в 1 с (design, «Уточнения apply», п. 7). - Отказы: шаг 1 —
docker compose stop worker, 3 загрузки,docker startтого же контейнера через 145 с; шаг 2 —docker compose stop relay,docker start; шаг 3 — сессия psql сapplication_name = mtusi-alerts-checkциклами держитLOCK TABLE messages IN SHARE MODE2 с с паузой 0,2 с, затемpg_terminate_backend; шаг 4 — 3 потока входа по Basic 45 с (сравнение, отказа нет); шаг 5 —docker compose stop postgres, затемstart. Вfinally— нагрузка остановлена, сессия блокировки завершена, PostgreSQL, worker и relay запущены, LB готов (последняя строка).
Сценарии → строки вывода
Сценарий (observability, если не указано) |
Строки |
|---|---|
| «Правила загружены» | rule groups loaded, rules with evaluation errors, alerts loaded, Grafana shows rules…, rule change through Grafana rejected, alert delivery addresses, prometheus config has alertmanagers |
| «Остановленный worker» | [worker stopped]: pending через 84 с, firing через 144 с после первой загрузки (≤ 240), outbox + published +3, снято через 29 с после запуска (≤ 120), 3 ready |
| «Перезапуск worker» | [worker stopped] time to ready count…: первая точка нового процесса — 0 (время начала), приращение по точкам ровно 3; worker стоял 145 с (> 2 мин) |
| «Остановленный relay» | [relay stopped] MessengerQueueMetricsMissing: pending через 69 с, firing через 190 с (≤ 300), снято через 45 с (≤ 120) |
| «Остановленный relay уходит из рядов» | [relay stopped] queue depth and age series gone within 120 s — ряды глубины и возраста ушли через 60 с |
| «Медленная БД» | [slow database]: pending через 75 с, firing через 135 с (≤ 240), доля span PostgreSQL в медленных трассах отправки 0,99–1,0, пул 6 против 0, CPU реплик 0,09 предела, снято через 74 с (≤ 240), MessengerHighErrorRate нет |
| «Медленный handler не путается с БД» | [slow handler]: доля span PostgreSQL 0,0–0,2 (медиана 0,01), пул ни одной реплики не занят целиком (не больше 0,20), отказов по сроку 0, pending — наблюдение (0), CPU реплик 0,81 предела, MessengerSlowRequests неактивно (доля медленных до 0,072) |
| «Тяжёлая ручка вне SLI задержки» | [slow handler] login p95 on the route panel slower than 0.25 s (p95 входа 0,598 с) при неактивном MessengerSlowRequests |
| «Недоступная БД» | [database down]: MessengerNoReadyReplicas pending через 29 с, firing через 59 с, MessengerHighErrorRate pending через 44 с, firing через 105 с (≤ 180), сняты через 30 и 135 с (≤ 240) |
| «Ответы самого балансировщика» | [database down]: готовых реплик 0, клиент получил 2497 ответов 503 от LB (без X-Instance-Id), 5xx на LB +2605, у API +67 |
| «Отказ БД виден в SLI доступности» | availability SLI over the outage window: LB 0.5557, client 0.5581 |
| «SLO считается» | [SLI and panels]: три SLI за 7 суток в [0, 1]; за час по рядам правил и по исходным счётчикам в тех же окнах — 0, 0 и 0,085 п. п. (≤ 1); у доступности и задержки и против increase одним окном часа — 0,58 и 0,066 п. п. (≤ 1) |
| «История оповещения на панели», «Новые панели после трафика» | alert history panel shows firing of the steps, new panel queries (31) with data: [] (у 31 запроса панелей с id от 25 данных хватает) |
«Без потерь» (TelemetryDataLoss) |
other alerts firing: [] на каждом шаге |
Лишних firing нет ни на одном шаге (other alerts firing: []); MessengerQueueMetricsMissing в
pending во время недоступной БД — допустимо (relay не делает снимок без PostgreSQL).
Вывод
alerts-check 2026-11-14T01:09:30Z: rules, quiet stand
rule groups loaded: ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rules with evaluation errors: []
alerts loaded: ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests', 'TelemetryDataLoss', 'TelemetryTargetDown']
Grafana shows rules of the Prometheus datasource: 200 ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rule change through Grafana rejected: yes
alert delivery addresses: []
prometheus config has alertmanagers: no
load: 6 writers every 0.25 s and 1 reader every 1.0 s
observed: pool used before the steps (max): 0
uploads ready before the steps: 20
[worker stopped] stand quiet before the step: yes
[worker stopped]
observed: queue by stage and age before: {'outbox': 0.0, 'processing': 0.0, 'published': 0.0, 'scheduled': 0.0, 'oldest_age_s': 0.0}
observed: worker stopped at 46 s of the drill
[worker stopped] uploads accepted (202): 3
observed: MessengerQueueLagHigh pending after 84 s
[worker stopped] MessengerQueueLagHigh firing after the first upload: after 144 s
observed: queue by stage and age: {'outbox': 0.0, 'processing': 0.0, 'published': 3.0, 'scheduled': 0.0, 'oldest_age_s': 131.9}
[worker stopped] outbox + published grew by the uploads: 3.0
[worker stopped] consumer spans after the stop: 0
observed: last worker record in Loki: 'Attachment became ready' 1 s before the stop
observed: 'Stopped worker' in stdout: yes
[worker stopped] worker series gone before the start: 0
observed: worker down for 145 s
[worker stopped] uploads ready after the start: 3
[worker stopped] MessengerQueueLagHigh resolved after the start: after 29 s
observed: time to ready count, first points of the new process: [(2.3, 0.0)]
[worker stopped] time to ready count grew by the uploads (by points): 3.0
[worker stopped] other alerts firing: []
[relay stopped] stand quiet before the step: yes
[relay stopped]
[relay stopped] queue depth and age series gone within 120 s: yes
observed: queue series gone after 60 s
observed: MessengerQueueMetricsMissing pending after 69 s
[relay stopped] MessengerQueueMetricsMissing firing: after 190 s
[relay stopped] MessengerQueueMetricsMissing resolved after the start: after 45 s
[relay stopped] other alerts firing: []
[slow database] stand quiet before the step: yes
[slow database]
observed: MessengerSlowRequests pending after 75 s
[slow database] MessengerSlowRequests firing: after 135 s
observed: slow light requests share: 0.427
observed: p95 by route before: {'/api/v1/chats/{chat_id}/messages': 0.016}
observed: p95 by route during: {'/api/v1/chats/{chat_id}/messages': 2.324}
observed: PostgreSQL share of slow send traces: [1.0, 0.99, 1.0, 1.0, 1.0]
[slow database] PostgreSQL spans over half of slow send traces (median): yes
observed: pool used during (max): 6, before 0
[slow database] pool busier than before: yes
observed: CPU of replicas to limit (max): 0.09
[slow database] CPU of replicas below half of the limit: yes
[slow database] MessengerSlowRequests resolved after the unlock: after 74 s
[slow database] other alerts firing: []
[slow handler] stand quiet before the step: yes
[slow handler]
observed: logins: 515, median 0.25 s, max 0.73 s
observed: PostgreSQL share of slow login traces: [0.0, 0.2, 0.01, 0.0, 0.01]
[slow handler] PostgreSQL spans under a fifth of slow login traces (median): yes
observed: pool: max share used 0.20, requests in checkout (max) 0, timeouts 0
[slow handler] no request waits for a free pool connection: yes
observed: CPU of replicas to limit (max): 0.81
observed: p95 by route: {'/api/v1/chats/{chat_id}/messages': 0.085, '/api/v1/auth/sessions': 0.598}
[slow handler] login p95 on the route panel slower than 0.25 s: yes
observed: slow light requests share (max): 0.072
[slow handler] MessengerSlowRequests stayed inactive: yes
[slow handler] other alerts firing: []
[database down] stand quiet before the step: yes
[database down]
observed: MessengerNoReadyReplicas pending after 29 s
[database down] MessengerNoReadyReplicas firing: after 59 s
observed: MessengerHighErrorRate pending after 44 s
[database down] MessengerHighErrorRate firing: after 105 s
[database down] ready replicas in LB: 0
observed: 'Database is unavailable' records of replicas in Loki: 67
observed: API traces with an error server span: 50, with a PostgreSQL span in error: 0
[database down] LB ready after the start: yes
observed: client responses during the outage: {'5xx': 2556}, from LB: 2497
[database down] MessengerNoReadyReplicas resolved: after 30 s
[database down] MessengerHighErrorRate resolved: after 135 s
observed: 5xx grew: LB 2605, API 67; LB responses 5863
[database down] 5xx on LB more than 5xx of API: yes
[database down] LB 5xx grew by the LB's own 503 to the client or more: yes
observed: availability SLI over the outage window: LB 0.5557, client 0.5581
[database down] LB 5xx share equals the client's within 5%: yes
observed: MessengerQueueMetricsMissing was pending (allowed)
[database down] other alerts firing: []
[SLI and panels]
observed: client responses over the drill: {'2xx': 20911, '5xx': 2602}
observed: messenger-slo windows in the last hour: 10 of 12
SLI availability over 7 days within [0, 1]: yes
observed: SLI availability: 7 days 0.93726, hour by rules 0.96010, by raw counters in the same windows 0.96010 (58645 events), raw increase over the hour 0.96590
SLI availability over the hour, rules vs raw counters within 1 pp: yes
SLI availability over the hour, rules vs raw increase over one window within 1 pp: yes
observed: SLI availability over the drill: 0.88927
SLI latency over 7 days within [0, 1]: yes
observed: SLI latency: 7 days 0.98652, hour by rules 0.98658, by raw counters in the same windows 0.98658 (55462 events), raw increase over the hour 0.98724
SLI latency over the hour, rules vs raw counters within 1 pp: yes
SLI latency over the hour, rules vs raw increase over one window within 1 pp: yes
observed: SLI latency over the drill: 0.98180
SLI time to ready over 7 days within [0, 1]: yes
observed: SLI time to ready: 7 days 0.93162, hour by rules 0.93586, by raw counters in the same windows 0.93501 (49 events), raw increase over the hour 0.88000
SLI time to ready over the hour, rules vs raw counters within 1 pp: yes
observed: SLI time to ready over the drill: 0.86957
new panel queries (31) with data: []
alert history panel shows firing of the steps: ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests']
memory prometheus ad08a84f4d3a: peak 128 MiB, anon+kernel+shmem 82 MiB, file 16 MiB, limit 384 MiB
memory otel-collector 7209c9b86a7c: peak 83 MiB, anon+kernel+shmem 72 MiB, file 0 MiB, limit 256 MiB
rule groups loaded expected ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts'] got ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rules with evaluation errors expected [] got []
alerts loaded expected ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests', 'TelemetryDataLoss', 'TelemetryTargetDown'] got ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests', 'TelemetryDataLoss', 'TelemetryTargetDown']
Grafana shows rules of the Prometheus datasource expected 200 ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts'] got 200 ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rule change through Grafana rejected expected yes got yes
alert delivery addresses expected [] got []
prometheus config has alertmanagers expected no got no
uploads ready before the steps expected 20 got 20
[worker stopped] stand quiet before the step expected yes got yes
[worker stopped] uploads accepted (202) expected 3 got 3
[worker stopped] MessengerQueueLagHigh firing after the first upload within 240 s expected yes got yes
[worker stopped] outbox + published grew by the uploads expected 3.0 got 3.0
[worker stopped] consumer spans after the stop expected 0 got 0
[worker stopped] worker series gone before the start expected 0 got 0
[worker stopped] uploads ready after the start expected 3 got 3
[worker stopped] MessengerQueueLagHigh resolved after the start within 120 s expected yes got yes
[worker stopped] time to ready count grew by the uploads (by points) expected 3.0 got 3.0
[worker stopped] other alerts firing expected [] got []
[relay stopped] stand quiet before the step expected yes got yes
[relay stopped] queue depth and age series gone within 120 s expected yes got yes
[relay stopped] MessengerQueueMetricsMissing firing within 300 s expected yes got yes
[relay stopped] MessengerQueueMetricsMissing resolved after the start within 120 s expected yes got yes
[relay stopped] other alerts firing expected [] got []
[slow database] stand quiet before the step expected yes got yes
[slow database] MessengerSlowRequests firing within 240 s expected yes got yes
[slow database] PostgreSQL spans over half of slow send traces (median) expected yes got yes
[slow database] pool busier than before expected yes got yes
[slow database] CPU of replicas below half of the limit expected yes got yes
[slow database] MessengerSlowRequests resolved after the unlock within 240 s expected yes got yes
[slow database] other alerts firing expected [] got []
[slow handler] stand quiet before the step expected yes got yes
[slow handler] PostgreSQL spans under a fifth of slow login traces (median) expected yes got yes
[slow handler] no request waits for a free pool connection expected yes got yes
[slow handler] login p95 on the route panel slower than 0.25 s expected yes got yes
[slow handler] MessengerSlowRequests stayed inactive expected yes got yes
[slow handler] other alerts firing expected [] got []
[database down] stand quiet before the step expected yes got yes
[database down] MessengerNoReadyReplicas firing within 180 s expected yes got yes
[database down] MessengerHighErrorRate firing within 180 s expected yes got yes
[database down] ready replicas in LB expected 0 got 0
[database down] LB ready after the start expected yes got yes
[database down] MessengerNoReadyReplicas resolved within 240 s expected yes got yes
[database down] MessengerHighErrorRate resolved within 240 s expected yes got yes
[database down] 5xx on LB more than 5xx of API expected yes got yes
[database down] LB 5xx grew by the LB's own 503 to the client or more expected yes got yes
[database down] LB 5xx share equals the client's within 5% expected yes got yes
[database down] other alerts firing expected [] got []
SLI availability over 7 days within [0, 1] expected yes got yes
SLI availability over the hour, rules vs raw counters within 1 pp expected yes got yes
SLI availability over the hour, rules vs raw increase over one window within 1 pp expected yes got yes
SLI latency over 7 days within [0, 1] expected yes got yes
SLI latency over the hour, rules vs raw counters within 1 pp expected yes got yes
SLI latency over the hour, rules vs raw increase over one window within 1 pp expected yes got yes
SLI time to ready over 7 days within [0, 1] expected yes got yes
SLI time to ready over the hour, rules vs raw counters within 1 pp expected yes got yes
new panel queries (31) with data expected [] got []
alert history panel shows firing of the steps expected ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests'] got ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests']
PASS: each alert went firing on its fault and resolved after the recovery, no other alert fired; SLI agree by both paths
restored: load stopped, lock session ended, PostgreSQL started, worker and relay running: yes, LB ready: yes
Чтение чисел
- SLI за час — по последней точке ряда
messenger-slo(Prometheus пишет точку с временем предыдущего слота, design, «Уточнения apply», п. 11): окон за час 10 из 12 — Prometheus запущен в 00:42Z (шаг[prometheus]пробногоbackend-outageперед съёмкой), раньше рядов правил за этот час нет. В записанные окна попали пробные учения до съёмки и этот прогон, кроме последних 5–10 мин: доступность за час 0,96010. Окно учений целиком — строкиSLI … over the drillпо исходным счётчикам: доступность 0,88927 (отказ БД), задержка 0,98180 (медленная БД), время доready0,86957 (три загрузки при остановленном worker ждали дольше 30 с). - Сверка путей (спек «SLO считается», design, «Уточнения apply», пп. 12, 18): ряды правил против
суммы
increase(…[5m])исходных счётчиков в тех же окнах — доступность и задержка 0 п. п., время доready0,085 п. п. (0,93586 против 0,93501 при 49 событиях); у доступности и задержки ещё и противincreaseодним окном часа — 0,58 п. п. (0,96010 против 0,96590) и 0,066 п. п. (0,98658 против 0,98724), допуск 1 п. п. У времени доreadyодно окно (0,88000) — наблюдение: при единицах событий потеря приращения на границах непересекающихся окон меняет SLI на п. п. - SLI за 7 суток включает все учения и проверки этого и прошлых прогонов на рабочем стенде
(доступность 0,93726 при SLO 0,995 — бюджет исчерпан учениями; остаток бюджета на панели
отрицательный), это ожидаемо и названо в
REPORT.md(«Ограничения SLI»). - Span PostgreSQL с ошибкой при недоступной БД — 0 из 50 найденных трасс с ошибкой server span
(предел поиска — 50): отказ — на выдаче соединения из пула, до
before_cursor_execute(design, «Уточнения apply», п. 9; наблюдение, а не проверка). Причину называют записи «Database is unavailable» в Loki (67 — по одной на ответ 503 реплик до вывода из LB, у API +67 ответов 5xx). - Память Prometheus и Collector —
memory.peakс их запуска в 00:42Z и 00:44Z (пробный прогонalerts-checkи этот): Prometheus 128 МиБ (неосвобождаемая часть 82), Collector 83 МиБ при лимитах 384 и 256 — правило «лимит ≥ 1,5 × пика» выполнено (раздел changeslo-alertsвevidence/telemetry/budget.md).