Files
mtusi2026lab6/evidence/alerts/check.md
T

24 KiB
Raw Blame History

driver.py --port 8080 alerts-check (change slo-alerts, задачи 11.2 и 12.11)

  • Основа: commit ef60b47 (fix: address slo-alerts verify findings), рабочее дерево чистое (git status --short до прогона пуст). Файлы src/ и migrations/ (151) в контейнерах api (2: api-13, api-14), relay, worker и cleaner побайтно совпадают с деревом commit (SHA-256 каждого файла); ревизия схемы e4b8c2d17f90. /etc/prometheus/rules.yml и prometheus.yml в контейнере Prometheus побайтно равны файлам дерева ef60b47 (cmp). Код приложения с 15ea156 не менялся: ef60b47 правит правила, их тесты, драйвер и документы. Прежний прогон этого файла — на 15ea156 (до правок verify).
  • Дата: 2026-11-14, 01:09:30Z – 01:27:20Z (~17,8 мин); хост macOS, Docker 29.9.2, Compose 5.6.1, ВМ Docker Desktop — 12 CPU / 11,6 ГиБ. Рабочий стенд mtusi-lab6 (две реплики API, relay, worker, уборщик, Collector, Prometheus, Loki, Tempo, Grafana), команда из корня репозитория: python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 alerts-check, код выхода 0.
  • Всё — через HTTP API Grafana на 127.0.0.1:3000 с паролем из .env (источники-прокси uid prometheus, loki, tempo); моменты pending / firing / снятия — /api/v1/rules раз в 5 с, время — от начала шага (у MessengerQueueLagHigh — от первой загрузки, у снятия — от восстановления); момент pending печатается перед каждой строкой firing. Пользователи, пароли, токены и файлы — синтетические, в вывод не попадают. Фоновая нагрузка драйвера через LB — 6 потоков отправки сообщения с новым Idempotency-Key раз в 0,25 с и 1 поток чтения истории раз в 1 с (design, «Уточнения apply», п. 7).
  • Отказы: шаг 1 — docker compose stop worker, 3 загрузки, docker start того же контейнера через 145 с; шаг 2 — docker compose stop relay, docker start; шаг 3 — сессия psql с application_name = mtusi-alerts-check циклами держит LOCK TABLE messages IN SHARE MODE 2 с с паузой 0,2 с, затем pg_terminate_backend; шаг 4 — 3 потока входа по Basic 45 с (сравнение, отказа нет); шаг 5 — docker compose stop postgres, затем start. В finally — нагрузка остановлена, сессия блокировки завершена, PostgreSQL, worker и relay запущены, LB готов (последняя строка).

Сценарии → строки вывода

Сценарий (observability, если не указано) Строки
«Правила загружены» rule groups loaded, rules with evaluation errors, alerts loaded, Grafana shows rules…, rule change through Grafana rejected, alert delivery addresses, prometheus config has alertmanagers
«Остановленный worker» [worker stopped]: pending через 84 с, firing через 144 с после первой загрузки (≤ 240), outbox + published +3, снято через 29 с после запуска (≤ 120), 3 ready
«Перезапуск worker» [worker stopped] time to ready count…: первая точка нового процесса — 0 (время начала), приращение по точкам ровно 3; worker стоял 145 с (> 2 мин)
«Остановленный relay» [relay stopped] MessengerQueueMetricsMissing: pending через 69 с, firing через 190 с (≤ 300), снято через 45 с (≤ 120)
«Остановленный relay уходит из рядов» [relay stopped] queue depth and age series gone within 120 s — ряды глубины и возраста ушли через 60 с
«Медленная БД» [slow database]: pending через 75 с, firing через 135 с (≤ 240), доля span PostgreSQL в медленных трассах отправки 0,99–1,0, пул 6 против 0, CPU реплик 0,09 предела, снято через 74 с (≤ 240), MessengerHighErrorRate нет
«Медленный handler не путается с БД» [slow handler]: доля span PostgreSQL 0,0–0,2 (медиана 0,01), пул ни одной реплики не занят целиком (не больше 0,20), отказов по сроку 0, pending — наблюдение (0), CPU реплик 0,81 предела, MessengerSlowRequests неактивно (доля медленных до 0,072)
«Тяжёлая ручка вне SLI задержки» [slow handler] login p95 on the route panel slower than 0.25 s (p95 входа 0,598 с) при неактивном MessengerSlowRequests
«Недоступная БД» [database down]: MessengerNoReadyReplicas pending через 29 с, firing через 59 с, MessengerHighErrorRate pending через 44 с, firing через 105 с (≤ 180), сняты через 30 и 135 с (≤ 240)
«Ответы самого балансировщика» [database down]: готовых реплик 0, клиент получил 2497 ответов 503 от LB (без X-Instance-Id), 5xx на LB +2605, у API +67
«Отказ БД виден в SLI доступности» availability SLI over the outage window: LB 0.5557, client 0.5581
«SLO считается» [SLI and panels]: три SLI за 7 суток в [0, 1]; за час по рядам правил и по исходным счётчикам в тех же окнах — 0, 0 и 0,085 п. п. (≤ 1); у доступности и задержки и против increase одним окном часа — 0,58 и 0,066 п. п. (≤ 1)
«История оповещения на панели», «Новые панели после трафика» alert history panel shows firing of the steps, new panel queries (31) with data: [] (у 31 запроса панелей с id от 25 данных хватает)
«Без потерь» (TelemetryDataLoss) other alerts firing: [] на каждом шаге

Лишних firing нет ни на одном шаге (other alerts firing: []); MessengerQueueMetricsMissing в pending во время недоступной БД — допустимо (relay не делает снимок без PostgreSQL).

Вывод

alerts-check 2026-11-14T01:09:30Z: rules, quiet stand
rule groups loaded: ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rules with evaluation errors: []
alerts loaded: ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests', 'TelemetryDataLoss', 'TelemetryTargetDown']
Grafana shows rules of the Prometheus datasource: 200 ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rule change through Grafana rejected: yes
alert delivery addresses: []
prometheus config has alertmanagers: no
load: 6 writers every 0.25 s and 1 reader every 1.0 s
  observed: pool used before the steps (max): 0
uploads ready before the steps: 20
[worker stopped] stand quiet before the step: yes
[worker stopped]
  observed: queue by stage and age before: {'outbox': 0.0, 'processing': 0.0, 'published': 0.0, 'scheduled': 0.0, 'oldest_age_s': 0.0}
  observed: worker stopped at 46 s of the drill
[worker stopped] uploads accepted (202): 3
  observed: MessengerQueueLagHigh pending after 84 s
  [worker stopped] MessengerQueueLagHigh firing after the first upload: after 144 s
  observed: queue by stage and age: {'outbox': 0.0, 'processing': 0.0, 'published': 3.0, 'scheduled': 0.0, 'oldest_age_s': 131.9}
[worker stopped] outbox + published grew by the uploads: 3.0
[worker stopped] consumer spans after the stop: 0
  observed: last worker record in Loki: 'Attachment became ready' 1 s before the stop
  observed: 'Stopped worker' in stdout: yes
[worker stopped] worker series gone before the start: 0
  observed: worker down for 145 s
[worker stopped] uploads ready after the start: 3
  [worker stopped] MessengerQueueLagHigh resolved after the start: after 29 s
  observed: time to ready count, first points of the new process: [(2.3, 0.0)]
[worker stopped] time to ready count grew by the uploads (by points): 3.0
[worker stopped] other alerts firing: []
[relay stopped] stand quiet before the step: yes
[relay stopped]
[relay stopped] queue depth and age series gone within 120 s: yes
  observed: queue series gone after 60 s
  observed: MessengerQueueMetricsMissing pending after 69 s
  [relay stopped] MessengerQueueMetricsMissing firing: after 190 s
  [relay stopped] MessengerQueueMetricsMissing resolved after the start: after 45 s
[relay stopped] other alerts firing: []
[slow database] stand quiet before the step: yes
[slow database]
  observed: MessengerSlowRequests pending after 75 s
  [slow database] MessengerSlowRequests firing: after 135 s
  observed: slow light requests share: 0.427
  observed: p95 by route before: {'/api/v1/chats/{chat_id}/messages': 0.016}
  observed: p95 by route during: {'/api/v1/chats/{chat_id}/messages': 2.324}
  observed: PostgreSQL share of slow send traces: [1.0, 0.99, 1.0, 1.0, 1.0]
[slow database] PostgreSQL spans over half of slow send traces (median): yes
  observed: pool used during (max): 6, before 0
[slow database] pool busier than before: yes
  observed: CPU of replicas to limit (max): 0.09
[slow database] CPU of replicas below half of the limit: yes
  [slow database] MessengerSlowRequests resolved after the unlock: after 74 s
[slow database] other alerts firing: []
[slow handler] stand quiet before the step: yes
[slow handler]
  observed: logins: 515, median 0.25 s, max 0.73 s
  observed: PostgreSQL share of slow login traces: [0.0, 0.2, 0.01, 0.0, 0.01]
[slow handler] PostgreSQL spans under a fifth of slow login traces (median): yes
  observed: pool: max share used 0.20, requests in checkout (max) 0, timeouts 0
[slow handler] no request waits for a free pool connection: yes
  observed: CPU of replicas to limit (max): 0.81
  observed: p95 by route: {'/api/v1/chats/{chat_id}/messages': 0.085, '/api/v1/auth/sessions': 0.598}
[slow handler] login p95 on the route panel slower than 0.25 s: yes
  observed: slow light requests share (max): 0.072
[slow handler] MessengerSlowRequests stayed inactive: yes
[slow handler] other alerts firing: []
[database down] stand quiet before the step: yes
[database down]
  observed: MessengerNoReadyReplicas pending after 29 s
  [database down] MessengerNoReadyReplicas firing: after 59 s
  observed: MessengerHighErrorRate pending after 44 s
  [database down] MessengerHighErrorRate firing: after 105 s
[database down] ready replicas in LB: 0
  observed: 'Database is unavailable' records of replicas in Loki: 67
  observed: API traces with an error server span: 50, with a PostgreSQL span in error: 0
[database down] LB ready after the start: yes
  observed: client responses during the outage: {'5xx': 2556}, from LB: 2497
  [database down] MessengerNoReadyReplicas resolved: after 30 s
  [database down] MessengerHighErrorRate resolved: after 135 s
  observed: 5xx grew: LB 2605, API 67; LB responses 5863
[database down] 5xx on LB more than 5xx of API: yes
[database down] LB 5xx grew by the LB's own 503 to the client or more: yes
  observed: availability SLI over the outage window: LB 0.5557, client 0.5581
[database down] LB 5xx share equals the client's within 5%: yes
  observed: MessengerQueueMetricsMissing was pending (allowed)
[database down] other alerts firing: []
[SLI and panels]
  observed: client responses over the drill: {'2xx': 20911, '5xx': 2602}
  observed: messenger-slo windows in the last hour: 10 of 12
SLI availability over 7 days within [0, 1]: yes
  observed: SLI availability: 7 days 0.93726, hour by rules 0.96010, by raw counters in the same windows 0.96010 (58645 events), raw increase over the hour 0.96590
SLI availability over the hour, rules vs raw counters within 1 pp: yes
SLI availability over the hour, rules vs raw increase over one window within 1 pp: yes
  observed: SLI availability over the drill: 0.88927
SLI latency over 7 days within [0, 1]: yes
  observed: SLI latency: 7 days 0.98652, hour by rules 0.98658, by raw counters in the same windows 0.98658 (55462 events), raw increase over the hour 0.98724
SLI latency over the hour, rules vs raw counters within 1 pp: yes
SLI latency over the hour, rules vs raw increase over one window within 1 pp: yes
  observed: SLI latency over the drill: 0.98180
SLI time to ready over 7 days within [0, 1]: yes
  observed: SLI time to ready: 7 days 0.93162, hour by rules 0.93586, by raw counters in the same windows 0.93501 (49 events), raw increase over the hour 0.88000
SLI time to ready over the hour, rules vs raw counters within 1 pp: yes
  observed: SLI time to ready over the drill: 0.86957
new panel queries (31) with data: []
alert history panel shows firing of the steps: ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests']
memory prometheus ad08a84f4d3a: peak 128 MiB, anon+kernel+shmem 82 MiB, file 16 MiB, limit 384 MiB
memory otel-collector 7209c9b86a7c: peak 83 MiB, anon+kernel+shmem 72 MiB, file 0 MiB, limit 256 MiB
rule groups loaded                                                                 expected ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts'] got ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rules with evaluation errors                                                       expected []              got []
alerts loaded                                                                      expected ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests', 'TelemetryDataLoss', 'TelemetryTargetDown'] got ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests', 'TelemetryDataLoss', 'TelemetryTargetDown']
Grafana shows rules of the Prometheus datasource                                   expected 200 ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts'] got 200 ['messenger-alerts', 'messenger-sli', 'messenger-slo', 'telemetry-alerts']
rule change through Grafana rejected                                               expected yes             got yes
alert delivery addresses                                                           expected []              got []
prometheus config has alertmanagers                                                expected no              got no
uploads ready before the steps                                                     expected 20              got 20
[worker stopped] stand quiet before the step                                       expected yes             got yes
[worker stopped] uploads accepted (202)                                            expected 3               got 3
[worker stopped] MessengerQueueLagHigh firing after the first upload within 240 s  expected yes             got yes
[worker stopped] outbox + published grew by the uploads                            expected 3.0             got 3.0
[worker stopped] consumer spans after the stop                                     expected 0               got 0
[worker stopped] worker series gone before the start                               expected 0               got 0
[worker stopped] uploads ready after the start                                     expected 3               got 3
[worker stopped] MessengerQueueLagHigh resolved after the start within 120 s       expected yes             got yes
[worker stopped] time to ready count grew by the uploads (by points)               expected 3.0             got 3.0
[worker stopped] other alerts firing                                               expected []              got []
[relay stopped] stand quiet before the step                                        expected yes             got yes
[relay stopped] queue depth and age series gone within 120 s                       expected yes             got yes
[relay stopped] MessengerQueueMetricsMissing firing within 300 s                   expected yes             got yes
[relay stopped] MessengerQueueMetricsMissing resolved after the start within 120 s expected yes             got yes
[relay stopped] other alerts firing                                                expected []              got []
[slow database] stand quiet before the step                                        expected yes             got yes
[slow database] MessengerSlowRequests firing within 240 s                          expected yes             got yes
[slow database] PostgreSQL spans over half of slow send traces (median)            expected yes             got yes
[slow database] pool busier than before                                            expected yes             got yes
[slow database] CPU of replicas below half of the limit                            expected yes             got yes
[slow database] MessengerSlowRequests resolved after the unlock within 240 s       expected yes             got yes
[slow database] other alerts firing                                                expected []              got []
[slow handler] stand quiet before the step                                         expected yes             got yes
[slow handler] PostgreSQL spans under a fifth of slow login traces (median)        expected yes             got yes
[slow handler] no request waits for a free pool connection                         expected yes             got yes
[slow handler] login p95 on the route panel slower than 0.25 s                     expected yes             got yes
[slow handler] MessengerSlowRequests stayed inactive                               expected yes             got yes
[slow handler] other alerts firing                                                 expected []              got []
[database down] stand quiet before the step                                        expected yes             got yes
[database down] MessengerNoReadyReplicas firing within 180 s                       expected yes             got yes
[database down] MessengerHighErrorRate firing within 180 s                         expected yes             got yes
[database down] ready replicas in LB                                               expected 0               got 0
[database down] LB ready after the start                                           expected yes             got yes
[database down] MessengerNoReadyReplicas resolved within 240 s                     expected yes             got yes
[database down] MessengerHighErrorRate resolved within 240 s                       expected yes             got yes
[database down] 5xx on LB more than 5xx of API                                     expected yes             got yes
[database down] LB 5xx grew by the LB's own 503 to the client or more              expected yes             got yes
[database down] LB 5xx share equals the client's within 5%                         expected yes             got yes
[database down] other alerts firing                                                expected []              got []
SLI availability over 7 days within [0, 1]                                         expected yes             got yes
SLI availability over the hour, rules vs raw counters within 1 pp                  expected yes             got yes
SLI availability over the hour, rules vs raw increase over one window within 1 pp  expected yes             got yes
SLI latency over 7 days within [0, 1]                                              expected yes             got yes
SLI latency over the hour, rules vs raw counters within 1 pp                       expected yes             got yes
SLI latency over the hour, rules vs raw increase over one window within 1 pp       expected yes             got yes
SLI time to ready over 7 days within [0, 1]                                        expected yes             got yes
SLI time to ready over the hour, rules vs raw counters within 1 pp                 expected yes             got yes
new panel queries (31) with data                                                   expected []              got []
alert history panel shows firing of the steps                                      expected ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests'] got ['MessengerHighErrorRate', 'MessengerNoReadyReplicas', 'MessengerQueueLagHigh', 'MessengerQueueMetricsMissing', 'MessengerSlowRequests']
PASS: each alert went firing on its fault and resolved after the recovery, no other alert fired; SLI agree by both paths
restored: load stopped, lock session ended, PostgreSQL started, worker and relay running: yes, LB ready: yes

Чтение чисел

  • SLI за час — по последней точке ряда messenger-slo (Prometheus пишет точку с временем предыдущего слота, design, «Уточнения apply», п. 11): окон за час 10 из 12 — Prometheus запущен в 00:42Z (шаг [prometheus] пробного backend-outage перед съёмкой), раньше рядов правил за этот час нет. В записанные окна попали пробные учения до съёмки и этот прогон, кроме последних 5–10 мин: доступность за час 0,96010. Окно учений целиком — строки SLI … over the drill по исходным счётчикам: доступность 0,88927 (отказ БД), задержка 0,98180 (медленная БД), время до ready 0,86957 (три загрузки при остановленном worker ждали дольше 30 с).
  • Сверка путей (спек «SLO считается», design, «Уточнения apply», пп. 12, 18): ряды правил против суммы increase(…[5m]) исходных счётчиков в тех же окнах — доступность и задержка 0 п. п., время до ready 0,085 п. п. (0,93586 против 0,93501 при 49 событиях); у доступности и задержки ещё и против increase одним окном часа — 0,58 п. п. (0,96010 против 0,96590) и 0,066 п. п. (0,98658 против 0,98724), допуск 1 п. п. У времени до ready одно окно (0,88000) — наблюдение: при единицах событий потеря приращения на границах непересекающихся окон меняет SLI на п. п.
  • SLI за 7 суток включает все учения и проверки этого и прошлых прогонов на рабочем стенде (доступность 0,93726 при SLO 0,995 — бюджет исчерпан учениями; остаток бюджета на панели отрицательный), это ожидаемо и названо в REPORT.md («Ограничения SLI»).
  • Span PostgreSQL с ошибкой при недоступной БД — 0 из 50 найденных трасс с ошибкой server span (предел поиска — 50): отказ — на выдаче соединения из пула, до before_cursor_execute (design, «Уточнения apply», п. 9; наблюдение, а не проверка). Причину называют записи «Database is unavailable» в Loki (67 — по одной на ответ 503 реплик до вывода из LB, у API +67 ответов 5xx).
  • Память Prometheus и Collector — memory.peak с их запуска в 00:42Z и 00:44Z (пробный прогон alerts-check и этот): Prometheus 128 МиБ (неосвобождаемая часть 82), Collector 83 МиБ при лимитах 384 и 256 — правило «лимит ≥ 1,5 × пика» выполнено (раздел change slo-alerts в evidence/telemetry/budget.md).