# incident-telemetry: lab6: Tempo+Loki down -> API unaffected, collector memory bounded, losses counted
# date: 2026-11-07 19:27:09, commit: 20afc2c + uncommitted changes
# command: python.exe student-tests/incidents.py telemetry-down
# exit code: 0, 284.7 s

=== telemetry-down
[  22.2s] baseline: {'requests': 793, 'codes': {'201': 793}, 'success_ratio': 1.0, 'success_rps': 52.9, 'p95_ms_successful': 32.0}, collector heap 32.3 MiB, container 76.9 MiB
[  53.5s] docker compose stop tempo loki: stopped
[  55.6s]   {'t': 0.0, 'collector_heap_mib': 51.5, 'collector_rss_mib': 237.8, 'collector_container_mib': 76.9, 'queue': {'otlp_http/loki': 16.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 7.0}, 'alert': 'inactive'}
[  68.3s]   {'t': 12.1, 'collector_heap_mib': 60.3, 'collector_rss_mib': 237.8, 'collector_container_mib': 76.9, 'queue': {'otlp_http/loki': 16.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 7.0}, 'alert': 'inactive'}
[  80.4s]   {'t': 24.8, 'collector_heap_mib': 44.3, 'collector_rss_mib': 237.8, 'collector_container_mib': 76.9, 'queue': {'otlp_http/loki': 23.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 10.0}, 'alert': 'pending'}
[  92.4s]   {'t': 36.9, 'collector_heap_mib': 53.4, 'collector_rss_mib': 237.8, 'collector_container_mib': 77.0, 'queue': {'otlp_http/loki': 31.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 13.0}, 'alert': 'pending'}
[ 104.5s]   {'t': 49.0, 'collector_heap_mib': 62.1, 'collector_rss_mib': 239.8, 'collector_container_mib': 81.0, 'queue': {'otlp_http/loki': 38.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 16.0}, 'alert': 'pending'}
[ 117.3s]   {'t': 61.1, 'collector_heap_mib': 62.1, 'collector_rss_mib': 239.8, 'collector_container_mib': 87.1, 'queue': {'otlp_http/loki': 46.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 19.0}, 'alert': 'pending'}
[ 129.4s]   {'t': 73.8, 'collector_heap_mib': 45.9, 'collector_rss_mib': 247.8, 'collector_container_mib': 87.1, 'queue': {'otlp_http/loki': 46.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 19.0}, 'alert': 'pending'}
[ 141.4s]   {'t': 85.9, 'collector_heap_mib': 54.5, 'collector_rss_mib': 247.8, 'collector_container_mib': 87.2, 'queue': {'otlp_http/loki': 53.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 22.0}, 'alert': 'firing'}
[ 153.5s]   {'t': 98.0, 'collector_heap_mib': 66.1, 'collector_rss_mib': 248.0, 'collector_container_mib': 87.3, 'queue': {'otlp_http/loki': 59.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 24.0}, 'alert': 'firing'}
[ 165.6s]   {'t': 110.0, 'collector_heap_mib': 75.7, 'collector_rss_mib': 254.0, 'collector_container_mib': 95.4, 'queue': {'otlp_http/loki': 66.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 27.0}, 'alert': 'firing'}
[ 178.4s]   {'t': 122.1, 'collector_heap_mib': 84.7, 'collector_rss_mib': 262.2, 'collector_container_mib': 101.5, 'queue': {'otlp_http/loki': 74.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 30.0}, 'alert': 'firing'}
[ 190.4s]   {'t': 134.9, 'collector_heap_mib': 84.7, 'collector_rss_mib': 262.2, 'collector_container_mib': 103.7, 'queue': {'otlp_http/loki': 81.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 33.0}, 'alert': 'firing'}
[ 202.5s]   {'t': 147.0, 'collector_heap_mib': 64.8, 'collector_rss_mib': 264.2, 'collector_container_mib': 103.6, 'queue': {'otlp_http/loki': 81.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 33.0}, 'alert': 'firing'}
[ 214.0s] docker compose start tempo loki
[ 269.3s] exporter queues drained 55.3 s after start
[ 284.6s] API during outage: {'requests': 8486, 'codes': {'201': 8486}, 'success_ratio': 1.0, 'success_rps': 52.9, 'p95_ms_successful': 32.0}
[ 284.6s] probe traces: 13/13 reached Tempo (lost: sent at [] s)
[ 284.6s] lost in Collector (send_failed+enqueue_failed): {'spans': 2048, 'log_records': 213}; collector heap 32.3 -> max 84.7 MiB (memory_limiter 200 MiB), container max 103.7 MiB
[ 284.6s] saved C:\Users\Gigabyte\Desktop\lab7\mtusi2026lab7\evidence\incident-telemetry-down.json
PASS: Tempo+Loki down -> API unaffected, collector memory bounded, losses counted; recovery -> resolved
1 passed, 0 failed
