# incident-telemetry: lab6: Tempo+Loki down -> API unaffected, collector memory bounded, losses counted
# date: 2026-11-01 00:39:14, commit: unknown
# command: python student-tests/incidents.py telemetry-down
# exit code: 0, 374.8 s

=== telemetry-down
[  20.2s] baseline: {'requests': 955, 'codes': {'201': 955}, 'success_ratio': 1.0, 'success_rps': 63.7, 'p95_ms_successful': 17.9}, collector heap 44.1 MiB, container 224.4 MiB
[  50.9s] docker compose stop tempo loki: stopped
[  51.1s]   {'t': 0.0, 'collector_heap_mib': 39.8, 'collector_rss_mib': 228.5, 'collector_container_mib': 228.6, 'queue': {'otlp_http/loki': 0.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 6.0}, 'alert': 'inactive'}
[  61.2s]   {'t': 10.1, 'collector_heap_mib': 52.2, 'collector_rss_mib': 228.6, 'collector_container_mib': 228.6, 'queue': {'otlp_http/loki': 5.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 10.0}, 'alert': 'inactive'}
[  71.3s]   {'t': 20.3, 'collector_heap_mib': 52.2, 'collector_rss_mib': 228.6, 'collector_container_mib': 228.7, 'queue': {'otlp_http/loki': 5.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 10.0}, 'alert': 'inactive'}
[  81.5s]   {'t': 30.4, 'collector_heap_mib': 63.3, 'collector_rss_mib': 230.2, 'collector_container_mib': 233.6, 'queue': {'otlp_http/loki': 12.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 14.0}, 'alert': 'inactive'}
[  91.6s]   {'t': 40.6, 'collector_heap_mib': 48.5, 'collector_rss_mib': 234.5, 'collector_container_mib': 234.5, 'queue': {'otlp_http/loki': 20.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 18.0}, 'alert': 'pending'}
[ 101.8s]   {'t': 50.7, 'collector_heap_mib': 48.5, 'collector_rss_mib': 234.5, 'collector_container_mib': 234.6, 'queue': {'otlp_http/loki': 20.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 18.0}, 'alert': 'pending'}
[ 112.0s]   {'t': 60.8, 'collector_heap_mib': 59.5, 'collector_rss_mib': 234.6, 'collector_container_mib': 234.9, 'queue': {'otlp_http/loki': 27.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 22.0}, 'alert': 'pending'}
[ 122.1s]   {'t': 71.0, 'collector_heap_mib': 70.7, 'collector_rss_mib': 239.4, 'collector_container_mib': 242.2, 'queue': {'otlp_http/loki': 35.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 26.0}, 'alert': 'pending'}
[ 132.2s]   {'t': 81.2, 'collector_heap_mib': 70.7, 'collector_rss_mib': 239.4, 'collector_container_mib': 247.4, 'queue': {'otlp_http/loki': 35.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 26.0}, 'alert': 'pending'}
[ 142.4s]   {'t': 91.3, 'collector_heap_mib': 58.1, 'collector_rss_mib': 247.4, 'collector_container_mib': 247.4, 'queue': {'otlp_http/loki': 42.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 30.0}, 'alert': 'pending'}
[ 152.5s]   {'t': 101.4, 'collector_heap_mib': 69.3, 'collector_rss_mib': 247.5, 'collector_container_mib': 247.5, 'queue': {'otlp_http/loki': 50.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 32.0}, 'alert': 'firing'}
[ 162.7s]   {'t': 111.6, 'collector_heap_mib': 69.3, 'collector_rss_mib': 247.5, 'collector_container_mib': 249.7, 'queue': {'otlp_http/loki': 50.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 32.0}, 'alert': 'firing'}
[ 172.9s]   {'t': 121.8, 'collector_heap_mib': 80.2, 'collector_rss_mib': 249.9, 'collector_container_mib': 258.8, 'queue': {'otlp_http/loki': 57.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 36.0}, 'alert': 'firing'}
[ 183.0s]   {'t': 131.9, 'collector_heap_mib': 92.7, 'collector_rss_mib': 262.3, 'collector_container_mib': 265.4, 'queue': {'otlp_http/loki': 63.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 40.0}, 'alert': 'firing'}
[ 193.2s]   {'t': 142.1, 'collector_heap_mib': 92.7, 'collector_rss_mib': 262.3, 'collector_container_mib': 273.3, 'queue': {'otlp_http/loki': 63.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 40.0}, 'alert': 'firing'}
[ 219.3s] docker compose start tempo loki
[ 299.3s] exporter queues drained 80.0 s after start
[ 324.7s]   ... TelemetryExportFailing: firing (0 s)
[ 329.7s]   ... TelemetryExportFailing: firing (5 s)
[ 334.7s]   ... TelemetryExportFailing: firing (10 s)
[ 339.7s]   ... TelemetryExportFailing: firing (15 s)
[ 344.7s]   ... TelemetryExportFailing: firing (20 s)
[ 349.7s]   ... TelemetryExportFailing: firing (25 s)
[ 354.7s]   ... TelemetryExportFailing: firing (30 s)
[ 359.7s]   ... TelemetryExportFailing: firing (35 s)
[ 364.7s]   ... TelemetryExportFailing: firing (40 s)
[ 369.7s]   ... TelemetryExportFailing: firing (45 s)
[ 374.7s] API during outage: {'requests': 11500, 'codes': {'201': 11500}, 'success_ratio': 1.0, 'success_rps': 68.3, 'p95_ms_successful': 13.2}
[ 374.7s] probe traces: 14/15 reached Tempo (lost: sent at [142.1] s)
[ 374.7s] lost in Collector (send_failed+enqueue_failed): {'spans': 4096, 'log_records': 269}; collector heap 44.1 -> max 92.7 MiB (memory_limiter 200 MiB), container max 273.3 MiB
[ 374.7s] saved /home/claude/work/mtusi2026lab6/evidence/native-sandbox/incident-telemetry-down.json
PASS: Tempo+Loki down -> API unaffected, collector memory bounded, losses counted; recovery -> resolved
1 passed, 0 failed
