# incident-telemetry: lab6: Tempo+Loki down -> API unaffected, collector memory bounded, losses counted
# date: 2026-11-01 20:25:06, commit: 470d9d6 + uncommitted changes
# command: python.exe student-tests/incidents.py telemetry-down
# exit code: 0, 379.8 s

=== telemetry-down
[  22.5s] baseline: {'requests': 899, 'codes': {'201': 899}, 'success_ratio': 1.0, 'success_rps': 59.9, 'p95_ms_successful': 31.0}, collector heap 35.5 MiB, container 75.9 MiB
[  53.8s] docker compose stop tempo loki: stopped
[  55.9s]   {'t': 0.0, 'collector_heap_mib': 56.4, 'collector_rss_mib': 237.6, 'collector_container_mib': 76.0, 'queue': {'otlp_http/loki': 12.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 5.0}, 'alert': 'inactive'}
[  68.0s]   {'t': 12.1, 'collector_heap_mib': 45.4, 'collector_rss_mib': 237.8, 'collector_container_mib': 76.0, 'queue': {'otlp_http/loki': 20.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 9.0}, 'alert': 'pending'}
[  80.0s]   {'t': 24.2, 'collector_heap_mib': 54.2, 'collector_rss_mib': 237.8, 'collector_container_mib': 76.0, 'queue': {'otlp_http/loki': 27.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 12.0}, 'alert': 'pending'}
[  92.5s]   {'t': 36.2, 'collector_heap_mib': 54.2, 'collector_rss_mib': 237.8, 'collector_container_mib': 78.1, 'queue': {'otlp_http/loki': 35.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 15.0}, 'alert': 'pending'}
[ 104.5s]   {'t': 48.7, 'collector_heap_mib': 62.9, 'collector_rss_mib': 240.4, 'collector_container_mib': 82.2, 'queue': {'otlp_http/loki': 35.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 15.0}, 'alert': 'pending'}
[ 116.9s]   {'t': 60.8, 'collector_heap_mib': 51.2, 'collector_rss_mib': 244.4, 'collector_container_mib': 82.2, 'queue': {'otlp_http/loki': 42.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 19.0}, 'alert': 'pending'}
[ 128.9s]   {'t': 73.1, 'collector_heap_mib': 59.9, 'collector_rss_mib': 244.4, 'collector_container_mib': 82.3, 'queue': {'otlp_http/loki': 50.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 22.0}, 'alert': 'firing'}
[ 141.5s]   {'t': 85.2, 'collector_heap_mib': 70.5, 'collector_rss_mib': 248.4, 'collector_container_mib': 88.7, 'queue': {'otlp_http/loki': 57.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 26.0}, 'alert': 'firing'}
[ 153.5s]   {'t': 97.7, 'collector_heap_mib': 70.5, 'collector_rss_mib': 248.4, 'collector_container_mib': 94.5, 'queue': {'otlp_http/loki': 63.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 27.0}, 'alert': 'firing'}
[ 165.9s]   {'t': 109.8, 'collector_heap_mib': 56.3, 'collector_rss_mib': 256.7, 'collector_container_mib': 94.5, 'queue': {'otlp_http/loki': 63.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 27.0}, 'alert': 'firing'}
[ 178.0s]   {'t': 122.1, 'collector_heap_mib': 67.5, 'collector_rss_mib': 256.7, 'collector_container_mib': 94.5, 'queue': {'otlp_http/loki': 70.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 31.0}, 'alert': 'firing'}
[ 190.0s]   {'t': 134.2, 'collector_heap_mib': 76.1, 'collector_rss_mib': 256.9, 'collector_container_mib': 98.8, 'queue': {'otlp_http/loki': 78.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 34.0}, 'alert': 'firing'}
[ 202.6s]   {'t': 146.2, 'collector_heap_mib': 86.9, 'collector_rss_mib': 267.1, 'collector_container_mib': 107.0, 'queue': {'otlp_http/loki': 85.0, 'otlp_http/prometheus': 0.0, 'otlp_grpc/tempo': 38.0}, 'alert': 'firing'}
[ 214.0s] docker compose start tempo loki
[ 274.1s] exporter queues drained 60.1 s after start
[ 289.4s]   ... TelemetryExportFailing: firing (0 s)
[ 294.4s]   ... TelemetryExportFailing: firing (5 s)
[ 299.5s]   ... TelemetryExportFailing: firing (10 s)
[ 304.5s]   ... TelemetryExportFailing: firing (15 s)
[ 309.5s]   ... TelemetryExportFailing: firing (20 s)
[ 314.5s]   ... TelemetryExportFailing: firing (25 s)
[ 319.5s]   ... TelemetryExportFailing: firing (30 s)
[ 324.5s]   ... TelemetryExportFailing: firing (35 s)
[ 329.5s]   ... TelemetryExportFailing: firing (40 s)
[ 334.5s]   ... TelemetryExportFailing: firing (45 s)
[ 339.6s]   ... TelemetryExportFailing: firing (50 s)
[ 344.6s]   ... TelemetryExportFailing: firing (55 s)
[ 349.6s]   ... TelemetryExportFailing: firing (60 s)
[ 354.6s]   ... TelemetryExportFailing: firing (65 s)
[ 359.6s]   ... TelemetryExportFailing: firing (70 s)
[ 364.6s]   ... TelemetryExportFailing: firing (75 s)
[ 369.6s]   ... TelemetryExportFailing: firing (80 s)
[ 374.7s]   ... TelemetryExportFailing: firing (85 s)
[ 379.7s] API during outage: {'requests': 9555, 'codes': {'201': 9555}, 'success_ratio': 1.0, 'success_rps': 59.6, 'p95_ms_successful': 31.0}
[ 379.7s] probe traces: 13/13 reached Tempo (lost: sent at [] s)
[ 379.7s] lost in Collector (send_failed+enqueue_failed): {'spans': 4096, 'log_records': 235}; collector heap 35.5 -> max 86.9 MiB (memory_limiter 200 MiB), container max 107.0 MiB
[ 379.7s] saved C:\Users\Gigabyte\Desktop\lab6\mtusi2026lab6\evidence\incident-telemetry-down.json
PASS: Tempo+Loki down -> API unaffected, collector memory bounded, losses counted; recovery -> resolved
1 passed, 0 failed
