# telemetry: lab6: request<->log<->trace, upload->publish->consume->S3/DB chain, retry, replay link, no secrets
# date: 2026-11-01 00:23:48, commit: unknown
# command: python student-tests/test_telemetry.py request upload-chain retry-chain replay-link
# exit code: 0, 77.1 s

POST message: 201, X-Request-Id=4d281ed1-39de-484a-add2-8480ef33ab9c, trace_id=97ebaba763fec926fa1e1ed3fc0fae7a (client traceparent)
- [api] POST /api/v1/chats/{chat_id}/messages (SERVER, 5.8 ms) http.response.status_code=201
  - [api] redis GET (CLIENT, 0.3 ms)
  - [api] SELECT (CLIENT, 0.5 ms)
  - [api] INSERT (CLIENT, 0.8 ms)
Loki: 'POST /api/v1/chats/{chat_id}/messages -> 201 (5.9 ms)' event=http.request severity=INFO service=messenger-api trace_id=97ebaba763fec926fa1e1ed3fc0fae7a span_id=82ab66080cb50d2a request_id=4d281ed1-39de-484a-add2-8480ef33ab9c
PASS: request -> server span with DB/Redis children -> structured log with the same trace_id/span_id and request_id; no message text, password or token in logs/span attributes
- [api] POST /api/v1/chats/{chat_id}/attachments (SERVER, 64.6 ms) http.response.status_code=202
  - [api] redis GET (CLIENT, 0.4 ms)
  - [api] SELECT (CLIENT, 0.5 ms)
  - [api] inspect image (INTERNAL, 45.7 ms)
  - [api] INSERT (CLIENT, 1.1 ms)
  - [api] S3 PutObject (CLIENT, 5.1 ms) aws.s3.key=staging/f5bdb54e2b234121b7e13ef3007a03b1
  - [api] UPDATE (CLIENT, 0.9 ms)
  - [api] create attachments (PRODUCER, 3.5 ms) messenger.job.attempt=1
    - [api] INSERT (CLIENT, 0.9 ms)
    - [api] INSERT (CLIENT, 0.9 ms)
    - [api] INSERT (CLIENT, 0.8 ms)
    - [api] NOTIFY (CLIENT, 0.3 ms)
    - [dispatcher] publish attachments (PRODUCER, 4.4 ms) messenger.job.attempt=1
      - [dispatcher] UPDATE (CLIENT, 0.9 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 280.0 ms) messenger.job.attempt=1 messenger.job.outcome=ready
        - [worker] SELECT (CLIENT, 1.4 ms)
        - [worker] UPDATE (CLIENT, 0.8 ms)
        - [worker] UPDATE (CLIENT, 1.4 ms)
        - [worker] INSERT (CLIENT, 0.7 ms)
        - [worker] S3 GetObject (CLIENT, 2.9 ms) aws.s3.key=staging/f5bdb54e2b234121b7e13ef3007a03b1
        - [worker] verify source sha256 (INTERNAL, 0.1 ms)
        - [worker] imaging (INTERNAL, 238.2 ms)
        - [worker] S3 PutObject (CLIENT, 4.2 ms) aws.s3.key=attachments/016e0413-eeaa-481a-bef5-27270d212a71/v1/original
        - [worker] S3 PutObject (CLIENT, 3.4 ms) aws.s3.key=attachments/016e0413-eeaa-481a-bef5-27270d212a71/v1/optimized.webp
        - [worker] S3 PutObject (CLIENT, 3.2 ms) aws.s3.key=attachments/016e0413-eeaa-481a-bef5-27270d212a71/v1/thumbnail.jpg
        - [worker] S3 PutObject (CLIENT, 2.9 ms) aws.s3.key=attachments/016e0413-eeaa-481a-bef5-27270d212a71/v1/crop.jpg
        - [worker] commit result (INTERNAL, 7.5 ms)
          - [worker] SELECT (CLIENT, 0.7 ms)
          - [worker] INSERT (CLIENT, 1.0 ms)
          - [worker] INSERT (CLIENT, 0.6 ms)
          - [worker] INSERT (CLIENT, 0.7 ms)
          - [worker] UPDATE (CLIENT, 1.0 ms)
          - [worker] UPDATE (CLIENT, 0.7 ms)
          - [worker] INSERT (CLIENT, 0.4 ms)
        - [worker] cleanup staging (INTERNAL, 4.1 ms)
          - [worker] S3 DeleteObject (CLIENT, 2.4 ms) aws.s3.key=staging/f5bdb54e2b234121b7e13ef3007a03b1
          - [worker] UPDATE (CLIENT, 1.4 ms)
worker logs with trace_id 0eecc096d0191a19e41ad1547110183c: ['job.claimed', 'job.ready']
PASS: one trace 0eecc096d0191a19e41ad1547110183c: API 202 -> outbox create -> dispatcher publish -> worker consume -> 4 S3 objects + COMMIT ready (37 spans, services: messenger-api, messenger-dispatcher, messenger-worker)
- [api] POST /api/v1/chats/{chat_id}/attachments (SERVER, 17.4 ms) http.response.status_code=202
  - [api] redis GET (CLIENT, 0.3 ms)
  - [api] SELECT (CLIENT, 0.5 ms)
  - [api] inspect image (INTERNAL, 0.3 ms)
  - [api] INSERT (CLIENT, 0.7 ms)
  - [api] S3 PutObject (CLIENT, 5.0 ms) aws.s3.key=staging/e638ce1aa4c645a6bbc8e93427d644df
  - [api] UPDATE (CLIENT, 0.6 ms)
  - [api] create attachments (PRODUCER, 2.9 ms) messenger.job.attempt=1
    - [api] INSERT (CLIENT, 0.6 ms)
    - [api] INSERT (CLIENT, 0.5 ms)
    - [api] INSERT (CLIENT, 0.5 ms)
    - [api] NOTIFY (CLIENT, 0.2 ms)
    - [dispatcher] publish attachments (PRODUCER, 3.2 ms) messenger.job.attempt=1
      - [dispatcher] UPDATE (CLIENT, 0.5 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 255.7 ms) ERROR messenger.job.attempt=1 messenger.job.outcome=retry error.type=storage_unavailable
        - [worker] SELECT (CLIENT, 0.8 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] INSERT (CLIENT, 0.4 ms)
        - [worker] S3 GetObject (CLIENT, 3.4 ms) aws.s3.key=staging/e638ce1aa4c645a6bbc8e93427d644df
        - [worker] verify source sha256 (INTERNAL, 0.0 ms)
        - [worker] imaging (INTERNAL, 238.9 ms)
        - [worker] SELECT (CLIENT, 0.6 ms)
        - [worker] UPDATE (CLIENT, 0.8 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] create attachments (PRODUCER, 1.5 ms) messenger.job.attempt=2
          - [worker] INSERT (CLIENT, 0.8 ms)
          - [worker] NOTIFY (CLIENT, 0.2 ms)
          - [dispatcher] publish attachments (PRODUCER, 3.3 ms) messenger.job.attempt=2
            - [dispatcher] UPDATE (CLIENT, 0.6 ms)
            - [worker] process attachments.process.v1 (CONSUMER, 284.4 ms) messenger.job.attempt=2 messenger.job.outcome=ready
              - [worker] SELECT (CLIENT, 0.7 ms)
              - [worker] UPDATE (CLIENT, 0.8 ms)
              - [worker] UPDATE (CLIENT, 0.7 ms)
              - [worker] INSERT (CLIENT, 0.6 ms)
              - [worker] S3 GetObject (CLIENT, 3.5 ms) aws.s3.key=staging/e638ce1aa4c645a6bbc8e93427d644df
              - [worker] verify source sha256 (INTERNAL, 0.1 ms)
              - [worker] imaging (INTERNAL, 250.7 ms)
              - [worker] S3 PutObject (CLIENT, 4.7 ms) aws.s3.key=attachments/9e5dd23f-0ecc-4bdf-b8de-87d529fd54b0/v1/original
              - [worker] S3 PutObject (CLIENT, 3.2 ms) aws.s3.key=attachments/9e5dd23f-0ecc-4bdf-b8de-87d529fd54b0/v1/optimized.webp
              - [worker] S3 PutObject (CLIENT, 3.0 ms) aws.s3.key=attachments/9e5dd23f-0ecc-4bdf-b8de-87d529fd54b0/v1/thumbnail.jpg
              - [worker] S3 PutObject (CLIENT, 2.9 ms) aws.s3.key=attachments/9e5dd23f-0ecc-4bdf-b8de-87d529fd54b0/v1/crop.jpg
              - [worker] commit result (INTERNAL, 6.6 ms)
                - [worker] SELECT (CLIENT, 0.6 ms)
                - [worker] INSERT (CLIENT, 0.7 ms)
                - [worker] INSERT (CLIENT, 0.6 ms)
                - [worker] INSERT (CLIENT, 0.8 ms)
                - [worker] UPDATE (CLIENT, 0.8 ms)
                - [worker] UPDATE (CLIENT, 0.5 ms)
                - [worker] INSERT (CLIENT, 0.5 ms)
              - [worker] cleanup staging (INTERNAL, 4.6 ms)
                - [worker] S3 DeleteObject (CLIENT, 2.7 ms) aws.s3.key=staging/e638ce1aa4c645a6bbc8e93427d644df
                - [worker] UPDATE (CLIENT, 1.3 ms)
        - [worker] INSERT (CLIENT, 0.5 ms)
worker log: 'job 9a82f341-ec57-49e5-aa86-6263681529c1 attempt 1: storage_unavailable (temporary) -> retry' error_type=storage_unavailable
messenger_job_duration_seconds_count{outcome="retry"}: 0 -> 1
PASS: retry in one trace 1abe27ebd519ac0854a844d18d679bf6: attempt 1 ERROR storage_unavailable -> create -> publish -> attempt 2 ready; retry counted in metrics and logged
original trace 5b3ea90278cb5e18b6ad08cd32349c0e: 5 attempts, last outcome failed
- [ops] ops replay (INTERNAL, 16.5 ms)
  - [ops] SELECT (CLIENT, 2.3 ms)
  - [ops] UPDATE (CLIENT, 0.9 ms)
  - [ops] UPDATE (CLIENT, 0.8 ms)
  - [ops] INSERT (CLIENT, 0.7 ms)
  - [ops] create attachments (PRODUCER, 1.1 ms) links=1 messenger.job.attempt=6
    - [ops] INSERT (CLIENT, 0.7 ms)
    - [ops] NOTIFY (CLIENT, 0.2 ms)
    - [dispatcher] publish attachments (PRODUCER, 5.3 ms) messenger.job.attempt=6
      - [worker] process attachments.process.v1 (CONSUMER, 281.7 ms) messenger.job.attempt=6 messenger.job.outcome=ready
        - [worker] SELECT (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.6 ms)
        - [worker] INSERT (CLIENT, 0.2 ms)
        - [worker] S3 GetObject (CLIENT, 2.7 ms) aws.s3.key=staging/4a1c95c1e3ea46c1b51982aa45344f00
        - [worker] verify source sha256 (INTERNAL, 0.0 ms)
        - [worker] imaging (INTERNAL, 251.3 ms)
        - [worker] S3 PutObject (CLIENT, 3.9 ms) aws.s3.key=attachments/5f4e2b56-4497-4ff8-9c9f-df41939a2520/v1/original
        - [worker] S3 PutObject (CLIENT, 2.9 ms) aws.s3.key=attachments/5f4e2b56-4497-4ff8-9c9f-df41939a2520/v1/optimized.webp
        - [worker] S3 PutObject (CLIENT, 2.8 ms) aws.s3.key=attachments/5f4e2b56-4497-4ff8-9c9f-df41939a2520/v1/thumbnail.jpg
        - [worker] S3 PutObject (CLIENT, 2.5 ms) aws.s3.key=attachments/5f4e2b56-4497-4ff8-9c9f-df41939a2520/v1/crop.jpg
        - [worker] commit result (INTERNAL, 6.0 ms)
          - [worker] SELECT (CLIENT, 0.6 ms)
          - [worker] INSERT (CLIENT, 0.7 ms)
          - [worker] INSERT (CLIENT, 0.6 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
          - [worker] UPDATE (CLIENT, 0.7 ms)
          - [worker] UPDATE (CLIENT, 0.5 ms)
          - [worker] INSERT (CLIENT, 0.4 ms)
        - [worker] cleanup staging (INTERNAL, 5.9 ms)
          - [worker] S3 DeleteObject (CLIENT, 3.3 ms) aws.s3.key=staging/4a1c95c1e3ea46c1b51982aa45344f00
          - [worker] UPDATE (CLIENT, 2.1 ms)
      - [dispatcher] UPDATE (CLIENT, 1.3 ms)
replay trace 6d0327f64ceb5758c4a19b30aceed617: create span links -> ['5b3ea90278cb5e18b6ad08cd32349c0e'] (original upload trace 5b3ea90278cb5e18b6ad08cd32349c0e)
PASS: replay = new trace (root 'ops replay') with a link to the original upload; worker finished it
4 passed, 0 failed
