# telemetry: lab6: request<->log<->trace, upload->publish->consume->S3/DB chain, retry, replay link, no secrets
# date: 2026-11-07 19:14:19, commit: 20afc2c + uncommitted changes
# command: python.exe student-tests/test_telemetry.py request upload-chain retry-chain replay-link
# exit code: 0, 86.6 s

POST message: 201, X-Request-Id=847560b9-32a7-42e0-8c97-04c37a1919ce, trace_id=6f3e99921c71335cdc50bf5d3be240b1 (client traceparent)
- [api] POST /api/v1/chats/{chat_id}/messages (SERVER, 5.6 ms) http.response.status_code=201
  - [api] redis GET (CLIENT, 0.2 ms)
  - [api] SELECT (CLIENT, 0.2 ms)
  - [api] INSERT (CLIENT, 0.3 ms)
Loki: 'POST /api/v1/chats/{chat_id}/messages -> 201 (5.6 ms)' event=http.request severity=INFO service=messenger-api trace_id=6f3e99921c71335cdc50bf5d3be240b1 span_id=ace71ef24e2ec703 request_id=847560b9-32a7-42e0-8c97-04c37a1919ce
PASS: request -> server span with DB/Redis children -> structured log with the same trace_id/span_id and request_id; no message text, password or token in logs/span attributes
- [api] POST /api/v1/chats/{chat_id}/attachments (SERVER, 34.6 ms) http.response.status_code=202
  - [api] redis GET (CLIENT, 0.3 ms)
  - [api] SELECT (CLIENT, 0.5 ms)
  - [api] inspect image (INTERNAL, 0.6 ms)
  - [api] INSERT (CLIENT, 1.8 ms)
  - [api] S3 PutObject (CLIENT, 17.8 ms) aws.s3.key=staging/087e652e62ba42f28d08207795a8d2bc
  - [api] UPDATE (CLIENT, 0.7 ms)
  - [api] create attachments (PRODUCER, 1.9 ms) messenger.job.attempt=1
    - [api] INSERT (CLIENT, 0.5 ms)
    - [api] INSERT (CLIENT, 0.4 ms)
    - [api] INSERT (CLIENT, 0.4 ms)
    - [api] NOTIFY (CLIENT, 0.1 ms)
    - [dispatcher] publish attachments (PRODUCER, 8.2 ms) messenger.job.attempt=1
      - [dispatcher] UPDATE (CLIENT, 0.5 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 154.0 ms) messenger.job.attempt=1 messenger.job.outcome=ready
        - [worker] SELECT (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.6 ms)
        - [worker] INSERT (CLIENT, 0.5 ms)
        - [worker] S3 GetObject (CLIENT, 7.3 ms) aws.s3.key=staging/087e652e62ba42f28d08207795a8d2bc
        - [worker] verify source sha256 (INTERNAL, 0.1 ms)
        - [worker] imaging (INTERNAL, 117.4 ms)
        - [worker] S3 PutObject (CLIENT, 3.3 ms) aws.s3.key=attachments/9176b5fe-cabd-40b0-a7d4-9e3a3358a71b/v1/original
        - [worker] S3 PutObject (CLIENT, 2.2 ms) aws.s3.key=attachments/9176b5fe-cabd-40b0-a7d4-9e3a3358a71b/v1/optimized.webp
        - [worker] S3 PutObject (CLIENT, 2.2 ms) aws.s3.key=attachments/9176b5fe-cabd-40b0-a7d4-9e3a3358a71b/v1/thumbnail.jpg
        - [worker] S3 PutObject (CLIENT, 2.2 ms) aws.s3.key=attachments/9176b5fe-cabd-40b0-a7d4-9e3a3358a71b/v1/crop.jpg
        - [worker] commit result (INTERNAL, 7.0 ms)
          - [worker] SELECT (CLIENT, 0.5 ms)
          - [worker] INSERT (CLIENT, 0.5 ms)
          - [worker] INSERT (CLIENT, 0.4 ms)
          - [worker] INSERT (CLIENT, 0.5 ms)
          - [worker] UPDATE (CLIENT, 0.4 ms)
          - [worker] UPDATE (CLIENT, 0.4 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
        - [worker] cleanup staging (INTERNAL, 6.9 ms)
          - [worker] S3 DeleteObject (CLIENT, 2.3 ms) aws.s3.key=staging/087e652e62ba42f28d08207795a8d2bc
          - [worker] UPDATE (CLIENT, 4.4 ms)
worker logs with trace_id ebd0eee9cd2a35f48c1cf839af717ffd: ['job.claimed', 'job.ready']
PASS: one trace ebd0eee9cd2a35f48c1cf839af717ffd: API 202 -> outbox create -> dispatcher publish -> worker consume -> 4 S3 objects + COMMIT ready (37 spans, services: messenger-api, messenger-dispatcher, messenger-worker)
- [api] POST /api/v1/chats/{chat_id}/attachments (SERVER, 19.3 ms) http.response.status_code=202
  - [api] redis GET (CLIENT, 0.2 ms)
  - [api] SELECT (CLIENT, 0.6 ms)
  - [api] inspect image (INTERNAL, 0.2 ms)
  - [api] INSERT (CLIENT, 0.8 ms)
  - [api] S3 PutObject (CLIENT, 3.4 ms) aws.s3.key=staging/58c8bf4b54734ee08afd28090d6b1620
  - [api] UPDATE (CLIENT, 0.7 ms)
  - [api] create attachments (PRODUCER, 2.4 ms) messenger.job.attempt=1
    - [api] INSERT (CLIENT, 0.7 ms)
    - [api] INSERT (CLIENT, 0.7 ms)
    - [api] INSERT (CLIENT, 0.6 ms)
    - [api] NOTIFY (CLIENT, 0.2 ms)
    - [dispatcher] publish attachments (PRODUCER, 7.1 ms) messenger.job.attempt=1
      - [dispatcher] UPDATE (CLIENT, 0.3 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 122.8 ms) ERROR messenger.job.attempt=1 messenger.job.outcome=retry error.type=storage_unavailable
        - [worker] SELECT (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] INSERT (CLIENT, 0.4 ms)
        - [worker] S3 GetObject (CLIENT, 2.3 ms) aws.s3.key=staging/58c8bf4b54734ee08afd28090d6b1620
        - [worker] verify source sha256 (INTERNAL, 0.0 ms)
        - [worker] imaging (INTERNAL, 107.3 ms)
        - [worker] SELECT (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] create attachments (PRODUCER, 0.9 ms) messenger.job.attempt=2
          - [worker] INSERT (CLIENT, 0.6 ms)
          - [worker] NOTIFY (CLIENT, 0.1 ms)
          - [dispatcher] publish attachments (PRODUCER, 8.4 ms) messenger.job.attempt=2
            - [dispatcher] UPDATE (CLIENT, 0.7 ms)
            - [worker] process attachments.process.v1 (CONSUMER, 157.7 ms) messenger.job.attempt=2 messenger.job.outcome=ready
              - [worker] SELECT (CLIENT, 0.5 ms)
              - [worker] UPDATE (CLIENT, 0.5 ms)
              - [worker] UPDATE (CLIENT, 0.4 ms)
              - [worker] INSERT (CLIENT, 0.3 ms)
              - [worker] S3 GetObject (CLIENT, 2.4 ms) aws.s3.key=staging/58c8bf4b54734ee08afd28090d6b1620
              - [worker] verify source sha256 (INTERNAL, 0.0 ms)
              - [worker] imaging (INTERNAL, 110.7 ms)
              - [worker] S3 PutObject (CLIENT, 3.0 ms) aws.s3.key=attachments/313e5eb1-ca8c-478e-b7e9-650e07812bbb/v1/original
              - [worker] S3 PutObject (CLIENT, 2.2 ms) aws.s3.key=attachments/313e5eb1-ca8c-478e-b7e9-650e07812bbb/v1/optimized.webp
              - [worker] S3 PutObject (CLIENT, 23.5 ms) aws.s3.key=attachments/313e5eb1-ca8c-478e-b7e9-650e07812bbb/v1/thumbnail.jpg
              - [worker] S3 PutObject (CLIENT, 2.1 ms) aws.s3.key=attachments/313e5eb1-ca8c-478e-b7e9-650e07812bbb/v1/crop.jpg
              - [worker] commit result (INTERNAL, 6.2 ms)
                - [worker] SELECT (CLIENT, 0.4 ms)
                - [worker] INSERT (CLIENT, 0.4 ms)
                - [worker] INSERT (CLIENT, 0.3 ms)
                - [worker] INSERT (CLIENT, 0.2 ms)
                - [worker] UPDATE (CLIENT, 0.4 ms)
                - [worker] UPDATE (CLIENT, 0.3 ms)
                - [worker] INSERT (CLIENT, 0.4 ms)
              - [worker] cleanup staging (INTERNAL, 6.0 ms)
                - [worker] S3 DeleteObject (CLIENT, 2.1 ms) aws.s3.key=staging/58c8bf4b54734ee08afd28090d6b1620
                - [worker] UPDATE (CLIENT, 3.6 ms)
        - [worker] INSERT (CLIENT, 0.4 ms)
worker log: 'job 4f3e152c-ba8c-43d4-a83a-7534b92b22bf attempt 1: storage_unavailable (temporary) -> retry' error_type=storage_unavailable
messenger_job_duration_seconds_count{outcome="retry"}: 0 -> 1
PASS: retry in one trace 1406d9f311c8ad0b5c630c831069730d: attempt 1 ERROR storage_unavailable -> create -> publish -> attempt 2 ready; retry counted in metrics and logged
original trace ad6df269b6aa0ed78f91643bc615c79b: 5 attempts, last outcome failed
- [ops] ops replay (INTERNAL, 108.2 ms)
  - [ops] SELECT (CLIENT, 2.1 ms)
  - [ops] UPDATE (CLIENT, 0.7 ms)
  - [ops] UPDATE (CLIENT, 0.5 ms)
  - [ops] INSERT (CLIENT, 0.7 ms)
  - [ops] create attachments (PRODUCER, 1.0 ms) links=1 messenger.job.attempt=6
    - [ops] INSERT (CLIENT, 0.6 ms)
    - [ops] NOTIFY (CLIENT, 0.2 ms)
    - [dispatcher] publish attachments (PRODUCER, 64.9 ms) messenger.job.attempt=6
      - [dispatcher] UPDATE (CLIENT, 58.2 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 132.3 ms) messenger.job.attempt=6 messenger.job.outcome=ready
        - [worker] SELECT (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] INSERT (CLIENT, 0.3 ms)
        - [worker] S3 GetObject (CLIENT, 2.4 ms) aws.s3.key=staging/ab7d94352d1b44b6a515ecd1947771b0
        - [worker] verify source sha256 (INTERNAL, 0.0 ms)
        - [worker] imaging (INTERNAL, 107.9 ms)
        - [worker] S3 PutObject (CLIENT, 3.0 ms) aws.s3.key=attachments/d6e5faa2-7a4b-4797-b56c-ecc6c5577b42/v1/original
        - [worker] S3 PutObject (CLIENT, 2.0 ms) aws.s3.key=attachments/d6e5faa2-7a4b-4797-b56c-ecc6c5577b42/v1/optimized.webp
        - [worker] S3 PutObject (CLIENT, 2.0 ms) aws.s3.key=attachments/d6e5faa2-7a4b-4797-b56c-ecc6c5577b42/v1/thumbnail.jpg
        - [worker] S3 PutObject (CLIENT, 2.1 ms) aws.s3.key=attachments/d6e5faa2-7a4b-4797-b56c-ecc6c5577b42/v1/crop.jpg
        - [worker] commit result (INTERNAL, 5.8 ms)
          - [worker] SELECT (CLIENT, 0.5 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
          - [worker] INSERT (CLIENT, 0.2 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
          - [worker] UPDATE (CLIENT, 0.4 ms)
          - [worker] UPDATE (CLIENT, 0.3 ms)
          - [worker] INSERT (CLIENT, 0.2 ms)
        - [worker] cleanup staging (INTERNAL, 6.5 ms)
          - [worker] S3 DeleteObject (CLIENT, 2.1 ms) aws.s3.key=staging/ab7d94352d1b44b6a515ecd1947771b0
          - [worker] UPDATE (CLIENT, 4.0 ms)
replay trace a363348f79137b4754f30274b251de47: create span links -> ['ad6df269b6aa0ed78f91643bc615c79b'] (original upload trace ad6df269b6aa0ed78f91643bc615c79b)
PASS: replay = new trace (root 'ops replay') with a link to the original upload; worker finished it
4 passed, 0 failed
