# telemetry: lab6: request<->log<->trace, upload->publish->consume->S3/DB chain, retry, replay link, no secrets
# date: 2026-11-01 20:09:11, commit: 470d9d6 + uncommitted changes
# command: python.exe student-tests/test_telemetry.py request upload-chain retry-chain replay-link
# exit code: 0, 78.6 s

POST message: 201, X-Request-Id=df174aef-1d0d-4340-b2d7-907241684f5e, trace_id=6be6bfe68dce05dd2bbbf0048a53c7ce (client traceparent)
- [api] POST /api/v1/chats/{chat_id}/messages (SERVER, 4.6 ms) http.response.status_code=201
  - [api] redis GET (CLIENT, 0.2 ms)
  - [api] SELECT (CLIENT, 0.3 ms)
  - [api] INSERT (CLIENT, 0.4 ms)
Loki: 'POST /api/v1/chats/{chat_id}/messages -> 201 (4.6 ms)' event=http.request severity=INFO service=messenger-api trace_id=6be6bfe68dce05dd2bbbf0048a53c7ce span_id=78cd8621fce4f3b8 request_id=df174aef-1d0d-4340-b2d7-907241684f5e
PASS: request -> server span with DB/Redis children -> structured log with the same trace_id/span_id and request_id; no message text, password or token in logs/span attributes
- [api] POST /api/v1/chats/{chat_id}/attachments (SERVER, 13.5 ms) http.response.status_code=202
  - [api] redis GET (CLIENT, 0.2 ms)
  - [api] SELECT (CLIENT, 0.4 ms)
  - [api] inspect image (INTERNAL, 0.5 ms)
  - [api] INSERT (CLIENT, 0.6 ms)
  - [api] S3 PutObject (CLIENT, 2.9 ms) aws.s3.key=staging/1b42fa0c1dd84589a58369be9f034238
  - [api] UPDATE (CLIENT, 0.7 ms)
  - [api] create attachments (PRODUCER, 2.4 ms) messenger.job.attempt=1
    - [api] INSERT (CLIENT, 0.7 ms)
    - [api] INSERT (CLIENT, 0.6 ms)
    - [api] INSERT (CLIENT, 0.6 ms)
    - [api] NOTIFY (CLIENT, 0.1 ms)
    - [dispatcher] publish attachments (PRODUCER, 4.0 ms) messenger.job.attempt=1
      - [dispatcher] UPDATE (CLIENT, 0.4 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 161.4 ms) messenger.job.attempt=1 messenger.job.outcome=ready
        - [worker] SELECT (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.3 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] INSERT (CLIENT, 0.3 ms)
        - [worker] S3 GetObject (CLIENT, 2.0 ms) aws.s3.key=staging/1b42fa0c1dd84589a58369be9f034238
        - [worker] verify source sha256 (INTERNAL, 0.0 ms)
        - [worker] imaging (INTERNAL, 139.8 ms)
        - [worker] S3 PutObject (CLIENT, 2.8 ms) aws.s3.key=attachments/98cd3d9b-6ee6-4d65-8c63-9586acb34da1/v1/original
        - [worker] S3 PutObject (CLIENT, 1.9 ms) aws.s3.key=attachments/98cd3d9b-6ee6-4d65-8c63-9586acb34da1/v1/optimized.webp
        - [worker] S3 PutObject (CLIENT, 2.3 ms) aws.s3.key=attachments/98cd3d9b-6ee6-4d65-8c63-9586acb34da1/v1/thumbnail.jpg
        - [worker] S3 PutObject (CLIENT, 1.8 ms) aws.s3.key=attachments/98cd3d9b-6ee6-4d65-8c63-9586acb34da1/v1/crop.jpg
        - [worker] commit result (INTERNAL, 4.9 ms)
          - [worker] SELECT (CLIENT, 0.5 ms)
          - [worker] INSERT (CLIENT, 0.5 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
          - [worker] INSERT (CLIENT, 0.4 ms)
          - [worker] UPDATE (CLIENT, 0.4 ms)
          - [worker] UPDATE (CLIENT, 0.3 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
        - [worker] cleanup staging (INTERNAL, 4.7 ms)
          - [worker] S3 DeleteObject (CLIENT, 2.2 ms) aws.s3.key=staging/1b42fa0c1dd84589a58369be9f034238
          - [worker] UPDATE (CLIENT, 2.2 ms)
worker logs with trace_id f5d1897588244cad5fc6ded83b930725: ['job.claimed', 'job.ready']
PASS: one trace f5d1897588244cad5fc6ded83b930725: API 202 -> outbox create -> dispatcher publish -> worker consume -> 4 S3 objects + COMMIT ready (37 spans, services: messenger-api, messenger-dispatcher, messenger-worker)
- [api] POST /api/v1/chats/{chat_id}/attachments (SERVER, 12.6 ms) http.response.status_code=202
  - [api] redis GET (CLIENT, 0.2 ms)
  - [api] SELECT (CLIENT, 0.5 ms)
  - [api] inspect image (INTERNAL, 0.3 ms)
  - [api] INSERT (CLIENT, 0.6 ms)
  - [api] S3 PutObject (CLIENT, 2.5 ms) aws.s3.key=staging/d44ab652329d41458bd477c6cf3ba29c
  - [api] UPDATE (CLIENT, 0.6 ms)
  - [api] create attachments (PRODUCER, 1.4 ms) messenger.job.attempt=1
    - [api] INSERT (CLIENT, 0.4 ms)
    - [api] INSERT (CLIENT, 0.3 ms)
    - [api] INSERT (CLIENT, 0.3 ms)
    - [api] NOTIFY (CLIENT, 0.1 ms)
    - [dispatcher] publish attachments (PRODUCER, 4.2 ms) messenger.job.attempt=1
      - [dispatcher] UPDATE (CLIENT, 0.4 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 135.6 ms) ERROR messenger.job.attempt=1 messenger.job.outcome=retry error.type=storage_unavailable
        - [worker] SELECT (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.4 ms)
        - [worker] INSERT (CLIENT, 0.3 ms)
        - [worker] S3 GetObject (CLIENT, 2.6 ms) aws.s3.key=staging/d44ab652329d41458bd477c6cf3ba29c
        - [worker] verify source sha256 (INTERNAL, 0.0 ms)
        - [worker] imaging (INTERNAL, 123.0 ms)
        - [worker] SELECT (CLIENT, 0.4 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.3 ms)
        - [worker] create attachments (PRODUCER, 1.0 ms) messenger.job.attempt=2
          - [worker] INSERT (CLIENT, 0.6 ms)
          - [worker] NOTIFY (CLIENT, 0.2 ms)
          - [dispatcher] publish attachments (PRODUCER, 5.2 ms) messenger.job.attempt=2
            - [dispatcher] UPDATE (CLIENT, 0.5 ms)
            - [worker] process attachments.process.v1 (CONSUMER, 144.9 ms) messenger.job.attempt=2 messenger.job.outcome=ready
              - [worker] SELECT (CLIENT, 0.4 ms)
              - [worker] UPDATE (CLIENT, 0.4 ms)
              - [worker] UPDATE (CLIENT, 0.4 ms)
              - [worker] INSERT (CLIENT, 0.3 ms)
              - [worker] S3 GetObject (CLIENT, 1.7 ms) aws.s3.key=staging/d44ab652329d41458bd477c6cf3ba29c
              - [worker] verify source sha256 (INTERNAL, 0.0 ms)
              - [worker] imaging (INTERNAL, 125.7 ms)
              - [worker] S3 PutObject (CLIENT, 2.5 ms) aws.s3.key=attachments/214e2e86-8b1d-488b-99d8-5c60fd849c0c/v1/original
              - [worker] S3 PutObject (CLIENT, 1.8 ms) aws.s3.key=attachments/214e2e86-8b1d-488b-99d8-5c60fd849c0c/v1/optimized.webp
              - [worker] S3 PutObject (CLIENT, 1.7 ms) aws.s3.key=attachments/214e2e86-8b1d-488b-99d8-5c60fd849c0c/v1/thumbnail.jpg
              - [worker] S3 PutObject (CLIENT, 1.6 ms) aws.s3.key=attachments/214e2e86-8b1d-488b-99d8-5c60fd849c0c/v1/crop.jpg
              - [worker] commit result (INTERNAL, 4.4 ms)
                - [worker] SELECT (CLIENT, 0.4 ms)
                - [worker] INSERT (CLIENT, 0.4 ms)
                - [worker] INSERT (CLIENT, 0.3 ms)
                - [worker] INSERT (CLIENT, 0.2 ms)
                - [worker] UPDATE (CLIENT, 0.4 ms)
                - [worker] UPDATE (CLIENT, 0.3 ms)
                - [worker] INSERT (CLIENT, 0.4 ms)
              - [worker] cleanup staging (INTERNAL, 4.5 ms)
                - [worker] S3 DeleteObject (CLIENT, 1.7 ms) aws.s3.key=staging/d44ab652329d41458bd477c6cf3ba29c
                - [worker] UPDATE (CLIENT, 2.5 ms)
        - [worker] INSERT (CLIENT, 0.4 ms)
worker log: 'job 7bd96513-9f85-471e-b553-ec49f9fe4560 attempt 1: storage_unavailable (temporary) -> retry' error_type=storage_unavailable
messenger_job_duration_seconds_count{outcome="retry"}: 0 -> 1
PASS: retry in one trace 5d34bb1eecbc90702d2f5e7de5b9bca6: attempt 1 ERROR storage_unavailable -> create -> publish -> attempt 2 ready; retry counted in metrics and logged
original trace 38319800cdce5da47cb2b821dc03bf6e: 5 attempts, last outcome failed
- [ops] ops replay (INTERNAL, 12.3 ms)
  - [ops] SELECT (CLIENT, 1.9 ms)
  - [ops] UPDATE (CLIENT, 0.6 ms)
  - [ops] UPDATE (CLIENT, 0.4 ms)
  - [ops] INSERT (CLIENT, 0.5 ms)
  - [ops] create attachments (PRODUCER, 0.8 ms) links=1 messenger.job.attempt=6
    - [ops] INSERT (CLIENT, 0.5 ms)
    - [ops] NOTIFY (CLIENT, 0.2 ms)
    - [dispatcher] publish attachments (PRODUCER, 70.0 ms) messenger.job.attempt=6
      - [dispatcher] UPDATE (CLIENT, 66.1 ms)
      - [worker] process attachments.process.v1 (CONSUMER, 145.2 ms) messenger.job.attempt=6 messenger.job.outcome=ready
        - [worker] SELECT (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] UPDATE (CLIENT, 0.5 ms)
        - [worker] INSERT (CLIENT, 0.2 ms)
        - [worker] S3 GetObject (CLIENT, 1.7 ms) aws.s3.key=staging/31f520a1f9d2445689ed0344a495998f
        - [worker] verify source sha256 (INTERNAL, 0.0 ms)
        - [worker] imaging (INTERNAL, 125.7 ms)
        - [worker] S3 PutObject (CLIENT, 2.5 ms) aws.s3.key=attachments/ebdeb0f4-a519-42b0-808b-3e6cf118b861/v1/original
        - [worker] S3 PutObject (CLIENT, 2.0 ms) aws.s3.key=attachments/ebdeb0f4-a519-42b0-808b-3e6cf118b861/v1/optimized.webp
        - [worker] S3 PutObject (CLIENT, 1.6 ms) aws.s3.key=attachments/ebdeb0f4-a519-42b0-808b-3e6cf118b861/v1/thumbnail.jpg
        - [worker] S3 PutObject (CLIENT, 1.6 ms) aws.s3.key=attachments/ebdeb0f4-a519-42b0-808b-3e6cf118b861/v1/crop.jpg
        - [worker] commit result (INTERNAL, 4.4 ms)
          - [worker] SELECT (CLIENT, 0.5 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
          - [worker] INSERT (CLIENT, 0.2 ms)
          - [worker] INSERT (CLIENT, 0.2 ms)
          - [worker] UPDATE (CLIENT, 0.4 ms)
          - [worker] UPDATE (CLIENT, 0.3 ms)
          - [worker] INSERT (CLIENT, 0.3 ms)
        - [worker] cleanup staging (INTERNAL, 4.1 ms)
          - [worker] S3 DeleteObject (CLIENT, 2.1 ms) aws.s3.key=staging/31f520a1f9d2445689ed0344a495998f
          - [worker] UPDATE (CLIENT, 1.8 ms)
replay trace 9417040b8f913c26894caaaf0996e442: create span links -> ['38319800cdce5da47cb2b821dc03bf6e'] (original upload trace 38319800cdce5da47cb2b821dc03bf6e)
PASS: replay = new trace (root 'ops replay') with a link to the original upload; worker finished it
4 passed, 0 failed
