09:16:45 message sent request_id=07f7f48bf1c74c01b3f546bd98c19492 (text marker kept private)
09:16:47 upload 5791d45a-87cb-465a-b8af-47a02fcfe006: queued -> queued -> ready
09:16:51 Tempo: trace ef3b4c403f734bcb04ee9b5ce085aa68, 42 spans; tree (DB client spans omitted):
    api      SERVER    POST /api/v1/chats/{chat_id}/attachments         10.2 ms UNSET {"http.response.status_code": "202", "messenger.request_id": "f33563a1565d46bea878a83207f42bef", "messenger.instance_id": "api-1", "messenger.attachment_id": "5791d45a-87cb-465a-b8af-47a02fcfe006"}
      api      CLIENT    S3.PutObject                                      3.9 ms UNSET {"rpc.method": "PutObject"}
      api      PRODUCER  attachments.process publish                       1.5 ms UNSET {"messaging.message.id": "e0be9d72-9d4a-4b43-9f2f-1765a668b857", "messenger.job.attempt": "1", "messenger.attachment_id": "5791d45a-87cb-465a-b8af-47a02fcfe006"}
        worker   CONSUMER  attachments.process process                       4.9 ms ERROR {"messaging.message.id": "e0be9d72-9d4a-4b43-9f2f-1765a668b857", "messenger.attachment_id": "5791d45a-87cb-465a-b8af-47a02fcfe006", "messenger.job.attempt": "1", "messenger.job.outcome": "retry"}
          api      PRODUCER  attachments.process publish                       4.6 ms UNSET {"messaging.message.id": "e0be9d72-9d4a-4b43-9f2f-1765a668b857", "messenger.job.attempt": "2", "messenger.attachment_id": "5791d45a-87cb-465a-b8af-47a02fcfe006"}
            worker   CONSUMER  attachments.process process                     169.0 ms UNSET {"messaging.message.id": "e0be9d72-9d4a-4b43-9f2f-1765a668b857", "messenger.attachment_id": "5791d45a-87cb-465a-b8af-47a02fcfe006", "messenger.job.attempt": "2", "messenger.job.outcome": "succeeded"}
              worker   CLIENT    S3.GetObject                                      6.8 ms UNSET {"rpc.method": "GetObject"}
              worker   INTERNAL  imageproc                                       137.9 ms UNSET {"messenger.pipeline_version": "1"}
              worker   CLIENT    S3.CopyObject                                     3.8 ms UNSET {"rpc.method": "CopyObject"}
              worker   CLIENT    S3.PutObject                                      2.2 ms UNSET {"rpc.method": "PutObject"}
              worker   CLIENT    S3.PutObject                                      1.8 ms UNSET {"rpc.method": "PutObject"}
              worker   CLIENT    S3.PutObject                                      1.8 ms UNSET {"rpc.method": "PutObject"}
              worker   CLIENT    S3.DeleteObject                                   1.0 ms UNSET {"rpc.method": "DeleteObject"}
09:16:51 PASS: W3C context API -> outbox -> broker -> worker (retry) -> S3/DB is one trace with producer/consumer links
09:16:51 Loki: 6 records with trace_id=ef3b4c403f734bcb04ee9b5ce085aa68: [('api', 'attachment_queued'), ('worker', 'job_claimed'), ('worker', 'job_attempt_failed'), ('worker', 'job_finished'), ('worker', 'job_claimed'), ('worker', 'job_finished')]
09:16:51 Loki record fields: ['code_filepath', 'code_function', 'code_lineno', 'deployment_environment_name', 'detected_level', 'event', 'flags', 'observed_timestamp', 'scope_name', 'service_instance_id', 'service_name', 'service_namespace', 'service_version', 'severity_number', 'severity_text', 'span_id', 'telemetry_sdk_language', 'telemetry_sdk_name', 'telemetry_sdk_version', 'trace_id', 'ts']
09:16:55 HTTP 500 X-Request-Id=9302b0182c0243df8acc92865370b939 -> Loki event=unhandled_error trace_id=6a254a70c77d5419df95d6f8d2e681ce
09:16:55 Tempo: GET /api/v1/chats status=ERROR request_id matches; exception event recorded=True
09:16:55 message POST trace 26d27e3a639403ae1576707024358b6e: ['POST /api/v1/chats/{chat_id}/messages', 'SELECT', 'SELECT chat_members', 'SELECT idempotency_keys', 'INSERT messages', 'INSERT idempotency_keys']
09:16:55 PASS: password, bearer token, message text, filename, Authorization, signed URLs, image bytes absent from traces and logs
09:16:55 PASS: 994 series, label names: ['__name__', 'db_client_connection_state', 'decision', 'deployment_environment_name', 'error_type', 'http_request_method', 'http_response_status_code', 'http_route', 'instance', 'job', 'job_type', 'le', 'network_protocol_version', 'outcome', 'process_cpu_owner', 'queue', 'rule', 'service_instance_id', 'service_name', 'service_version', 'state', 'url_scheme']
09:16:55 http_route values (templates only): ['/api/v1/attachments/{attachment_id}', '/api/v1/auth/sessions', '/api/v1/chats', '/api/v1/chats/{chat_id}/attachments', '/api/v1/chats/{chat_id}/messages', '/api/v1/users']
09:17:07 PASS: required metrics present: http_server_request_duration_seconds_bucket, messenger_jobs_processed_total, messenger_job_duration_seconds_bucket, messenger_attachment_time_to_ready_seconds_bucket, messenger_queue_depth, messenger_queue_oldest_age_seconds, messenger_worker_busy, messenger_worker_capacity, db_client_connection_count, db_client_connection_pending_requests, process_cpu_time_seconds_total, process_memory_usage_bytes, otelcol_receiver_accepted_spans_total, otelcol_exporter_sent_spans_total, otelcol_exporter_queue_size
PASS: observability levels 3-4 (traces/logs/metrics correlation and privacy)
