10:32:44 message sent request_id=94e97b26265542fb9d208a905b5d5701 (text marker kept private)
10:32:47 upload 913f8afd-4a07-4a58-b943-ea623e6ec0d0: queued -> queued -> processing -> ready
10:32:51 Tempo: trace 715b953a7f7536b44b99c48205a5c639, 42 spans; tree (DB client spans omitted):
    api      SERVER    POST /api/v1/chats/{chat_id}/attachments         15.3 ms UNSET {"http.response.status_code": "202", "messenger.request_id": "95e2a7eb2f9e46319830eca2e087ccfb", "messenger.instance_id": "api-2", "messenger.attachment_id": "913f8afd-4a07-4a58-b943-ea623e6ec0d0"}
      api      CLIENT    S3.PutObject                                      8.4 ms UNSET {"rpc.method": "PutObject"}
      api      PRODUCER  attachments.process publish                       3.1 ms UNSET {"messaging.message.id": "52bd90b1-34f0-4737-86e7-50e164191b3f", "messenger.job.attempt": "1", "messenger.attachment_id": "913f8afd-4a07-4a58-b943-ea623e6ec0d0"}
        worker   CONSUMER  attachments.process process                       8.3 ms ERROR {"messaging.message.id": "52bd90b1-34f0-4737-86e7-50e164191b3f", "messenger.attachment_id": "913f8afd-4a07-4a58-b943-ea623e6ec0d0", "messenger.job.attempt": "1", "messenger.job.outcome": "retry"}
          api      PRODUCER  attachments.process publish                       4.8 ms UNSET {"messaging.message.id": "52bd90b1-34f0-4737-86e7-50e164191b3f", "messenger.job.attempt": "2", "messenger.attachment_id": "913f8afd-4a07-4a58-b943-ea623e6ec0d0"}
            worker   CONSUMER  attachments.process process                     174.7 ms UNSET {"messaging.message.id": "52bd90b1-34f0-4737-86e7-50e164191b3f", "messenger.attachment_id": "913f8afd-4a07-4a58-b943-ea623e6ec0d0", "messenger.job.attempt": "2", "messenger.job.outcome": "succeeded"}
              worker   CLIENT    S3.GetObject                                     15.2 ms UNSET {"rpc.method": "GetObject"}
              worker   INTERNAL  imageproc                                       128.0 ms UNSET {"messenger.pipeline_version": "1"}
              worker   CLIENT    S3.CopyObject                                     5.3 ms UNSET {"rpc.method": "CopyObject"}
              worker   CLIENT    S3.PutObject                                      3.0 ms UNSET {"rpc.method": "PutObject"}
              worker   CLIENT    S3.PutObject                                      2.1 ms UNSET {"rpc.method": "PutObject"}
              worker   CLIENT    S3.PutObject                                      1.8 ms UNSET {"rpc.method": "PutObject"}
              worker   CLIENT    S3.DeleteObject                                   1.0 ms UNSET {"rpc.method": "DeleteObject"}
10:32:51 PASS: W3C context API -> outbox -> broker -> worker (retry) -> S3/DB is one trace with producer/consumer links
10:32:51 Loki: 4 records with trace_id=715b953a7f7536b44b99c48205a5c639: [('api', 'attachment_queued'), ('worker', 'job_claimed'), ('worker', 'job_attempt_failed'), ('worker', 'job_finished')]
10:32:51 Loki record fields: ['code_filepath', 'code_function', 'code_lineno', 'deployment_environment_name', 'detected_level', 'event', 'flags', 'observed_timestamp', 'scope_name', 'service_instance_id', 'service_name', 'service_namespace', 'service_version', 'severity_number', 'severity_text', 'span_id', 'telemetry_sdk_language', 'telemetry_sdk_name', 'telemetry_sdk_version', 'trace_id', 'ts']
10:32:55 HTTP 500 X-Request-Id=2df37bc19c3d4676804688b4abcb2ae1 -> Loki event=unhandled_error trace_id=f63428fc5f86c4f25be261c5db8ecc08
10:32:55 Tempo: GET /api/v1/chats status=ERROR request_id matches; exception event recorded=True
10:32:55 message POST trace 3157b85474b226259a91fb7e6ddeec5a: ['POST /api/v1/chats/{chat_id}/messages', 'SELECT', 'SELECT chat_members', 'SELECT idempotency_keys', 'INSERT messages', 'INSERT idempotency_keys']
10:32:55 PASS: password, bearer token, message text, filename, Authorization, signed URLs, image bytes absent from traces and logs
10:32:55 PASS: 1921 series, label names: ['__name__', 'db_client_connection_state', 'decision', 'deployment_environment_name', 'error_type', 'http_method', 'http_request_method', 'http_response_status_code', 'http_route', 'http_scheme', 'http_status_code', 'instance', 'job', 'job_type', 'le', 'net_host_name', 'net_host_port', 'net_protocol_name', 'net_protocol_version', 'network_protocol_version', 'otel_scope_name', 'otel_scope_version', 'outcome', 'process_cpu_owner', 'queue', 'rule', 'service_instance_id', 'service_name', 'service_version', 'state', 'url_scheme']
10:32:55 http_route values (templates only): ['/api/v1/attachments/{attachment_id}', '/api/v1/attachments/{attachment_id}/content', '/api/v1/auth/sessions', '/api/v1/auth/sessions/current', '/api/v1/chats', '/api/v1/chats/{chat_id}', '/api/v1/chats/{chat_id}/attachments', '/api/v1/chats/{chat_id}/messages', '/api/v1/users', '/api/v1/users/me']
10:33:07 PASS: required metrics present: http_server_request_duration_seconds_bucket, messenger_jobs_processed_total, messenger_job_duration_seconds_bucket, messenger_attachment_time_to_ready_seconds_bucket, messenger_queue_depth, messenger_queue_oldest_age_seconds, messenger_worker_busy, messenger_worker_capacity, db_client_connection_count, db_client_connection_pending_requests, process_cpu_time_seconds_total, process_memory_usage_bytes, otelcol_receiver_accepted_spans_total, otelcol_exporter_sent_spans_total, otelcol_exporter_queue_size
PASS: observability levels 3-4 (traces/logs/metrics correlation and privacy)
