27 Commits
Author SHA1 Message Date
Artyom cd4d64a111 Merge completed lab 6 for Gitea submission
Application / Линт, типы и собственные тесты (pull_request) Canceled after 0s
Application / Стенд Compose с RabbitMQ, relay и worker за HAProxy, smoke, pipeline, отказ реплики, рестарт и отказы зависимостей (pull_request) Canceled after 0s
Skeleton contracts / contracts (pull_request) Canceled after 0s
Preserve the exact lab 6 tree and original history from f60297b8e2.
Connect it to the instructor master for a cross-fork pull request.
2026-11-08 00:57:06 +03:00
Artyom f60297b8e2 Validate preserved load evidence and finalize release checks 2026-11-07 04:41:45 +03:00
Artyom 1e00d861a8 Finalize lab 6 report and evidence index
Document grade 3 and 4 evidence, the observability cost study, and review limitations. Separate historical measurements and record static report validation without rerunning Docker workloads.
2026-11-07 04:33:20 +03:00
Artyom 6ca1c75499 Verify late lab 5 fixes and refresh observability evidence 2026-11-07 04:13:55 +03:00
Artyom bc1d7aa771 Port final lab 5 validation, session and PNG recovery fixes 2026-11-07 02:35:40 +03:00
Artyom 97feb04b3f Propose final lab 5 fixes as an OpenSpec change 2026-11-07 02:31:02 +03:00
Artyom 237a7c93e5 Record remaining nonblocking evidence formatting findings 2026-11-06 22:02:43 +03:00
Artyom ebafbc3099 Complete observability cost study with fifteen measured series 2026-11-06 22:02:13 +03:00
Artyom bc0b162082 Handle Docker stats without embedded container names 2026-11-06 21:26:45 +03:00
Artyom 202b985dea Add reproducible observability cost modes and external measurements 2026-11-06 21:22:38 +03:00
Artyom 20e7759f87 Finalize grade 4 verification and record observed limitations 2026-11-06 21:18:21 +03:00
Artyom 0fc02496fe Archive grade 4 traces, alerts and bounded outage evidence 2026-11-06 21:17:45 +03:00
Artyom cd9d91550a Exercise error alerts with a bounded PostgreSQL lock incident 2026-11-06 21:12:24 +03:00
Artyom 447a6507f9 Keep S3 stopped during the telemetry retry scenario 2026-11-06 13:01:28 +03:00
Artyom 44d81fd80e Preserve distributed job context and expose grade 4 operational signals 2026-11-06 12:57:10 +03:00
Artyom 1a171d0d0f Archive verified grade 3 observability requirements 2026-11-06 12:47:37 +03:00
Artyom 2903d9dc73 Verify the grade 3 telemetry stack and inherited API guarantees 2026-11-06 12:41:06 +03:00
Artyom 95511002b1 Archive the lab 5 foundation forwarding change 2026-11-06 12:30:02 +03:00
Artyom 12db77133f Forward the lab 5 foundation while preserving lab 6 telemetry
Source: immutable lab5 snapshot f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071; forward the allowed diff from 64d5c67. Carry SQLAlchemy, Alembic, reliability fixes, versioned jobs, infrastructure and student tests without changing protected coursework files.

Validation: make check, validate, lint; 515 student tests passed with one Linux-only skip; public smoke with telemetry enabled and repeat migration passed. Stop the owned stack and retain volumes. Preserve the existing OTel work in progress; grade implementation remains a separate phase.
2026-11-06 12:29:35 +03:00
Artyom dc9256baf9 Propose forwarding the lab 5 foundation as an openspec change 2026-11-06 12:17:32 +03:00
ArtyomandClaude Opus 5.5 a8836ce8b5 Instrument the API, the relay and the worker
Start the telemetry in create_app, which gunicorn runs in every worker
after fork because preload_app is off, and in the relay and worker
processes with their own service names, and stop it after the rest of
the shutdown within TELEMETRY_SHUTDOWN_SECONDS. The API gets server
spans and the http.server.request.duration histogram in seconds from
the FastAPI instrumentation with stable HTTP semantic conventions,
without /health probes and without the internal send and receive
spans, and the request pipeline puts X-Request-Id on the server span
as messenger.request_id. The worker broker gets the FastStream
RabbitTelemetryMiddleware, so every delivery runs inside a consumer
span.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:45:26 +03:00
ArtyomandClaude Opus 5.5 e2ec3fe9af Count and trace attachment jobs through a job observer port
Add the JobObserver port to the attachments zone and its adapter on the
OpenTelemetry API: every delivery adds one to messenger.jobs.processed
and records messenger.job.duration in seconds with buckets from 50 ms to
60 s, both labelled only by job.type and outcome, and puts the job id,
the attachment id, the attempt and the outcome on the current consumer
span, which turns to ERROR on retry_scheduled, failed and error. The
worker entrypoint measures each delivery, reports the outcome of the
handler or error when it raised, and writes a job.processed record.
Without the SDK the adapter runs on the no-op providers of the API.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:45:20 +03:00
ArtyomandClaude Opus 5.5 e60b4eac98 Start the OpenTelemetry SDK in every process
Add the OpenTelemetry API and SDK, the OTLP over HTTP exporter and the
FastAPI and asyncpg instrumentations to the lock, and a platform module
that builds one resource per process (service.name, service.version,
service.instance.id as hostname and pid, deployment.environment.name
from OTEL_RESOURCE_ATTRIBUTES), batch processors for traces and logs,
a periodic metric reader, PostgreSQL client spans and a bounded
shutdown in a daemon thread instead of the 30 s flush of the SDK. The
SDK stays off without a Collector address or with OTEL_SDK_DISABLED.

The sampler lets only server, consumer and producer spans open a
trace, so readiness probes, outbox polls and sweeps do not become
one-span traces. Log records carry trace_id and span_id in stdout and
go to OTLP through a handler that keeps only the allowlisted fields,
and deferred event subscribers now run in the context the event was
emitted in, so audit records stay linked to their request or job.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:45:14 +03:00
ArtyomandClaude Opus 5.5 a41d6d39e8 Propose the telemetry pipeline as an openspec change
Add the change add-lab6-telemetry-pipeline for level 3 of lab 6: every
API worker process, the relay and the worker start the OpenTelemetry
SDK after fork with service.name, service.version, a per-process
service.instance.id and deployment.environment.name, export OTLP over
HTTP with batch processors only, record HTTP server spans and the
seconds histogram by route template, consumer spans and the
jobs.processed counter and job.duration histogram by job type and
outcome, PostgreSQL client spans inside requests and jobs, and send
every allowlisted log record to Loki with trace_id and span_id. The
stand gains the Collector, Prometheus, Loki, Tempo and Grafana with
volumes and provisioning from Git, built from the verified .prep/lab6
configuration.

The specs add lab6-service-telemetry and lab6-telemetry-stack and
extend the unified log requirement of lab1-service-operability.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 04:37:41 +03:00
ArtyomandClaude Opus 5.5 5ae3d77152 Carry over the lab 5 solution
Bring the application, own tests, Dockerfile, toolchain, compose stack,
HAProxy, k6, RustFS and RabbitMQ configuration, migrations, hooks,
scenario scripts and the CI job from the lab 5 solution commit 64d5c67,
keep the lab 5 Makefile targets on top of the unchanged skeleton
targets, and append the lab 5 solution variables to the skeleton
.env.example. The package and the application follow the contract
version 1.6.0, and the image, the throwaway test containers and the
broker vhost description are tagged lab6 so they do not collide with a
lab 5 stand on the same host.

This is the starting point of the lab 6 cycle: lint, own tests and the
offline skeleton check pass before any lab 6 change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:51:58 +03:00
ArtyomandClaude Opus 5.5 a0d3947710 Carry over the agent context and openspec history from lab 5
Bring the Claude Code settings, the lint hook, the opsx commands and
skills, the architecture notes, the lab 1 to lab 5 specs and the
archived lab 1 to lab 5 changes into the lab 6 skeleton, so lab 6
continues the same spec history instead of starting from scratch. The
files come from the lab 5 solution commit 64d5c67.

Rewrite AGENTS.md and the openspec context and rules for lab 6: the
business API stays at contract 1.6.0, API, relay and worker start the
OpenTelemetry SDK in every process after fork and export OTLP over HTTP
to a separate Collector, Prometheus, Loki and Tempo store metrics, logs
and traces on volumes, Grafana only reads them through provisioning
kept in Git, telemetry never blocks a request and never feeds
/health/ready, metric labels stay low-cardinality and message bodies
and secrets never leave the process. Out of scope moves to lab 7,
track 2 and clustered telemetry backends.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-29 03:46:20 +03:00
Mikhail Verkovykh 470d9d6f1e Подготовить skeleton лабораторной 6 по мессенджеру 2026-09-10 18:01:36 +03:00
699 changed files with 333932 additions and 74 deletions
+181
View File
@@ -0,0 +1,181 @@
---
name: "OPSX: Apply"
description: "Implement tasks from an OpenSpec change (Experimental)"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "artifacts", "experimental"]
---
Implement tasks from an OpenSpec change.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name (e.g., `/opsx:apply add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:apply <other>`).
2. **Check status to understand the schema**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand:
- `schemaName`: The workflow being used (e.g., "spec-driven")
- `planningHome`, `changeRoot`, and `actionContext`: planning scope and edit constraints
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
3. **Get apply instructions**
```bash
openspec instructions apply --change "<name>" --json
```
This returns:
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
- Progress (total, complete, remaining)
- Task list with status
- Dynamic instruction based on current state
- Optional `context`: current required project instruction input from the selected root
- Optional `operationGuidance`: current advisory guidance for apply
**Handle states:**
- If `state: "blocked"` (missing artifacts): show message, suggest using `/opsx:continue` (if it is not installed, run `openspec status --change "<name>" --json` to see the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` for how to create it)
- If `state: "all_done"`: congratulate, suggest archive
- Otherwise: proceed to implementation
Treat `context` as a required prompt-level input. Read and consider it, and
apply relevant project facts, conventions, and constraints while implementing.
Treat `operationGuidance` as optional additive advice. Read and consider every
entry, and follow entries that are applicable and compatible with the built-in
workflow.
Keep both fields separate from CLI-returned state, missing artifacts, tasks,
progress, `contextFiles`, and the built-in `instruction`. They are not
evidence of task completion, do not replace the built-in instruction, and do
not permit bypassing a blocked state. If context conflicts with the built-in
instruction, an explicit user choice, or a CLI-controlled value, report the
conflict and preserve the controlling value. If guidance is inapplicable or
conflicts with those controlling inputs, do not follow it and explain why.
These are prompt-level behavior contracts, not enforceable checks.
4. **Read context files**
Read every file path listed under `contextFiles` from the apply instructions output.
The files depend on the schema being used:
- **spec-driven**: proposal, specs, design, tasks
- Other schemas: follow the contextFiles from CLI output
Do not copy `context` or `operationGuidance` verbatim into implementation
files or planning artifacts unless the user separately asks for that content.
5. **Show current progress**
Display:
- Schema being used
- Progress: "N/M tasks complete"
- Remaining tasks overview
- Dynamic instruction from CLI
6. **Implement tasks (loop until done or blocked)**
For each pending task:
- Show which task is being worked on
- Make the code changes required
- Keep changes minimal and focused
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
- Continue to next task
**Pause if:**
- Task is unclear → ask for clarification
- Implementation reveals a design issue → suggest updating artifacts
- Error or blocker encountered → report and wait for guidance
- User interrupts
7. **On completion or pause, show status**
Display:
- Tasks completed this session
- Overall progress: "N/M tasks complete"
- If all done: suggest archive
- If paused: explain why and wait for guidance
**Output During Implementation**
```
## Implementing: <change-name> (schema: <schema-name>)
Working on task 3/7: <task description>
[...implementation happening...]
✓ Task complete
Working on task 4/7: <task description>
[...implementation happening...]
✓ Task complete
```
**Output On Completion**
```
## Implementation Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 7/7 tasks complete ✓
### Completed This Session
- [x] Task 1
- [x] Task 2
...
All tasks complete! You can archive this change with `/opsx:archive`.
```
**Output On Pause (Issue Encountered)**
```
## Implementation Paused
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 4/7 tasks complete
### Issue Encountered
<description of the issue>
**Options:**
1. <option 1>
2. <option 2>
3. Other approach
What would you like to do?
```
**Guardrails**
- Keep going through tasks until done or blocked
- Always read context files before starting (from the apply instructions output)
- If task is ambiguous, pause and ask before implementing
- If implementation reveals issues, pause and suggest artifact updates
- Keep code changes minimal and scoped to each task
- Update task checkbox immediately after completing each task
- Pause on errors, blockers, or unclear requirements - don't guess
- Use contextFiles from CLI output, don't assume specific file names
- Do not use context or operation guidance as proof that a task is complete
- Apply relevant project context; report conflicts with controlling workflow inputs
- Consider every guidance entry; explain any inapplicable or conflicting advice
- Do not copy runtime context or operation guidance into implementation files or planning artifacts
- Preserve CLI-controlled blocked/ready/all-done behavior and completion criteria
**Fluid Workflow Integration**
This skill supports the "actions on a change" model:
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
+225
View File
@@ -0,0 +1,225 @@
---
name: "OPSX: Archive"
description: "Archive a completed change in the experimental workflow"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "archive", "experimental"]
---
Archive a completed change in the experimental workflow.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name after `/opsx:archive` (e.g., `/opsx:archive add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show only active changes (not already archived).
Include the schema used for each change if available.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:archive <other>`).
**Load current archive inputs before the existing archive checks:**
After resolving the selected change and planning root, run:
```bash
openspec instructions archive --change "<name>" --json
```
Keep the same selected-root flags on this command. This lookup is advisory and
optional: it only supplies extra prompt inputs, so it must never block archiving.
If it exits non-zero or returns invalid JSON — for example on an older CLI that
does not support this command yet — continue the archive workflow with no
context and no operation guidance. Do not report an error and do not stop.
A successful response may omit both optional fields. Treat `context` as a
required prompt-level input: read and consider it, and apply relevant project
facts, conventions, and constraints. Treat `operationGuidance` as optional
additive advice: read and consider every entry, and follow entries that are
applicable and compatible with the built-in archive workflow.
Keep both fields separate from built-in steps, explicit user choices, resolved
paths, CLI checks, and command contracts. If context conflicts with one of those
controlling inputs, report the conflict and preserve the controlling value. If
guidance is inapplicable or conflicts with a controlling input, do not follow it
and explain why. Do not infer replacement paths, skipped prompts, or flags from
either field, and do not copy their text verbatim into specs, change artifacts,
or archive summaries unless the user separately asks for it. These are
prompt-level behavior contracts, not enforceable checks.
2. **Check artifact completion status**
Run `openspec status --change "<name>" --json` to check artifact completion.
Parse the JSON to understand:
- `schemaName`: The workflow being used
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context
- `artifacts`: List of artifacts with their status (`done`, `skipped`, or other)
**If any artifacts are neither `done` nor `skipped`** (skipped artifacts satisfy the requirement - the change declares skip_specs):
- Display warning listing incomplete artifacts
- Prompt user for confirmation to continue
- Proceed if user confirms
3. **Check task completion status**
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
**If incomplete tasks found:**
- Display warning showing count of incomplete tasks
- Prompt user for confirmation to continue
- Proceed if user confirms
**If no tasks file exists:** Proceed without task-related warning.
4. **Assess delta spec sync state**
Use `artifactPaths.specs.existingOutputPaths` from status JSON as the only
delta-spec source. If the `specs` entry is missing or
`existingOutputPaths` is empty, proceed without a sync prompt and do not infer
delta specs from other artifacts.
**If delta specs exist:**
- Compare each delta spec with its corresponding main spec at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (use the store-aware `planningHome.root` from step 2, not a hardcoded repo path)
- Determine what changes would be applied (adds, modifications, removals, renames)
- Show a combined summary before prompting
**Prompt options:**
- If changes needed: "Sync now (recommended)", "Archive without syncing"
- If already synced: "Archive now", "Sync anyway", "Cancel"
Route on the answer:
- "Cancel" — stop, do not archive
- "Archive without syncing" or "Archive now" — proceed to archive
- "Sync now" or "Sync anyway" — sync, then verify (below)
- Anything else — ask again rather than archiving
Before a selected sync writes any main spec, run
`openspec instructions specs --change "<name>" --json` once with the same
selected-root flags. Require a zero exit status and valid artifact-instruction
JSON. If the lookup fails or returns invalid JSON, report the error and stop
before writing any main spec or moving the change. A valid response with omitted
`rules` is the no-rules case. Apply returned `rules` only to the content and
form of main specs produced by this merge; do not use them as archive guidance,
change CLI behavior, or copy the rule text into any output file.
Then run the `/opsx:sync` workflow inline (agent-driven intelligent merge) for change '<name>', passing the delta spec analysis and the fetched specs-rule snapshot from above, and wait for it to finish. The inline sync must reuse that snapshot without fetching `specs` instructions again. Do not delegate it to a background task — step 5 would move `changeRoot` out from under a sync that is still reading it, leaving the change archived and the main specs never updated. If your agent can only run it by delegation, delegate synchronously and wait for the result.
Then re-run the comparison from the top of this step against every capability that has a delta spec in `artifactPaths.specs.existingOutputPaths` — not only the ones the sync reports it touched. A successful sync leaves nothing left to apply, so each capability must now read as already synced:
- ADDED requirements present
- MODIFIED requirements carrying the scenario and description changes named in the delta, with their other scenarios intact
- REMOVED requirements gone — and where this sync retired a capability (removed its last requirement, leaving `## Requirements` empty), its main spec deleted rather than left empty; a spec the sync deliberately kept and reported is also a match
- RENAMED requirements present under the new name and absent under the old one
If the sync failed, or any capability does not match, report what differs and stop — do not archive. Nothing has moved and `changeRoot` is intact, so the user can fix the mismatch or re-run the sync and start the archive again.
5. **Perform the archive**
Create an `archive` directory under `planningHome.changesDir` if it doesn't exist:
```bash
mkdir -p "<planningHome.changesDir>/archive"
```
Generate the target name: use the change name as-is when it already starts with a `YYYY-MM-DD-` prefix; otherwise prepend the current date as `YYYY-MM-DD-<change-name>`. Never stack a second date (same rule as `openspec archive`).
**Check if target already exists:**
- If yes: Fail with error, suggest renaming existing archive or using different date
- If no: Move `changeRoot` to the archive directory
```bash
mv "<changeRoot>" "<planningHome.changesDir>/archive/<target-name>"
```
6. **Display summary**
Show archive completion summary including:
- Change name
- Schema that was used
- Archive location
- Spec sync status (synced / sync skipped / no delta specs)
- Note about any warnings (incomplete artifacts/tasks)
**Output On Success**
```markdown
## Archive Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** ✓ Synced to main specs
All artifacts complete. All tasks complete.
```
**Output On Success (No Delta Specs)**
```markdown
## Archive Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** No delta specs
All artifacts complete. All tasks complete.
```
**Output On Success With Warnings**
```markdown
## Archive Complete (with warnings)
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** Sync skipped (user chose to skip)
**Warnings:**
- Archived with 2 incomplete artifacts
- Archived with 3 incomplete tasks
- Delta spec sync was skipped (user chose to skip)
Review the archive if this was not intentional.
```
**Output On Error (Archive Exists)**
```markdown
## Archive Failed
**Change:** <change-name>
**Target:** the archive path derived from `planningHome.changesDir`/<target-name>/
Target archive directory already exists.
**Options:**
1. Rename the existing archive
2. Delete the existing archive if it's a duplicate
3. Wait until a different date to archive
```
**Guardrails**
- Announce the selected change; prompt for selection when it is ambiguous
- Use artifact graph (openspec status --json) for completion checking
- Don't block archive on warnings - just inform and confirm
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
- Show clear summary of what happened
- If sync is requested, run the `/opsx:sync` workflow inline (agent-driven)
- Never archive while a spec sync is still in flight — run the sync inline and verify the main specs before moving `changeRoot`
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
- Apply relevant runtime context and report conflicts; operation guidance remains advisory
- Consider every guidance entry and explain any inapplicable or conflicting advice
- Existing CLI checks, resolved paths, prompts, and command contracts are unchanged
- Artifact rules constrain only the specs being written and are never operation guidance
- Never copy runtime context, operation guidance, or artifact-rule text verbatim into output files
+193
View File
@@ -0,0 +1,193 @@
---
name: "OPSX: Explore"
description: "Enter explore mode - think through ideas, investigate problems, clarify requirements"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "explore", "experimental", "thinking"]
---
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing. For a new change, scaffold it first as described below.
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: The argument after `/opsx:explore` is whatever the user wants to think about. Could be:
- A vague idea: "real-time collaboration"
- A specific problem: "the auth system is getting unwieldy"
- A change name: "add-dark-mode" (to explore in context of that change)
- A comparison: "postgres vs sqlite for this"
- Nothing (just enter explore mode)
---
## The Stance
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
- **Adaptive** - Follow interesting threads, pivot when new information emerges
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
---
## What You Might Do
Depending on what the user brings, you might:
**Explore the problem space**
- Ask clarifying questions that emerge from what they said
- Challenge assumptions
- Reframe the problem
- Find analogies
**Investigate the codebase**
- Map existing architecture relevant to the discussion
- Find integration points
- Identify patterns already in use
- Surface hidden complexity
**Compare options**
- Brainstorm multiple approaches
- Build comparison tables
- Sketch tradeoffs
- Recommend a path (if asked)
**Visualize**
```
┌─────────────────────────────────────────┐
│ Use ASCII diagrams liberally │
├─────────────────────────────────────────┤
│ │
│ ┌────────┐ ┌────────┐ │
│ │ State │────────▶│ State │ │
│ │ A │ │ B │ │
│ └────────┘ └────────┘ │
│ │
│ System diagrams, state machines, │
│ data flows, architecture sketches, │
│ dependency graphs, comparison tables │
│ │
└─────────────────────────────────────────┘
```
**Surface risks and unknowns**
- Identify what could go wrong
- Find gaps in understanding
- Suggest spikes or investigations
---
## OpenSpec Awareness
You have full context of the OpenSpec system. Use it naturally, don't force it.
### Check for context
At the start, quickly check what exists:
```bash
openspec list --json
```
This tells you:
- If there are active changes
- Their names, schemas, and status
- What the user might be working on
Then read the project's own context from the resolved root - `<root.path>/openspec/config.yaml` (or `config.yml`). Use the `root.path` returned above, and skip this if neither file exists:
- `context`: project background - tech stack, conventions, constraints
- `rules`: keyed by artifact id - the entries for an artifact apply only when you write that artifact
Ground your thinking in these. They are constraints for you to follow, not content to reproduce: do NOT copy them into the conversation or into any artifact you create.
If the user mentioned a specific change name, read its artifacts for context.
### When no change exists
Think freely. When insights crystallize, you might offer:
- "This feels solid enough to start a change. Want me to create a proposal?"
- Or keep exploring - no pressure to formalize
If the user asks you to capture the exploration as a new change, transition seamlessly into the requested capture:
1. Run `openspec new change "<name>"` (with `--store <id>` when applicable) before creating any artifacts. Never create a new change directory under `openspec/changes/` by hand; the CLI scaffold creates required metadata such as `.openspec.yaml`. Keep the selected `--store <id>` on every applicable follow-up `status` and `instructions` command.
2. Run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store), then process the requested artifacts in dependency order. For each requested artifact that is `ready`, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store). Before creating a requested artifact, evaluate any condition in its own `instruction` against the explored change; record a deliberate skip instead when the condition does not apply. If a requested artifact is blocked by a direct prerequisite the user did not request, run `openspec instructions "<prerequisite-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) for that prerequisite whether it is `ready` or `blocked`. If its own `instruction` states a condition, evaluate that condition against the explored change and record a deliberate skip only when the condition does not apply. If the condition applies, or the prerequisite is not conditional, treat it as a normal prerequisite and ask before expanding the capture. Do not create an unrequested prerequisite unless the user approves.
3. Follow the returned `template` and `instruction` fields. Read completed dependency files listed in `dependencies`, and apply `context` and `rules` as constraints without copying them into the artifact. If the instruction delegates creation to a specific skill or command, invoke it; otherwise write the artifact to `resolvedOutputPath`, using the instruction to choose a concrete path when it is a glob. Verify that the selected concrete output exists.
4. After creating each artifact, re-run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) and continue until every requested artifact is `done`, `skipped`, or was deliberately skipped because its own `instruction` stated a condition that did not apply. Tell the user about a deliberate conditional skip, remember it, and do not reconsider it. Dependencies are enablers, not gates: if a requested artifact is still `blocked` only because you deliberately skipped a conditional prerequisite, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) despite the blocked status, then create it using step 3 only when those recorded conditional skips are its sole missing dependencies. If a requested artifact is blocked by a prerequisite the user did not ask to capture and cannot be conditionally skipped, explain that dependency and ask before expanding the capture.
Capture the artifact(s) the user requested without asking them to invoke another workflow command. If they asked only to start a change, stop after scaffolding and show its status.
### When a change exists
If the user mentions a change or you detect one is relevant:
1. **Resolve and read existing artifacts for context**
- Run `openspec status --change "<name>" --json`.
- Use `changeRoot`, `artifactPaths`, and `actionContext` from the status JSON.
- Read existing files from `artifactPaths.<artifact>.existingOutputPaths`.
2. **Reference them naturally in conversation**
- "Your design mentions using Redis, but we just realized SQLite fits better..."
- "The proposal scopes this to premium users, but we're now thinking everyone..."
3. **Offer to capture when decisions are made**
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
| Insight Type | Where to Capture |
|----------------------------|-------------------------------------|
| New requirement discovered | `specs/<capability-path>/spec.md` |
| Requirement changed | `specs/<capability-path>/spec.md` |
| Design decision made | `design.md` |
| Scope changed | `proposal.md` |
| New work identified | `tasks.md` |
| Assumption invalidated | Relevant artifact |
Example offers:
- "That's a design decision. Capture it in design.md?"
- "This is a new requirement. Add it to specs?"
- "This changes scope. Update the proposal?"
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
---
## What You Don't Have To Do
- Follow a script
- Ask the same questions every time
- Produce a specific artifact
- Reach a conclusion
- Stay on topic if a tangent is valuable
- Be brief (this is thinking time)
---
## Ending Discovery
There's no required ending. Discovery might:
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
- **Result in artifact updates**: "Updated design.md with these decisions"
- **Just provide clarity**: User has what they need, moves on
- **Continue later**: "We can pick this up anytime"
When things crystallize, you might offer a summary - but it's optional. Sometimes the thinking IS the value.
---
## Guardrails
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
- **Don't fake understanding** - If something is unclear, dig deeper
- **Don't rush** - Discovery is thinking time, not task time
- **Don't force structure** - Let patterns emerge naturally
- **Don't auto-capture** - Offer to save insights, don't just do it
- **Don't manually scaffold changes** - Never create a new change directory under `openspec/changes/` by hand. Always use `openspec new change "<name>"` (with `--store <id>` when applicable) so required metadata such as `.openspec.yaml` is created before writing artifacts.
- **Do visualize** - A good diagram is worth many paragraphs
- **Do explore the codebase** - Ground discussions in reality
- **Do question assumptions** - Including the user's and your own
+145
View File
@@ -0,0 +1,145 @@
---
name: "OPSX: Propose"
description: "Propose a new change - create it and generate all artifacts in one step"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "artifacts", "experimental"]
---
Propose a new change - create the change and generate all artifacts in one step.
**Planning boundary**: This workflow creates planning artifacts only. The user request that selected or triggered this workflow authorizes planning only, even if it asks to build or fix something. Do not edit project code. After the planning artifacts are complete, stop. Do not start implementation in the same response, even if the initial request asks for it. Wait for a new user request after the artifacts are presented; then start the apply workflow.
I'll create a change with the artifacts your schema defines. With the default spec-driven schema that is:
- proposal.md (what & why)
- `specs/<capability-path>/spec.md` (what the system must do - a delta, not the main spec)
- design.md (how)
- tasks.md (implementation steps)
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
When the user is ready to implement, they must start the apply workflow explicitly.
---
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: The argument after `/opsx:propose` is the change name (kebab-case), OR a description of what the user wants to build.
**Steps**
1. **Understand the request and clarify material ambiguity**
If no input is provided, ask the user (open-ended, no preset options):
> "What change do you want to work on? Describe what you want to build or fix."
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
If the request contains ambiguity that would materially affect scope, externally observable behavior, compatibility, or acceptance criteria, ask the user before creating the change. For minor details, make a reasonable assumption and record it in the planning artifacts.
2. **Determine the workflow schema**
Use the configured default schema unless the user explicitly requests a different workflow.
**Use a different schema only if the user:**
- Explicitly requests a specific schema by name → use `--schema <schema-name>`
- Asks to "show workflows" or asks "what workflows" exist → resolve the authoritative root by running `openspec context --json` from the current working directory. If the user explicitly selected a registered store, use `openspec context --json --store "<store-id>"`. Then run `openspec schemas --json` with its working directory set to the returned `root.path` and let them choose. This preserves roots selected by a local `store:` pointer or the global `defaultStore`; `schemas` does not accept `--store`. If context reports only `no_openspec_root`, run `openspec schemas --json` from the current working directory instead. Do not use this fallback for invalid or unavailable stores.
Otherwise, omit `--schema` to preserve the configured default.
3. **Create the change directory**
Choose one schema form below. If a registered store is selected, append `--store "<store-id>"` to that command and each later OpenSpec command shown below that accepts `--store`.
Using the configured default:
```bash
openspec new change "<name>"
```
Using an explicitly requested schema:
```bash
openspec new change "<name>" --schema "<schema-name>"
```
This creates a scaffolded change in the planning home resolved by the CLI with `.openspec.yaml`.
4. **Get the artifact build order**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to get:
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
- `artifacts`: list of all artifacts, each with its `status` and its `requires` edges (the artifact IDs it directly depends on)
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
5. **Create every artifact in the required set**
Use a todo list to track progress through the artifacts.
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
a. **For each artifact that is `ready` (dependencies satisfied)**:
- Get instructions:
```bash
openspec instructions <artifact-id> --change "<name>" --json
```
- The instructions JSON includes:
- `context`: Project background (constraints for you - do NOT include in output)
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
- `template`: The structure to use for your output file
- `instruction`: Schema-specific guidance for this artifact type
- `skipped`/`warning`: present when the change declares skip_specs and this artifact must NOT be created - stop and pick another artifact
- `resolvedOutputPath`: Resolved path or pattern to write the artifact
- `dependencies`: Completed artifacts to read for context
- Read any completed dependency files for context - always re-read them from disk, even if you saw them earlier in the conversation (the user may have edited them)
- If the `instruction` field delegates creation to a specific skill or command, invoke it to produce the artifact instead of writing the file yourself, then verify the artifact file exists at `resolvedOutputPath`
- Otherwise create the artifact file using `template` as the structure and write it to `resolvedOutputPath`. If `resolvedOutputPath` is a glob, follow `instruction` to choose the concrete file path
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
- Show brief progress: "Created <artifact-id>"
b. **Continue until every artifact in the required set exists (not just `apply.requires`)**
- After creating each artifact, re-run `openspec status --change "<name>" --json`
- The required set is `applyRequires` plus every artifact reachable from those by following the `requires` edges in `status --json` - walk them transitively (spec-driven closes over proposal, specs, design, tasks). Leave artifacts outside that set alone
- `status` is file-existence only, so an `applyRequires` artifact reading `done` does NOT mean its dependencies exist - writing `tasks.md` early marks `tasks` done while `specs` was never written. Use each artifact's `requires` edges, not its `status`, to build the required set: a `done` artifact still lists what it depends on
- An artifact already reading `status: "skipped"` is satisfied: the change declares `skip_specs` in `.openspec.yaml`, so its files must NOT exist. Never try to create one
- Create every artifact in the required set that is missing, then re-check - creating one can unblock others
- Skip one only when `status` already reports it `skipped`, or when its own `instruction` says it is conditional: run `openspec instructions <artifact-id> --change "<name>" --json` and skip only if its `instruction` field marks it optional (e.g. "create only if..."). Spec-driven's `design.md` qualifies; `specs` qualifies only via the `skipped` status above, never by your own judgment. Tell the user, and do not reconsider it
- Dependencies are enablers, not gates: if a required artifact is still `blocked` only because you skipped a conditional dependency, write it anyway
- Stop when every artifact in the required set is `done`, `skipped`, or was deliberately skipped
c. **If an artifact requires user input** (unclear context):
- Ask the user to clarify
- Then continue with creation
6. **Show final status**
```bash
openspec status --change "<name>"
```
**Output**
After completing all artifacts, summarize:
- Change name and location
- List of artifacts created with brief descriptions, plus any conditional artifact you skipped and why
- What's ready: "All artifacts needed for implementation are ready."
- Prompt: "The artifacts are ready for review. When you are ready, run `/opsx:apply`."
**Artifact Creation Guidelines**
- Follow the `instruction` field from `openspec instructions` for each artifact type - it is the authoritative guidance, even for familiar artifact names
- If the `instruction` field directs you to use a specific skill or command to create the artifact, invoke it instead of writing the artifact directly
- The schema defines what each artifact should contain - follow it
- Read dependency artifacts for context before creating new ones
- Use `template` as the structure for your output file - fill in its sections
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
- These guide what you write, but should never appear in the output
**Guardrails**
- The request that invoked this workflow authorizes planning only. Any implementation or apply instruction in that request does not carry forward. Do NOT implement the change, start the apply workflow, or edit project code during this workflow. After presenting the artifacts, stop and wait for a new user request to start the apply workflow
- Create every artifact the apply phase transitively depends on, not just the ids listed in `apply.requires`
- Always read dependency artifacts before creating a new one - re-read from disk, not from conversation memory (files may have changed since you last saw them)
- Ask about ambiguities that would materially change scope, externally observable behavior, compatibility, or acceptance criteria; for minor details, make reasonable assumptions and record them
- If a change with that name already exists, ask if user wants to continue it or create a new one
- Verify each artifact file exists after writing before proceeding to next
+258
View File
@@ -0,0 +1,258 @@
---
name: "OPSX: Sync"
description: "Sync delta specs from a change to main specs"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "specs", "experimental"]
---
Sync delta specs from a change to main specs.
This is an **agent-driven** operation - you will read delta specs and directly edit main specs to apply the changes. This allows intelligent merging (e.g., adding a scenario without copying the entire requirement).
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name after `/opsx:sync` (e.g., `/opsx:sync add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show changes that have delta specs (under `specs/` directory).
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:sync <other>`).
2. **Resolve change context**
Run:
```bash
openspec status --change "<name>" --json
```
The JSON includes `planningHome.root`. Main specs live under `<planningHome.root>/openspec/specs/` — use that (store-aware) root for every main-spec path below, not a hardcoded repo path. When a store is selected it points at the store, not the current repository.
3. **Find delta specs**
Use `artifactPaths.specs.existingOutputPaths` from the status JSON as the
only source of delta spec paths. If the `specs` entry is missing or
`existingOutputPaths` is empty, report that there are no delta specs to sync,
do not infer them from other artifacts, and stop without requesting artifact
instructions or writing a main spec.
Sync every path in `existingOutputPaths` unless the caller narrowed the set.
A caller narrows it by naming an explicit list of complete entries from
`existingOutputPaths` — copy those absolute values verbatim. Archive does
this inline, and a user can too (for example, by selecting the entry ending
in `/specs/billing/invoices/spec.md`).
Then sync only the named paths and leave the remaining delta specs untouched:
bulk archive excludes a delta whose implementation it could not find, and
syncing it anyway would write a main spec the caller deliberately withheld.
Carry that narrowed selection through step 4; never widen it back to the full
list. If a named path is not in `existingOutputPaths`, do not sync it —
report it and stop, rather than dropping it silently. If the named list is
empty, report that there is nothing to sync and stop without writing a main
spec.
Each delta spec file contains sections like:
- `## ADDED Requirements` - New requirements to add
- `## MODIFIED Requirements` - Changes to existing requirements
- `## REMOVED Requirements` - Requirements to remove
- `## RENAMED Requirements` - Requirements to rename (FROM:/TO: format)
If no delta specs found, inform user and stop.
4. **For each delta spec, apply changes to main specs**
Before the first main-spec write, obtain one current specs-rule snapshot:
- If archive invoked this workflow inline and supplied a valid snapshot from
`openspec instructions specs --change "<name>" --json`, reuse it and do not
fetch the same instructions again.
- Otherwise run that command once now with the same selected-root flags.
- If the direct lookup exits non-zero or returns invalid artifact-instruction
JSON, report the error and stop before writing any main spec. Do not treat the
failure as an absent rule set.
- A valid response with omitted `rules` means no artifact rules are configured
and the existing semantic merge continues.
Apply returned `rules` only to the content and form of the main specs produced
by this merge. Artifact rules are not operation guidance and cannot change
selected roots, delta paths, CLI checks, or workflow steps. Use their text as
constraints without copying it verbatim into a main spec or summary.
For each capability delta spec path selected in step 3 — the full `existingOutputPaths` list, or the narrowed subset when a caller supplied one (these may belong to a selected store, not the repo):
a. **Read the delta spec** to understand the intended changes
b. **Read the main spec** at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (may not exist yet)
c. **Apply changes intelligently**:
**ADDED Requirements:**
- If requirement doesn't exist in main spec → add it
- If requirement already exists → update it to match (treat as implicit MODIFIED)
**MODIFIED Requirements:**
- Find the requirement in main spec
- Apply the changes - this can be:
- Adding new scenarios the main spec does not have yet
- Modifying existing scenarios
- Changing the requirement description
- Preserve scenarios/content not mentioned in the delta
**REMOVED Requirements:**
- Remove the entire requirement block from main spec
- Retiring the capability. Delete the whole `spec.md` - and the directory once
nothing else is left in it - only when ALL of these hold:
1. removing the requirements *this run* left no requirement blocks;
2. the rest of the spec is well-formed (it still has a `## Purpose`);
3. the main spec was not already empty before this sync - if you removed
nothing, change nothing;
4. every other nonblank line in the whole file is accounted for as the
title, Purpose, Requirements header, or a canonical requirement's
statement, scenarios, or fenced examples;
5. the change's `.openspec.yaml` declares `retire_capabilities: true`;
6. the `spec.md` resolves inside the real specs root (do not follow a
capability-directory symlink to delete an external file).
If removing the selected requirements would leave no requirement blocks and
any retirement condition is not satisfied, do not modify the main spec. Stop
the sync for that capability, report the blocking condition, and tell the user
how to resolve it. Never write or leave an empty `## Requirements` section.
When only the marker is missing, say that too - it is the one thing the user
can add to make the retirement go through.
- Deleting the file also deletes its `## Purpose`; any other section blocks
retirement. Name Purpose when you report the retirement. Include a pasteable
`git checkout` only when the spec lived in the caller's checkout;
otherwise give checkout-scoped recovery guidance.
**RENAMED Requirements:**
- Find the FROM requirement, rename to TO
**`## Purpose` in the delta:**
- The main spec already has one and it is authoritative - leave it alone
(this is what `openspec archive` does; it warns and moves on)
d. **Create new main spec** if capability doesn't exist yet:
- Create `<planningHome.root>/openspec/specs/<capability-path>/spec.md`
- Add Purpose section: copy the delta's `## Purpose` body verbatim when it has one
(this is what `openspec archive` does); only write a brief TBD placeholder when it does not
- Add Requirements section with the ADDED requirements
- Follow the **Main Spec Format Reference** below
5. **Validate updated main specs**
Run `openspec validate --specs` with the same selected-root flags used earlier.
If validation fails, report the problems and do not claim the sync succeeded.
6. **Show summary**
After applying all changes, summarize:
- Which capabilities were updated
- What changes were made (requirements added/modified/removed/renamed)
- Any new main spec left with a TBD Purpose placeholder, so it gets written
now rather than lingering
- Any capability retired, naming the deleted `spec.md`, its Purpose, and
either a pasteable `git checkout` or checkout-scoped recovery guidance
**Delta Spec Format Reference**
```markdown
## Purpose
Only on a delta that introduces a brand-new capability. Seeds the new main spec.
## ADDED Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
## MODIFIED Requirements
### Requirement: Existing Feature
The system SHALL keep doing the existing thing, now also handling A.
#### Scenario: Scenario the main spec already has
- **WHEN** user does X
- **THEN** system does Y
#### Scenario: New scenario to add
- **WHEN** user does A
- **THEN** system does B
## REMOVED Requirements
### Requirement: Deprecated Feature
## RENAMED Requirements
- FROM: `### Requirement: Old Name`
- TO: `### Requirement: New Name`
```
**Main Spec Format Reference**
Main specs are what the delta merges INTO. They must never contain delta operation headers (`## ADDED/MODIFIED/REMOVED/RENAMED Requirements`) - after syncing, every requirement lives under a single `## Requirements` section:
```markdown
# <capability> Specification
## Purpose
Short description of what this capability does and why it exists.
## Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
```
**Key Principle: Intelligent Merging**
Unlike programmatic merging, you merge rather than overwrite:
- A MODIFIED block carries the whole requirement - body plus every scenario that survives the change. `openspec validate` and `openspec archive` both reject one that drops a scenario the main spec still has.
- Keep anything the delta does not mention, in the main spec's existing order
- Use your judgment to merge changes sensibly
**Output On Success**
```markdown
## Specs Synced: <change-name>
Updated main specs:
**<capability-1>**:
- Added requirement: "New Feature"
- Modified requirement: "Existing Feature" (added 1 scenario)
**<capability-2>**:
- Created new spec file
- Added requirement: "Another Feature"
Main specs are now updated. The change remains active - archive when implementation is complete.
```
**Guardrails**
- Read both delta and main specs before making changes
- Preserve existing content not mentioned in delta
- Never copy a delta file into a main spec as-is - merge its content so the main spec keeps the Main Spec Format Reference structure, with no delta operation headers
- If something is unclear, ask for clarification
- Show what you're changing as you go
- The operation should be idempotent - running twice should give same result
- Use only `artifactPaths.specs.existingOutputPaths`; never infer delta specs from unrelated artifacts
- Honor a caller-supplied subset of `existingOutputPaths`; never widen it back to the full list
- Fetch specs instructions once for direct sync, or reuse the archive-supplied snapshot inline
- Stop before every main-spec write on a non-zero or invalid JSON specs-instruction response
- Artifact rules constrain only the specs being written and are never copied into output files
+87
View File
@@ -0,0 +1,87 @@
---
name: "OPSX: Update"
description: "Update a change - revise existing planning artifacts and keep them coherent (Experimental)"
allowed-tools: Bash(openspec:*)
category: "Workflow"
tags: ["workflow", "artifacts", "experimental"]
---
Revise a change's existing planning artifacts and keep them coherent. Never edit code.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name after `/opsx:update` (e.g., `/opsx:update add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
`/opsx:continue` is an expanded-profile workflow and may not be installed. Before suggesting it anywhere below, verify that it is available. If it is unavailable, `openspec status --change "<name>" --json` shows the next artifact and `openspec instructions "<artifact-id>" --change "<name>" --json` explains how to create it.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes sorted by most recently modified, and ask the user to select one
When prompting, present the top 3-4 most recently modified changes as options, showing:
- Change name
- Schema (from `schema` field if present, otherwise "spec-driven")
- Status (e.g., "0/5 tasks", "complete", "no tasks")
- How recently it was modified (from `lastModified` field)
Mark the most recently modified change as "(Recommended)" since it's likely what the user wants to update.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:update <other>`).
2. **Get the change's artifacts**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand current state. The response includes:
- `schemaName`: The workflow schema being used (e.g., "spec-driven")
- `artifacts`: Array of artifacts with their status ("done", "skipped", "ready", "blocked")
- `isPlanningComplete`: Boolean indicating if all planning artifacts are complete. Older CLI versions expose the same value as `isComplete`.
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
The artifact ids and paths come from the active schema - do NOT assume them, and do NOT branch on hardcoded artifact names. Custom schemas must work unchanged.
The files to edit are `artifactPaths.<id>.existingOutputPaths` - the concrete files that exist on disk, already glob-expanded for glob artifacts (e.g. `specs/**/*.md`). Do NOT write to `resolvedOutputPath`: for a glob artifact it is still the glob pattern, not a real file.
3. **Understand the request**
- If the user asked for a specific revision ("the design now uses X"), that is the starting edit.
- If they only said "update" / "make this coherent", treat it as a coherence review: read the existing artifacts and check them against each other for contradictions, gaps, and duplication.
4. **Read and reconcile**
- Read the artifact(s) the request touches and the change's other existing artifacts.
- Apply the requested edit. Then check every other existing artifact against it - in ANY direction: an edit to a later artifact may require revising an earlier one, not only the other way around. Build order is a useful reading order, not a constraint on which artifacts may be revised.
- Note everything that is now inconsistent, missing, or contradictory.
- Revise only files that already exist (`existingOutputPaths`). Do NOT create artifacts that don't exist yet, and do NOT invent new files under a glob artifact - note them and point the user to `/opsx:continue` to create them.
- If the change is already coherent, say so and make no edits.
5. **Confirm and apply, one artifact at a time**
- Show each proposed revision and why. Write only after the user confirms.
- If the user rejects a revision, do not write it - leave that artifact unchanged.
- When a substantial rewrite is needed, get that artifact's rules and template first:
```bash
openspec instructions "<artifact-id>" --change "<name>" --json
```
6. **Point to the next step (guidance only - NEVER act on it)**
- Artifacts still missing -> suggest `/opsx:continue` to create them.
- Change already implemented (tasks checked off / already applied) -> the code may no longer match the revised plan; suggest `/opsx:apply` to carry the delta into code.
- Everything done and implemented -> suggest `/opsx:archive`.
**Output**
After each invocation, show:
- Which artifacts were revised (and which proposed revisions were rejected)
- Anything deferred to `/opsx:continue` (not-yet-created artifacts or files)
- Where the change stands and the recommended next command
**Guardrails**
- Planning artifacts only - NEVER edit implementation code. If the revised plan implies code changes, stop and point to `/opsx:apply`.
- Use the artifact ids and paths reported by `openspec status`; never branch on hardcoded artifact names.
- Edit only the concrete files in `existingOutputPaths`; never write to a glob `resolvedOutputPath`.
- Do not advance the build frontier: no new artifacts, no new files under glob artifacts - that is `/opsx:continue`'s job.
- Confirm every edit with the user before writing.
- If the request changes the change's *intent* rather than refining it, first verify whether the expanded-profile `/opsx:new` workflow is available. If it is, recommend starting fresh with `/opsx:new` (the "Update vs. Start Fresh" heuristic). If it is unavailable, ask for a distinct unused change name and recommend `openspec new change "<new-change-name>"` instead.
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
set -uo pipefail
ROOT="${CLAUDE_PROJECT_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
PAYLOAD=$(cat)
FILE=$(printf '%s' "$PAYLOAD" | /usr/bin/python3 -c 'import json,sys
try:
d = json.load(sys.stdin)
except Exception:
sys.exit(0)
t = d.get("tool_input") or {}
r = d.get("tool_response") or {}
p = r.get("filePath") or t.get("file_path") or ""
print(p)
' 2>/dev/null)
case "$FILE" in
"") exit 0 ;;
*.py) ;;
*) exit 0 ;;
esac
case "$FILE" in
"$ROOT"/src/*) ;;
src/*) FILE="$ROOT/$FILE" ;;
*) exit 0 ;;
esac
[ -f "$FILE" ] || exit 0
[ -x "$ROOT/.venv/bin/ruff" ] || exit 0
OUT=$(
cd "$ROOT" || exit 0
"$ROOT/.venv/bin/ruff" check --fix --quiet "$FILE" 2>&1
"$ROOT/.venv/bin/ruff" format --quiet "$FILE" 2>&1
if [ -x "$ROOT/.venv/bin/ty" ]; then
"$ROOT/.venv/bin/ty" check --quiet "$FILE" 2>&1 | grep -v "^All checks passed" || true
fi
)
if [ -n "$OUT" ]; then
printf 'ruff/ty по %s:\n%s\n' "${FILE#"$ROOT"/}" "$OUT" >&2
exit 2
fi
exit 0
+17
View File
@@ -0,0 +1,17 @@
{
"hooks": {
"PostToolUse": [
{
"matcher": "Write|Edit|MultiEdit|NotebookEdit",
"hooks": [
{
"type": "command",
"command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/lint-python.sh",
"timeout": 60,
"statusMessage": "ruff + ty по src/"
}
]
}
]
}
}
@@ -0,0 +1,185 @@
---
name: openspec-apply-change
description: Implement tasks from an OpenSpec change. Use when the user wants to start implementing, continue implementation, or work through tasks.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Implement tasks from an OpenSpec change.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name (e.g., `/opsx:apply add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:apply <other>`).
2. **Check status to understand the schema**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand:
- `schemaName`: The workflow being used (e.g., "spec-driven")
- `planningHome`, `changeRoot`, and `actionContext`: planning scope and edit constraints
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
3. **Get apply instructions**
```bash
openspec instructions apply --change "<name>" --json
```
This returns:
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
- Progress (total, complete, remaining)
- Task list with status
- Dynamic instruction based on current state
- Optional `context`: current required project instruction input from the selected root
- Optional `operationGuidance`: current advisory guidance for apply
**Handle states:**
- If `state: "blocked"` (missing artifacts): show message, suggest using `/opsx:continue` (if it is not installed, run `openspec status --change "<name>" --json` to see the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` for how to create it)
- If `state: "all_done"`: congratulate, suggest archive
- Otherwise: proceed to implementation
Treat `context` as a required prompt-level input. Read and consider it, and
apply relevant project facts, conventions, and constraints while implementing.
Treat `operationGuidance` as optional additive advice. Read and consider every
entry, and follow entries that are applicable and compatible with the built-in
workflow.
Keep both fields separate from CLI-returned state, missing artifacts, tasks,
progress, `contextFiles`, and the built-in `instruction`. They are not
evidence of task completion, do not replace the built-in instruction, and do
not permit bypassing a blocked state. If context conflicts with the built-in
instruction, an explicit user choice, or a CLI-controlled value, report the
conflict and preserve the controlling value. If guidance is inapplicable or
conflicts with those controlling inputs, do not follow it and explain why.
These are prompt-level behavior contracts, not enforceable checks.
4. **Read context files**
Read every file path listed under `contextFiles` from the apply instructions output.
The files depend on the schema being used:
- **spec-driven**: proposal, specs, design, tasks
- Other schemas: follow the contextFiles from CLI output
Do not copy `context` or `operationGuidance` verbatim into implementation
files or planning artifacts unless the user separately asks for that content.
5. **Show current progress**
Display:
- Schema being used
- Progress: "N/M tasks complete"
- Remaining tasks overview
- Dynamic instruction from CLI
6. **Implement tasks (loop until done or blocked)**
For each pending task:
- Show which task is being worked on
- Make the code changes required
- Keep changes minimal and focused
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
- Continue to next task
**Pause if:**
- Task is unclear → ask for clarification
- Implementation reveals a design issue → suggest updating artifacts
- Error or blocker encountered → report and wait for guidance
- User interrupts
7. **On completion or pause, show status**
Display:
- Tasks completed this session
- Overall progress: "N/M tasks complete"
- If all done: suggest archive
- If paused: explain why and wait for guidance
**Output During Implementation**
```
## Implementing: <change-name> (schema: <schema-name>)
Working on task 3/7: <task description>
[...implementation happening...]
✓ Task complete
Working on task 4/7: <task description>
[...implementation happening...]
✓ Task complete
```
**Output On Completion**
```
## Implementation Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 7/7 tasks complete ✓
### Completed This Session
- [x] Task 1
- [x] Task 2
...
All tasks complete! You can archive this change with `/opsx:archive`.
```
**Output On Pause (Issue Encountered)**
```
## Implementation Paused
**Change:** <change-name>
**Schema:** <schema-name>
**Progress:** 4/7 tasks complete
### Issue Encountered
<description of the issue>
**Options:**
1. <option 1>
2. <option 2>
3. Other approach
What would you like to do?
```
**Guardrails**
- Keep going through tasks until done or blocked
- Always read context files before starting (from the apply instructions output)
- If task is ambiguous, pause and ask before implementing
- If implementation reveals issues, pause and suggest artifact updates
- Keep code changes minimal and scoped to each task
- Update task checkbox immediately after completing each task
- Pause on errors, blockers, or unclear requirements - don't guess
- Use contextFiles from CLI output, don't assume specific file names
- Do not use context or operation guidance as proof that a task is complete
- Apply relevant project context; report conflicts with controlling workflow inputs
- Consider every guidance entry; explain any inapplicable or conflicting advice
- Do not copy runtime context or operation guidance into implementation files or planning artifacts
- Preserve CLI-controlled blocked/ready/all-done behavior and completion criteria
**Fluid Workflow Integration**
This skill supports the "actions on a change" model:
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
@@ -0,0 +1,182 @@
---
name: openspec-archive-change
description: Archive a completed change in the experimental workflow. Use when the user wants to finalize and archive a change after implementation is complete.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Archive a completed change in the experimental workflow.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show only active changes (not already archived).
Include the schema used for each change if available.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:archive <other>`).
**Load current archive inputs before the existing archive checks:**
After resolving the selected change and planning root, run:
```bash
openspec instructions archive --change "<name>" --json
```
Keep the same selected-root flags on this command. This lookup is advisory and
optional: it only supplies extra prompt inputs, so it must never block archiving.
If it exits non-zero or returns invalid JSON — for example on an older CLI that
does not support this command yet — continue the archive workflow with no
context and no operation guidance. Do not report an error and do not stop.
A successful response may omit both optional fields. Treat `context` as a
required prompt-level input: read and consider it, and apply relevant project
facts, conventions, and constraints. Treat `operationGuidance` as optional
additive advice: read and consider every entry, and follow entries that are
applicable and compatible with the built-in archive workflow.
Keep both fields separate from built-in steps, explicit user choices, resolved
paths, CLI checks, and command contracts. If context conflicts with one of those
controlling inputs, report the conflict and preserve the controlling value. If
guidance is inapplicable or conflicts with a controlling input, do not follow it
and explain why. Do not infer replacement paths, skipped prompts, or flags from
either field, and do not copy their text verbatim into specs, change artifacts,
or archive summaries unless the user separately asks for it. These are
prompt-level behavior contracts, not enforceable checks.
2. **Check artifact completion status**
Run `openspec status --change "<name>" --json` to check artifact completion.
Parse the JSON to understand:
- `schemaName`: The workflow being used
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context
- `artifacts`: List of artifacts with their status (`done`, `skipped`, or other)
**If any artifacts are neither `done` nor `skipped`** (skipped artifacts satisfy the requirement - the change declares skip_specs):
- Display warning listing incomplete artifacts
- Ask the user to confirm they want to proceed
- Proceed if user confirms
3. **Check task completion status**
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
**If incomplete tasks found:**
- Display warning showing count of incomplete tasks
- Ask the user to confirm they want to proceed
- Proceed if user confirms
**If no tasks file exists:** Proceed without task-related warning.
4. **Assess delta spec sync state**
Use `artifactPaths.specs.existingOutputPaths` from status JSON as the only
delta-spec source. If the `specs` entry is missing or
`existingOutputPaths` is empty, proceed without a sync prompt and do not infer
delta specs from other artifacts.
**If delta specs exist:**
- Compare each delta spec with its corresponding main spec at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (use the store-aware `planningHome.root` from step 2, not a hardcoded repo path)
- Determine what changes would be applied (adds, modifications, removals, renames)
- Show a combined summary before prompting
**Prompt options:**
- If changes needed: "Sync now (recommended)", "Archive without syncing"
- If already synced: "Archive now", "Sync anyway", "Cancel"
Route on the answer:
- "Cancel" — stop, do not archive
- "Archive without syncing" or "Archive now" — proceed to archive
- "Sync now" or "Sync anyway" — sync, then verify (below)
- Anything else — ask again rather than archiving
Before a selected sync writes any main spec, run
`openspec instructions specs --change "<name>" --json` once with the same
selected-root flags. Require a zero exit status and valid artifact-instruction
JSON. If the lookup fails or returns invalid JSON, report the error and stop
before writing any main spec or moving the change. A valid response with omitted
`rules` is the no-rules case. Apply returned `rules` only to the content and
form of main specs produced by this merge; do not use them as archive guidance,
change CLI behavior, or copy the rule text into any output file.
Then run the `openspec-sync-specs` workflow inline (agent-driven intelligent merge) for change '<name>', passing the delta spec analysis and the fetched specs-rule snapshot from above, and wait for it to finish. The inline sync must reuse that snapshot without fetching `specs` instructions again. Do not delegate it to a background task — step 5 would move `changeRoot` out from under a sync that is still reading it, leaving the change archived and the main specs never updated. If your agent can only run it by delegation, delegate synchronously and wait for the result.
Then re-run the comparison from the top of this step against every capability that has a delta spec in `artifactPaths.specs.existingOutputPaths` — not only the ones the sync reports it touched. A successful sync leaves nothing left to apply, so each capability must now read as already synced:
- ADDED requirements present
- MODIFIED requirements carrying the scenario and description changes named in the delta, with their other scenarios intact
- REMOVED requirements gone — and where this sync retired a capability (removed its last requirement, leaving `## Requirements` empty), its main spec deleted rather than left empty; a spec the sync deliberately kept and reported is also a match
- RENAMED requirements present under the new name and absent under the old one
If the sync failed, or any capability does not match, report what differs and stop — do not archive. Nothing has moved and `changeRoot` is intact, so the user can fix the mismatch or re-run the sync and start the archive again.
5. **Perform the archive**
Create an `archive` directory under `planningHome.changesDir` if it doesn't exist:
```bash
mkdir -p "<planningHome.changesDir>/archive"
```
Generate the target name: use the change name as-is when it already starts with a `YYYY-MM-DD-` prefix; otherwise prepend the current date as `YYYY-MM-DD-<change-name>`. Never stack a second date (same rule as `openspec archive`).
**Check if target already exists:**
- If yes: Fail with error, suggest renaming existing archive or using different date
- If no: Move `changeRoot` to the archive directory
```bash
mv "<changeRoot>" "<planningHome.changesDir>/archive/<target-name>"
```
6. **Display summary**
Show archive completion summary including:
- Change name
- Schema that was used
- Archive location
- Whether specs were synced (if applicable)
- Note about any warnings (incomplete artifacts/tasks)
**Output On Success**
```markdown
## Archive Complete
**Change:** <change-name>
**Schema:** <schema-name>
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
**Specs:** <"✓ Synced to main specs" only if the step 4 verification passed; otherwise "No delta specs" or "Sync skipped">
<"All artifacts complete. All tasks complete." — or, if archived with warnings, list them instead (e.g. "Archived with 2 incomplete tasks")>
```
**Guardrails**
- Announce the selected change; prompt for selection when it is ambiguous
- Use artifact graph (openspec status --json) for completion checking
- Don't block archive on warnings - just inform and confirm
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
- Show clear summary of what happened
- If sync is requested, run the `openspec-sync-specs` workflow inline (agent-driven)
- Never archive while a spec sync is still in flight — run the sync inline and verify the main specs before moving `changeRoot`
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
- Apply relevant runtime context and report conflicts; operation guidance remains advisory
- Consider every guidance entry and explain any inapplicable or conflicting advice
- Existing CLI checks, resolved paths, prompts, and command contracts are unchanged
- Artifact rules constrain only the specs being written and are never operation guidance
- Never copy runtime context, operation guidance, or artifact-rule text verbatim into output files
+308
View File
@@ -0,0 +1,308 @@
---
name: openspec-explore
description: Enter explore mode - a thinking partner for exploring ideas, investigating problems, and clarifying requirements. Use when the user wants to think through something before or during a change.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing. For a new change, scaffold it first as described below.
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
---
## The Stance
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
- **Adaptive** - Follow interesting threads, pivot when new information emerges
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
---
## What You Might Do
Depending on what the user brings, you might:
**Explore the problem space**
- Ask clarifying questions that emerge from what they said
- Challenge assumptions
- Reframe the problem
- Find analogies
**Investigate the codebase**
- Map existing architecture relevant to the discussion
- Find integration points
- Identify patterns already in use
- Surface hidden complexity
**Compare options**
- Brainstorm multiple approaches
- Build comparison tables
- Sketch tradeoffs
- Recommend a path (if asked)
**Visualize**
```
┌─────────────────────────────────────────┐
│ Use ASCII diagrams liberally │
├─────────────────────────────────────────┤
│ │
│ ┌────────┐ ┌────────┐ │
│ │ State │────────▶│ State │ │
│ │ A │ │ B │ │
│ └────────┘ └────────┘ │
│ │
│ System diagrams, state machines, │
│ data flows, architecture sketches, │
│ dependency graphs, comparison tables │
│ │
└─────────────────────────────────────────┘
```
**Surface risks and unknowns**
- Identify what could go wrong
- Find gaps in understanding
- Suggest spikes or investigations
---
## OpenSpec Awareness
You have full context of the OpenSpec system. Use it naturally, don't force it.
### Check for context
At the start, quickly check what exists:
```bash
openspec list --json
```
This tells you:
- If there are active changes
- Their names, schemas, and status
- What the user might be working on
Then read the project's own context from the resolved root - `<root.path>/openspec/config.yaml` (or `config.yml`). Use the `root.path` returned above, and skip this if neither file exists:
- `context`: project background - tech stack, conventions, constraints
- `rules`: keyed by artifact id - the entries for an artifact apply only when you write that artifact
Ground your thinking in these. They are constraints for you to follow, not content to reproduce: do NOT copy them into the conversation or into any artifact you create.
### When no change exists
Think freely. When insights crystallize, you might offer:
- "This feels solid enough to start a change. Want me to create a proposal?"
- Or keep exploring - no pressure to formalize
If the user asks you to capture the exploration as a new change, transition seamlessly into the requested capture:
1. Run `openspec new change "<name>"` (with `--store <id>` when applicable) before creating any artifacts. Never create a new change directory under `openspec/changes/` by hand; the CLI scaffold creates required metadata such as `.openspec.yaml`. Keep the selected `--store <id>` on every applicable follow-up `status` and `instructions` command.
2. Run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store), then process the requested artifacts in dependency order. For each requested artifact that is `ready`, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store). Before creating a requested artifact, evaluate any condition in its own `instruction` against the explored change; record a deliberate skip instead when the condition does not apply. If a requested artifact is blocked by a direct prerequisite the user did not request, run `openspec instructions "<prerequisite-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) for that prerequisite whether it is `ready` or `blocked`. If its own `instruction` states a condition, evaluate that condition against the explored change and record a deliberate skip only when the condition does not apply. If the condition applies, or the prerequisite is not conditional, treat it as a normal prerequisite and ask before expanding the capture. Do not create an unrequested prerequisite unless the user approves.
3. Follow the returned `template` and `instruction` fields. Read completed dependency files listed in `dependencies`, and apply `context` and `rules` as constraints without copying them into the artifact. If the instruction delegates creation to a specific skill or command, invoke it; otherwise write the artifact to `resolvedOutputPath`, using the instruction to choose a concrete path when it is a glob. Verify that the selected concrete output exists.
4. After creating each artifact, re-run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) and continue until every requested artifact is `done`, `skipped`, or was deliberately skipped because its own `instruction` stated a condition that did not apply. Tell the user about a deliberate conditional skip, remember it, and do not reconsider it. Dependencies are enablers, not gates: if a requested artifact is still `blocked` only because you deliberately skipped a conditional prerequisite, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) despite the blocked status, then create it using step 3 only when those recorded conditional skips are its sole missing dependencies. If a requested artifact is blocked by a prerequisite the user did not ask to capture and cannot be conditionally skipped, explain that dependency and ask before expanding the capture.
Capture the artifact(s) the user requested without asking them to invoke another workflow command. If they asked only to start a change, stop after scaffolding and show its status.
### When a change exists
If the user mentions a change or you detect one is relevant:
1. **Resolve and read existing artifacts for context**
- Run `openspec status --change "<name>" --json`.
- Use `changeRoot`, `artifactPaths`, and `actionContext` from the status JSON.
- Read existing files from `artifactPaths.<artifact>.existingOutputPaths`.
2. **Reference them naturally in conversation**
- "Your design mentions using Redis, but we just realized SQLite fits better..."
- "The proposal scopes this to premium users, but we're now thinking everyone..."
3. **Offer to capture when decisions are made**
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
| Insight Type | Where to Capture |
|----------------------------|-------------------------------------|
| New requirement discovered | `specs/<capability-path>/spec.md` |
| Requirement changed | `specs/<capability-path>/spec.md` |
| Design decision made | `design.md` |
| Scope changed | `proposal.md` |
| New work identified | `tasks.md` |
| Assumption invalidated | Relevant artifact |
Example offers:
- "That's a design decision. Capture it in design.md?"
- "This is a new requirement. Add it to specs?"
- "This changes scope. Update the proposal?"
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
---
## What You Don't Have To Do
- Follow a script
- Ask the same questions every time
- Produce a specific artifact
- Reach a conclusion
- Stay on topic if a tangent is valuable
- Be brief (this is thinking time)
---
## Handling Different Entry Points
**User brings a vague idea:**
```
User: I'm thinking about adding real-time collaboration
You: Real-time collab is a big space. Let me think about this...
COLLABORATION SPECTRUM
════════════════════════════════════════════
Awareness Coordination Sync
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│Presence│ │Cursors │ │ CRDT │
│ "3 │ │ Multi │ │Conflict│
│online" │ │ select │ │ free │
└────────┘ └────────┘ └────────┘
│ │ │
trivial moderate complex
Where's your head at?
```
**User brings a specific problem:**
```
User: The auth system is a mess
You: [reads codebase]
┌─────────────────────────────────────────────┐
│ CURRENT AUTH FLOW │
└─────────────────────────────────────────────┘
│
┌─────────────┼─────────────┐
▼ ▼ ▼
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Google │ │ GitHub │ │ Email │
│ OAuth │ │ OAuth │ │ Magic │
└────┬────┘ └────┬────┘ └────┬────┘
│ │ │
└─────────────┼─────────────┘
▼
┌───────────┐
│ Session │
└─────┬─────┘
│
▼
┌───────────┐
│ Perms │
└───────────┘
I see three tangles. Which one's burning?
```
**User is stuck mid-implementation:**
```
User: /opsx:explore add-auth-system
The OAuth integration is more complex than expected
You: [reads change artifacts]
You're on task 4: "Implement OAuth flow"
Let me trace what's involved...
[draws diagram, explores options, suggests paths]
Want to update the design to reflect this?
Or add a spike task to investigate?
```
**User wants to compare options:**
```
User: Should we use Postgres or SQLite?
You: Generic answer is boring. What's the context?
User: A CLI tool that tracks local dev environments
You: That changes everything.
┌─────────────────────────────────────────────────┐
│ CLI TOOL DATA STORAGE │
└─────────────────────────────────────────────────┘
Key constraints:
• No daemon running
• Must work offline
• Single user
SQLite Postgres
Deployment embedded ✓ needs server ✗
Offline yes ✓ no ✗
Single file yes ✓ no ✗
SQLite. Not even close.
Unless... is there a sync component?
```
---
## Ending Discovery
There's no required ending. Discovery might:
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
- **Result in artifact updates**: "Updated design.md with these decisions"
- **Just provide clarity**: User has what they need, moves on
- **Continue later**: "We can pick this up anytime"
When it feels like things are crystallizing, you might summarize:
```
## What We Figured Out
**The problem**: [crystallized understanding]
**The approach**: [if one emerged]
**Open questions**: [if any remain]
**Next steps** (if ready):
- Create a change proposal
- Keep exploring: just keep talking
```
But this summary is optional. Sometimes the thinking IS the value.
---
## Guardrails
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
- **Don't fake understanding** - If something is unclear, dig deeper
- **Don't rush** - Discovery is thinking time, not task time
- **Don't force structure** - Let patterns emerge naturally
- **Don't auto-capture** - Offer to save insights, don't just do it
- **Don't manually scaffold changes** - Never create a new change directory under `openspec/changes/` by hand. Always use `openspec new change "<name>"` (with `--store <id>` when applicable) so required metadata such as `.openspec.yaml` is created before writing artifacts.
- **Do visualize** - A good diagram is worth many paragraphs
- **Do explore the codebase** - Ground discussions in reality
- **Do question assumptions** - Including the user's and your own
+149
View File
@@ -0,0 +1,149 @@
---
name: openspec-propose
description: Propose a new change with all artifacts generated in one step. Use when the user wants to quickly describe what they want to build and get a complete proposal with design, specs, and tasks ready for implementation.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Propose a new change - create the change and generate all artifacts in one step.
**Planning boundary**: This workflow creates planning artifacts only. The user request that selected or triggered this workflow authorizes planning only, even if it asks to build or fix something. Do not edit project code. After the planning artifacts are complete, stop. Do not start implementation in the same response, even if the initial request asks for it. Wait for a new user request after the artifacts are presented; then start the apply workflow.
I'll create a change with the artifacts your schema defines. With the default spec-driven schema that is:
- proposal.md (what & why)
- `specs/<capability-path>/spec.md` (what the system must do - a delta, not the main spec)
- design.md (how)
- tasks.md (implementation steps)
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
When the user is ready to implement, they must start the apply workflow explicitly.
---
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: The user's request should include a change name (kebab-case) OR a description of what they want to build.
**Steps**
1. **Understand the request and clarify material ambiguity**
If no clear input is provided, ask the user (open-ended, no preset options):
> "What change do you want to work on? Describe what you want to build or fix."
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
If the request contains ambiguity that would materially affect scope, externally observable behavior, compatibility, or acceptance criteria, ask the user before creating the change. For minor details, make a reasonable assumption and record it in the planning artifacts.
2. **Determine the workflow schema**
Use the configured default schema unless the user explicitly requests a different workflow.
**Use a different schema only if the user:**
- Explicitly requests a specific schema by name → use `--schema <schema-name>`
- Asks to "show workflows" or asks "what workflows" exist → resolve the authoritative root by running `openspec context --json` from the current working directory. If the user explicitly selected a registered store, use `openspec context --json --store "<store-id>"`. Then run `openspec schemas --json` with its working directory set to the returned `root.path` and let them choose. This preserves roots selected by a local `store:` pointer or the global `defaultStore`; `schemas` does not accept `--store`. If context reports only `no_openspec_root`, run `openspec schemas --json` from the current working directory instead. Do not use this fallback for invalid or unavailable stores.
Otherwise, omit `--schema` to preserve the configured default.
3. **Create the change directory**
Choose one schema form below. If a registered store is selected, append `--store "<store-id>"` to that command and each later OpenSpec command shown below that accepts `--store`.
Using the configured default:
```bash
openspec new change "<name>"
```
Using an explicitly requested schema:
```bash
openspec new change "<name>" --schema "<schema-name>"
```
This creates a scaffolded change in the planning home resolved by the CLI with `.openspec.yaml`.
4. **Get the artifact build order**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to get:
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
- `artifacts`: list of all artifacts, each with its `status` and its `requires` edges (the artifact IDs it directly depends on)
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
5. **Create every artifact in the required set**
Use a todo list to track progress through the artifacts.
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
a. **For each artifact that is `ready` (dependencies satisfied)**:
- Get instructions:
```bash
openspec instructions <artifact-id> --change "<name>" --json
```
- The instructions JSON includes:
- `context`: Project background (constraints for you - do NOT include in output)
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
- `template`: The structure to use for your output file
- `instruction`: Schema-specific guidance for this artifact type
- `skipped`/`warning`: present when the change declares skip_specs and this artifact must NOT be created - stop and pick another artifact
- `resolvedOutputPath`: Resolved path or pattern to write the artifact
- `dependencies`: Completed artifacts to read for context
- Read any completed dependency files for context - always re-read them from disk, even if you saw them earlier in the conversation (the user may have edited them)
- If the `instruction` field delegates creation to a specific skill or command, invoke it to produce the artifact instead of writing the file yourself, then verify the artifact file exists at `resolvedOutputPath`
- Otherwise create the artifact file using `template` as the structure and write it to `resolvedOutputPath`. If `resolvedOutputPath` is a glob, follow `instruction` to choose the concrete file path
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
- Show brief progress: "Created <artifact-id>"
b. **Continue until every artifact in the required set exists (not just `apply.requires`)**
- After creating each artifact, re-run `openspec status --change "<name>" --json`
- The required set is `applyRequires` plus every artifact reachable from those by following the `requires` edges in `status --json` - walk them transitively (spec-driven closes over proposal, specs, design, tasks). Leave artifacts outside that set alone
- `status` is file-existence only, so an `applyRequires` artifact reading `done` does NOT mean its dependencies exist - writing `tasks.md` early marks `tasks` done while `specs` was never written. Use each artifact's `requires` edges, not its `status`, to build the required set: a `done` artifact still lists what it depends on
- An artifact already reading `status: "skipped"` is satisfied: the change declares `skip_specs` in `.openspec.yaml`, so its files must NOT exist. Never try to create one
- Create every artifact in the required set that is missing, then re-check - creating one can unblock others
- Skip one only when `status` already reports it `skipped`, or when its own `instruction` says it is conditional: run `openspec instructions <artifact-id> --change "<name>" --json` and skip only if its `instruction` field marks it optional (e.g. "create only if..."). Spec-driven's `design.md` qualifies; `specs` qualifies only via the `skipped` status above, never by your own judgment. Tell the user, and do not reconsider it
- Dependencies are enablers, not gates: if a required artifact is still `blocked` only because you skipped a conditional dependency, write it anyway
- Stop when every artifact in the required set is `done`, `skipped`, or was deliberately skipped
c. **If an artifact requires user input** (unclear context):
- Ask the user to clarify
- Then continue with creation
6. **Show final status**
```bash
openspec status --change "<name>"
```
**Output**
After completing all artifacts, summarize:
- Change name and location
- List of artifacts created with brief descriptions, plus any conditional artifact you skipped and why
- What's ready: "All artifacts needed for implementation are ready."
- Prompt: "The artifacts are ready for review. When you are ready, run `/opsx:apply` or ask me to apply this change."
**Artifact Creation Guidelines**
- Follow the `instruction` field from `openspec instructions` for each artifact type - it is the authoritative guidance, even for familiar artifact names
- If the `instruction` field directs you to use a specific skill or command to create the artifact, invoke it instead of writing the artifact directly
- The schema defines what each artifact should contain - follow it
- Read dependency artifacts for context before creating new ones
- Use `template` as the structure for your output file - fill in its sections
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
- These guide what you write, but should never appear in the output
**Guardrails**
- The request that invoked this workflow authorizes planning only. Any implementation or apply instruction in that request does not carry forward. Do NOT implement the change, start the apply workflow, or edit project code during this workflow. After presenting the artifacts, stop and wait for a new user request to start the apply workflow
- Create every artifact the apply phase transitively depends on, not just the ids listed in `apply.requires`
- Always read dependency artifacts before creating a new one - re-read from disk, not from conversation memory (files may have changed since you last saw them)
- Ask about ambiguities that would materially change scope, externally observable behavior, compatibility, or acceptance criteria; for minor details, make reasonable assumptions and record them
- If a change with that name already exists, ask if user wants to continue it or create a new one
- Verify each artifact file exists after writing before proceeding to next
+262
View File
@@ -0,0 +1,262 @@
---
name: openspec-sync-specs
description: Sync delta specs from a change to main specs. Use when the user wants to update main specs with changes from a delta spec, without archiving the change.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Sync delta specs from a change to main specs.
This is an **agent-driven** operation - you will read delta specs and directly edit main specs to apply the changes. This allows intelligent merging (e.g., adding a scenario without copying the entire requirement).
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
When prompting, show changes that have delta specs (under `specs/` directory).
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:sync <other>`).
2. **Resolve change context**
Run:
```bash
openspec status --change "<name>" --json
```
The JSON includes `planningHome.root`. Main specs live under `<planningHome.root>/openspec/specs/` — use that (store-aware) root for every main-spec path below, not a hardcoded repo path. When a store is selected it points at the store, not the current repository.
3. **Find delta specs**
Use `artifactPaths.specs.existingOutputPaths` from the status JSON as the
only source of delta spec paths. If the `specs` entry is missing or
`existingOutputPaths` is empty, report that there are no delta specs to sync,
do not infer them from other artifacts, and stop without requesting artifact
instructions or writing a main spec.
Sync every path in `existingOutputPaths` unless the caller narrowed the set.
A caller narrows it by naming an explicit list of complete entries from
`existingOutputPaths` — copy those absolute values verbatim. Archive does
this inline, and a user can too (for example, by selecting the entry ending
in `/specs/billing/invoices/spec.md`).
Then sync only the named paths and leave the remaining delta specs untouched:
bulk archive excludes a delta whose implementation it could not find, and
syncing it anyway would write a main spec the caller deliberately withheld.
Carry that narrowed selection through step 4; never widen it back to the full
list. If a named path is not in `existingOutputPaths`, do not sync it —
report it and stop, rather than dropping it silently. If the named list is
empty, report that there is nothing to sync and stop without writing a main
spec.
Each delta spec file contains sections like:
- `## ADDED Requirements` - New requirements to add
- `## MODIFIED Requirements` - Changes to existing requirements
- `## REMOVED Requirements` - Requirements to remove
- `## RENAMED Requirements` - Requirements to rename (FROM:/TO: format)
If no delta specs found, inform user and stop.
4. **For each delta spec, apply changes to main specs**
Before the first main-spec write, obtain one current specs-rule snapshot:
- If archive invoked this workflow inline and supplied a valid snapshot from
`openspec instructions specs --change "<name>" --json`, reuse it and do not
fetch the same instructions again.
- Otherwise run that command once now with the same selected-root flags.
- If the direct lookup exits non-zero or returns invalid artifact-instruction
JSON, report the error and stop before writing any main spec. Do not treat the
failure as an absent rule set.
- A valid response with omitted `rules` means no artifact rules are configured
and the existing semantic merge continues.
Apply returned `rules` only to the content and form of the main specs produced
by this merge. Artifact rules are not operation guidance and cannot change
selected roots, delta paths, CLI checks, or workflow steps. Use their text as
constraints without copying it verbatim into a main spec or summary.
For each capability delta spec path selected in step 3 — the full `existingOutputPaths` list, or the narrowed subset when a caller supplied one (these may belong to a selected store, not the repo):
a. **Read the delta spec** to understand the intended changes
b. **Read the main spec** at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (may not exist yet)
c. **Apply changes intelligently**:
**ADDED Requirements:**
- If requirement doesn't exist in main spec → add it
- If requirement already exists → update it to match (treat as implicit MODIFIED)
**MODIFIED Requirements:**
- Find the requirement in main spec
- Apply the changes - this can be:
- Adding new scenarios the main spec does not have yet
- Modifying existing scenarios
- Changing the requirement description
- Preserve scenarios/content not mentioned in the delta
**REMOVED Requirements:**
- Remove the entire requirement block from main spec
- Retiring the capability. Delete the whole `spec.md` - and the directory once
nothing else is left in it - only when ALL of these hold:
1. removing the requirements *this run* left no requirement blocks;
2. the rest of the spec is well-formed (it still has a `## Purpose`);
3. the main spec was not already empty before this sync - if you removed
nothing, change nothing;
4. every other nonblank line in the whole file is accounted for as the
title, Purpose, Requirements header, or a canonical requirement's
statement, scenarios, or fenced examples;
5. the change's `.openspec.yaml` declares `retire_capabilities: true`;
6. the `spec.md` resolves inside the real specs root (do not follow a
capability-directory symlink to delete an external file).
If removing the selected requirements would leave no requirement blocks and
any retirement condition is not satisfied, do not modify the main spec. Stop
the sync for that capability, report the blocking condition, and tell the user
how to resolve it. Never write or leave an empty `## Requirements` section.
When only the marker is missing, say that too - it is the one thing the user
can add to make the retirement go through.
- Deleting the file also deletes its `## Purpose`; any other section blocks
retirement. Name Purpose when you report the retirement. Include a pasteable
`git checkout` only when the spec lived in the caller's checkout;
otherwise give checkout-scoped recovery guidance.
**RENAMED Requirements:**
- Find the FROM requirement, rename to TO
**`## Purpose` in the delta:**
- The main spec already has one and it is authoritative - leave it alone
(this is what `openspec archive` does; it warns and moves on)
d. **Create new main spec** if capability doesn't exist yet:
- Create `<planningHome.root>/openspec/specs/<capability-path>/spec.md`
- Add Purpose section: copy the delta's `## Purpose` body verbatim when it has one
(this is what `openspec archive` does); only write a brief TBD placeholder when it does not
- Add Requirements section with the ADDED requirements
- Follow the **Main Spec Format Reference** below
5. **Validate updated main specs**
Run `openspec validate --specs` with the same selected-root flags used earlier.
If validation fails, report the problems and do not claim the sync succeeded.
6. **Show summary**
After applying all changes, summarize:
- Which capabilities were updated
- What changes were made (requirements added/modified/removed/renamed)
- Any new main spec left with a TBD Purpose placeholder, so it gets written
now rather than lingering
- Any capability retired, naming the deleted `spec.md`, its Purpose, and
either a pasteable `git checkout` or checkout-scoped recovery guidance
**Delta Spec Format Reference**
```markdown
## Purpose
Only on a delta that introduces a brand-new capability. Seeds the new main spec.
## ADDED Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
## MODIFIED Requirements
### Requirement: Existing Feature
The system SHALL keep doing the existing thing, now also handling A.
#### Scenario: Scenario the main spec already has
- **WHEN** user does X
- **THEN** system does Y
#### Scenario: New scenario to add
- **WHEN** user does A
- **THEN** system does B
## REMOVED Requirements
### Requirement: Deprecated Feature
## RENAMED Requirements
- FROM: `### Requirement: Old Name`
- TO: `### Requirement: New Name`
```
**Main Spec Format Reference**
Main specs are what the delta merges INTO. They must never contain delta operation headers (`## ADDED/MODIFIED/REMOVED/RENAMED Requirements`) - after syncing, every requirement lives under a single `## Requirements` section:
```markdown
# <capability> Specification
## Purpose
Short description of what this capability does and why it exists.
## Requirements
### Requirement: New Feature
The system SHALL do something new.
#### Scenario: Basic case
- **WHEN** user does X
- **THEN** system does Y
```
**Key Principle: Intelligent Merging**
Unlike programmatic merging, you merge rather than overwrite:
- A MODIFIED block carries the whole requirement - body plus every scenario that survives the change. `openspec validate` and `openspec archive` both reject one that drops a scenario the main spec still has.
- Keep anything the delta does not mention, in the main spec's existing order
- Use your judgment to merge changes sensibly
**Output On Success**
```markdown
## Specs Synced: <change-name>
Updated main specs:
**<capability-1>**:
- Added requirement: "New Feature"
- Modified requirement: "Existing Feature" (added 1 scenario)
**<capability-2>**:
- Created new spec file
- Added requirement: "Another Feature"
Main specs are now updated. The change remains active - archive when implementation is complete.
```
**Guardrails**
- Read both delta and main specs before making changes
- Preserve existing content not mentioned in delta
- Never copy a delta file into a main spec as-is - merge its content so the main spec keeps the Main Spec Format Reference structure, with no delta operation headers
- If something is unclear, ask for clarification
- Show what you're changing as you go
- The operation should be idempotent - running twice should give same result
- Use only `artifactPaths.specs.existingOutputPaths`; never infer delta specs from unrelated artifacts
- Honor a caller-supplied subset of `existingOutputPaths`; never widen it back to the full list
- Fetch specs instructions once for direct sync, or reuse the archive-supplied snapshot inline
- Stop before every main-spec write on a non-zero or invalid JSON specs-instruction response
- Artifact rules constrain only the specs being written and are never copied into output files
@@ -0,0 +1,91 @@
---
name: openspec-update-change
description: Update an OpenSpec change by revising its existing planning artifacts and keeping them coherent with one another. Use when the user wants to revise a change's plan, fold new decisions into it, or reconcile its artifacts after an edit. Never edits code.
allowed-tools: Bash(openspec:*)
license: MIT
compatibility: Requires openspec CLI.
metadata:
author: openspec
version: "1.0"
generatedBy: "1.8.0"
---
Revise a change's existing planning artifacts and keep them coherent. Never edit code.
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
`/opsx:continue` is an expanded-profile workflow and may not be installed. Before suggesting it anywhere below, verify that it is available. If it is unavailable, `openspec status --change "<name>" --json` shows the next artifact and `openspec instructions "<artifact-id>" --change "<name>" --json` explains how to create it.
**Steps**
1. **Select the change**
If a name is provided, use it. Otherwise:
- Infer from conversation context if the user mentioned a change
- Auto-select if only one active change exists
- If ambiguous, run `openspec list --json` to get available changes sorted by most recently modified, and ask the user to select one
When prompting, present the top 3-4 most recently modified changes as options, showing:
- Change name
- Schema (from `schema` field if present, otherwise "spec-driven")
- Status (e.g., "0/5 tasks", "complete", "no tasks")
- How recently it was modified (from `lastModified` field)
Mark the most recently modified change as "(Recommended)" since it's likely what the user wants to update.
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:update <other>`).
2. **Get the change's artifacts**
```bash
openspec status --change "<name>" --json
```
Parse the JSON to understand current state. The response includes:
- `schemaName`: The workflow schema being used (e.g., "spec-driven")
- `artifacts`: Array of artifacts with their status ("done", "skipped", "ready", "blocked")
- `isPlanningComplete`: Boolean indicating if all planning artifacts are complete. Older CLI versions expose the same value as `isComplete`.
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
The artifact ids and paths come from the active schema - do NOT assume them, and do NOT branch on hardcoded artifact names. Custom schemas must work unchanged.
The files to edit are `artifactPaths.<id>.existingOutputPaths` - the concrete files that exist on disk, already glob-expanded for glob artifacts (e.g. `specs/**/*.md`). Do NOT write to `resolvedOutputPath`: for a glob artifact it is still the glob pattern, not a real file.
3. **Understand the request**
- If the user asked for a specific revision ("the design now uses X"), that is the starting edit.
- If they only said "update" / "make this coherent", treat it as a coherence review: read the existing artifacts and check them against each other for contradictions, gaps, and duplication.
4. **Read and reconcile**
- Read the artifact(s) the request touches and the change's other existing artifacts.
- Apply the requested edit. Then check every other existing artifact against it - in ANY direction: an edit to a later artifact may require revising an earlier one, not only the other way around. Build order is a useful reading order, not a constraint on which artifacts may be revised.
- Note everything that is now inconsistent, missing, or contradictory.
- Revise only files that already exist (`existingOutputPaths`). Do NOT create artifacts that don't exist yet, and do NOT invent new files under a glob artifact - note them and point the user to `/opsx:continue` to create them.
- If the change is already coherent, say so and make no edits.
5. **Confirm and apply, one artifact at a time**
- Show each proposed revision and why. Write only after the user confirms.
- If the user rejects a revision, do not write it - leave that artifact unchanged.
- When a substantial rewrite is needed, get that artifact's rules and template first:
```bash
openspec instructions "<artifact-id>" --change "<name>" --json
```
6. **Point to the next step (guidance only - NEVER act on it)**
- Artifacts still missing -> suggest `/opsx:continue` to create them.
- Change already implemented (tasks checked off / already applied) -> the code may no longer match the revised plan; suggest `/opsx:apply` to carry the delta into code.
- Everything done and implemented -> suggest `/opsx:archive`.
**Output**
After each invocation, show:
- Which artifacts were revised (and which proposed revisions were rejected)
- Anything deferred to `/opsx:continue` (not-yet-created artifacts or files)
- Where the change stands and the recommended next command
**Guardrails**
- Planning artifacts only - NEVER edit implementation code. If the revised plan implies code changes, stop and point to `/opsx:apply`.
- Use the artifact ids and paths reported by `openspec status`; never branch on hardcoded artifact names.
- Edit only the concrete files in `existingOutputPaths`; never write to a glob `resolvedOutputPath`.
- Do not advance the build frontier: no new artifacts, no new files under glob artifacts - that is `/opsx:continue`'s job.
- Confirm every edit with the user before writing.
- If the request changes the change's *intent* rather than refining it, first verify whether the expanded-profile `/opsx:new` workflow is available. If it is, recommend starting fresh with `/opsx:new` (the "Update vs. Start Fresh" heuristic). If it is unavailable, ask for a distinct unused change name and recommend `openspec new change "<new-change-name>"` instead.
+26
View File
@@ -14,7 +14,33 @@ backups
*.pfx
*.jks
.venv
venv
__pycache__
*.pyc
*.pyo
.pytest_cache
.ruff_cache
.mypy_cache
.ty_cache
node_modules
evidence
openspec
student-tests
tests
scripts
examples
contracts
infra
*.md
!README.md
Makefile
Dockerfile*
compose*.yaml
.dockerignore
.gitignore
.DS_Store
lab.json
requirements-dev.txt
.claude
.verification/
+95
View File
@@ -9,3 +9,98 @@ DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
S3_ENDPOINT=http://s3:9000
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
BROKER_URL_FILE=/run/secrets/broker_url
WORKER_CONCURRENCY=2
JOB_MAX_ATTEMPTS=5
JOB_TIMEOUT_SECONDS=60
STAGING_RETENTION_SECONDS=86400
OTEL_SERVICE_NAME=messenger-api
# Worker must set a distinct service.name.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
# Solution-specific: published LB ports, replica limits, workers and pool per replica,
# LB queue per replica, drain window before closing the listener on SIGTERM, PostgreSQL
# limits, load generator limits (profile bench), cursor key file. API_PORT/SHORTTTL_PORT of lab2 are gone:
# only the load balancer is published.
LB_HTTP_PORT=18086
LB_STATS_PORT=26085
API_CPUS=1
API_MEMORY=512m
WEB_CONCURRENCY=2
DB_POOL_MAX_SIZE=8
API_SERVER_MAXCONN=64
DRAIN_SECONDS=2
PG_MAX_CONNECTIONS=200
PG_CPUS=2
PG_MEMORY=1g
K6_CPUS=2
K6_MEMORY=512m
CURSOR_SECRET_FILE=/run/secrets/cursor_secret
# Solution-specific for lab4: S3 client timeouts (connect + read, two attempts stay below
# REQUEST_TIMEOUT_SECONDS=8), parallel image checks per worker, RustFS memory limit.
S3_CONNECT_TIMEOUT_SECONDS=1
S3_READ_TIMEOUT_SECONDS=2
IMAGE_CHECK_CONCURRENCY=2
S3_MEMORY=512m
# Solution-specific for lab4 grade 4: upload bodies buffered at once per worker, grace period
# before an abandoned pending upload is swept, sweep interval and rows per sweep pass.
UPLOAD_CONCURRENCY=3
UPLOAD_GRACE_SECONDS=900
UPLOAD_SWEEP_INTERVAL_SECONDS=30
UPLOAD_SWEEP_BATCH=100
# Fixed glibc mmap threshold for the API: buffers above 128 KiB are returned to the system on free.
MALLOC_MMAP_THRESHOLD_=131072
# Solution-specific for lab5: RabbitMQ management UI on 127.0.0.1, backoff base of a transient
# retry (doubles per attempt up to 60 s), relay poll and sweep intervals, worker limits and the
# PostgreSQL pool of relay and worker. FAULT_POINT is set only by infra/compose.faults.yaml.
BROKER_MANAGEMENT_PORT=26086
JOB_BACKOFF_BASE_SECONDS=2
OUTBOX_POLL_SECONDS=1
SWEEP_INTERVAL_SECONDS=5
WORKER_CPUS=2
WORKER_MEMORY=1g
PIPELINE_DB_POOL_MAX_SIZE=4
# Solution-specific for lab5 grade 4: render time per job (the render process is killed after it),
# address space of one render process (WORKER_MEMORY >= WORKER_CONCURRENCY * RENDER_MEMORY_MB plus
# ~120 MiB of the worker itself), jobs per render process before it is replaced.
RENDER_TIMEOUT_SECONDS=30
RENDER_MEMORY_MB=448
RENDER_TASKS_PER_CHILD=100
# Solution-specific for lab6: host ports of Prometheus, Loki, Tempo and Grafana (127.0.0.1 only),
# Prometheus retention, OTLP export timeout in seconds, metric export interval in ms, SDK queue
# sizes, the bounded flush on shutdown; OTEL_SDK_DISABLED=true turns the SDK and instrumentations off.
# relay and worker get OTEL_SERVICE_NAME messenger-relay and messenger-worker in compose.yaml.
PROMETHEUS_PORT=26090
LOKI_PORT=26310
TEMPO_PORT=26320
GRAFANA_PORT=26300
PROMETHEUS_RETENTION=7d
OTEL_SDK_DISABLED=false
OTEL_EXPORTER_OTLP_TIMEOUT=2
OTEL_METRIC_EXPORT_INTERVAL=10000
OTEL_BSP_MAX_QUEUE_SIZE=2048
OTEL_BLRP_MAX_QUEUE_SIZE=2048
TELEMETRY_SHUTDOWN_SECONDS=2
# Minimal client transfer rate, bytes/s: every body byte moved in either direction extends the
# request deadline by 1/rate, so slow but steady uploads and downloads outlive REQUEST_TIMEOUT_SECONDS.
MIN_TRANSFER_RATE_BYTES=131072
# Solution-specific for lab4 track 2: limits of the one-off reconciler (profile reconcile), its S3
# read timeout for listing pages, optional CPU pinning of the measured services (empty = no pinning)
# and the CPU limit of RustFS (make reconcile-cost raises it with its memory for the 100k point).
RECONCILE_CPUS=1
RECONCILE_MEMORY=256m
RECONCILE_S3_READ_TIMEOUT_SECONDS=10
RECONCILE_CPUSET=
S3_CPUSET=
PG_CPUSET=
S3_CPUS=1
+1 -1
View File
@@ -1,4 +1,4 @@
## Лабораторная 2
## Лабораторная 6
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
+122
View File
@@ -0,0 +1,122 @@
name: Application
on: [push, pull_request]
permissions:
contents: read
jobs:
quality:
name: Линт, типы и собственные тесты
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- run: uv sync --locked --group dev
- run: make lint VENV=.venv
- run: make student-test VENV=.venv
acceptance:
name: Стенд Compose с RabbitMQ, relay и worker за HAProxy, smoke, pipeline, отказ реплики, рестарт и отказы зависимостей
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v5
with:
enable-cache: true
- run: uv sync --locked --group dev
- name: Поднять стенд из чистого clone
run: make up
- name: Публичный smoke через балансировщик
run: make test
- name: Топология брокера, consumer с prefetch и права клиентов
run: make topology
- name: Уровни «3» и «4» pipeline — worker остановлен, рестарт брокера, дубль, S3, попытки, варианты и пределы, падения в окнах 2–5, аренда, replay, DLQ, jitter, retention
run: |
make suite-preflight
make level4
- name: Сообщение задания в очереди, 409 до ready и SIGTERM relay и worker
run: make pipeline-evidence
- name: Вложение после остановки принявшей его реплики и при недоступном S3
run: |
make scenario SCENARIO=failover-upload
make scenario SCENARIO=s3-outage
- name: Приватный bucket, ограниченный ключ API и повторный bootstrap
run: make s3-check
- name: Негативный suite вложений, компенсация, уборка после grace, обрыв и зависание загрузки
run: make upload-faults
- name: Несколько реплик на одном токене и гонка одного ключа
run: make lb-check
- name: Повтор неподтверждённой записи после потерянного ответа и убитой реплики
run: make retry-check
- name: Остановка реплики под запросами, затем возврат
run: |
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
docker compose ps -a api
make restore
- name: Рестарт LB, API и хранилищ с сохранением volumes
run: make test-persistence ACTION_SCRIPT=scripts/restart.sh
- name: Отказы Valkey и PostgreSQL, гонка и rollback
run: make test-faults
- name: Лимиты, порты, секреты, volumes и сети по политике
run: make limits
- name: Время обнаружения отказа балансировщиком (SIGTERM, kill, pause, PostgreSQL)
run: make detection
- name: Наружу опубликован только балансировщик
run: |
docker compose ps --format '{{.Service}} {{.Ports}}' | awk '$2 != "" && $1 != "lb" && $1 != "broker" {print; bad=1} END {exit bad}'
docker compose ps --format '{{.Service}} {{.Ports}}' | awk '$1 == "broker" && $2 !~ /^127\.0\.0\.1:/ {print; bad=1} END {exit bad}'
- name: Процессы работают не от root
run: |
for id in $(docker compose ps -q); do
docker top "$id" -eo pid,uid,comm | awk 'NR>1 && $2==0 {print; bad=1} END {exit bad}' || exit 1
done
- name: Одноразовые сервисы не от root
run: |
for service in migrate s3-init; do
user=$(docker inspect -f '{{.Config.User}}' "$(docker compose ps -aq "$service")")
echo "$service: $user"
case "${user%%:*}" in ""|0|root) exit 1 ;; esac
done
- name: Корректная остановка relay и worker по SIGTERM
run: |
start=$(date +%s)
docker compose stop -t 10 relay worker
elapsed=$(( $(date +%s) - start ))
codes=$(docker inspect -f '{{.State.ExitCode}}' $(docker compose ps -aq relay worker) | sort -u)
echo "остановка за ${elapsed} с, коды завершения ${codes}"
test "$elapsed" -le 10 && test "$codes" = "0"
- name: Корректная остановка реплик API по SIGTERM
run: |
start=$(date +%s)
docker compose kill --signal SIGTERM api
codes=$(docker wait $(docker compose ps -aq api) | sort -u)
elapsed=$(( $(date +%s) - start ))
echo "остановка за ${elapsed} с, коды завершения ${codes}"
test "$elapsed" -le 10 && test "$codes" = "0"
- name: Журналы стенда
if: always()
run: docker compose logs --tail 80
+10
View File
@@ -30,3 +30,13 @@ coverage/
.vscode/
# Keep small, sanitized evidence summaries in evidence/ under version control.
# Inspect git diff --cached: ignore rules are not a secret scanner.
# Тулчейн студенческой реализации
.pytest_cache/
.ruff_cache/
.ty_cache/
# Локальные настройки агента
.claude/settings.local.json
.verification/
+75
View File
@@ -0,0 +1,75 @@
## Что это за репозиторий
Skeleton лабораторной 6 курса «Мессенджер под нагрузкой» (МТУСИ, 7 независимых лаб) и решение в ветке `solution/bpi2303-kostylev`. Skeleton даёт задание (`README.md`), нормативный контракт (`contracts/`, включая `contracts/observability.md`) и публичные black-box проверки (`tests/`). Реализация API, relay, worker, Dockerfile, compose-стенд, конфигурация балансировщика, S3, брокера и стека наблюдаемости, миграции, хуки отказов и собственные тесты — студенческие; решение лабы 6 построено переносом решения лабы 5.
Лаба 6: наблюдаемость всей системы, включая фоновую обработку. Бизнес-API не меняется (контракт 1.6.0). API, relay и worker инициализируют **OpenTelemetry SDK** в каждом процессе после fork (gunicorn-воркер, процесс relay, процесс worker) и отдают OTLP/HTTP в **OTel Collector** (`otelcol-contrib`); Collector раскладывает метрики в **Prometheus 3** (OTLP-приём), логи в **Loki 3** (OTLP `/otlp`), трассы в **Tempo 2**; **Grafana 12** только показывает и связывает их, datasources и dashboards приходят из provisioning в Git. HTTP-метрики и server spans даёт инструментация FastAPI, client spans БД — инструментация asyncpg под SQLAlchemy 2.0.45 async (ORM/Core, async_sessionmaker, DeclarativeBase); миграции — Alembic 1.16.5, consumer spans — `RabbitTelemetryMiddleware` FastStream. Логи идут через порт `Logger` (structlog → stdout JSON и OTLP) с `trace_id`/`span_id`, без тел сообщений и секретов. Остаётся всё из лаб 1–5: HAProxy перед ≥2 репликами, PostgreSQL, Valkey, RustFS, RabbitMQ, outbox, relay, worker с арендой и вариантами, хуки остановки реплики и рестарта стенда.
## Команды
```sh
make check # offline: структура skeleton, JSON, локальные $ref, синтаксис Python, версия 1.6.0
make validate PYTHON=.venv/bin/python # полная валидация OpenAPI 3.1 и jobs.schema.json (venv с requirements-dev.txt)
make install # uv sync --group dev
make lint # ruff check + ruff format --check + ty check; обязательно перед commit
make student-test # собственные тесты (одноразовые PostgreSQL, Valkey, RustFS и RabbitMQ в Docker)
make secrets # локальные secrets/* (БД, кэш, S3, broker, Grafana) и .env
make up # docker compose build, затем up -d --wait: хранилища, broker, стек наблюдаемости → migrate, s3-init → api ×2, relay, worker → lb
make obs-validate # otelcol validate, promtool check config, loki/tempo verify, JSON dashboards
make telemetry-check # Grafana provisioning, запрос → лог → трасса, job → трасса и лог worker, утечки
make test # публичный smoke через LB (через scripts/lab6.sh: http://127.0.0.1:18086)
make test-persistence ACTION_SCRIPT=scripts/restart.sh # рестарт lb, api, relay, worker и postgres, valkey, s3, broker с volumes
make test-failover ACTION_SCRIPT=scripts/stop-one.sh # остановка реплики TARGET_INSTANCE_ID под запросами
make restore # вернуть остановленные реплики и дождаться их в ротации LB
make level3 # сценарии уровня «3» лабы 5 (scripts/pipeline_suite.py) → evidence/level3.json
make down # остановка без удаления данных; make reset-data — намеренное удаление volumes
```
Команды стенда выше запускаются через `scripts/lab6.sh`, например `scripts/lab6.sh make up`. Локальные `make check validate lint` запускаются напрямую; `make student-test` создаёт одноразовые Docker-зависимости и не выполняется в документационной фазе.
`tests/*.py` и `scripts/*.py` — только stdlib, синтаксис Python 3.9 (`make check` разбирает их системным `python3`). Проверки сами не поднимают стенд. Хуки `scripts/stop-one.sh` и `scripts/restart.sh` запускаются тестом без shell, поэтому исполняемые и с shebang. Адреса Prometheus, Loki, Tempo и Grafana для `scripts/obs_check.py` — переменные `PROMETHEUS_URL`, `LOKI_URL`, `TEMPO_URL`, `GRAFANA_URL` (по умолчанию стандартные порты на `127.0.0.1`).
## Архитектура skeleton
- `lab.json` — единственный источник номера лабы (6). От него зависят `scripts/check_contract.py` (`info.version == 1.6.0`) и `tests/client.py` (ветки `LAB>=5`: upload `202 queued`, ожидание `ready` до 60 с, ровно один `thumbnail` 256×160). Публичных проверок телеметрии нет: наблюдаемость доказывается своими скриптами и `evidence/`.
- `tests/` байт в байт как в лабе 5. `contracts/openapi.json` отличается только `info.title` и `info.version`.
- `contracts/observability.md` — нормативный набор сигналов (уровни «3» и «4»), resource attributes, запрещённые labels метрик и запрещённое содержимое telemetry, alerts и инциденты. `infra/otel/README.md` — что лежит в Git (Collector, storage, provisioning, dashboards, rules).
- `contracts/infrastructure.json` — роли лабы 5 плюс `otel_collector`, `telemetry_storage` (метрики, логи, трассы, persistent volume) и `grafana` (provisioning в Git). `compose.example.yaml` — только worksheet ролей.
## Ключевые правила контракта (lab6)
- Resource attributes: `service.name` (`messenger-api`, `messenger-worker`, `messenger-relay`), `service.version`, `service.instance.id` (уникален на процесс: `<hostname>-<pid>`), `deployment.environment.name=lab`.
- Метрики не получают labels `user_id`, `chat_id`, `message_id`, `attachment_id`, `request_id`, сырой URL, filename, текст; маршрут — шаблон (`/api/v1/chats/{chat_id}/messages`). HTTP duration — histogram в секундах (`http.server.request.duration`, `OTEL_SEMCONV_STABILITY_OPT_IN=http`).
- Логи: timestamp, severity, `service.name`, `event`, `trace_id`/`span_id` при активном span. В telemetry никогда не попадают пароли, `Authorization`, токен сессии, тело сообщения, байты изображения, filename, ключи и подписанные URL; захват HTTP-заголовков выключен, Collector дополнительно редактирует подозрительные ключи.
- Экспорт телеметрии никогда не блокирует запрос: только `Batch*Processor` и `PeriodicExportingMetricReader`, короткий таймаут OTLP. `/health/ready` не зависит от Collector, хранилищ телеметрии и Grafana. `OTEL_SDK_DISABLED=true` выключает SDK целиком.
- Тело ошибки остаётся `{"error":{"code","message","request_id"}}` с `additionalProperties: false`: `trace_id` в тело и схемы не добавляется.
- Сохраняются правила lab5 (outbox, `202 queued`, `ack` после `COMMIT`, ключ эффекта, `409` на `/content` до `ready`), lab4, lab3 (`X-Instance-Id`, `Idempotency-Key`), lab2 и lab1.
## Правила работы с репозиторием
- Не менять и не ослаблять `contracts/`, `tests/`, `README.md`, `COURSE.md`, `infra/README.md`, `infra/otel/README.md`, рубрику. Расхождение оформляется вопросом преподавателю.
- Код — `src/messenger/`, миграции — `alembic/versions/`, async `alembic/env.py`, one-shot `alembic upgrade head`; работа с БД — только SQLAlchemy ORM/Core, asyncpg используется как драйвер, конфигурация LB, нагрузки, S3, брокера и наблюдаемости — `infra/` (`infra/otel`, `infra/prometheus`, `infra/loki`, `infra/tempo`, `infra/grafana`), хуки и обвязка стенда — `scripts/`, собственные тесты — `student-tests/`, обезличенные доказательства — `evidence/`.
- Без комментариев в коде; отступы — пробелы (`[tool.ruff.format]`); Python 3.14 без `from __future__ import annotations` и `TYPE_CHECKING`.
- Наружу публикуются только LB, UI брокера, Prometheus, Loki, Tempo и Grafana на `127.0.0.1`; порты реплик API, OTLP и self-metrics Collector не публикуются. Порты задаются переменными `.env`/shell: API 18086, HAProxy stats 26085, broker UI 26086, Prometheus 26090, Loki 26310, Tempo 26320, Grafana 26300. Для команд стенда используй `scripts/lab6.sh`, который согласует project, порты, URL проверок и NO_PROXY.
- `REPORT.md`: команды из чистого clone, архитектура, таблица «критерий → команда → наблюдение → файл», таблица отказов, сопоставление сигналов `observability.md` реальным именам метрик. PASS без команды не доказательство.
- Ветка `solution/bpi2303-kostylev`, коммиты на английском, перед каждым — `make lint`. Никогда не коммитить `.env`, `secrets/`, токены, пароли Grafana, ключи S3, пароли брокера, volumes, дампы.
## Уровни оценки
Решение заявлено на **«5»** с треком 1 «Цена наблюдаемости»; таблица доказательств — в `REPORT.md`.
- **3** — минимум lab5 работает при включённой телеметрии; Collector, Prometheus, Loki, Tempo и Grafana в общем стенде с volumes и provisioning из Git; API, relay и worker отдают OTLP; RPS/errors/duration API, jobs по исходу и duration worker; структурные логи с `trace_id`/`span_id`; реальные server и consumer spans; импортируемые dashboards; нет секретов и тел в telemetry; запрос → трасса и лог, job → трасса и лог worker.
- **4** — W3C trace context API → outbox → broker → worker (producer/consumer), цепочка до S3/DB с retry; p50/p95/p99 из histogram, доля ошибок, queue depth и возраст старейшей job, DB pool, saturation worker, CPU/RAM; SLI/SLO, alert rules error rate и queue lag (firing → resolved); memory_limiter, batch, ограниченные очереди, отказ хранилищ телеметрии; два разобранных инцидента.
- **5** — трек 1: telemetry off / on / head sampling / tail sampling / сбалансированная конфигурация при одинаковой open-loop нагрузке — latency с клиента, CPU/RAM вне телеметрии, объём ingest, потери spans и захват вброшенных ошибок и медленных запросов.
## Актуальное состояние и границы этапов
- Автор: Костылев Артём, БПИ2303. Итоговый отчёт — [REPORT.md](REPORT.md), индекс — [evidence/README.md](evidence/README.md).
- Актуальный runtime и измеритель: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`; успешное независимое ревью: `6ca1c7549978af3127966540385195ab0587f2e1`. Поздние исправления ЛР5 доведены до immutable source `c7b8feb805d151e878d77b0dff47320a9436b172`, решения по путям — [forward-port.md](evidence/forward-port.md).
- Текущий набор — `evidence/forward-published/`: 995 passed, 2 skipped, 19 deselected в сохранённом полном suite; 15 пригодных серий, 12000 запросов, balanced −92,2% медианного Tempo ingest, 48/48 error и 48/48 slow. Исторические `grade3/grade4/track5` не выдаются за измерение позднего кода. Незатронутые storage/history проверки переиспользуются только с основанием из `preserved-mechanisms.json`.
- Пять minor после ревью — [review-summary.md](evidence/review-summary.md) и ограничения REPORT. Исправлять только major: доказанная угроза безопасности, потеря/повреждение данных, недоступность сервиса, неработающий обязательный сценарий. Minor сохранять, не блокировать ими отчёт/release; не объявлять неуспешный тест успешным.
- Этапы запускает только runner: один агент переноса/уровней 3→4→5/evidence → review (отдельный fix major и re-review при необходимости) → отдельный отчёт → один агент нагрузки и публикации. Ревью/отчёт предыдущей лабы может идти параллельно реализации следующей. Самостоятельно фоновых агентов не создавать, runner/state не менять.
- Обычные Docker-этапы используют отдельные project/порты/volumes совместно с другими лабами; эксклюзивный доступ получает только финальная нагрузка/release. Разрешён `mtusi2026lab6` в основном репозитории либо `hlpipe-lab6-*` в каталоге под ним или `/Users/artemkostylev/projects/learning/HL/labs/.pipeline/.runs/labs`. Чужие контейнеры не трогать; в конце остановить свои без удаления постоянных volumes. Не выполнять reset/clean и удаление пользовательских данных.
- Фон и условия измерения фиксировать. Пригодные сохранённые серии переиспользовать после проверки SHA/diff; повторять лишь отсутствующие, неуспешные или признанные непригодными. Изменение только документации не требует новой нагрузки.
- Отчётная фаза меняет только REPORT.md, AGENTS.md и Markdown внутри evidence; OpenSpec/config, код и raw measurements не меняет. `openspec/config.yaml` уже содержит актуальный стек; циклы propose/apply/archive закрыты, проверка — `openspec validate --all --strict --no-interactive`.
- Публикация только в private MTUCI-Labs на явно назначенном этапе PUBLICATION. Самостоятельные push и Telegram вне назначенной фазы запрещены; для разрешённого Telegram — только `telegram-send`. ЛР1/2 заморожены; другие репозитории читать только через предоставленный snapshot.
+51
View File
@@ -0,0 +1,51 @@
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS builder
COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
ENV UV_COMPILE_BYTECODE=1 \
UV_LINK_MODE=copy \
UV_PYTHON_DOWNLOADS=never \
UV_PROJECT_ENVIRONMENT=/opt/venv
WORKDIR /build
COPY pyproject.toml uv.lock README.md ./
RUN uv sync --locked --no-dev --no-install-project --no-editable
COPY src ./src
RUN uv sync --locked --no-dev --no-editable
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS runtime
ENV PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
PATH="/opt/venv/bin:$PATH" \
HOST=0.0.0.0 \
PORT=8080 \
LOG_LEVEL=INFO \
WEB_CONCURRENCY=2 \
REQUEST_TIMEOUT_SECONDS=8 \
DRAIN_SECONDS=2 \
SHUTDOWN_TIMEOUT_SECONDS=6
RUN groupadd --system --gid 10001 messenger \
&& useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
COPY --from=builder --chown=root:root /opt/venv /opt/venv
WORKDIR /app
COPY --chown=root:root alembic.ini ./
COPY --chown=root:root alembic ./alembic
USER 10001:10001
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
CMD ["python", "-c", "import urllib.request,sys;sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).status==200 else 1)"]
CMD ["python", "-m", "gunicorn", \
"--config", "python:messenger.gunicorn_conf", \
"messenger.main:create_app()"]
+1 -1
View File
@@ -1,4 +1,4 @@
# Памятка преподавателю — лабораторная 2
# Памятка преподавателю — лабораторная 6
## Что подготовлено
+284 -2
View File
@@ -1,17 +1,72 @@
PYTHON ?= python3
BASE_URL ?= http://localhost:8080
BASE_URL ?= http://$(shell docker compose port lb 8080 2>/dev/null || echo localhost:8080)
CA_FILE ?=
ACTION_SCRIPT ?=
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
.PHONY: help check validate test load test-persistence
VENV ?= .venv
UV ?= uv
COMPOSE ?= docker compose
RATE ?= 100
FROM_REV ?=
SHIFT ?= 0
SCRIPTS_PY = scripts/obs_cost_summary.py scripts/obs_cost.py scripts/obs_incidents.py scripts/obs_recovery.py scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py \
scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py \
scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
SCENARIO ?= failover-upload
PSQL = $(COMPOSE) exec -T postgres psql -U messenger -d messenger -X -q
OBS_CHECK = $(PYTHON) scripts/obs_check.py
APP_METRICS = {job=~"messenger-.*"}
TRACK2_OUTPUT ?= evidence/track2.json
TRACK2_SCENARIOS = smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version
PIPELINE_VERSION ?= 2
PYTEST_ARGS ?=
.PHONY: help check validate test load test-persistence test-failover \
install lint format typecheck student-test \
secrets up down reset-data migrate ps logs restore lb-check failover-evidence stack-evidence \
persistence-evidence storage-evidence cache-evidence test-faults seed explain \
limits detection failover-each rolling-update lab4-level4 \
scenario s3-check attachments-evidence upload-faults upload-memory \
images queues topology worker-stop worker-start broker-restart suite-preflight level3 \
pipeline-evidence level4 replay failed dlq \
obs-validate telemetry-check \
faults-evidence retry-check reconcile reconcile-demo reconcile-cost load-evidence \
clean-clone-evidence overload-leftovers
.NOTPARALLEL:
help:
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
@echo "install/lint/typecheck/student-test: студенческий тулчейн"
@echo "secrets/up/down/migrate/ps/logs: стенд Docker Compose; reset-data: намеренное удаление volumes"
@echo "restore: вернуть остановленные реплики; lb-check: реплики на одном токене и 20 гонок одного ключа"
@echo "lab4-level4 = limits, lb-check, detection, failover-each, rolling-update [FROM_REV=<коммит>]; RATE=$(RATE) rps"
@echo "scenario SCENARIO=failover-upload|s3-outage → evidence/<сценарий>.json; s3-check; attachments-evidence"
@echo "upload-faults: негативный suite, компенсация, уборка, обрыв и зависание загрузки против стенда"
@echo "upload-memory: пик памяти реплик при 10/20/40 одновременных загрузках 1 и 10 MiB → evidence/upload-memory/"
@echo "lab5: queues, topology, worker-stop/worker-start, broker-restart, images, suite-preflight, level3, level4"
@echo "lab5 оператор: failed (реестр), replay ID=<attachment_id>, dlq (просмотр DLQ без извлечения)"
@echo "lab6: obs-validate (конфиги Collector, Prometheus, Loki, Tempo, dashboards); telemetry-check → evidence/*.json"
@echo "retry-check: потерянный ответ на живой и на убитой реплике, повтор тем же ключом"
@echo "lab4-level4 = limits, lb-check, detection, failover-each, rolling-update [FROM_REV=<коммит>]; RATE=$(RATE) rps"
@echo "scenario SCENARIO=failover-upload|s3-outage → evidence/<сценарий>.json; s3-check; attachments-evidence"
@echo "upload-faults: негативный suite, компенсация, уборка, обрыв и зависание загрузки против стенда"
@echo "upload-memory ARGS='--out evidence/upload-memory/runN': пик памяти реплик при 10/20/40 загрузках 1 и 10 MiB"
@echo "load-evidence: tests/load.py ×3 клиентом-контейнером в сети edge стенда → evidence/load.txt, evidence/load/"
@echo "overload-leftovers: после upload-memory — намерения от 503, сверка и уборка после grace → evidence/overload-leftovers.txt"
@echo "clean-clone-evidence: блок «Запуск из чистого clone» в свежем клоне рядом → evidence/clean-clone.txt"
@echo "reconcile ARGS='--apply --deep': сверка PostgreSQL ↔ S3 (без ARGS — dry-run); трек 2:"
@echo "reconcile-demo: внедрение ошибок, dry-run/apply/повтор, безопасность, нагрузка → evidence/reconcile/"
@echo "reconcile-cost: стоимость обхода 1k/10k/30k/100k объектов × 3 прогона на чистом стенде → evidence/reconcile/cost/"
@echo "BASE_URL=$(BASE_URL) (порт lb из docker compose); failover-evidence SHIFT=1 — сдвиг round robin на вторую реплику"
check:
$(PYTHON) scripts/check_contract.py
validate: PYTHON := $(if $(wildcard $(VENV)/bin/python),$(VENV)/bin/python,$(PYTHON))
validate:
$(PYTHON) scripts/validate_openapi.py
@@ -24,3 +79,230 @@ load:
test-persistence:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable restart hook"; exit 2)
$(PYTHON) tests/resilience.py --mode persistence --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
install:
$(UV) sync --group dev
lint:
$(VENV)/bin/ruff check src student-tests $(SCRIPTS_PY)
$(VENV)/bin/ruff format --check src student-tests $(SCRIPTS_PY)
$(VENV)/bin/ty check src student-tests alembic
format:
$(VENV)/bin/ruff check --fix src student-tests $(SCRIPTS_PY)
$(VENV)/bin/ruff format src student-tests $(SCRIPTS_PY)
typecheck:
$(VENV)/bin/ty check src student-tests alembic
student-test:
$(VENV)/bin/pytest $(PYTEST_ARGS)
secrets:
sh scripts/secrets.sh
sh scripts/broker-secrets.sh
sh scripts/secrets-lab6.sh
up: secrets
$(COMPOSE) build
$(COMPOSE) up -d --wait
scripts/lb-wait.sh
down:
$(COMPOSE) --profile bench --profile tools --profile reconcile down
reset-data:
@echo "Удаляются volumes с учебными данными стенда"
$(COMPOSE) --profile bench --profile tools --profile reconcile down -v --remove-orphans
migrate:
$(COMPOSE) run --rm migrate
ps:
$(COMPOSE) ps
logs:
$(COMPOSE) logs --tail 200
restore:
scripts/restore.sh
lb-check:
$(PYTHON) scripts/probe.py check --rounds 20 --out evidence/lb-check.json
retry-check:
$(PYTHON) scripts/probe.py retry --out evidence/retry.json
limits:
$(PYTHON) scripts/limits.py --out evidence/limits.txt
detection:
rm -rf evidence/detection
for run in 1 2 3; do sh scripts/detection.sh evidence/detection/run$$run || exit 1; done
failover-each:
RATE=$(RATE) sh scripts/failover-each.sh evidence/failover-each
rolling-update:
ifneq ($(FROM_REV),)
git archive "$(FROM_REV)" | docker build -q -t "mtusi-messenger:$(FROM_REV)" -
API_IMAGE="mtusi-messenger:$(FROM_REV)" $(COMPOSE) up -d --no-deps --force-recreate --wait api
scripts/lb-wait.sh
endif
$(COMPOSE) build -q api
RATE=$(RATE) $(PYTHON) scripts/probe.py load --expect replace --duration 90 \
--out evidence/rolling$(if $(FROM_REV),-from-$(FROM_REV)) \
-- scripts/rolling-update.sh
lab4-level4: limits lb-check detection failover-each rolling-update
scenario:
$(PYTHON) scripts/lab4_scenarios.py $(SCENARIO) --base-url "$(BASE_URL)" --restore --output evidence/$(SCENARIO).json
scripts/lb-wait.sh
s3-check:
sh scripts/s3-check.sh > evidence/s3-security.txt; status=$$?; cat evidence/s3-security.txt; exit $$status
attachments-evidence:
sh scripts/evidence-attachments.sh
upload-faults:
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD) BASE_URL=$(BASE_URL)"; \
BASE_URL="$(BASE_URL)" $(VENV)/bin/pytest -m stack -v -s --no-header --tb=short \
-p no:cacheprovider student-tests/stack/test_attachment_faults.py; } > evidence/upload-faults.txt; \
status=$$?; cat evidence/upload-faults.txt; exit $$status
upload-memory:
$(PYTHON) scripts/upload_memory.py --base-url "$(BASE_URL)" $(ARGS)
load-evidence:
sh scripts/load-evidence.sh
overload-leftovers:
sh scripts/overload-leftovers.sh
clean-clone-evidence:
sh scripts/clean-clone.sh
reconcile:
$(COMPOSE) run --rm --no-deps reconciler $(ARGS)
reconcile-demo:
$(PYTHON) scripts/reconcile_track.py demo --base-url "$(BASE_URL)" $(ARGS)
reconcile-cost:
$(PYTHON) scripts/reconcile_track.py cost $(ARGS)
failover-evidence:
sh scripts/failover-evidence.sh "$(BASE_URL)" "$(SHIFT)"
stack-evidence:
sh scripts/evidence-stack.sh
persistence-evidence:
sh scripts/persistence-evidence.sh "$(BASE_URL)" > evidence/persistence.txt
cat evidence/persistence.txt
storage-evidence:
sh scripts/evidence-storage.sh
cache-evidence:
sh scripts/evidence-cache.sh > evidence/cache.txt
cat evidence/cache.txt
test-faults:
BASE_URL="$(BASE_URL)" $(VENV)/bin/pytest -m stack -v -s student-tests/stack
faults-evidence:
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; echo "\$$ make test-faults"; \
$(MAKE) --no-print-directory test-faults 2>&1; } | sed 's#$(CURDIR)#<repo>#g' > evidence/faults.txt
cat evidence/faults.txt
grep -qE '^=+ [0-9]+ passed in' evidence/faults.txt
seed:
$(PSQL) -f - < scripts/sql/seed-history.sql
explain: seed
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; \
echo "\$$ make explain # scripts/sql/explain-history.sql"; \
$(PSQL) -f - < scripts/sql/explain-history.sql; } > evidence/explain-history.txt
cat evidence/explain-history.txt
images:
$(PYTHON) scripts/make_images.py
queues:
sh scripts/broker.sh queues
topology:
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; \
echo "$$ rabbitmqctl list_queues -p messenger name type durable policy messages"; \
$(COMPOSE) exec -T broker rabbitmqctl -q list_queues -p messenger name type durable policy messages; \
echo "$$ rabbitmqctl list_consumers -p messenger"; \
$(COMPOSE) exec -T broker rabbitmqctl -q list_consumers -p messenger queue_name ack_required prefetch_count; \
echo "$$ rabbitmqctl list_permissions -p messenger"; \
$(COMPOSE) exec -T broker rabbitmqctl -q list_permissions -p messenger; \
echo "$$ rabbitmqctl list_users"; \
$(COMPOSE) exec -T broker rabbitmqctl -q list_users; \
echo "$$ docker compose ps"; \
$(COMPOSE) ps --format '{{.Service}}\t{{.Status}}\t{{.Ports}}'; } > evidence/topology.txt
cat evidence/topology.txt
worker-stop:
sh scripts/worker.sh stop
worker-start:
sh scripts/worker.sh start
broker-restart:
sh scripts/broker.sh restart
suite-preflight:
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --preflight
level3:
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --level 3 --output evidence/level3.json
level4:
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --level 4 --output evidence/level4.json
failed:
$(COMPOSE) run --rm --no-deps -T worker python -m messenger.admin failed
replay:
@test -n "$(ID)" || (echo "Set ID=<attachment_id>"; exit 2)
$(COMPOSE) run --rm --no-deps -T worker python -m messenger.admin replay --attachment-id "$(ID)"
dlq:
sh scripts/broker.sh dlq
pipeline-evidence:
$(PYTHON) scripts/pipeline_evidence.py --base-url "$(BASE_URL)" > evidence/pipeline-stand.txt; \
status=$$?; cat evidence/pipeline-stand.txt; exit $$status
obs-validate:
sh scripts/obs-validate.sh
telemetry-check:
$(OBS_CHECK) grafana > evidence/grafana.json
$(OBS_CHECK) dashboards --strict > evidence/dashboards.json
$(OBS_CHECK) request --base-url "$(BASE_URL)" > evidence/trace-request.json
$(OBS_CHECK) job --base-url "$(BASE_URL)" > evidence/trace-job.json
$(OBS_CHECK) leaks --base-url "$(BASE_URL)" > evidence/leaks.json
$(OBS_CHECK) query 'count by (__name__, job) ($(APP_METRICS))' \
'count by (http_route, http_request_method) (http_server_request_duration_seconds_count)' \
'count by (job_type, outcome) (messenger_jobs_processed_total)' > evidence/cardinality.json
@echo "telemetry-check: evidence/grafana.json dashboards.json trace-request.json trace-job.json leaks.json cardinality.json"
.PHONY: track2 reprocess
track2:
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --level 5 --only "$(TRACK2_SCENARIOS)" --output "$(TRACK2_OUTPUT)"
reprocess:
@test -n "$(ID)" || (echo "Set ID=<attachment_id>"; exit 2)
$(COMPOSE) run --rm --no-deps -T worker python -m messenger.admin reprocess --attachment-id "$(ID)" --pipeline-version "$(PIPELINE_VERSION)"
+33 -33
View File
@@ -1,6 +1,6 @@
# Лабораторная 2. PostgreSQL, сессии и Redis/Valkey
# Лабораторная 6. Наблюдаемость: OpenTelemetry и Grafana
Перенести состояние в постоянное хранилище и связать запросы с аутентифицированным пользователем. Подтверждённые сообщения и действующие сессии переживают перезапуск.
Научиться находить причину деградации через метрики, логи и трассы всей системы, включая фоновую обработку.
Это **skeleton**, а не готовый сервер. Здесь есть задание, контракт и публичные проверки. Реализацию приложения, Dockerfile и требуемую инфраструктуру пишет студент. Язык и framework свободные; UI не обязателен.
@@ -8,41 +8,41 @@
## Откуда начинаем
Перенесите собственный минимум lab1. Изменения API: password при регистрации, Basic для создания сессии, Bearer вместо X-User-Id для бизнес-запросов.
Перенесите минимум lab5. Бизнес-API не меняется. Должны оставаться API-реплики, БД, S3, broker и worker.
**Архитектура:** Клиент → API → PostgreSQL; на 4 появляется Redis ИЛИ Valkey для сессий/кэша.
**Архитектура:** API/worker → OTLP → OTel Collector → хранилища metrics/logs/traces → Grafana; queue context связывает API и worker.
## Что сделать
1. Опишите таблицы, ограничения, индексы и транзакционные границы до написания запросов. Добавьте версионируемые миграции.
2. Перенесите пользователей, чаты, членство и сообщения в PostgreSQL. Не используйте SQLite как замену PostgreSQL в принимаемом стенде.
3. Реализуйте регистрацию с password KDF, вход через Basic, opaque Bearer-сессию, expiry и logout. Действующие сессии тоже должны сохраняться.
4. Подготовьте единый локальный запуск API и БД (Compose рекомендуется уже здесь), подключите именованные volumes.
5. Пройдите smoke и тест полного перезапуска. На 4 добавьте Redis/Valkey с конкретной ролью и повторите эксперимент отказа.
1. Выберите хранилища: например Prometheus для метрик, Loki для логов, Tempo для трасс. Разрешена другая совместимая комбинация с Grafana; Grafana сама не хранит все три сигнала.
2. Добавьте SDK-инструментацию API и worker и отдельный Collector в общий Compose/Swarm; настройте receivers/processors/exporters и хранение данных.
3. Реализуйте набор сигналов contracts/observability.md и импортируемые dashboard/datasources. Не используйте user_id/chat_id/message_id как labels метрик.
4. Проверьте связь запроса, структурного лога и обработки задания. Секреты, тела сообщений и изображения не должны попадать в telemetry.
5. Создайте нагрузку, внедрите два ограниченных отказа и найдите их причину только с помощью сохранённых наблюдений; приложите доказательства.
## Оценка «3»: работающий минимум
- Минимум lab1 сохранён с объявленным переходом на новую аутентификацию. Все обязательные данные и ещё действующие сессии переживают рестарт API и хранилищ без удаления volumes.
- PostgreSQL содержит пользователей, чаты, участников и сообщения. Есть первичные/внешние ключи, уникальность username и миграции с нуля. Составная операция создания чата выполняется атомарно.
- Basic применяется только на POST /auth/sessions, остальные endpoint требуют Bearer. Есть срок жизни сессии и отзыв; X-User-Id не даёт доступ. Пароли уже хешируются password KDF.
- make test и make test-persistence ACTION_SCRIPT=scripts/restart.sh проходят. Миграции повторно запускаются без дублирования данных. Secrets/volumes не попадают в Git.
- Весь минимум lab5 работает. Collector, хранилища трёх сигналов и Grafana включены в общий воспроизводимый стенд; конфиги и provisioning лежат в Git, persistent volumes сохраняют нужную историю.
- API и worker отдают OTLP в Collector. В Grafana видны: RPS/errors/duration API, успешные/ошибочные jobs и duration worker; доступны структурные логи и реальные трассы API и worker.
- Есть импортируемый dashboard, а не только screenshot. Сохраняются низкокардинальные атрибуты и идентификаторы trace/span в логах; содержимое сообщений, пароли/токены не экспортируются.
- На контролируемом запросе можно найти trace и связанный лог; на job — worker trace и лог. make test проходит при включённой телеметрии. Объяснено назначение каждого хранилища.
## Оценка «4»: уровень стажёра/джуна
Весь уровень 3 **и все** пункты ниже:
- Redis ИЛИ Valkey реально используется для сессий с TTL либо осмысленного кэша. Опишите источник истины, политику expiry и инвалидации. Если это единственное хранилище сессий, включите persistence, выдерживающее проверяемый рестарт; один TTL не сохраняет данные.
- Используются параметризованные запросы/безопасный ORM, ограниченный connection pool и таймауты. Проверены гонка регистрации одинакового username и rollback составной операции.
- Индекс истории обоснован запросом и EXPLAIN (ANALYZE, BUFFERS) на ≥10 000 сообщениях; пагинация не использует полный scan всей истории на каждую страницу.
- Есть автоматическая проверка expiry при коротком SESSION_TTL_SECONDS, logout, неверного пароля и недоступности Redis/БД. При потере сервиса приложение возвращает ограниченную ошибку/корректно обходит кэш, не принимает запрос от анонимного пользователя как доверенный.
- W3C trace context переносится API → broker → worker; видны producer/consumer spans с корректной связью parent или link. По одному upload найдена цепочка до S3/DB результата, включая retry.
- Dashboard показывает p50/p95/p99, долю ошибок, queue depth/возраст старейшей job, DB pool и saturation worker. Приведены запросы и единицы измерения; percentile вычисляется из histogram, не средних.
- Задан проверяемый SLI/SLO и окно; сохранены alert rules минимум для высокого error rate и растущего queue lag. Alerts проверены искусственным отказом; доставка наружу не нужна, достаточно локального состояния firing/resolved.
- Collector имеет memory limiter/batch и ограниченную буферизацию/поведение при недоступном backend. Остановка telemetry-хранилища не ломает сообщения и не вызывает бесконечный рост RAM. Отчёт содержит разбор двух инцидентов и шаги диагностики.
## Оценка «5»: исследование и доказанный результат
Весь уровень 4 **и один законченный трек на выбор**. Приведите гипотезу, повторяемую методику, результаты и ограничения.
**Трек 1. Согласованность кэша.** Покажите на конкурентном сценарии stale read или cache stampede, реализуйте защиту и измерьте число SQL-запросов/latency до и после. Докажите read-after-write из контракта и восстановление после рестарта кэша; одной установки Redis недостаточно.
**Трек 1. Цена наблюдаемости.** Сравните telemetry off/on и ≥2 sampling/aggregation настройки при одинаковой нагрузке: latency, CPU/RAM, объём ingest, потери spans. Объясните, какие ошибки и редкие медленные запросы теряются; предложите и проверьте сбалансированную конфигурацию.
**Трек 2. План запроса и восстановление.** Исследуйте ≥100 000 сообщений с неравномерными размерами чатов, сравните ≥2 индекса/запроса и стоимость записи. Сделайте backup → восстановление в новое хранилище → сверку количества и выбранных сообщений. Обсудите время восстановления и границы потери данных.
**Трек 2. SLO и обнаружение инцидентов.** Обоснуйте SLO для API и времени до ready, реализуйте burn-rate alerts с несколькими окнами, воспроизведите краткий всплеск и длительную деградацию. Измерьте время обнаружения/восстановления, ложные срабатывания и покажите runbook, по которому другой студент находит причину.
## Как начать и проверить
@@ -78,28 +78,28 @@ Smoke не выставляет оценку автоматически. Про
## Сценарий защиты
1. Применить миграции к пустому volume, создать пользователей, войти, отправить сообщение.
2. Запустить persistence-сценарий: он держит токен только в памяти тестового процесса; action script перезапускает API и все используемые хранилища, сохраняя volumes.
3. После готовности тот же токен читает те же данные; после logout — 401. Повторить с отдельным коротким TTL для проверки expiry.
4. Для 4: показать роль Redis/Valkey, план SQL и ошибку зависимости. Для 5: выполнить выбранный эксперимент.
1. С нуля поднять стенд, импортировать provisioning без ручного набора dashboard, запустить smoke и нагрузку.
2. Показать свежие метрики, одну трассу и соответствующий структурный лог API и worker; проверить, что это данные своего приложения.
3. Для 4: пройти от upload до worker в одном trace/связанных traces, остановить worker или ограничить DB, увидеть firing и затем resolved.
4. Остановить telemetry backend, проверить сохранение работы API и ограниченное поведение exporter. Для 5: повторить эксперимент.
## Что должно быть в PR
Исходники, Dockerfile, миграции, compose.yaml либо эквивалентный воспроизводимый запуск, scripts/restart.sh, собственные тесты и отчёт.
Collector configuration, конфиги backend, Grafana provisioning/dashboards, alert rules по уровню, схема сигналов и отчёт инцидентов.
Заполните `REPORT.md` и [шаблон PR](.github/pull_request_template.md), укажите целевую оценку/трек. Не меняйте обязательный контракт и публичные tests ради зелёного результата. Сохраняйте возможности предыдущих лабораторных в пределах [объявленных переходов](COURSE.md).
## Вопросы на защите
- Что подтверждает COMMIT и какие гарантии зависят от настройки durability?
- Где хранятся сессии и что будет после flush/restart Redis?
- Чем аутентификация отличается от доступа к конкретному чату?
- Почему hash пароля и быстрый checksum имеют разные задачи?
- Чем лог, метрика и trace отвечают на разные вопросы об одном инциденте?
- Почему chat_id в labels может исчерпать хранилище?
- Как очередь переносит причинную связь между процессами?
- Почему среднее latency не заменяет p99 и почему нельзя усреднять p99 реплик?
## Первичные материалы
- [PostgreSQL transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html)
- [PostgreSQL EXPLAIN](https://www.postgresql.org/docs/current/using-explain.html)
- [Valkey persistence](https://valkey.io/topics/persistence/)
- [Redis persistence](https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/)
- [Basic authentication](https://www.rfc-editor.org/info/rfc7617/)
- [OTel Collector configuration](https://opentelemetry.io/docs/collector/configuration/)
- [OTel propagation](https://opentelemetry.io/docs/concepts/context-propagation/)
- [W3C Trace Context](https://www.w3.org/TR/trace-context/)
- [Prometheus histograms](https://prometheus.io/docs/practices/histograms/)
- [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/)
+263 -19
View File
@@ -1,37 +1,281 @@
# Отчёт — лабораторная 2
# Отчёт — лабораторная 6
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Автор: **Костылев Артём, БПИ2303**. Заявленная оценка — **«5»**, трек 1 «Цена наблюдаемости»: уровни 3 и 4 подтверждены таблицей ниже, пять режимов наблюдаемости — 15 пригодными сериями. Это заявка по рубрике, а не автоматически выставленная оценка.
Ветка: `solution/bpi2303-kostylev`. Проверенный ревью SHA: `6ca1c7549978af3127966540385195ab0587f2e1`; измеряемый код приложения, инфраструктуры и генератора: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`. Между ними эти файлы не менялись. Документационный коммит определяется через `git log -1 --format=%H -- REPORT.md`; публикация выполняется отдельным этапом, здесь не заявлены.
Актуальные результаты: [индекс evidence](evidence/README.md), [перенос поздних исправлений](evidence/forward-port.md), [сводка ревью](evidence/review-summary.md), [проверки отчёта](evidence/report-audit.md), [финальный аудит нагрузки](evidence/load-validation.md). Основа ЛР5 — `f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071`, поздний перенос доведён до immutable source `c7b8feb805d151e878d77b0dff47320a9436b172`; учтены 281 путь source diff. Результаты ЛР5 не выдаются за проверки ЛР6.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Нужны Git, Docker с Compose v2, uv, Python 3.14 для приложения и Python 3.9+ для stdlib-проверок. OpenSpec CLI нужен только для валидации спецификаций. Проверенный host: macOS arm64, 16 CPU; Docker 28.3.2, 16 vCPU и 8 217 190 400 bytes RAM. Версии зависимостей закреплены в [uv.lock](uv.lock), образы — тегом и digest в [Dockerfile](Dockerfile) и [compose.yaml](compose.yaml).
`REPOSITORY_URL` оператор задаёт адресом своего private fork в MTUCI-Labs. Команды ниже — инструкция воспроизведения; на этапе отчёта Docker и нагрузки не запускались.
```sh
git clone "$REPOSITORY_URL" mtusi2026lab6
cd mtusi2026lab6
git switch -c solution/bpi2303-kostylev origin/main
uv python install 3.14
uv sync --locked --group dev
make check validate lint
openspec validate --all --strict --no-interactive
scripts/lab6.sh make secrets
scripts/lab6.sh docker compose config --quiet
scripts/lab6.sh make up
scripts/lab6.sh make ps
curl --fail http://127.0.0.1:18086/health/ready
scripts/lab6.sh make obs-validate
scripts/lab6.sh make test
scripts/lab6.sh make telemetry-check
scripts/lab6.sh make test-persistence ACTION_SCRIPT=scripts/restart.sh
scripts/lab6.sh make test-failover ACTION_SCRIPT=scripts/stop-one.sh
scripts/lab6.sh make restore
scripts/lab6.sh make down
```
`make secrets` создаёт локальные `.env` и `secrets/`, не печатая значения; существующие значения сохраняются. `make up` собирает образ и ждёт healthy: PostgreSQL → one-shot `alembic upgrade head`, S3 → `s3-init` (bucket, политики и отдельные роли), RabbitMQ → API ×2 / relay / worker → LB; параллельно поднимаются Collector и хранилища. Ручной SQL, ручное создание datasource/dashboard и внешний S3 не требуются. Повторный запуск миграций: `scripts/lab6.sh make migrate`.
Полные собственные тесты запускаются отдельно: `scripts/lab6.sh make student-test`. Обвязка задаёт обход localhost через `NO_PROXY`, в том числе при системном прокси macOS. Тесты создают одноразовые Docker-зависимости и не должны идти одновременно с измерениями трека. В отчётном этапе использован сохранённый результат; свежий запуск фазы validation описан в [финальном аудите](evidence/load-validation.md). Данные публичного API синтетические. Для повторных экспериментов сохраняйте вывод в новый каталог, не перезаписывая принятые evidence.
`make down` останавливает только выбранный Compose project и сохраняет volumes. Существующая цель `scripts/lab6.sh make reset-data` намеренно удаляет его учебные volumes; она не входит в обычный запуск и в отчётном этапе не выполнялась. Не останавливайте чужие контейнеры. Основной project — `mtusi2026lab6` в этом репозитории; временный — только `hlpipe-lab6-*` в разрешённом координатором каталоге, с отдельными портами/volumes.
### Порты и настройки
Все опубликованные порты привязаны к `127.0.0.1`; API-реплики, PostgreSQL, Valkey, S3, AMQP, OTLP и self-metrics Collector доступны только внутри стенда. [scripts/lab6.sh](scripts/lab6.sh) задаёт project, порты, URL проверок и `NO_PROXY`, затем выполняет переданную команду. Перед запуском оператор проверяет доступность портов и отсутствие конфликтующих переменных своего shell.
| Назначение | Переменная | Host port |
| --- | --- | ---: |
| API через HAProxy | `LB_HTTP_PORT` | 18086 |
| HAProxy stats | `LB_STATS_PORT` | 26085 |
| RabbitMQ management | `BROKER_MANAGEMENT_PORT` | 26086 |
| Prometheus | `PROMETHEUS_PORT` | 26090 |
| Loki | `LOKI_PORT` | 26310 |
| Tempo query | `TEMPO_PORT` | 26320 |
| Grafana | `GRAFANA_PORT` | 26300 |
Grafana: пользователь `admin`, пароль из локального `secrets/grafana_admin_password`; его значение не входит в отчёт. Provisioning создаёт datasources `prometheus`, `loki`, `tempo` и dashboards API/pipeline, переходы exemplar → trace и trace ↔ logs.
| Настройки из `.env.example` | Фактическое применение |
| --- | --- |
| `APP_HOST`, `APP_PORT`, `APP_ENV` | Логические поля skeleton; Compose не передаёт их приложению. Образ слушает `HOST=0.0.0.0`, `PORT=8080`; окружение OTel задаёт `deployment.environment.name=lab` |
| `SESSION_STORE=postgres`, `CACHE_URL_FILE`, `SESSION_TTL_SECONDS` | PostgreSQL — источник сессий; Valkey — адаптер кэша. Переключателя backend через `SESSION_STORE` нет; TTL по умолчанию 3600 s |
| `DATABASE_URL_FILE`, `CURSOR_SECRET_FILE`, S3 `*_FILE` | Compose монтирует роли в `/run/secrets`; SQLAlchemy переводит DSN в `postgresql+asyncpg` |
| `BROKER_URL_FILE` | В relay и worker одинаковый путь `/run/secrets/broker_url`, но разные secrets `broker_url_api` / `broker_url_worker` и права |
| `API_REPLICAS`, `WEB_CONCURRENCY`, `DB_POOL_MAX_SIZE` | 2 реплики ×2 процесса, пул по 8 соединений на процесс; relay/worker по `PIPELINE_DB_POOL_MAX_SIZE=4` |
| `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS`, `WORKER_CONCURRENCY` | 10 MiB, 40 MP, 2 задания; render timeout 30 s, лимит адресного пространства renderer 448 MiB |
| `OTEL_*`, `TELEMETRY_SHUTDOWN_SECONDS` | OTLP/HTTP, timeout 2 s, метрики каждые 10 s, очереди SDK по 2048, bounded shutdown 2 s; SDK off через `OTEL_SDK_DISABLED=true` |
| `OTEL_TRACE_RATIO`, `OTEL_COLLECTOR_CONFIG`, `COST_EXPERIMENT` | Дополнительные Compose-параметры: по умолчанию 1, `collector.yaml`, false. Генератор включает fault hook лишь на серию и восстанавливает default в `finally` |
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab1, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
```mermaid
flowchart LR
Client[Клиент] --> LB[HAProxy]
LB --> API[API ×2]
API --> PG[(PostgreSQL)]
API --> Cache[(Valkey)]
API --> S3[(RustFS / S3)]
PG --> Relay[Outbox relay]
Relay --> Broker[(RabbitMQ)]
Broker --> Worker[FastStream worker]
Worker --> PG
Worker --> S3
API -. OTLP .-> Collector[OTel Collector]
Relay -. OTLP .-> Collector
Worker -. OTLP .-> Collector
Collector --> Prom[(Prometheus)]
Collector --> Loki[(Loki)]
Collector --> Tempo[(Tempo)]
Grafana[Grafana] --> Prom
Grafana --> Loki
Grafana --> Tempo
```
Python 3.14; SQLAlchemy **2.0.45 async**, Alembic **1.16.5**, FastAPI, dishka, structlog, FastStream **0.7.8** + aio-pika **10.2.1**; OTel SDK 1.45.1 и инструментации 0.66b1. Инфраструктура: HAProxy 3.2.26, PostgreSQL 17.11, Valkey 8.1.11, RustFS 1.0.4, RabbitMQ 4.1.8, otelcol-contrib 0.163.2, Prometheus 3.15.1, Loki 3.7.9, Tempo 2.10.10, Grafana 12.4.14. Фактические image IDs, non-root users и лимиты: [runtime.json](evidence/forward-published/runtime.json).
Зоны `identity`, `messaging`, `attachments` разделены на domain/application/infrastructure/entrypoints; домен зависит от портов. `shared` содержит только платформу. `main.py`, `relay.py`, `worker.py` собирают процессы. PostgreSQL хранит пользователей, сессии, сообщения, состояния вложений, jobs и outbox; RustFS — байты, RabbitMQ — доставку, Valkey — ускорение чтения. Prometheus хранит численные временные ряды, Loki — структурные логи, Tempo — spans. Collector принимает/фильтрует/буферизует сигналы, Grafana запрашивает их и связывает; ни один из них не источник бизнес-данных.
Все runtime-запросы PostgreSQL проходят через SQLAlchemy ORM/Core: `DeclarativeBase`, `create_async_engine`, `async_sessionmaker`, отдельная `AsyncSession` на UoW. `asyncpg` остаётся драйвером и точкой инструментации. Миграции — [alembic/env.py](alembic/env.py) и [alembic/versions](alembic/versions): async engine, `run_sync`, advisory lock, legacy adoption, транзакция на ревизию, head `0007` для trace context. Старого исполняемого SQL-мигратора в инструкции запуска нет.
Upload возвращает `202 queued` после durable acceptance; задание и outbox фиксируются атомарно. Relay публикует persistent сообщение с confirms через FastStream/aio-pika, затем отмечает outbox. Неустранимое окно между publish и отметкой допускает повторную доставку. Worker использует manual ack **после COMMIT**; lease/fencing и ключ эффекта `(attachment_id, pipeline_version, variant)` не допускают повторной публикации результата. До ready `/content` отвечает 409. Retry/backoff, DLQ, реестр failed и replay сохранены. Цена outbox — дополнительный poll/таблица и eventual consistency; брокер не даёт exactly-once.
ЛР6 добавила SDK после fork (без preload приложения в gunicorn), OTLP, корреляцию, resource attributes и наблюдение фоновых задач. `traceparent`/`tracestate` сохраняются отдельно от payload в jobs/outbox и передаются в headers; producer/consumer spans создаёт `RabbitTelemetryMiddleware`. S3 spans показывают операцию без ключей, SQL client spans — работу БД. Retry сохраняет причинную связь, что видно в одной цепочке из 86 spans.
Экспорт выполняют `BatchSpanProcessor`, `BatchLogRecordProcessor`, `PeriodicExportingMetricReader`; network timeout короткий, readiness не проверяет наблюдаемость. Collector: memory limiter 75% / spike 20%, batch 1024/2048 с timeout 2 s, очереди 20 000 spans, 20 000 logs, 50 000 metric points, 2 consumers, retry до 60 s. Буферы конечны и не долговременны: при длительном отказе telemetry теряется, бизнес-запросы продолжаются. `Logger` ограничивает поля, HTTP headers не захватываются, Collector редактирует подозрительные ключи/значения. Границы проверки утечек явно указаны ниже.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Команды в таблице выполняются с префиксом `scripts/lab6.sh`, если работают со стендом. Полные команды, UTC, SHA и exit code сохранены в указанных логах. **Актуальный набор — `forward-published`**; исторические исключения помечены явно и применяются только к неизменным механизмам согласно [preserved-mechanisms.json](evidence/forward-published/preserved-mechanisms.json).
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Критерий | Команда / сценарий | Наблюдение | Файл |
| --- | --- | --- | --- |
| Структура, контракт, качество | `make check validate lint` | OpenAPI 1.6.0 / JSON Schema, ruff, 216 форматированных файлов, ty — PASS | [final-quality.txt](evidence/forward-published/final-quality.txt), [свежая проверка отчёта](evidence/report-audit.md) |
| Собственные регрессии, SQLAlchemy/Alembic | `scripts/lab6.sh make student-test` | Сохранённый полный: 995 passed, 2 Linux-only skipped, 19 stack deselected. Свежий validation: 993 passed, 2 ошибки подготовки S3; адресный повтор с NO_PROXY — 2 passed | [прежний полный](evidence/forward-published/student-test.txt), [свежий полный](evidence/validation/student-test.txt), [повтор S3](evidence/validation/student-test-s3-retry.txt), [разбор](evidence/load-validation.md) |
| 3: минимум lab5 при telemetry on | `make test` | Public API PASS; final smoke после возврата defaults | [smoke.txt](evidence/forward-published/smoke.txt), [final-smoke.txt](evidence/forward-published/final-smoke.txt) |
| 3: durable pipeline и повторы | `python3 scripts/pipeline_suite.py --level 4 --only smoke_pipeline,worker_down_broker_restart,duplicate_after_ready,poison_image,s3_outage_recovers,retry_exhausted,crash_after_s3_before_commit,crash_after_commit_before_ack,malformed_message_dead_lettered --output evidence/reproduction-pipeline.json` | 9 сценариев; 7 принятых заданий достигают ожидаемых состояний, single effect и другие инварианты PASS | [pipeline.txt](evidence/forward-published/pipeline.txt), [pipeline.json](evidence/forward-published/pipeline.json) |
| 3: сохранение данных и сессий | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | Подтверждённые данные и прежняя сессия сохранены, 14,386 s на сценарий | [persistence.txt](evidence/forward-published/persistence.txt), [hook](scripts/restart.sh) |
| 3: отказ реплики и возврат | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`; `make restore` | 2 исходные реплики, 130 успешных записей, 0 failed attempts; первый успех через 2,876 s, api-2 возвращена healthy | [failover.txt](evidence/forward-published/failover.txt), [restore.txt](evidence/forward-published/restore.txt) |
| 3: воспроизводимый общий стенд | `make up`; `make obs-validate` | Сервисы готовы, Collector/Prometheus/Loki/Tempo и JSON dashboards валидны | [up.txt](evidence/forward-published/up.txt), [obs-config.txt](evidence/forward-published/obs-config.txt), [runtime.json](evidence/forward-published/runtime.json) |
| 3: история наблюдаемости в volumes | `python3 scripts/obs_recovery.py` | Исторический прогон: 7 spans до/после, 1 лог, значение метрики 352 сохранено при рестарте storage | [telemetry-recovery.txt](evidence/grade3/telemetry-recovery.txt), [основание применимости](evidence/forward-published/preserved-mechanisms.json) |
| 3: API → лог → trace, реальные server/client spans | `python3 scripts/obs_check.py request --base-url http://127.0.0.1:18086` | `request.handled`, trace/span IDs; 7 spans, включая HTTP server и DB client | [telemetry-request.txt](evidence/forward-published/telemetry-request.txt) |
| 3: worker → лог → trace, consumer spans | `python3 scripts/obs_check.py job --base-url http://127.0.0.1:18086` | Attachment ready, 52 spans, `job.processed` / `audit.job_settled` с одним trace_id | [telemetry-job.txt](evidence/forward-published/telemetry-job.txt) |
| 3: RED API, jobs outcome/duration, provisioning | `python3 scripts/obs_check.py dashboards --strict` | 32 targets, problems=[], реальные series; два dashboard JSON в Git | [dashboards.txt](evidence/forward-published/dashboards.txt), [dashboards](infra/grafana/dashboards) |
| 3: низкая кардинальность / resource attributes | `python3 scripts/obs_check.py query 'count by (__name__, job) ({job=~"messenger-.*"})'` | Шаблон маршрута, method/status, job_type/outcome, service/instance; IDs не используются как labels | [исторические series неизменных инструментов](evidence/grade4/cardinality-series.json), [свежие dashboard queries](evidence/forward-published/dashboards.txt) |
| 3: отсутствие запрещённых данных | `python3 scripts/obs_check.py leaks --base-url http://127.0.0.1:18086` | 0 совпадений password/token/text/filename в проверенных log lines и 8 traces за 20 s; metadata Loki этим тестом не покрыта | [telemetry-leaks.txt](evidence/forward-published/telemetry-leaks.txt), [minor M1](evidence/review-summary.md) |
| 4: W3C API → outbox → broker → worker → S3/DB | `python3 scripts/obs_incidents.py chain` | 55 spans, producer/consumer, итог done | [incident-chain.txt](evidence/forward-published/incident-chain.txt) |
| 4: та же цепочка с retry | `python3 scripts/obs_incidents.py retry` | 86 spans; `retry_scheduled` → `done`, один trace | [incident-retry.txt](evidence/forward-published/incident-retry.txt) |
| 4: p50/p95/p99, error ratio, backlog, pool, saturation, CPU/RAM, time-to-ready | `python3 scripts/obs_check.py dashboards --strict` | Все используемые запросы разрешаются; имена/единицы и ограничения приведены ниже | [dashboards.txt](evidence/forward-published/dashboards.txt), [recording rules](infra/prometheus/rules/messenger.recording.yml) |
| 4: SLI/SLO, error rate alert | `python3 scripts/obs_incidents.py database` | SQL client ERROR 2002,866 ms, error ratio 0,9181, firing → resolved | [incident-database.txt](evidence/forward-published/incident-database.txt), [alert rules](infra/prometheus/rules/messenger.alerts.yml) |
| 4: queue lag alert и восстановление | `python3 scripts/obs_incidents.py worker` | 8 jobs, age 34,470 s, firing → resolved, depth 8 → 0 | [incident-worker.txt](evidence/forward-published/incident-worker.txt) |
| 4: ограниченные очереди, отказ telemetry storage | `python3 scripts/obs_incidents.py storage` | Исторический неизменный механизм: smoke PASS, 74 успешных probes, потеря 2 traces, RAM ≤252,7/256 MiB, очереди после recovery =0 | [storage-incident.txt](evidence/grade4/storage-incident.txt), [основание применимости](evidence/forward-published/preserved-mechanisms.json) |
| 4: readiness и shutdown при отказе Collector | `python3 scripts/obs_recovery.py` | Исторический прогон: ready=200, smoke PASS, shutdown API 4,592 s, exit 0 | [telemetry-recovery.txt](evidence/grade3/telemetry-recovery.txt) |
| Поздние исправления сессий/HTTP | `.venv/bin/python student-tests/tools/forward_network.py`; stack pytest из логов | Негативный Basic/NUL, logout обеих реплик, идемпотентность; Valkey bypass и logout503→204 | [network.txt](evidence/forward-published/network.txt), [cache-outage.txt](evidence/forward-published/cache-outage.txt) |
| Поздние исправления доставки/изображений | `env PYTHONPATH=.:scripts:student-tests .venv/bin/python student-tests/tools/forward_runtime.py postgres-outage` (также `png-http`, `png-boundary`) | 12 requeue без роста DLQ, тот же job/attempt; PNG 40 MP, две версии, конкурентный peak 913 227 776 <1 073 741 824 bytes, OOM=0 | [postgres-outage.json](evidence/forward-published/postgres-outage.json), [png-http.json](evidence/forward-published/png-http.json), [png-boundary.json](evidence/forward-published/png-boundary.json) |
| 5: off/on/head/tail/balanced, цена и сохранность | `python3 scripts/obs_cost.py --mode MODE --repeat N --duration 40 --rate 20 --output evidence/reproduction/MODE-N.json` | 15 пригодных серий, 12 000 запросов; balanced −92,2% Tempo ingest, 48/48 error и 48/48 slow | [методика](evidence/forward-published/cost/method.md), [comparison.json](evidence/forward-published/cost/comparison.json), [результаты](evidence/forward-published/cost/results.md) |
| OpenSpec propose/apply/archive | `openspec list --json`; `openspec validate --all --strict --no-interactive` | 0 active changes, 26 specs PASS; циклы 3/4/5 и поздний перенос архивированы | [архивы](openspec/changes/archive), [spec-sync.json](evidence/forward-published/spec-sync.json), [проверка отчёта](evidence/report-audit.md) |
`restart.sh` останавливает `lb api relay worker`, перезапускает `postgres valkey s3 broker`, затем ждёт Compose healthy с прежними volumes. Публичный тест подавляет stdout hook: 14,386 s — время всего сценария, а не измеренная длительность недоступности каждого процесса; отдельного timestamp-аудита всех перезапусков в этом логе нет. `restore.txt` непосредственно подтверждает запуск остановленной api-2. История telemetry проверена отдельным `obs_recovery.py`. Старый запуск из чистого clone на `2903d9d` — [clean-clone.txt](evidence/grade3/clean/clean-clone.txt); он не является свежим clean-clone прогоном позднего кода.
### Сигналы и запросы
Resource: `service.name=messenger-api|messenger-relay|messenger-worker`, `service.version=1.6.0`, уникальный `service.instance.id=<hostname>-<pid>`, `deployment.environment.name=lab`. В Prometheus ресурс виден через `job`, `instance` и `target_info`; в Loki — нормализованные поля, включая `service_name`. Технические request/job IDs допустимы в логах/spans для корреляции синтетических данных, но не в labels метрик.
| Логический сигнал контракта | Реальное имя / поля | Единицы и измерения / запрос |
| --- | --- | --- |
| HTTP requests | `http_server_request_duration_seconds_count` | counter наблюдений histogram; `http_request_method`, `http_route`, `http_response_status_code`; `sum(messenger:http_requests:rate2m)` — RPS; recording rule формирует `status_class` |
| HTTP duration | `http_server_request_duration_seconds_bucket` | секунды; `histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job="messenger-api",http_route!~"/health/.*"}[5m])))`; для p50/p99 заменить 0.95 |
| HTTP errors | `messenger:http_errors:ratio_rate2m` | 5xx / все бизнес-запросы, без health; 4xx остаются в знаменателе, не считаются отказами |
| Jobs processed / failed | `messenger_jobs_processed_total` | доставки, `job_type`, `outcome`; `sum by(job_type,outcome)(rate(messenger_jobs_processed_total[5m]))`; outcome содержит done/retry_scheduled/failed и другие ветки |
| Job duration | `messenger_job_duration_seconds_bucket` | секунды обработки доставки, `job_type`, `outcome`; percentile из суммы bucket rates |
| Time to ready | `messenger_job_time_to_ready_seconds_bucket` | секунды от Job.created_at до done после commit, `job_type`; учитывает ожидание и retries |
| Queue depth / oldest age | `messenger_queue_depth`, `messenger_queue_oldest_age_seconds` | jobs / секунды, `queue_name=attachments.process`; SQLAlchemy poll 5 s, durable queued/processing включая leased/retry, не только AMQP messages_ready |
| DB pool | `messenger_db_pool_used`, `_waiting`, `_capacity` | занятые connections / acquisitions / лимит; waiting включает получение/создание соединения, не только очередь слотов |
| Worker saturation | `messenger_worker_busy`, `messenger_worker_capacity` | активные deliveries / доступная capacity; отношение сумм даёт долю занятости |
| Process CPU / RAM | `process_cpu_time_seconds_total`, `process_memory_usage_bytes` | `sum by(job)(rate(process_cpu_time_seconds_total[2m]))` — cores; `sum by(job)(process_memory_usage_bytes)` — bytes RSS; CPU renderer-детей не входит |
| Logs | timestamp, severity, resource service.name, event, trace_id, span_id | `request.handled`, `job.processed`; LogQL `{service_name="messenger-api"} | trace_id="<trace_id>"`; поля OTel представлены structured metadata |
| Traces | SERVER, PRODUCER, CONSUMER, CLIENT | Tempo по trace_id; DB/S3 операции, duration/status, связь parent; ссылки из Grafana provisioning |
HTTP histogram (секунды): 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10, +Inf. Job/time-to-ready: 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, +Inf. Они различают быстрый HTTP, таймауты зависимостей и 30-секундный SLO job. Квантили агрегируются по buckets всех реплик; среднее арифметическое p99 реплик не используется. CPU/RAM трека дополнительно измерены извне по cgroups, независимо от состояния SDK.
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
### SLI/SLO и диагностика
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
API availability: `1 − 5xx/все бизнес-HTTP` за **1 h**, цель **≥99%**, health исключён. Это SLI приложения: ответы самого HAProxy не входят. Job SLI: доля **завершённых** jobs с time-to-ready **≤30 s** за **1 h**, цель **≥95%**; незавершённые отдельно контролируются возрастом backlog. Эти цели заданы и измеримы, но короткие опыты не доказывают месячный production SLO.
`MessengerHighErrorRate`: доля 5xx >5% за 1 min, `for: 15s`. `MessengerQueueLag`: max age >15 s и depth >0, `for: 15s`. Проверка rules каждые 5 s; доставка наружу не требуется. [Runbooks](infra/prometheus/runbooks.md) связывают alert с запросами и действиями.
| Воздействие и доказательство | Ожидание и наблюдение | Время / потери |
| --- | --- | --- |
| Остановка worker; [свежий лог](evidence/forward-published/incident-worker.txt) | API принимает 8 jobs, backlog=8, oldest=34,470 s; после возврата depth=0, alert resolved | UTC 06.11: stop 23:47:05, firing 23:47:49 (**44 s**), restore 23:47:52, resolved 23:48:04 (**12 s** после возврата). Все 8 jobs обработаны |
| Ограниченная SQL-блокировка users; [свежий лог](evidence/forward-published/incident-database.txt) | 168 контролируемых 503; error ratio=0,9181, pool waiting=0; SELECT client ERROR длится 2002,866 ms | UTC 06.11: lock 23:48:16, firing/снятие lock 23:48:59 (**43 s**), resolved 23:49:54 (**55 s**). Блокировка сама данные не меняет; полноценный RPO здесь не измерялся |
| S3 временно недоступен; [retry](evidence/forward-published/incident-retry.txt) | Одна цепочка из 86 spans показывает retry_scheduled → done, затем DB commit | Сценарий 23:46:51–23:47:04 UTC (≈13 s); принятое вложение ready, отдельное время unavailable не выделено |
| PostgreSQL недоступен worker; [JSON](evidence/forward-published/postgres-outage.json) | 12 requeue, delivery-limit=-1; job_id и attempt=1 сохранены, DLQ 0→0, после восстановления done без replay | Targeted case 20,39 s; это время кейса, не точный RTO. Потери принятого задания не обнаружены |
| Остановка одной API-реплики; [failover](evidence/forward-published/failover.txt) | 130 writes, 0 failed attempts, подтверждённые данные и сессия сохранились | Первый успешный запрос после воздействия через 2,876 s; restore завершился за 3,660 s |
| Рестарт приложений и business stores; [persistence](evidence/forward-published/persistence.txt) | Сессия и подтверждённые данные сохранены; volumes не удаляются | 14,386 s на весь сценарий; отдельный RTO не измерен |
| Остановка Prometheus/Loki/Tempo; **исторический** [storage](evidence/grade4/storage-incident.txt), неизменные механизмы | Smoke PASS, 74 успешных probes; queue logs/metrics/traces=381/1287/465, после recovery 0/0/0; Collector без наблюдаемого OOM | UTC 06.11: 18:14:29–18:16:16 (**107 s**), проверка закончилась 18:16:39. 72/74 traces найдены, **2 потеряны**; RAM 252,7/256 MiB (98,7%) |
| Остановка Collector / рестарт telemetry stores; **исторический** [recovery](evidence/grade3/telemetry-recovery.txt) | Ready=200, smoke PASS, ранее сохранённая история доступна | API shutdown 4,592 s, exit 0; 7/7 прежних spans и лог найдены после рестарта |
Первый инцидент диагностируется по растущим depth/age при отсутствии обработки: после возврата worker backlog исчезает, значит узкое место — consumer. Во втором 503 server span содержит PostgreSQL client ERROR около 2 s, при этом pool waiting=0: причина в выполнении SQL, а не в очереди пула или просто медленном HTTP handler. Снятие lock и последующий resolved подтверждают гипотезу. Это разбор сохранённых наблюдений контролируемых отказов; слепой runbook-прогон другого оператора (трек 2) не заявлен.
При отказе telemetry storage retry ограничен 60 s. В сохранённых counters есть 440 failed span exports, 11 log records и 72 metric points; эти единицы не равны числу потерянных запросов/полных трасс. Непосредственный контроль trace IDs показал 2 потерянные трассы из 74 успешных запросов. Рост памяти ограничен наблюдаемым окном и лимитом контейнера; запаса Collector почти нет, бесконечная устойчивость не доказана.
### Трек 1: цена наблюдаемости
Гипотеза: head sampling снижает ingest, но теряет важные события; tail sampling оставляет ERROR/slow и уменьшает запись в Tempo, сохраняя расходы SDK. Сбалансированный режим должен сохранить контрольные события при меньшей записи обычных traces и более редком экспорте метрик.
Измеряемый SHA: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`, image ID `sha256:66c7d75903f514e14fea21689d039eeba1d9d85a0d27ef3d70d4ab41288e2f6d`. Серии выполнены 06–07.11.2026 UTC. Нагрузка — GET `/api/v1/users/me` по одному созданному для серии пользователю/сессии; это HTTP/DB-профиль, не массовый рендер изображений. Существующие volumes между режимами сохраняются, полный размер БД не нормировался. Условия, host load и список контейнеров сохранены в [conditions-start.json](evidence/forward-published/cost/conditions-start.json), снимках каждого круга и raw JSON. В снимках других Docker projects нет; эксклюзивность всего host не заявляется.
| Режим | SDK / выбор traces | Метрики |
| --- | --- | --- |
| off | SDK полностью выключен | Нет метрик приложения; self-scrapes storage продолжаются |
| on | Все server roots и их дочерние spans | Каждые 10 s |
| head | ParentBased root ratio 10% | Каждые 10 s |
| tail | Все spans до Collector; сохранить ERROR, latency ≥300 ms, 10% остальных | Каждые 10 s |
| balanced | Все spans до Collector; ERROR, latency ≥300 ms, 5% остальных | Каждые 30 s |
По три повтора каждого режима, последовательно off/on/head/tail/balanced. 20 прогревочных GET и 35 s ожидания экспорта; затем **40 s open-loop, 20 arrivals/s, 800 запросов**, максимум 64 клиентских потока. В каждой серии 768 обычных, 16 ожидаемых 503 и 16 задержек 350 ms; после — 35 s drain. Всего **12 000 запросов**, 11 760 ожидаемых 200 и 240 намеренных 503, **0 неожиданных ответов**. Фактическая скорость 20,012–20,020 req/s; максимум schedule lag 61,055 ms. Это заданная нагрузка, не поиск предельного throughput.
Команда серии и разбор всех параметров — [method.md](evidence/forward-published/cost/method.md). CPU: delta cgroup CPU time / длительность внешних samples; RAM: working set usage−inactive_file, 21 sample с периодом около 2 s. Application = API/relay/worker; observability = Collector/Prometheus/Loki/Tempo/Grafana; business stores/LB считаются отдельно. CPU-окно охватывает нагрузку, но не всю работу tail во время drain. Ingest включает drain; bytes Tempo/Loki и samples Prometheus не складываются в одну единицу.
Медианы трёх повторов; диапазон normal p95 — min/max, не доверительный интервал. Общие p50/p95/p99 посчитаны клиентом по всем категориям, поэтому p99 отражает вброшенную задержку.
| Режим | p50 / p95 / p99, ms | normal p95, ms [min;max] | CPU app / obs / stores+LB, cores | RAM app / obs / stores+LB, MiB (median peak) |
| --- | --- | --- | --- | --- |
| off | 8,052 / 12,102 / 358,732 | 11,583 [11,572;11,856] | 0,1404 / 0,0256 / 0,1085 | 655,5 / 1013,3 / 419,9 |
| on | 7,460 / 13,148 / 359,469 | 12,709 [12,554;12,993] | 0,1726 / 0,0287 / 0,1030 | 706,8 / 1058,5 / 425,5 |
| head | 9,195 / 13,116 / 358,373 | 12,556 [12,354;12,652] | 0,1657 / 0,0261 / 0,1023 | 723,8 / 1076,1 / 347,1 |
| tail | 8,942 / 13,281 / 359,109 | 12,781 [12,555;12,920] | 0,1717 / 0,0272 / 0,1026 | 721,0 / 1021,2 / 430,6 |
| balanced | 8,269 / 13,230 / 359,753 | 12,895 [12,833;13,031] | 0,1749 / 0,0258 / 0,1030 | 707,1 / 1027,9 / 423,1 |
Фактические лимиты: API 2×(1 CPU/512 MiB), relay 0,5 CPU/192 MiB, worker 2 CPU/1 GiB; Collector 0,5/256 MiB, Prometheus 0,5/512 MiB, Loki/Tempo/Grafana каждый 0,5/384 MiB; PostgreSQL 2/768 MiB, S3 1/512 MiB, RabbitMQ 1/512 MiB, Valkey 0,5/128 MiB, LB 1/128 MiB. **PG_MEMORY в измерении 768m, в `.env.example` — 1g**: для повторения именно этих условий задайте `PG_MEMORY=768m` до `make up`. Остальные значения сверяйте с [runtime.json](evidence/forward-published/runtime.json).
| Режим | Tempo / Loki, KiB (median) | Prom samples (median) | SDK ended / sent spans (median) | Найдено normal /2304 | error /48 | slow /48 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| off | 0,0 / 0,0 | 33118 | 0 / 0 | 0 | 0 | 0 |
| on | 1662,2 / 288,3 | 36284 | 5489 / 5489 | 2304 | 48 | 48 |
| head | 176,2 / 288,4 | 37234 | 543 / 543 | 237 | 6 | 3 |
| tail | 251,8 / 289,7 | 36657 | 5492 / 699 | 235 | 48 | 48 |
| balanced | 130,2 / 289,8 | 34459 | 5486 / 352 | 101 | 48 | 48 |
Во всех пригодных сериях **SDK ended = Collector accepted**, exporter failures=0. Head намеренно не создал 42/48 error и 45/48 slow server traces; число несозданных дочерних spans неизвестно и не выдумывается. Tail/balanced фильтруют уже созданные spans; accepted−sent здесь не означает неисправность доставки. Off означает отключение SDK. В режиме on найдены все 2400 server traces трёх повторов.
**Balanced снизил медианный Tempo ingest на 92,2% относительно on**, сохранив 48/48 error и 48/48 slow. Loki почти не изменился; Prometheus включает постоянный фон self-scrapes. Normal p95 у on выше off примерно на 1,127 ms (+9,7%), CPU приложения — 0,1726 против 0,1404 cores; это описательные разности данного опыта, не доказательство статистической значимости. Balanced не устраняет цену SDK и не показал ускорения HTTP. Проверенная рекомендация относится к сокращению записи traces при этом профиле и окне решения tail 10 s.
![Повторы, медианы и диапазоны цены наблюдаемости](evidence/forward-published/cost/comparison.png)
[Сырые серии и comparison.json](evidence/forward-published/cost/comparison.json), [сводка](evidence/forward-published/cost/results.md), [аудит пригодности](evidence/forward-published/cost/measurement-audit.json). Первоначальные on-3/head-3 (ещё 1600 запросов) исключены из сравнения из-за разрывов wall clock: 932,714 s после нагрузки и 938,154 s вместо 40 s, gap samples до 900,354 s. Их HTTP PASS не означает пригодность CPU/ingest. Оригиналы с hashes сохранены в [invalid-series.json](evidence/forward-published/cost/invalid-series.json) и [invalid-time-gap](evidence/forward-published/cost/invalid-time-gap); повторены только эти две серии. Причина разрыва времени достоверно не установлена.
Проверенный запуск balanced:
```sh
OTEL_COLLECTOR_CONFIG=collector-balanced.yaml OTEL_METRIC_EXPORT_INTERVAL=30000 scripts/lab6.sh docker compose up -d --no-deps --wait otel-collector api relay worker
```
После опытов возвращены SDK on/ratio1/interval10s, `collector.yaml`, `COST_EXPERIMENT=false`: [defaults-restored.json](evidence/forward-published/defaults-restored.json). Стенд остановлен без удаления данных, 9 volumes сохранены: [shutdown.txt](evidence/forward-published/shutdown.txt), [closure.json](evidence/forward-published/closure.json). Это зафиксированное состояние окончания реализации, не новое обращение отчётного этапа к Docker.
### Финальная validation перед публикацией
На входном HEAD `1e00d861a803a6d49ef443175b8231cbf048fc02` аудит подтвердил применимость всех 15 серий: runtime и измеритель совпадают с прошедшим ревью кодом. Все raw JSON и comparison пересчитаны, временные окна и hashes исключённых оригиналов проверены. **Нагрузку повторно не запускали.** [Аудит](evidence/load-validation.md) содержит решение по каждой серии, конфигурации, dataset, фону и ограничениям.
Свежие `make check validate lint` и OpenSpec strict (26 specs) прошли. Свежий полный `make student-test` завершился с 993 passed, 2 skipped, 19 deselected и 2 ошибками подготовки RustFS. В окружении был системный прокси macOS без обхода localhost; только два затронутых теста повторены штатной обвязкой с `NO_PROXY`: 2 passed. Полный первый запуск не объявляется PASS; оба лога сохранены. Код и прежние сырые измерения не менялись; публикация в фазе validation не выполнялась.
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
Ревью `6ca1c7549978af3127966540385195ab0587f2e1` не выявило major. Пять minor сохранены; по правилу пользователя они не блокируют отчёт/release и в документационном этапе не исправляются:
| ID | Ограничение и последствие | Файлы / доказательства |
| --- | --- | --- |
| M1 | Loki leak-check ищет маркеры через line filter, но не во всех attributes/structured metadata. Нулевые hits — ограниченная проверка, не полный аудит отсутствия секретов. Фактическая утечка не обнаружена | [obs_check.py](scripts/obs_check.py), [telemetry.py](src/messenger/shared/infrastructure/telemetry.py), [leaks](evidence/forward-published/telemetry-leaks.txt) |
| M2 | Унаследованные allowlists `make limits` и published-port audit CI не учитывают OTel/helper services и отклоняют штатную ЛР6. Весь CI не объявляется зелёным; лимиты проверены отдельно | [limits.py](scripts/limits.py), [workflow](.github/workflows/application.yml), [runtime](evidence/forward-published/runtime.json) |
| M3 | API SLI не учитывает 503 HAProxy, rate новой серии может пропустить первый всплеск. Полная пауза БД не вызвала alert; успешен именно отдельный SQL-lock сценарий | [alerts](infra/prometheus/rules/messenger.alerts.yml), [неуспешный старый опыт](evidence/grade4/database-incident.txt), [свежий SQL-lock](evidence/forward-published/incident-database.txt) |
| M4 | Collector достиг 98,7% лимита (252,7/256 MiB) без наблюдаемого OOM. Запас мал; опыт не доказывает устойчивость при любой нагрузке/длительности отказа | [collector.yaml](infra/otel/collector.yaml), [compose](compose.yaml), [storage](evidence/grade4/storage-incident.txt) |
| M5 | Whitespace в SVG и сыром stdout: ревью получило `git show --check` exit 2, 2320 замечаний. Это косметика артефактов; `make lint` исходников проходит | [comparison.svg](evidence/forward-published/cost/comparison.svg), [whitespace-check.txt](evidence/forward-published/whitespace-check.txt) |
Полный текст и происхождение замечаний — [review-summary.md](evidence/review-summary.md). Улучшения охвата leak-check, CI allowlists, edge SLI и запаса памяти остаются будущей работой; результат их реализации не заявлен.
Дополнительные границы: один host, одиночные PostgreSQL/RabbitMQ/S3/Collector и локальные volumes, без HA этих компонентов и disaster recovery. Сбой узла и удаление volumes не покрыты. TLS/mTLS, внешние receivers alerts и multi-window burn-rate относятся к другим задачам. Три повтора, фиксированный порядок, общий host и накопленная история storage ограничивают выводы о CPU/RAM/latency; CPU во время drain и непрерывный пик RAM не измерены. Tail window 10 s не гарантирует полноту очень поздних retry. Queue gauge — durable backlog; time-to-ready SLI не включает pending; process CPU не включает renderer. Full suite имеет 2 macOS skip и 19 deselected stack, поэтому дополнен targeted Docker evidence, но не выдаётся за исполнение всех тестов.
Старые `grade3`, `grade4`, `track5`, `level3.json` привязаны к своим SHA в [индексе](evidence/README.md); прежние 519 passed и −91,1% Tempo не являются результатом позднего кода. Неуспешные/прерванные smoke, retry, persistence, тесты и измерения сохранены и размечены в исторических notes. Новый clean-clone и динамический прогон в отчётном этапе не выполнялись. Сырые evidence, OpenSpec и реализация не менялись. Конфигурация OpenSpec уже актуальна; новая proposal/apply/archive для документационного этапа не создавалась по его явному ограничению на изменения.
## Вопросы защиты
1. **Чем отличаются три сигнала?** Метрика показывает рост error rate/backlog, лог даёт событие и технический ID, trace показывает длительность зависимостей и причинную цепочку.
2. **Почему нельзя chat_id в labels?** Число series растёт с чатами и комбинациями labels; память/индекс хранилища расходуются даже при малом RPS. Нужен route template; ID оставляют для выборочного поиска логов/traces.
3. **Как связаны API и worker?** W3C carrier сохраняется вместе с durable job/outbox, затем идёт в AMQP headers; consumer извлекает parent, retry сохраняет связь. Сбой relay не стирает контекст из БД.
4. **Почему ack после COMMIT?** Ранний ack может потерять работу при падении до commit; поздний допускает повтор, который нейтрализуют effect key и fencing.
5. **Почему SQLAlchemy и Alembic?** ORM/Core задают единые транзакции/блокировки и пул, Alembic — упорядоченный head с lock/legacy adoption. OTel asyncpg инструментация наблюдает драйвер под SQLAlchemy, не заменяя его.
6. **Почему среднее не заменяет p99?** Редкая задержка скрывается средним; p99 реплик нельзя усреднять. Сначала складываются bucket rates, затем вычисляется quantile.
7. **Как отличили DB от медленного handler?** В сохранённом trace SELECT client ERROR около 2 s внутри server 503, pool waiting=0; снятие SQL lock убрало причину и затем alert.
8. **Почему head потерял ошибки, а balanced сохранил?** Head решает до исхода, tail — после получения ERROR/duration. Цена tail — создание всех spans и ограниченное ожидание в памяти Collector.
9. **Почему наблюдаемость может теряться без потери сообщения?** Её очереди и retry конечны и отделены от бизнес-коммита; 2/74 traces потеряны при продолжающем работать API.
10. **Что доказано треком?** Сокращение записи в Tempo на 92,2% с сохранением контрольных событий при 20 req/s. Production capacity, статистическое ускорение и полнота всех поздних retry не доказаны.
## Источники и помощь
Нормы: [README](README.md), [контракт наблюдаемости](contracts/observability.md), [API](contracts/openapi.json), [COURSE](COURSE.md). Архитектура портов/UoW — [конспект Персиваля—Грегори](openspec/notes/architecture-patterns-python.md). Код и проверенные конфигурации версионируются в этом репозитории.
SQLAlchemy async sessions/engine и Alembic async `run_sync` сверены с [SQLAlchemy 2.0](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html) и [Alembic cookbook](https://alembic.sqlalchemy.org/en/latest/cookbook.html#using-asyncio-with-alembic). Context7 в этом этапе ответил `Monthly quota exceeded`, поэтому использована официальная документация; закреплённые версии проекта не обновлялись. Остальные первичные материалы OTel, W3C, Prometheus и Grafana перечислены в README; фактические имена/настройки сверены с локальным кодом и evidence.
Codex использован как инструмент подготовки реализации, проверок, анализа и отчёта; автор работы — Костылев Артём. Независимое ревью получено отдельным этапом runner. Динамические результаты в этом документе — сохранённые измерения исполнителей, а не повторные прогоны агента отчёта.
+4
View File
@@ -0,0 +1,4 @@
[alembic]
script_location = %(here)s/alembic
prepend_sys_path = %(here)s/src %(here)s/alembic
path_separator = space
+102
View File
@@ -0,0 +1,102 @@
import asyncio
from alembic import context
from alembic.script import ScriptDirectory
from legacy import adopt_legacy
from sqlalchemy import Connection, func, make_url, pool, select
from sqlalchemy.ext.asyncio import create_async_engine
from messenger.attachments.infrastructure.postgres import (
AttachmentRow,
JobEventRow,
JobRow,
OutboxRow,
VariantRow,
)
from messenger.identity.infrastructure.postgres import SessionRow, UserRow
from messenger.messaging.infrastructure.postgres import (
ChatMemberRow,
ChatRow,
IdempotencyKeyRow,
MessageAttachmentRow,
MessageRow,
)
from messenger.shared.config import PlatformSettings
LOCK_KEY = 20_260_002
MODELS = (
UserRow,
SessionRow,
ChatRow,
ChatMemberRow,
MessageRow,
IdempotencyKeyRow,
AttachmentRow,
JobRow,
OutboxRow,
JobEventRow,
VariantRow,
MessageAttachmentRow,
)
target_metadata = UserRow.metadata
config = context.config
def include_object(_obj, name, _type, _reflected, _compare_to):
return name != "schema_migrations"
def run_migrations(connection: Connection) -> None:
with connection.engine.connect() as lock:
lock.execute(select(func.pg_advisory_lock(LOCK_KEY)))
lock.commit()
try:
with connection.begin():
adopt_legacy(connection, ScriptDirectory.from_config(config))
context.configure(
connection=connection,
target_metadata=target_metadata,
transaction_per_migration=True,
compare_type=True,
include_object=include_object,
)
with context.begin_transaction():
context.run_migrations()
finally:
lock.execute(select(func.pg_advisory_unlock(LOCK_KEY)))
lock.commit()
async def run_async_migrations() -> None:
dsn = config.attributes.get("database_url")
if dsn is None:
dsn = PlatformSettings().database_url.get_secret_value()
engine = create_async_engine(
make_url(dsn).set(drivername="postgresql+asyncpg"),
poolclass=pool.NullPool,
hide_parameters=True,
connect_args={
"timeout": 10,
"server_settings": {"lock_timeout": "10000", "statement_timeout": "60000"},
},
)
try:
async with engine.connect() as connection:
await connection.run_sync(run_migrations)
finally:
await engine.dispose()
if context.is_offline_mode():
context.configure(
dialect_name="postgresql",
target_metadata=target_metadata,
literal_binds=True,
transaction_per_migration=True,
)
with context.begin_transaction():
context.run_migrations()
elif config.attributes.get("connection") is not None:
run_migrations(config.attributes["connection"])
else:
asyncio.run(run_async_migrations())
+94
View File
@@ -0,0 +1,94 @@
from uuid import uuid4
from alembic.migration import MigrationContext
from alembic.operations import Operations
from alembic.script import ScriptDirectory
from sqlalchemy import Connection, MetaData, Table, func, inspect, select, text
from sqlalchemy.schema import CreateSchema
LEGACY = {
"0001_init": "9a13618f078ae9056431391e2e5197a31b981ae1fff047a9b3421ac750d3fd0d",
"0002_idempotency_keys": "d9159a35553249a1e51af5f2e4e7c39b014e97593c49748e13f3799839063dd0",
"0003_attachments": "c4417f560538a083efba1c67f660147dc94532fcbb2fa91f3f3e8258a0f01f2e",
"0004_jobs": "c3887fa7a886ae8153a216aa43fd4c3c6190a2074e8ce20c3e77f29bfb01550e",
"0005_failed_registry": "02b80061a08a6929feb092148c453ad970e3d001901c651b5a73b90b1e40f145",
"0006_attachment_quarantine": (
"541f5399d87fac3584c4102b4e0bf92c23cd372cc20a7d56dad03e771773202c"
),
}
SNAPSHOT = text("""
WITH relations AS (
SELECT c.* FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE n.nspname = :schema AND c.relkind IN ('r', 'p')
AND c.relname NOT IN ('schema_migrations', 'alembic_version')
)
SELECT 'table' AS kind, c.relname AS name,
concat_ws('|', c.relkind, c.relpersistence,
c.relrowsecurity, c.relforcerowsecurity) AS definition
FROM relations c
UNION ALL
SELECT 'column', c.relname || '.' || a.attname,
concat_ws('|', a.attnum, format_type(a.atttypid, a.atttypmod), a.attnotnull,
a.attidentity, a.attgenerated, a.attcollation::regcollation,
pg_get_expr(d.adbin, d.adrelid))
FROM relations c JOIN pg_attribute a ON a.attrelid = c.oid
LEFT JOIN pg_attrdef d ON d.adrelid = c.oid AND d.adnum = a.attnum
WHERE a.attnum > 0 AND NOT a.attisdropped
UNION ALL
SELECT 'constraint', c.relname || '.' || k.conname,
concat_ws('|', pg_get_constraintdef(k.oid), k.convalidated)
FROM relations c JOIN pg_constraint k ON k.conrelid = c.oid
UNION ALL
SELECT 'index', c.relname || '.' || x.relname,
concat_ws('|', pg_get_indexdef(i.indexrelid), i.indisvalid, i.indisready)
FROM relations c JOIN pg_index i ON i.indrelid = c.oid
JOIN pg_class x ON x.oid = i.indexrelid
UNION ALL
SELECT 'trigger', c.relname || '.' || t.tgname,
concat_ws('|', pg_get_triggerdef(t.oid), t.tgenabled)
FROM relations c JOIN pg_trigger t ON t.tgrelid = c.oid WHERE NOT t.tgisinternal
ORDER BY kind, name
""")
def snapshot(connection: Connection, schema: str) -> list[tuple[str, str, str]]:
return [
(kind, name, definition.replace(f"{schema}.", ""))
for kind, name, definition in connection.execute(SNAPSHOT, {"schema": schema})
]
def adopt_legacy(connection: Connection, scripts: ScriptDirectory) -> None:
migration = MigrationContext.configure(connection)
if migration.get_current_heads():
return
tables = set(inspect(connection).get_table_names(schema="public"))
if tables <= {"alembic_version"}:
return
if "schema_migrations" not in tables:
raise RuntimeError("Existing schema has no legacy migration journal")
journal = Table("schema_migrations", MetaData(), schema="public", autoload_with=connection)
applied = dict(connection.execute(select(journal.c.version, journal.c.checksum)).tuples().all())
expected = dict(list(LEGACY.items())[: len(applied)])
if applied != expected:
raise RuntimeError("Unknown or changed legacy migrations")
if not applied:
if tables - {"schema_migrations", "alembic_version"}:
raise RuntimeError("Existing schema has an empty migration journal")
return
revision = list(expected)[-1]
actual = snapshot(connection, "public")
reference = f"alembic_verify_{uuid4().hex}"
savepoint = connection.begin_nested()
try:
connection.execute(CreateSchema(reference))
connection.execute(select(func.set_config("search_path", reference, True)))
with Operations.context(MigrationContext.configure(connection)):
for script in reversed(list(scripts.walk_revisions(base="base", head=revision))):
script.module.upgrade()
wanted = snapshot(connection, reference)
finally:
savepoint.rollback()
if actual != wanted:
raise RuntimeError("Legacy schema differs from the recorded migrations")
migration.stamp(scripts, revision)
+16
View File
@@ -0,0 +1,16 @@
from alembic import op
import sqlalchemy as sa
${imports if imports else ""}
revision = ${repr(up_revision)}
down_revision = ${repr(down_revision)}
branch_labels = ${repr(branch_labels)}
depends_on = ${repr(depends_on)}
def upgrade() -> None:
${upgrades if upgrades else "pass"}
def downgrade() -> None:
${downgrades if downgrades else "pass"}
+87
View File
@@ -0,0 +1,87 @@
import sqlalchemy as sa
from alembic import op
revision = "0001_init"
down_revision = None
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"users",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("username", sa.Text(), nullable=False),
sa.Column("display_name", sa.Text(), nullable=False),
sa.Column("password_hash", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("password_hash LIKE '$argon2id$%'", name="users_password_hash_check"),
sa.CheckConstraint("username ~ '^[a-z0-9_]{3,32}$'", name="users_username_format"),
sa.CheckConstraint(
"char_length(display_name) BETWEEN 1 AND 100", name="users_display_name_check"
),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("username", name="users_username_key"),
)
op.create_table(
"sessions",
sa.Column("token_hash", sa.LargeBinary(), nullable=False),
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("expires_at > created_at", name="sessions_check"),
sa.CheckConstraint("octet_length(token_hash) = 32", name="sessions_token_hash_check"),
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("token_hash"),
)
op.create_index("sessions_user_idx", "sessions", ["user_id"], unique=False)
op.create_table(
"chats",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("title", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("last_message_at", sa.DateTime(timezone=True), nullable=True),
sa.CheckConstraint("char_length(title) BETWEEN 1 AND 100", name="chats_title_check"),
sa.PrimaryKeyConstraint("id"),
)
op.create_table(
"chat_members",
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.Column("position", sa.SmallInteger(), nullable=False),
sa.Column("chat_created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("position BETWEEN 1 AND 100", name="chat_members_position_check"),
sa.ForeignKeyConstraint(["chat_id"], ["chats.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(
["user_id"],
["users.id"],
),
sa.PrimaryKeyConstraint("chat_id", "user_id"),
)
op.create_index(
"chat_members_user_idx",
"chat_members",
["user_id", "chat_created_at", "chat_id"],
unique=False,
)
op.create_table(
"messages",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("sender_id", sa.Uuid(), nullable=False),
sa.Column("text", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("char_length(text) BETWEEN 1 AND 4000", name="messages_text_check"),
sa.ForeignKeyConstraint(
["chat_id", "sender_id"],
["chat_members.chat_id", "chat_members.user_id"],
),
sa.PrimaryKeyConstraint("id"),
)
op.create_index(
"messages_history_idx", "messages", ["chat_id", "created_at", "id"], unique=False
)
def downgrade() -> None:
raise RuntimeError("Downgrade would delete application data")
+33
View File
@@ -0,0 +1,33 @@
import sqlalchemy as sa
from alembic import op
revision = "0002_idempotency_keys"
down_revision = "0001_init"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"idempotency_keys",
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("user_id", sa.Uuid(), nullable=False),
sa.Column("key", sa.Text(), nullable=False),
sa.Column("fingerprint", sa.LargeBinary(), nullable=False),
sa.Column("message_id", sa.Uuid(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.CheckConstraint("key ~ '^[A-Za-z0-9._:-]{1,128}$'", name="idempotency_keys_key_check"),
sa.CheckConstraint(
"octet_length(fingerprint) = 32", name="idempotency_keys_fingerprint_check"
),
sa.ForeignKeyConstraint(
["chat_id", "user_id"],
["chat_members.chat_id", "chat_members.user_id"],
),
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
sa.PrimaryKeyConstraint("chat_id", "user_id", "key"),
)
def downgrade() -> None:
raise RuntimeError("Downgrade would delete application data")
+77
View File
@@ -0,0 +1,77 @@
import sqlalchemy as sa
from alembic import op
revision = "0003_attachments"
down_revision = "0002_idempotency_keys"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.drop_constraint("messages_text_check", "messages", type_="check")
op.create_check_constraint("messages_text_check", "messages", "char_length(text) <= 4000")
op.create_table(
"attachments",
sa.Column("id", sa.Uuid(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("owner_id", sa.Uuid(), nullable=False),
sa.Column("filename", sa.Text(), nullable=False),
sa.Column("content_type", sa.Text(), nullable=False),
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
sa.Column("sha256", sa.Text(), nullable=False),
sa.Column("width", sa.Integer(), nullable=False),
sa.Column("height", sa.Integer(), nullable=False),
sa.Column("object_key", sa.Text(collation="C"), nullable=False),
sa.Column("status", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("id"),
sa.UniqueConstraint("object_key", name="attachments_object_key_key"),
sa.UniqueConstraint("id", "chat_id", name="attachments_chat_key"),
sa.ForeignKeyConstraint(
["chat_id", "owner_id"],
["chat_members.chat_id", "chat_members.user_id"],
name="attachments_owner_fkey",
),
sa.CheckConstraint(
"char_length(filename) BETWEEN 1 AND 255", name="attachments_filename_check"
),
sa.CheckConstraint(
"content_type IN ('image/png', 'image/jpeg')", name="attachments_content_type_check"
),
sa.CheckConstraint(
"size_bytes BETWEEN 1 AND 10485760", name="attachments_size_bytes_check"
),
sa.CheckConstraint("sha256 ~ '^[0-9a-f]{64}$'", name="attachments_sha256_check"),
sa.CheckConstraint("width > 0", name="attachments_width_check"),
sa.CheckConstraint("height > 0", name="attachments_height_check"),
sa.CheckConstraint("status IN ('pending', 'ready')", name="attachments_status_check"),
)
op.create_index(
"attachments_pending_idx",
"attachments",
["created_at"],
postgresql_where=sa.text("status = 'pending'"),
)
op.create_table(
"message_attachments",
sa.Column("message_id", sa.Uuid(), nullable=False),
sa.Column("position", sa.SmallInteger(), nullable=False),
sa.Column("chat_id", sa.Uuid(), nullable=False),
sa.Column("attachment_id", sa.Uuid(), nullable=False),
sa.PrimaryKeyConstraint("message_id", "position"),
sa.UniqueConstraint("message_id", "attachment_id"),
sa.CheckConstraint("position BETWEEN 0 AND 9", name="message_attachments_position_check"),
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
sa.ForeignKeyConstraint(
["attachment_id", "chat_id"],
["attachments.id", "attachments.chat_id"],
name="message_attachments_attachment_fkey",
ondelete="RESTRICT",
),
)
op.create_index("message_attachments_attachment_idx", "message_attachments", ["attachment_id"])
def downgrade() -> None:
raise RuntimeError("Downgrade would delete application data")
+146
View File
@@ -0,0 +1,146 @@
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects import postgresql
revision = "0004_jobs"
down_revision = "0003_attachments"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.drop_constraint("attachments_status_check", "attachments", type_="check")
op.create_check_constraint(
"attachments_status_check",
"attachments",
"status IN ('pending', 'queued', 'processing', 'ready', 'failed')",
)
for name in ("staging_key", "original_key"):
op.add_column("attachments", sa.Column(name, sa.Text(collation="C"), nullable=True))
op.create_unique_constraint(f"attachments_{name}_key", "attachments", [name])
op.add_column(
"attachments",
sa.Column("published_version", sa.Integer(), nullable=False, server_default="0"),
)
op.create_check_constraint(
"attachments_published_version_check", "attachments", "published_version >= 0"
)
op.add_column("attachments", sa.Column("error_code", sa.Text(), nullable=True))
op.create_check_constraint(
"attachments_error_code_check",
"attachments",
"error_code IN ('invalid_image', 'processing_failed', 'retry_exhausted')",
)
op.alter_column("attachments", "object_key", nullable=True)
attachments = sa.table(
"attachments",
sa.column("status"),
sa.column("object_key"),
sa.column("original_key"),
sa.column("staging_key"),
sa.column("published_version"),
)
op.execute(
attachments.update()
.where(attachments.c.status == "ready")
.values(original_key=attachments.c.object_key, published_version=1)
)
op.execute(
attachments.update()
.where(attachments.c.status == "pending")
.values(staging_key=attachments.c.object_key)
)
op.create_index(
"attachments_staged_idx",
"attachments",
["updated_at"],
postgresql_where=sa.column("staging_key").is_not(None),
)
op.create_table(
"attachment_variants",
sa.Column("attachment_id", sa.Uuid(), sa.ForeignKey("attachments.id"), nullable=False),
sa.Column("pipeline_version", sa.Integer(), nullable=False),
sa.Column("name", sa.Text(), nullable=False),
sa.Column("object_key", sa.Text(collation="C"), nullable=False),
sa.Column("content_type", sa.Text(), nullable=False),
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
sa.Column("width", sa.Integer(), nullable=False),
sa.Column("height", sa.Integer(), nullable=False),
sa.Column("sha256", sa.Text(), nullable=False),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.PrimaryKeyConstraint("attachment_id", "pipeline_version", "name"),
sa.UniqueConstraint("object_key", name="attachment_variants_object_key_key"),
sa.CheckConstraint(
"pipeline_version >= 1", name="attachment_variants_pipeline_version_check"
),
sa.CheckConstraint(
"name IN ('thumbnail', 'crop', 'optimized')", name="attachment_variants_name_check"
),
sa.CheckConstraint(
"content_type IN ('image/jpeg', 'image/png', 'image/webp')",
name="attachment_variants_content_type_check",
),
sa.CheckConstraint("size_bytes >= 1", name="attachment_variants_size_bytes_check"),
sa.CheckConstraint("width >= 1", name="attachment_variants_width_check"),
sa.CheckConstraint("height >= 1", name="attachment_variants_height_check"),
sa.CheckConstraint("sha256 ~ '^[0-9a-f]{64}$'", name="attachment_variants_sha256_check"),
)
op.create_table(
"jobs",
sa.Column("job_id", sa.Uuid(), primary_key=True),
sa.Column("attachment_id", sa.Uuid(), sa.ForeignKey("attachments.id"), nullable=False),
sa.Column("pipeline_version", sa.Integer(), nullable=False),
sa.Column("status", sa.Text(), nullable=False),
sa.Column("attempt", sa.Integer(), nullable=False),
sa.Column("claims", sa.Integer(), nullable=False, server_default="0"),
sa.Column("lease_token", sa.Uuid(), nullable=True),
sa.Column("lease_until", sa.DateTime(timezone=True), nullable=True),
sa.Column("next_attempt_at", sa.DateTime(timezone=True), nullable=True),
sa.Column("last_error", sa.Text(), nullable=True),
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
sa.UniqueConstraint("attachment_id", "pipeline_version", name="jobs_effect_key"),
sa.CheckConstraint("pipeline_version >= 1", name="jobs_pipeline_version_check"),
sa.CheckConstraint(
"status IN ('queued', 'processing', 'done', 'failed', 'superseded')",
name="jobs_status_check",
),
sa.CheckConstraint("attempt >= 1", name="jobs_attempt_check"),
sa.CheckConstraint("claims >= 0", name="jobs_claims_check"),
)
op.create_index(
"jobs_lease_idx",
"jobs",
["lease_until"],
postgresql_where=sa.column("status") == "processing",
)
op.create_table(
"outbox",
sa.Column("id", sa.BigInteger(), sa.Identity(always=True), primary_key=True),
sa.Column("job_id", sa.Uuid(), sa.ForeignKey("jobs.job_id"), nullable=False),
sa.Column("payload", postgresql.JSONB(), nullable=False),
sa.Column("available_at", sa.DateTime(timezone=True), nullable=False),
sa.Column("published_at", sa.DateTime(timezone=True), nullable=True),
)
op.create_index(
"outbox_pending_idx",
"outbox",
["available_at", "id"],
postgresql_where=sa.column("published_at").is_(None),
)
op.create_index("outbox_job_idx", "outbox", ["job_id"])
op.create_table(
"job_events",
sa.Column("id", sa.BigInteger(), sa.Identity(always=True), primary_key=True),
sa.Column("job_id", sa.Uuid(), nullable=False),
sa.Column("attachment_id", sa.Uuid(), nullable=False),
sa.Column("kind", sa.Text(), nullable=False),
sa.Column("attempt", sa.Integer(), nullable=True),
sa.Column("detail", postgresql.JSONB(), nullable=False, server_default="{}"),
sa.Column("at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()),
)
op.create_index("job_events_attachment_idx", "job_events", ["attachment_id", "id"])
def downgrade() -> None:
raise RuntimeError("Downgrade would delete pipeline data")
+20
View File
@@ -0,0 +1,20 @@
import sqlalchemy as sa
from alembic import op
revision = "0005_failed_registry"
down_revision = "0004_jobs"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_index(
"jobs_failed_idx",
"jobs",
[sa.column("updated_at").desc()],
postgresql_where=sa.column("status") == "failed",
)
def downgrade() -> None:
op.drop_index("jobs_failed_idx", table_name="jobs")
@@ -0,0 +1,28 @@
import sqlalchemy as sa
from alembic import op
revision = "0006_attachment_quarantine"
down_revision = "0005_failed_registry"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("attachments", sa.Column("quarantine_reason", sa.Text(), nullable=True))
op.create_check_constraint(
"attachments_quarantine_reason_check",
"attachments",
"quarantine_reason IN ('missing', 'size_mismatch', 'sha_mismatch')",
)
op.add_column(
"attachments", sa.Column("quarantined_at", sa.DateTime(timezone=True), nullable=True)
)
op.create_check_constraint(
"attachments_quarantine_check",
"attachments",
"(quarantine_reason IS NULL) = (quarantined_at IS NULL)",
)
def downgrade() -> None:
raise RuntimeError("Downgrade would delete quarantine data")
+19
View File
@@ -0,0 +1,19 @@
import sqlalchemy as sa
from alembic import op
from sqlalchemy.dialects.postgresql import JSONB
revision = "0007_trace_context"
down_revision = "0006_attachment_quarantine"
branch_labels = None
depends_on = None
def upgrade() -> None:
for table in ("jobs", "outbox"):
op.add_column(
table, sa.Column("trace_context", JSONB(), nullable=False, server_default="{}")
)
def downgrade() -> None:
raise RuntimeError("Downgrade would delete trace correlation data")
+8 -1
View File
@@ -1,10 +1,17 @@
# Topology worksheet, deliberately NOT a runnable stack (no application supplied).
# Implement compose.yaml or stack.yaml; do not submit this empty file as deployment.
# Service names below are logical roles, not imposed container names.
x-lab-number: 2
x-lab-number: 6
x-required-roles:
- api
- postgres
- load_balancer
- s3
- broker
- worker
- otel_collector
- telemetry_storage
- grafana
# TODO: implement real images/builds, network wiring, volumes, healthchecks,
# credentials and bootstrap. Details: contracts/infrastructure.json and README.md.
services: {}
+638
View File
@@ -0,0 +1,638 @@
x-app: &app
image: ${API_IMAGE:-mtusi-messenger:${COMPOSE_PROJECT_NAME}}
networks: [backend]
x-s3-env: &s3-env
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_REGION: ${S3_REGION:-us-east-1}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
S3_CONNECT_TIMEOUT_SECONDS: ${S3_CONNECT_TIMEOUT_SECONDS:-1}
S3_READ_TIMEOUT_SECONDS: ${S3_READ_TIMEOUT_SECONDS:-2}
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
IMAGE_CHECK_CONCURRENCY: ${IMAGE_CHECK_CONCURRENCY:-2}
UPLOAD_CONCURRENCY: ${UPLOAD_CONCURRENCY:-3}
UPLOAD_GRACE_SECONDS: ${UPLOAD_GRACE_SECONDS:-900}
UPLOAD_SWEEP_INTERVAL_SECONDS: ${UPLOAD_SWEEP_INTERVAL_SECONDS:-30}
UPLOAD_SWEEP_BATCH: ${UPLOAD_SWEEP_BATCH:-100}
x-pipeline: &pipeline
image: ${API_IMAGE:-mtusi-messenger:${COMPOSE_PROJECT_NAME}}
pull_policy: never
restart: unless-stopped
init: true
read_only: true
tmpfs: [/tmp]
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
stop_grace_period: 10s
networks: [backend]
healthcheck:
test: ["CMD", "python", "-c", "import os,sys,time;sys.exit(time.time()-os.stat('/tmp/heartbeat').st_mtime>30)"]
interval: 10s
timeout: 5s
retries: 3
start_period: 30s
start_interval: 2s
depends_on:
migrate:
condition: service_completed_successfully
s3-init:
condition: service_completed_successfully
broker:
condition: service_healthy
x-otel-env: &otel-env
OTEL_TRACE_RATIO: ${OTEL_TRACE_RATIO:-1}
OTEL_SDK_DISABLED: ${OTEL_SDK_DISABLED:-false}
OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:-http://otel-collector:4318}
OTEL_EXPORTER_OTLP_PROTOCOL: ${OTEL_EXPORTER_OTLP_PROTOCOL:-http/protobuf}
OTEL_EXPORTER_OTLP_TIMEOUT: ${OTEL_EXPORTER_OTLP_TIMEOUT:-2}
OTEL_RESOURCE_ATTRIBUTES: ${OTEL_RESOURCE_ATTRIBUTES:-deployment.environment.name=lab}
OTEL_SEMCONV_STABILITY_OPT_IN: http
OTEL_METRIC_EXPORT_INTERVAL: ${OTEL_METRIC_EXPORT_INTERVAL:-10000}
OTEL_BSP_MAX_QUEUE_SIZE: ${OTEL_BSP_MAX_QUEUE_SIZE:-2048}
OTEL_BLRP_MAX_QUEUE_SIZE: ${OTEL_BLRP_MAX_QUEUE_SIZE:-2048}
TELEMETRY_SHUTDOWN_SECONDS: ${TELEMETRY_SHUTDOWN_SECONDS:-2}
x-obs-logging: &obs-logging
driver: json-file
options:
max-size: 10m
max-file: "3"
x-obs-hardening: &obs-hardening
restart: unless-stopped
stop_grace_period: 10s
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
logging: *obs-logging
x-pipeline-env: &pipeline-env
<<: [*s3-env, *otel-env]
DATABASE_URL_FILE: /run/secrets/database_url
BROKER_URL_FILE: /run/secrets/broker_url
DB_POOL_MAX_SIZE: ${PIPELINE_DB_POOL_MAX_SIZE:-4}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
HEARTBEAT_FILE: /tmp/heartbeat
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
JOB_BACKOFF_BASE_SECONDS: ${JOB_BACKOFF_BASE_SECONDS:-2}
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
x-s3-client: &s3-client
image: rustfs/rc:v0.1.36@sha256:ab024bfebee49a750ce886b4c70963ccd9ddaa03f491704a90710641d7a26699
read_only: true
tmpfs: ["/tmp:size=16m"]
environment:
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
RC_CONFIG_DIR: /tmp/rc
user: "65534:65534"
volumes: ["./infra/s3:/s3:ro"]
secrets:
- s3_root_user
- s3_root_password
- s3_access_key_id
- s3_secret_access_key
- s3_reconciler_access_key_id
- s3_reconciler_secret_access_key
- s3_worker_access_key_id
- s3_worker_secret_access_key
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
restart: "no"
networks: [backend]
deploy:
resources:
limits: {cpus: "0.5", memory: 64m}
services:
lb:
image: haproxy:3.2.26-alpine@sha256:5893840576071e6217cd52ebdcc14f454c6f7a631056fa6f4e0f40d3cac4215b
environment:
API_SERVER_MAXCONN: ${API_SERVER_MAXCONN:-64}
volumes: [./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro]
ports:
- "127.0.0.1:${LB_HTTP_PORT:-18086}:8080"
- "127.0.0.1:${LB_STATS_PORT:-26085}:8404"
networks: [edge, backend]
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8404/healthz"]
interval: 2s
timeout: 2s
retries: 3
start_period: 60s
start_interval: 1s
sysctls:
net.ipv4.ip_local_port_range: "10240 65535"
net.ipv4.tcp_tw_reuse: "1"
stop_signal: SIGUSR1
stop_grace_period: 8s
restart: unless-stopped
read_only: true
cap_drop: [ALL]
deploy:
resources:
limits: {cpus: "${LB_CPUS:-1}", memory: 128m}
api:
<<: *app
build: .
environment:
<<: [*s3-env, *otel-env]
OTEL_SERVICE_NAME: ${OTEL_SERVICE_NAME:-messenger-api}
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
COST_EXPERIMENT: ${COST_EXPERIMENT:-false}
WEB_CONCURRENCY: ${WEB_CONCURRENCY:-2}
DB_POOL_MAX_SIZE: ${DB_POOL_MAX_SIZE:-8}
REQUEST_TIMEOUT_SECONDS: ${REQUEST_TIMEOUT_SECONDS:-8}
MIN_TRANSFER_RATE_BYTES: ${MIN_TRANSFER_RATE_BYTES:-131072}
DB_ACQUIRE_TIMEOUT_SECONDS: ${DB_ACQUIRE_TIMEOUT_SECONDS:-1}
DB_COMMAND_TIMEOUT_SECONDS: ${DB_COMMAND_TIMEOUT_SECONDS:-2}
DRAIN_SECONDS: ${DRAIN_SECONDS:-2}
SHUTDOWN_TIMEOUT_SECONDS: ${SHUTDOWN_TIMEOUT_SECONDS:-6}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
MALLOC_MMAP_THRESHOLD_: ${MALLOC_MMAP_THRESHOLD_:-131072}
secrets: [database_url, cache_url, cursor_secret, s3_access_key_id, s3_secret_access_key]
healthcheck:
test: ["CMD", "python", "-c", "import sys,urllib.request;sys.exit(urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=2).status!=200)"]
interval: 5s
timeout: 3s
retries: 3
start_period: 30s
start_interval: 200ms
stop_grace_period: 8s
restart: unless-stopped
read_only: true
tmpfs: [/tmp]
sysctls:
net.ipv4.tcp_rmem: "4096 65536 262144"
net.ipv4.tcp_wmem: "4096 65536 262144"
depends_on:
migrate:
condition: service_completed_successfully
valkey:
condition: service_started
s3-init:
condition: service_completed_successfully
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits: {cpus: "${API_CPUS:-1}", memory: "${API_MEMORY:-512m}"}
migrate:
<<: *app
pull_policy: never
init: true
command: ["alembic", "upgrade", "head"]
environment:
DATABASE_URL_FILE: /run/secrets/database_url
LOG_LEVEL: ${LOG_LEVEL:-INFO}
secrets: [database_url]
healthcheck:
disable: true
restart: "no"
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits: {cpus: "0.5", memory: 256m}
postgres:
image: postgres:17.11-alpine@sha256:266d3c257594e195b309ae9d1805e54c788f1d7b375f56bf6c367aac6c917e7b
user: "70:70"
command:
- postgres
- -c
- max_connections=${PG_MAX_CONNECTIONS:-200}
- -c
- shared_buffers=${PG_SHARED_BUFFERS:-256MB}
- -c
- shared_preload_libraries=pg_stat_statements
- -c
- pg_stat_statements.track=top
- -c
- track_io_timing=on
- -c
- idle_in_transaction_session_timeout=10s
environment:
POSTGRES_DB: messenger
POSTGRES_USER: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
volumes: [pgdata:/var/lib/postgresql/data]
shm_size: 256m
cpuset: ${PG_CPUSET:-}
healthcheck:
test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "messenger", "-d", "messenger"]
interval: 2s
timeout: 3s
retries: 30
start_period: 10s
start_interval: 200ms
stop_grace_period: 10s
restart: unless-stopped
networks: [backend]
deploy:
resources:
limits: {cpus: "${PG_CPUS:-2}", memory: "${PG_MEMORY:-1g}"}
valkey:
image: valkey/valkey:8.1.11-alpine@sha256:2602ace2be6569bc2bbb06f873776516ab9089821525be34f4305c67eb9db02a
user: "999:1000"
command:
- valkey-server
- /run/secrets/valkey.conf
- --appendonly
- "yes"
- --appendfsync
- always
- --maxmemory
- 64mb
- --maxmemory-policy
- volatile-lru
secrets: [valkey.conf]
volumes: [valkeydata:/data]
healthcheck:
test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$(sed -n 's/^requirepass //p' /run/secrets/valkey.conf)\" valkey-cli ping | grep -q PONG"]
interval: 2s
timeout: 3s
retries: 30
stop_grace_period: 10s
restart: unless-stopped
networks: [backend]
deploy:
resources:
limits: {cpus: "0.5", memory: 128m}
s3:
image: rustfs/rustfs:1.0.4@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2
environment:
RUSTFS_ACCESS_KEY_FILE: /run/secrets/s3_root_user
RUSTFS_SECRET_KEY_FILE: /run/secrets/s3_root_password
RUSTFS_ADDRESS: ":9000"
RUSTFS_CONSOLE_ENABLE: "false"
RUSTFS_REGION: ${S3_REGION:-us-east-1}
RUSTFS_OBS_LOG_DIRECTORY: ""
RUSTFS_OBS_LOGGER_LEVEL: ${S3_LOG_LEVEL:-warn}
secrets: [s3_root_user, s3_root_password]
volumes: [s3data:/data]
cpuset: ${S3_CPUSET:-}
read_only: true
tmpfs: ["/tmp:size=64m"]
cap_drop: [ALL]
security_opt: ["no-new-privileges:true"]
healthcheck:
test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:9000/health/ready"]
interval: 2s
timeout: 3s
retries: 30
start_period: 20s
stop_grace_period: 10s
restart: unless-stopped
networks: [backend]
deploy:
resources:
limits: {cpus: "${S3_CPUS:-1}", memory: "${S3_MEMORY:-512m}"}
reconciler:
<<: *app
pull_policy: never
profiles: [reconcile]
init: true
entrypoint: ["python", "-m", "messenger.attachments.entrypoints.reconcile"]
command: ["--dry-run"]
environment:
<<: *s3-env
DATABASE_URL_FILE: /run/secrets/database_url
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_reconciler_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_reconciler_secret_access_key
S3_READ_TIMEOUT_SECONDS: ${RECONCILE_S3_READ_TIMEOUT_SECONDS:-10}
LOG_LEVEL: ${LOG_LEVEL:-INFO}
secrets: [database_url, s3_reconciler_access_key_id, s3_reconciler_secret_access_key]
healthcheck:
disable: true
restart: "no"
read_only: true
cpuset: ${RECONCILE_CPUSET:-}
depends_on:
postgres:
condition: service_healthy
s3:
condition: service_healthy
deploy:
resources:
limits: {cpus: "${RECONCILE_CPUS:-1}", memory: "${RECONCILE_MEMORY:-256m}"}
s3-init:
<<: *s3-client
entrypoint: ["/bin/sh", "/s3/init.sh"]
depends_on:
s3:
condition: service_healthy
s3-cli:
<<: *s3-client
profiles: [tools]
entrypoint: ["/bin/sh", "/s3/cli.sh"]
broker:
image: rabbitmq:4.1.8-management-alpine@sha256:54ef5ff505352c1beef05a97ee9bd6ab95b3559a76629ba0119b1d63d8d6e4b2
hostname: broker
user: "100:101"
restart: unless-stopped
read_only: true
tmpfs: [/tmp]
security_opt: ["no-new-privileges:true"]
cap_drop: [ALL]
ports:
- "127.0.0.1:${BROKER_MANAGEMENT_PORT:-26086}:15672"
networks: [backend, edge]
volumes:
- broker-data:/var/lib/rabbitmq
- ./infra/rabbitmq/rabbitmq.conf:/etc/rabbitmq/conf.d/20-messenger.conf:ro
- ./infra/rabbitmq/enabled_plugins:/etc/rabbitmq/enabled_plugins:ro
- ./infra/rabbitmq/definitions.json:/etc/rabbitmq/definitions.d/10-topology.json:ro
secrets:
- source: rabbitmq_users
target: /etc/rabbitmq/definitions.d/20-users.json
- rabbitmq_ops_password
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "-q", "check_port_connectivity"]
interval: 10s
timeout: 10s
retries: 5
start_period: 60s
start_interval: 2s
stop_grace_period: 10s
deploy:
resources:
limits: {cpus: "1", memory: 512m}
relay:
<<: *pipeline
command: ["python", "-m", "messenger.relay"]
environment:
<<: *pipeline-env
OTEL_SERVICE_NAME: messenger-relay
OUTBOX_POLL_SECONDS: ${OUTBOX_POLL_SECONDS:-1}
SWEEP_INTERVAL_SECONDS: ${SWEEP_INTERVAL_SECONDS:-5}
secrets:
- database_url
- source: broker_url_api
target: broker_url
- s3_access_key_id
- s3_secret_access_key
deploy:
resources:
limits: {cpus: "0.5", memory: 192m}
worker:
<<: *pipeline
command: ["python", "-m", "messenger.worker"]
environment:
<<: *pipeline-env
OTEL_SERVICE_NAME: messenger-worker
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
RENDER_TIMEOUT_SECONDS: ${RENDER_TIMEOUT_SECONDS:-30}
RENDER_MEMORY_MB: ${RENDER_MEMORY_MB:-448}
RENDER_TASKS_PER_CHILD: ${RENDER_TASKS_PER_CHILD:-100}
secrets:
- database_url
- source: broker_url_worker
target: broker_url
- source: s3_worker_access_key_id
target: s3_access_key_id
- source: s3_worker_secret_access_key
target: s3_secret_access_key
deploy:
resources:
limits: {cpus: "${WORKER_CPUS:-2}", memory: "${WORKER_MEMORY:-1g}"}
k6:
image: grafana/k6:2.3.0@sha256:2dd4756ec6e14e62778916e1eb850108e4b499304c02df56760fa0bdb0fba0b8
profiles: [bench]
environment:
BASE_URL: http://lb:8080
volumes: [./infra/k6:/scripts:ro]
networks: [edge]
deploy:
resources:
limits: {cpus: "${K6_CPUS:-2}", memory: "${K6_MEMORY:-512m}"}
healthtools:
image: busybox:1.37.0-musl@sha256:5cec3fc171c87218698e85a52af7087de727372aae264a787b8112901a5b0092
command: ["cp", "/bin/busybox", "/hc/busybox"]
restart: "no"
network_mode: none
volumes:
- healthtools:/hc
logging: *obs-logging
otel-collector:
<<: *obs-hardening
image: otel/opentelemetry-collector-contrib:0.163.2@sha256:7f22d2a6748110ffb6a59bc0df448bccbf501cdb667d610ede4c61d3863d5849
command: ["--config=/etc/otelcol/${OTEL_COLLECTOR_CONFIG:-collector.yaml}"]
networks: [backend, edge]
environment:
GOMEMLIMIT: 200MiB
volumes:
- ./infra/otel:/etc/otelcol:ro
- healthtools:/hc:ro
depends_on:
healthtools:
condition: service_completed_successfully
healthcheck:
test: ["CMD", "/hc/busybox", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:13133/"]
interval: 10s
timeout: 3s
retries: 6
start_period: 10s
start_interval: 2s
deploy:
resources:
limits: {cpus: "0.50", memory: 256M}
prometheus:
<<: *obs-hardening
image: prom/prometheus:v3.15.1@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=${PROMETHEUS_RETENTION:-7d}
- --storage.tsdb.retention.size=1GB
- --web.enable-otlp-receiver
- --enable-feature=exemplar-storage
networks: [backend, edge]
volumes:
- ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./infra/prometheus/rules:/etc/prometheus/rules:ro
- prometheus-data:/prometheus
ports:
- "127.0.0.1:${PROMETHEUS_PORT:-26090}:9090"
healthcheck:
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:9090/-/ready"]
interval: 10s
timeout: 3s
retries: 6
start_period: 15s
start_interval: 2s
deploy:
resources:
limits: {cpus: "0.50", memory: 512M}
loki:
<<: *obs-hardening
image: grafana/loki:3.7.9@sha256:0db216b630ae1f348a1584c67f3cead8b4c1fc839169a39501db664744158e28
command: ["-config.file=/etc/loki/loki.yaml"]
networks: [edge]
environment:
GOMEMLIMIT: 300MiB
volumes:
- ./infra/loki/loki.yaml:/etc/loki/loki.yaml:ro
- loki-data:/loki
- healthtools:/hc:ro
depends_on:
healthtools:
condition: service_completed_successfully
ports:
- "127.0.0.1:${LOKI_PORT:-26310}:3100"
healthcheck:
test: ["CMD", "/hc/busybox", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:3100/ready"]
interval: 10s
timeout: 3s
retries: 10
start_period: 30s
start_interval: 2s
deploy:
resources:
limits: {cpus: "0.50", memory: 384M}
tempo:
<<: *obs-hardening
image: grafana/tempo:2.10.10@sha256:12b0820f13ffa10ce956b6e9acc3b03f3cffded99109be6cc09d106fe56e5d38
command: ["-config.file=/etc/tempo/tempo.yaml"]
networks: [edge]
environment:
GOMEMLIMIT: 300MiB
volumes:
- ./infra/tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
- tempo-data:/var/tempo
- healthtools:/hc:ro
depends_on:
healthtools:
condition: service_completed_successfully
ports:
- "127.0.0.1:${TEMPO_PORT:-26320}:3200"
healthcheck:
test: ["CMD", "/hc/busybox", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:3200/ready"]
interval: 10s
timeout: 3s
retries: 10
start_period: 30s
start_interval: 2s
deploy:
resources:
limits: {cpus: "0.50", memory: 384M}
grafana:
<<: *obs-hardening
image: grafana/grafana:12.4.14@sha256:ae509d9ae8c1021ca32ec370a9ca975d40165eda9fccd1e1102d317ff8787cd7
networks: [edge]
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
GF_USERS_ALLOW_SIGN_UP: "false"
GF_AUTH_ANONYMOUS_ENABLED: "false"
GF_ANALYTICS_REPORTING_ENABLED: "false"
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES: "false"
GF_NEWS_NEWS_FEED_ENABLED: "false"
GF_PLUGINS_PREINSTALL_DISABLED: "true"
GF_LOG_LEVEL: warn
secrets: [grafana_admin_password]
volumes:
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
- ./infra/grafana/dashboards:/etc/grafana/dashboards:ro
- grafana-data:/var/lib/grafana
depends_on:
prometheus:
condition: service_started
loki:
condition: service_started
tempo:
condition: service_started
ports:
- "127.0.0.1:${GRAFANA_PORT:-26300}:3000"
healthcheck:
test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:3000/api/health"]
interval: 10s
timeout: 3s
retries: 10
start_period: 30s
start_interval: 2s
deploy:
resources:
limits: {cpus: "0.50", memory: 384M}
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
valkey.conf:
file: ./secrets/valkey.conf
cache_url:
file: ./secrets/cache_url
cursor_secret:
file: ./secrets/cursor_secret
s3_root_user:
file: ./secrets/s3_root_user
s3_root_password:
file: ./secrets/s3_root_password
s3_access_key_id:
file: ./secrets/s3_access_key_id
s3_secret_access_key:
file: ./secrets/s3_secret_access_key
s3_worker_access_key_id:
file: ./secrets/s3_worker_access_key_id
s3_worker_secret_access_key:
file: ./secrets/s3_worker_secret_access_key
rabbitmq_users:
file: ./secrets/rabbitmq_users.json
rabbitmq_ops_password:
file: ./secrets/rabbitmq_ops_password
broker_url_api:
file: ./secrets/broker_url_api
broker_url_worker:
file: ./secrets/broker_url_worker
grafana_admin_password:
file: ./secrets/grafana_admin_password
s3_reconciler_access_key_id:
file: ./secrets/s3_reconciler_access_key_id
s3_reconciler_secret_access_key:
file: ./secrets/s3_reconciler_secret_access_key
volumes:
pgdata:
valkeydata:
s3data:
broker-data:
healthtools:
prometheus-data:
loki-data:
tempo-data:
grafana-data:
networks:
edge:
backend:
internal: true
+21
View File
@@ -0,0 +1,21 @@
# Состояния вложения и договор между producer/worker
Сообщение broker соответствует [jobs.schema.json](jobs.schema.json). Можно применять native headers брокера для trace context, но семантика carrier сохраняется; для проверки покажите отображение полей. `job_id` идентифицирует одну логическую работу и не меняется при redelivery; `attempt` начинается с 1 и отражает управляемую попытку обработки (broker redelivery сам по себе не создаёт новый job_id). Ключ эффекта — `(attachment_id, pipeline_version, variant)`.
| Переход | Условие | Что устойчиво сохранено |
| --- | --- | --- |
| нет → queued | Принимаем upload; только после durable handoff отвечаем 202 | Полный проверенный оригинал в staging, метаданные, намерение обработки |
| queued → processing | Worker захватил/арендовал работу | Попытка, lease/deadline или эквивалентная защита |
| processing → queued | Временная ошибка и остались попытки | Причина без секретов, время следующей попытки |
| processing → ready | Все обязательные объекты записаны и результат атомарно опубликован | Оригинал, thumbnail (на 4 ещё crop/optimized), метаданные/checksums |
| processing → failed | Ошибка постоянная или попытки исчерпаны | error_code и запись, доступная оператору |
| processing → queued | Истёк lease после смерти worker | Восстановленная работа, без дублирования эффектов |
| failed → queued | Явный операторский replay, доступен на 4 | Audit, новая попытка, прежний логический эффект защищён от дубля |
Повторная доставка уже ready-задачи — no-op с ack после проверки завершённого эффекта. Конкурентные попытки не перезаписывают более новый pipeline_version. Основной курс использует pipeline_version=1; версионирование нескольких pipeline — трек 5. Заявляйте готовность только когда все обещанные уровнем варианты доступны. Нельзя откатывать ready в processing из-за старого дубля.
**Окна отказа для защиты:** после staging до записи задания; после COMMIT до publish; после publish до confirm; после записи S3 до COMMIT результата; после COMMIT результата до ack. На 3 покажите сохранность успешно принятой работы и повторную доставку; на 4 автоматическое восстановление разрыва DB/broker (outbox/эквивалент) и зависшего processing.
Staging не удаляется до безопасной публикации результатов. Неуспешные/непривязанные исходники имеют документированный retention и cleanup; он не удаляет текущую работу. Broker принимает только ID/служебные данные, не произвольный URL для скачивания и не путь из клиентского filename. Worker берёт доверенные storage metadata из БД.
Ack подтверждает устойчивую обработку, publisher confirm — приём брокером; это разные границы. См. [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability). At-least-once + идемпотентный эффект не означает exactly-once доставку.
+4 -2
View File
@@ -34,7 +34,8 @@
"chat_id": "33333333-3333-4333-8333-333333333333",
"sender_id": "11111111-1111-4111-8111-111111111111",
"text": "Привет, Боб!",
"created_at": "2026-09-01T12:00:00Z"
"created_at": "2026-09-01T12:00:00Z",
"attachment_ids": []
},
"MessagePage": {
"items": [
@@ -43,7 +44,8 @@
"chat_id": "33333333-3333-4333-8333-333333333333",
"sender_id": "11111111-1111-4111-8111-111111111111",
"text": "Привет, Боб!",
"created_at": "2026-09-01T12:00:00Z"
"created_at": "2026-09-01T12:00:00Z",
"attachment_ids": []
}
],
"next_cursor": null
+59 -2
View File
@@ -1,10 +1,10 @@
{
"lab": 2,
"lab": 6,
"kind": "requirements, not runnable deployment",
"required_for_grade_3": [
{
"role": "api",
"count_min": 1,
"count_min": 2,
"student_implements": true,
"persistent_local_state": false
},
@@ -12,6 +12,63 @@
"role": "postgres",
"purpose": "users/chats/membership/messages/metadata",
"persistent_volume": true
},
{
"role": "load_balancer",
"public_entrypoint": true,
"backend_readiness": true
},
{
"role": "s3",
"choose_one": [
"MinIO",
"SeaweedFS",
"RustFS"
],
"private_bucket": true,
"persistent_volume": true
},
{
"role": "broker",
"choose_one": [
"RabbitMQ",
"Redis Streams",
"Valkey Streams",
"NATS JetStream",
"documented durable equivalent"
],
"persistent_volume": true,
"delivery": "at-least-once"
},
{
"role": "worker",
"separate_process": true
},
{
"role": "otel_collector",
"signals": [
"metrics",
"logs",
"traces"
]
},
{
"role": "telemetry_storage",
"signals": [
"metrics",
"logs",
"traces"
],
"possible_combination": [
"Prometheus",
"Loki",
"Tempo"
],
"persistent_volume": true
},
{
"role": "grafana",
"provisioning_in_git": true
}
],
"optional_components": [
+63
View File
@@ -0,0 +1,63 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "urn:mtusi:messenger:attachment-job:v1",
"title": "Durable attachment processing job v1",
"type": "object",
"additionalProperties": false,
"required": [
"schema_version",
"job_id",
"type",
"attachment_id",
"pipeline_version",
"attempt",
"created_at"
],
"properties": {
"schema_version": {
"const": 1
},
"job_id": {
"type": "string",
"format": "uuid"
},
"type": {
"const": "attachment.process.v1"
},
"attachment_id": {
"type": "string",
"format": "uuid"
},
"pipeline_version": {
"type": "integer",
"minimum": 1
},
"attempt": {
"type": "integer",
"minimum": 1
},
"created_at": {
"type": "string",
"format": "date-time"
},
"traceparent": {
"type": "string",
"pattern": "^00-[0-9a-f]{32}-[0-9a-f]{16}-[0-9a-f]{2}$"
},
"tracestate": {
"type": "string",
"maxLength": 512
}
},
"examples": [
{
"schema_version": 1,
"job_id": "11111111-1111-4111-8111-111111111111",
"type": "attachment.process.v1",
"attachment_id": "22222222-2222-4222-8222-222222222222",
"pipeline_version": 1,
"attempt": 1,
"created_at": "2026-09-01T12:00:00Z"
}
]
}
+28
View File
@@ -0,0 +1,28 @@
# Контракт наблюдаемости
Ниже логические сигналы. Допустимы стандартные semantic conventions выбранного SDK и их native metric names: зафиксируйте соответствие в `REPORT.md`, dashboard обязан ссылаться на реально существующие имена/единицы. Переименование SDK не повод дублировать одну метрику двумя инструментами. Версии SDK, semantic conventions, Collector distribution и backend закрепляются в решении.
| Сигнал | Минимальные поля/измерения | Уровень |
| --- | --- | --- |
| HTTP requests | method, route template, status class; counter | 3 |
| HTTP duration | histogram в секундах или документированное преобразование из ms | 3 |
| Jobs processed / failed | job type, outcome, counter | 3 |
| Job processing duration | job type, outcome, histogram | 3 |
| API и worker logs | timestamp, severity, service.name, event, trace_id, span_id при активном span | 3 |
| API и worker traces | реальные server/consumer spans, duration, outcome; не искусственный статический trace | 3 |
| Queue depth / oldest job age | queue name, количество / секунды | 4 |
| Pool/saturation | DB pool used/waiting, worker busy/capacity, process/container CPU/RAM | 4 |
| Сквозная связь upload → publish → consume → S3/DB | W3C traceparent/tracestate; parent-child или link по выбранной семантике | 4 |
| End-to-end time to ready | от durable acceptance до публикации ready, histogram | 4 |
Resource attributes: `service.name` (messenger-api / messenger-worker), `service.version`, `service.instance.id`, `deployment.environment.name=lab`. Список имён может отличаться, если явно сопоставлен. `X-Request-Id` коррелирует HTTP-ошибку с логом; trace_id — не замена всех request IDs.
**Не labels метрик:** user_id, chat_id, message_id, attachment_id, request_id, raw URL, filename, текст. Для маршрута используйте `/api/v1/chats/{chat_id}/messages`, не реальный UUID. В логах/трассах допустимы ограниченные технические ID для учебных синтетических данных, если обоснована необходимость; пароли, Authorization, session token, body сообщения, картинка, private key и signed URL запрещены.
Для HTTP error ratio заранее определите numerator/denominator: например 5xx/все бизнес-запросы, исключая health; не смешивайте неверный пароль клиента с отказом БД. p95 агрегируется из histogram buckets всех нужных реплик, а не средних p95. Границы buckets должны соответствовать ожидаемым задержкам. Backlog — не queue processing latency.
В Git находятся Collector config, backend configs, datasources/dashboard provisioning, dashboard JSON и на 4 alert rules. У alerts есть условие, окно, порог, `for` при необходимости, описание и локальный runbook. Отправка сообщений в почту/мессенджер не требуется.
Минимальные контролируемые инциденты для 4: (1) остановить worker и увидеть рост возраста задач; (2) сделать DB медленной/недоступной и отличить её от медленного HTTP handler по trace/метрикам. После восстановления backlog уменьшается, alerts resolved. Третий сценарий — недоступность backend telemetry: SDK/Collector ограничивают RAM/очередь, API сохраняет работу, потери telemetry считаются и объясняются.
Collector — получатель/обработчик/экспортёр, не долговременная БД. Grafana — интерфейс и запросы к источникам. Логи можно отправлять OTLP в совместимый backend; не закладывайте удалённый/устаревший exporter без проверки текущей сборки. См. [Collector configuration](https://opentelemetry.io/docs/collector/configuration/), [Trace Context](https://www.w3.org/TR/trace-context/) и [histograms](https://prometheus.io/docs/practices/histograms/).
+1493 -7
View File
File diff suppressed because it is too large Load Diff
+62 -3
View File
@@ -1,5 +1,64 @@
# Доказательства
# Индекс доказательств ЛР6
Храните небольшие обезличенные результаты: команды и stdout проверок, JSON/CSV агрегатов нагрузки, EXPLAIN, dashboard JSON, текст разбора отказа. Укажите дату, commit, аппаратные лимиты и профиль нагрузки.
Автор: Костылев Артём, БПИ2303. Итог: [REPORT.md](../REPORT.md), заявлены уровни 3/4 и трек 1 «Цена наблюдаемости». **Основной актуальный набор — `forward-published/`**. Имя каталога описывает перенос опубликованного источника ЛР5 и не означает публикацию ЛР6.
Не добавляйте .env, session tokens, cookies, приватные ключи, credentials, реальные сообщения, raw dumps и signed URLs. Для скриншотов скрывайте секреты. Большие логи/бинарные traces сохраняйте отдельно по правилам преподавателя; одного скриншота без методики недостаточно.
## Происхождение и актуальность
| Набор | SHA и назначение | Как использовать |
| --- | --- | --- |
| [forward-port.md](forward-port.md), [source-inventory.json](forward-published/source-inventory.json) | ЛР5 `f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071` → `c7b8feb805d151e878d77b0dff47320a9436b172`; все 281 путь diff учтены | Карта позднего переноса, исключения и причина повторения измерений |
| [forward-published](forward-published) | Runtime/измеритель `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`; evidence-коммит/ревью `6ca1c7549978af3127966540385195ab0587f2e1` | Свежие доказательства после исправлений; runtime между этими SHA не менялся |
| [grade3/notes.md](grade3/notes.md), [level3.json](level3.json) | **Исторический код** `95511002b12510178a3cc2a22566b43e2b2b6c81`; чистый clone `2903d9d` | Не свежие smoke/нагрузка. История telemetry и независимость от Collector переиспользуются только для незатронутых механизмов |
| [grade4/notes.md](grade4/notes.md) | **Исторический код** `44d81fd80ee13faa42bca069245ee0943f88b2df`, retry harness `447a6507f973596e612f40c1484dda922bf0e50d`; storage incident `cd9d91550aa4eeba31489e07b71ee2767acfc7d8` | Отказ storage и кардинальность применимы в пределах [preserved-mechanisms.json](forward-published/preserved-mechanisms.json); chain/retry/alerts заменены свежими |
| [track5/notes.md](track5/notes.md) | **Исторический код** `202b985deacda79b6d8df65dcefe8a1af43f1a8e`, измеритель `bc0b1620825b53fd3a3019623326ebf68cbbbeec` | Старые 519 passed, −91,1% Tempo и 15 серий не смешивать с актуальными 995 passed / −92,2%; auth/cache путь изменился и был перемерен |
| [review-summary.md](review-summary.md) | Результат независимого review из runner на `6ca1c75…` | 0 major, 5 minor; полный список ограничений, не новое ревью |
| [report-audit.md](report-audit.md) | Документационный этап поверх `6ca1c75…` | Только статические проверки и сверка сохранённых чисел; Docker/нагрузка не запускались |
| [load-validation.md](load-validation.md) | Финальная validation поверх `1e00d861a803a6d49ef443175b8231cbf048fc02` | Аудит применимости всех 15 серий, свежие проверки в [validation](validation); новых нагрузочных запусков нет, публикация не выполнялась |
[closure.json](forward-published/closure.json) фиксирует окончание реализации: runtime не менялся после замеров, 22 проверяемых тогда файла и прежний evidence сохранены, своих работающих контейнеров нет, 9 volumes оставлены. Число 22 включает документы, которые отчётный этап имеет право обновить; это исторический снимок, не запрет на REPORT/AGENTS/этот индекс. Нормативные contracts/tests/README/COURSE и infra README неизменны. В отчётном этапе raw evidence не переписывается; в старые notes добавлена только отметка исторического кода.
## Карта актуального набора
| Что проверено | Команда / средство | Доказательства |
| --- | --- | --- |
| Конфигурация, образы, условия и limits | `scripts/lab6.sh make up`; снимки runtime/host | [up.txt](forward-published/up.txt), [runtime.json](forward-published/runtime.json), [conditions-before.json](forward-published/conditions-before.json) |
| Структура, схемы, ruff/ty | `make check validate lint` | [quality.txt](forward-published/quality.txt), [final-quality.txt](forward-published/final-quality.txt) |
| Полный собственный suite | `make student-test` | [student-test.txt](forward-published/student-test.txt): 995 passed, 2 skipped, 19 deselected; запуск до runtime-коммита на WIP поверх `97feb04…`, зафиксированном затем в `bc1d7aa…` |
| Public API, persistent stores, failover | `make test`, `make test-persistence ACTION_SCRIPT=scripts/restart.sh`, `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make restore` | [smoke](forward-published/smoke.txt), [persistence](forward-published/persistence.txt), [failover](forward-published/failover.txt), [restore](forward-published/restore.txt), [final-smoke](forward-published/final-smoke.txt) |
| Поздние auth/cache/HTTP регрессии | `student-tests/tools/forward_network.py`, stack tests, точные команды внутри логов | [network](forward-published/network.txt), [cache-outage](forward-published/cache-outage.txt), [HAProxy errors](forward-published/haproxy-errors.txt), [negative upload](forward-published/upload-negative.txt) |
| Requeue при отказе PostgreSQL | `student-tests/tools/forward_runtime.py postgres-outage` | [JSON](forward-published/postgres-outage.json), [команда](forward-published/postgres-outage.txt), [worker](forward-published/postgres-outage-worker.txt), [broker policy](forward-published/policy.txt) |
| PNG корректность и ограничения памяти | `student-tests/tools/forward_runtime.py png-http` / `png-boundary` | [png-http.json](forward-published/png-http.json), [png-http.txt](forward-published/png-http.txt), [png-boundary.json](forward-published/png-boundary.json), [png-boundary.txt](forward-published/png-boundary.txt) |
| Pipeline, retry, COMMIT/ack, DLQ | `scripts/pipeline_suite.py --level 4 --only …` (9 кейсов, полный вызов в txt) | [pipeline.json](forward-published/pipeline.json), [pipeline.txt](forward-published/pipeline.txt) |
| Конфиги наблюдаемости и dashboard queries | `make obs-validate`; `scripts/obs_check.py dashboards --strict` | [obs-config](forward-published/obs-config.txt), [dashboards](forward-published/dashboards.txt) |
| Корреляция request/job | `scripts/obs_check.py request` / `job` | [telemetry-request](forward-published/telemetry-request.txt), [telemetry-job](forward-published/telemetry-job.txt) |
| Контроль запрещённых значений | `scripts/obs_check.py leaks` | [telemetry-leaks](forward-published/telemetry-leaks.txt); ограничение Loki metadata — M1 в review-summary |
| Upload trace и retry до результата | `scripts/obs_incidents.py chain` / `retry` | [chain](forward-published/incident-chain.txt), [retry](forward-published/incident-retry.txt): 55/86 spans |
| Worker/DB incidents, alerts | `scripts/obs_incidents.py worker` / `database` | [worker](forward-published/incident-worker.txt), [database](forward-published/incident-database.txt): firing → resolved |
| История и outage telemetry (исторический код) | `scripts/obs_recovery.py`; `scripts/obs_incidents.py storage` | [recovery](grade3/telemetry-recovery.txt), [storage](grade4/storage-incident.txt), [обоснование неизменности](forward-published/preserved-mechanisms.json) |
| OpenSpec | `openspec validate --all --strict --no-interactive`, sync/archive | [validation](forward-published/openspec-validation.txt), [archive](forward-published/openspec-archive.txt), [spec-sync](forward-published/spec-sync.json) |
| Завершение без удаления данных | Возврат defaults, `make down` | [defaults-restored](forward-published/defaults-restored.json), [shutdown](forward-published/shutdown.txt), [closure](forward-published/closure.json) |
## Трек 1: только пригодные серии
[Методика](forward-published/cost/method.md) · [результаты](forward-published/cost/results.md) · [comparison.json](forward-published/cost/comparison.json) · [PNG](forward-published/cost/comparison.png) · [SVG](forward-published/cost/comparison.svg). Все численные сводки берутся из 15 raw JSON на `bc1d7aa…`; TXT той же серии содержит точный вызов, дату/SHA и exit code.
| Режим | Raw JSON (повторы 1 / 2 / 3) | Команды/вывод |
| --- | --- | --- |
| off | [1](forward-published/cost/off-1.json) / [2](forward-published/cost/off-2.json) / [3](forward-published/cost/off-3.json) | [1](forward-published/cost/off-1.txt) / [2](forward-published/cost/off-2.txt) / [3](forward-published/cost/off-3.txt) |
| on | [1](forward-published/cost/on-1.json) / [2](forward-published/cost/on-2.json) / [3](forward-published/cost/on-3.json) | [1](forward-published/cost/on-1.txt) / [2](forward-published/cost/on-2.txt) / [3](forward-published/cost/on-3.txt) |
| head | [1](forward-published/cost/head-1.json) / [2](forward-published/cost/head-2.json) / [3](forward-published/cost/head-3.json) | [1](forward-published/cost/head-1.txt) / [2](forward-published/cost/head-2.txt) / [3](forward-published/cost/head-3.txt) |
| tail | [1](forward-published/cost/tail-1.json) / [2](forward-published/cost/tail-2.json) / [3](forward-published/cost/tail-3.json) | [1](forward-published/cost/tail-1.txt) / [2](forward-published/cost/tail-2.txt) / [3](forward-published/cost/tail-3.txt) |
| balanced | [1](forward-published/cost/balanced-1.json) / [2](forward-published/cost/balanced-2.json) / [3](forward-published/cost/balanced-3.json) | [1](forward-published/cost/balanced-1.txt) / [2](forward-published/cost/balanced-2.txt) / [3](forward-published/cost/balanced-3.txt) |
[Measurement audit](forward-published/cost/measurement-audit.json) проверяет окно 38..42 s, drain/report 30..45 s и max gap samples <5 s. Два непригодных оригинала — [on-3](forward-published/cost/invalid-time-gap/on-3.json), [head-3](forward-published/cost/invalid-time-gap/head-3.json); причина исключения, hashes и решение повторить только их — [invalid-series.json](forward-published/cost/invalid-series.json). Их 1600 запросов не входят в 12000 пригодных.
Фон: [start](forward-published/cost/conditions-start.json), [round 1](forward-published/cost/conditions-round-1.json), [round 2](forward-published/cost/conditions-round-2.json), [round 3](forward-published/cost/conditions-round-3.json), [repeat start](forward-published/cost/conditions-repeat-start.json), [repeat end](forward-published/cost/conditions-repeat-end.json). Других Docker projects в снимках нет; общий host остаётся ограничением. Пригодные измерения сохраняются для финального аудита; документационные изменения не основание прогонять их заново.
## Неуспешные попытки и границы доказательств
- В [grade3/notes.md](grade3/notes.md) размечены преждевременный smoke и job во время рестарта; успешные повторы лежат рядом. Clean-clone evidence относится только к старому `2903d9d`.
- В [grade4/notes.md](grade4/notes.md) сохранены ранние неуспешные student tests, retry без реального отказа, прерванный persistence, неправильный cardinality-вызов и DB pause без firing. Ни один не объявлен PASS итоговым отчётом.
- В [track5/notes.md](track5/notes.md) отмечены ранний smoke, ошибка Docker stats до нагрузки и пустые validation-файлы. Все результаты этого каталога исторические.
- [Старые findings позднего переноса](forward-published/findings.json) содержат четыре minor; [review-summary.md](review-summary.md) добавляет пятое, про Loki metadata. [Whitespaces](forward-published/whitespace-check.txt) — реальный неуспех косметической проверки, не неуспех `make lint`.
Не добавлять `.env`, tokens/cookies, credentials, реальные сообщения, raw dumps и signed URLs. Технические IDs в проверках синтетические; они служат корреляции, не являются секретами. Наличие файла/скриншота само по себе не доказательство: нужны команда, SHA, наблюдение и границы применимости.
+60
View File
@@ -0,0 +1,60 @@
# Поздний перенос ЛР5 → ЛР6
Источник только immutable snapshot `lab5-c7b8feb805d151e878d77b0dff47320a9436b172`; диапазон `f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071..c7b8feb805d151e878d77b0dff47320a9436b172`. Исходный HEAD ЛР6 `237a7c93e507b984add5b49c51c6720a6bfca505`, дерево было чистым. Полный перечень всех изменённых путей, SHA-256 источника и решение по каждому: [source-inventory.json](forward-published/source-inventory.json).
## Перенос
| Группа | Изменение | Разрешение |
|---|---|---|
| identity handlers/postgres/valkey | Отзыв сессии, блокировка чтения, nonce заполнения кэша | Полный diff; SQLAlchemy и Lua, схема без изменений |
| identity authentication/values, messaging values | NUL и не-ASCII Basic | Полный diff; ошибки по существующему контракту |
| attachments images.py и png.py | Проверка IDAT, исходная tRNS глубина, tiles памяти | Полный diff; OTel S3/worker не изменён |
| infra/haproxy/haproxy.cfg | JSON escaping request_id в 502/503/504 | Полный diff |
| infra/rabbitmq/definitions.json | delivery-limit=-1 | Сохранено описание lab6 и существующая топология/DLX |
| student-tests (все пути inventory) | Регрессии источника | Сохранены OTelJobObserver/provider ЛР6; прямые handle_job получают observer. Boundary PNG генерируются локально, без зависимости от evidence ЛР5; default project/port заменены на ЛР6 |
| openspec/specs (10 capabilities) | 11 изменённых requirements | Полные delta blocks, исходные префиксы capabilities сохранены |
| openspec/config.yaml | Версии SQLAlchemy/Alembic, provenance | Применимые строки влиты, контекст ЛР6 сохранён |
## Неприменимые артефакты
AGENTS.md, REPORT.md и evidence/README.md источника относятся к ЛР5 и не заменяют документы ЛР6. Результаты evidence источника относятся к другому приложению/SHA и не копируются как доказательство ЛР6. Исполняемая логика двух targeted.py адаптирована в student-tests/tools/forward_runtime.py; их файлы результатов не импортированы. Четыре новых source OpenSpec archives — история выполненных этапов ЛР5; их требования перенесены объединённым change `forward-lab5-published-fixes`, их завершение не выдаётся за проверку ЛР6. Каждый исключённый путь перечислен в inventory с причиной. Применимых runtime-изменений не пропущено; source diff не затрагивает Dockerfile, зависимости, Alembic или compose runtime.
## Документация
Context7 resolve для SQLAlchemy и RabbitMQ вернул `Monthly quota exceeded`. Использован официальный fallback: [SQLAlchemy 2.0 with_for_update](https://docs.sqlalchemy.org/en/20/core/selectable.html#sqlalchemy.sql.expression.GenerativeSelect.with_for_update) подтверждает read+key_share; [RabbitMQ 4.1 quorum delivery limit](https://www.rabbitmq.com/docs/4.1/quorum-queues#disabling-the-limit) содержит policy delivery-limit=-1. Новых зависимостей нет; версии остаются закреплёнными.
## Проверки и затронутые измерения
Изменение auth/cache влияет на HTTP/DB путь трека, поэтому повторяются все 15 серий с прежними параметрами; `evidence/track5` остаётся историческим результатом bc0b1620825b53fd3a3019623326ebf68cbbbeec. Новые PNG/доставка проверяются через runtime ЛР6 с включённой OTel, включая память, recovery и трассы. Наблюдения, сырые результаты и завершённый повтор трека приведены ниже.
## Реальные регрессии на новом runtime
Runtime commit: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`, образ/ресурсы — forward-published/runtime.json. `make student-test`: 995 passed, 2 Linux-only skipped, 19 stack deselected; `make check validate lint`: PASS. Эти локальные проверки выполнены до runtime-коммита, на WIP поверх 97feb04b3f2a6008f636e1fdbdf284b1c97282a1 (точный SHA и команды в логах); затем код зафиксирован в bc1d7aa. Linux memory paths проверены отдельно реальным worker в Docker с 448 MiB/30 s, concurrency2 и cgroup1GiB.
- network.txt: некорректный Basic/NUL, атомарность и ключ идемпотентности, logout на обеих репликах.
- cache-outage.txt: работа при отказе Valkey, logout503 и успешный204 после восстановления.
- postgres-outage.json/txt и postgres-outage-worker.txt: 12 реальных requeue, тот же job_id/attempt1, DLQ без прироста, recovery без replay. policy.txt подтверждает delivery-limit=-1.
- png-http.json/txt: восемь PNG tRNS/Adam7/filter/EXIF случаев, v1/v2 и byte-exact сравнение с независимым белым эталоном; duplicate/stale не меняют результат.
- png-boundary.json/txt: 40MP gray2/4/16/RGB16 и opaque, обе версии; два RGB16 рендера пересекались по времени, peak913227776 bytes <1073741824, oom/oom_kill0. Forced timeout и memory дали processing_failed без retry, следующее обычное задание ready.
- haproxy-errors.txt: 12 сетевых проверок502/503/504 с request_id, JSON валиден и echo совпадает. Временный project hlpipe-lab6-forward-lb остановлен finally.
- upload-negative.txt: 56 ответов негативного suite, 0 ответов5xx.
- pipeline.json/txt: девять затронутых сценариев, включая retry, восстановление брокера/worker, COMMIT/ack и DLQ, с инвариантами.
- persistence.txt, failover.txt, restore.txt: публичные проверки и восстановление реплики PASS.
- telemetry-request/job/leaks.txt, incident-chain/retry.txt: свежие spans и логи приложения, сохранённая цепочка retry, 0 утечек.
- incident-worker/database.txt: alerts firing→resolved; dashboards.txt и obs-config.txt PASS.
Прежние проверки отказа telemetry storage и сохранения истории уровней3/4 остаются применимыми: preserved-mechanisms.json подтверждает неизменность21 файла сигналов/хранилищ. 22 защищённых файла также неизменны. Эти незатронутые серии не прогоняются повторно. Новые cost-series хранятся отдельно в forward-published/cost; метод и ограничения — cost/method.md.
## Сохраняемые ограничения для следующего отчёта
Четыре ранее установленных minor-замечания не исправляются по правилу пользователя «только критичное»: неполные allowlists вспомогательного аудита лимитов, границы API SLI для ответов HAProxy/первой серии, малый запас памяти Collector при отказе storage и whitespace сгенерированных артефактов. Файлы, доказательства и актуальность перечислены в [findings.json](forward-published/findings.json). Новый REPORT здесь не составляется; эти ограничения передаются следующему агенту отчёта. Ни один обязательный неуспешный тест не объявлен успешным.
## Итог повторного трека и завершение
15 пригодных последовательных серий (по три на off/on/head/tail/balanced), 12000 запросов, 0 неожиданных ответов. SDK ended = Collector accepted во всех сериях, экспортных ошибок0. Balanced сохранил48/48 ошибок и48/48 slow; медианный ingest Tempo130,2 KiB против1662,2 KiB у on (−92,2%). Head сохранил6/48 ошибок и3/48 slow. Это наблюдение данного workload, не доказательство статистически значимого изменения latency/CPU. Сводка, диапазоны и графики: [cost/results.md](forward-published/cost/results.md), comparison.json/png/svg. PNG графика визуально проверен.
Два первоначальных прогона on-3/head-3 с разрывом wall clock исключены; HTTP PASS не выдан за пригодное измерение. Их raw JSON/вывод сохранены с SHA-256 в invalid-time-gap и invalid-series.json. Повторены только эти две серии. measurement-audit.json подтверждает штатные окна всех15 включённых серий. Фактический фон записан при старте, после каждого круга и повторения; сторонние проекты в этих снимках отсутствуют, общий host остаётся ограничением.
Штатные SDK on/100%/10s, Collector collector.yaml и COST_EXPERIMENT=false проверены в defaults-restored.json. Финальный публичный smoke прошёл (final-smoke.txt). Собственный стенд остановлен make down без удаления volumes (shutdown.txt); closure.json подтверждает9 volumes, отсутствие своих работающих контейнеров, неизменность22 защищённых файлов, предыдущего evidence и runtime после измерений. Все нагрузки и временный caffeinate завершены. Финальные offline/check/validate/lint — final-quality.txt. Публикаций не выполнялось.
OpenSpec: 11 requirements в10 capabilities синхронизированы и побайтно сверены по блокам с immutable source и delta (spec-sync.json). Архив: openspec/changes/archive/2026-11-07-forward-lab5-published-fixes, archived2026-11-07/lab6;26 specs strict PASS, активных changes0. git diff --cached --check завершился с кодом2 из-за whitespace в сыром выводе/сгенерированных артефактах (whitespace-check.txt), замечание minor сохранено; make lint прошёл.
@@ -0,0 +1,10 @@
date: 2026-11-06T23:37:24Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/pytest -m stack -s -q student-tests/stack/test_faults.py::test_valkey_stopped_is_bypassed_but_logout_waits
valkey остановлен, действующий токен: 200 за 0.008 с
valkey остановлен, неизвестный токен: 401 за 0.004 с
valkey остановлен: ready 200, вход 201, выход 503
valkey снова запущен: повторный выход 204
.
1 passed in 4.06s
exit_code: 0 elapsed_seconds: 4.831
+21
View File
@@ -0,0 +1,21 @@
{
"at": "2026-11-07T01:12:37.041033+00:00",
"implementation_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"source_sha": "c7b8feb805d151e878d77b0dff47320a9436b172",
"protected_files": 22,
"changed_protected": [],
"changed_old_evidence": [],
"changed_runtime_since_measurements": [],
"own_running_containers": [],
"preserved_volumes": [
"mtusi2026lab6_broker-data",
"mtusi2026lab6_grafana-data",
"mtusi2026lab6_healthtools",
"mtusi2026lab6_loki-data",
"mtusi2026lab6_pgdata",
"mtusi2026lab6_prometheus-data",
"mtusi2026lab6_s3data",
"mtusi2026lab6_tempo-data",
"mtusi2026lab6_valkeydata"
]
}
@@ -0,0 +1,102 @@
{
"at": "2026-11-06T23:37:22.532499+00:00",
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"platform": "Darwin",
"architecture": "arm64",
"host_cpus": 16,
"host_load": [
3.3583984375,
2.19873046875,
2.24169921875
],
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
"own_project": "mtusi2026lab6",
"own": [
{
"Name": "mtusi2026lab6-relay-1",
"CPUPerc": "0.14%",
"MemUsage": "116.9MiB / 192MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-api-2",
"CPUPerc": "0.97%",
"MemUsage": "193.6MiB / 512MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-api-1",
"CPUPerc": "0.78%",
"MemUsage": "193.4MiB / 512MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-worker-1",
"CPUPerc": "0.14%",
"MemUsage": "110.9MiB / 1GiB",
"PIDs": "6"
},
{
"Name": "mtusi2026lab6-grafana-1",
"CPUPerc": "0.31%",
"MemUsage": "314.9MiB / 384MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-loki-1",
"CPUPerc": "0.84%",
"MemUsage": "151.9MiB / 384MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-tempo-1",
"CPUPerc": "0.25%",
"MemUsage": "106.5MiB / 384MiB",
"PIDs": "9"
},
{
"Name": "mtusi2026lab6-otel-collector-1",
"CPUPerc": "0.32%",
"MemUsage": "238.1MiB / 256MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-broker-1",
"CPUPerc": "0.13%",
"MemUsage": "94.06MiB / 512MiB",
"PIDs": "53"
},
{
"Name": "mtusi2026lab6-postgres-1",
"CPUPerc": "0.94%",
"MemUsage": "45.65MiB / 768MiB",
"PIDs": "13"
},
{
"Name": "mtusi2026lab6-lb-1",
"CPUPerc": "0.10%",
"MemUsage": "22.77MiB / 128MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-prometheus-1",
"CPUPerc": "0.23%",
"MemUsage": "173.6MiB / 512MiB",
"PIDs": "6"
},
{
"Name": "mtusi2026lab6-s3-1",
"CPUPerc": "0.06%",
"MemUsage": "71.48MiB / 512MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-valkey-1",
"CPUPerc": "1.37%",
"MemUsage": "12.66MiB / 128MiB",
"PIDs": "6"
}
],
"other_projects": {},
"sampling": "Shared Docker snapshot; no throughput claim"
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-06T23:59:06Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode balanced --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/balanced-1.json
{"mode": "balanced", "repeat": 1, "result": "PASS", "retained": {"normal": 39, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 126.615
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:09:36Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode balanced --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/balanced-2.json
{"mode": "balanced", "repeat": 2, "result": "PASS", "retained": {"normal": 32, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 126.494
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T01:05:01Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode balanced --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/balanced-3.json
{"mode": "balanced", "repeat": 3, "result": "PASS", "retained": {"normal": 30, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 126.531
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 128 KiB

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 99 KiB

@@ -0,0 +1,102 @@
{
"at": "2026-11-07T01:11:40.821011+00:00",
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"platform": "Darwin",
"architecture": "arm64",
"host_cpus": 16,
"host_load": [
2.5986328125,
2.326171875,
2.35888671875
],
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
"own_project": "mtusi2026lab6",
"own": [
{
"Name": "mtusi2026lab6-api-1",
"CPUPerc": "0.71%",
"MemUsage": "184.3MiB / 512MiB",
"PIDs": "13"
},
{
"Name": "mtusi2026lab6-api-2",
"CPUPerc": "1.70%",
"MemUsage": "192.5MiB / 512MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-relay-1",
"CPUPerc": "0.67%",
"MemUsage": "116.6MiB / 192MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-worker-1",
"CPUPerc": "0.14%",
"MemUsage": "108.3MiB / 1GiB",
"PIDs": "6"
},
{
"Name": "mtusi2026lab6-otel-collector-1",
"CPUPerc": "0.15%",
"MemUsage": "52.7MiB / 256MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-grafana-1",
"CPUPerc": "0.39%",
"MemUsage": "320.1MiB / 384MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-loki-1",
"CPUPerc": "0.75%",
"MemUsage": "225.5MiB / 384MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-tempo-1",
"CPUPerc": "0.25%",
"MemUsage": "227MiB / 384MiB",
"PIDs": "9"
},
{
"Name": "mtusi2026lab6-broker-1",
"CPUPerc": "24.52%",
"MemUsage": "95.97MiB / 512MiB",
"PIDs": "53"
},
{
"Name": "mtusi2026lab6-postgres-1",
"CPUPerc": "0.53%",
"MemUsage": "45.99MiB / 768MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-lb-1",
"CPUPerc": "0.66%",
"MemUsage": "25.02MiB / 128MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-prometheus-1",
"CPUPerc": "0.14%",
"MemUsage": "158.4MiB / 512MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-s3-1",
"CPUPerc": "0.03%",
"MemUsage": "160.2MiB / 512MiB",
"PIDs": "25"
},
{
"Name": "mtusi2026lab6-valkey-1",
"CPUPerc": "0.13%",
"MemUsage": "13.07MiB / 128MiB",
"PIDs": "6"
}
],
"other_projects": {},
"sampling": "Shared Docker snapshot; no throughput claim"
}
@@ -0,0 +1,102 @@
{
"at": "2026-11-07T01:07:27.961141+00:00",
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"platform": "Darwin",
"architecture": "arm64",
"host_cpus": 16,
"host_load": [
1.646484375,
1.9970703125,
2.29052734375
],
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
"own_project": "mtusi2026lab6",
"own": [
{
"Name": "mtusi2026lab6-api-1",
"CPUPerc": "1.17%",
"MemUsage": "192.9MiB / 512MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-api-2",
"CPUPerc": "1.03%",
"MemUsage": "192.8MiB / 512MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-worker-1",
"CPUPerc": "0.62%",
"MemUsage": "108.4MiB / 1GiB",
"PIDs": "6"
},
{
"Name": "mtusi2026lab6-relay-1",
"CPUPerc": "0.80%",
"MemUsage": "116.7MiB / 192MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-otel-collector-1",
"CPUPerc": "0.27%",
"MemUsage": "42.03MiB / 256MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-grafana-1",
"CPUPerc": "1.83%",
"MemUsage": "320MiB / 384MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-loki-1",
"CPUPerc": "0.62%",
"MemUsage": "228.5MiB / 384MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-tempo-1",
"CPUPerc": "0.33%",
"MemUsage": "287.3MiB / 384MiB",
"PIDs": "9"
},
{
"Name": "mtusi2026lab6-broker-1",
"CPUPerc": "36.74%",
"MemUsage": "183.4MiB / 512MiB",
"PIDs": "103"
},
{
"Name": "mtusi2026lab6-postgres-1",
"CPUPerc": "0.94%",
"MemUsage": "47.72MiB / 768MiB",
"PIDs": "13"
},
{
"Name": "mtusi2026lab6-lb-1",
"CPUPerc": "0.61%",
"MemUsage": "24.96MiB / 128MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-prometheus-1",
"CPUPerc": "0.59%",
"MemUsage": "157.3MiB / 512MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-s3-1",
"CPUPerc": "0.93%",
"MemUsage": "160.2MiB / 512MiB",
"PIDs": "25"
},
{
"Name": "mtusi2026lab6-valkey-1",
"CPUPerc": "0.78%",
"MemUsage": "13.07MiB / 128MiB",
"PIDs": "6"
}
],
"other_projects": {},
"sampling": "Shared Docker snapshot; no throughput claim"
}
@@ -0,0 +1,102 @@
{
"at": "2026-11-07T00:01:16.214846+00:00",
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"platform": "Darwin",
"architecture": "arm64",
"host_cpus": 16,
"host_load": [
1.6005859375,
1.84521484375,
2.02685546875
],
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
"own_project": "mtusi2026lab6",
"own": [
{
"Name": "mtusi2026lab6-api-1",
"CPUPerc": "1.28%",
"MemUsage": "184MiB / 512MiB",
"PIDs": "13"
},
{
"Name": "mtusi2026lab6-api-2",
"CPUPerc": "0.83%",
"MemUsage": "184.3MiB / 512MiB",
"PIDs": "13"
},
{
"Name": "mtusi2026lab6-worker-1",
"CPUPerc": "0.15%",
"MemUsage": "108.3MiB / 1GiB",
"PIDs": "6"
},
{
"Name": "mtusi2026lab6-relay-1",
"CPUPerc": "0.19%",
"MemUsage": "116.5MiB / 192MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-otel-collector-1",
"CPUPerc": "0.36%",
"MemUsage": "46.17MiB / 256MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-grafana-1",
"CPUPerc": "0.32%",
"MemUsage": "319.3MiB / 384MiB",
"PIDs": "10"
},
{
"Name": "mtusi2026lab6-loki-1",
"CPUPerc": "0.97%",
"MemUsage": "214.4MiB / 384MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-tempo-1",
"CPUPerc": "0.23%",
"MemUsage": "224.8MiB / 384MiB",
"PIDs": "9"
},
{
"Name": "mtusi2026lab6-broker-1",
"CPUPerc": "0.24%",
"MemUsage": "95.74MiB / 512MiB",
"PIDs": "53"
},
{
"Name": "mtusi2026lab6-postgres-1",
"CPUPerc": "1.49%",
"MemUsage": "44.05MiB / 768MiB",
"PIDs": "11"
},
{
"Name": "mtusi2026lab6-lb-1",
"CPUPerc": "1.03%",
"MemUsage": "23.41MiB / 128MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-prometheus-1",
"CPUPerc": "1.11%",
"MemUsage": "154.9MiB / 512MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-s3-1",
"CPUPerc": "0.05%",
"MemUsage": "160.2MiB / 512MiB",
"PIDs": "25"
},
{
"Name": "mtusi2026lab6-valkey-1",
"CPUPerc": "1.18%",
"MemUsage": "13.05MiB / 128MiB",
"PIDs": "6"
}
],
"other_projects": {},
"sampling": "Shared Docker snapshot; no throughput claim"
}
@@ -0,0 +1,102 @@
{
"at": "2026-11-07T00:11:45.236693+00:00",
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"platform": "Darwin",
"architecture": "arm64",
"host_cpus": 16,
"host_load": [
4.9462890625,
3.5625,
2.7451171875
],
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
"own_project": "mtusi2026lab6",
"own": [
{
"Name": "mtusi2026lab6-api-2",
"CPUPerc": "1.19%",
"MemUsage": "184.1MiB / 512MiB",
"PIDs": "13"
},
{
"Name": "mtusi2026lab6-api-1",
"CPUPerc": "3.14%",
"MemUsage": "191.7MiB / 512MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-relay-1",
"CPUPerc": "0.62%",
"MemUsage": "116.5MiB / 192MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-worker-1",
"CPUPerc": "0.17%",
"MemUsage": "108.3MiB / 1GiB",
"PIDs": "6"
},
{
"Name": "mtusi2026lab6-otel-collector-1",
"CPUPerc": "0.28%",
"MemUsage": "42.81MiB / 256MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-grafana-1",
"CPUPerc": "0.30%",
"MemUsage": "320MiB / 384MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-loki-1",
"CPUPerc": "1.56%",
"MemUsage": "221.1MiB / 384MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-tempo-1",
"CPUPerc": "1.27%",
"MemUsage": "233.4MiB / 384MiB",
"PIDs": "9"
},
{
"Name": "mtusi2026lab6-broker-1",
"CPUPerc": "0.25%",
"MemUsage": "95.85MiB / 512MiB",
"PIDs": "53"
},
{
"Name": "mtusi2026lab6-postgres-1",
"CPUPerc": "0.73%",
"MemUsage": "45.72MiB / 768MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-lb-1",
"CPUPerc": "0.17%",
"MemUsage": "24.55MiB / 128MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-prometheus-1",
"CPUPerc": "0.17%",
"MemUsage": "155MiB / 512MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-s3-1",
"CPUPerc": "0.02%",
"MemUsage": "160.2MiB / 512MiB",
"PIDs": "25"
},
{
"Name": "mtusi2026lab6-valkey-1",
"CPUPerc": "1.13%",
"MemUsage": "13.12MiB / 128MiB",
"PIDs": "6"
}
],
"other_projects": {},
"sampling": "Shared Docker snapshot; no throughput claim"
}
@@ -0,0 +1,102 @@
{
"at": "2026-11-07T01:07:10.780508+00:00",
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"platform": "Darwin",
"architecture": "arm64",
"host_cpus": 16,
"host_load": [
1.6572265625,
2.015625,
2.30224609375
],
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
"own_project": "mtusi2026lab6",
"own": [
{
"Name": "mtusi2026lab6-api-1",
"CPUPerc": "1.54%",
"MemUsage": "183.9MiB / 512MiB",
"PIDs": "13"
},
{
"Name": "mtusi2026lab6-api-2",
"CPUPerc": "3.32%",
"MemUsage": "192.2MiB / 512MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-worker-1",
"CPUPerc": "0.18%",
"MemUsage": "108.3MiB / 1GiB",
"PIDs": "6"
},
{
"Name": "mtusi2026lab6-relay-1",
"CPUPerc": "0.53%",
"MemUsage": "116.5MiB / 192MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-otel-collector-1",
"CPUPerc": "0.40%",
"MemUsage": "40.87MiB / 256MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-grafana-1",
"CPUPerc": "0.42%",
"MemUsage": "320MiB / 384MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-loki-1",
"CPUPerc": "0.70%",
"MemUsage": "228.8MiB / 384MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-tempo-1",
"CPUPerc": "0.31%",
"MemUsage": "269.3MiB / 384MiB",
"PIDs": "9"
},
{
"Name": "mtusi2026lab6-broker-1",
"CPUPerc": "0.23%",
"MemUsage": "96.34MiB / 512MiB",
"PIDs": "53"
},
{
"Name": "mtusi2026lab6-postgres-1",
"CPUPerc": "0.71%",
"MemUsage": "45.93MiB / 768MiB",
"PIDs": "12"
},
{
"Name": "mtusi2026lab6-lb-1",
"CPUPerc": "0.17%",
"MemUsage": "24.96MiB / 128MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-prometheus-1",
"CPUPerc": "0.14%",
"MemUsage": "157.3MiB / 512MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-s3-1",
"CPUPerc": "0.04%",
"MemUsage": "160.2MiB / 512MiB",
"PIDs": "25"
},
{
"Name": "mtusi2026lab6-valkey-1",
"CPUPerc": "0.17%",
"MemUsage": "13.07MiB / 128MiB",
"PIDs": "6"
}
],
"other_projects": {},
"sampling": "Shared Docker snapshot; no throughput claim"
}
@@ -0,0 +1,102 @@
{
"at": "2026-11-06T23:50:46.782935+00:00",
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
"platform": "Darwin",
"architecture": "arm64",
"host_cpus": 16,
"host_load": [
2.43505859375,
2.41015625,
2.33154296875
],
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
"own_project": "mtusi2026lab6",
"own": [
{
"Name": "mtusi2026lab6-worker-1",
"CPUPerc": "0.25%",
"MemUsage": "339.7MiB / 1GiB",
"PIDs": "15"
},
{
"Name": "mtusi2026lab6-relay-1",
"CPUPerc": "0.15%",
"MemUsage": "118.1MiB / 192MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-api-2",
"CPUPerc": "0.80%",
"MemUsage": "250.3MiB / 512MiB",
"PIDs": "18"
},
{
"Name": "mtusi2026lab6-api-1",
"CPUPerc": "8.79%",
"MemUsage": "255.3MiB / 512MiB",
"PIDs": "19"
},
{
"Name": "mtusi2026lab6-grafana-1",
"CPUPerc": "0.34%",
"MemUsage": "320.6MiB / 384MiB",
"PIDs": "10"
},
{
"Name": "mtusi2026lab6-loki-1",
"CPUPerc": "0.51%",
"MemUsage": "210.6MiB / 384MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-tempo-1",
"CPUPerc": "0.23%",
"MemUsage": "200.6MiB / 384MiB",
"PIDs": "9"
},
{
"Name": "mtusi2026lab6-otel-collector-1",
"CPUPerc": "0.07%",
"MemUsage": "107.4MiB / 256MiB",
"PIDs": "8"
},
{
"Name": "mtusi2026lab6-broker-1",
"CPUPerc": "1.14%",
"MemUsage": "87.95MiB / 512MiB",
"PIDs": "53"
},
{
"Name": "mtusi2026lab6-postgres-1",
"CPUPerc": "1.74%",
"MemUsage": "76.54MiB / 768MiB",
"PIDs": "29"
},
{
"Name": "mtusi2026lab6-lb-1",
"CPUPerc": "0.99%",
"MemUsage": "18.9MiB / 128MiB",
"PIDs": "17"
},
{
"Name": "mtusi2026lab6-prometheus-1",
"CPUPerc": "0.12%",
"MemUsage": "154.4MiB / 512MiB",
"PIDs": "7"
},
{
"Name": "mtusi2026lab6-s3-1",
"CPUPerc": "1.27%",
"MemUsage": "160.9MiB / 512MiB",
"PIDs": "25"
},
{
"Name": "mtusi2026lab6-valkey-1",
"CPUPerc": "1.20%",
"MemUsage": "13.05MiB / 128MiB",
"PIDs": "6"
}
],
"other_projects": {},
"sampling": "Shared Docker snapshot; no throughput claim"
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-06T23:54:56Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode head --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/head-1.json
{"mode": "head", "repeat": 1, "result": "PASS", "retained": {"normal": 85, "error": 1, "slow": 1}}
exit_code: 0 elapsed_seconds: 124.106
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:05:25Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode head --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/head-2.json
{"mode": "head", "repeat": 2, "result": "PASS", "retained": {"normal": 76, "error": 4, "slow": 1}}
exit_code: 0 elapsed_seconds: 124.746
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T01:09:32Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode head --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/head-3.json
{"mode": "head", "repeat": 3, "result": "PASS", "retained": {"normal": 76, "error": 1, "slow": 1}}
exit_code: 0 elapsed_seconds: 125.216
@@ -0,0 +1,28 @@
{
"decision_at": "2026-11-07T01:05:32.987756+00:00",
"interpretation": "Observed wall-clock gaps while monotonic elapsed remains nominal; likely host suspension or clock discontinuity. Cause not asserted as proven. Only these two completed series are repeated; raw originals preserved byte-for-byte.",
"runs": [
{
"series": "on-3",
"reason": "Wall-clock discontinuity inside the measurement/accounting window; excluded from resource/ingest comparison, HTTP PASS alone is insufficient.",
"load_seconds": 39.970131158828735,
"finish_to_report_seconds": 932.7143309116364,
"max_resource_gap_seconds": 2.047463893890381,
"files": {
"on-3.json": "d0c4799cea3f5a5c819c175c738b6099158bc5ca9b6d2d840a89c9d73d394953",
"on-3.txt": "64199cc40d2a81d15a219e63c04aac875929eba41110bfbf3909a4d43518e010"
}
},
{
"series": "head-3",
"reason": "Wall-clock discontinuity inside the measurement/accounting window; excluded from resource/ingest comparison, HTTP PASS alone is insufficient.",
"load_seconds": 938.1542620658875,
"finish_to_report_seconds": 34.9133198261261,
"max_resource_gap_seconds": 900.3541407585144,
"files": {
"head-3.json": "3dbecccf017b26bc6a256860b795f8cc9e4c47099da2d46e8303120f49686f45",
"head-3.txt": "0b4f1607837921746893700a5897b1a1c94db115d59c750f3ec9631266e95a69"
}
}
]
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:30:52Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode head --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/head-3.json
{"mode": "head", "repeat": 3, "result": "PASS", "retained": {"normal": 70, "error": 1, "slow": 3}}
exit_code: 0 elapsed_seconds: 125.007
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:13:50Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode on --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/on-3.json
{"mode": "on", "repeat": 3, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 123.841
@@ -0,0 +1,115 @@
{
"at": "2026-11-07T01:11:53.670099+00:00",
"policy": "40s workload: wall window38..42s, finish-to-report30..45s, maximum2s-sampling gap<5s. All three conditions required independently of HTTP PASS.",
"runs": [
{
"file": "off-1.json",
"load_seconds": 39.9768009185791,
"finish_to_report_seconds": 34.378885984420776,
"max_resource_gap_seconds": 2.047776937484741,
"result": "usable"
},
{
"file": "off-2.json",
"load_seconds": 39.96762013435364,
"finish_to_report_seconds": 35.243422985076904,
"max_resource_gap_seconds": 2.048405885696411,
"result": "usable"
},
{
"file": "off-3.json",
"load_seconds": 39.97038507461548,
"finish_to_report_seconds": 35.21114897727966,
"max_resource_gap_seconds": 2.0483851432800293,
"result": "usable"
},
{
"file": "on-1.json",
"load_seconds": 39.960899114608765,
"finish_to_report_seconds": 35.31035900115967,
"max_resource_gap_seconds": 2.0440289974212646,
"result": "usable"
},
{
"file": "on-2.json",
"load_seconds": 39.967061042785645,
"finish_to_report_seconds": 34.75409412384033,
"max_resource_gap_seconds": 2.045710802078247,
"result": "usable"
},
{
"file": "on-3.json",
"load_seconds": 39.961934089660645,
"finish_to_report_seconds": 35.3386869430542,
"max_resource_gap_seconds": 2.0448179244995117,
"result": "usable"
},
{
"file": "head-1.json",
"load_seconds": 39.960408210754395,
"finish_to_report_seconds": 34.749109983444214,
"max_resource_gap_seconds": 2.0449328422546387,
"result": "usable"
},
{
"file": "head-2.json",
"load_seconds": 39.95907402038574,
"finish_to_report_seconds": 34.537447929382324,
"max_resource_gap_seconds": 2.0501198768615723,
"result": "usable"
},
{
"file": "head-3.json",
"load_seconds": 39.96534299850464,
"finish_to_report_seconds": 35.03631401062012,
"max_resource_gap_seconds": 2.0513830184936523,
"result": "usable"
},
{
"file": "tail-1.json",
"load_seconds": 39.969640254974365,
"finish_to_report_seconds": 35.31044888496399,
"max_resource_gap_seconds": 2.0470268726348877,
"result": "usable"
},
{
"file": "tail-2.json",
"load_seconds": 39.96142888069153,
"finish_to_report_seconds": 34.63654613494873,
"max_resource_gap_seconds": 2.0448548793792725,
"result": "usable"
},
{
"file": "tail-3.json",
"load_seconds": 39.968284130096436,
"finish_to_report_seconds": 35.04265284538269,
"max_resource_gap_seconds": 2.047273874282837,
"result": "usable"
},
{
"file": "balanced-1.json",
"load_seconds": 39.965638160705566,
"finish_to_report_seconds": 34.83575201034546,
"max_resource_gap_seconds": 2.0490190982818604,
"result": "usable"
},
{
"file": "balanced-2.json",
"load_seconds": 39.96572422981262,
"finish_to_report_seconds": 35.22300386428833,
"max_resource_gap_seconds": 2.0449130535125732,
"result": "usable"
},
{
"file": "balanced-3.json",
"load_seconds": 39.96644401550293,
"finish_to_report_seconds": 35.353606939315796,
"max_resource_gap_seconds": 2.0450639724731445,
"result": "usable"
}
],
"excluded": [
"invalid-time-gap/on-3.json",
"invalid-time-gap/head-3.json"
]
}
@@ -0,0 +1,115 @@
{
"at": "2026-11-07T01:11:53.670099+00:00",
"policy": "40s workload: wall window38..42s, finish-to-report30..45s, maximum2s-sampling gap<5s. All three conditions required independently of HTTP PASS.",
"runs": [
{
"file": "off-1.json",
"load_seconds": 39.9768009185791,
"finish_to_report_seconds": 34.378885984420776,
"max_resource_gap_seconds": 2.047776937484741,
"result": "usable"
},
{
"file": "off-2.json",
"load_seconds": 39.96762013435364,
"finish_to_report_seconds": 35.243422985076904,
"max_resource_gap_seconds": 2.048405885696411,
"result": "usable"
},
{
"file": "off-3.json",
"load_seconds": 39.97038507461548,
"finish_to_report_seconds": 35.21114897727966,
"max_resource_gap_seconds": 2.0483851432800293,
"result": "usable"
},
{
"file": "on-1.json",
"load_seconds": 39.960899114608765,
"finish_to_report_seconds": 35.31035900115967,
"max_resource_gap_seconds": 2.0440289974212646,
"result": "usable"
},
{
"file": "on-2.json",
"load_seconds": 39.967061042785645,
"finish_to_report_seconds": 34.75409412384033,
"max_resource_gap_seconds": 2.045710802078247,
"result": "usable"
},
{
"file": "on-3.json",
"load_seconds": 39.961934089660645,
"finish_to_report_seconds": 35.3386869430542,
"max_resource_gap_seconds": 2.0448179244995117,
"result": "usable"
},
{
"file": "head-1.json",
"load_seconds": 39.960408210754395,
"finish_to_report_seconds": 34.749109983444214,
"max_resource_gap_seconds": 2.0449328422546387,
"result": "usable"
},
{
"file": "head-2.json",
"load_seconds": 39.95907402038574,
"finish_to_report_seconds": 34.537447929382324,
"max_resource_gap_seconds": 2.0501198768615723,
"result": "usable"
},
{
"file": "head-3.json",
"load_seconds": 39.96534299850464,
"finish_to_report_seconds": 35.03631401062012,
"max_resource_gap_seconds": 2.0513830184936523,
"result": "usable"
},
{
"file": "tail-1.json",
"load_seconds": 39.969640254974365,
"finish_to_report_seconds": 35.31044888496399,
"max_resource_gap_seconds": 2.0470268726348877,
"result": "usable"
},
{
"file": "tail-2.json",
"load_seconds": 39.96142888069153,
"finish_to_report_seconds": 34.63654613494873,
"max_resource_gap_seconds": 2.0448548793792725,
"result": "usable"
},
{
"file": "tail-3.json",
"load_seconds": 39.968284130096436,
"finish_to_report_seconds": 35.04265284538269,
"max_resource_gap_seconds": 2.047273874282837,
"result": "usable"
},
{
"file": "balanced-1.json",
"load_seconds": 39.965638160705566,
"finish_to_report_seconds": 34.83575201034546,
"max_resource_gap_seconds": 2.0490190982818604,
"result": "usable"
},
{
"file": "balanced-2.json",
"load_seconds": 39.96572422981262,
"finish_to_report_seconds": 35.22300386428833,
"max_resource_gap_seconds": 2.0449130535125732,
"result": "usable"
},
{
"file": "balanced-3.json",
"load_seconds": 39.96644401550293,
"finish_to_report_seconds": 35.353606939315796,
"max_resource_gap_seconds": 2.0450639724731445,
"result": "usable"
}
],
"excluded": [
"invalid-time-gap/on-3.json",
"invalid-time-gap/head-3.json"
]
}
+21
View File
@@ -0,0 +1,21 @@
# Повтор трека после позднего переноса
Приложение и измеритель: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`. Образ/лимиты: `../runtime.json`. Причина повторения — изменён путь auth/cache, используемый GET users/me; исторические серии `evidence/track5` на bc0b162 остаются неизменными и не выдаются за результат нового кода.
Гипотеза: head 10% уменьшает ingest, теряя редкие события; tail сохраняет ошибки и slow, сокращая запись в Tempo, но сохраняет расходы SDK. Balanced: tail 5% обычных трасс, все ERROR и latency >=300 ms; период метрик 30 s вместо 10 s.
Команда каждой серии: `scripts/lab6.sh python3 scripts/obs_cost.py --mode MODE --repeat N --duration 40 --rate 20 --output evidence/forward-published/cost/MODE-N.json`. Порядок: off/on/head/tail/balanced, три круга. 20 прогревочных GET, 35 s ожидания экспорта; 40 s open-loop, 800 arrivals с периодом 50 ms, 64 клиентских потока. 768 обычных запросов, 16 контролируемых 503, 16 задержек 350 ms. Затем 35 s drain. finally возвращает обычный Collector, SDK on/ratio1/interval10s и отключает fault hook. Все серии идут последовательно, отдельные проверки и другие нагрузки этого агента одновременно не запускаются.
Latency измерена клиентом, schedule_lag записан отдельно. CPU — delta cgroup total_usage / время внешних Docker samples; RAM — usage−inactive_file. Это не SDK-метрики. Группы application (API, relay, worker), observability, business stores/LB раздельны. Окно CPU охватывает нагрузку, не последнюю работу tail sampler во время drain; RAM — пик наблюдаемых samples, не непрерывный максимум.
Ingest: Tempo/Loki received bytes и Prometheus appended samples; разные единицы не складываются. Окно включает drain и постоянный фон self-scrapes. SDK ended сравнивается с Collector accepted, export failures учитываются отдельно; для tail accepted−sent включает намеренную фильтрацию. Отбор server traces по уникальному префиксу каждой серии и категории, search limit5000 >800. Для head неизвестное число несозданных DB spans не оценивается вымышленным числом; наблюдается доля server roots. Off — отсутствие SDK, а не ошибка доставки.
Фон Docker/host записан в conditions-start и conditions-round-N, inventory дополнительно есть в каждом raw JSON. Host не эксклюзивен, поэтому малые различия latency/CPU не доказывают статистическую значимость. Три повтора дают наблюдаемый диапазон, не доверительный интервал. Workload HTTP/DB, worker ожидает заданий; массовая обработка изображений отдельно проверена в png-boundary.json. Tail decision_wait10s не доказывает полноту цепочек с очень поздними retry. Успешные завершённые серии сохраняются для финального аудита без автоматического повторного прогона.
Сводка: `python3 scripts/obs_cost_summary.py evidence/forward-published/cost`. Графики: `uv run --no-project --with matplotlib==3.11.2 python tools/plot_observability.py evidence/forward-published/cost`.
## Пригодность серий и повтор двух прогонов
После двух первых кругов обнаружен разрыв настенных часов относительно монотонного таймера (вероятен сон host, причина не утверждается доказанной). В первоначальном on-3 окно от конца нагрузки до отчёта стало 932,714 s, в head-3 сама нагрузка по wall clock — 938,154 s, максимальный разрыв внешних samples — 900,354 s. HTTP PASS этих прогонов не означает пригодность CPU/ingest. Оба оригинала сохранены без изменения в invalid-time-gap, SHA-256 и решение — invalid-series.json; они исключены из сравнения. Остальные завершённые серии не перезапускаются.
Для оставшихся измерений включён ограниченный по времени `caffeinate -i -s`; повторяются только on-3 и head-3 с теми же параметрами и SHA, последовательно. Проверка пригодности всех включённых серий: finished−started 38..42 s, utc−finished 30..45 s (35 s drain плюс запросы/округление), max соседнего resource.at gap <5 s при периоде2 s. Результат — measurement-audit.json. Отдельные паузы до начала измеряемого окна не включаются в CPU/ingest и сами по себе не бракуют серию. Контекст общего host и фактический фон остаются ограничениями.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-06T23:50:47Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode off --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/off-1.json
{"mode": "off", "repeat": 1, "result": "PASS", "retained": {"normal": 0, "error": 0, "slow": 0}}
exit_code: 0 elapsed_seconds: 125.202
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:01:16Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode off --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/off-2.json
{"mode": "off", "repeat": 2, "result": "PASS", "retained": {"normal": 0, "error": 0, "slow": 0}}
exit_code: 0 elapsed_seconds: 125.016
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:11:45Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode off --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/off-3.json
{"mode": "off", "repeat": 3, "result": "PASS", "retained": {"normal": 0, "error": 0, "slow": 0}}
exit_code: 0 elapsed_seconds: 125.038
File diff suppressed because it is too large Load Diff
+5
View File
@@ -0,0 +1,5 @@
date: 2026-11-06T23:52:52Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode on --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/on-1.json
{"mode": "on", "repeat": 1, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 124.436
File diff suppressed because it is too large Load Diff
+5
View File
@@ -0,0 +1,5 @@
date: 2026-11-07T00:03:21Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode on --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/on-2.json
{"mode": "on", "repeat": 2, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 123.922
File diff suppressed because it is too large Load Diff
+5
View File
@@ -0,0 +1,5 @@
date: 2026-11-07T01:07:28Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode on --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/on-3.json
{"mode": "on", "repeat": 3, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 124.718
@@ -0,0 +1,29 @@
# Результаты повторного трека
Приложение и измеритель: bc1d7aa7713858acb5d65b61d07e69a640f1ee97. Все 15 пригодных серий завершены: 12000 запросов, 0 неожиданных ответов. Два первоначальных прогона (ещё 1600 запросов) исключены по разрывам времени, сохранены в invalid-time-gap и повторены; подробности в method.md и invalid-series.json. Все включённые серии прошли measurement-audit.json. Во всех сериях SDK ended = Collector accepted, exporter failures = 0.
| Режим | normal p95 ms: median [min;max] | общий p99 ms | CPU app / obs cores | RAM app / obs MiB, median peak |
|---|---:|---:|---:|---:|
| off | 11.583 [11.572; 11.856] | 358.73 | 0.1404 / 0.0256 | 655.5 / 1013.3 |
| on | 12.709 [12.554; 12.993] | 359.47 | 0.1726 / 0.0287 | 706.8 / 1058.5 |
| head | 12.556 [12.354; 12.652] | 358.37 | 0.1657 / 0.0261 | 723.8 / 1076.1 |
| tail | 12.781 [12.555; 12.920] | 359.11 | 0.1717 / 0.0272 | 721.0 / 1021.2 |
| balanced | 12.895 [12.833; 13.031] | 359.75 | 0.1749 / 0.0258 | 707.1 / 1027.9 |
| Режим | Tempo KiB | Loki KiB | Prom samples | SDK / sent spans, median | normal /2304 | error /48 | slow /48 |
|---|---:|---:|---:|---:|---:|---:|---:|
| off | 0.0 | 0.0 | 33118 | 0 / 0 | 0 | 0 | 0 |
| on | 1662.2 | 288.3 | 36284 | 5489 / 5489 | 2304 | 48 | 48 |
| head | 176.2 | 288.4 | 37234 | 543 / 543 | 237 | 6 | 3 |
| tail | 251.8 | 289.7 | 36657 | 5492 / 699 | 235 | 48 | 48 |
| balanced | 130.2 | 289.8 | 34459 | 5486 / 352 | 101 | 48 | 48 |
Balanced сохраняет 48/48 errors и 48/48 slow, медианный Tempo ingest ниже on на 92.2%. Head пропустил 42 ошибок и 45 slow. Это отбор sampling, а не сбой доставки.
Фактическая скорость 20.012–20.020 requests/s. Максимальный schedule lag 61.055 ms. Внешних samples на серию: 21–21.
Рекомендация в пределах этого workload: balanced сокращает запись трасс и сохраняет контрольные ERROR/slow, но не устраняет цену SDK. Все spans создаются до tail-решения. Малые различия latency/CPU не доказывают статистическую значимость; ограничения общего host, HTTP/DB workload и окна CPU приведены в method.md. Поздние spans за пределами decision window отдельно не доказаны.
Проверенная конфигурация: `OTEL_COLLECTOR_CONFIG=collector-balanced.yaml OTEL_METRIC_EXPORT_INTERVAL=30000 scripts/lab6.sh docker compose up -d --no-deps --wait otel-collector api relay worker`. Экспериментальные fault hooks по умолчанию выключены. После серий восстановлен default on/100%, рекомендация не включается скрыто.
comparison.json рассчитан из 15 raw JSON; comparison.png/svg показывают median, отдельные повторы и min/max. Старые evidence/track5 не изменены и не смешаны с этим набором.
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-06T23:57:01Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode tail --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/tail-1.json
{"mode": "tail", "repeat": 1, "result": "PASS", "retained": {"normal": 82, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 125.775
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:07:30Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode tail --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/tail-2.json
{"mode": "tail", "repeat": 2, "result": "PASS", "retained": {"normal": 65, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 125.706
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,5 @@
date: 2026-11-07T00:47:55Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_cost.py --mode tail --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/tail-3.json
{"mode": "tail", "repeat": 3, "result": "PASS", "retained": {"normal": 88, "error": 16, "slow": 16}}
exit_code: 0 elapsed_seconds: 127.417
+230
View File
@@ -0,0 +1,230 @@
date: 2026-11-06T23:49:55Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ python3 scripts/obs_check.py dashboards --strict
{
"checked_at": "2026-11-06T23:49:56+00:00",
"targets": [
{
"dashboard": "messenger-api.json",
"panel": "RPS бизнес-запросов",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Доля 5xx (2m)",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "p95 latency (5m)",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Доля не-5xx (окно 1h)",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "RPS по маршруту и классу статуса",
"datasource": "prometheus",
"series": 11,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Доля ошибок",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Доля ошибок",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Latency p50 / p95 / p99",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Latency p50 / p95 / p99",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Latency p50 / p95 / p99",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "p95 по маршруту",
"datasource": "prometheus",
"series": 9,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "RPS по реплике",
"datasource": "prometheus",
"series": 4,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Активные запросы",
"datasource": "prometheus",
"series": 6,
"missing_metrics": []
},
{
"dashboard": "messenger-api.json",
"panel": "Логи API (ошибки и предупреждения)",
"datasource": "loki",
"lines": 5
},
{
"dashboard": "messenger-api.json",
"panel": "Трассы API с ошибкой",
"datasource": "tempo",
"traces": 5
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Jobs по исходу",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Длительность обработки job",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Длительность обработки job",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Длительность обработки job",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Логи worker",
"datasource": "loki",
"lines": 5
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Трассы worker",
"datasource": "tempo",
"traces": 5
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Durable queue depth (includes retry)",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Oldest durable job age",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "DB pool used / waiting / capacity",
"datasource": "prometheus",
"series": 9,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Worker busy / capacity",
"datasource": "prometheus",
"series": 2,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Process CPU cores",
"datasource": "prometheus",
"series": 3,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Process resident memory",
"datasource": "prometheus",
"series": 3,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Time to ready p50 / p95 / p99",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Time to ready p50 / p95 / p99",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Time to ready p50 / p95 / p99",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "Ready within 30s, SLO 95% / 1h",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
},
{
"dashboard": "messenger-pipeline.json",
"panel": "API availability, SLO 99% / 1h",
"datasource": "prometheus",
"series": 1,
"missing_metrics": []
}
],
"problems": []
}
exit_code: 0 elapsed_seconds: 0.112
@@ -0,0 +1,45 @@
{
"at": "2026-11-07T01:11:54.405624+00:00",
"process_settings": [
{
"name": "/mtusi2026lab6-api-1",
"settings": {
"OTEL_SDK_DISABLED": "false",
"OTEL_TRACE_RATIO": "1",
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
"COST_EXPERIMENT": "false"
}
},
{
"name": "/mtusi2026lab6-api-2",
"settings": {
"OTEL_SDK_DISABLED": "false",
"OTEL_TRACE_RATIO": "1",
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
"COST_EXPERIMENT": "false"
}
},
{
"name": "/mtusi2026lab6-relay-1",
"settings": {
"OTEL_SDK_DISABLED": "false",
"OTEL_TRACE_RATIO": "1",
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
"COST_EXPERIMENT": null
}
},
{
"name": "/mtusi2026lab6-worker-1",
"settings": {
"OTEL_SDK_DISABLED": "false",
"OTEL_TRACE_RATIO": "1",
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
"COST_EXPERIMENT": null
}
}
],
"collector_command": [
"--config=/etc/otelcol/collector.yaml"
],
"result": "PASS"
}
@@ -0,0 +1,6 @@
date: 2026-11-06T23:44:50Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/python -c 'import sqlalchemy,alembic; from sqlalchemy import select,literal; from sqlalchemy.dialects import postgresql; print("SQLAlchemy",sqlalchemy.__version__,"Alembic",alembic.__version__); print(select(literal(1)).with_for_update(read=True,key_share=True).compile(dialect=postgresql.dialect()))'
SQLAlchemy 2.0.45 Alembic 1.16.5
SELECT %(param_1)s AS anon_1 FOR KEY SHARE
exit_code: 0 elapsed_seconds: 0.182

Some files were not shown because too many files have changed in this diff Show More