forked from mtusi2026/mtusi2026lab6
Compare commits
25
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f60297b8e2 | ||
|
|
1e00d861a8 | ||
|
|
6ca1c75499 | ||
|
|
bc1d7aa771 | ||
|
|
97feb04b3f | ||
|
|
237a7c93e5 | ||
|
|
ebafbc3099 | ||
|
|
bc0b162082 | ||
|
|
202b985dea | ||
|
|
20e7759f87 | ||
|
|
0fc02496fe | ||
|
|
cd9d91550a | ||
|
|
447a6507f9 | ||
|
|
44d81fd80e | ||
|
|
1a171d0d0f | ||
|
|
2903d9dc73 | ||
|
|
95511002b1 | ||
|
|
12db77133f | ||
|
|
dc9256baf9 | ||
|
|
a8836ce8b5 | ||
|
|
e2ec3fe9af | ||
|
|
e60b4eac98 | ||
|
|
a41d6d39e8 | ||
|
|
5ae3d77152 | ||
|
|
a0d3947710 |
@@ -0,0 +1,181 @@
|
||||
---
|
||||
name: "OPSX: Apply"
|
||||
description: "Implement tasks from an OpenSpec change (Experimental)"
|
||||
allowed-tools: Bash(openspec:*)
|
||||
category: "Workflow"
|
||||
tags: ["workflow", "artifacts", "experimental"]
|
||||
---
|
||||
|
||||
Implement tasks from an OpenSpec change.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name (e.g., `/opsx:apply add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:apply <other>`).
|
||||
|
||||
2. **Check status to understand the schema**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used (e.g., "spec-driven")
|
||||
- `planningHome`, `changeRoot`, and `actionContext`: planning scope and edit constraints
|
||||
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
|
||||
|
||||
3. **Get apply instructions**
|
||||
|
||||
```bash
|
||||
openspec instructions apply --change "<name>" --json
|
||||
```
|
||||
|
||||
This returns:
|
||||
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
|
||||
- Progress (total, complete, remaining)
|
||||
- Task list with status
|
||||
- Dynamic instruction based on current state
|
||||
- Optional `context`: current required project instruction input from the selected root
|
||||
- Optional `operationGuidance`: current advisory guidance for apply
|
||||
|
||||
**Handle states:**
|
||||
- If `state: "blocked"` (missing artifacts): show message, suggest using `/opsx:continue` (if it is not installed, run `openspec status --change "<name>" --json` to see the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` for how to create it)
|
||||
- If `state: "all_done"`: congratulate, suggest archive
|
||||
- Otherwise: proceed to implementation
|
||||
|
||||
Treat `context` as a required prompt-level input. Read and consider it, and
|
||||
apply relevant project facts, conventions, and constraints while implementing.
|
||||
Treat `operationGuidance` as optional additive advice. Read and consider every
|
||||
entry, and follow entries that are applicable and compatible with the built-in
|
||||
workflow.
|
||||
|
||||
Keep both fields separate from CLI-returned state, missing artifacts, tasks,
|
||||
progress, `contextFiles`, and the built-in `instruction`. They are not
|
||||
evidence of task completion, do not replace the built-in instruction, and do
|
||||
not permit bypassing a blocked state. If context conflicts with the built-in
|
||||
instruction, an explicit user choice, or a CLI-controlled value, report the
|
||||
conflict and preserve the controlling value. If guidance is inapplicable or
|
||||
conflicts with those controlling inputs, do not follow it and explain why.
|
||||
These are prompt-level behavior contracts, not enforceable checks.
|
||||
|
||||
4. **Read context files**
|
||||
|
||||
Read every file path listed under `contextFiles` from the apply instructions output.
|
||||
The files depend on the schema being used:
|
||||
- **spec-driven**: proposal, specs, design, tasks
|
||||
- Other schemas: follow the contextFiles from CLI output
|
||||
|
||||
Do not copy `context` or `operationGuidance` verbatim into implementation
|
||||
files or planning artifacts unless the user separately asks for that content.
|
||||
|
||||
5. **Show current progress**
|
||||
|
||||
Display:
|
||||
- Schema being used
|
||||
- Progress: "N/M tasks complete"
|
||||
- Remaining tasks overview
|
||||
- Dynamic instruction from CLI
|
||||
|
||||
6. **Implement tasks (loop until done or blocked)**
|
||||
|
||||
For each pending task:
|
||||
- Show which task is being worked on
|
||||
- Make the code changes required
|
||||
- Keep changes minimal and focused
|
||||
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
|
||||
- Continue to next task
|
||||
|
||||
**Pause if:**
|
||||
- Task is unclear → ask for clarification
|
||||
- Implementation reveals a design issue → suggest updating artifacts
|
||||
- Error or blocker encountered → report and wait for guidance
|
||||
- User interrupts
|
||||
|
||||
7. **On completion or pause, show status**
|
||||
|
||||
Display:
|
||||
- Tasks completed this session
|
||||
- Overall progress: "N/M tasks complete"
|
||||
- If all done: suggest archive
|
||||
- If paused: explain why and wait for guidance
|
||||
|
||||
**Output During Implementation**
|
||||
|
||||
```
|
||||
## Implementing: <change-name> (schema: <schema-name>)
|
||||
|
||||
Working on task 3/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
|
||||
Working on task 4/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
```
|
||||
|
||||
**Output On Completion**
|
||||
|
||||
```
|
||||
## Implementation Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 7/7 tasks complete ✓
|
||||
|
||||
### Completed This Session
|
||||
- [x] Task 1
|
||||
- [x] Task 2
|
||||
...
|
||||
|
||||
All tasks complete! You can archive this change with `/opsx:archive`.
|
||||
```
|
||||
|
||||
**Output On Pause (Issue Encountered)**
|
||||
|
||||
```
|
||||
## Implementation Paused
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 4/7 tasks complete
|
||||
|
||||
### Issue Encountered
|
||||
<description of the issue>
|
||||
|
||||
**Options:**
|
||||
1. <option 1>
|
||||
2. <option 2>
|
||||
3. Other approach
|
||||
|
||||
What would you like to do?
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Keep going through tasks until done or blocked
|
||||
- Always read context files before starting (from the apply instructions output)
|
||||
- If task is ambiguous, pause and ask before implementing
|
||||
- If implementation reveals issues, pause and suggest artifact updates
|
||||
- Keep code changes minimal and scoped to each task
|
||||
- Update task checkbox immediately after completing each task
|
||||
- Pause on errors, blockers, or unclear requirements - don't guess
|
||||
- Use contextFiles from CLI output, don't assume specific file names
|
||||
- Do not use context or operation guidance as proof that a task is complete
|
||||
- Apply relevant project context; report conflicts with controlling workflow inputs
|
||||
- Consider every guidance entry; explain any inapplicable or conflicting advice
|
||||
- Do not copy runtime context or operation guidance into implementation files or planning artifacts
|
||||
- Preserve CLI-controlled blocked/ready/all-done behavior and completion criteria
|
||||
|
||||
**Fluid Workflow Integration**
|
||||
|
||||
This skill supports the "actions on a change" model:
|
||||
|
||||
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
|
||||
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
|
||||
@@ -0,0 +1,225 @@
|
||||
---
|
||||
name: "OPSX: Archive"
|
||||
description: "Archive a completed change in the experimental workflow"
|
||||
allowed-tools: Bash(openspec:*)
|
||||
category: "Workflow"
|
||||
tags: ["workflow", "archive", "experimental"]
|
||||
---
|
||||
|
||||
Archive a completed change in the experimental workflow.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
|
||||
|
||||
**Input**: Optionally specify a change name after `/opsx:archive` (e.g., `/opsx:archive add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
When prompting, show only active changes (not already archived).
|
||||
Include the schema used for each change if available.
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:archive <other>`).
|
||||
|
||||
**Load current archive inputs before the existing archive checks:**
|
||||
|
||||
After resolving the selected change and planning root, run:
|
||||
```bash
|
||||
openspec instructions archive --change "<name>" --json
|
||||
```
|
||||
Keep the same selected-root flags on this command. This lookup is advisory and
|
||||
optional: it only supplies extra prompt inputs, so it must never block archiving.
|
||||
If it exits non-zero or returns invalid JSON — for example on an older CLI that
|
||||
does not support this command yet — continue the archive workflow with no
|
||||
context and no operation guidance. Do not report an error and do not stop.
|
||||
|
||||
A successful response may omit both optional fields. Treat `context` as a
|
||||
required prompt-level input: read and consider it, and apply relevant project
|
||||
facts, conventions, and constraints. Treat `operationGuidance` as optional
|
||||
additive advice: read and consider every entry, and follow entries that are
|
||||
applicable and compatible with the built-in archive workflow.
|
||||
|
||||
Keep both fields separate from built-in steps, explicit user choices, resolved
|
||||
paths, CLI checks, and command contracts. If context conflicts with one of those
|
||||
controlling inputs, report the conflict and preserve the controlling value. If
|
||||
guidance is inapplicable or conflicts with a controlling input, do not follow it
|
||||
and explain why. Do not infer replacement paths, skipped prompts, or flags from
|
||||
either field, and do not copy their text verbatim into specs, change artifacts,
|
||||
or archive summaries unless the user separately asks for it. These are
|
||||
prompt-level behavior contracts, not enforceable checks.
|
||||
|
||||
2. **Check artifact completion status**
|
||||
|
||||
Run `openspec status --change "<name>" --json` to check artifact completion.
|
||||
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context
|
||||
- `artifacts`: List of artifacts with their status (`done`, `skipped`, or other)
|
||||
|
||||
**If any artifacts are neither `done` nor `skipped`** (skipped artifacts satisfy the requirement - the change declares skip_specs):
|
||||
- Display warning listing incomplete artifacts
|
||||
- Prompt user for confirmation to continue
|
||||
- Proceed if user confirms
|
||||
|
||||
3. **Check task completion status**
|
||||
|
||||
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
|
||||
|
||||
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
|
||||
|
||||
**If incomplete tasks found:**
|
||||
- Display warning showing count of incomplete tasks
|
||||
- Prompt user for confirmation to continue
|
||||
- Proceed if user confirms
|
||||
|
||||
**If no tasks file exists:** Proceed without task-related warning.
|
||||
|
||||
4. **Assess delta spec sync state**
|
||||
|
||||
Use `artifactPaths.specs.existingOutputPaths` from status JSON as the only
|
||||
delta-spec source. If the `specs` entry is missing or
|
||||
`existingOutputPaths` is empty, proceed without a sync prompt and do not infer
|
||||
delta specs from other artifacts.
|
||||
|
||||
**If delta specs exist:**
|
||||
- Compare each delta spec with its corresponding main spec at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (use the store-aware `planningHome.root` from step 2, not a hardcoded repo path)
|
||||
- Determine what changes would be applied (adds, modifications, removals, renames)
|
||||
- Show a combined summary before prompting
|
||||
|
||||
**Prompt options:**
|
||||
- If changes needed: "Sync now (recommended)", "Archive without syncing"
|
||||
- If already synced: "Archive now", "Sync anyway", "Cancel"
|
||||
|
||||
Route on the answer:
|
||||
- "Cancel" — stop, do not archive
|
||||
- "Archive without syncing" or "Archive now" — proceed to archive
|
||||
- "Sync now" or "Sync anyway" — sync, then verify (below)
|
||||
- Anything else — ask again rather than archiving
|
||||
|
||||
Before a selected sync writes any main spec, run
|
||||
`openspec instructions specs --change "<name>" --json` once with the same
|
||||
selected-root flags. Require a zero exit status and valid artifact-instruction
|
||||
JSON. If the lookup fails or returns invalid JSON, report the error and stop
|
||||
before writing any main spec or moving the change. A valid response with omitted
|
||||
`rules` is the no-rules case. Apply returned `rules` only to the content and
|
||||
form of main specs produced by this merge; do not use them as archive guidance,
|
||||
change CLI behavior, or copy the rule text into any output file.
|
||||
|
||||
Then run the `/opsx:sync` workflow inline (agent-driven intelligent merge) for change '<name>', passing the delta spec analysis and the fetched specs-rule snapshot from above, and wait for it to finish. The inline sync must reuse that snapshot without fetching `specs` instructions again. Do not delegate it to a background task — step 5 would move `changeRoot` out from under a sync that is still reading it, leaving the change archived and the main specs never updated. If your agent can only run it by delegation, delegate synchronously and wait for the result.
|
||||
|
||||
Then re-run the comparison from the top of this step against every capability that has a delta spec in `artifactPaths.specs.existingOutputPaths` — not only the ones the sync reports it touched. A successful sync leaves nothing left to apply, so each capability must now read as already synced:
|
||||
- ADDED requirements present
|
||||
- MODIFIED requirements carrying the scenario and description changes named in the delta, with their other scenarios intact
|
||||
- REMOVED requirements gone — and where this sync retired a capability (removed its last requirement, leaving `## Requirements` empty), its main spec deleted rather than left empty; a spec the sync deliberately kept and reported is also a match
|
||||
- RENAMED requirements present under the new name and absent under the old one
|
||||
|
||||
If the sync failed, or any capability does not match, report what differs and stop — do not archive. Nothing has moved and `changeRoot` is intact, so the user can fix the mismatch or re-run the sync and start the archive again.
|
||||
|
||||
5. **Perform the archive**
|
||||
|
||||
Create an `archive` directory under `planningHome.changesDir` if it doesn't exist:
|
||||
```bash
|
||||
mkdir -p "<planningHome.changesDir>/archive"
|
||||
```
|
||||
|
||||
Generate the target name: use the change name as-is when it already starts with a `YYYY-MM-DD-` prefix; otherwise prepend the current date as `YYYY-MM-DD-<change-name>`. Never stack a second date (same rule as `openspec archive`).
|
||||
|
||||
**Check if target already exists:**
|
||||
- If yes: Fail with error, suggest renaming existing archive or using different date
|
||||
- If no: Move `changeRoot` to the archive directory
|
||||
|
||||
```bash
|
||||
mv "<changeRoot>" "<planningHome.changesDir>/archive/<target-name>"
|
||||
```
|
||||
|
||||
6. **Display summary**
|
||||
|
||||
Show archive completion summary including:
|
||||
- Change name
|
||||
- Schema that was used
|
||||
- Archive location
|
||||
- Spec sync status (synced / sync skipped / no delta specs)
|
||||
- Note about any warnings (incomplete artifacts/tasks)
|
||||
|
||||
**Output On Success**
|
||||
|
||||
```markdown
|
||||
## Archive Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
|
||||
**Specs:** ✓ Synced to main specs
|
||||
|
||||
All artifacts complete. All tasks complete.
|
||||
```
|
||||
|
||||
**Output On Success (No Delta Specs)**
|
||||
|
||||
```markdown
|
||||
## Archive Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
|
||||
**Specs:** No delta specs
|
||||
|
||||
All artifacts complete. All tasks complete.
|
||||
```
|
||||
|
||||
**Output On Success With Warnings**
|
||||
|
||||
```markdown
|
||||
## Archive Complete (with warnings)
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
|
||||
**Specs:** Sync skipped (user chose to skip)
|
||||
|
||||
**Warnings:**
|
||||
- Archived with 2 incomplete artifacts
|
||||
- Archived with 3 incomplete tasks
|
||||
- Delta spec sync was skipped (user chose to skip)
|
||||
|
||||
Review the archive if this was not intentional.
|
||||
```
|
||||
|
||||
**Output On Error (Archive Exists)**
|
||||
|
||||
```markdown
|
||||
## Archive Failed
|
||||
|
||||
**Change:** <change-name>
|
||||
**Target:** the archive path derived from `planningHome.changesDir`/<target-name>/
|
||||
|
||||
Target archive directory already exists.
|
||||
|
||||
**Options:**
|
||||
1. Rename the existing archive
|
||||
2. Delete the existing archive if it's a duplicate
|
||||
3. Wait until a different date to archive
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Announce the selected change; prompt for selection when it is ambiguous
|
||||
- Use artifact graph (openspec status --json) for completion checking
|
||||
- Don't block archive on warnings - just inform and confirm
|
||||
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
|
||||
- Show clear summary of what happened
|
||||
- If sync is requested, run the `/opsx:sync` workflow inline (agent-driven)
|
||||
- Never archive while a spec sync is still in flight — run the sync inline and verify the main specs before moving `changeRoot`
|
||||
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
|
||||
- Apply relevant runtime context and report conflicts; operation guidance remains advisory
|
||||
- Consider every guidance entry and explain any inapplicable or conflicting advice
|
||||
- Existing CLI checks, resolved paths, prompts, and command contracts are unchanged
|
||||
- Artifact rules constrain only the specs being written and are never operation guidance
|
||||
- Never copy runtime context, operation guidance, or artifact-rule text verbatim into output files
|
||||
@@ -0,0 +1,193 @@
|
||||
---
|
||||
name: "OPSX: Explore"
|
||||
description: "Enter explore mode - think through ideas, investigate problems, clarify requirements"
|
||||
allowed-tools: Bash(openspec:*)
|
||||
category: "Workflow"
|
||||
tags: ["workflow", "explore", "experimental", "thinking"]
|
||||
---
|
||||
|
||||
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
|
||||
|
||||
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing. For a new change, scaffold it first as described below.
|
||||
|
||||
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: The argument after `/opsx:explore` is whatever the user wants to think about. Could be:
|
||||
- A vague idea: "real-time collaboration"
|
||||
- A specific problem: "the auth system is getting unwieldy"
|
||||
- A change name: "add-dark-mode" (to explore in context of that change)
|
||||
- A comparison: "postgres vs sqlite for this"
|
||||
- Nothing (just enter explore mode)
|
||||
|
||||
---
|
||||
|
||||
## The Stance
|
||||
|
||||
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
|
||||
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
|
||||
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
|
||||
- **Adaptive** - Follow interesting threads, pivot when new information emerges
|
||||
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
|
||||
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
|
||||
|
||||
---
|
||||
|
||||
## What You Might Do
|
||||
|
||||
Depending on what the user brings, you might:
|
||||
|
||||
**Explore the problem space**
|
||||
- Ask clarifying questions that emerge from what they said
|
||||
- Challenge assumptions
|
||||
- Reframe the problem
|
||||
- Find analogies
|
||||
|
||||
**Investigate the codebase**
|
||||
- Map existing architecture relevant to the discussion
|
||||
- Find integration points
|
||||
- Identify patterns already in use
|
||||
- Surface hidden complexity
|
||||
|
||||
**Compare options**
|
||||
- Brainstorm multiple approaches
|
||||
- Build comparison tables
|
||||
- Sketch tradeoffs
|
||||
- Recommend a path (if asked)
|
||||
|
||||
**Visualize**
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Use ASCII diagrams liberally │
|
||||
├─────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌────────┐ ┌────────┐ │
|
||||
│ │ State │────────▶│ State │ │
|
||||
│ │ A │ │ B │ │
|
||||
│ └────────┘ └────────┘ │
|
||||
│ │
|
||||
│ System diagrams, state machines, │
|
||||
│ data flows, architecture sketches, │
|
||||
│ dependency graphs, comparison tables │
|
||||
│ │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Surface risks and unknowns**
|
||||
- Identify what could go wrong
|
||||
- Find gaps in understanding
|
||||
- Suggest spikes or investigations
|
||||
|
||||
---
|
||||
|
||||
## OpenSpec Awareness
|
||||
|
||||
You have full context of the OpenSpec system. Use it naturally, don't force it.
|
||||
|
||||
### Check for context
|
||||
|
||||
At the start, quickly check what exists:
|
||||
```bash
|
||||
openspec list --json
|
||||
```
|
||||
|
||||
This tells you:
|
||||
- If there are active changes
|
||||
- Their names, schemas, and status
|
||||
- What the user might be working on
|
||||
|
||||
Then read the project's own context from the resolved root - `<root.path>/openspec/config.yaml` (or `config.yml`). Use the `root.path` returned above, and skip this if neither file exists:
|
||||
- `context`: project background - tech stack, conventions, constraints
|
||||
- `rules`: keyed by artifact id - the entries for an artifact apply only when you write that artifact
|
||||
|
||||
Ground your thinking in these. They are constraints for you to follow, not content to reproduce: do NOT copy them into the conversation or into any artifact you create.
|
||||
|
||||
If the user mentioned a specific change name, read its artifacts for context.
|
||||
|
||||
### When no change exists
|
||||
|
||||
Think freely. When insights crystallize, you might offer:
|
||||
|
||||
- "This feels solid enough to start a change. Want me to create a proposal?"
|
||||
- Or keep exploring - no pressure to formalize
|
||||
|
||||
If the user asks you to capture the exploration as a new change, transition seamlessly into the requested capture:
|
||||
|
||||
1. Run `openspec new change "<name>"` (with `--store <id>` when applicable) before creating any artifacts. Never create a new change directory under `openspec/changes/` by hand; the CLI scaffold creates required metadata such as `.openspec.yaml`. Keep the selected `--store <id>` on every applicable follow-up `status` and `instructions` command.
|
||||
2. Run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store), then process the requested artifacts in dependency order. For each requested artifact that is `ready`, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store). Before creating a requested artifact, evaluate any condition in its own `instruction` against the explored change; record a deliberate skip instead when the condition does not apply. If a requested artifact is blocked by a direct prerequisite the user did not request, run `openspec instructions "<prerequisite-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) for that prerequisite whether it is `ready` or `blocked`. If its own `instruction` states a condition, evaluate that condition against the explored change and record a deliberate skip only when the condition does not apply. If the condition applies, or the prerequisite is not conditional, treat it as a normal prerequisite and ask before expanding the capture. Do not create an unrequested prerequisite unless the user approves.
|
||||
3. Follow the returned `template` and `instruction` fields. Read completed dependency files listed in `dependencies`, and apply `context` and `rules` as constraints without copying them into the artifact. If the instruction delegates creation to a specific skill or command, invoke it; otherwise write the artifact to `resolvedOutputPath`, using the instruction to choose a concrete path when it is a glob. Verify that the selected concrete output exists.
|
||||
4. After creating each artifact, re-run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) and continue until every requested artifact is `done`, `skipped`, or was deliberately skipped because its own `instruction` stated a condition that did not apply. Tell the user about a deliberate conditional skip, remember it, and do not reconsider it. Dependencies are enablers, not gates: if a requested artifact is still `blocked` only because you deliberately skipped a conditional prerequisite, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) despite the blocked status, then create it using step 3 only when those recorded conditional skips are its sole missing dependencies. If a requested artifact is blocked by a prerequisite the user did not ask to capture and cannot be conditionally skipped, explain that dependency and ask before expanding the capture.
|
||||
|
||||
Capture the artifact(s) the user requested without asking them to invoke another workflow command. If they asked only to start a change, stop after scaffolding and show its status.
|
||||
|
||||
### When a change exists
|
||||
|
||||
If the user mentions a change or you detect one is relevant:
|
||||
|
||||
1. **Resolve and read existing artifacts for context**
|
||||
- Run `openspec status --change "<name>" --json`.
|
||||
- Use `changeRoot`, `artifactPaths`, and `actionContext` from the status JSON.
|
||||
- Read existing files from `artifactPaths.<artifact>.existingOutputPaths`.
|
||||
|
||||
2. **Reference them naturally in conversation**
|
||||
- "Your design mentions using Redis, but we just realized SQLite fits better..."
|
||||
- "The proposal scopes this to premium users, but we're now thinking everyone..."
|
||||
|
||||
3. **Offer to capture when decisions are made**
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
|
||||
|
||||
| Insight Type | Where to Capture |
|
||||
|----------------------------|-------------------------------------|
|
||||
| New requirement discovered | `specs/<capability-path>/spec.md` |
|
||||
| Requirement changed | `specs/<capability-path>/spec.md` |
|
||||
| Design decision made | `design.md` |
|
||||
| Scope changed | `proposal.md` |
|
||||
| New work identified | `tasks.md` |
|
||||
| Assumption invalidated | Relevant artifact |
|
||||
|
||||
Example offers:
|
||||
- "That's a design decision. Capture it in design.md?"
|
||||
- "This is a new requirement. Add it to specs?"
|
||||
- "This changes scope. Update the proposal?"
|
||||
|
||||
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
|
||||
|
||||
---
|
||||
|
||||
## What You Don't Have To Do
|
||||
|
||||
- Follow a script
|
||||
- Ask the same questions every time
|
||||
- Produce a specific artifact
|
||||
- Reach a conclusion
|
||||
- Stay on topic if a tangent is valuable
|
||||
- Be brief (this is thinking time)
|
||||
|
||||
---
|
||||
|
||||
## Ending Discovery
|
||||
|
||||
There's no required ending. Discovery might:
|
||||
|
||||
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
|
||||
- **Result in artifact updates**: "Updated design.md with these decisions"
|
||||
- **Just provide clarity**: User has what they need, moves on
|
||||
- **Continue later**: "We can pick this up anytime"
|
||||
|
||||
When things crystallize, you might offer a summary - but it's optional. Sometimes the thinking IS the value.
|
||||
|
||||
---
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
|
||||
- **Don't fake understanding** - If something is unclear, dig deeper
|
||||
- **Don't rush** - Discovery is thinking time, not task time
|
||||
- **Don't force structure** - Let patterns emerge naturally
|
||||
- **Don't auto-capture** - Offer to save insights, don't just do it
|
||||
- **Don't manually scaffold changes** - Never create a new change directory under `openspec/changes/` by hand. Always use `openspec new change "<name>"` (with `--store <id>` when applicable) so required metadata such as `.openspec.yaml` is created before writing artifacts.
|
||||
- **Do visualize** - A good diagram is worth many paragraphs
|
||||
- **Do explore the codebase** - Ground discussions in reality
|
||||
- **Do question assumptions** - Including the user's and your own
|
||||
@@ -0,0 +1,145 @@
|
||||
---
|
||||
name: "OPSX: Propose"
|
||||
description: "Propose a new change - create it and generate all artifacts in one step"
|
||||
allowed-tools: Bash(openspec:*)
|
||||
category: "Workflow"
|
||||
tags: ["workflow", "artifacts", "experimental"]
|
||||
---
|
||||
|
||||
Propose a new change - create the change and generate all artifacts in one step.
|
||||
|
||||
**Planning boundary**: This workflow creates planning artifacts only. The user request that selected or triggered this workflow authorizes planning only, even if it asks to build or fix something. Do not edit project code. After the planning artifacts are complete, stop. Do not start implementation in the same response, even if the initial request asks for it. Wait for a new user request after the artifacts are presented; then start the apply workflow.
|
||||
|
||||
I'll create a change with the artifacts your schema defines. With the default spec-driven schema that is:
|
||||
- proposal.md (what & why)
|
||||
- `specs/<capability-path>/spec.md` (what the system must do - a delta, not the main spec)
|
||||
- design.md (how)
|
||||
- tasks.md (implementation steps)
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
|
||||
|
||||
When the user is ready to implement, they must start the apply workflow explicitly.
|
||||
|
||||
---
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: The argument after `/opsx:propose` is the change name (kebab-case), OR a description of what the user wants to build.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Understand the request and clarify material ambiguity**
|
||||
|
||||
If no input is provided, ask the user (open-ended, no preset options):
|
||||
> "What change do you want to work on? Describe what you want to build or fix."
|
||||
|
||||
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
|
||||
|
||||
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
|
||||
|
||||
If the request contains ambiguity that would materially affect scope, externally observable behavior, compatibility, or acceptance criteria, ask the user before creating the change. For minor details, make a reasonable assumption and record it in the planning artifacts.
|
||||
|
||||
2. **Determine the workflow schema**
|
||||
|
||||
Use the configured default schema unless the user explicitly requests a different workflow.
|
||||
|
||||
**Use a different schema only if the user:**
|
||||
- Explicitly requests a specific schema by name → use `--schema <schema-name>`
|
||||
- Asks to "show workflows" or asks "what workflows" exist → resolve the authoritative root by running `openspec context --json` from the current working directory. If the user explicitly selected a registered store, use `openspec context --json --store "<store-id>"`. Then run `openspec schemas --json` with its working directory set to the returned `root.path` and let them choose. This preserves roots selected by a local `store:` pointer or the global `defaultStore`; `schemas` does not accept `--store`. If context reports only `no_openspec_root`, run `openspec schemas --json` from the current working directory instead. Do not use this fallback for invalid or unavailable stores.
|
||||
|
||||
Otherwise, omit `--schema` to preserve the configured default.
|
||||
|
||||
3. **Create the change directory**
|
||||
|
||||
Choose one schema form below. If a registered store is selected, append `--store "<store-id>"` to that command and each later OpenSpec command shown below that accepts `--store`.
|
||||
|
||||
Using the configured default:
|
||||
```bash
|
||||
openspec new change "<name>"
|
||||
```
|
||||
|
||||
Using an explicitly requested schema:
|
||||
```bash
|
||||
openspec new change "<name>" --schema "<schema-name>"
|
||||
```
|
||||
This creates a scaffolded change in the planning home resolved by the CLI with `.openspec.yaml`.
|
||||
|
||||
4. **Get the artifact build order**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to get:
|
||||
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
|
||||
- `artifacts`: list of all artifacts, each with its `status` and its `requires` edges (the artifact IDs it directly depends on)
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
|
||||
|
||||
5. **Create every artifact in the required set**
|
||||
|
||||
Use a todo list to track progress through the artifacts.
|
||||
|
||||
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
|
||||
|
||||
a. **For each artifact that is `ready` (dependencies satisfied)**:
|
||||
- Get instructions:
|
||||
```bash
|
||||
openspec instructions <artifact-id> --change "<name>" --json
|
||||
```
|
||||
- The instructions JSON includes:
|
||||
- `context`: Project background (constraints for you - do NOT include in output)
|
||||
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
|
||||
- `template`: The structure to use for your output file
|
||||
- `instruction`: Schema-specific guidance for this artifact type
|
||||
- `skipped`/`warning`: present when the change declares skip_specs and this artifact must NOT be created - stop and pick another artifact
|
||||
- `resolvedOutputPath`: Resolved path or pattern to write the artifact
|
||||
- `dependencies`: Completed artifacts to read for context
|
||||
- Read any completed dependency files for context - always re-read them from disk, even if you saw them earlier in the conversation (the user may have edited them)
|
||||
- If the `instruction` field delegates creation to a specific skill or command, invoke it to produce the artifact instead of writing the file yourself, then verify the artifact file exists at `resolvedOutputPath`
|
||||
- Otherwise create the artifact file using `template` as the structure and write it to `resolvedOutputPath`. If `resolvedOutputPath` is a glob, follow `instruction` to choose the concrete file path
|
||||
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
|
||||
- Show brief progress: "Created <artifact-id>"
|
||||
|
||||
b. **Continue until every artifact in the required set exists (not just `apply.requires`)**
|
||||
- After creating each artifact, re-run `openspec status --change "<name>" --json`
|
||||
- The required set is `applyRequires` plus every artifact reachable from those by following the `requires` edges in `status --json` - walk them transitively (spec-driven closes over proposal, specs, design, tasks). Leave artifacts outside that set alone
|
||||
- `status` is file-existence only, so an `applyRequires` artifact reading `done` does NOT mean its dependencies exist - writing `tasks.md` early marks `tasks` done while `specs` was never written. Use each artifact's `requires` edges, not its `status`, to build the required set: a `done` artifact still lists what it depends on
|
||||
- An artifact already reading `status: "skipped"` is satisfied: the change declares `skip_specs` in `.openspec.yaml`, so its files must NOT exist. Never try to create one
|
||||
- Create every artifact in the required set that is missing, then re-check - creating one can unblock others
|
||||
- Skip one only when `status` already reports it `skipped`, or when its own `instruction` says it is conditional: run `openspec instructions <artifact-id> --change "<name>" --json` and skip only if its `instruction` field marks it optional (e.g. "create only if..."). Spec-driven's `design.md` qualifies; `specs` qualifies only via the `skipped` status above, never by your own judgment. Tell the user, and do not reconsider it
|
||||
- Dependencies are enablers, not gates: if a required artifact is still `blocked` only because you skipped a conditional dependency, write it anyway
|
||||
- Stop when every artifact in the required set is `done`, `skipped`, or was deliberately skipped
|
||||
|
||||
c. **If an artifact requires user input** (unclear context):
|
||||
- Ask the user to clarify
|
||||
- Then continue with creation
|
||||
|
||||
6. **Show final status**
|
||||
```bash
|
||||
openspec status --change "<name>"
|
||||
```
|
||||
|
||||
**Output**
|
||||
|
||||
After completing all artifacts, summarize:
|
||||
- Change name and location
|
||||
- List of artifacts created with brief descriptions, plus any conditional artifact you skipped and why
|
||||
- What's ready: "All artifacts needed for implementation are ready."
|
||||
- Prompt: "The artifacts are ready for review. When you are ready, run `/opsx:apply`."
|
||||
|
||||
**Artifact Creation Guidelines**
|
||||
|
||||
- Follow the `instruction` field from `openspec instructions` for each artifact type - it is the authoritative guidance, even for familiar artifact names
|
||||
- If the `instruction` field directs you to use a specific skill or command to create the artifact, invoke it instead of writing the artifact directly
|
||||
- The schema defines what each artifact should contain - follow it
|
||||
- Read dependency artifacts for context before creating new ones
|
||||
- Use `template` as the structure for your output file - fill in its sections
|
||||
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
|
||||
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
|
||||
- These guide what you write, but should never appear in the output
|
||||
|
||||
**Guardrails**
|
||||
- The request that invoked this workflow authorizes planning only. Any implementation or apply instruction in that request does not carry forward. Do NOT implement the change, start the apply workflow, or edit project code during this workflow. After presenting the artifacts, stop and wait for a new user request to start the apply workflow
|
||||
- Create every artifact the apply phase transitively depends on, not just the ids listed in `apply.requires`
|
||||
- Always read dependency artifacts before creating a new one - re-read from disk, not from conversation memory (files may have changed since you last saw them)
|
||||
- Ask about ambiguities that would materially change scope, externally observable behavior, compatibility, or acceptance criteria; for minor details, make reasonable assumptions and record them
|
||||
- If a change with that name already exists, ask if user wants to continue it or create a new one
|
||||
- Verify each artifact file exists after writing before proceeding to next
|
||||
@@ -0,0 +1,258 @@
|
||||
---
|
||||
name: "OPSX: Sync"
|
||||
description: "Sync delta specs from a change to main specs"
|
||||
allowed-tools: Bash(openspec:*)
|
||||
category: "Workflow"
|
||||
tags: ["workflow", "specs", "experimental"]
|
||||
---
|
||||
|
||||
Sync delta specs from a change to main specs.
|
||||
|
||||
This is an **agent-driven** operation - you will read delta specs and directly edit main specs to apply the changes. This allows intelligent merging (e.g., adding a scenario without copying the entire requirement).
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
|
||||
|
||||
**Input**: Optionally specify a change name after `/opsx:sync` (e.g., `/opsx:sync add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
When prompting, show changes that have delta specs (under `specs/` directory).
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:sync <other>`).
|
||||
|
||||
2. **Resolve change context**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
|
||||
The JSON includes `planningHome.root`. Main specs live under `<planningHome.root>/openspec/specs/` — use that (store-aware) root for every main-spec path below, not a hardcoded repo path. When a store is selected it points at the store, not the current repository.
|
||||
|
||||
3. **Find delta specs**
|
||||
|
||||
Use `artifactPaths.specs.existingOutputPaths` from the status JSON as the
|
||||
only source of delta spec paths. If the `specs` entry is missing or
|
||||
`existingOutputPaths` is empty, report that there are no delta specs to sync,
|
||||
do not infer them from other artifacts, and stop without requesting artifact
|
||||
instructions or writing a main spec.
|
||||
|
||||
Sync every path in `existingOutputPaths` unless the caller narrowed the set.
|
||||
A caller narrows it by naming an explicit list of complete entries from
|
||||
`existingOutputPaths` — copy those absolute values verbatim. Archive does
|
||||
this inline, and a user can too (for example, by selecting the entry ending
|
||||
in `/specs/billing/invoices/spec.md`).
|
||||
Then sync only the named paths and leave the remaining delta specs untouched:
|
||||
bulk archive excludes a delta whose implementation it could not find, and
|
||||
syncing it anyway would write a main spec the caller deliberately withheld.
|
||||
Carry that narrowed selection through step 4; never widen it back to the full
|
||||
list. If a named path is not in `existingOutputPaths`, do not sync it —
|
||||
report it and stop, rather than dropping it silently. If the named list is
|
||||
empty, report that there is nothing to sync and stop without writing a main
|
||||
spec.
|
||||
|
||||
Each delta spec file contains sections like:
|
||||
- `## ADDED Requirements` - New requirements to add
|
||||
- `## MODIFIED Requirements` - Changes to existing requirements
|
||||
- `## REMOVED Requirements` - Requirements to remove
|
||||
- `## RENAMED Requirements` - Requirements to rename (FROM:/TO: format)
|
||||
|
||||
If no delta specs found, inform user and stop.
|
||||
|
||||
4. **For each delta spec, apply changes to main specs**
|
||||
|
||||
Before the first main-spec write, obtain one current specs-rule snapshot:
|
||||
- If archive invoked this workflow inline and supplied a valid snapshot from
|
||||
`openspec instructions specs --change "<name>" --json`, reuse it and do not
|
||||
fetch the same instructions again.
|
||||
- Otherwise run that command once now with the same selected-root flags.
|
||||
- If the direct lookup exits non-zero or returns invalid artifact-instruction
|
||||
JSON, report the error and stop before writing any main spec. Do not treat the
|
||||
failure as an absent rule set.
|
||||
- A valid response with omitted `rules` means no artifact rules are configured
|
||||
and the existing semantic merge continues.
|
||||
|
||||
Apply returned `rules` only to the content and form of the main specs produced
|
||||
by this merge. Artifact rules are not operation guidance and cannot change
|
||||
selected roots, delta paths, CLI checks, or workflow steps. Use their text as
|
||||
constraints without copying it verbatim into a main spec or summary.
|
||||
|
||||
For each capability delta spec path selected in step 3 — the full `existingOutputPaths` list, or the narrowed subset when a caller supplied one (these may belong to a selected store, not the repo):
|
||||
|
||||
a. **Read the delta spec** to understand the intended changes
|
||||
|
||||
b. **Read the main spec** at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (may not exist yet)
|
||||
|
||||
c. **Apply changes intelligently**:
|
||||
|
||||
**ADDED Requirements:**
|
||||
- If requirement doesn't exist in main spec → add it
|
||||
- If requirement already exists → update it to match (treat as implicit MODIFIED)
|
||||
|
||||
**MODIFIED Requirements:**
|
||||
- Find the requirement in main spec
|
||||
- Apply the changes - this can be:
|
||||
- Adding new scenarios the main spec does not have yet
|
||||
- Modifying existing scenarios
|
||||
- Changing the requirement description
|
||||
- Preserve scenarios/content not mentioned in the delta
|
||||
|
||||
**REMOVED Requirements:**
|
||||
- Remove the entire requirement block from main spec
|
||||
- Retiring the capability. Delete the whole `spec.md` - and the directory once
|
||||
nothing else is left in it - only when ALL of these hold:
|
||||
1. removing the requirements *this run* left no requirement blocks;
|
||||
2. the rest of the spec is well-formed (it still has a `## Purpose`);
|
||||
3. the main spec was not already empty before this sync - if you removed
|
||||
nothing, change nothing;
|
||||
4. every other nonblank line in the whole file is accounted for as the
|
||||
title, Purpose, Requirements header, or a canonical requirement's
|
||||
statement, scenarios, or fenced examples;
|
||||
5. the change's `.openspec.yaml` declares `retire_capabilities: true`;
|
||||
6. the `spec.md` resolves inside the real specs root (do not follow a
|
||||
capability-directory symlink to delete an external file).
|
||||
If removing the selected requirements would leave no requirement blocks and
|
||||
any retirement condition is not satisfied, do not modify the main spec. Stop
|
||||
the sync for that capability, report the blocking condition, and tell the user
|
||||
how to resolve it. Never write or leave an empty `## Requirements` section.
|
||||
When only the marker is missing, say that too - it is the one thing the user
|
||||
can add to make the retirement go through.
|
||||
- Deleting the file also deletes its `## Purpose`; any other section blocks
|
||||
retirement. Name Purpose when you report the retirement. Include a pasteable
|
||||
`git checkout` only when the spec lived in the caller's checkout;
|
||||
otherwise give checkout-scoped recovery guidance.
|
||||
|
||||
**RENAMED Requirements:**
|
||||
- Find the FROM requirement, rename to TO
|
||||
|
||||
**`## Purpose` in the delta:**
|
||||
- The main spec already has one and it is authoritative - leave it alone
|
||||
(this is what `openspec archive` does; it warns and moves on)
|
||||
|
||||
d. **Create new main spec** if capability doesn't exist yet:
|
||||
- Create `<planningHome.root>/openspec/specs/<capability-path>/spec.md`
|
||||
- Add Purpose section: copy the delta's `## Purpose` body verbatim when it has one
|
||||
(this is what `openspec archive` does); only write a brief TBD placeholder when it does not
|
||||
- Add Requirements section with the ADDED requirements
|
||||
- Follow the **Main Spec Format Reference** below
|
||||
|
||||
5. **Validate updated main specs**
|
||||
|
||||
Run `openspec validate --specs` with the same selected-root flags used earlier.
|
||||
If validation fails, report the problems and do not claim the sync succeeded.
|
||||
|
||||
6. **Show summary**
|
||||
|
||||
After applying all changes, summarize:
|
||||
- Which capabilities were updated
|
||||
- What changes were made (requirements added/modified/removed/renamed)
|
||||
- Any new main spec left with a TBD Purpose placeholder, so it gets written
|
||||
now rather than lingering
|
||||
- Any capability retired, naming the deleted `spec.md`, its Purpose, and
|
||||
either a pasteable `git checkout` or checkout-scoped recovery guidance
|
||||
|
||||
**Delta Spec Format Reference**
|
||||
|
||||
```markdown
|
||||
## Purpose
|
||||
|
||||
Only on a delta that introduces a brand-new capability. Seeds the new main spec.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: New Feature
|
||||
The system SHALL do something new.
|
||||
|
||||
#### Scenario: Basic case
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Existing Feature
|
||||
The system SHALL keep doing the existing thing, now also handling A.
|
||||
|
||||
#### Scenario: Scenario the main spec already has
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
|
||||
#### Scenario: New scenario to add
|
||||
- **WHEN** user does A
|
||||
- **THEN** system does B
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Deprecated Feature
|
||||
|
||||
## RENAMED Requirements
|
||||
|
||||
- FROM: `### Requirement: Old Name`
|
||||
- TO: `### Requirement: New Name`
|
||||
```
|
||||
|
||||
**Main Spec Format Reference**
|
||||
|
||||
Main specs are what the delta merges INTO. They must never contain delta operation headers (`## ADDED/MODIFIED/REMOVED/RENAMED Requirements`) - after syncing, every requirement lives under a single `## Requirements` section:
|
||||
|
||||
```markdown
|
||||
# <capability> Specification
|
||||
|
||||
## Purpose
|
||||
Short description of what this capability does and why it exists.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: New Feature
|
||||
The system SHALL do something new.
|
||||
|
||||
#### Scenario: Basic case
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
```
|
||||
|
||||
**Key Principle: Intelligent Merging**
|
||||
|
||||
Unlike programmatic merging, you merge rather than overwrite:
|
||||
- A MODIFIED block carries the whole requirement - body plus every scenario that survives the change. `openspec validate` and `openspec archive` both reject one that drops a scenario the main spec still has.
|
||||
- Keep anything the delta does not mention, in the main spec's existing order
|
||||
- Use your judgment to merge changes sensibly
|
||||
|
||||
**Output On Success**
|
||||
|
||||
```markdown
|
||||
## Specs Synced: <change-name>
|
||||
|
||||
Updated main specs:
|
||||
|
||||
**<capability-1>**:
|
||||
- Added requirement: "New Feature"
|
||||
- Modified requirement: "Existing Feature" (added 1 scenario)
|
||||
|
||||
**<capability-2>**:
|
||||
- Created new spec file
|
||||
- Added requirement: "Another Feature"
|
||||
|
||||
Main specs are now updated. The change remains active - archive when implementation is complete.
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Read both delta and main specs before making changes
|
||||
- Preserve existing content not mentioned in delta
|
||||
- Never copy a delta file into a main spec as-is - merge its content so the main spec keeps the Main Spec Format Reference structure, with no delta operation headers
|
||||
- If something is unclear, ask for clarification
|
||||
- Show what you're changing as you go
|
||||
- The operation should be idempotent - running twice should give same result
|
||||
- Use only `artifactPaths.specs.existingOutputPaths`; never infer delta specs from unrelated artifacts
|
||||
- Honor a caller-supplied subset of `existingOutputPaths`; never widen it back to the full list
|
||||
- Fetch specs instructions once for direct sync, or reuse the archive-supplied snapshot inline
|
||||
- Stop before every main-spec write on a non-zero or invalid JSON specs-instruction response
|
||||
- Artifact rules constrain only the specs being written and are never copied into output files
|
||||
@@ -0,0 +1,87 @@
|
||||
---
|
||||
name: "OPSX: Update"
|
||||
description: "Update a change - revise existing planning artifacts and keep them coherent (Experimental)"
|
||||
allowed-tools: Bash(openspec:*)
|
||||
category: "Workflow"
|
||||
tags: ["workflow", "artifacts", "experimental"]
|
||||
---
|
||||
|
||||
Revise a change's existing planning artifacts and keep them coherent. Never edit code.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name after `/opsx:update` (e.g., `/opsx:update add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
`/opsx:continue` is an expanded-profile workflow and may not be installed. Before suggesting it anywhere below, verify that it is available. If it is unavailable, `openspec status --change "<name>" --json` shows the next artifact and `openspec instructions "<artifact-id>" --change "<name>" --json` explains how to create it.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes sorted by most recently modified, and ask the user to select one
|
||||
|
||||
When prompting, present the top 3-4 most recently modified changes as options, showing:
|
||||
- Change name
|
||||
- Schema (from `schema` field if present, otherwise "spec-driven")
|
||||
- Status (e.g., "0/5 tasks", "complete", "no tasks")
|
||||
- How recently it was modified (from `lastModified` field)
|
||||
|
||||
Mark the most recently modified change as "(Recommended)" since it's likely what the user wants to update.
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:update <other>`).
|
||||
|
||||
2. **Get the change's artifacts**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to understand current state. The response includes:
|
||||
- `schemaName`: The workflow schema being used (e.g., "spec-driven")
|
||||
- `artifacts`: Array of artifacts with their status ("done", "skipped", "ready", "blocked")
|
||||
- `isPlanningComplete`: Boolean indicating if all planning artifacts are complete. Older CLI versions expose the same value as `isComplete`.
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
|
||||
|
||||
The artifact ids and paths come from the active schema - do NOT assume them, and do NOT branch on hardcoded artifact names. Custom schemas must work unchanged.
|
||||
|
||||
The files to edit are `artifactPaths.<id>.existingOutputPaths` - the concrete files that exist on disk, already glob-expanded for glob artifacts (e.g. `specs/**/*.md`). Do NOT write to `resolvedOutputPath`: for a glob artifact it is still the glob pattern, not a real file.
|
||||
|
||||
3. **Understand the request**
|
||||
- If the user asked for a specific revision ("the design now uses X"), that is the starting edit.
|
||||
- If they only said "update" / "make this coherent", treat it as a coherence review: read the existing artifacts and check them against each other for contradictions, gaps, and duplication.
|
||||
|
||||
4. **Read and reconcile**
|
||||
- Read the artifact(s) the request touches and the change's other existing artifacts.
|
||||
- Apply the requested edit. Then check every other existing artifact against it - in ANY direction: an edit to a later artifact may require revising an earlier one, not only the other way around. Build order is a useful reading order, not a constraint on which artifacts may be revised.
|
||||
- Note everything that is now inconsistent, missing, or contradictory.
|
||||
- Revise only files that already exist (`existingOutputPaths`). Do NOT create artifacts that don't exist yet, and do NOT invent new files under a glob artifact - note them and point the user to `/opsx:continue` to create them.
|
||||
- If the change is already coherent, say so and make no edits.
|
||||
|
||||
5. **Confirm and apply, one artifact at a time**
|
||||
- Show each proposed revision and why. Write only after the user confirms.
|
||||
- If the user rejects a revision, do not write it - leave that artifact unchanged.
|
||||
- When a substantial rewrite is needed, get that artifact's rules and template first:
|
||||
```bash
|
||||
openspec instructions "<artifact-id>" --change "<name>" --json
|
||||
```
|
||||
|
||||
6. **Point to the next step (guidance only - NEVER act on it)**
|
||||
- Artifacts still missing -> suggest `/opsx:continue` to create them.
|
||||
- Change already implemented (tasks checked off / already applied) -> the code may no longer match the revised plan; suggest `/opsx:apply` to carry the delta into code.
|
||||
- Everything done and implemented -> suggest `/opsx:archive`.
|
||||
|
||||
**Output**
|
||||
|
||||
After each invocation, show:
|
||||
- Which artifacts were revised (and which proposed revisions were rejected)
|
||||
- Anything deferred to `/opsx:continue` (not-yet-created artifacts or files)
|
||||
- Where the change stands and the recommended next command
|
||||
|
||||
**Guardrails**
|
||||
- Planning artifacts only - NEVER edit implementation code. If the revised plan implies code changes, stop and point to `/opsx:apply`.
|
||||
- Use the artifact ids and paths reported by `openspec status`; never branch on hardcoded artifact names.
|
||||
- Edit only the concrete files in `existingOutputPaths`; never write to a glob `resolvedOutputPath`.
|
||||
- Do not advance the build frontier: no new artifacts, no new files under glob artifacts - that is `/opsx:continue`'s job.
|
||||
- Confirm every edit with the user before writing.
|
||||
- If the request changes the change's *intent* rather than refining it, first verify whether the expanded-profile `/opsx:new` workflow is available. If it is, recommend starting fresh with `/opsx:new` (the "Update vs. Start Fresh" heuristic). If it is unavailable, ask for a distinct unused change name and recommend `openspec new change "<new-change-name>"` instead.
|
||||
Executable
+45
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="${CLAUDE_PROJECT_DIR:-$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)}"
|
||||
PAYLOAD=$(cat)
|
||||
FILE=$(printf '%s' "$PAYLOAD" | /usr/bin/python3 -c 'import json,sys
|
||||
try:
|
||||
d = json.load(sys.stdin)
|
||||
except Exception:
|
||||
sys.exit(0)
|
||||
t = d.get("tool_input") or {}
|
||||
r = d.get("tool_response") or {}
|
||||
p = r.get("filePath") or t.get("file_path") or ""
|
||||
print(p)
|
||||
' 2>/dev/null)
|
||||
|
||||
case "$FILE" in
|
||||
"") exit 0 ;;
|
||||
*.py) ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
|
||||
case "$FILE" in
|
||||
"$ROOT"/src/*) ;;
|
||||
src/*) FILE="$ROOT/$FILE" ;;
|
||||
*) exit 0 ;;
|
||||
esac
|
||||
|
||||
[ -f "$FILE" ] || exit 0
|
||||
[ -x "$ROOT/.venv/bin/ruff" ] || exit 0
|
||||
|
||||
OUT=$(
|
||||
cd "$ROOT" || exit 0
|
||||
"$ROOT/.venv/bin/ruff" check --fix --quiet "$FILE" 2>&1
|
||||
"$ROOT/.venv/bin/ruff" format --quiet "$FILE" 2>&1
|
||||
if [ -x "$ROOT/.venv/bin/ty" ]; then
|
||||
"$ROOT/.venv/bin/ty" check --quiet "$FILE" 2>&1 | grep -v "^All checks passed" || true
|
||||
fi
|
||||
)
|
||||
|
||||
if [ -n "$OUT" ]; then
|
||||
printf 'ruff/ty по %s:\n%s\n' "${FILE#"$ROOT"/}" "$OUT" >&2
|
||||
exit 2
|
||||
fi
|
||||
exit 0
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"hooks": {
|
||||
"PostToolUse": [
|
||||
{
|
||||
"matcher": "Write|Edit|MultiEdit|NotebookEdit",
|
||||
"hooks": [
|
||||
{
|
||||
"type": "command",
|
||||
"command": "\"$CLAUDE_PROJECT_DIR\"/.claude/hooks/lint-python.sh",
|
||||
"timeout": 60,
|
||||
"statusMessage": "ruff + ty по src/"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,185 @@
|
||||
---
|
||||
name: openspec-apply-change
|
||||
description: Implement tasks from an OpenSpec change. Use when the user wants to start implementing, continue implementation, or work through tasks.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.8.0"
|
||||
---
|
||||
|
||||
Implement tasks from an OpenSpec change.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name (e.g., `/opsx:apply add-auth`). If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:apply <other>`).
|
||||
|
||||
2. **Check status to understand the schema**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used (e.g., "spec-driven")
|
||||
- `planningHome`, `changeRoot`, and `actionContext`: planning scope and edit constraints
|
||||
- Which artifact contains the tasks (typically "tasks" for spec-driven, check status for others)
|
||||
|
||||
3. **Get apply instructions**
|
||||
|
||||
```bash
|
||||
openspec instructions apply --change "<name>" --json
|
||||
```
|
||||
|
||||
This returns:
|
||||
- `contextFiles`: artifact ID -> array of concrete file paths (varies by schema - could be proposal/specs/design/tasks or spec/tests/implementation/docs)
|
||||
- Progress (total, complete, remaining)
|
||||
- Task list with status
|
||||
- Dynamic instruction based on current state
|
||||
- Optional `context`: current required project instruction input from the selected root
|
||||
- Optional `operationGuidance`: current advisory guidance for apply
|
||||
|
||||
**Handle states:**
|
||||
- If `state: "blocked"` (missing artifacts): show message, suggest using `/opsx:continue` (if it is not installed, run `openspec status --change "<name>" --json` to see the next artifact and `openspec instructions <artifact-id> --change "<name>" --json` for how to create it)
|
||||
- If `state: "all_done"`: congratulate, suggest archive
|
||||
- Otherwise: proceed to implementation
|
||||
|
||||
Treat `context` as a required prompt-level input. Read and consider it, and
|
||||
apply relevant project facts, conventions, and constraints while implementing.
|
||||
Treat `operationGuidance` as optional additive advice. Read and consider every
|
||||
entry, and follow entries that are applicable and compatible with the built-in
|
||||
workflow.
|
||||
|
||||
Keep both fields separate from CLI-returned state, missing artifacts, tasks,
|
||||
progress, `contextFiles`, and the built-in `instruction`. They are not
|
||||
evidence of task completion, do not replace the built-in instruction, and do
|
||||
not permit bypassing a blocked state. If context conflicts with the built-in
|
||||
instruction, an explicit user choice, or a CLI-controlled value, report the
|
||||
conflict and preserve the controlling value. If guidance is inapplicable or
|
||||
conflicts with those controlling inputs, do not follow it and explain why.
|
||||
These are prompt-level behavior contracts, not enforceable checks.
|
||||
|
||||
4. **Read context files**
|
||||
|
||||
Read every file path listed under `contextFiles` from the apply instructions output.
|
||||
The files depend on the schema being used:
|
||||
- **spec-driven**: proposal, specs, design, tasks
|
||||
- Other schemas: follow the contextFiles from CLI output
|
||||
|
||||
Do not copy `context` or `operationGuidance` verbatim into implementation
|
||||
files or planning artifacts unless the user separately asks for that content.
|
||||
|
||||
5. **Show current progress**
|
||||
|
||||
Display:
|
||||
- Schema being used
|
||||
- Progress: "N/M tasks complete"
|
||||
- Remaining tasks overview
|
||||
- Dynamic instruction from CLI
|
||||
|
||||
6. **Implement tasks (loop until done or blocked)**
|
||||
|
||||
For each pending task:
|
||||
- Show which task is being worked on
|
||||
- Make the code changes required
|
||||
- Keep changes minimal and focused
|
||||
- Mark task complete in the tasks file: `- [ ]` → `- [x]`
|
||||
- Continue to next task
|
||||
|
||||
**Pause if:**
|
||||
- Task is unclear → ask for clarification
|
||||
- Implementation reveals a design issue → suggest updating artifacts
|
||||
- Error or blocker encountered → report and wait for guidance
|
||||
- User interrupts
|
||||
|
||||
7. **On completion or pause, show status**
|
||||
|
||||
Display:
|
||||
- Tasks completed this session
|
||||
- Overall progress: "N/M tasks complete"
|
||||
- If all done: suggest archive
|
||||
- If paused: explain why and wait for guidance
|
||||
|
||||
**Output During Implementation**
|
||||
|
||||
```
|
||||
## Implementing: <change-name> (schema: <schema-name>)
|
||||
|
||||
Working on task 3/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
|
||||
Working on task 4/7: <task description>
|
||||
[...implementation happening...]
|
||||
✓ Task complete
|
||||
```
|
||||
|
||||
**Output On Completion**
|
||||
|
||||
```
|
||||
## Implementation Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 7/7 tasks complete ✓
|
||||
|
||||
### Completed This Session
|
||||
- [x] Task 1
|
||||
- [x] Task 2
|
||||
...
|
||||
|
||||
All tasks complete! You can archive this change with `/opsx:archive`.
|
||||
```
|
||||
|
||||
**Output On Pause (Issue Encountered)**
|
||||
|
||||
```
|
||||
## Implementation Paused
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Progress:** 4/7 tasks complete
|
||||
|
||||
### Issue Encountered
|
||||
<description of the issue>
|
||||
|
||||
**Options:**
|
||||
1. <option 1>
|
||||
2. <option 2>
|
||||
3. Other approach
|
||||
|
||||
What would you like to do?
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Keep going through tasks until done or blocked
|
||||
- Always read context files before starting (from the apply instructions output)
|
||||
- If task is ambiguous, pause and ask before implementing
|
||||
- If implementation reveals issues, pause and suggest artifact updates
|
||||
- Keep code changes minimal and scoped to each task
|
||||
- Update task checkbox immediately after completing each task
|
||||
- Pause on errors, blockers, or unclear requirements - don't guess
|
||||
- Use contextFiles from CLI output, don't assume specific file names
|
||||
- Do not use context or operation guidance as proof that a task is complete
|
||||
- Apply relevant project context; report conflicts with controlling workflow inputs
|
||||
- Consider every guidance entry; explain any inapplicable or conflicting advice
|
||||
- Do not copy runtime context or operation guidance into implementation files or planning artifacts
|
||||
- Preserve CLI-controlled blocked/ready/all-done behavior and completion criteria
|
||||
|
||||
**Fluid Workflow Integration**
|
||||
|
||||
This skill supports the "actions on a change" model:
|
||||
|
||||
- **Can be invoked anytime**: Before all artifacts are done (if tasks exist), after partial implementation, interleaved with other actions
|
||||
- **Allows artifact updates**: If implementation reveals design issues, suggest updating artifacts - not phase-locked, work fluidly
|
||||
@@ -0,0 +1,182 @@
|
||||
---
|
||||
name: openspec-archive-change
|
||||
description: Archive a completed change in the experimental workflow. Use when the user wants to finalize and archive a change after implementation is complete.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.8.0"
|
||||
---
|
||||
|
||||
Archive a completed change in the experimental workflow.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
When prompting, show only active changes (not already archived).
|
||||
Include the schema used for each change if available.
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:archive <other>`).
|
||||
|
||||
**Load current archive inputs before the existing archive checks:**
|
||||
|
||||
After resolving the selected change and planning root, run:
|
||||
```bash
|
||||
openspec instructions archive --change "<name>" --json
|
||||
```
|
||||
Keep the same selected-root flags on this command. This lookup is advisory and
|
||||
optional: it only supplies extra prompt inputs, so it must never block archiving.
|
||||
If it exits non-zero or returns invalid JSON — for example on an older CLI that
|
||||
does not support this command yet — continue the archive workflow with no
|
||||
context and no operation guidance. Do not report an error and do not stop.
|
||||
|
||||
A successful response may omit both optional fields. Treat `context` as a
|
||||
required prompt-level input: read and consider it, and apply relevant project
|
||||
facts, conventions, and constraints. Treat `operationGuidance` as optional
|
||||
additive advice: read and consider every entry, and follow entries that are
|
||||
applicable and compatible with the built-in archive workflow.
|
||||
|
||||
Keep both fields separate from built-in steps, explicit user choices, resolved
|
||||
paths, CLI checks, and command contracts. If context conflicts with one of those
|
||||
controlling inputs, report the conflict and preserve the controlling value. If
|
||||
guidance is inapplicable or conflicts with a controlling input, do not follow it
|
||||
and explain why. Do not infer replacement paths, skipped prompts, or flags from
|
||||
either field, and do not copy their text verbatim into specs, change artifacts,
|
||||
or archive summaries unless the user separately asks for it. These are
|
||||
prompt-level behavior contracts, not enforceable checks.
|
||||
|
||||
2. **Check artifact completion status**
|
||||
|
||||
Run `openspec status --change "<name>" --json` to check artifact completion.
|
||||
|
||||
Parse the JSON to understand:
|
||||
- `schemaName`: The workflow being used
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context
|
||||
- `artifacts`: List of artifacts with their status (`done`, `skipped`, or other)
|
||||
|
||||
**If any artifacts are neither `done` nor `skipped`** (skipped artifacts satisfy the requirement - the change declares skip_specs):
|
||||
- Display warning listing incomplete artifacts
|
||||
- Ask the user to confirm they want to proceed
|
||||
- Proceed if user confirms
|
||||
|
||||
3. **Check task completion status**
|
||||
|
||||
Read the tasks file (typically `tasks.md`) to check for incomplete tasks.
|
||||
|
||||
Count tasks marked with `- [ ]` (incomplete) vs `- [x]` (complete).
|
||||
|
||||
**If incomplete tasks found:**
|
||||
- Display warning showing count of incomplete tasks
|
||||
- Ask the user to confirm they want to proceed
|
||||
- Proceed if user confirms
|
||||
|
||||
**If no tasks file exists:** Proceed without task-related warning.
|
||||
|
||||
4. **Assess delta spec sync state**
|
||||
|
||||
Use `artifactPaths.specs.existingOutputPaths` from status JSON as the only
|
||||
delta-spec source. If the `specs` entry is missing or
|
||||
`existingOutputPaths` is empty, proceed without a sync prompt and do not infer
|
||||
delta specs from other artifacts.
|
||||
|
||||
**If delta specs exist:**
|
||||
- Compare each delta spec with its corresponding main spec at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (use the store-aware `planningHome.root` from step 2, not a hardcoded repo path)
|
||||
- Determine what changes would be applied (adds, modifications, removals, renames)
|
||||
- Show a combined summary before prompting
|
||||
|
||||
**Prompt options:**
|
||||
- If changes needed: "Sync now (recommended)", "Archive without syncing"
|
||||
- If already synced: "Archive now", "Sync anyway", "Cancel"
|
||||
|
||||
Route on the answer:
|
||||
- "Cancel" — stop, do not archive
|
||||
- "Archive without syncing" or "Archive now" — proceed to archive
|
||||
- "Sync now" or "Sync anyway" — sync, then verify (below)
|
||||
- Anything else — ask again rather than archiving
|
||||
|
||||
Before a selected sync writes any main spec, run
|
||||
`openspec instructions specs --change "<name>" --json` once with the same
|
||||
selected-root flags. Require a zero exit status and valid artifact-instruction
|
||||
JSON. If the lookup fails or returns invalid JSON, report the error and stop
|
||||
before writing any main spec or moving the change. A valid response with omitted
|
||||
`rules` is the no-rules case. Apply returned `rules` only to the content and
|
||||
form of main specs produced by this merge; do not use them as archive guidance,
|
||||
change CLI behavior, or copy the rule text into any output file.
|
||||
|
||||
Then run the `openspec-sync-specs` workflow inline (agent-driven intelligent merge) for change '<name>', passing the delta spec analysis and the fetched specs-rule snapshot from above, and wait for it to finish. The inline sync must reuse that snapshot without fetching `specs` instructions again. Do not delegate it to a background task — step 5 would move `changeRoot` out from under a sync that is still reading it, leaving the change archived and the main specs never updated. If your agent can only run it by delegation, delegate synchronously and wait for the result.
|
||||
|
||||
Then re-run the comparison from the top of this step against every capability that has a delta spec in `artifactPaths.specs.existingOutputPaths` — not only the ones the sync reports it touched. A successful sync leaves nothing left to apply, so each capability must now read as already synced:
|
||||
- ADDED requirements present
|
||||
- MODIFIED requirements carrying the scenario and description changes named in the delta, with their other scenarios intact
|
||||
- REMOVED requirements gone — and where this sync retired a capability (removed its last requirement, leaving `## Requirements` empty), its main spec deleted rather than left empty; a spec the sync deliberately kept and reported is also a match
|
||||
- RENAMED requirements present under the new name and absent under the old one
|
||||
|
||||
If the sync failed, or any capability does not match, report what differs and stop — do not archive. Nothing has moved and `changeRoot` is intact, so the user can fix the mismatch or re-run the sync and start the archive again.
|
||||
|
||||
5. **Perform the archive**
|
||||
|
||||
Create an `archive` directory under `planningHome.changesDir` if it doesn't exist:
|
||||
```bash
|
||||
mkdir -p "<planningHome.changesDir>/archive"
|
||||
```
|
||||
|
||||
Generate the target name: use the change name as-is when it already starts with a `YYYY-MM-DD-` prefix; otherwise prepend the current date as `YYYY-MM-DD-<change-name>`. Never stack a second date (same rule as `openspec archive`).
|
||||
|
||||
**Check if target already exists:**
|
||||
- If yes: Fail with error, suggest renaming existing archive or using different date
|
||||
- If no: Move `changeRoot` to the archive directory
|
||||
|
||||
```bash
|
||||
mv "<changeRoot>" "<planningHome.changesDir>/archive/<target-name>"
|
||||
```
|
||||
|
||||
6. **Display summary**
|
||||
|
||||
Show archive completion summary including:
|
||||
- Change name
|
||||
- Schema that was used
|
||||
- Archive location
|
||||
- Whether specs were synced (if applicable)
|
||||
- Note about any warnings (incomplete artifacts/tasks)
|
||||
|
||||
**Output On Success**
|
||||
|
||||
```markdown
|
||||
## Archive Complete
|
||||
|
||||
**Change:** <change-name>
|
||||
**Schema:** <schema-name>
|
||||
**Archived to:** the archive path derived from `planningHome.changesDir`/<target-name>/
|
||||
**Specs:** <"✓ Synced to main specs" only if the step 4 verification passed; otherwise "No delta specs" or "Sync skipped">
|
||||
|
||||
<"All artifacts complete. All tasks complete." — or, if archived with warnings, list them instead (e.g. "Archived with 2 incomplete tasks")>
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Announce the selected change; prompt for selection when it is ambiguous
|
||||
- Use artifact graph (openspec status --json) for completion checking
|
||||
- Don't block archive on warnings - just inform and confirm
|
||||
- Preserve .openspec.yaml when moving to archive (it moves with the directory)
|
||||
- Show clear summary of what happened
|
||||
- If sync is requested, run the `openspec-sync-specs` workflow inline (agent-driven)
|
||||
- Never archive while a spec sync is still in flight — run the sync inline and verify the main specs before moving `changeRoot`
|
||||
- If delta specs exist, always run the sync assessment and show the combined summary before prompting
|
||||
- Apply relevant runtime context and report conflicts; operation guidance remains advisory
|
||||
- Consider every guidance entry and explain any inapplicable or conflicting advice
|
||||
- Existing CLI checks, resolved paths, prompts, and command contracts are unchanged
|
||||
- Artifact rules constrain only the specs being written and are never operation guidance
|
||||
- Never copy runtime context, operation guidance, or artifact-rule text verbatim into output files
|
||||
@@ -0,0 +1,308 @@
|
||||
---
|
||||
name: openspec-explore
|
||||
description: Enter explore mode - a thinking partner for exploring ideas, investigating problems, and clarifying requirements. Use when the user wants to think through something before or during a change.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.8.0"
|
||||
---
|
||||
|
||||
Enter explore mode. Think deeply. Visualize freely. Follow the conversation wherever it goes.
|
||||
|
||||
**IMPORTANT: Explore mode is for thinking, not implementing.** You may read files, search code, and investigate the codebase, but you must NEVER write code or implement features. If the user asks you to implement something, remind them to exit explore mode first and create a change proposal. You MAY create OpenSpec artifacts (proposals, designs, specs) if the user asks—that's capturing thinking, not implementing. For a new change, scaffold it first as described below.
|
||||
|
||||
**This is a stance, not a workflow.** There are no fixed steps, no required sequence, no mandatory outputs. You're a thinking partner helping the user explore.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
---
|
||||
|
||||
## The Stance
|
||||
|
||||
- **Curious, not prescriptive** - Ask questions that emerge naturally, don't follow a script
|
||||
- **Open threads, not interrogations** - Surface multiple interesting directions and let the user follow what resonates. Don't funnel them through a single path of questions.
|
||||
- **Visual** - Use ASCII diagrams liberally when they'd help clarify thinking
|
||||
- **Adaptive** - Follow interesting threads, pivot when new information emerges
|
||||
- **Patient** - Don't rush to conclusions, let the shape of the problem emerge
|
||||
- **Grounded** - Explore the actual codebase when relevant, don't just theorize
|
||||
|
||||
---
|
||||
|
||||
## What You Might Do
|
||||
|
||||
Depending on what the user brings, you might:
|
||||
|
||||
**Explore the problem space**
|
||||
- Ask clarifying questions that emerge from what they said
|
||||
- Challenge assumptions
|
||||
- Reframe the problem
|
||||
- Find analogies
|
||||
|
||||
**Investigate the codebase**
|
||||
- Map existing architecture relevant to the discussion
|
||||
- Find integration points
|
||||
- Identify patterns already in use
|
||||
- Surface hidden complexity
|
||||
|
||||
**Compare options**
|
||||
- Brainstorm multiple approaches
|
||||
- Build comparison tables
|
||||
- Sketch tradeoffs
|
||||
- Recommend a path (if asked)
|
||||
|
||||
**Visualize**
|
||||
```
|
||||
┌─────────────────────────────────────────┐
|
||||
│ Use ASCII diagrams liberally │
|
||||
├─────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌────────┐ ┌────────┐ │
|
||||
│ │ State │────────▶│ State │ │
|
||||
│ │ A │ │ B │ │
|
||||
│ └────────┘ └────────┘ │
|
||||
│ │
|
||||
│ System diagrams, state machines, │
|
||||
│ data flows, architecture sketches, │
|
||||
│ dependency graphs, comparison tables │
|
||||
│ │
|
||||
└─────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Surface risks and unknowns**
|
||||
- Identify what could go wrong
|
||||
- Find gaps in understanding
|
||||
- Suggest spikes or investigations
|
||||
|
||||
---
|
||||
|
||||
## OpenSpec Awareness
|
||||
|
||||
You have full context of the OpenSpec system. Use it naturally, don't force it.
|
||||
|
||||
### Check for context
|
||||
|
||||
At the start, quickly check what exists:
|
||||
```bash
|
||||
openspec list --json
|
||||
```
|
||||
|
||||
This tells you:
|
||||
- If there are active changes
|
||||
- Their names, schemas, and status
|
||||
- What the user might be working on
|
||||
|
||||
Then read the project's own context from the resolved root - `<root.path>/openspec/config.yaml` (or `config.yml`). Use the `root.path` returned above, and skip this if neither file exists:
|
||||
- `context`: project background - tech stack, conventions, constraints
|
||||
- `rules`: keyed by artifact id - the entries for an artifact apply only when you write that artifact
|
||||
|
||||
Ground your thinking in these. They are constraints for you to follow, not content to reproduce: do NOT copy them into the conversation or into any artifact you create.
|
||||
|
||||
### When no change exists
|
||||
|
||||
Think freely. When insights crystallize, you might offer:
|
||||
|
||||
- "This feels solid enough to start a change. Want me to create a proposal?"
|
||||
- Or keep exploring - no pressure to formalize
|
||||
|
||||
If the user asks you to capture the exploration as a new change, transition seamlessly into the requested capture:
|
||||
|
||||
1. Run `openspec new change "<name>"` (with `--store <id>` when applicable) before creating any artifacts. Never create a new change directory under `openspec/changes/` by hand; the CLI scaffold creates required metadata such as `.openspec.yaml`. Keep the selected `--store <id>` on every applicable follow-up `status` and `instructions` command.
|
||||
2. Run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store), then process the requested artifacts in dependency order. For each requested artifact that is `ready`, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store). Before creating a requested artifact, evaluate any condition in its own `instruction` against the explored change; record a deliberate skip instead when the condition does not apply. If a requested artifact is blocked by a direct prerequisite the user did not request, run `openspec instructions "<prerequisite-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) for that prerequisite whether it is `ready` or `blocked`. If its own `instruction` states a condition, evaluate that condition against the explored change and record a deliberate skip only when the condition does not apply. If the condition applies, or the prerequisite is not conditional, treat it as a normal prerequisite and ask before expanding the capture. Do not create an unrequested prerequisite unless the user approves.
|
||||
3. Follow the returned `template` and `instruction` fields. Read completed dependency files listed in `dependencies`, and apply `context` and `rules` as constraints without copying them into the artifact. If the instruction delegates creation to a specific skill or command, invoke it; otherwise write the artifact to `resolvedOutputPath`, using the instruction to choose a concrete path when it is a glob. Verify that the selected concrete output exists.
|
||||
4. After creating each artifact, re-run `openspec status --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) and continue until every requested artifact is `done`, `skipped`, or was deliberately skipped because its own `instruction` stated a condition that did not apply. Tell the user about a deliberate conditional skip, remember it, and do not reconsider it. Dependencies are enablers, not gates: if a requested artifact is still `blocked` only because you deliberately skipped a conditional prerequisite, run `openspec instructions "<artifact-id>" --change "<name>" --json` (append the confirmed `--store "<id>"` only for a registered standalone store) despite the blocked status, then create it using step 3 only when those recorded conditional skips are its sole missing dependencies. If a requested artifact is blocked by a prerequisite the user did not ask to capture and cannot be conditionally skipped, explain that dependency and ask before expanding the capture.
|
||||
|
||||
Capture the artifact(s) the user requested without asking them to invoke another workflow command. If they asked only to start a change, stop after scaffolding and show its status.
|
||||
|
||||
### When a change exists
|
||||
|
||||
If the user mentions a change or you detect one is relevant:
|
||||
|
||||
1. **Resolve and read existing artifacts for context**
|
||||
- Run `openspec status --change "<name>" --json`.
|
||||
- Use `changeRoot`, `artifactPaths`, and `actionContext` from the status JSON.
|
||||
- Read existing files from `artifactPaths.<artifact>.existingOutputPaths`.
|
||||
|
||||
2. **Reference them naturally in conversation**
|
||||
- "Your design mentions using Redis, but we just realized SQLite fits better..."
|
||||
- "The proposal scopes this to premium users, but we're now thinking everyone..."
|
||||
|
||||
3. **Offer to capture when decisions are made**
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
|
||||
|
||||
| Insight Type | Where to Capture |
|
||||
|----------------------------|-------------------------------------|
|
||||
| New requirement discovered | `specs/<capability-path>/spec.md` |
|
||||
| Requirement changed | `specs/<capability-path>/spec.md` |
|
||||
| Design decision made | `design.md` |
|
||||
| Scope changed | `proposal.md` |
|
||||
| New work identified | `tasks.md` |
|
||||
| Assumption invalidated | Relevant artifact |
|
||||
|
||||
Example offers:
|
||||
- "That's a design decision. Capture it in design.md?"
|
||||
- "This is a new requirement. Add it to specs?"
|
||||
- "This changes scope. Update the proposal?"
|
||||
|
||||
4. **The user decides** - Offer and move on. Don't pressure. Don't auto-capture.
|
||||
|
||||
---
|
||||
|
||||
## What You Don't Have To Do
|
||||
|
||||
- Follow a script
|
||||
- Ask the same questions every time
|
||||
- Produce a specific artifact
|
||||
- Reach a conclusion
|
||||
- Stay on topic if a tangent is valuable
|
||||
- Be brief (this is thinking time)
|
||||
|
||||
---
|
||||
|
||||
## Handling Different Entry Points
|
||||
|
||||
**User brings a vague idea:**
|
||||
```
|
||||
User: I'm thinking about adding real-time collaboration
|
||||
|
||||
You: Real-time collab is a big space. Let me think about this...
|
||||
|
||||
COLLABORATION SPECTRUM
|
||||
════════════════════════════════════════════
|
||||
|
||||
Awareness Coordination Sync
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
┌────────┐ ┌────────┐ ┌────────┐
|
||||
│Presence│ │Cursors │ │ CRDT │
|
||||
│ "3 │ │ Multi │ │Conflict│
|
||||
│online" │ │ select │ │ free │
|
||||
└────────┘ └────────┘ └────────┘
|
||||
│ │ │
|
||||
trivial moderate complex
|
||||
|
||||
Where's your head at?
|
||||
```
|
||||
|
||||
**User brings a specific problem:**
|
||||
```
|
||||
User: The auth system is a mess
|
||||
|
||||
You: [reads codebase]
|
||||
|
||||
┌─────────────────────────────────────────────┐
|
||||
│ CURRENT AUTH FLOW │
|
||||
└─────────────────────────────────────────────┘
|
||||
│
|
||||
┌─────────────┼─────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ Google │ │ GitHub │ │ Email │
|
||||
│ OAuth │ │ OAuth │ │ Magic │
|
||||
└────┬────┘ └────┬────┘ └────┬────┘
|
||||
│ │ │
|
||||
└─────────────┼─────────────┘
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Session │
|
||||
└─────┬─────┘
|
||||
│
|
||||
▼
|
||||
┌───────────┐
|
||||
│ Perms │
|
||||
└───────────┘
|
||||
|
||||
I see three tangles. Which one's burning?
|
||||
```
|
||||
|
||||
**User is stuck mid-implementation:**
|
||||
```
|
||||
User: /opsx:explore add-auth-system
|
||||
The OAuth integration is more complex than expected
|
||||
|
||||
You: [reads change artifacts]
|
||||
|
||||
You're on task 4: "Implement OAuth flow"
|
||||
|
||||
Let me trace what's involved...
|
||||
|
||||
[draws diagram, explores options, suggests paths]
|
||||
|
||||
Want to update the design to reflect this?
|
||||
Or add a spike task to investigate?
|
||||
```
|
||||
|
||||
**User wants to compare options:**
|
||||
```
|
||||
User: Should we use Postgres or SQLite?
|
||||
|
||||
You: Generic answer is boring. What's the context?
|
||||
|
||||
User: A CLI tool that tracks local dev environments
|
||||
|
||||
You: That changes everything.
|
||||
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ CLI TOOL DATA STORAGE │
|
||||
└─────────────────────────────────────────────────┘
|
||||
|
||||
Key constraints:
|
||||
• No daemon running
|
||||
• Must work offline
|
||||
• Single user
|
||||
|
||||
SQLite Postgres
|
||||
Deployment embedded ✓ needs server ✗
|
||||
Offline yes ✓ no ✗
|
||||
Single file yes ✓ no ✗
|
||||
|
||||
SQLite. Not even close.
|
||||
|
||||
Unless... is there a sync component?
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ending Discovery
|
||||
|
||||
There's no required ending. Discovery might:
|
||||
|
||||
- **Flow into a proposal**: "Ready to start? I can create a change proposal."
|
||||
- **Result in artifact updates**: "Updated design.md with these decisions"
|
||||
- **Just provide clarity**: User has what they need, moves on
|
||||
- **Continue later**: "We can pick this up anytime"
|
||||
|
||||
When it feels like things are crystallizing, you might summarize:
|
||||
|
||||
```
|
||||
## What We Figured Out
|
||||
|
||||
**The problem**: [crystallized understanding]
|
||||
|
||||
**The approach**: [if one emerged]
|
||||
|
||||
**Open questions**: [if any remain]
|
||||
|
||||
**Next steps** (if ready):
|
||||
- Create a change proposal
|
||||
- Keep exploring: just keep talking
|
||||
```
|
||||
|
||||
But this summary is optional. Sometimes the thinking IS the value.
|
||||
|
||||
---
|
||||
|
||||
## Guardrails
|
||||
|
||||
- **Don't implement** - Never write code or implement features. Creating OpenSpec artifacts is fine, writing application code is not.
|
||||
- **Don't fake understanding** - If something is unclear, dig deeper
|
||||
- **Don't rush** - Discovery is thinking time, not task time
|
||||
- **Don't force structure** - Let patterns emerge naturally
|
||||
- **Don't auto-capture** - Offer to save insights, don't just do it
|
||||
- **Don't manually scaffold changes** - Never create a new change directory under `openspec/changes/` by hand. Always use `openspec new change "<name>"` (with `--store <id>` when applicable) so required metadata such as `.openspec.yaml` is created before writing artifacts.
|
||||
- **Do visualize** - A good diagram is worth many paragraphs
|
||||
- **Do explore the codebase** - Ground discussions in reality
|
||||
- **Do question assumptions** - Including the user's and your own
|
||||
@@ -0,0 +1,149 @@
|
||||
---
|
||||
name: openspec-propose
|
||||
description: Propose a new change with all artifacts generated in one step. Use when the user wants to quickly describe what they want to build and get a complete proposal with design, specs, and tasks ready for implementation.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.8.0"
|
||||
---
|
||||
|
||||
Propose a new change - create the change and generate all artifacts in one step.
|
||||
|
||||
**Planning boundary**: This workflow creates planning artifacts only. The user request that selected or triggered this workflow authorizes planning only, even if it asks to build or fix something. Do not edit project code. After the planning artifacts are complete, stop. Do not start implementation in the same response, even if the initial request asks for it. Wait for a new user request after the artifacts are presented; then start the apply workflow.
|
||||
|
||||
I'll create a change with the artifacts your schema defines. With the default spec-driven schema that is:
|
||||
- proposal.md (what & why)
|
||||
- `specs/<capability-path>/spec.md` (what the system must do - a delta, not the main spec)
|
||||
- design.md (how)
|
||||
- tasks.md (implementation steps)
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve an existing capability's full path and follow the project's established organization for new capabilities.
|
||||
|
||||
When the user is ready to implement, they must start the apply workflow explicitly.
|
||||
|
||||
---
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: The user's request should include a change name (kebab-case) OR a description of what they want to build.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Understand the request and clarify material ambiguity**
|
||||
|
||||
If no clear input is provided, ask the user (open-ended, no preset options):
|
||||
> "What change do you want to work on? Describe what you want to build or fix."
|
||||
|
||||
From their description, derive a kebab-case name (e.g., "add user authentication" → `add-user-auth`).
|
||||
|
||||
**IMPORTANT**: Do NOT proceed without understanding what the user wants to build.
|
||||
|
||||
If the request contains ambiguity that would materially affect scope, externally observable behavior, compatibility, or acceptance criteria, ask the user before creating the change. For minor details, make a reasonable assumption and record it in the planning artifacts.
|
||||
|
||||
2. **Determine the workflow schema**
|
||||
|
||||
Use the configured default schema unless the user explicitly requests a different workflow.
|
||||
|
||||
**Use a different schema only if the user:**
|
||||
- Explicitly requests a specific schema by name → use `--schema <schema-name>`
|
||||
- Asks to "show workflows" or asks "what workflows" exist → resolve the authoritative root by running `openspec context --json` from the current working directory. If the user explicitly selected a registered store, use `openspec context --json --store "<store-id>"`. Then run `openspec schemas --json` with its working directory set to the returned `root.path` and let them choose. This preserves roots selected by a local `store:` pointer or the global `defaultStore`; `schemas` does not accept `--store`. If context reports only `no_openspec_root`, run `openspec schemas --json` from the current working directory instead. Do not use this fallback for invalid or unavailable stores.
|
||||
|
||||
Otherwise, omit `--schema` to preserve the configured default.
|
||||
|
||||
3. **Create the change directory**
|
||||
|
||||
Choose one schema form below. If a registered store is selected, append `--store "<store-id>"` to that command and each later OpenSpec command shown below that accepts `--store`.
|
||||
|
||||
Using the configured default:
|
||||
```bash
|
||||
openspec new change "<name>"
|
||||
```
|
||||
|
||||
Using an explicitly requested schema:
|
||||
```bash
|
||||
openspec new change "<name>" --schema "<schema-name>"
|
||||
```
|
||||
This creates a scaffolded change in the planning home resolved by the CLI with `.openspec.yaml`.
|
||||
|
||||
4. **Get the artifact build order**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to get:
|
||||
- `applyRequires`: array of artifact IDs needed before implementation (e.g., `["tasks"]`)
|
||||
- `artifacts`: list of all artifacts, each with its `status` and its `requires` edges (the artifact IDs it directly depends on)
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
|
||||
|
||||
5. **Create every artifact in the required set**
|
||||
|
||||
Use a todo list to track progress through the artifacts.
|
||||
|
||||
Loop through artifacts in dependency order (artifacts with no pending dependencies first):
|
||||
|
||||
a. **For each artifact that is `ready` (dependencies satisfied)**:
|
||||
- Get instructions:
|
||||
```bash
|
||||
openspec instructions <artifact-id> --change "<name>" --json
|
||||
```
|
||||
- The instructions JSON includes:
|
||||
- `context`: Project background (constraints for you - do NOT include in output)
|
||||
- `rules`: Artifact-specific rules (constraints for you - do NOT include in output)
|
||||
- `template`: The structure to use for your output file
|
||||
- `instruction`: Schema-specific guidance for this artifact type
|
||||
- `skipped`/`warning`: present when the change declares skip_specs and this artifact must NOT be created - stop and pick another artifact
|
||||
- `resolvedOutputPath`: Resolved path or pattern to write the artifact
|
||||
- `dependencies`: Completed artifacts to read for context
|
||||
- Read any completed dependency files for context - always re-read them from disk, even if you saw them earlier in the conversation (the user may have edited them)
|
||||
- If the `instruction` field delegates creation to a specific skill or command, invoke it to produce the artifact instead of writing the file yourself, then verify the artifact file exists at `resolvedOutputPath`
|
||||
- Otherwise create the artifact file using `template` as the structure and write it to `resolvedOutputPath`. If `resolvedOutputPath` is a glob, follow `instruction` to choose the concrete file path
|
||||
- Apply `context` and `rules` as constraints - but do NOT copy them into the file
|
||||
- Show brief progress: "Created <artifact-id>"
|
||||
|
||||
b. **Continue until every artifact in the required set exists (not just `apply.requires`)**
|
||||
- After creating each artifact, re-run `openspec status --change "<name>" --json`
|
||||
- The required set is `applyRequires` plus every artifact reachable from those by following the `requires` edges in `status --json` - walk them transitively (spec-driven closes over proposal, specs, design, tasks). Leave artifacts outside that set alone
|
||||
- `status` is file-existence only, so an `applyRequires` artifact reading `done` does NOT mean its dependencies exist - writing `tasks.md` early marks `tasks` done while `specs` was never written. Use each artifact's `requires` edges, not its `status`, to build the required set: a `done` artifact still lists what it depends on
|
||||
- An artifact already reading `status: "skipped"` is satisfied: the change declares `skip_specs` in `.openspec.yaml`, so its files must NOT exist. Never try to create one
|
||||
- Create every artifact in the required set that is missing, then re-check - creating one can unblock others
|
||||
- Skip one only when `status` already reports it `skipped`, or when its own `instruction` says it is conditional: run `openspec instructions <artifact-id> --change "<name>" --json` and skip only if its `instruction` field marks it optional (e.g. "create only if..."). Spec-driven's `design.md` qualifies; `specs` qualifies only via the `skipped` status above, never by your own judgment. Tell the user, and do not reconsider it
|
||||
- Dependencies are enablers, not gates: if a required artifact is still `blocked` only because you skipped a conditional dependency, write it anyway
|
||||
- Stop when every artifact in the required set is `done`, `skipped`, or was deliberately skipped
|
||||
|
||||
c. **If an artifact requires user input** (unclear context):
|
||||
- Ask the user to clarify
|
||||
- Then continue with creation
|
||||
|
||||
6. **Show final status**
|
||||
```bash
|
||||
openspec status --change "<name>"
|
||||
```
|
||||
|
||||
**Output**
|
||||
|
||||
After completing all artifacts, summarize:
|
||||
- Change name and location
|
||||
- List of artifacts created with brief descriptions, plus any conditional artifact you skipped and why
|
||||
- What's ready: "All artifacts needed for implementation are ready."
|
||||
- Prompt: "The artifacts are ready for review. When you are ready, run `/opsx:apply` or ask me to apply this change."
|
||||
|
||||
**Artifact Creation Guidelines**
|
||||
|
||||
- Follow the `instruction` field from `openspec instructions` for each artifact type - it is the authoritative guidance, even for familiar artifact names
|
||||
- If the `instruction` field directs you to use a specific skill or command to create the artifact, invoke it instead of writing the artifact directly
|
||||
- The schema defines what each artifact should contain - follow it
|
||||
- Read dependency artifacts for context before creating new ones
|
||||
- Use `template` as the structure for your output file - fill in its sections
|
||||
- **IMPORTANT**: `context` and `rules` are constraints for YOU, not content for the file
|
||||
- Do NOT copy `<context>`, `<rules>`, `<project_context>` blocks into the artifact
|
||||
- These guide what you write, but should never appear in the output
|
||||
|
||||
**Guardrails**
|
||||
- The request that invoked this workflow authorizes planning only. Any implementation or apply instruction in that request does not carry forward. Do NOT implement the change, start the apply workflow, or edit project code during this workflow. After presenting the artifacts, stop and wait for a new user request to start the apply workflow
|
||||
- Create every artifact the apply phase transitively depends on, not just the ids listed in `apply.requires`
|
||||
- Always read dependency artifacts before creating a new one - re-read from disk, not from conversation memory (files may have changed since you last saw them)
|
||||
- Ask about ambiguities that would materially change scope, externally observable behavior, compatibility, or acceptance criteria; for minor details, make reasonable assumptions and record them
|
||||
- If a change with that name already exists, ask if user wants to continue it or create a new one
|
||||
- Verify each artifact file exists after writing before proceeding to next
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
name: openspec-sync-specs
|
||||
description: Sync delta specs from a change to main specs. Use when the user wants to update main specs with changes from a delta spec, without archiving the change.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.8.0"
|
||||
---
|
||||
|
||||
Sync delta specs from a change to main specs.
|
||||
|
||||
This is an **agent-driven** operation - you will read delta specs and directly edit main specs to apply the changes. This allows intelligent merging (e.g., adding a scenario without copying the entire requirement).
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
`<capability-path>` is the spec directory relative to `specs/` (for example, `user-auth` or `identity/user-auth`). Preserve the full path from each delta spec when resolving its main spec.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes and ask the user to select one
|
||||
|
||||
When prompting, show changes that have delta specs (under `specs/` directory).
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:sync <other>`).
|
||||
|
||||
2. **Resolve change context**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
|
||||
The JSON includes `planningHome.root`. Main specs live under `<planningHome.root>/openspec/specs/` — use that (store-aware) root for every main-spec path below, not a hardcoded repo path. When a store is selected it points at the store, not the current repository.
|
||||
|
||||
3. **Find delta specs**
|
||||
|
||||
Use `artifactPaths.specs.existingOutputPaths` from the status JSON as the
|
||||
only source of delta spec paths. If the `specs` entry is missing or
|
||||
`existingOutputPaths` is empty, report that there are no delta specs to sync,
|
||||
do not infer them from other artifacts, and stop without requesting artifact
|
||||
instructions or writing a main spec.
|
||||
|
||||
Sync every path in `existingOutputPaths` unless the caller narrowed the set.
|
||||
A caller narrows it by naming an explicit list of complete entries from
|
||||
`existingOutputPaths` — copy those absolute values verbatim. Archive does
|
||||
this inline, and a user can too (for example, by selecting the entry ending
|
||||
in `/specs/billing/invoices/spec.md`).
|
||||
Then sync only the named paths and leave the remaining delta specs untouched:
|
||||
bulk archive excludes a delta whose implementation it could not find, and
|
||||
syncing it anyway would write a main spec the caller deliberately withheld.
|
||||
Carry that narrowed selection through step 4; never widen it back to the full
|
||||
list. If a named path is not in `existingOutputPaths`, do not sync it —
|
||||
report it and stop, rather than dropping it silently. If the named list is
|
||||
empty, report that there is nothing to sync and stop without writing a main
|
||||
spec.
|
||||
|
||||
Each delta spec file contains sections like:
|
||||
- `## ADDED Requirements` - New requirements to add
|
||||
- `## MODIFIED Requirements` - Changes to existing requirements
|
||||
- `## REMOVED Requirements` - Requirements to remove
|
||||
- `## RENAMED Requirements` - Requirements to rename (FROM:/TO: format)
|
||||
|
||||
If no delta specs found, inform user and stop.
|
||||
|
||||
4. **For each delta spec, apply changes to main specs**
|
||||
|
||||
Before the first main-spec write, obtain one current specs-rule snapshot:
|
||||
- If archive invoked this workflow inline and supplied a valid snapshot from
|
||||
`openspec instructions specs --change "<name>" --json`, reuse it and do not
|
||||
fetch the same instructions again.
|
||||
- Otherwise run that command once now with the same selected-root flags.
|
||||
- If the direct lookup exits non-zero or returns invalid artifact-instruction
|
||||
JSON, report the error and stop before writing any main spec. Do not treat the
|
||||
failure as an absent rule set.
|
||||
- A valid response with omitted `rules` means no artifact rules are configured
|
||||
and the existing semantic merge continues.
|
||||
|
||||
Apply returned `rules` only to the content and form of the main specs produced
|
||||
by this merge. Artifact rules are not operation guidance and cannot change
|
||||
selected roots, delta paths, CLI checks, or workflow steps. Use their text as
|
||||
constraints without copying it verbatim into a main spec or summary.
|
||||
|
||||
For each capability delta spec path selected in step 3 — the full `existingOutputPaths` list, or the narrowed subset when a caller supplied one (these may belong to a selected store, not the repo):
|
||||
|
||||
a. **Read the delta spec** to understand the intended changes
|
||||
|
||||
b. **Read the main spec** at `<planningHome.root>/openspec/specs/<capability-path>/spec.md` (may not exist yet)
|
||||
|
||||
c. **Apply changes intelligently**:
|
||||
|
||||
**ADDED Requirements:**
|
||||
- If requirement doesn't exist in main spec → add it
|
||||
- If requirement already exists → update it to match (treat as implicit MODIFIED)
|
||||
|
||||
**MODIFIED Requirements:**
|
||||
- Find the requirement in main spec
|
||||
- Apply the changes - this can be:
|
||||
- Adding new scenarios the main spec does not have yet
|
||||
- Modifying existing scenarios
|
||||
- Changing the requirement description
|
||||
- Preserve scenarios/content not mentioned in the delta
|
||||
|
||||
**REMOVED Requirements:**
|
||||
- Remove the entire requirement block from main spec
|
||||
- Retiring the capability. Delete the whole `spec.md` - and the directory once
|
||||
nothing else is left in it - only when ALL of these hold:
|
||||
1. removing the requirements *this run* left no requirement blocks;
|
||||
2. the rest of the spec is well-formed (it still has a `## Purpose`);
|
||||
3. the main spec was not already empty before this sync - if you removed
|
||||
nothing, change nothing;
|
||||
4. every other nonblank line in the whole file is accounted for as the
|
||||
title, Purpose, Requirements header, or a canonical requirement's
|
||||
statement, scenarios, or fenced examples;
|
||||
5. the change's `.openspec.yaml` declares `retire_capabilities: true`;
|
||||
6. the `spec.md` resolves inside the real specs root (do not follow a
|
||||
capability-directory symlink to delete an external file).
|
||||
If removing the selected requirements would leave no requirement blocks and
|
||||
any retirement condition is not satisfied, do not modify the main spec. Stop
|
||||
the sync for that capability, report the blocking condition, and tell the user
|
||||
how to resolve it. Never write or leave an empty `## Requirements` section.
|
||||
When only the marker is missing, say that too - it is the one thing the user
|
||||
can add to make the retirement go through.
|
||||
- Deleting the file also deletes its `## Purpose`; any other section blocks
|
||||
retirement. Name Purpose when you report the retirement. Include a pasteable
|
||||
`git checkout` only when the spec lived in the caller's checkout;
|
||||
otherwise give checkout-scoped recovery guidance.
|
||||
|
||||
**RENAMED Requirements:**
|
||||
- Find the FROM requirement, rename to TO
|
||||
|
||||
**`## Purpose` in the delta:**
|
||||
- The main spec already has one and it is authoritative - leave it alone
|
||||
(this is what `openspec archive` does; it warns and moves on)
|
||||
|
||||
d. **Create new main spec** if capability doesn't exist yet:
|
||||
- Create `<planningHome.root>/openspec/specs/<capability-path>/spec.md`
|
||||
- Add Purpose section: copy the delta's `## Purpose` body verbatim when it has one
|
||||
(this is what `openspec archive` does); only write a brief TBD placeholder when it does not
|
||||
- Add Requirements section with the ADDED requirements
|
||||
- Follow the **Main Spec Format Reference** below
|
||||
|
||||
5. **Validate updated main specs**
|
||||
|
||||
Run `openspec validate --specs` with the same selected-root flags used earlier.
|
||||
If validation fails, report the problems and do not claim the sync succeeded.
|
||||
|
||||
6. **Show summary**
|
||||
|
||||
After applying all changes, summarize:
|
||||
- Which capabilities were updated
|
||||
- What changes were made (requirements added/modified/removed/renamed)
|
||||
- Any new main spec left with a TBD Purpose placeholder, so it gets written
|
||||
now rather than lingering
|
||||
- Any capability retired, naming the deleted `spec.md`, its Purpose, and
|
||||
either a pasteable `git checkout` or checkout-scoped recovery guidance
|
||||
|
||||
**Delta Spec Format Reference**
|
||||
|
||||
```markdown
|
||||
## Purpose
|
||||
|
||||
Only on a delta that introduces a brand-new capability. Seeds the new main spec.
|
||||
|
||||
## ADDED Requirements
|
||||
|
||||
### Requirement: New Feature
|
||||
The system SHALL do something new.
|
||||
|
||||
#### Scenario: Basic case
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
|
||||
## MODIFIED Requirements
|
||||
|
||||
### Requirement: Existing Feature
|
||||
The system SHALL keep doing the existing thing, now also handling A.
|
||||
|
||||
#### Scenario: Scenario the main spec already has
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
|
||||
#### Scenario: New scenario to add
|
||||
- **WHEN** user does A
|
||||
- **THEN** system does B
|
||||
|
||||
## REMOVED Requirements
|
||||
|
||||
### Requirement: Deprecated Feature
|
||||
|
||||
## RENAMED Requirements
|
||||
|
||||
- FROM: `### Requirement: Old Name`
|
||||
- TO: `### Requirement: New Name`
|
||||
```
|
||||
|
||||
**Main Spec Format Reference**
|
||||
|
||||
Main specs are what the delta merges INTO. They must never contain delta operation headers (`## ADDED/MODIFIED/REMOVED/RENAMED Requirements`) - after syncing, every requirement lives under a single `## Requirements` section:
|
||||
|
||||
```markdown
|
||||
# <capability> Specification
|
||||
|
||||
## Purpose
|
||||
Short description of what this capability does and why it exists.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement: New Feature
|
||||
The system SHALL do something new.
|
||||
|
||||
#### Scenario: Basic case
|
||||
- **WHEN** user does X
|
||||
- **THEN** system does Y
|
||||
```
|
||||
|
||||
**Key Principle: Intelligent Merging**
|
||||
|
||||
Unlike programmatic merging, you merge rather than overwrite:
|
||||
- A MODIFIED block carries the whole requirement - body plus every scenario that survives the change. `openspec validate` and `openspec archive` both reject one that drops a scenario the main spec still has.
|
||||
- Keep anything the delta does not mention, in the main spec's existing order
|
||||
- Use your judgment to merge changes sensibly
|
||||
|
||||
**Output On Success**
|
||||
|
||||
```markdown
|
||||
## Specs Synced: <change-name>
|
||||
|
||||
Updated main specs:
|
||||
|
||||
**<capability-1>**:
|
||||
- Added requirement: "New Feature"
|
||||
- Modified requirement: "Existing Feature" (added 1 scenario)
|
||||
|
||||
**<capability-2>**:
|
||||
- Created new spec file
|
||||
- Added requirement: "Another Feature"
|
||||
|
||||
Main specs are now updated. The change remains active - archive when implementation is complete.
|
||||
```
|
||||
|
||||
**Guardrails**
|
||||
- Read both delta and main specs before making changes
|
||||
- Preserve existing content not mentioned in delta
|
||||
- Never copy a delta file into a main spec as-is - merge its content so the main spec keeps the Main Spec Format Reference structure, with no delta operation headers
|
||||
- If something is unclear, ask for clarification
|
||||
- Show what you're changing as you go
|
||||
- The operation should be idempotent - running twice should give same result
|
||||
- Use only `artifactPaths.specs.existingOutputPaths`; never infer delta specs from unrelated artifacts
|
||||
- Honor a caller-supplied subset of `existingOutputPaths`; never widen it back to the full list
|
||||
- Fetch specs instructions once for direct sync, or reuse the archive-supplied snapshot inline
|
||||
- Stop before every main-spec write on a non-zero or invalid JSON specs-instruction response
|
||||
- Artifact rules constrain only the specs being written and are never copied into output files
|
||||
@@ -0,0 +1,91 @@
|
||||
---
|
||||
name: openspec-update-change
|
||||
description: Update an OpenSpec change by revising its existing planning artifacts and keeping them coherent with one another. Use when the user wants to revise a change's plan, fold new decisions into it, or reconcile its artifacts after an edit. Never edits code.
|
||||
allowed-tools: Bash(openspec:*)
|
||||
license: MIT
|
||||
compatibility: Requires openspec CLI.
|
||||
metadata:
|
||||
author: openspec
|
||||
version: "1.0"
|
||||
generatedBy: "1.8.0"
|
||||
---
|
||||
|
||||
Revise a change's existing planning artifacts and keep them coherent. Never edit code.
|
||||
|
||||
**Store selection:** If the user names a store (a store is a standalone OpenSpec repo registered on this machine) or the work lives in one, run `openspec store list --json` to discover registered store ids, then pass `--store <id>` on the commands that read or write specs and changes (`new change`, `status`, `instructions`, `list`, `show`, `validate`, `archive`, `doctor`, `context`, `view`). Once selected, treat `--store <id>` as sticky for the rest of the workflow. Every unscoped example of those commands below is shorthand: before running it, append the flag. For example, run `openspec status --change "<name>" --json --store "<id>"`, not the unscoped form shown below. Other commands do not take the flag. Hints printed by commands already carry the flag; keep it on follow-ups. Without a store, commands act on the nearest local `openspec/` root.
|
||||
|
||||
**Input**: Optionally specify a change name. If omitted, check if it can be inferred from conversation context. If vague or ambiguous you MUST prompt for available changes.
|
||||
|
||||
`/opsx:continue` is an expanded-profile workflow and may not be installed. Before suggesting it anywhere below, verify that it is available. If it is unavailable, `openspec status --change "<name>" --json` shows the next artifact and `openspec instructions "<artifact-id>" --change "<name>" --json` explains how to create it.
|
||||
|
||||
**Steps**
|
||||
|
||||
1. **Select the change**
|
||||
|
||||
If a name is provided, use it. Otherwise:
|
||||
- Infer from conversation context if the user mentioned a change
|
||||
- Auto-select if only one active change exists
|
||||
- If ambiguous, run `openspec list --json` to get available changes sorted by most recently modified, and ask the user to select one
|
||||
|
||||
When prompting, present the top 3-4 most recently modified changes as options, showing:
|
||||
- Change name
|
||||
- Schema (from `schema` field if present, otherwise "spec-driven")
|
||||
- Status (e.g., "0/5 tasks", "complete", "no tasks")
|
||||
- How recently it was modified (from `lastModified` field)
|
||||
|
||||
Mark the most recently modified change as "(Recommended)" since it's likely what the user wants to update.
|
||||
|
||||
Always announce: "Using change: <name>" and how to override (e.g., `/opsx:update <other>`).
|
||||
|
||||
2. **Get the change's artifacts**
|
||||
```bash
|
||||
openspec status --change "<name>" --json
|
||||
```
|
||||
Parse the JSON to understand current state. The response includes:
|
||||
- `schemaName`: The workflow schema being used (e.g., "spec-driven")
|
||||
- `artifacts`: Array of artifacts with their status ("done", "skipped", "ready", "blocked")
|
||||
- `isPlanningComplete`: Boolean indicating if all planning artifacts are complete. Older CLI versions expose the same value as `isComplete`.
|
||||
- `planningHome`, `changeRoot`, `artifactPaths`, and `actionContext`: path and scope context. Use these instead of assuming repo-local paths.
|
||||
|
||||
The artifact ids and paths come from the active schema - do NOT assume them, and do NOT branch on hardcoded artifact names. Custom schemas must work unchanged.
|
||||
|
||||
The files to edit are `artifactPaths.<id>.existingOutputPaths` - the concrete files that exist on disk, already glob-expanded for glob artifacts (e.g. `specs/**/*.md`). Do NOT write to `resolvedOutputPath`: for a glob artifact it is still the glob pattern, not a real file.
|
||||
|
||||
3. **Understand the request**
|
||||
- If the user asked for a specific revision ("the design now uses X"), that is the starting edit.
|
||||
- If they only said "update" / "make this coherent", treat it as a coherence review: read the existing artifacts and check them against each other for contradictions, gaps, and duplication.
|
||||
|
||||
4. **Read and reconcile**
|
||||
- Read the artifact(s) the request touches and the change's other existing artifacts.
|
||||
- Apply the requested edit. Then check every other existing artifact against it - in ANY direction: an edit to a later artifact may require revising an earlier one, not only the other way around. Build order is a useful reading order, not a constraint on which artifacts may be revised.
|
||||
- Note everything that is now inconsistent, missing, or contradictory.
|
||||
- Revise only files that already exist (`existingOutputPaths`). Do NOT create artifacts that don't exist yet, and do NOT invent new files under a glob artifact - note them and point the user to `/opsx:continue` to create them.
|
||||
- If the change is already coherent, say so and make no edits.
|
||||
|
||||
5. **Confirm and apply, one artifact at a time**
|
||||
- Show each proposed revision and why. Write only after the user confirms.
|
||||
- If the user rejects a revision, do not write it - leave that artifact unchanged.
|
||||
- When a substantial rewrite is needed, get that artifact's rules and template first:
|
||||
```bash
|
||||
openspec instructions "<artifact-id>" --change "<name>" --json
|
||||
```
|
||||
|
||||
6. **Point to the next step (guidance only - NEVER act on it)**
|
||||
- Artifacts still missing -> suggest `/opsx:continue` to create them.
|
||||
- Change already implemented (tasks checked off / already applied) -> the code may no longer match the revised plan; suggest `/opsx:apply` to carry the delta into code.
|
||||
- Everything done and implemented -> suggest `/opsx:archive`.
|
||||
|
||||
**Output**
|
||||
|
||||
After each invocation, show:
|
||||
- Which artifacts were revised (and which proposed revisions were rejected)
|
||||
- Anything deferred to `/opsx:continue` (not-yet-created artifacts or files)
|
||||
- Where the change stands and the recommended next command
|
||||
|
||||
**Guardrails**
|
||||
- Planning artifacts only - NEVER edit implementation code. If the revised plan implies code changes, stop and point to `/opsx:apply`.
|
||||
- Use the artifact ids and paths reported by `openspec status`; never branch on hardcoded artifact names.
|
||||
- Edit only the concrete files in `existingOutputPaths`; never write to a glob `resolvedOutputPath`.
|
||||
- Do not advance the build frontier: no new artifacts, no new files under glob artifacts - that is `/opsx:continue`'s job.
|
||||
- Confirm every edit with the user before writing.
|
||||
- If the request changes the change's *intent* rather than refining it, first verify whether the expanded-profile `/opsx:new` workflow is available. If it is, recommend starting fresh with `/opsx:new` (the "Update vs. Start Fresh" heuristic). If it is unavailable, ask for a distinct unused change name and recommend `openspec new change "<new-change-name>"` instead.
|
||||
@@ -14,7 +14,33 @@ backups
|
||||
*.pfx
|
||||
*.jks
|
||||
.venv
|
||||
venv
|
||||
__pycache__
|
||||
*.pyc
|
||||
*.pyo
|
||||
.pytest_cache
|
||||
.ruff_cache
|
||||
.mypy_cache
|
||||
.ty_cache
|
||||
node_modules
|
||||
evidence
|
||||
openspec
|
||||
student-tests
|
||||
tests
|
||||
scripts
|
||||
examples
|
||||
contracts
|
||||
infra
|
||||
*.md
|
||||
!README.md
|
||||
Makefile
|
||||
Dockerfile*
|
||||
compose*.yaml
|
||||
.dockerignore
|
||||
.gitignore
|
||||
.DS_Store
|
||||
lab.json
|
||||
requirements-dev.txt
|
||||
.claude
|
||||
|
||||
.verification/
|
||||
|
||||
@@ -28,3 +28,79 @@ OTEL_SERVICE_NAME=messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
|
||||
# Solution-specific: published LB ports, replica limits, workers and pool per replica,
|
||||
# LB queue per replica, drain window before closing the listener on SIGTERM, PostgreSQL
|
||||
# limits, load generator limits (profile bench), cursor key file. API_PORT/SHORTTTL_PORT of lab2 are gone:
|
||||
# only the load balancer is published.
|
||||
LB_HTTP_PORT=18086
|
||||
LB_STATS_PORT=26085
|
||||
API_CPUS=1
|
||||
API_MEMORY=512m
|
||||
WEB_CONCURRENCY=2
|
||||
DB_POOL_MAX_SIZE=8
|
||||
API_SERVER_MAXCONN=64
|
||||
DRAIN_SECONDS=2
|
||||
PG_MAX_CONNECTIONS=200
|
||||
PG_CPUS=2
|
||||
PG_MEMORY=1g
|
||||
K6_CPUS=2
|
||||
K6_MEMORY=512m
|
||||
CURSOR_SECRET_FILE=/run/secrets/cursor_secret
|
||||
# Solution-specific for lab4: S3 client timeouts (connect + read, two attempts stay below
|
||||
# REQUEST_TIMEOUT_SECONDS=8), parallel image checks per worker, RustFS memory limit.
|
||||
S3_CONNECT_TIMEOUT_SECONDS=1
|
||||
S3_READ_TIMEOUT_SECONDS=2
|
||||
IMAGE_CHECK_CONCURRENCY=2
|
||||
S3_MEMORY=512m
|
||||
# Solution-specific for lab4 grade 4: upload bodies buffered at once per worker, grace period
|
||||
# before an abandoned pending upload is swept, sweep interval and rows per sweep pass.
|
||||
UPLOAD_CONCURRENCY=3
|
||||
UPLOAD_GRACE_SECONDS=900
|
||||
UPLOAD_SWEEP_INTERVAL_SECONDS=30
|
||||
UPLOAD_SWEEP_BATCH=100
|
||||
# Fixed glibc mmap threshold for the API: buffers above 128 KiB are returned to the system on free.
|
||||
MALLOC_MMAP_THRESHOLD_=131072
|
||||
# Solution-specific for lab5: RabbitMQ management UI on 127.0.0.1, backoff base of a transient
|
||||
# retry (doubles per attempt up to 60 s), relay poll and sweep intervals, worker limits and the
|
||||
# PostgreSQL pool of relay and worker. FAULT_POINT is set only by infra/compose.faults.yaml.
|
||||
BROKER_MANAGEMENT_PORT=26086
|
||||
JOB_BACKOFF_BASE_SECONDS=2
|
||||
OUTBOX_POLL_SECONDS=1
|
||||
SWEEP_INTERVAL_SECONDS=5
|
||||
WORKER_CPUS=2
|
||||
WORKER_MEMORY=1g
|
||||
PIPELINE_DB_POOL_MAX_SIZE=4
|
||||
# Solution-specific for lab5 grade 4: render time per job (the render process is killed after it),
|
||||
# address space of one render process (WORKER_MEMORY >= WORKER_CONCURRENCY * RENDER_MEMORY_MB plus
|
||||
# ~120 MiB of the worker itself), jobs per render process before it is replaced.
|
||||
RENDER_TIMEOUT_SECONDS=30
|
||||
RENDER_MEMORY_MB=448
|
||||
RENDER_TASKS_PER_CHILD=100
|
||||
# Solution-specific for lab6: host ports of Prometheus, Loki, Tempo and Grafana (127.0.0.1 only),
|
||||
# Prometheus retention, OTLP export timeout in seconds, metric export interval in ms, SDK queue
|
||||
# sizes, the bounded flush on shutdown; OTEL_SDK_DISABLED=true turns the SDK and instrumentations off.
|
||||
# relay and worker get OTEL_SERVICE_NAME messenger-relay and messenger-worker in compose.yaml.
|
||||
PROMETHEUS_PORT=26090
|
||||
LOKI_PORT=26310
|
||||
TEMPO_PORT=26320
|
||||
GRAFANA_PORT=26300
|
||||
PROMETHEUS_RETENTION=7d
|
||||
OTEL_SDK_DISABLED=false
|
||||
OTEL_EXPORTER_OTLP_TIMEOUT=2
|
||||
OTEL_METRIC_EXPORT_INTERVAL=10000
|
||||
OTEL_BSP_MAX_QUEUE_SIZE=2048
|
||||
OTEL_BLRP_MAX_QUEUE_SIZE=2048
|
||||
TELEMETRY_SHUTDOWN_SECONDS=2
|
||||
# Minimal client transfer rate, bytes/s: every body byte moved in either direction extends the
|
||||
# request deadline by 1/rate, so slow but steady uploads and downloads outlive REQUEST_TIMEOUT_SECONDS.
|
||||
MIN_TRANSFER_RATE_BYTES=131072
|
||||
# Solution-specific for lab4 track 2: limits of the one-off reconciler (profile reconcile), its S3
|
||||
# read timeout for listing pages, optional CPU pinning of the measured services (empty = no pinning)
|
||||
# and the CPU limit of RustFS (make reconcile-cost raises it with its memory for the 100k point).
|
||||
RECONCILE_CPUS=1
|
||||
RECONCILE_MEMORY=256m
|
||||
RECONCILE_S3_READ_TIMEOUT_SECONDS=10
|
||||
RECONCILE_CPUSET=
|
||||
S3_CPUSET=
|
||||
PG_CPUSET=
|
||||
S3_CPUS=1
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
name: Application
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
quality:
|
||||
name: Линт, типы и собственные тесты
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
enable-cache: true
|
||||
- run: uv sync --locked --group dev
|
||||
- run: make lint VENV=.venv
|
||||
- run: make student-test VENV=.venv
|
||||
|
||||
acceptance:
|
||||
name: Стенд Compose с RabbitMQ, relay и worker за HAProxy, smoke, pipeline, отказ реплики, рестарт и отказы зависимостей
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: astral-sh/setup-uv@v5
|
||||
with:
|
||||
enable-cache: true
|
||||
- run: uv sync --locked --group dev
|
||||
|
||||
- name: Поднять стенд из чистого clone
|
||||
run: make up
|
||||
|
||||
- name: Публичный smoke через балансировщик
|
||||
run: make test
|
||||
|
||||
- name: Топология брокера, consumer с prefetch и права клиентов
|
||||
run: make topology
|
||||
|
||||
- name: Уровни «3» и «4» pipeline — worker остановлен, рестарт брокера, дубль, S3, попытки, варианты и пределы, падения в окнах 2–5, аренда, replay, DLQ, jitter, retention
|
||||
run: |
|
||||
make suite-preflight
|
||||
make level4
|
||||
|
||||
- name: Сообщение задания в очереди, 409 до ready и SIGTERM relay и worker
|
||||
run: make pipeline-evidence
|
||||
|
||||
- name: Вложение после остановки принявшей его реплики и при недоступном S3
|
||||
run: |
|
||||
make scenario SCENARIO=failover-upload
|
||||
make scenario SCENARIO=s3-outage
|
||||
|
||||
- name: Приватный bucket, ограниченный ключ API и повторный bootstrap
|
||||
run: make s3-check
|
||||
|
||||
- name: Негативный suite вложений, компенсация, уборка после grace, обрыв и зависание загрузки
|
||||
run: make upload-faults
|
||||
|
||||
- name: Несколько реплик на одном токене и гонка одного ключа
|
||||
run: make lb-check
|
||||
|
||||
- name: Повтор неподтверждённой записи после потерянного ответа и убитой реплики
|
||||
run: make retry-check
|
||||
|
||||
- name: Остановка реплики под запросами, затем возврат
|
||||
run: |
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
docker compose ps -a api
|
||||
make restore
|
||||
|
||||
- name: Рестарт LB, API и хранилищ с сохранением volumes
|
||||
run: make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
|
||||
- name: Отказы Valkey и PostgreSQL, гонка и rollback
|
||||
run: make test-faults
|
||||
|
||||
- name: Лимиты, порты, секреты, volumes и сети по политике
|
||||
run: make limits
|
||||
|
||||
- name: Время обнаружения отказа балансировщиком (SIGTERM, kill, pause, PostgreSQL)
|
||||
run: make detection
|
||||
|
||||
- name: Наружу опубликован только балансировщик
|
||||
run: |
|
||||
docker compose ps --format '{{.Service}} {{.Ports}}' | awk '$2 != "" && $1 != "lb" && $1 != "broker" {print; bad=1} END {exit bad}'
|
||||
docker compose ps --format '{{.Service}} {{.Ports}}' | awk '$1 == "broker" && $2 !~ /^127\.0\.0\.1:/ {print; bad=1} END {exit bad}'
|
||||
|
||||
- name: Процессы работают не от root
|
||||
run: |
|
||||
for id in $(docker compose ps -q); do
|
||||
docker top "$id" -eo pid,uid,comm | awk 'NR>1 && $2==0 {print; bad=1} END {exit bad}' || exit 1
|
||||
done
|
||||
|
||||
- name: Одноразовые сервисы не от root
|
||||
run: |
|
||||
for service in migrate s3-init; do
|
||||
user=$(docker inspect -f '{{.Config.User}}' "$(docker compose ps -aq "$service")")
|
||||
echo "$service: $user"
|
||||
case "${user%%:*}" in ""|0|root) exit 1 ;; esac
|
||||
done
|
||||
|
||||
- name: Корректная остановка relay и worker по SIGTERM
|
||||
run: |
|
||||
start=$(date +%s)
|
||||
docker compose stop -t 10 relay worker
|
||||
elapsed=$(( $(date +%s) - start ))
|
||||
codes=$(docker inspect -f '{{.State.ExitCode}}' $(docker compose ps -aq relay worker) | sort -u)
|
||||
echo "остановка за ${elapsed} с, коды завершения ${codes}"
|
||||
test "$elapsed" -le 10 && test "$codes" = "0"
|
||||
|
||||
- name: Корректная остановка реплик API по SIGTERM
|
||||
run: |
|
||||
start=$(date +%s)
|
||||
docker compose kill --signal SIGTERM api
|
||||
codes=$(docker wait $(docker compose ps -aq api) | sort -u)
|
||||
elapsed=$(( $(date +%s) - start ))
|
||||
echo "остановка за ${elapsed} с, коды завершения ${codes}"
|
||||
test "$elapsed" -le 10 && test "$codes" = "0"
|
||||
|
||||
- name: Журналы стенда
|
||||
if: always()
|
||||
run: docker compose logs --tail 80
|
||||
+10
@@ -30,3 +30,13 @@ coverage/
|
||||
.vscode/
|
||||
# Keep small, sanitized evidence summaries in evidence/ under version control.
|
||||
# Inspect git diff --cached: ignore rules are not a secret scanner.
|
||||
|
||||
# Тулчейн студенческой реализации
|
||||
.pytest_cache/
|
||||
.ruff_cache/
|
||||
.ty_cache/
|
||||
|
||||
# Локальные настройки агента
|
||||
.claude/settings.local.json
|
||||
|
||||
.verification/
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
## Что это за репозиторий
|
||||
|
||||
Skeleton лабораторной 6 курса «Мессенджер под нагрузкой» (МТУСИ, 7 независимых лаб) и решение в ветке `solution/bpi2303-kostylev`. Skeleton даёт задание (`README.md`), нормативный контракт (`contracts/`, включая `contracts/observability.md`) и публичные black-box проверки (`tests/`). Реализация API, relay, worker, Dockerfile, compose-стенд, конфигурация балансировщика, S3, брокера и стека наблюдаемости, миграции, хуки отказов и собственные тесты — студенческие; решение лабы 6 построено переносом решения лабы 5.
|
||||
|
||||
Лаба 6: наблюдаемость всей системы, включая фоновую обработку. Бизнес-API не меняется (контракт 1.6.0). API, relay и worker инициализируют **OpenTelemetry SDK** в каждом процессе после fork (gunicorn-воркер, процесс relay, процесс worker) и отдают OTLP/HTTP в **OTel Collector** (`otelcol-contrib`); Collector раскладывает метрики в **Prometheus 3** (OTLP-приём), логи в **Loki 3** (OTLP `/otlp`), трассы в **Tempo 2**; **Grafana 12** только показывает и связывает их, datasources и dashboards приходят из provisioning в Git. HTTP-метрики и server spans даёт инструментация FastAPI, client spans БД — инструментация asyncpg под SQLAlchemy 2.0.45 async (ORM/Core, async_sessionmaker, DeclarativeBase); миграции — Alembic 1.16.5, consumer spans — `RabbitTelemetryMiddleware` FastStream. Логи идут через порт `Logger` (structlog → stdout JSON и OTLP) с `trace_id`/`span_id`, без тел сообщений и секретов. Остаётся всё из лаб 1–5: HAProxy перед ≥2 репликами, PostgreSQL, Valkey, RustFS, RabbitMQ, outbox, relay, worker с арендой и вариантами, хуки остановки реплики и рестарта стенда.
|
||||
|
||||
## Команды
|
||||
|
||||
```sh
|
||||
make check # offline: структура skeleton, JSON, локальные $ref, синтаксис Python, версия 1.6.0
|
||||
make validate PYTHON=.venv/bin/python # полная валидация OpenAPI 3.1 и jobs.schema.json (venv с requirements-dev.txt)
|
||||
make install # uv sync --group dev
|
||||
make lint # ruff check + ruff format --check + ty check; обязательно перед commit
|
||||
make student-test # собственные тесты (одноразовые PostgreSQL, Valkey, RustFS и RabbitMQ в Docker)
|
||||
|
||||
make secrets # локальные secrets/* (БД, кэш, S3, broker, Grafana) и .env
|
||||
make up # docker compose build, затем up -d --wait: хранилища, broker, стек наблюдаемости → migrate, s3-init → api ×2, relay, worker → lb
|
||||
make obs-validate # otelcol validate, promtool check config, loki/tempo verify, JSON dashboards
|
||||
make telemetry-check # Grafana provisioning, запрос → лог → трасса, job → трасса и лог worker, утечки
|
||||
make test # публичный smoke через LB (через scripts/lab6.sh: http://127.0.0.1:18086)
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh # рестарт lb, api, relay, worker и postgres, valkey, s3, broker с volumes
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh # остановка реплики TARGET_INSTANCE_ID под запросами
|
||||
make restore # вернуть остановленные реплики и дождаться их в ротации LB
|
||||
make level3 # сценарии уровня «3» лабы 5 (scripts/pipeline_suite.py) → evidence/level3.json
|
||||
make down # остановка без удаления данных; make reset-data — намеренное удаление volumes
|
||||
```
|
||||
|
||||
Команды стенда выше запускаются через `scripts/lab6.sh`, например `scripts/lab6.sh make up`. Локальные `make check validate lint` запускаются напрямую; `make student-test` создаёт одноразовые Docker-зависимости и не выполняется в документационной фазе.
|
||||
|
||||
`tests/*.py` и `scripts/*.py` — только stdlib, синтаксис Python 3.9 (`make check` разбирает их системным `python3`). Проверки сами не поднимают стенд. Хуки `scripts/stop-one.sh` и `scripts/restart.sh` запускаются тестом без shell, поэтому исполняемые и с shebang. Адреса Prometheus, Loki, Tempo и Grafana для `scripts/obs_check.py` — переменные `PROMETHEUS_URL`, `LOKI_URL`, `TEMPO_URL`, `GRAFANA_URL` (по умолчанию стандартные порты на `127.0.0.1`).
|
||||
|
||||
## Архитектура skeleton
|
||||
|
||||
- `lab.json` — единственный источник номера лабы (6). От него зависят `scripts/check_contract.py` (`info.version == 1.6.0`) и `tests/client.py` (ветки `LAB>=5`: upload `202 queued`, ожидание `ready` до 60 с, ровно один `thumbnail` 256×160). Публичных проверок телеметрии нет: наблюдаемость доказывается своими скриптами и `evidence/`.
|
||||
- `tests/` байт в байт как в лабе 5. `contracts/openapi.json` отличается только `info.title` и `info.version`.
|
||||
- `contracts/observability.md` — нормативный набор сигналов (уровни «3» и «4»), resource attributes, запрещённые labels метрик и запрещённое содержимое telemetry, alerts и инциденты. `infra/otel/README.md` — что лежит в Git (Collector, storage, provisioning, dashboards, rules).
|
||||
- `contracts/infrastructure.json` — роли лабы 5 плюс `otel_collector`, `telemetry_storage` (метрики, логи, трассы, persistent volume) и `grafana` (provisioning в Git). `compose.example.yaml` — только worksheet ролей.
|
||||
|
||||
## Ключевые правила контракта (lab6)
|
||||
|
||||
- Resource attributes: `service.name` (`messenger-api`, `messenger-worker`, `messenger-relay`), `service.version`, `service.instance.id` (уникален на процесс: `<hostname>-<pid>`), `deployment.environment.name=lab`.
|
||||
- Метрики не получают labels `user_id`, `chat_id`, `message_id`, `attachment_id`, `request_id`, сырой URL, filename, текст; маршрут — шаблон (`/api/v1/chats/{chat_id}/messages`). HTTP duration — histogram в секундах (`http.server.request.duration`, `OTEL_SEMCONV_STABILITY_OPT_IN=http`).
|
||||
- Логи: timestamp, severity, `service.name`, `event`, `trace_id`/`span_id` при активном span. В telemetry никогда не попадают пароли, `Authorization`, токен сессии, тело сообщения, байты изображения, filename, ключи и подписанные URL; захват HTTP-заголовков выключен, Collector дополнительно редактирует подозрительные ключи.
|
||||
- Экспорт телеметрии никогда не блокирует запрос: только `Batch*Processor` и `PeriodicExportingMetricReader`, короткий таймаут OTLP. `/health/ready` не зависит от Collector, хранилищ телеметрии и Grafana. `OTEL_SDK_DISABLED=true` выключает SDK целиком.
|
||||
- Тело ошибки остаётся `{"error":{"code","message","request_id"}}` с `additionalProperties: false`: `trace_id` в тело и схемы не добавляется.
|
||||
- Сохраняются правила lab5 (outbox, `202 queued`, `ack` после `COMMIT`, ключ эффекта, `409` на `/content` до `ready`), lab4, lab3 (`X-Instance-Id`, `Idempotency-Key`), lab2 и lab1.
|
||||
|
||||
## Правила работы с репозиторием
|
||||
|
||||
- Не менять и не ослаблять `contracts/`, `tests/`, `README.md`, `COURSE.md`, `infra/README.md`, `infra/otel/README.md`, рубрику. Расхождение оформляется вопросом преподавателю.
|
||||
- Код — `src/messenger/`, миграции — `alembic/versions/`, async `alembic/env.py`, one-shot `alembic upgrade head`; работа с БД — только SQLAlchemy ORM/Core, asyncpg используется как драйвер, конфигурация LB, нагрузки, S3, брокера и наблюдаемости — `infra/` (`infra/otel`, `infra/prometheus`, `infra/loki`, `infra/tempo`, `infra/grafana`), хуки и обвязка стенда — `scripts/`, собственные тесты — `student-tests/`, обезличенные доказательства — `evidence/`.
|
||||
- Без комментариев в коде; отступы — пробелы (`[tool.ruff.format]`); Python 3.14 без `from __future__ import annotations` и `TYPE_CHECKING`.
|
||||
- Наружу публикуются только LB, UI брокера, Prometheus, Loki, Tempo и Grafana на `127.0.0.1`; порты реплик API, OTLP и self-metrics Collector не публикуются. Порты задаются переменными `.env`/shell: API 18086, HAProxy stats 26085, broker UI 26086, Prometheus 26090, Loki 26310, Tempo 26320, Grafana 26300. Для команд стенда используй `scripts/lab6.sh`, который согласует project, порты, URL проверок и NO_PROXY.
|
||||
- `REPORT.md`: команды из чистого clone, архитектура, таблица «критерий → команда → наблюдение → файл», таблица отказов, сопоставление сигналов `observability.md` реальным именам метрик. PASS без команды не доказательство.
|
||||
- Ветка `solution/bpi2303-kostylev`, коммиты на английском, перед каждым — `make lint`. Никогда не коммитить `.env`, `secrets/`, токены, пароли Grafana, ключи S3, пароли брокера, volumes, дампы.
|
||||
|
||||
## Уровни оценки
|
||||
|
||||
Решение заявлено на **«5»** с треком 1 «Цена наблюдаемости»; таблица доказательств — в `REPORT.md`.
|
||||
|
||||
- **3** — минимум lab5 работает при включённой телеметрии; Collector, Prometheus, Loki, Tempo и Grafana в общем стенде с volumes и provisioning из Git; API, relay и worker отдают OTLP; RPS/errors/duration API, jobs по исходу и duration worker; структурные логи с `trace_id`/`span_id`; реальные server и consumer spans; импортируемые dashboards; нет секретов и тел в telemetry; запрос → трасса и лог, job → трасса и лог worker.
|
||||
- **4** — W3C trace context API → outbox → broker → worker (producer/consumer), цепочка до S3/DB с retry; p50/p95/p99 из histogram, доля ошибок, queue depth и возраст старейшей job, DB pool, saturation worker, CPU/RAM; SLI/SLO, alert rules error rate и queue lag (firing → resolved); memory_limiter, batch, ограниченные очереди, отказ хранилищ телеметрии; два разобранных инцидента.
|
||||
- **5** — трек 1: telemetry off / on / head sampling / tail sampling / сбалансированная конфигурация при одинаковой open-loop нагрузке — latency с клиента, CPU/RAM вне телеметрии, объём ingest, потери spans и захват вброшенных ошибок и медленных запросов.
|
||||
|
||||
## Актуальное состояние и границы этапов
|
||||
|
||||
- Автор: Костылев Артём, БПИ2303. Итоговый отчёт — [REPORT.md](REPORT.md), индекс — [evidence/README.md](evidence/README.md).
|
||||
- Актуальный runtime и измеритель: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`; успешное независимое ревью: `6ca1c7549978af3127966540385195ab0587f2e1`. Поздние исправления ЛР5 доведены до immutable source `c7b8feb805d151e878d77b0dff47320a9436b172`, решения по путям — [forward-port.md](evidence/forward-port.md).
|
||||
- Текущий набор — `evidence/forward-published/`: 995 passed, 2 skipped, 19 deselected в сохранённом полном suite; 15 пригодных серий, 12000 запросов, balanced −92,2% медианного Tempo ingest, 48/48 error и 48/48 slow. Исторические `grade3/grade4/track5` не выдаются за измерение позднего кода. Незатронутые storage/history проверки переиспользуются только с основанием из `preserved-mechanisms.json`.
|
||||
- Пять minor после ревью — [review-summary.md](evidence/review-summary.md) и ограничения REPORT. Исправлять только major: доказанная угроза безопасности, потеря/повреждение данных, недоступность сервиса, неработающий обязательный сценарий. Minor сохранять, не блокировать ими отчёт/release; не объявлять неуспешный тест успешным.
|
||||
- Этапы запускает только runner: один агент переноса/уровней 3→4→5/evidence → review (отдельный fix major и re-review при необходимости) → отдельный отчёт → один агент нагрузки и публикации. Ревью/отчёт предыдущей лабы может идти параллельно реализации следующей. Самостоятельно фоновых агентов не создавать, runner/state не менять.
|
||||
- Обычные Docker-этапы используют отдельные project/порты/volumes совместно с другими лабами; эксклюзивный доступ получает только финальная нагрузка/release. Разрешён `mtusi2026lab6` в основном репозитории либо `hlpipe-lab6-*` в каталоге под ним или `/Users/artemkostylev/projects/learning/HL/labs/.pipeline/.runs/labs`. Чужие контейнеры не трогать; в конце остановить свои без удаления постоянных volumes. Не выполнять reset/clean и удаление пользовательских данных.
|
||||
- Фон и условия измерения фиксировать. Пригодные сохранённые серии переиспользовать после проверки SHA/diff; повторять лишь отсутствующие, неуспешные или признанные непригодными. Изменение только документации не требует новой нагрузки.
|
||||
- Отчётная фаза меняет только REPORT.md, AGENTS.md и Markdown внутри evidence; OpenSpec/config, код и raw measurements не меняет. `openspec/config.yaml` уже содержит актуальный стек; циклы propose/apply/archive закрыты, проверка — `openspec validate --all --strict --no-interactive`.
|
||||
- Публикация только в private MTUCI-Labs на явно назначенном этапе PUBLICATION. Самостоятельные push и Telegram вне назначенной фазы запрещены; для разрешённого Telegram — только `telegram-send`. ЛР1/2 заморожены; другие репозитории читать только через предоставленный snapshot.
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS builder
|
||||
|
||||
COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
|
||||
|
||||
ENV UV_COMPILE_BYTECODE=1 \
|
||||
UV_LINK_MODE=copy \
|
||||
UV_PYTHON_DOWNLOADS=never \
|
||||
UV_PROJECT_ENVIRONMENT=/opt/venv
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
COPY pyproject.toml uv.lock README.md ./
|
||||
RUN uv sync --locked --no-dev --no-install-project --no-editable
|
||||
|
||||
COPY src ./src
|
||||
RUN uv sync --locked --no-dev --no-editable
|
||||
|
||||
|
||||
FROM python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d AS runtime
|
||||
|
||||
ENV PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
PATH="/opt/venv/bin:$PATH" \
|
||||
HOST=0.0.0.0 \
|
||||
PORT=8080 \
|
||||
LOG_LEVEL=INFO \
|
||||
WEB_CONCURRENCY=2 \
|
||||
REQUEST_TIMEOUT_SECONDS=8 \
|
||||
DRAIN_SECONDS=2 \
|
||||
SHUTDOWN_TIMEOUT_SECONDS=6
|
||||
|
||||
RUN groupadd --system --gid 10001 messenger \
|
||||
&& useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
|
||||
|
||||
COPY --from=builder --chown=root:root /opt/venv /opt/venv
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --chown=root:root alembic.ini ./
|
||||
COPY --chown=root:root alembic ./alembic
|
||||
|
||||
USER 10001:10001
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD ["python", "-c", "import urllib.request,sys;sys.exit(0 if urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).status==200 else 1)"]
|
||||
|
||||
CMD ["python", "-m", "gunicorn", \
|
||||
"--config", "python:messenger.gunicorn_conf", \
|
||||
"messenger.main:create_app()"]
|
||||
@@ -1,17 +1,72 @@
|
||||
PYTHON ?= python3
|
||||
BASE_URL ?= http://localhost:8080
|
||||
BASE_URL ?= http://$(shell docker compose port lb 8080 2>/dev/null || echo localhost:8080)
|
||||
CA_FILE ?=
|
||||
ACTION_SCRIPT ?=
|
||||
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
|
||||
|
||||
.PHONY: help check validate test load test-persistence test-failover
|
||||
VENV ?= .venv
|
||||
UV ?= uv
|
||||
COMPOSE ?= docker compose
|
||||
RATE ?= 100
|
||||
FROM_REV ?=
|
||||
SHIFT ?= 0
|
||||
SCRIPTS_PY = scripts/obs_cost_summary.py scripts/obs_cost.py scripts/obs_incidents.py scripts/obs_recovery.py scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py \
|
||||
scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py \
|
||||
scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
|
||||
SCENARIO ?= failover-upload
|
||||
PSQL = $(COMPOSE) exec -T postgres psql -U messenger -d messenger -X -q
|
||||
OBS_CHECK = $(PYTHON) scripts/obs_check.py
|
||||
APP_METRICS = {job=~"messenger-.*"}
|
||||
|
||||
TRACK2_OUTPUT ?= evidence/track2.json
|
||||
TRACK2_SCENARIOS = smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version
|
||||
PIPELINE_VERSION ?= 2
|
||||
PYTEST_ARGS ?=
|
||||
|
||||
.PHONY: help check validate test load test-persistence test-failover \
|
||||
install lint format typecheck student-test \
|
||||
secrets up down reset-data migrate ps logs restore lb-check failover-evidence stack-evidence \
|
||||
persistence-evidence storage-evidence cache-evidence test-faults seed explain \
|
||||
limits detection failover-each rolling-update lab4-level4 \
|
||||
scenario s3-check attachments-evidence upload-faults upload-memory \
|
||||
images queues topology worker-stop worker-start broker-restart suite-preflight level3 \
|
||||
pipeline-evidence level4 replay failed dlq \
|
||||
obs-validate telemetry-check \
|
||||
faults-evidence retry-check reconcile reconcile-demo reconcile-cost load-evidence \
|
||||
clean-clone-evidence overload-leftovers
|
||||
|
||||
.NOTPARALLEL:
|
||||
|
||||
help:
|
||||
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
|
||||
@echo "install/lint/typecheck/student-test: студенческий тулчейн"
|
||||
@echo "secrets/up/down/migrate/ps/logs: стенд Docker Compose; reset-data: намеренное удаление volumes"
|
||||
@echo "restore: вернуть остановленные реплики; lb-check: реплики на одном токене и 20 гонок одного ключа"
|
||||
@echo "lab4-level4 = limits, lb-check, detection, failover-each, rolling-update [FROM_REV=<коммит>]; RATE=$(RATE) rps"
|
||||
@echo "scenario SCENARIO=failover-upload|s3-outage → evidence/<сценарий>.json; s3-check; attachments-evidence"
|
||||
@echo "upload-faults: негативный suite, компенсация, уборка, обрыв и зависание загрузки против стенда"
|
||||
@echo "upload-memory: пик памяти реплик при 10/20/40 одновременных загрузках 1 и 10 MiB → evidence/upload-memory/"
|
||||
@echo "lab5: queues, topology, worker-stop/worker-start, broker-restart, images, suite-preflight, level3, level4"
|
||||
@echo "lab5 оператор: failed (реестр), replay ID=<attachment_id>, dlq (просмотр DLQ без извлечения)"
|
||||
@echo "lab6: obs-validate (конфиги Collector, Prometheus, Loki, Tempo, dashboards); telemetry-check → evidence/*.json"
|
||||
|
||||
@echo "retry-check: потерянный ответ на живой и на убитой реплике, повтор тем же ключом"
|
||||
@echo "lab4-level4 = limits, lb-check, detection, failover-each, rolling-update [FROM_REV=<коммит>]; RATE=$(RATE) rps"
|
||||
@echo "scenario SCENARIO=failover-upload|s3-outage → evidence/<сценарий>.json; s3-check; attachments-evidence"
|
||||
@echo "upload-faults: негативный suite, компенсация, уборка, обрыв и зависание загрузки против стенда"
|
||||
@echo "upload-memory ARGS='--out evidence/upload-memory/runN': пик памяти реплик при 10/20/40 загрузках 1 и 10 MiB"
|
||||
@echo "load-evidence: tests/load.py ×3 клиентом-контейнером в сети edge стенда → evidence/load.txt, evidence/load/"
|
||||
@echo "overload-leftovers: после upload-memory — намерения от 503, сверка и уборка после grace → evidence/overload-leftovers.txt"
|
||||
@echo "clean-clone-evidence: блок «Запуск из чистого clone» в свежем клоне рядом → evidence/clean-clone.txt"
|
||||
@echo "reconcile ARGS='--apply --deep': сверка PostgreSQL ↔ S3 (без ARGS — dry-run); трек 2:"
|
||||
@echo "reconcile-demo: внедрение ошибок, dry-run/apply/повтор, безопасность, нагрузка → evidence/reconcile/"
|
||||
@echo "reconcile-cost: стоимость обхода 1k/10k/30k/100k объектов × 3 прогона на чистом стенде → evidence/reconcile/cost/"
|
||||
@echo "BASE_URL=$(BASE_URL) (порт lb из docker compose); failover-evidence SHIFT=1 — сдвиг round robin на вторую реплику"
|
||||
|
||||
check:
|
||||
$(PYTHON) scripts/check_contract.py
|
||||
|
||||
validate: PYTHON := $(if $(wildcard $(VENV)/bin/python),$(VENV)/bin/python,$(PYTHON))
|
||||
validate:
|
||||
$(PYTHON) scripts/validate_openapi.py
|
||||
|
||||
@@ -28,3 +83,226 @@ test-persistence:
|
||||
test-failover:
|
||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
|
||||
install:
|
||||
$(UV) sync --group dev
|
||||
|
||||
lint:
|
||||
$(VENV)/bin/ruff check src student-tests $(SCRIPTS_PY)
|
||||
$(VENV)/bin/ruff format --check src student-tests $(SCRIPTS_PY)
|
||||
$(VENV)/bin/ty check src student-tests alembic
|
||||
|
||||
format:
|
||||
$(VENV)/bin/ruff check --fix src student-tests $(SCRIPTS_PY)
|
||||
$(VENV)/bin/ruff format src student-tests $(SCRIPTS_PY)
|
||||
|
||||
typecheck:
|
||||
$(VENV)/bin/ty check src student-tests alembic
|
||||
|
||||
student-test:
|
||||
$(VENV)/bin/pytest $(PYTEST_ARGS)
|
||||
|
||||
secrets:
|
||||
sh scripts/secrets.sh
|
||||
sh scripts/broker-secrets.sh
|
||||
sh scripts/secrets-lab6.sh
|
||||
|
||||
up: secrets
|
||||
$(COMPOSE) build
|
||||
$(COMPOSE) up -d --wait
|
||||
scripts/lb-wait.sh
|
||||
|
||||
down:
|
||||
$(COMPOSE) --profile bench --profile tools --profile reconcile down
|
||||
|
||||
reset-data:
|
||||
@echo "Удаляются volumes с учебными данными стенда"
|
||||
$(COMPOSE) --profile bench --profile tools --profile reconcile down -v --remove-orphans
|
||||
|
||||
migrate:
|
||||
$(COMPOSE) run --rm migrate
|
||||
|
||||
ps:
|
||||
$(COMPOSE) ps
|
||||
|
||||
logs:
|
||||
$(COMPOSE) logs --tail 200
|
||||
|
||||
restore:
|
||||
scripts/restore.sh
|
||||
|
||||
lb-check:
|
||||
$(PYTHON) scripts/probe.py check --rounds 20 --out evidence/lb-check.json
|
||||
|
||||
retry-check:
|
||||
$(PYTHON) scripts/probe.py retry --out evidence/retry.json
|
||||
|
||||
limits:
|
||||
$(PYTHON) scripts/limits.py --out evidence/limits.txt
|
||||
|
||||
detection:
|
||||
rm -rf evidence/detection
|
||||
for run in 1 2 3; do sh scripts/detection.sh evidence/detection/run$$run || exit 1; done
|
||||
|
||||
failover-each:
|
||||
RATE=$(RATE) sh scripts/failover-each.sh evidence/failover-each
|
||||
|
||||
rolling-update:
|
||||
ifneq ($(FROM_REV),)
|
||||
git archive "$(FROM_REV)" | docker build -q -t "mtusi-messenger:$(FROM_REV)" -
|
||||
API_IMAGE="mtusi-messenger:$(FROM_REV)" $(COMPOSE) up -d --no-deps --force-recreate --wait api
|
||||
scripts/lb-wait.sh
|
||||
endif
|
||||
$(COMPOSE) build -q api
|
||||
RATE=$(RATE) $(PYTHON) scripts/probe.py load --expect replace --duration 90 \
|
||||
--out evidence/rolling$(if $(FROM_REV),-from-$(FROM_REV)) \
|
||||
-- scripts/rolling-update.sh
|
||||
|
||||
lab4-level4: limits lb-check detection failover-each rolling-update
|
||||
|
||||
scenario:
|
||||
$(PYTHON) scripts/lab4_scenarios.py $(SCENARIO) --base-url "$(BASE_URL)" --restore --output evidence/$(SCENARIO).json
|
||||
scripts/lb-wait.sh
|
||||
|
||||
s3-check:
|
||||
sh scripts/s3-check.sh > evidence/s3-security.txt; status=$$?; cat evidence/s3-security.txt; exit $$status
|
||||
|
||||
attachments-evidence:
|
||||
sh scripts/evidence-attachments.sh
|
||||
|
||||
upload-faults:
|
||||
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD) BASE_URL=$(BASE_URL)"; \
|
||||
BASE_URL="$(BASE_URL)" $(VENV)/bin/pytest -m stack -v -s --no-header --tb=short \
|
||||
-p no:cacheprovider student-tests/stack/test_attachment_faults.py; } > evidence/upload-faults.txt; \
|
||||
status=$$?; cat evidence/upload-faults.txt; exit $$status
|
||||
|
||||
upload-memory:
|
||||
$(PYTHON) scripts/upload_memory.py --base-url "$(BASE_URL)" $(ARGS)
|
||||
|
||||
load-evidence:
|
||||
sh scripts/load-evidence.sh
|
||||
|
||||
overload-leftovers:
|
||||
sh scripts/overload-leftovers.sh
|
||||
|
||||
clean-clone-evidence:
|
||||
sh scripts/clean-clone.sh
|
||||
|
||||
reconcile:
|
||||
$(COMPOSE) run --rm --no-deps reconciler $(ARGS)
|
||||
|
||||
reconcile-demo:
|
||||
$(PYTHON) scripts/reconcile_track.py demo --base-url "$(BASE_URL)" $(ARGS)
|
||||
|
||||
reconcile-cost:
|
||||
$(PYTHON) scripts/reconcile_track.py cost $(ARGS)
|
||||
|
||||
failover-evidence:
|
||||
sh scripts/failover-evidence.sh "$(BASE_URL)" "$(SHIFT)"
|
||||
|
||||
stack-evidence:
|
||||
sh scripts/evidence-stack.sh
|
||||
|
||||
persistence-evidence:
|
||||
sh scripts/persistence-evidence.sh "$(BASE_URL)" > evidence/persistence.txt
|
||||
cat evidence/persistence.txt
|
||||
|
||||
storage-evidence:
|
||||
sh scripts/evidence-storage.sh
|
||||
|
||||
cache-evidence:
|
||||
sh scripts/evidence-cache.sh > evidence/cache.txt
|
||||
cat evidence/cache.txt
|
||||
|
||||
test-faults:
|
||||
BASE_URL="$(BASE_URL)" $(VENV)/bin/pytest -m stack -v -s student-tests/stack
|
||||
|
||||
faults-evidence:
|
||||
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; echo "\$$ make test-faults"; \
|
||||
$(MAKE) --no-print-directory test-faults 2>&1; } | sed 's#$(CURDIR)#<repo>#g' > evidence/faults.txt
|
||||
cat evidence/faults.txt
|
||||
grep -qE '^=+ [0-9]+ passed in' evidence/faults.txt
|
||||
|
||||
seed:
|
||||
$(PSQL) -f - < scripts/sql/seed-history.sql
|
||||
|
||||
explain: seed
|
||||
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; \
|
||||
echo "\$$ make explain # scripts/sql/explain-history.sql"; \
|
||||
$(PSQL) -f - < scripts/sql/explain-history.sql; } > evidence/explain-history.txt
|
||||
cat evidence/explain-history.txt
|
||||
|
||||
images:
|
||||
$(PYTHON) scripts/make_images.py
|
||||
|
||||
queues:
|
||||
sh scripts/broker.sh queues
|
||||
|
||||
topology:
|
||||
{ echo "date: $$(date -u +%FT%TZ) commit: $$(git rev-parse --short HEAD)"; \
|
||||
echo "$$ rabbitmqctl list_queues -p messenger name type durable policy messages"; \
|
||||
$(COMPOSE) exec -T broker rabbitmqctl -q list_queues -p messenger name type durable policy messages; \
|
||||
echo "$$ rabbitmqctl list_consumers -p messenger"; \
|
||||
$(COMPOSE) exec -T broker rabbitmqctl -q list_consumers -p messenger queue_name ack_required prefetch_count; \
|
||||
echo "$$ rabbitmqctl list_permissions -p messenger"; \
|
||||
$(COMPOSE) exec -T broker rabbitmqctl -q list_permissions -p messenger; \
|
||||
echo "$$ rabbitmqctl list_users"; \
|
||||
$(COMPOSE) exec -T broker rabbitmqctl -q list_users; \
|
||||
echo "$$ docker compose ps"; \
|
||||
$(COMPOSE) ps --format '{{.Service}}\t{{.Status}}\t{{.Ports}}'; } > evidence/topology.txt
|
||||
cat evidence/topology.txt
|
||||
|
||||
worker-stop:
|
||||
sh scripts/worker.sh stop
|
||||
|
||||
worker-start:
|
||||
sh scripts/worker.sh start
|
||||
|
||||
broker-restart:
|
||||
sh scripts/broker.sh restart
|
||||
|
||||
suite-preflight:
|
||||
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --preflight
|
||||
|
||||
level3:
|
||||
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --level 3 --output evidence/level3.json
|
||||
|
||||
level4:
|
||||
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --level 4 --output evidence/level4.json
|
||||
|
||||
failed:
|
||||
$(COMPOSE) run --rm --no-deps -T worker python -m messenger.admin failed
|
||||
|
||||
replay:
|
||||
@test -n "$(ID)" || (echo "Set ID=<attachment_id>"; exit 2)
|
||||
$(COMPOSE) run --rm --no-deps -T worker python -m messenger.admin replay --attachment-id "$(ID)"
|
||||
|
||||
dlq:
|
||||
sh scripts/broker.sh dlq
|
||||
|
||||
pipeline-evidence:
|
||||
$(PYTHON) scripts/pipeline_evidence.py --base-url "$(BASE_URL)" > evidence/pipeline-stand.txt; \
|
||||
status=$$?; cat evidence/pipeline-stand.txt; exit $$status
|
||||
|
||||
obs-validate:
|
||||
sh scripts/obs-validate.sh
|
||||
|
||||
telemetry-check:
|
||||
$(OBS_CHECK) grafana > evidence/grafana.json
|
||||
$(OBS_CHECK) dashboards --strict > evidence/dashboards.json
|
||||
$(OBS_CHECK) request --base-url "$(BASE_URL)" > evidence/trace-request.json
|
||||
$(OBS_CHECK) job --base-url "$(BASE_URL)" > evidence/trace-job.json
|
||||
$(OBS_CHECK) leaks --base-url "$(BASE_URL)" > evidence/leaks.json
|
||||
$(OBS_CHECK) query 'count by (__name__, job) ($(APP_METRICS))' \
|
||||
'count by (http_route, http_request_method) (http_server_request_duration_seconds_count)' \
|
||||
'count by (job_type, outcome) (messenger_jobs_processed_total)' > evidence/cardinality.json
|
||||
@echo "telemetry-check: evidence/grafana.json dashboards.json trace-request.json trace-job.json leaks.json cardinality.json"
|
||||
|
||||
.PHONY: track2 reprocess
|
||||
|
||||
track2:
|
||||
BASE_URL="$(BASE_URL)" $(PYTHON) scripts/pipeline_suite.py --level 5 --only "$(TRACK2_SCENARIOS)" --output "$(TRACK2_OUTPUT)"
|
||||
|
||||
reprocess:
|
||||
@test -n "$(ID)" || (echo "Set ID=<attachment_id>"; exit 2)
|
||||
$(COMPOSE) run --rm --no-deps -T worker python -m messenger.admin reprocess --attachment-id "$(ID)" --pipeline-version "$(PIPELINE_VERSION)"
|
||||
|
||||
@@ -1,37 +1,281 @@
|
||||
# Отчёт — лабораторная 6
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
Автор: **Костылев Артём, БПИ2303**. Заявленная оценка — **«5»**, трек 1 «Цена наблюдаемости»: уровни 3 и 4 подтверждены таблицей ниже, пять режимов наблюдаемости — 15 пригодными сериями. Это заявка по рубрике, а не автоматически выставленная оценка.
|
||||
|
||||
Ветка: `solution/bpi2303-kostylev`. Проверенный ревью SHA: `6ca1c7549978af3127966540385195ab0587f2e1`; измеряемый код приложения, инфраструктуры и генератора: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`. Между ними эти файлы не менялись. Документационный коммит определяется через `git log -1 --format=%H -- REPORT.md`; публикация выполняется отдельным этапом, здесь не заявлены.
|
||||
|
||||
Актуальные результаты: [индекс evidence](evidence/README.md), [перенос поздних исправлений](evidence/forward-port.md), [сводка ревью](evidence/review-summary.md), [проверки отчёта](evidence/report-audit.md), [финальный аудит нагрузки](evidence/load-validation.md). Основа ЛР5 — `f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071`, поздний перенос доведён до immutable source `c7b8feb805d151e878d77b0dff47320a9436b172`; учтены 281 путь source diff. Результаты ЛР5 не выдаются за проверки ЛР6.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Git, Docker с Compose v2, uv, Python 3.14 для приложения и Python 3.9+ для stdlib-проверок. OpenSpec CLI нужен только для валидации спецификаций. Проверенный host: macOS arm64, 16 CPU; Docker 28.3.2, 16 vCPU и 8 217 190 400 bytes RAM. Версии зависимостей закреплены в [uv.lock](uv.lock), образы — тегом и digest в [Dockerfile](Dockerfile) и [compose.yaml](compose.yaml).
|
||||
|
||||
`REPOSITORY_URL` оператор задаёт адресом своего private fork в MTUCI-Labs. Команды ниже — инструкция воспроизведения; на этапе отчёта Docker и нагрузки не запускались.
|
||||
|
||||
```sh
|
||||
git clone "$REPOSITORY_URL" mtusi2026lab6
|
||||
cd mtusi2026lab6
|
||||
git switch -c solution/bpi2303-kostylev origin/main
|
||||
uv python install 3.14
|
||||
uv sync --locked --group dev
|
||||
make check validate lint
|
||||
openspec validate --all --strict --no-interactive
|
||||
scripts/lab6.sh make secrets
|
||||
scripts/lab6.sh docker compose config --quiet
|
||||
scripts/lab6.sh make up
|
||||
scripts/lab6.sh make ps
|
||||
curl --fail http://127.0.0.1:18086/health/ready
|
||||
scripts/lab6.sh make obs-validate
|
||||
scripts/lab6.sh make test
|
||||
scripts/lab6.sh make telemetry-check
|
||||
scripts/lab6.sh make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
scripts/lab6.sh make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
scripts/lab6.sh make restore
|
||||
scripts/lab6.sh make down
|
||||
```
|
||||
|
||||
`make secrets` создаёт локальные `.env` и `secrets/`, не печатая значения; существующие значения сохраняются. `make up` собирает образ и ждёт healthy: PostgreSQL → one-shot `alembic upgrade head`, S3 → `s3-init` (bucket, политики и отдельные роли), RabbitMQ → API ×2 / relay / worker → LB; параллельно поднимаются Collector и хранилища. Ручной SQL, ручное создание datasource/dashboard и внешний S3 не требуются. Повторный запуск миграций: `scripts/lab6.sh make migrate`.
|
||||
|
||||
Полные собственные тесты запускаются отдельно: `scripts/lab6.sh make student-test`. Обвязка задаёт обход localhost через `NO_PROXY`, в том числе при системном прокси macOS. Тесты создают одноразовые Docker-зависимости и не должны идти одновременно с измерениями трека. В отчётном этапе использован сохранённый результат; свежий запуск фазы validation описан в [финальном аудите](evidence/load-validation.md). Данные публичного API синтетические. Для повторных экспериментов сохраняйте вывод в новый каталог, не перезаписывая принятые evidence.
|
||||
|
||||
`make down` останавливает только выбранный Compose project и сохраняет volumes. Существующая цель `scripts/lab6.sh make reset-data` намеренно удаляет его учебные volumes; она не входит в обычный запуск и в отчётном этапе не выполнялась. Не останавливайте чужие контейнеры. Основной project — `mtusi2026lab6` в этом репозитории; временный — только `hlpipe-lab6-*` в разрешённом координатором каталоге, с отдельными портами/volumes.
|
||||
|
||||
### Порты и настройки
|
||||
|
||||
Все опубликованные порты привязаны к `127.0.0.1`; API-реплики, PostgreSQL, Valkey, S3, AMQP, OTLP и self-metrics Collector доступны только внутри стенда. [scripts/lab6.sh](scripts/lab6.sh) задаёт project, порты, URL проверок и `NO_PROXY`, затем выполняет переданную команду. Перед запуском оператор проверяет доступность портов и отсутствие конфликтующих переменных своего shell.
|
||||
|
||||
| Назначение | Переменная | Host port |
|
||||
| --- | --- | ---: |
|
||||
| API через HAProxy | `LB_HTTP_PORT` | 18086 |
|
||||
| HAProxy stats | `LB_STATS_PORT` | 26085 |
|
||||
| RabbitMQ management | `BROKER_MANAGEMENT_PORT` | 26086 |
|
||||
| Prometheus | `PROMETHEUS_PORT` | 26090 |
|
||||
| Loki | `LOKI_PORT` | 26310 |
|
||||
| Tempo query | `TEMPO_PORT` | 26320 |
|
||||
| Grafana | `GRAFANA_PORT` | 26300 |
|
||||
|
||||
Grafana: пользователь `admin`, пароль из локального `secrets/grafana_admin_password`; его значение не входит в отчёт. Provisioning создаёт datasources `prometheus`, `loki`, `tempo` и dashboards API/pipeline, переходы exemplar → trace и trace ↔ logs.
|
||||
|
||||
| Настройки из `.env.example` | Фактическое применение |
|
||||
| --- | --- |
|
||||
| `APP_HOST`, `APP_PORT`, `APP_ENV` | Логические поля skeleton; Compose не передаёт их приложению. Образ слушает `HOST=0.0.0.0`, `PORT=8080`; окружение OTel задаёт `deployment.environment.name=lab` |
|
||||
| `SESSION_STORE=postgres`, `CACHE_URL_FILE`, `SESSION_TTL_SECONDS` | PostgreSQL — источник сессий; Valkey — адаптер кэша. Переключателя backend через `SESSION_STORE` нет; TTL по умолчанию 3600 s |
|
||||
| `DATABASE_URL_FILE`, `CURSOR_SECRET_FILE`, S3 `*_FILE` | Compose монтирует роли в `/run/secrets`; SQLAlchemy переводит DSN в `postgresql+asyncpg` |
|
||||
| `BROKER_URL_FILE` | В relay и worker одинаковый путь `/run/secrets/broker_url`, но разные secrets `broker_url_api` / `broker_url_worker` и права |
|
||||
| `API_REPLICAS`, `WEB_CONCURRENCY`, `DB_POOL_MAX_SIZE` | 2 реплики ×2 процесса, пул по 8 соединений на процесс; relay/worker по `PIPELINE_DB_POOL_MAX_SIZE=4` |
|
||||
| `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS`, `WORKER_CONCURRENCY` | 10 MiB, 40 MP, 2 задания; render timeout 30 s, лимит адресного пространства renderer 448 MiB |
|
||||
| `OTEL_*`, `TELEMETRY_SHUTDOWN_SECONDS` | OTLP/HTTP, timeout 2 s, метрики каждые 10 s, очереди SDK по 2048, bounded shutdown 2 s; SDK off через `OTEL_SDK_DISABLED=true` |
|
||||
| `OTEL_TRACE_RATIO`, `OTEL_COLLECTOR_CONFIG`, `COST_EXPERIMENT` | Дополнительные Compose-параметры: по умолчанию 1, `collector.yaml`, false. Генератор включает fault hook лишь на серию и восстанавливает default в `finally` |
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Client[Клиент] --> LB[HAProxy]
|
||||
LB --> API[API ×2]
|
||||
API --> PG[(PostgreSQL)]
|
||||
API --> Cache[(Valkey)]
|
||||
API --> S3[(RustFS / S3)]
|
||||
PG --> Relay[Outbox relay]
|
||||
Relay --> Broker[(RabbitMQ)]
|
||||
Broker --> Worker[FastStream worker]
|
||||
Worker --> PG
|
||||
Worker --> S3
|
||||
API -. OTLP .-> Collector[OTel Collector]
|
||||
Relay -. OTLP .-> Collector
|
||||
Worker -. OTLP .-> Collector
|
||||
Collector --> Prom[(Prometheus)]
|
||||
Collector --> Loki[(Loki)]
|
||||
Collector --> Tempo[(Tempo)]
|
||||
Grafana[Grafana] --> Prom
|
||||
Grafana --> Loki
|
||||
Grafana --> Tempo
|
||||
```
|
||||
|
||||
Python 3.14; SQLAlchemy **2.0.45 async**, Alembic **1.16.5**, FastAPI, dishka, structlog, FastStream **0.7.8** + aio-pika **10.2.1**; OTel SDK 1.45.1 и инструментации 0.66b1. Инфраструктура: HAProxy 3.2.26, PostgreSQL 17.11, Valkey 8.1.11, RustFS 1.0.4, RabbitMQ 4.1.8, otelcol-contrib 0.163.2, Prometheus 3.15.1, Loki 3.7.9, Tempo 2.10.10, Grafana 12.4.14. Фактические image IDs, non-root users и лимиты: [runtime.json](evidence/forward-published/runtime.json).
|
||||
|
||||
Зоны `identity`, `messaging`, `attachments` разделены на domain/application/infrastructure/entrypoints; домен зависит от портов. `shared` содержит только платформу. `main.py`, `relay.py`, `worker.py` собирают процессы. PostgreSQL хранит пользователей, сессии, сообщения, состояния вложений, jobs и outbox; RustFS — байты, RabbitMQ — доставку, Valkey — ускорение чтения. Prometheus хранит численные временные ряды, Loki — структурные логи, Tempo — spans. Collector принимает/фильтрует/буферизует сигналы, Grafana запрашивает их и связывает; ни один из них не источник бизнес-данных.
|
||||
|
||||
Все runtime-запросы PostgreSQL проходят через SQLAlchemy ORM/Core: `DeclarativeBase`, `create_async_engine`, `async_sessionmaker`, отдельная `AsyncSession` на UoW. `asyncpg` остаётся драйвером и точкой инструментации. Миграции — [alembic/env.py](alembic/env.py) и [alembic/versions](alembic/versions): async engine, `run_sync`, advisory lock, legacy adoption, транзакция на ревизию, head `0007` для trace context. Старого исполняемого SQL-мигратора в инструкции запуска нет.
|
||||
|
||||
Upload возвращает `202 queued` после durable acceptance; задание и outbox фиксируются атомарно. Relay публикует persistent сообщение с confirms через FastStream/aio-pika, затем отмечает outbox. Неустранимое окно между publish и отметкой допускает повторную доставку. Worker использует manual ack **после COMMIT**; lease/fencing и ключ эффекта `(attachment_id, pipeline_version, variant)` не допускают повторной публикации результата. До ready `/content` отвечает 409. Retry/backoff, DLQ, реестр failed и replay сохранены. Цена outbox — дополнительный poll/таблица и eventual consistency; брокер не даёт exactly-once.
|
||||
|
||||
ЛР6 добавила SDK после fork (без preload приложения в gunicorn), OTLP, корреляцию, resource attributes и наблюдение фоновых задач. `traceparent`/`tracestate` сохраняются отдельно от payload в jobs/outbox и передаются в headers; producer/consumer spans создаёт `RabbitTelemetryMiddleware`. S3 spans показывают операцию без ключей, SQL client spans — работу БД. Retry сохраняет причинную связь, что видно в одной цепочке из 86 spans.
|
||||
|
||||
Экспорт выполняют `BatchSpanProcessor`, `BatchLogRecordProcessor`, `PeriodicExportingMetricReader`; network timeout короткий, readiness не проверяет наблюдаемость. Collector: memory limiter 75% / spike 20%, batch 1024/2048 с timeout 2 s, очереди 20 000 spans, 20 000 logs, 50 000 metric points, 2 consumers, retry до 60 s. Буферы конечны и не долговременны: при длительном отказе telemetry теряется, бизнес-запросы продолжаются. `Logger` ограничивает поля, HTTP headers не захватываются, Collector редактирует подозрительные ключи/значения. Границы проверки утечек явно указаны ниже.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Команды в таблице выполняются с префиксом `scripts/lab6.sh`, если работают со стендом. Полные команды, UTC, SHA и exit code сохранены в указанных логах. **Актуальный набор — `forward-published`**; исторические исключения помечены явно и применяются только к неизменным механизмам согласно [preserved-mechanisms.json](evidence/forward-published/preserved-mechanisms.json).
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Критерий | Команда / сценарий | Наблюдение | Файл |
|
||||
| --- | --- | --- | --- |
|
||||
| Структура, контракт, качество | `make check validate lint` | OpenAPI 1.6.0 / JSON Schema, ruff, 216 форматированных файлов, ty — PASS | [final-quality.txt](evidence/forward-published/final-quality.txt), [свежая проверка отчёта](evidence/report-audit.md) |
|
||||
| Собственные регрессии, SQLAlchemy/Alembic | `scripts/lab6.sh make student-test` | Сохранённый полный: 995 passed, 2 Linux-only skipped, 19 stack deselected. Свежий validation: 993 passed, 2 ошибки подготовки S3; адресный повтор с NO_PROXY — 2 passed | [прежний полный](evidence/forward-published/student-test.txt), [свежий полный](evidence/validation/student-test.txt), [повтор S3](evidence/validation/student-test-s3-retry.txt), [разбор](evidence/load-validation.md) |
|
||||
| 3: минимум lab5 при telemetry on | `make test` | Public API PASS; final smoke после возврата defaults | [smoke.txt](evidence/forward-published/smoke.txt), [final-smoke.txt](evidence/forward-published/final-smoke.txt) |
|
||||
| 3: durable pipeline и повторы | `python3 scripts/pipeline_suite.py --level 4 --only smoke_pipeline,worker_down_broker_restart,duplicate_after_ready,poison_image,s3_outage_recovers,retry_exhausted,crash_after_s3_before_commit,crash_after_commit_before_ack,malformed_message_dead_lettered --output evidence/reproduction-pipeline.json` | 9 сценариев; 7 принятых заданий достигают ожидаемых состояний, single effect и другие инварианты PASS | [pipeline.txt](evidence/forward-published/pipeline.txt), [pipeline.json](evidence/forward-published/pipeline.json) |
|
||||
| 3: сохранение данных и сессий | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | Подтверждённые данные и прежняя сессия сохранены, 14,386 s на сценарий | [persistence.txt](evidence/forward-published/persistence.txt), [hook](scripts/restart.sh) |
|
||||
| 3: отказ реплики и возврат | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`; `make restore` | 2 исходные реплики, 130 успешных записей, 0 failed attempts; первый успех через 2,876 s, api-2 возвращена healthy | [failover.txt](evidence/forward-published/failover.txt), [restore.txt](evidence/forward-published/restore.txt) |
|
||||
| 3: воспроизводимый общий стенд | `make up`; `make obs-validate` | Сервисы готовы, Collector/Prometheus/Loki/Tempo и JSON dashboards валидны | [up.txt](evidence/forward-published/up.txt), [obs-config.txt](evidence/forward-published/obs-config.txt), [runtime.json](evidence/forward-published/runtime.json) |
|
||||
| 3: история наблюдаемости в volumes | `python3 scripts/obs_recovery.py` | Исторический прогон: 7 spans до/после, 1 лог, значение метрики 352 сохранено при рестарте storage | [telemetry-recovery.txt](evidence/grade3/telemetry-recovery.txt), [основание применимости](evidence/forward-published/preserved-mechanisms.json) |
|
||||
| 3: API → лог → trace, реальные server/client spans | `python3 scripts/obs_check.py request --base-url http://127.0.0.1:18086` | `request.handled`, trace/span IDs; 7 spans, включая HTTP server и DB client | [telemetry-request.txt](evidence/forward-published/telemetry-request.txt) |
|
||||
| 3: worker → лог → trace, consumer spans | `python3 scripts/obs_check.py job --base-url http://127.0.0.1:18086` | Attachment ready, 52 spans, `job.processed` / `audit.job_settled` с одним trace_id | [telemetry-job.txt](evidence/forward-published/telemetry-job.txt) |
|
||||
| 3: RED API, jobs outcome/duration, provisioning | `python3 scripts/obs_check.py dashboards --strict` | 32 targets, problems=[], реальные series; два dashboard JSON в Git | [dashboards.txt](evidence/forward-published/dashboards.txt), [dashboards](infra/grafana/dashboards) |
|
||||
| 3: низкая кардинальность / resource attributes | `python3 scripts/obs_check.py query 'count by (__name__, job) ({job=~"messenger-.*"})'` | Шаблон маршрута, method/status, job_type/outcome, service/instance; IDs не используются как labels | [исторические series неизменных инструментов](evidence/grade4/cardinality-series.json), [свежие dashboard queries](evidence/forward-published/dashboards.txt) |
|
||||
| 3: отсутствие запрещённых данных | `python3 scripts/obs_check.py leaks --base-url http://127.0.0.1:18086` | 0 совпадений password/token/text/filename в проверенных log lines и 8 traces за 20 s; metadata Loki этим тестом не покрыта | [telemetry-leaks.txt](evidence/forward-published/telemetry-leaks.txt), [minor M1](evidence/review-summary.md) |
|
||||
| 4: W3C API → outbox → broker → worker → S3/DB | `python3 scripts/obs_incidents.py chain` | 55 spans, producer/consumer, итог done | [incident-chain.txt](evidence/forward-published/incident-chain.txt) |
|
||||
| 4: та же цепочка с retry | `python3 scripts/obs_incidents.py retry` | 86 spans; `retry_scheduled` → `done`, один trace | [incident-retry.txt](evidence/forward-published/incident-retry.txt) |
|
||||
| 4: p50/p95/p99, error ratio, backlog, pool, saturation, CPU/RAM, time-to-ready | `python3 scripts/obs_check.py dashboards --strict` | Все используемые запросы разрешаются; имена/единицы и ограничения приведены ниже | [dashboards.txt](evidence/forward-published/dashboards.txt), [recording rules](infra/prometheus/rules/messenger.recording.yml) |
|
||||
| 4: SLI/SLO, error rate alert | `python3 scripts/obs_incidents.py database` | SQL client ERROR 2002,866 ms, error ratio 0,9181, firing → resolved | [incident-database.txt](evidence/forward-published/incident-database.txt), [alert rules](infra/prometheus/rules/messenger.alerts.yml) |
|
||||
| 4: queue lag alert и восстановление | `python3 scripts/obs_incidents.py worker` | 8 jobs, age 34,470 s, firing → resolved, depth 8 → 0 | [incident-worker.txt](evidence/forward-published/incident-worker.txt) |
|
||||
| 4: ограниченные очереди, отказ telemetry storage | `python3 scripts/obs_incidents.py storage` | Исторический неизменный механизм: smoke PASS, 74 успешных probes, потеря 2 traces, RAM ≤252,7/256 MiB, очереди после recovery =0 | [storage-incident.txt](evidence/grade4/storage-incident.txt), [основание применимости](evidence/forward-published/preserved-mechanisms.json) |
|
||||
| 4: readiness и shutdown при отказе Collector | `python3 scripts/obs_recovery.py` | Исторический прогон: ready=200, smoke PASS, shutdown API 4,592 s, exit 0 | [telemetry-recovery.txt](evidence/grade3/telemetry-recovery.txt) |
|
||||
| Поздние исправления сессий/HTTP | `.venv/bin/python student-tests/tools/forward_network.py`; stack pytest из логов | Негативный Basic/NUL, logout обеих реплик, идемпотентность; Valkey bypass и logout503→204 | [network.txt](evidence/forward-published/network.txt), [cache-outage.txt](evidence/forward-published/cache-outage.txt) |
|
||||
| Поздние исправления доставки/изображений | `env PYTHONPATH=.:scripts:student-tests .venv/bin/python student-tests/tools/forward_runtime.py postgres-outage` (также `png-http`, `png-boundary`) | 12 requeue без роста DLQ, тот же job/attempt; PNG 40 MP, две версии, конкурентный peak 913 227 776 <1 073 741 824 bytes, OOM=0 | [postgres-outage.json](evidence/forward-published/postgres-outage.json), [png-http.json](evidence/forward-published/png-http.json), [png-boundary.json](evidence/forward-published/png-boundary.json) |
|
||||
| 5: off/on/head/tail/balanced, цена и сохранность | `python3 scripts/obs_cost.py --mode MODE --repeat N --duration 40 --rate 20 --output evidence/reproduction/MODE-N.json` | 15 пригодных серий, 12 000 запросов; balanced −92,2% Tempo ingest, 48/48 error и 48/48 slow | [методика](evidence/forward-published/cost/method.md), [comparison.json](evidence/forward-published/cost/comparison.json), [результаты](evidence/forward-published/cost/results.md) |
|
||||
| OpenSpec propose/apply/archive | `openspec list --json`; `openspec validate --all --strict --no-interactive` | 0 active changes, 26 specs PASS; циклы 3/4/5 и поздний перенос архивированы | [архивы](openspec/changes/archive), [spec-sync.json](evidence/forward-published/spec-sync.json), [проверка отчёта](evidence/report-audit.md) |
|
||||
|
||||
`restart.sh` останавливает `lb api relay worker`, перезапускает `postgres valkey s3 broker`, затем ждёт Compose healthy с прежними volumes. Публичный тест подавляет stdout hook: 14,386 s — время всего сценария, а не измеренная длительность недоступности каждого процесса; отдельного timestamp-аудита всех перезапусков в этом логе нет. `restore.txt` непосредственно подтверждает запуск остановленной api-2. История telemetry проверена отдельным `obs_recovery.py`. Старый запуск из чистого clone на `2903d9d` — [clean-clone.txt](evidence/grade3/clean/clean-clone.txt); он не является свежим clean-clone прогоном позднего кода.
|
||||
|
||||
### Сигналы и запросы
|
||||
|
||||
Resource: `service.name=messenger-api|messenger-relay|messenger-worker`, `service.version=1.6.0`, уникальный `service.instance.id=<hostname>-<pid>`, `deployment.environment.name=lab`. В Prometheus ресурс виден через `job`, `instance` и `target_info`; в Loki — нормализованные поля, включая `service_name`. Технические request/job IDs допустимы в логах/spans для корреляции синтетических данных, но не в labels метрик.
|
||||
|
||||
| Логический сигнал контракта | Реальное имя / поля | Единицы и измерения / запрос |
|
||||
| --- | --- | --- |
|
||||
| HTTP requests | `http_server_request_duration_seconds_count` | counter наблюдений histogram; `http_request_method`, `http_route`, `http_response_status_code`; `sum(messenger:http_requests:rate2m)` — RPS; recording rule формирует `status_class` |
|
||||
| HTTP duration | `http_server_request_duration_seconds_bucket` | секунды; `histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{job="messenger-api",http_route!~"/health/.*"}[5m])))`; для p50/p99 заменить 0.95 |
|
||||
| HTTP errors | `messenger:http_errors:ratio_rate2m` | 5xx / все бизнес-запросы, без health; 4xx остаются в знаменателе, не считаются отказами |
|
||||
| Jobs processed / failed | `messenger_jobs_processed_total` | доставки, `job_type`, `outcome`; `sum by(job_type,outcome)(rate(messenger_jobs_processed_total[5m]))`; outcome содержит done/retry_scheduled/failed и другие ветки |
|
||||
| Job duration | `messenger_job_duration_seconds_bucket` | секунды обработки доставки, `job_type`, `outcome`; percentile из суммы bucket rates |
|
||||
| Time to ready | `messenger_job_time_to_ready_seconds_bucket` | секунды от Job.created_at до done после commit, `job_type`; учитывает ожидание и retries |
|
||||
| Queue depth / oldest age | `messenger_queue_depth`, `messenger_queue_oldest_age_seconds` | jobs / секунды, `queue_name=attachments.process`; SQLAlchemy poll 5 s, durable queued/processing включая leased/retry, не только AMQP messages_ready |
|
||||
| DB pool | `messenger_db_pool_used`, `_waiting`, `_capacity` | занятые connections / acquisitions / лимит; waiting включает получение/создание соединения, не только очередь слотов |
|
||||
| Worker saturation | `messenger_worker_busy`, `messenger_worker_capacity` | активные deliveries / доступная capacity; отношение сумм даёт долю занятости |
|
||||
| Process CPU / RAM | `process_cpu_time_seconds_total`, `process_memory_usage_bytes` | `sum by(job)(rate(process_cpu_time_seconds_total[2m]))` — cores; `sum by(job)(process_memory_usage_bytes)` — bytes RSS; CPU renderer-детей не входит |
|
||||
| Logs | timestamp, severity, resource service.name, event, trace_id, span_id | `request.handled`, `job.processed`; LogQL `{service_name="messenger-api"} | trace_id="<trace_id>"`; поля OTel представлены structured metadata |
|
||||
| Traces | SERVER, PRODUCER, CONSUMER, CLIENT | Tempo по trace_id; DB/S3 операции, duration/status, связь parent; ссылки из Grafana provisioning |
|
||||
|
||||
HTTP histogram (секунды): 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10, +Inf. Job/time-to-ready: 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 30, 60, +Inf. Они различают быстрый HTTP, таймауты зависимостей и 30-секундный SLO job. Квантили агрегируются по buckets всех реплик; среднее арифметическое p99 реплик не используется. CPU/RAM трека дополнительно измерены извне по cgroups, независимо от состояния SDK.
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
### SLI/SLO и диагностика
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
API availability: `1 − 5xx/все бизнес-HTTP` за **1 h**, цель **≥99%**, health исключён. Это SLI приложения: ответы самого HAProxy не входят. Job SLI: доля **завершённых** jobs с time-to-ready **≤30 s** за **1 h**, цель **≥95%**; незавершённые отдельно контролируются возрастом backlog. Эти цели заданы и измеримы, но короткие опыты не доказывают месячный production SLO.
|
||||
|
||||
`MessengerHighErrorRate`: доля 5xx >5% за 1 min, `for: 15s`. `MessengerQueueLag`: max age >15 s и depth >0, `for: 15s`. Проверка rules каждые 5 s; доставка наружу не требуется. [Runbooks](infra/prometheus/runbooks.md) связывают alert с запросами и действиями.
|
||||
|
||||
| Воздействие и доказательство | Ожидание и наблюдение | Время / потери |
|
||||
| --- | --- | --- |
|
||||
| Остановка worker; [свежий лог](evidence/forward-published/incident-worker.txt) | API принимает 8 jobs, backlog=8, oldest=34,470 s; после возврата depth=0, alert resolved | UTC 06.11: stop 23:47:05, firing 23:47:49 (**44 s**), restore 23:47:52, resolved 23:48:04 (**12 s** после возврата). Все 8 jobs обработаны |
|
||||
| Ограниченная SQL-блокировка users; [свежий лог](evidence/forward-published/incident-database.txt) | 168 контролируемых 503; error ratio=0,9181, pool waiting=0; SELECT client ERROR длится 2002,866 ms | UTC 06.11: lock 23:48:16, firing/снятие lock 23:48:59 (**43 s**), resolved 23:49:54 (**55 s**). Блокировка сама данные не меняет; полноценный RPO здесь не измерялся |
|
||||
| S3 временно недоступен; [retry](evidence/forward-published/incident-retry.txt) | Одна цепочка из 86 spans показывает retry_scheduled → done, затем DB commit | Сценарий 23:46:51–23:47:04 UTC (≈13 s); принятое вложение ready, отдельное время unavailable не выделено |
|
||||
| PostgreSQL недоступен worker; [JSON](evidence/forward-published/postgres-outage.json) | 12 requeue, delivery-limit=-1; job_id и attempt=1 сохранены, DLQ 0→0, после восстановления done без replay | Targeted case 20,39 s; это время кейса, не точный RTO. Потери принятого задания не обнаружены |
|
||||
| Остановка одной API-реплики; [failover](evidence/forward-published/failover.txt) | 130 writes, 0 failed attempts, подтверждённые данные и сессия сохранились | Первый успешный запрос после воздействия через 2,876 s; restore завершился за 3,660 s |
|
||||
| Рестарт приложений и business stores; [persistence](evidence/forward-published/persistence.txt) | Сессия и подтверждённые данные сохранены; volumes не удаляются | 14,386 s на весь сценарий; отдельный RTO не измерен |
|
||||
| Остановка Prometheus/Loki/Tempo; **исторический** [storage](evidence/grade4/storage-incident.txt), неизменные механизмы | Smoke PASS, 74 успешных probes; queue logs/metrics/traces=381/1287/465, после recovery 0/0/0; Collector без наблюдаемого OOM | UTC 06.11: 18:14:29–18:16:16 (**107 s**), проверка закончилась 18:16:39. 72/74 traces найдены, **2 потеряны**; RAM 252,7/256 MiB (98,7%) |
|
||||
| Остановка Collector / рестарт telemetry stores; **исторический** [recovery](evidence/grade3/telemetry-recovery.txt) | Ready=200, smoke PASS, ранее сохранённая история доступна | API shutdown 4,592 s, exit 0; 7/7 прежних spans и лог найдены после рестарта |
|
||||
|
||||
Первый инцидент диагностируется по растущим depth/age при отсутствии обработки: после возврата worker backlog исчезает, значит узкое место — consumer. Во втором 503 server span содержит PostgreSQL client ERROR около 2 s, при этом pool waiting=0: причина в выполнении SQL, а не в очереди пула или просто медленном HTTP handler. Снятие lock и последующий resolved подтверждают гипотезу. Это разбор сохранённых наблюдений контролируемых отказов; слепой runbook-прогон другого оператора (трек 2) не заявлен.
|
||||
|
||||
При отказе telemetry storage retry ограничен 60 s. В сохранённых counters есть 440 failed span exports, 11 log records и 72 metric points; эти единицы не равны числу потерянных запросов/полных трасс. Непосредственный контроль trace IDs показал 2 потерянные трассы из 74 успешных запросов. Рост памяти ограничен наблюдаемым окном и лимитом контейнера; запаса Collector почти нет, бесконечная устойчивость не доказана.
|
||||
|
||||
### Трек 1: цена наблюдаемости
|
||||
|
||||
Гипотеза: head sampling снижает ingest, но теряет важные события; tail sampling оставляет ERROR/slow и уменьшает запись в Tempo, сохраняя расходы SDK. Сбалансированный режим должен сохранить контрольные события при меньшей записи обычных traces и более редком экспорте метрик.
|
||||
|
||||
Измеряемый SHA: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`, image ID `sha256:66c7d75903f514e14fea21689d039eeba1d9d85a0d27ef3d70d4ab41288e2f6d`. Серии выполнены 06–07.11.2026 UTC. Нагрузка — GET `/api/v1/users/me` по одному созданному для серии пользователю/сессии; это HTTP/DB-профиль, не массовый рендер изображений. Существующие volumes между режимами сохраняются, полный размер БД не нормировался. Условия, host load и список контейнеров сохранены в [conditions-start.json](evidence/forward-published/cost/conditions-start.json), снимках каждого круга и raw JSON. В снимках других Docker projects нет; эксклюзивность всего host не заявляется.
|
||||
|
||||
| Режим | SDK / выбор traces | Метрики |
|
||||
| --- | --- | --- |
|
||||
| off | SDK полностью выключен | Нет метрик приложения; self-scrapes storage продолжаются |
|
||||
| on | Все server roots и их дочерние spans | Каждые 10 s |
|
||||
| head | ParentBased root ratio 10% | Каждые 10 s |
|
||||
| tail | Все spans до Collector; сохранить ERROR, latency ≥300 ms, 10% остальных | Каждые 10 s |
|
||||
| balanced | Все spans до Collector; ERROR, latency ≥300 ms, 5% остальных | Каждые 30 s |
|
||||
|
||||
По три повтора каждого режима, последовательно off/on/head/tail/balanced. 20 прогревочных GET и 35 s ожидания экспорта; затем **40 s open-loop, 20 arrivals/s, 800 запросов**, максимум 64 клиентских потока. В каждой серии 768 обычных, 16 ожидаемых 503 и 16 задержек 350 ms; после — 35 s drain. Всего **12 000 запросов**, 11 760 ожидаемых 200 и 240 намеренных 503, **0 неожиданных ответов**. Фактическая скорость 20,012–20,020 req/s; максимум schedule lag 61,055 ms. Это заданная нагрузка, не поиск предельного throughput.
|
||||
|
||||
Команда серии и разбор всех параметров — [method.md](evidence/forward-published/cost/method.md). CPU: delta cgroup CPU time / длительность внешних samples; RAM: working set usage−inactive_file, 21 sample с периодом около 2 s. Application = API/relay/worker; observability = Collector/Prometheus/Loki/Tempo/Grafana; business stores/LB считаются отдельно. CPU-окно охватывает нагрузку, но не всю работу tail во время drain. Ingest включает drain; bytes Tempo/Loki и samples Prometheus не складываются в одну единицу.
|
||||
|
||||
Медианы трёх повторов; диапазон normal p95 — min/max, не доверительный интервал. Общие p50/p95/p99 посчитаны клиентом по всем категориям, поэтому p99 отражает вброшенную задержку.
|
||||
|
||||
| Режим | p50 / p95 / p99, ms | normal p95, ms [min;max] | CPU app / obs / stores+LB, cores | RAM app / obs / stores+LB, MiB (median peak) |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| off | 8,052 / 12,102 / 358,732 | 11,583 [11,572;11,856] | 0,1404 / 0,0256 / 0,1085 | 655,5 / 1013,3 / 419,9 |
|
||||
| on | 7,460 / 13,148 / 359,469 | 12,709 [12,554;12,993] | 0,1726 / 0,0287 / 0,1030 | 706,8 / 1058,5 / 425,5 |
|
||||
| head | 9,195 / 13,116 / 358,373 | 12,556 [12,354;12,652] | 0,1657 / 0,0261 / 0,1023 | 723,8 / 1076,1 / 347,1 |
|
||||
| tail | 8,942 / 13,281 / 359,109 | 12,781 [12,555;12,920] | 0,1717 / 0,0272 / 0,1026 | 721,0 / 1021,2 / 430,6 |
|
||||
| balanced | 8,269 / 13,230 / 359,753 | 12,895 [12,833;13,031] | 0,1749 / 0,0258 / 0,1030 | 707,1 / 1027,9 / 423,1 |
|
||||
|
||||
Фактические лимиты: API 2×(1 CPU/512 MiB), relay 0,5 CPU/192 MiB, worker 2 CPU/1 GiB; Collector 0,5/256 MiB, Prometheus 0,5/512 MiB, Loki/Tempo/Grafana каждый 0,5/384 MiB; PostgreSQL 2/768 MiB, S3 1/512 MiB, RabbitMQ 1/512 MiB, Valkey 0,5/128 MiB, LB 1/128 MiB. **PG_MEMORY в измерении 768m, в `.env.example` — 1g**: для повторения именно этих условий задайте `PG_MEMORY=768m` до `make up`. Остальные значения сверяйте с [runtime.json](evidence/forward-published/runtime.json).
|
||||
|
||||
| Режим | Tempo / Loki, KiB (median) | Prom samples (median) | SDK ended / sent spans (median) | Найдено normal /2304 | error /48 | slow /48 |
|
||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| off | 0,0 / 0,0 | 33118 | 0 / 0 | 0 | 0 | 0 |
|
||||
| on | 1662,2 / 288,3 | 36284 | 5489 / 5489 | 2304 | 48 | 48 |
|
||||
| head | 176,2 / 288,4 | 37234 | 543 / 543 | 237 | 6 | 3 |
|
||||
| tail | 251,8 / 289,7 | 36657 | 5492 / 699 | 235 | 48 | 48 |
|
||||
| balanced | 130,2 / 289,8 | 34459 | 5486 / 352 | 101 | 48 | 48 |
|
||||
|
||||
Во всех пригодных сериях **SDK ended = Collector accepted**, exporter failures=0. Head намеренно не создал 42/48 error и 45/48 slow server traces; число несозданных дочерних spans неизвестно и не выдумывается. Tail/balanced фильтруют уже созданные spans; accepted−sent здесь не означает неисправность доставки. Off означает отключение SDK. В режиме on найдены все 2400 server traces трёх повторов.
|
||||
|
||||
**Balanced снизил медианный Tempo ingest на 92,2% относительно on**, сохранив 48/48 error и 48/48 slow. Loki почти не изменился; Prometheus включает постоянный фон self-scrapes. Normal p95 у on выше off примерно на 1,127 ms (+9,7%), CPU приложения — 0,1726 против 0,1404 cores; это описательные разности данного опыта, не доказательство статистической значимости. Balanced не устраняет цену SDK и не показал ускорения HTTP. Проверенная рекомендация относится к сокращению записи traces при этом профиле и окне решения tail 10 s.
|
||||
|
||||

|
||||
|
||||
[Сырые серии и comparison.json](evidence/forward-published/cost/comparison.json), [сводка](evidence/forward-published/cost/results.md), [аудит пригодности](evidence/forward-published/cost/measurement-audit.json). Первоначальные on-3/head-3 (ещё 1600 запросов) исключены из сравнения из-за разрывов wall clock: 932,714 s после нагрузки и 938,154 s вместо 40 s, gap samples до 900,354 s. Их HTTP PASS не означает пригодность CPU/ingest. Оригиналы с hashes сохранены в [invalid-series.json](evidence/forward-published/cost/invalid-series.json) и [invalid-time-gap](evidence/forward-published/cost/invalid-time-gap); повторены только эти две серии. Причина разрыва времени достоверно не установлена.
|
||||
|
||||
Проверенный запуск balanced:
|
||||
|
||||
```sh
|
||||
OTEL_COLLECTOR_CONFIG=collector-balanced.yaml OTEL_METRIC_EXPORT_INTERVAL=30000 scripts/lab6.sh docker compose up -d --no-deps --wait otel-collector api relay worker
|
||||
```
|
||||
|
||||
После опытов возвращены SDK on/ratio1/interval10s, `collector.yaml`, `COST_EXPERIMENT=false`: [defaults-restored.json](evidence/forward-published/defaults-restored.json). Стенд остановлен без удаления данных, 9 volumes сохранены: [shutdown.txt](evidence/forward-published/shutdown.txt), [closure.json](evidence/forward-published/closure.json). Это зафиксированное состояние окончания реализации, не новое обращение отчётного этапа к Docker.
|
||||
|
||||
### Финальная validation перед публикацией
|
||||
|
||||
На входном HEAD `1e00d861a803a6d49ef443175b8231cbf048fc02` аудит подтвердил применимость всех 15 серий: runtime и измеритель совпадают с прошедшим ревью кодом. Все raw JSON и comparison пересчитаны, временные окна и hashes исключённых оригиналов проверены. **Нагрузку повторно не запускали.** [Аудит](evidence/load-validation.md) содержит решение по каждой серии, конфигурации, dataset, фону и ограничениям.
|
||||
|
||||
Свежие `make check validate lint` и OpenSpec strict (26 specs) прошли. Свежий полный `make student-test` завершился с 993 passed, 2 skipped, 19 deselected и 2 ошибками подготовки RustFS. В окружении был системный прокси macOS без обхода localhost; только два затронутых теста повторены штатной обвязкой с `NO_PROXY`: 2 passed. Полный первый запуск не объявляется PASS; оба лога сохранены. Код и прежние сырые измерения не менялись; публикация в фазе validation не выполнялась.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Ревью `6ca1c7549978af3127966540385195ab0587f2e1` не выявило major. Пять minor сохранены; по правилу пользователя они не блокируют отчёт/release и в документационном этапе не исправляются:
|
||||
|
||||
| ID | Ограничение и последствие | Файлы / доказательства |
|
||||
| --- | --- | --- |
|
||||
| M1 | Loki leak-check ищет маркеры через line filter, но не во всех attributes/structured metadata. Нулевые hits — ограниченная проверка, не полный аудит отсутствия секретов. Фактическая утечка не обнаружена | [obs_check.py](scripts/obs_check.py), [telemetry.py](src/messenger/shared/infrastructure/telemetry.py), [leaks](evidence/forward-published/telemetry-leaks.txt) |
|
||||
| M2 | Унаследованные allowlists `make limits` и published-port audit CI не учитывают OTel/helper services и отклоняют штатную ЛР6. Весь CI не объявляется зелёным; лимиты проверены отдельно | [limits.py](scripts/limits.py), [workflow](.github/workflows/application.yml), [runtime](evidence/forward-published/runtime.json) |
|
||||
| M3 | API SLI не учитывает 503 HAProxy, rate новой серии может пропустить первый всплеск. Полная пауза БД не вызвала alert; успешен именно отдельный SQL-lock сценарий | [alerts](infra/prometheus/rules/messenger.alerts.yml), [неуспешный старый опыт](evidence/grade4/database-incident.txt), [свежий SQL-lock](evidence/forward-published/incident-database.txt) |
|
||||
| M4 | Collector достиг 98,7% лимита (252,7/256 MiB) без наблюдаемого OOM. Запас мал; опыт не доказывает устойчивость при любой нагрузке/длительности отказа | [collector.yaml](infra/otel/collector.yaml), [compose](compose.yaml), [storage](evidence/grade4/storage-incident.txt) |
|
||||
| M5 | Whitespace в SVG и сыром stdout: ревью получило `git show --check` exit 2, 2320 замечаний. Это косметика артефактов; `make lint` исходников проходит | [comparison.svg](evidence/forward-published/cost/comparison.svg), [whitespace-check.txt](evidence/forward-published/whitespace-check.txt) |
|
||||
|
||||
Полный текст и происхождение замечаний — [review-summary.md](evidence/review-summary.md). Улучшения охвата leak-check, CI allowlists, edge SLI и запаса памяти остаются будущей работой; результат их реализации не заявлен.
|
||||
|
||||
Дополнительные границы: один host, одиночные PostgreSQL/RabbitMQ/S3/Collector и локальные volumes, без HA этих компонентов и disaster recovery. Сбой узла и удаление volumes не покрыты. TLS/mTLS, внешние receivers alerts и multi-window burn-rate относятся к другим задачам. Три повтора, фиксированный порядок, общий host и накопленная история storage ограничивают выводы о CPU/RAM/latency; CPU во время drain и непрерывный пик RAM не измерены. Tail window 10 s не гарантирует полноту очень поздних retry. Queue gauge — durable backlog; time-to-ready SLI не включает pending; process CPU не включает renderer. Full suite имеет 2 macOS skip и 19 deselected stack, поэтому дополнен targeted Docker evidence, но не выдаётся за исполнение всех тестов.
|
||||
|
||||
Старые `grade3`, `grade4`, `track5`, `level3.json` привязаны к своим SHA в [индексе](evidence/README.md); прежние 519 passed и −91,1% Tempo не являются результатом позднего кода. Неуспешные/прерванные smoke, retry, persistence, тесты и измерения сохранены и размечены в исторических notes. Новый clean-clone и динамический прогон в отчётном этапе не выполнялись. Сырые evidence, OpenSpec и реализация не менялись. Конфигурация OpenSpec уже актуальна; новая proposal/apply/archive для документационного этапа не создавалась по его явному ограничению на изменения.
|
||||
|
||||
## Вопросы защиты
|
||||
|
||||
1. **Чем отличаются три сигнала?** Метрика показывает рост error rate/backlog, лог даёт событие и технический ID, trace показывает длительность зависимостей и причинную цепочку.
|
||||
2. **Почему нельзя chat_id в labels?** Число series растёт с чатами и комбинациями labels; память/индекс хранилища расходуются даже при малом RPS. Нужен route template; ID оставляют для выборочного поиска логов/traces.
|
||||
3. **Как связаны API и worker?** W3C carrier сохраняется вместе с durable job/outbox, затем идёт в AMQP headers; consumer извлекает parent, retry сохраняет связь. Сбой relay не стирает контекст из БД.
|
||||
4. **Почему ack после COMMIT?** Ранний ack может потерять работу при падении до commit; поздний допускает повтор, который нейтрализуют effect key и fencing.
|
||||
5. **Почему SQLAlchemy и Alembic?** ORM/Core задают единые транзакции/блокировки и пул, Alembic — упорядоченный head с lock/legacy adoption. OTel asyncpg инструментация наблюдает драйвер под SQLAlchemy, не заменяя его.
|
||||
6. **Почему среднее не заменяет p99?** Редкая задержка скрывается средним; p99 реплик нельзя усреднять. Сначала складываются bucket rates, затем вычисляется quantile.
|
||||
7. **Как отличили DB от медленного handler?** В сохранённом trace SELECT client ERROR около 2 s внутри server 503, pool waiting=0; снятие SQL lock убрало причину и затем alert.
|
||||
8. **Почему head потерял ошибки, а balanced сохранил?** Head решает до исхода, tail — после получения ERROR/duration. Цена tail — создание всех spans и ограниченное ожидание в памяти Collector.
|
||||
9. **Почему наблюдаемость может теряться без потери сообщения?** Её очереди и retry конечны и отделены от бизнес-коммита; 2/74 traces потеряны при продолжающем работать API.
|
||||
10. **Что доказано треком?** Сокращение записи в Tempo на 92,2% с сохранением контрольных событий при 20 req/s. Production capacity, статистическое ускорение и полнота всех поздних retry не доказаны.
|
||||
|
||||
## Источники и помощь
|
||||
|
||||
Нормы: [README](README.md), [контракт наблюдаемости](contracts/observability.md), [API](contracts/openapi.json), [COURSE](COURSE.md). Архитектура портов/UoW — [конспект Персиваля—Грегори](openspec/notes/architecture-patterns-python.md). Код и проверенные конфигурации версионируются в этом репозитории.
|
||||
|
||||
SQLAlchemy async sessions/engine и Alembic async `run_sync` сверены с [SQLAlchemy 2.0](https://docs.sqlalchemy.org/en/20/orm/extensions/asyncio.html) и [Alembic cookbook](https://alembic.sqlalchemy.org/en/latest/cookbook.html#using-asyncio-with-alembic). Context7 в этом этапе ответил `Monthly quota exceeded`, поэтому использована официальная документация; закреплённые версии проекта не обновлялись. Остальные первичные материалы OTel, W3C, Prometheus и Grafana перечислены в README; фактические имена/настройки сверены с локальным кодом и evidence.
|
||||
|
||||
Codex использован как инструмент подготовки реализации, проверок, анализа и отчёта; автор работы — Костылев Артём. Независимое ревью получено отдельным этапом runner. Динамические результаты в этом документе — сохранённые измерения исполнителей, а не повторные прогоны агента отчёта.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
[alembic]
|
||||
script_location = %(here)s/alembic
|
||||
prepend_sys_path = %(here)s/src %(here)s/alembic
|
||||
path_separator = space
|
||||
+102
@@ -0,0 +1,102 @@
|
||||
import asyncio
|
||||
|
||||
from alembic import context
|
||||
from alembic.script import ScriptDirectory
|
||||
from legacy import adopt_legacy
|
||||
from sqlalchemy import Connection, func, make_url, pool, select
|
||||
from sqlalchemy.ext.asyncio import create_async_engine
|
||||
|
||||
from messenger.attachments.infrastructure.postgres import (
|
||||
AttachmentRow,
|
||||
JobEventRow,
|
||||
JobRow,
|
||||
OutboxRow,
|
||||
VariantRow,
|
||||
)
|
||||
from messenger.identity.infrastructure.postgres import SessionRow, UserRow
|
||||
from messenger.messaging.infrastructure.postgres import (
|
||||
ChatMemberRow,
|
||||
ChatRow,
|
||||
IdempotencyKeyRow,
|
||||
MessageAttachmentRow,
|
||||
MessageRow,
|
||||
)
|
||||
from messenger.shared.config import PlatformSettings
|
||||
|
||||
LOCK_KEY = 20_260_002
|
||||
MODELS = (
|
||||
UserRow,
|
||||
SessionRow,
|
||||
ChatRow,
|
||||
ChatMemberRow,
|
||||
MessageRow,
|
||||
IdempotencyKeyRow,
|
||||
AttachmentRow,
|
||||
JobRow,
|
||||
OutboxRow,
|
||||
JobEventRow,
|
||||
VariantRow,
|
||||
MessageAttachmentRow,
|
||||
)
|
||||
target_metadata = UserRow.metadata
|
||||
config = context.config
|
||||
|
||||
|
||||
def include_object(_obj, name, _type, _reflected, _compare_to):
|
||||
return name != "schema_migrations"
|
||||
|
||||
|
||||
def run_migrations(connection: Connection) -> None:
|
||||
with connection.engine.connect() as lock:
|
||||
lock.execute(select(func.pg_advisory_lock(LOCK_KEY)))
|
||||
lock.commit()
|
||||
try:
|
||||
with connection.begin():
|
||||
adopt_legacy(connection, ScriptDirectory.from_config(config))
|
||||
context.configure(
|
||||
connection=connection,
|
||||
target_metadata=target_metadata,
|
||||
transaction_per_migration=True,
|
||||
compare_type=True,
|
||||
include_object=include_object,
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
finally:
|
||||
lock.execute(select(func.pg_advisory_unlock(LOCK_KEY)))
|
||||
lock.commit()
|
||||
|
||||
|
||||
async def run_async_migrations() -> None:
|
||||
dsn = config.attributes.get("database_url")
|
||||
if dsn is None:
|
||||
dsn = PlatformSettings().database_url.get_secret_value()
|
||||
engine = create_async_engine(
|
||||
make_url(dsn).set(drivername="postgresql+asyncpg"),
|
||||
poolclass=pool.NullPool,
|
||||
hide_parameters=True,
|
||||
connect_args={
|
||||
"timeout": 10,
|
||||
"server_settings": {"lock_timeout": "10000", "statement_timeout": "60000"},
|
||||
},
|
||||
)
|
||||
try:
|
||||
async with engine.connect() as connection:
|
||||
await connection.run_sync(run_migrations)
|
||||
finally:
|
||||
await engine.dispose()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
context.configure(
|
||||
dialect_name="postgresql",
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
transaction_per_migration=True,
|
||||
)
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
elif config.attributes.get("connection") is not None:
|
||||
run_migrations(config.attributes["connection"])
|
||||
else:
|
||||
asyncio.run(run_async_migrations())
|
||||
@@ -0,0 +1,94 @@
|
||||
from uuid import uuid4
|
||||
|
||||
from alembic.migration import MigrationContext
|
||||
from alembic.operations import Operations
|
||||
from alembic.script import ScriptDirectory
|
||||
from sqlalchemy import Connection, MetaData, Table, func, inspect, select, text
|
||||
from sqlalchemy.schema import CreateSchema
|
||||
|
||||
LEGACY = {
|
||||
"0001_init": "9a13618f078ae9056431391e2e5197a31b981ae1fff047a9b3421ac750d3fd0d",
|
||||
"0002_idempotency_keys": "d9159a35553249a1e51af5f2e4e7c39b014e97593c49748e13f3799839063dd0",
|
||||
"0003_attachments": "c4417f560538a083efba1c67f660147dc94532fcbb2fa91f3f3e8258a0f01f2e",
|
||||
"0004_jobs": "c3887fa7a886ae8153a216aa43fd4c3c6190a2074e8ce20c3e77f29bfb01550e",
|
||||
"0005_failed_registry": "02b80061a08a6929feb092148c453ad970e3d001901c651b5a73b90b1e40f145",
|
||||
"0006_attachment_quarantine": (
|
||||
"541f5399d87fac3584c4102b4e0bf92c23cd372cc20a7d56dad03e771773202c"
|
||||
),
|
||||
}
|
||||
SNAPSHOT = text("""
|
||||
WITH relations AS (
|
||||
SELECT c.* FROM pg_class c JOIN pg_namespace n ON n.oid = c.relnamespace
|
||||
WHERE n.nspname = :schema AND c.relkind IN ('r', 'p')
|
||||
AND c.relname NOT IN ('schema_migrations', 'alembic_version')
|
||||
)
|
||||
SELECT 'table' AS kind, c.relname AS name,
|
||||
concat_ws('|', c.relkind, c.relpersistence,
|
||||
c.relrowsecurity, c.relforcerowsecurity) AS definition
|
||||
FROM relations c
|
||||
UNION ALL
|
||||
SELECT 'column', c.relname || '.' || a.attname,
|
||||
concat_ws('|', a.attnum, format_type(a.atttypid, a.atttypmod), a.attnotnull,
|
||||
a.attidentity, a.attgenerated, a.attcollation::regcollation,
|
||||
pg_get_expr(d.adbin, d.adrelid))
|
||||
FROM relations c JOIN pg_attribute a ON a.attrelid = c.oid
|
||||
LEFT JOIN pg_attrdef d ON d.adrelid = c.oid AND d.adnum = a.attnum
|
||||
WHERE a.attnum > 0 AND NOT a.attisdropped
|
||||
UNION ALL
|
||||
SELECT 'constraint', c.relname || '.' || k.conname,
|
||||
concat_ws('|', pg_get_constraintdef(k.oid), k.convalidated)
|
||||
FROM relations c JOIN pg_constraint k ON k.conrelid = c.oid
|
||||
UNION ALL
|
||||
SELECT 'index', c.relname || '.' || x.relname,
|
||||
concat_ws('|', pg_get_indexdef(i.indexrelid), i.indisvalid, i.indisready)
|
||||
FROM relations c JOIN pg_index i ON i.indrelid = c.oid
|
||||
JOIN pg_class x ON x.oid = i.indexrelid
|
||||
UNION ALL
|
||||
SELECT 'trigger', c.relname || '.' || t.tgname,
|
||||
concat_ws('|', pg_get_triggerdef(t.oid), t.tgenabled)
|
||||
FROM relations c JOIN pg_trigger t ON t.tgrelid = c.oid WHERE NOT t.tgisinternal
|
||||
ORDER BY kind, name
|
||||
""")
|
||||
|
||||
|
||||
def snapshot(connection: Connection, schema: str) -> list[tuple[str, str, str]]:
|
||||
return [
|
||||
(kind, name, definition.replace(f"{schema}.", ""))
|
||||
for kind, name, definition in connection.execute(SNAPSHOT, {"schema": schema})
|
||||
]
|
||||
|
||||
|
||||
def adopt_legacy(connection: Connection, scripts: ScriptDirectory) -> None:
|
||||
migration = MigrationContext.configure(connection)
|
||||
if migration.get_current_heads():
|
||||
return
|
||||
tables = set(inspect(connection).get_table_names(schema="public"))
|
||||
if tables <= {"alembic_version"}:
|
||||
return
|
||||
if "schema_migrations" not in tables:
|
||||
raise RuntimeError("Existing schema has no legacy migration journal")
|
||||
journal = Table("schema_migrations", MetaData(), schema="public", autoload_with=connection)
|
||||
applied = dict(connection.execute(select(journal.c.version, journal.c.checksum)).tuples().all())
|
||||
expected = dict(list(LEGACY.items())[: len(applied)])
|
||||
if applied != expected:
|
||||
raise RuntimeError("Unknown or changed legacy migrations")
|
||||
if not applied:
|
||||
if tables - {"schema_migrations", "alembic_version"}:
|
||||
raise RuntimeError("Existing schema has an empty migration journal")
|
||||
return
|
||||
revision = list(expected)[-1]
|
||||
actual = snapshot(connection, "public")
|
||||
reference = f"alembic_verify_{uuid4().hex}"
|
||||
savepoint = connection.begin_nested()
|
||||
try:
|
||||
connection.execute(CreateSchema(reference))
|
||||
connection.execute(select(func.set_config("search_path", reference, True)))
|
||||
with Operations.context(MigrationContext.configure(connection)):
|
||||
for script in reversed(list(scripts.walk_revisions(base="base", head=revision))):
|
||||
script.module.upgrade()
|
||||
wanted = snapshot(connection, reference)
|
||||
finally:
|
||||
savepoint.rollback()
|
||||
if actual != wanted:
|
||||
raise RuntimeError("Legacy schema differs from the recorded migrations")
|
||||
migration.stamp(scripts, revision)
|
||||
@@ -0,0 +1,16 @@
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
revision = ${repr(up_revision)}
|
||||
down_revision = ${repr(down_revision)}
|
||||
branch_labels = ${repr(branch_labels)}
|
||||
depends_on = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -0,0 +1,87 @@
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0001_init"
|
||||
down_revision = None
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"users",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("username", sa.Text(), nullable=False),
|
||||
sa.Column("display_name", sa.Text(), nullable=False),
|
||||
sa.Column("password_hash", sa.Text(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.CheckConstraint("password_hash LIKE '$argon2id$%'", name="users_password_hash_check"),
|
||||
sa.CheckConstraint("username ~ '^[a-z0-9_]{3,32}$'", name="users_username_format"),
|
||||
sa.CheckConstraint(
|
||||
"char_length(display_name) BETWEEN 1 AND 100", name="users_display_name_check"
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("username", name="users_username_key"),
|
||||
)
|
||||
op.create_table(
|
||||
"sessions",
|
||||
sa.Column("token_hash", sa.LargeBinary(), nullable=False),
|
||||
sa.Column("user_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.CheckConstraint("expires_at > created_at", name="sessions_check"),
|
||||
sa.CheckConstraint("octet_length(token_hash) = 32", name="sessions_token_hash_check"),
|
||||
sa.ForeignKeyConstraint(["user_id"], ["users.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("token_hash"),
|
||||
)
|
||||
op.create_index("sessions_user_idx", "sessions", ["user_id"], unique=False)
|
||||
op.create_table(
|
||||
"chats",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("title", sa.Text(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("last_message_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.CheckConstraint("char_length(title) BETWEEN 1 AND 100", name="chats_title_check"),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_table(
|
||||
"chat_members",
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("user_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("position", sa.SmallInteger(), nullable=False),
|
||||
sa.Column("chat_created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.CheckConstraint("position BETWEEN 1 AND 100", name="chat_members_position_check"),
|
||||
sa.ForeignKeyConstraint(["chat_id"], ["chats.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(
|
||||
["user_id"],
|
||||
["users.id"],
|
||||
),
|
||||
sa.PrimaryKeyConstraint("chat_id", "user_id"),
|
||||
)
|
||||
op.create_index(
|
||||
"chat_members_user_idx",
|
||||
"chat_members",
|
||||
["user_id", "chat_created_at", "chat_id"],
|
||||
unique=False,
|
||||
)
|
||||
op.create_table(
|
||||
"messages",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("sender_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("text", sa.Text(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.CheckConstraint("char_length(text) BETWEEN 1 AND 4000", name="messages_text_check"),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id", "sender_id"],
|
||||
["chat_members.chat_id", "chat_members.user_id"],
|
||||
),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
)
|
||||
op.create_index(
|
||||
"messages_history_idx", "messages", ["chat_id", "created_at", "id"], unique=False
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
raise RuntimeError("Downgrade would delete application data")
|
||||
@@ -0,0 +1,33 @@
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0002_idempotency_keys"
|
||||
down_revision = "0001_init"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"idempotency_keys",
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("user_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("key", sa.Text(), nullable=False),
|
||||
sa.Column("fingerprint", sa.LargeBinary(), nullable=False),
|
||||
sa.Column("message_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.CheckConstraint("key ~ '^[A-Za-z0-9._:-]{1,128}$'", name="idempotency_keys_key_check"),
|
||||
sa.CheckConstraint(
|
||||
"octet_length(fingerprint) = 32", name="idempotency_keys_fingerprint_check"
|
||||
),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id", "user_id"],
|
||||
["chat_members.chat_id", "chat_members.user_id"],
|
||||
),
|
||||
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
|
||||
sa.PrimaryKeyConstraint("chat_id", "user_id", "key"),
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
raise RuntimeError("Downgrade would delete application data")
|
||||
@@ -0,0 +1,77 @@
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0003_attachments"
|
||||
down_revision = "0002_idempotency_keys"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.drop_constraint("messages_text_check", "messages", type_="check")
|
||||
op.create_check_constraint("messages_text_check", "messages", "char_length(text) <= 4000")
|
||||
op.create_table(
|
||||
"attachments",
|
||||
sa.Column("id", sa.Uuid(), nullable=False),
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("owner_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("filename", sa.Text(), nullable=False),
|
||||
sa.Column("content_type", sa.Text(), nullable=False),
|
||||
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
|
||||
sa.Column("sha256", sa.Text(), nullable=False),
|
||||
sa.Column("width", sa.Integer(), nullable=False),
|
||||
sa.Column("height", sa.Integer(), nullable=False),
|
||||
sa.Column("object_key", sa.Text(collation="C"), nullable=False),
|
||||
sa.Column("status", sa.Text(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("id"),
|
||||
sa.UniqueConstraint("object_key", name="attachments_object_key_key"),
|
||||
sa.UniqueConstraint("id", "chat_id", name="attachments_chat_key"),
|
||||
sa.ForeignKeyConstraint(
|
||||
["chat_id", "owner_id"],
|
||||
["chat_members.chat_id", "chat_members.user_id"],
|
||||
name="attachments_owner_fkey",
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"char_length(filename) BETWEEN 1 AND 255", name="attachments_filename_check"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"content_type IN ('image/png', 'image/jpeg')", name="attachments_content_type_check"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"size_bytes BETWEEN 1 AND 10485760", name="attachments_size_bytes_check"
|
||||
),
|
||||
sa.CheckConstraint("sha256 ~ '^[0-9a-f]{64}$'", name="attachments_sha256_check"),
|
||||
sa.CheckConstraint("width > 0", name="attachments_width_check"),
|
||||
sa.CheckConstraint("height > 0", name="attachments_height_check"),
|
||||
sa.CheckConstraint("status IN ('pending', 'ready')", name="attachments_status_check"),
|
||||
)
|
||||
op.create_index(
|
||||
"attachments_pending_idx",
|
||||
"attachments",
|
||||
["created_at"],
|
||||
postgresql_where=sa.text("status = 'pending'"),
|
||||
)
|
||||
op.create_table(
|
||||
"message_attachments",
|
||||
sa.Column("message_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("position", sa.SmallInteger(), nullable=False),
|
||||
sa.Column("chat_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("attachment_id", sa.Uuid(), nullable=False),
|
||||
sa.PrimaryKeyConstraint("message_id", "position"),
|
||||
sa.UniqueConstraint("message_id", "attachment_id"),
|
||||
sa.CheckConstraint("position BETWEEN 0 AND 9", name="message_attachments_position_check"),
|
||||
sa.ForeignKeyConstraint(["message_id"], ["messages.id"], ondelete="CASCADE"),
|
||||
sa.ForeignKeyConstraint(
|
||||
["attachment_id", "chat_id"],
|
||||
["attachments.id", "attachments.chat_id"],
|
||||
name="message_attachments_attachment_fkey",
|
||||
ondelete="RESTRICT",
|
||||
),
|
||||
)
|
||||
op.create_index("message_attachments_attachment_idx", "message_attachments", ["attachment_id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
raise RuntimeError("Downgrade would delete application data")
|
||||
@@ -0,0 +1,146 @@
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects import postgresql
|
||||
|
||||
revision = "0004_jobs"
|
||||
down_revision = "0003_attachments"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.drop_constraint("attachments_status_check", "attachments", type_="check")
|
||||
op.create_check_constraint(
|
||||
"attachments_status_check",
|
||||
"attachments",
|
||||
"status IN ('pending', 'queued', 'processing', 'ready', 'failed')",
|
||||
)
|
||||
for name in ("staging_key", "original_key"):
|
||||
op.add_column("attachments", sa.Column(name, sa.Text(collation="C"), nullable=True))
|
||||
op.create_unique_constraint(f"attachments_{name}_key", "attachments", [name])
|
||||
op.add_column(
|
||||
"attachments",
|
||||
sa.Column("published_version", sa.Integer(), nullable=False, server_default="0"),
|
||||
)
|
||||
op.create_check_constraint(
|
||||
"attachments_published_version_check", "attachments", "published_version >= 0"
|
||||
)
|
||||
op.add_column("attachments", sa.Column("error_code", sa.Text(), nullable=True))
|
||||
op.create_check_constraint(
|
||||
"attachments_error_code_check",
|
||||
"attachments",
|
||||
"error_code IN ('invalid_image', 'processing_failed', 'retry_exhausted')",
|
||||
)
|
||||
op.alter_column("attachments", "object_key", nullable=True)
|
||||
attachments = sa.table(
|
||||
"attachments",
|
||||
sa.column("status"),
|
||||
sa.column("object_key"),
|
||||
sa.column("original_key"),
|
||||
sa.column("staging_key"),
|
||||
sa.column("published_version"),
|
||||
)
|
||||
op.execute(
|
||||
attachments.update()
|
||||
.where(attachments.c.status == "ready")
|
||||
.values(original_key=attachments.c.object_key, published_version=1)
|
||||
)
|
||||
op.execute(
|
||||
attachments.update()
|
||||
.where(attachments.c.status == "pending")
|
||||
.values(staging_key=attachments.c.object_key)
|
||||
)
|
||||
op.create_index(
|
||||
"attachments_staged_idx",
|
||||
"attachments",
|
||||
["updated_at"],
|
||||
postgresql_where=sa.column("staging_key").is_not(None),
|
||||
)
|
||||
op.create_table(
|
||||
"attachment_variants",
|
||||
sa.Column("attachment_id", sa.Uuid(), sa.ForeignKey("attachments.id"), nullable=False),
|
||||
sa.Column("pipeline_version", sa.Integer(), nullable=False),
|
||||
sa.Column("name", sa.Text(), nullable=False),
|
||||
sa.Column("object_key", sa.Text(collation="C"), nullable=False),
|
||||
sa.Column("content_type", sa.Text(), nullable=False),
|
||||
sa.Column("size_bytes", sa.BigInteger(), nullable=False),
|
||||
sa.Column("width", sa.Integer(), nullable=False),
|
||||
sa.Column("height", sa.Integer(), nullable=False),
|
||||
sa.Column("sha256", sa.Text(), nullable=False),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.PrimaryKeyConstraint("attachment_id", "pipeline_version", "name"),
|
||||
sa.UniqueConstraint("object_key", name="attachment_variants_object_key_key"),
|
||||
sa.CheckConstraint(
|
||||
"pipeline_version >= 1", name="attachment_variants_pipeline_version_check"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"name IN ('thumbnail', 'crop', 'optimized')", name="attachment_variants_name_check"
|
||||
),
|
||||
sa.CheckConstraint(
|
||||
"content_type IN ('image/jpeg', 'image/png', 'image/webp')",
|
||||
name="attachment_variants_content_type_check",
|
||||
),
|
||||
sa.CheckConstraint("size_bytes >= 1", name="attachment_variants_size_bytes_check"),
|
||||
sa.CheckConstraint("width >= 1", name="attachment_variants_width_check"),
|
||||
sa.CheckConstraint("height >= 1", name="attachment_variants_height_check"),
|
||||
sa.CheckConstraint("sha256 ~ '^[0-9a-f]{64}$'", name="attachment_variants_sha256_check"),
|
||||
)
|
||||
op.create_table(
|
||||
"jobs",
|
||||
sa.Column("job_id", sa.Uuid(), primary_key=True),
|
||||
sa.Column("attachment_id", sa.Uuid(), sa.ForeignKey("attachments.id"), nullable=False),
|
||||
sa.Column("pipeline_version", sa.Integer(), nullable=False),
|
||||
sa.Column("status", sa.Text(), nullable=False),
|
||||
sa.Column("attempt", sa.Integer(), nullable=False),
|
||||
sa.Column("claims", sa.Integer(), nullable=False, server_default="0"),
|
||||
sa.Column("lease_token", sa.Uuid(), nullable=True),
|
||||
sa.Column("lease_until", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("next_attempt_at", sa.DateTime(timezone=True), nullable=True),
|
||||
sa.Column("last_error", sa.Text(), nullable=True),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.UniqueConstraint("attachment_id", "pipeline_version", name="jobs_effect_key"),
|
||||
sa.CheckConstraint("pipeline_version >= 1", name="jobs_pipeline_version_check"),
|
||||
sa.CheckConstraint(
|
||||
"status IN ('queued', 'processing', 'done', 'failed', 'superseded')",
|
||||
name="jobs_status_check",
|
||||
),
|
||||
sa.CheckConstraint("attempt >= 1", name="jobs_attempt_check"),
|
||||
sa.CheckConstraint("claims >= 0", name="jobs_claims_check"),
|
||||
)
|
||||
op.create_index(
|
||||
"jobs_lease_idx",
|
||||
"jobs",
|
||||
["lease_until"],
|
||||
postgresql_where=sa.column("status") == "processing",
|
||||
)
|
||||
op.create_table(
|
||||
"outbox",
|
||||
sa.Column("id", sa.BigInteger(), sa.Identity(always=True), primary_key=True),
|
||||
sa.Column("job_id", sa.Uuid(), sa.ForeignKey("jobs.job_id"), nullable=False),
|
||||
sa.Column("payload", postgresql.JSONB(), nullable=False),
|
||||
sa.Column("available_at", sa.DateTime(timezone=True), nullable=False),
|
||||
sa.Column("published_at", sa.DateTime(timezone=True), nullable=True),
|
||||
)
|
||||
op.create_index(
|
||||
"outbox_pending_idx",
|
||||
"outbox",
|
||||
["available_at", "id"],
|
||||
postgresql_where=sa.column("published_at").is_(None),
|
||||
)
|
||||
op.create_index("outbox_job_idx", "outbox", ["job_id"])
|
||||
op.create_table(
|
||||
"job_events",
|
||||
sa.Column("id", sa.BigInteger(), sa.Identity(always=True), primary_key=True),
|
||||
sa.Column("job_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("attachment_id", sa.Uuid(), nullable=False),
|
||||
sa.Column("kind", sa.Text(), nullable=False),
|
||||
sa.Column("attempt", sa.Integer(), nullable=True),
|
||||
sa.Column("detail", postgresql.JSONB(), nullable=False, server_default="{}"),
|
||||
sa.Column("at", sa.DateTime(timezone=True), nullable=False, server_default=sa.func.now()),
|
||||
)
|
||||
op.create_index("job_events_attachment_idx", "job_events", ["attachment_id", "id"])
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
raise RuntimeError("Downgrade would delete pipeline data")
|
||||
@@ -0,0 +1,20 @@
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0005_failed_registry"
|
||||
down_revision = "0004_jobs"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_index(
|
||||
"jobs_failed_idx",
|
||||
"jobs",
|
||||
[sa.column("updated_at").desc()],
|
||||
postgresql_where=sa.column("status") == "failed",
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("jobs_failed_idx", table_name="jobs")
|
||||
@@ -0,0 +1,28 @@
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
|
||||
revision = "0006_attachment_quarantine"
|
||||
down_revision = "0005_failed_registry"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column("attachments", sa.Column("quarantine_reason", sa.Text(), nullable=True))
|
||||
op.create_check_constraint(
|
||||
"attachments_quarantine_reason_check",
|
||||
"attachments",
|
||||
"quarantine_reason IN ('missing', 'size_mismatch', 'sha_mismatch')",
|
||||
)
|
||||
op.add_column(
|
||||
"attachments", sa.Column("quarantined_at", sa.DateTime(timezone=True), nullable=True)
|
||||
)
|
||||
op.create_check_constraint(
|
||||
"attachments_quarantine_check",
|
||||
"attachments",
|
||||
"(quarantine_reason IS NULL) = (quarantined_at IS NULL)",
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
raise RuntimeError("Downgrade would delete quarantine data")
|
||||
@@ -0,0 +1,19 @@
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from sqlalchemy.dialects.postgresql import JSONB
|
||||
|
||||
revision = "0007_trace_context"
|
||||
down_revision = "0006_attachment_quarantine"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
for table in ("jobs", "outbox"):
|
||||
op.add_column(
|
||||
table, sa.Column("trace_context", JSONB(), nullable=False, server_default="{}")
|
||||
)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
raise RuntimeError("Downgrade would delete trace correlation data")
|
||||
+638
@@ -0,0 +1,638 @@
|
||||
x-app: &app
|
||||
image: ${API_IMAGE:-mtusi-messenger:${COMPOSE_PROJECT_NAME}}
|
||||
networks: [backend]
|
||||
|
||||
x-s3-env: &s3-env
|
||||
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
S3_CONNECT_TIMEOUT_SECONDS: ${S3_CONNECT_TIMEOUT_SECONDS:-1}
|
||||
S3_READ_TIMEOUT_SECONDS: ${S3_READ_TIMEOUT_SECONDS:-2}
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
IMAGE_CHECK_CONCURRENCY: ${IMAGE_CHECK_CONCURRENCY:-2}
|
||||
UPLOAD_CONCURRENCY: ${UPLOAD_CONCURRENCY:-3}
|
||||
UPLOAD_GRACE_SECONDS: ${UPLOAD_GRACE_SECONDS:-900}
|
||||
UPLOAD_SWEEP_INTERVAL_SECONDS: ${UPLOAD_SWEEP_INTERVAL_SECONDS:-30}
|
||||
UPLOAD_SWEEP_BATCH: ${UPLOAD_SWEEP_BATCH:-100}
|
||||
|
||||
x-pipeline: &pipeline
|
||||
image: ${API_IMAGE:-mtusi-messenger:${COMPOSE_PROJECT_NAME}}
|
||||
pull_policy: never
|
||||
restart: unless-stopped
|
||||
init: true
|
||||
read_only: true
|
||||
tmpfs: [/tmp]
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
cap_drop: [ALL]
|
||||
stop_grace_period: 10s
|
||||
networks: [backend]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import os,sys,time;sys.exit(time.time()-os.stat('/tmp/heartbeat').st_mtime>30)"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
start_interval: 2s
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
broker:
|
||||
condition: service_healthy
|
||||
|
||||
x-otel-env: &otel-env
|
||||
OTEL_TRACE_RATIO: ${OTEL_TRACE_RATIO:-1}
|
||||
OTEL_SDK_DISABLED: ${OTEL_SDK_DISABLED:-false}
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT:-http://otel-collector:4318}
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL: ${OTEL_EXPORTER_OTLP_PROTOCOL:-http/protobuf}
|
||||
OTEL_EXPORTER_OTLP_TIMEOUT: ${OTEL_EXPORTER_OTLP_TIMEOUT:-2}
|
||||
OTEL_RESOURCE_ATTRIBUTES: ${OTEL_RESOURCE_ATTRIBUTES:-deployment.environment.name=lab}
|
||||
OTEL_SEMCONV_STABILITY_OPT_IN: http
|
||||
OTEL_METRIC_EXPORT_INTERVAL: ${OTEL_METRIC_EXPORT_INTERVAL:-10000}
|
||||
OTEL_BSP_MAX_QUEUE_SIZE: ${OTEL_BSP_MAX_QUEUE_SIZE:-2048}
|
||||
OTEL_BLRP_MAX_QUEUE_SIZE: ${OTEL_BLRP_MAX_QUEUE_SIZE:-2048}
|
||||
TELEMETRY_SHUTDOWN_SECONDS: ${TELEMETRY_SHUTDOWN_SECONDS:-2}
|
||||
|
||||
x-obs-logging: &obs-logging
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: 10m
|
||||
max-file: "3"
|
||||
|
||||
x-obs-hardening: &obs-hardening
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 10s
|
||||
cap_drop: [ALL]
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
logging: *obs-logging
|
||||
|
||||
x-pipeline-env: &pipeline-env
|
||||
<<: [*s3-env, *otel-env]
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
DB_POOL_MAX_SIZE: ${PIPELINE_DB_POOL_MAX_SIZE:-4}
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
HEARTBEAT_FILE: /tmp/heartbeat
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
JOB_BACKOFF_BASE_SECONDS: ${JOB_BACKOFF_BASE_SECONDS:-2}
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
|
||||
x-s3-client: &s3-client
|
||||
image: rustfs/rc:v0.1.36@sha256:ab024bfebee49a750ce886b4c70963ccd9ddaa03f491704a90710641d7a26699
|
||||
read_only: true
|
||||
tmpfs: ["/tmp:size=16m"]
|
||||
environment:
|
||||
S3_ENDPOINT: ${S3_ENDPOINT:-http://s3:9000}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
RC_CONFIG_DIR: /tmp/rc
|
||||
user: "65534:65534"
|
||||
volumes: ["./infra/s3:/s3:ro"]
|
||||
secrets:
|
||||
- s3_root_user
|
||||
- s3_root_password
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
- s3_reconciler_access_key_id
|
||||
- s3_reconciler_secret_access_key
|
||||
- s3_worker_access_key_id
|
||||
- s3_worker_secret_access_key
|
||||
cap_drop: [ALL]
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
restart: "no"
|
||||
networks: [backend]
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.5", memory: 64m}
|
||||
|
||||
services:
|
||||
lb:
|
||||
image: haproxy:3.2.26-alpine@sha256:5893840576071e6217cd52ebdcc14f454c6f7a631056fa6f4e0f40d3cac4215b
|
||||
environment:
|
||||
API_SERVER_MAXCONN: ${API_SERVER_MAXCONN:-64}
|
||||
volumes: [./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro]
|
||||
ports:
|
||||
- "127.0.0.1:${LB_HTTP_PORT:-18086}:8080"
|
||||
- "127.0.0.1:${LB_STATS_PORT:-26085}:8404"
|
||||
networks: [edge, backend]
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8404/healthz"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 3
|
||||
start_period: 60s
|
||||
start_interval: 1s
|
||||
sysctls:
|
||||
net.ipv4.ip_local_port_range: "10240 65535"
|
||||
net.ipv4.tcp_tw_reuse: "1"
|
||||
stop_signal: SIGUSR1
|
||||
stop_grace_period: 8s
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
cap_drop: [ALL]
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "${LB_CPUS:-1}", memory: 128m}
|
||||
|
||||
api:
|
||||
<<: *app
|
||||
build: .
|
||||
environment:
|
||||
<<: [*s3-env, *otel-env]
|
||||
OTEL_SERVICE_NAME: ${OTEL_SERVICE_NAME:-messenger-api}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
COST_EXPERIMENT: ${COST_EXPERIMENT:-false}
|
||||
WEB_CONCURRENCY: ${WEB_CONCURRENCY:-2}
|
||||
DB_POOL_MAX_SIZE: ${DB_POOL_MAX_SIZE:-8}
|
||||
REQUEST_TIMEOUT_SECONDS: ${REQUEST_TIMEOUT_SECONDS:-8}
|
||||
MIN_TRANSFER_RATE_BYTES: ${MIN_TRANSFER_RATE_BYTES:-131072}
|
||||
DB_ACQUIRE_TIMEOUT_SECONDS: ${DB_ACQUIRE_TIMEOUT_SECONDS:-1}
|
||||
DB_COMMAND_TIMEOUT_SECONDS: ${DB_COMMAND_TIMEOUT_SECONDS:-2}
|
||||
DRAIN_SECONDS: ${DRAIN_SECONDS:-2}
|
||||
SHUTDOWN_TIMEOUT_SECONDS: ${SHUTDOWN_TIMEOUT_SECONDS:-6}
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
MALLOC_MMAP_THRESHOLD_: ${MALLOC_MMAP_THRESHOLD_:-131072}
|
||||
secrets: [database_url, cache_url, cursor_secret, s3_access_key_id, s3_secret_access_key]
|
||||
healthcheck:
|
||||
test: ["CMD", "python", "-c", "import sys,urllib.request;sys.exit(urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=2).status!=200)"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 30s
|
||||
start_interval: 200ms
|
||||
stop_grace_period: 8s
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
tmpfs: [/tmp]
|
||||
sysctls:
|
||||
net.ipv4.tcp_rmem: "4096 65536 262144"
|
||||
net.ipv4.tcp_wmem: "4096 65536 262144"
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
valkey:
|
||||
condition: service_started
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
resources:
|
||||
limits: {cpus: "${API_CPUS:-1}", memory: "${API_MEMORY:-512m}"}
|
||||
|
||||
migrate:
|
||||
<<: *app
|
||||
pull_policy: never
|
||||
init: true
|
||||
command: ["alembic", "upgrade", "head"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
secrets: [database_url]
|
||||
healthcheck:
|
||||
disable: true
|
||||
restart: "no"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.5", memory: 256m}
|
||||
|
||||
postgres:
|
||||
image: postgres:17.11-alpine@sha256:266d3c257594e195b309ae9d1805e54c788f1d7b375f56bf6c367aac6c917e7b
|
||||
user: "70:70"
|
||||
command:
|
||||
- postgres
|
||||
- -c
|
||||
- max_connections=${PG_MAX_CONNECTIONS:-200}
|
||||
- -c
|
||||
- shared_buffers=${PG_SHARED_BUFFERS:-256MB}
|
||||
- -c
|
||||
- shared_preload_libraries=pg_stat_statements
|
||||
- -c
|
||||
- pg_stat_statements.track=top
|
||||
- -c
|
||||
- track_io_timing=on
|
||||
- -c
|
||||
- idle_in_transaction_session_timeout=10s
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
volumes: [pgdata:/var/lib/postgresql/data]
|
||||
shm_size: 256m
|
||||
cpuset: ${PG_CPUSET:-}
|
||||
healthcheck:
|
||||
test: ["CMD", "pg_isready", "-h", "127.0.0.1", "-U", "messenger", "-d", "messenger"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
start_period: 10s
|
||||
start_interval: 200ms
|
||||
stop_grace_period: 10s
|
||||
restart: unless-stopped
|
||||
networks: [backend]
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "${PG_CPUS:-2}", memory: "${PG_MEMORY:-1g}"}
|
||||
|
||||
valkey:
|
||||
image: valkey/valkey:8.1.11-alpine@sha256:2602ace2be6569bc2bbb06f873776516ab9089821525be34f4305c67eb9db02a
|
||||
user: "999:1000"
|
||||
command:
|
||||
- valkey-server
|
||||
- /run/secrets/valkey.conf
|
||||
- --appendonly
|
||||
- "yes"
|
||||
- --appendfsync
|
||||
- always
|
||||
- --maxmemory
|
||||
- 64mb
|
||||
- --maxmemory-policy
|
||||
- volatile-lru
|
||||
secrets: [valkey.conf]
|
||||
volumes: [valkeydata:/data]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "REDISCLI_AUTH=\"$$(sed -n 's/^requirepass //p' /run/secrets/valkey.conf)\" valkey-cli ping | grep -q PONG"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
stop_grace_period: 10s
|
||||
restart: unless-stopped
|
||||
networks: [backend]
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.5", memory: 128m}
|
||||
|
||||
s3:
|
||||
image: rustfs/rustfs:1.0.4@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2
|
||||
environment:
|
||||
RUSTFS_ACCESS_KEY_FILE: /run/secrets/s3_root_user
|
||||
RUSTFS_SECRET_KEY_FILE: /run/secrets/s3_root_password
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
RUSTFS_REGION: ${S3_REGION:-us-east-1}
|
||||
RUSTFS_OBS_LOG_DIRECTORY: ""
|
||||
RUSTFS_OBS_LOGGER_LEVEL: ${S3_LOG_LEVEL:-warn}
|
||||
secrets: [s3_root_user, s3_root_password]
|
||||
volumes: [s3data:/data]
|
||||
cpuset: ${S3_CPUSET:-}
|
||||
read_only: true
|
||||
tmpfs: ["/tmp:size=64m"]
|
||||
cap_drop: [ALL]
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
stop_grace_period: 10s
|
||||
restart: unless-stopped
|
||||
networks: [backend]
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "${S3_CPUS:-1}", memory: "${S3_MEMORY:-512m}"}
|
||||
|
||||
reconciler:
|
||||
<<: *app
|
||||
pull_policy: never
|
||||
profiles: [reconcile]
|
||||
init: true
|
||||
entrypoint: ["python", "-m", "messenger.attachments.entrypoints.reconcile"]
|
||||
command: ["--dry-run"]
|
||||
environment:
|
||||
<<: *s3-env
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_reconciler_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_reconciler_secret_access_key
|
||||
S3_READ_TIMEOUT_SECONDS: ${RECONCILE_S3_READ_TIMEOUT_SECONDS:-10}
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
secrets: [database_url, s3_reconciler_access_key_id, s3_reconciler_secret_access_key]
|
||||
healthcheck:
|
||||
disable: true
|
||||
restart: "no"
|
||||
read_only: true
|
||||
cpuset: ${RECONCILE_CPUSET:-}
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
s3:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "${RECONCILE_CPUS:-1}", memory: "${RECONCILE_MEMORY:-256m}"}
|
||||
|
||||
s3-init:
|
||||
<<: *s3-client
|
||||
entrypoint: ["/bin/sh", "/s3/init.sh"]
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
|
||||
s3-cli:
|
||||
<<: *s3-client
|
||||
profiles: [tools]
|
||||
entrypoint: ["/bin/sh", "/s3/cli.sh"]
|
||||
|
||||
broker:
|
||||
image: rabbitmq:4.1.8-management-alpine@sha256:54ef5ff505352c1beef05a97ee9bd6ab95b3559a76629ba0119b1d63d8d6e4b2
|
||||
hostname: broker
|
||||
user: "100:101"
|
||||
restart: unless-stopped
|
||||
read_only: true
|
||||
tmpfs: [/tmp]
|
||||
security_opt: ["no-new-privileges:true"]
|
||||
cap_drop: [ALL]
|
||||
ports:
|
||||
- "127.0.0.1:${BROKER_MANAGEMENT_PORT:-26086}:15672"
|
||||
networks: [backend, edge]
|
||||
volumes:
|
||||
- broker-data:/var/lib/rabbitmq
|
||||
- ./infra/rabbitmq/rabbitmq.conf:/etc/rabbitmq/conf.d/20-messenger.conf:ro
|
||||
- ./infra/rabbitmq/enabled_plugins:/etc/rabbitmq/enabled_plugins:ro
|
||||
- ./infra/rabbitmq/definitions.json:/etc/rabbitmq/definitions.d/10-topology.json:ro
|
||||
secrets:
|
||||
- source: rabbitmq_users
|
||||
target: /etc/rabbitmq/definitions.d/20-users.json
|
||||
- rabbitmq_ops_password
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "check_port_connectivity"]
|
||||
interval: 10s
|
||||
timeout: 10s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
start_interval: 2s
|
||||
stop_grace_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "1", memory: 512m}
|
||||
|
||||
relay:
|
||||
<<: *pipeline
|
||||
command: ["python", "-m", "messenger.relay"]
|
||||
environment:
|
||||
<<: *pipeline-env
|
||||
OTEL_SERVICE_NAME: messenger-relay
|
||||
OUTBOX_POLL_SECONDS: ${OUTBOX_POLL_SECONDS:-1}
|
||||
SWEEP_INTERVAL_SECONDS: ${SWEEP_INTERVAL_SECONDS:-5}
|
||||
secrets:
|
||||
- database_url
|
||||
- source: broker_url_api
|
||||
target: broker_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.5", memory: 192m}
|
||||
|
||||
worker:
|
||||
<<: *pipeline
|
||||
command: ["python", "-m", "messenger.worker"]
|
||||
environment:
|
||||
<<: *pipeline-env
|
||||
OTEL_SERVICE_NAME: messenger-worker
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
RENDER_TIMEOUT_SECONDS: ${RENDER_TIMEOUT_SECONDS:-30}
|
||||
RENDER_MEMORY_MB: ${RENDER_MEMORY_MB:-448}
|
||||
RENDER_TASKS_PER_CHILD: ${RENDER_TASKS_PER_CHILD:-100}
|
||||
secrets:
|
||||
- database_url
|
||||
- source: broker_url_worker
|
||||
target: broker_url
|
||||
- source: s3_worker_access_key_id
|
||||
target: s3_access_key_id
|
||||
- source: s3_worker_secret_access_key
|
||||
target: s3_secret_access_key
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "${WORKER_CPUS:-2}", memory: "${WORKER_MEMORY:-1g}"}
|
||||
|
||||
k6:
|
||||
image: grafana/k6:2.3.0@sha256:2dd4756ec6e14e62778916e1eb850108e4b499304c02df56760fa0bdb0fba0b8
|
||||
profiles: [bench]
|
||||
environment:
|
||||
BASE_URL: http://lb:8080
|
||||
volumes: [./infra/k6:/scripts:ro]
|
||||
networks: [edge]
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "${K6_CPUS:-2}", memory: "${K6_MEMORY:-512m}"}
|
||||
|
||||
healthtools:
|
||||
image: busybox:1.37.0-musl@sha256:5cec3fc171c87218698e85a52af7087de727372aae264a787b8112901a5b0092
|
||||
command: ["cp", "/bin/busybox", "/hc/busybox"]
|
||||
restart: "no"
|
||||
network_mode: none
|
||||
volumes:
|
||||
- healthtools:/hc
|
||||
logging: *obs-logging
|
||||
|
||||
otel-collector:
|
||||
<<: *obs-hardening
|
||||
image: otel/opentelemetry-collector-contrib:0.163.2@sha256:7f22d2a6748110ffb6a59bc0df448bccbf501cdb667d610ede4c61d3863d5849
|
||||
command: ["--config=/etc/otelcol/${OTEL_COLLECTOR_CONFIG:-collector.yaml}"]
|
||||
networks: [backend, edge]
|
||||
environment:
|
||||
GOMEMLIMIT: 200MiB
|
||||
volumes:
|
||||
- ./infra/otel:/etc/otelcol:ro
|
||||
- healthtools:/hc:ro
|
||||
depends_on:
|
||||
healthtools:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
test: ["CMD", "/hc/busybox", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:13133/"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 6
|
||||
start_period: 10s
|
||||
start_interval: 2s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.50", memory: 256M}
|
||||
|
||||
prometheus:
|
||||
<<: *obs-hardening
|
||||
image: prom/prometheus:v3.15.1@sha256:fa3302279710be2f21a8301ec17a8b4f2301fd829bd113839b3b6b02b6f191a2
|
||||
command:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
- --storage.tsdb.retention.time=${PROMETHEUS_RETENTION:-7d}
|
||||
- --storage.tsdb.retention.size=1GB
|
||||
- --web.enable-otlp-receiver
|
||||
- --enable-feature=exemplar-storage
|
||||
networks: [backend, edge]
|
||||
volumes:
|
||||
- ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- ./infra/prometheus/rules:/etc/prometheus/rules:ro
|
||||
- prometheus-data:/prometheus
|
||||
ports:
|
||||
- "127.0.0.1:${PROMETHEUS_PORT:-26090}:9090"
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:9090/-/ready"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 6
|
||||
start_period: 15s
|
||||
start_interval: 2s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.50", memory: 512M}
|
||||
|
||||
loki:
|
||||
<<: *obs-hardening
|
||||
image: grafana/loki:3.7.9@sha256:0db216b630ae1f348a1584c67f3cead8b4c1fc839169a39501db664744158e28
|
||||
command: ["-config.file=/etc/loki/loki.yaml"]
|
||||
networks: [edge]
|
||||
environment:
|
||||
GOMEMLIMIT: 300MiB
|
||||
volumes:
|
||||
- ./infra/loki/loki.yaml:/etc/loki/loki.yaml:ro
|
||||
- loki-data:/loki
|
||||
- healthtools:/hc:ro
|
||||
depends_on:
|
||||
healthtools:
|
||||
condition: service_completed_successfully
|
||||
ports:
|
||||
- "127.0.0.1:${LOKI_PORT:-26310}:3100"
|
||||
healthcheck:
|
||||
test: ["CMD", "/hc/busybox", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:3100/ready"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
start_interval: 2s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.50", memory: 384M}
|
||||
|
||||
tempo:
|
||||
<<: *obs-hardening
|
||||
image: grafana/tempo:2.10.10@sha256:12b0820f13ffa10ce956b6e9acc3b03f3cffded99109be6cc09d106fe56e5d38
|
||||
command: ["-config.file=/etc/tempo/tempo.yaml"]
|
||||
networks: [edge]
|
||||
environment:
|
||||
GOMEMLIMIT: 300MiB
|
||||
volumes:
|
||||
- ./infra/tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
|
||||
- tempo-data:/var/tempo
|
||||
- healthtools:/hc:ro
|
||||
depends_on:
|
||||
healthtools:
|
||||
condition: service_completed_successfully
|
||||
ports:
|
||||
- "127.0.0.1:${TEMPO_PORT:-26320}:3200"
|
||||
healthcheck:
|
||||
test: ["CMD", "/hc/busybox", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:3200/ready"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
start_interval: 2s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.50", memory: 384M}
|
||||
|
||||
grafana:
|
||||
<<: *obs-hardening
|
||||
image: grafana/grafana:12.4.14@sha256:ae509d9ae8c1021ca32ec370a9ca975d40165eda9fccd1e1102d317ff8787cd7
|
||||
networks: [edge]
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
|
||||
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||
GF_AUTH_ANONYMOUS_ENABLED: "false"
|
||||
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
||||
GF_ANALYTICS_CHECK_FOR_PLUGIN_UPDATES: "false"
|
||||
GF_NEWS_NEWS_FEED_ENABLED: "false"
|
||||
GF_PLUGINS_PREINSTALL_DISABLED: "true"
|
||||
GF_LOG_LEVEL: warn
|
||||
secrets: [grafana_admin_password]
|
||||
volumes:
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/etc/grafana/dashboards:ro
|
||||
- grafana-data:/var/lib/grafana
|
||||
depends_on:
|
||||
prometheus:
|
||||
condition: service_started
|
||||
loki:
|
||||
condition: service_started
|
||||
tempo:
|
||||
condition: service_started
|
||||
ports:
|
||||
- "127.0.0.1:${GRAFANA_PORT:-26300}:3000"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "-o", "/dev/null", "http://127.0.0.1:3000/api/health"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
start_interval: 2s
|
||||
deploy:
|
||||
resources:
|
||||
limits: {cpus: "0.50", memory: 384M}
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
valkey.conf:
|
||||
file: ./secrets/valkey.conf
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
cursor_secret:
|
||||
file: ./secrets/cursor_secret
|
||||
s3_root_user:
|
||||
file: ./secrets/s3_root_user
|
||||
s3_root_password:
|
||||
file: ./secrets/s3_root_password
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
s3_worker_access_key_id:
|
||||
file: ./secrets/s3_worker_access_key_id
|
||||
s3_worker_secret_access_key:
|
||||
file: ./secrets/s3_worker_secret_access_key
|
||||
rabbitmq_users:
|
||||
file: ./secrets/rabbitmq_users.json
|
||||
rabbitmq_ops_password:
|
||||
file: ./secrets/rabbitmq_ops_password
|
||||
broker_url_api:
|
||||
file: ./secrets/broker_url_api
|
||||
broker_url_worker:
|
||||
file: ./secrets/broker_url_worker
|
||||
grafana_admin_password:
|
||||
file: ./secrets/grafana_admin_password
|
||||
s3_reconciler_access_key_id:
|
||||
file: ./secrets/s3_reconciler_access_key_id
|
||||
s3_reconciler_secret_access_key:
|
||||
file: ./secrets/s3_reconciler_secret_access_key
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
valkeydata:
|
||||
s3data:
|
||||
broker-data:
|
||||
healthtools:
|
||||
prometheus-data:
|
||||
loki-data:
|
||||
tempo-data:
|
||||
grafana-data:
|
||||
|
||||
networks:
|
||||
edge:
|
||||
backend:
|
||||
internal: true
|
||||
+62
-3
@@ -1,5 +1,64 @@
|
||||
# Доказательства
|
||||
# Индекс доказательств ЛР6
|
||||
|
||||
Храните небольшие обезличенные результаты: команды и stdout проверок, JSON/CSV агрегатов нагрузки, EXPLAIN, dashboard JSON, текст разбора отказа. Укажите дату, commit, аппаратные лимиты и профиль нагрузки.
|
||||
Автор: Костылев Артём, БПИ2303. Итог: [REPORT.md](../REPORT.md), заявлены уровни 3/4 и трек 1 «Цена наблюдаемости». **Основной актуальный набор — `forward-published/`**. Имя каталога описывает перенос опубликованного источника ЛР5 и не означает публикацию ЛР6.
|
||||
|
||||
Не добавляйте .env, session tokens, cookies, приватные ключи, credentials, реальные сообщения, raw dumps и signed URLs. Для скриншотов скрывайте секреты. Большие логи/бинарные traces сохраняйте отдельно по правилам преподавателя; одного скриншота без методики недостаточно.
|
||||
## Происхождение и актуальность
|
||||
|
||||
| Набор | SHA и назначение | Как использовать |
|
||||
| --- | --- | --- |
|
||||
| [forward-port.md](forward-port.md), [source-inventory.json](forward-published/source-inventory.json) | ЛР5 `f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071` → `c7b8feb805d151e878d77b0dff47320a9436b172`; все 281 путь diff учтены | Карта позднего переноса, исключения и причина повторения измерений |
|
||||
| [forward-published](forward-published) | Runtime/измеритель `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`; evidence-коммит/ревью `6ca1c7549978af3127966540385195ab0587f2e1` | Свежие доказательства после исправлений; runtime между этими SHA не менялся |
|
||||
| [grade3/notes.md](grade3/notes.md), [level3.json](level3.json) | **Исторический код** `95511002b12510178a3cc2a22566b43e2b2b6c81`; чистый clone `2903d9d` | Не свежие smoke/нагрузка. История telemetry и независимость от Collector переиспользуются только для незатронутых механизмов |
|
||||
| [grade4/notes.md](grade4/notes.md) | **Исторический код** `44d81fd80ee13faa42bca069245ee0943f88b2df`, retry harness `447a6507f973596e612f40c1484dda922bf0e50d`; storage incident `cd9d91550aa4eeba31489e07b71ee2767acfc7d8` | Отказ storage и кардинальность применимы в пределах [preserved-mechanisms.json](forward-published/preserved-mechanisms.json); chain/retry/alerts заменены свежими |
|
||||
| [track5/notes.md](track5/notes.md) | **Исторический код** `202b985deacda79b6d8df65dcefe8a1af43f1a8e`, измеритель `bc0b1620825b53fd3a3019623326ebf68cbbbeec` | Старые 519 passed, −91,1% Tempo и 15 серий не смешивать с актуальными 995 passed / −92,2%; auth/cache путь изменился и был перемерен |
|
||||
| [review-summary.md](review-summary.md) | Результат независимого review из runner на `6ca1c75…` | 0 major, 5 minor; полный список ограничений, не новое ревью |
|
||||
| [report-audit.md](report-audit.md) | Документационный этап поверх `6ca1c75…` | Только статические проверки и сверка сохранённых чисел; Docker/нагрузка не запускались |
|
||||
| [load-validation.md](load-validation.md) | Финальная validation поверх `1e00d861a803a6d49ef443175b8231cbf048fc02` | Аудит применимости всех 15 серий, свежие проверки в [validation](validation); новых нагрузочных запусков нет, публикация не выполнялась |
|
||||
|
||||
[closure.json](forward-published/closure.json) фиксирует окончание реализации: runtime не менялся после замеров, 22 проверяемых тогда файла и прежний evidence сохранены, своих работающих контейнеров нет, 9 volumes оставлены. Число 22 включает документы, которые отчётный этап имеет право обновить; это исторический снимок, не запрет на REPORT/AGENTS/этот индекс. Нормативные contracts/tests/README/COURSE и infra README неизменны. В отчётном этапе raw evidence не переписывается; в старые notes добавлена только отметка исторического кода.
|
||||
|
||||
## Карта актуального набора
|
||||
|
||||
| Что проверено | Команда / средство | Доказательства |
|
||||
| --- | --- | --- |
|
||||
| Конфигурация, образы, условия и limits | `scripts/lab6.sh make up`; снимки runtime/host | [up.txt](forward-published/up.txt), [runtime.json](forward-published/runtime.json), [conditions-before.json](forward-published/conditions-before.json) |
|
||||
| Структура, схемы, ruff/ty | `make check validate lint` | [quality.txt](forward-published/quality.txt), [final-quality.txt](forward-published/final-quality.txt) |
|
||||
| Полный собственный suite | `make student-test` | [student-test.txt](forward-published/student-test.txt): 995 passed, 2 skipped, 19 deselected; запуск до runtime-коммита на WIP поверх `97feb04…`, зафиксированном затем в `bc1d7aa…` |
|
||||
| Public API, persistent stores, failover | `make test`, `make test-persistence ACTION_SCRIPT=scripts/restart.sh`, `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make restore` | [smoke](forward-published/smoke.txt), [persistence](forward-published/persistence.txt), [failover](forward-published/failover.txt), [restore](forward-published/restore.txt), [final-smoke](forward-published/final-smoke.txt) |
|
||||
| Поздние auth/cache/HTTP регрессии | `student-tests/tools/forward_network.py`, stack tests, точные команды внутри логов | [network](forward-published/network.txt), [cache-outage](forward-published/cache-outage.txt), [HAProxy errors](forward-published/haproxy-errors.txt), [negative upload](forward-published/upload-negative.txt) |
|
||||
| Requeue при отказе PostgreSQL | `student-tests/tools/forward_runtime.py postgres-outage` | [JSON](forward-published/postgres-outage.json), [команда](forward-published/postgres-outage.txt), [worker](forward-published/postgres-outage-worker.txt), [broker policy](forward-published/policy.txt) |
|
||||
| PNG корректность и ограничения памяти | `student-tests/tools/forward_runtime.py png-http` / `png-boundary` | [png-http.json](forward-published/png-http.json), [png-http.txt](forward-published/png-http.txt), [png-boundary.json](forward-published/png-boundary.json), [png-boundary.txt](forward-published/png-boundary.txt) |
|
||||
| Pipeline, retry, COMMIT/ack, DLQ | `scripts/pipeline_suite.py --level 4 --only …` (9 кейсов, полный вызов в txt) | [pipeline.json](forward-published/pipeline.json), [pipeline.txt](forward-published/pipeline.txt) |
|
||||
| Конфиги наблюдаемости и dashboard queries | `make obs-validate`; `scripts/obs_check.py dashboards --strict` | [obs-config](forward-published/obs-config.txt), [dashboards](forward-published/dashboards.txt) |
|
||||
| Корреляция request/job | `scripts/obs_check.py request` / `job` | [telemetry-request](forward-published/telemetry-request.txt), [telemetry-job](forward-published/telemetry-job.txt) |
|
||||
| Контроль запрещённых значений | `scripts/obs_check.py leaks` | [telemetry-leaks](forward-published/telemetry-leaks.txt); ограничение Loki metadata — M1 в review-summary |
|
||||
| Upload trace и retry до результата | `scripts/obs_incidents.py chain` / `retry` | [chain](forward-published/incident-chain.txt), [retry](forward-published/incident-retry.txt): 55/86 spans |
|
||||
| Worker/DB incidents, alerts | `scripts/obs_incidents.py worker` / `database` | [worker](forward-published/incident-worker.txt), [database](forward-published/incident-database.txt): firing → resolved |
|
||||
| История и outage telemetry (исторический код) | `scripts/obs_recovery.py`; `scripts/obs_incidents.py storage` | [recovery](grade3/telemetry-recovery.txt), [storage](grade4/storage-incident.txt), [обоснование неизменности](forward-published/preserved-mechanisms.json) |
|
||||
| OpenSpec | `openspec validate --all --strict --no-interactive`, sync/archive | [validation](forward-published/openspec-validation.txt), [archive](forward-published/openspec-archive.txt), [spec-sync](forward-published/spec-sync.json) |
|
||||
| Завершение без удаления данных | Возврат defaults, `make down` | [defaults-restored](forward-published/defaults-restored.json), [shutdown](forward-published/shutdown.txt), [closure](forward-published/closure.json) |
|
||||
|
||||
## Трек 1: только пригодные серии
|
||||
|
||||
[Методика](forward-published/cost/method.md) · [результаты](forward-published/cost/results.md) · [comparison.json](forward-published/cost/comparison.json) · [PNG](forward-published/cost/comparison.png) · [SVG](forward-published/cost/comparison.svg). Все численные сводки берутся из 15 raw JSON на `bc1d7aa…`; TXT той же серии содержит точный вызов, дату/SHA и exit code.
|
||||
|
||||
| Режим | Raw JSON (повторы 1 / 2 / 3) | Команды/вывод |
|
||||
| --- | --- | --- |
|
||||
| off | [1](forward-published/cost/off-1.json) / [2](forward-published/cost/off-2.json) / [3](forward-published/cost/off-3.json) | [1](forward-published/cost/off-1.txt) / [2](forward-published/cost/off-2.txt) / [3](forward-published/cost/off-3.txt) |
|
||||
| on | [1](forward-published/cost/on-1.json) / [2](forward-published/cost/on-2.json) / [3](forward-published/cost/on-3.json) | [1](forward-published/cost/on-1.txt) / [2](forward-published/cost/on-2.txt) / [3](forward-published/cost/on-3.txt) |
|
||||
| head | [1](forward-published/cost/head-1.json) / [2](forward-published/cost/head-2.json) / [3](forward-published/cost/head-3.json) | [1](forward-published/cost/head-1.txt) / [2](forward-published/cost/head-2.txt) / [3](forward-published/cost/head-3.txt) |
|
||||
| tail | [1](forward-published/cost/tail-1.json) / [2](forward-published/cost/tail-2.json) / [3](forward-published/cost/tail-3.json) | [1](forward-published/cost/tail-1.txt) / [2](forward-published/cost/tail-2.txt) / [3](forward-published/cost/tail-3.txt) |
|
||||
| balanced | [1](forward-published/cost/balanced-1.json) / [2](forward-published/cost/balanced-2.json) / [3](forward-published/cost/balanced-3.json) | [1](forward-published/cost/balanced-1.txt) / [2](forward-published/cost/balanced-2.txt) / [3](forward-published/cost/balanced-3.txt) |
|
||||
|
||||
[Measurement audit](forward-published/cost/measurement-audit.json) проверяет окно 38..42 s, drain/report 30..45 s и max gap samples <5 s. Два непригодных оригинала — [on-3](forward-published/cost/invalid-time-gap/on-3.json), [head-3](forward-published/cost/invalid-time-gap/head-3.json); причина исключения, hashes и решение повторить только их — [invalid-series.json](forward-published/cost/invalid-series.json). Их 1600 запросов не входят в 12000 пригодных.
|
||||
|
||||
Фон: [start](forward-published/cost/conditions-start.json), [round 1](forward-published/cost/conditions-round-1.json), [round 2](forward-published/cost/conditions-round-2.json), [round 3](forward-published/cost/conditions-round-3.json), [repeat start](forward-published/cost/conditions-repeat-start.json), [repeat end](forward-published/cost/conditions-repeat-end.json). Других Docker projects в снимках нет; общий host остаётся ограничением. Пригодные измерения сохраняются для финального аудита; документационные изменения не основание прогонять их заново.
|
||||
|
||||
## Неуспешные попытки и границы доказательств
|
||||
|
||||
- В [grade3/notes.md](grade3/notes.md) размечены преждевременный smoke и job во время рестарта; успешные повторы лежат рядом. Clean-clone evidence относится только к старому `2903d9d`.
|
||||
- В [grade4/notes.md](grade4/notes.md) сохранены ранние неуспешные student tests, retry без реального отказа, прерванный persistence, неправильный cardinality-вызов и DB pause без firing. Ни один не объявлен PASS итоговым отчётом.
|
||||
- В [track5/notes.md](track5/notes.md) отмечены ранний smoke, ошибка Docker stats до нагрузки и пустые validation-файлы. Все результаты этого каталога исторические.
|
||||
- [Старые findings позднего переноса](forward-published/findings.json) содержат четыре minor; [review-summary.md](review-summary.md) добавляет пятое, про Loki metadata. [Whitespaces](forward-published/whitespace-check.txt) — реальный неуспех косметической проверки, не неуспех `make lint`.
|
||||
|
||||
Не добавлять `.env`, tokens/cookies, credentials, реальные сообщения, raw dumps и signed URLs. Технические IDs в проверках синтетические; они служат корреляции, не являются секретами. Наличие файла/скриншота само по себе не доказательство: нужны команда, SHA, наблюдение и границы применимости.
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# Поздний перенос ЛР5 → ЛР6
|
||||
|
||||
Источник только immutable snapshot `lab5-c7b8feb805d151e878d77b0dff47320a9436b172`; диапазон `f88e9ba43cdfd7eb6a5a7ae5feb4247b4e820071..c7b8feb805d151e878d77b0dff47320a9436b172`. Исходный HEAD ЛР6 `237a7c93e507b984add5b49c51c6720a6bfca505`, дерево было чистым. Полный перечень всех изменённых путей, SHA-256 источника и решение по каждому: [source-inventory.json](forward-published/source-inventory.json).
|
||||
|
||||
## Перенос
|
||||
|
||||
| Группа | Изменение | Разрешение |
|
||||
|---|---|---|
|
||||
| identity handlers/postgres/valkey | Отзыв сессии, блокировка чтения, nonce заполнения кэша | Полный diff; SQLAlchemy и Lua, схема без изменений |
|
||||
| identity authentication/values, messaging values | NUL и не-ASCII Basic | Полный diff; ошибки по существующему контракту |
|
||||
| attachments images.py и png.py | Проверка IDAT, исходная tRNS глубина, tiles памяти | Полный diff; OTel S3/worker не изменён |
|
||||
| infra/haproxy/haproxy.cfg | JSON escaping request_id в 502/503/504 | Полный diff |
|
||||
| infra/rabbitmq/definitions.json | delivery-limit=-1 | Сохранено описание lab6 и существующая топология/DLX |
|
||||
| student-tests (все пути inventory) | Регрессии источника | Сохранены OTelJobObserver/provider ЛР6; прямые handle_job получают observer. Boundary PNG генерируются локально, без зависимости от evidence ЛР5; default project/port заменены на ЛР6 |
|
||||
| openspec/specs (10 capabilities) | 11 изменённых requirements | Полные delta blocks, исходные префиксы capabilities сохранены |
|
||||
| openspec/config.yaml | Версии SQLAlchemy/Alembic, provenance | Применимые строки влиты, контекст ЛР6 сохранён |
|
||||
|
||||
## Неприменимые артефакты
|
||||
|
||||
AGENTS.md, REPORT.md и evidence/README.md источника относятся к ЛР5 и не заменяют документы ЛР6. Результаты evidence источника относятся к другому приложению/SHA и не копируются как доказательство ЛР6. Исполняемая логика двух targeted.py адаптирована в student-tests/tools/forward_runtime.py; их файлы результатов не импортированы. Четыре новых source OpenSpec archives — история выполненных этапов ЛР5; их требования перенесены объединённым change `forward-lab5-published-fixes`, их завершение не выдаётся за проверку ЛР6. Каждый исключённый путь перечислен в inventory с причиной. Применимых runtime-изменений не пропущено; source diff не затрагивает Dockerfile, зависимости, Alembic или compose runtime.
|
||||
|
||||
## Документация
|
||||
|
||||
Context7 resolve для SQLAlchemy и RabbitMQ вернул `Monthly quota exceeded`. Использован официальный fallback: [SQLAlchemy 2.0 with_for_update](https://docs.sqlalchemy.org/en/20/core/selectable.html#sqlalchemy.sql.expression.GenerativeSelect.with_for_update) подтверждает read+key_share; [RabbitMQ 4.1 quorum delivery limit](https://www.rabbitmq.com/docs/4.1/quorum-queues#disabling-the-limit) содержит policy delivery-limit=-1. Новых зависимостей нет; версии остаются закреплёнными.
|
||||
|
||||
## Проверки и затронутые измерения
|
||||
|
||||
Изменение auth/cache влияет на HTTP/DB путь трека, поэтому повторяются все 15 серий с прежними параметрами; `evidence/track5` остаётся историческим результатом bc0b1620825b53fd3a3019623326ebf68cbbbeec. Новые PNG/доставка проверяются через runtime ЛР6 с включённой OTel, включая память, recovery и трассы. Наблюдения, сырые результаты и завершённый повтор трека приведены ниже.
|
||||
|
||||
## Реальные регрессии на новом runtime
|
||||
|
||||
Runtime commit: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`, образ/ресурсы — forward-published/runtime.json. `make student-test`: 995 passed, 2 Linux-only skipped, 19 stack deselected; `make check validate lint`: PASS. Эти локальные проверки выполнены до runtime-коммита, на WIP поверх 97feb04b3f2a6008f636e1fdbdf284b1c97282a1 (точный SHA и команды в логах); затем код зафиксирован в bc1d7aa. Linux memory paths проверены отдельно реальным worker в Docker с 448 MiB/30 s, concurrency2 и cgroup1GiB.
|
||||
|
||||
- network.txt: некорректный Basic/NUL, атомарность и ключ идемпотентности, logout на обеих репликах.
|
||||
- cache-outage.txt: работа при отказе Valkey, logout503 и успешный204 после восстановления.
|
||||
- postgres-outage.json/txt и postgres-outage-worker.txt: 12 реальных requeue, тот же job_id/attempt1, DLQ без прироста, recovery без replay. policy.txt подтверждает delivery-limit=-1.
|
||||
- png-http.json/txt: восемь PNG tRNS/Adam7/filter/EXIF случаев, v1/v2 и byte-exact сравнение с независимым белым эталоном; duplicate/stale не меняют результат.
|
||||
- png-boundary.json/txt: 40MP gray2/4/16/RGB16 и opaque, обе версии; два RGB16 рендера пересекались по времени, peak913227776 bytes <1073741824, oom/oom_kill0. Forced timeout и memory дали processing_failed без retry, следующее обычное задание ready.
|
||||
- haproxy-errors.txt: 12 сетевых проверок502/503/504 с request_id, JSON валиден и echo совпадает. Временный project hlpipe-lab6-forward-lb остановлен finally.
|
||||
- upload-negative.txt: 56 ответов негативного suite, 0 ответов5xx.
|
||||
- pipeline.json/txt: девять затронутых сценариев, включая retry, восстановление брокера/worker, COMMIT/ack и DLQ, с инвариантами.
|
||||
- persistence.txt, failover.txt, restore.txt: публичные проверки и восстановление реплики PASS.
|
||||
- telemetry-request/job/leaks.txt, incident-chain/retry.txt: свежие spans и логи приложения, сохранённая цепочка retry, 0 утечек.
|
||||
- incident-worker/database.txt: alerts firing→resolved; dashboards.txt и obs-config.txt PASS.
|
||||
|
||||
Прежние проверки отказа telemetry storage и сохранения истории уровней3/4 остаются применимыми: preserved-mechanisms.json подтверждает неизменность21 файла сигналов/хранилищ. 22 защищённых файла также неизменны. Эти незатронутые серии не прогоняются повторно. Новые cost-series хранятся отдельно в forward-published/cost; метод и ограничения — cost/method.md.
|
||||
|
||||
## Сохраняемые ограничения для следующего отчёта
|
||||
|
||||
Четыре ранее установленных minor-замечания не исправляются по правилу пользователя «только критичное»: неполные allowlists вспомогательного аудита лимитов, границы API SLI для ответов HAProxy/первой серии, малый запас памяти Collector при отказе storage и whitespace сгенерированных артефактов. Файлы, доказательства и актуальность перечислены в [findings.json](forward-published/findings.json). Новый REPORT здесь не составляется; эти ограничения передаются следующему агенту отчёта. Ни один обязательный неуспешный тест не объявлен успешным.
|
||||
|
||||
## Итог повторного трека и завершение
|
||||
|
||||
15 пригодных последовательных серий (по три на off/on/head/tail/balanced), 12000 запросов, 0 неожиданных ответов. SDK ended = Collector accepted во всех сериях, экспортных ошибок0. Balanced сохранил48/48 ошибок и48/48 slow; медианный ingest Tempo130,2 KiB против1662,2 KiB у on (−92,2%). Head сохранил6/48 ошибок и3/48 slow. Это наблюдение данного workload, не доказательство статистически значимого изменения latency/CPU. Сводка, диапазоны и графики: [cost/results.md](forward-published/cost/results.md), comparison.json/png/svg. PNG графика визуально проверен.
|
||||
|
||||
Два первоначальных прогона on-3/head-3 с разрывом wall clock исключены; HTTP PASS не выдан за пригодное измерение. Их raw JSON/вывод сохранены с SHA-256 в invalid-time-gap и invalid-series.json. Повторены только эти две серии. measurement-audit.json подтверждает штатные окна всех15 включённых серий. Фактический фон записан при старте, после каждого круга и повторения; сторонние проекты в этих снимках отсутствуют, общий host остаётся ограничением.
|
||||
|
||||
Штатные SDK on/100%/10s, Collector collector.yaml и COST_EXPERIMENT=false проверены в defaults-restored.json. Финальный публичный smoke прошёл (final-smoke.txt). Собственный стенд остановлен make down без удаления volumes (shutdown.txt); closure.json подтверждает9 volumes, отсутствие своих работающих контейнеров, неизменность22 защищённых файлов, предыдущего evidence и runtime после измерений. Все нагрузки и временный caffeinate завершены. Финальные offline/check/validate/lint — final-quality.txt. Публикаций не выполнялось.
|
||||
|
||||
OpenSpec: 11 requirements в10 capabilities синхронизированы и побайтно сверены по блокам с immutable source и delta (spec-sync.json). Архив: openspec/changes/archive/2026-11-07-forward-lab5-published-fixes, archived2026-11-07/lab6;26 specs strict PASS, активных changes0. git diff --cached --check завершился с кодом2 из-за whitespace в сыром выводе/сгенерированных артефактах (whitespace-check.txt), замечание minor сохранено; make lint прошёл.
|
||||
@@ -0,0 +1,10 @@
|
||||
date: 2026-11-06T23:37:24Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ .venv/bin/pytest -m stack -s -q student-tests/stack/test_faults.py::test_valkey_stopped_is_bypassed_but_logout_waits
|
||||
valkey остановлен, действующий токен: 200 за 0.008 с
|
||||
valkey остановлен, неизвестный токен: 401 за 0.004 с
|
||||
valkey остановлен: ready 200, вход 201, выход 503
|
||||
valkey снова запущен: повторный выход 204
|
||||
.
|
||||
1 passed in 4.06s
|
||||
exit_code: 0 elapsed_seconds: 4.831
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"at": "2026-11-07T01:12:37.041033+00:00",
|
||||
"implementation_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"source_sha": "c7b8feb805d151e878d77b0dff47320a9436b172",
|
||||
"protected_files": 22,
|
||||
"changed_protected": [],
|
||||
"changed_old_evidence": [],
|
||||
"changed_runtime_since_measurements": [],
|
||||
"own_running_containers": [],
|
||||
"preserved_volumes": [
|
||||
"mtusi2026lab6_broker-data",
|
||||
"mtusi2026lab6_grafana-data",
|
||||
"mtusi2026lab6_healthtools",
|
||||
"mtusi2026lab6_loki-data",
|
||||
"mtusi2026lab6_pgdata",
|
||||
"mtusi2026lab6_prometheus-data",
|
||||
"mtusi2026lab6_s3data",
|
||||
"mtusi2026lab6_tempo-data",
|
||||
"mtusi2026lab6_valkeydata"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"at": "2026-11-06T23:37:22.532499+00:00",
|
||||
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"platform": "Darwin",
|
||||
"architecture": "arm64",
|
||||
"host_cpus": 16,
|
||||
"host_load": [
|
||||
3.3583984375,
|
||||
2.19873046875,
|
||||
2.24169921875
|
||||
],
|
||||
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
|
||||
"own_project": "mtusi2026lab6",
|
||||
"own": [
|
||||
{
|
||||
"Name": "mtusi2026lab6-relay-1",
|
||||
"CPUPerc": "0.14%",
|
||||
"MemUsage": "116.9MiB / 192MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-2",
|
||||
"CPUPerc": "0.97%",
|
||||
"MemUsage": "193.6MiB / 512MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-1",
|
||||
"CPUPerc": "0.78%",
|
||||
"MemUsage": "193.4MiB / 512MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-worker-1",
|
||||
"CPUPerc": "0.14%",
|
||||
"MemUsage": "110.9MiB / 1GiB",
|
||||
"PIDs": "6"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-grafana-1",
|
||||
"CPUPerc": "0.31%",
|
||||
"MemUsage": "314.9MiB / 384MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-loki-1",
|
||||
"CPUPerc": "0.84%",
|
||||
"MemUsage": "151.9MiB / 384MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-tempo-1",
|
||||
"CPUPerc": "0.25%",
|
||||
"MemUsage": "106.5MiB / 384MiB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-otel-collector-1",
|
||||
"CPUPerc": "0.32%",
|
||||
"MemUsage": "238.1MiB / 256MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-broker-1",
|
||||
"CPUPerc": "0.13%",
|
||||
"MemUsage": "94.06MiB / 512MiB",
|
||||
"PIDs": "53"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-postgres-1",
|
||||
"CPUPerc": "0.94%",
|
||||
"MemUsage": "45.65MiB / 768MiB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-lb-1",
|
||||
"CPUPerc": "0.10%",
|
||||
"MemUsage": "22.77MiB / 128MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-prometheus-1",
|
||||
"CPUPerc": "0.23%",
|
||||
"MemUsage": "173.6MiB / 512MiB",
|
||||
"PIDs": "6"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-s3-1",
|
||||
"CPUPerc": "0.06%",
|
||||
"MemUsage": "71.48MiB / 512MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-valkey-1",
|
||||
"CPUPerc": "1.37%",
|
||||
"MemUsage": "12.66MiB / 128MiB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
],
|
||||
"other_projects": {},
|
||||
"sampling": "Shared Docker snapshot; no throughput claim"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-06T23:59:06Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode balanced --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/balanced-1.json
|
||||
{"mode": "balanced", "repeat": 1, "result": "PASS", "retained": {"normal": 39, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 126.615
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:09:36Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode balanced --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/balanced-2.json
|
||||
{"mode": "balanced", "repeat": 2, "result": "PASS", "retained": {"normal": 32, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 126.494
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T01:05:01Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode balanced --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/balanced-3.json
|
||||
{"mode": "balanced", "repeat": 3, "result": "PASS", "retained": {"normal": 30, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 126.531
|
||||
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 128 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 99 KiB |
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"at": "2026-11-07T01:11:40.821011+00:00",
|
||||
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"platform": "Darwin",
|
||||
"architecture": "arm64",
|
||||
"host_cpus": 16,
|
||||
"host_load": [
|
||||
2.5986328125,
|
||||
2.326171875,
|
||||
2.35888671875
|
||||
],
|
||||
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
|
||||
"own_project": "mtusi2026lab6",
|
||||
"own": [
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-1",
|
||||
"CPUPerc": "0.71%",
|
||||
"MemUsage": "184.3MiB / 512MiB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-2",
|
||||
"CPUPerc": "1.70%",
|
||||
"MemUsage": "192.5MiB / 512MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-relay-1",
|
||||
"CPUPerc": "0.67%",
|
||||
"MemUsage": "116.6MiB / 192MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-worker-1",
|
||||
"CPUPerc": "0.14%",
|
||||
"MemUsage": "108.3MiB / 1GiB",
|
||||
"PIDs": "6"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-otel-collector-1",
|
||||
"CPUPerc": "0.15%",
|
||||
"MemUsage": "52.7MiB / 256MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-grafana-1",
|
||||
"CPUPerc": "0.39%",
|
||||
"MemUsage": "320.1MiB / 384MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-loki-1",
|
||||
"CPUPerc": "0.75%",
|
||||
"MemUsage": "225.5MiB / 384MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-tempo-1",
|
||||
"CPUPerc": "0.25%",
|
||||
"MemUsage": "227MiB / 384MiB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-broker-1",
|
||||
"CPUPerc": "24.52%",
|
||||
"MemUsage": "95.97MiB / 512MiB",
|
||||
"PIDs": "53"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-postgres-1",
|
||||
"CPUPerc": "0.53%",
|
||||
"MemUsage": "45.99MiB / 768MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-lb-1",
|
||||
"CPUPerc": "0.66%",
|
||||
"MemUsage": "25.02MiB / 128MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-prometheus-1",
|
||||
"CPUPerc": "0.14%",
|
||||
"MemUsage": "158.4MiB / 512MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-s3-1",
|
||||
"CPUPerc": "0.03%",
|
||||
"MemUsage": "160.2MiB / 512MiB",
|
||||
"PIDs": "25"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-valkey-1",
|
||||
"CPUPerc": "0.13%",
|
||||
"MemUsage": "13.07MiB / 128MiB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
],
|
||||
"other_projects": {},
|
||||
"sampling": "Shared Docker snapshot; no throughput claim"
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"at": "2026-11-07T01:07:27.961141+00:00",
|
||||
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"platform": "Darwin",
|
||||
"architecture": "arm64",
|
||||
"host_cpus": 16,
|
||||
"host_load": [
|
||||
1.646484375,
|
||||
1.9970703125,
|
||||
2.29052734375
|
||||
],
|
||||
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
|
||||
"own_project": "mtusi2026lab6",
|
||||
"own": [
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-1",
|
||||
"CPUPerc": "1.17%",
|
||||
"MemUsage": "192.9MiB / 512MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-2",
|
||||
"CPUPerc": "1.03%",
|
||||
"MemUsage": "192.8MiB / 512MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-worker-1",
|
||||
"CPUPerc": "0.62%",
|
||||
"MemUsage": "108.4MiB / 1GiB",
|
||||
"PIDs": "6"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-relay-1",
|
||||
"CPUPerc": "0.80%",
|
||||
"MemUsage": "116.7MiB / 192MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-otel-collector-1",
|
||||
"CPUPerc": "0.27%",
|
||||
"MemUsage": "42.03MiB / 256MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-grafana-1",
|
||||
"CPUPerc": "1.83%",
|
||||
"MemUsage": "320MiB / 384MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-loki-1",
|
||||
"CPUPerc": "0.62%",
|
||||
"MemUsage": "228.5MiB / 384MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-tempo-1",
|
||||
"CPUPerc": "0.33%",
|
||||
"MemUsage": "287.3MiB / 384MiB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-broker-1",
|
||||
"CPUPerc": "36.74%",
|
||||
"MemUsage": "183.4MiB / 512MiB",
|
||||
"PIDs": "103"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-postgres-1",
|
||||
"CPUPerc": "0.94%",
|
||||
"MemUsage": "47.72MiB / 768MiB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-lb-1",
|
||||
"CPUPerc": "0.61%",
|
||||
"MemUsage": "24.96MiB / 128MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-prometheus-1",
|
||||
"CPUPerc": "0.59%",
|
||||
"MemUsage": "157.3MiB / 512MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-s3-1",
|
||||
"CPUPerc": "0.93%",
|
||||
"MemUsage": "160.2MiB / 512MiB",
|
||||
"PIDs": "25"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-valkey-1",
|
||||
"CPUPerc": "0.78%",
|
||||
"MemUsage": "13.07MiB / 128MiB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
],
|
||||
"other_projects": {},
|
||||
"sampling": "Shared Docker snapshot; no throughput claim"
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"at": "2026-11-07T00:01:16.214846+00:00",
|
||||
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"platform": "Darwin",
|
||||
"architecture": "arm64",
|
||||
"host_cpus": 16,
|
||||
"host_load": [
|
||||
1.6005859375,
|
||||
1.84521484375,
|
||||
2.02685546875
|
||||
],
|
||||
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
|
||||
"own_project": "mtusi2026lab6",
|
||||
"own": [
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-1",
|
||||
"CPUPerc": "1.28%",
|
||||
"MemUsage": "184MiB / 512MiB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-2",
|
||||
"CPUPerc": "0.83%",
|
||||
"MemUsage": "184.3MiB / 512MiB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-worker-1",
|
||||
"CPUPerc": "0.15%",
|
||||
"MemUsage": "108.3MiB / 1GiB",
|
||||
"PIDs": "6"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-relay-1",
|
||||
"CPUPerc": "0.19%",
|
||||
"MemUsage": "116.5MiB / 192MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-otel-collector-1",
|
||||
"CPUPerc": "0.36%",
|
||||
"MemUsage": "46.17MiB / 256MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-grafana-1",
|
||||
"CPUPerc": "0.32%",
|
||||
"MemUsage": "319.3MiB / 384MiB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-loki-1",
|
||||
"CPUPerc": "0.97%",
|
||||
"MemUsage": "214.4MiB / 384MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-tempo-1",
|
||||
"CPUPerc": "0.23%",
|
||||
"MemUsage": "224.8MiB / 384MiB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-broker-1",
|
||||
"CPUPerc": "0.24%",
|
||||
"MemUsage": "95.74MiB / 512MiB",
|
||||
"PIDs": "53"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-postgres-1",
|
||||
"CPUPerc": "1.49%",
|
||||
"MemUsage": "44.05MiB / 768MiB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-lb-1",
|
||||
"CPUPerc": "1.03%",
|
||||
"MemUsage": "23.41MiB / 128MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-prometheus-1",
|
||||
"CPUPerc": "1.11%",
|
||||
"MemUsage": "154.9MiB / 512MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-s3-1",
|
||||
"CPUPerc": "0.05%",
|
||||
"MemUsage": "160.2MiB / 512MiB",
|
||||
"PIDs": "25"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-valkey-1",
|
||||
"CPUPerc": "1.18%",
|
||||
"MemUsage": "13.05MiB / 128MiB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
],
|
||||
"other_projects": {},
|
||||
"sampling": "Shared Docker snapshot; no throughput claim"
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"at": "2026-11-07T00:11:45.236693+00:00",
|
||||
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"platform": "Darwin",
|
||||
"architecture": "arm64",
|
||||
"host_cpus": 16,
|
||||
"host_load": [
|
||||
4.9462890625,
|
||||
3.5625,
|
||||
2.7451171875
|
||||
],
|
||||
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
|
||||
"own_project": "mtusi2026lab6",
|
||||
"own": [
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-2",
|
||||
"CPUPerc": "1.19%",
|
||||
"MemUsage": "184.1MiB / 512MiB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-1",
|
||||
"CPUPerc": "3.14%",
|
||||
"MemUsage": "191.7MiB / 512MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-relay-1",
|
||||
"CPUPerc": "0.62%",
|
||||
"MemUsage": "116.5MiB / 192MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-worker-1",
|
||||
"CPUPerc": "0.17%",
|
||||
"MemUsage": "108.3MiB / 1GiB",
|
||||
"PIDs": "6"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-otel-collector-1",
|
||||
"CPUPerc": "0.28%",
|
||||
"MemUsage": "42.81MiB / 256MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-grafana-1",
|
||||
"CPUPerc": "0.30%",
|
||||
"MemUsage": "320MiB / 384MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-loki-1",
|
||||
"CPUPerc": "1.56%",
|
||||
"MemUsage": "221.1MiB / 384MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-tempo-1",
|
||||
"CPUPerc": "1.27%",
|
||||
"MemUsage": "233.4MiB / 384MiB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-broker-1",
|
||||
"CPUPerc": "0.25%",
|
||||
"MemUsage": "95.85MiB / 512MiB",
|
||||
"PIDs": "53"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-postgres-1",
|
||||
"CPUPerc": "0.73%",
|
||||
"MemUsage": "45.72MiB / 768MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-lb-1",
|
||||
"CPUPerc": "0.17%",
|
||||
"MemUsage": "24.55MiB / 128MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-prometheus-1",
|
||||
"CPUPerc": "0.17%",
|
||||
"MemUsage": "155MiB / 512MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-s3-1",
|
||||
"CPUPerc": "0.02%",
|
||||
"MemUsage": "160.2MiB / 512MiB",
|
||||
"PIDs": "25"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-valkey-1",
|
||||
"CPUPerc": "1.13%",
|
||||
"MemUsage": "13.12MiB / 128MiB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
],
|
||||
"other_projects": {},
|
||||
"sampling": "Shared Docker snapshot; no throughput claim"
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"at": "2026-11-07T01:07:10.780508+00:00",
|
||||
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"platform": "Darwin",
|
||||
"architecture": "arm64",
|
||||
"host_cpus": 16,
|
||||
"host_load": [
|
||||
1.6572265625,
|
||||
2.015625,
|
||||
2.30224609375
|
||||
],
|
||||
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
|
||||
"own_project": "mtusi2026lab6",
|
||||
"own": [
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-1",
|
||||
"CPUPerc": "1.54%",
|
||||
"MemUsage": "183.9MiB / 512MiB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-2",
|
||||
"CPUPerc": "3.32%",
|
||||
"MemUsage": "192.2MiB / 512MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-worker-1",
|
||||
"CPUPerc": "0.18%",
|
||||
"MemUsage": "108.3MiB / 1GiB",
|
||||
"PIDs": "6"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-relay-1",
|
||||
"CPUPerc": "0.53%",
|
||||
"MemUsage": "116.5MiB / 192MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-otel-collector-1",
|
||||
"CPUPerc": "0.40%",
|
||||
"MemUsage": "40.87MiB / 256MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-grafana-1",
|
||||
"CPUPerc": "0.42%",
|
||||
"MemUsage": "320MiB / 384MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-loki-1",
|
||||
"CPUPerc": "0.70%",
|
||||
"MemUsage": "228.8MiB / 384MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-tempo-1",
|
||||
"CPUPerc": "0.31%",
|
||||
"MemUsage": "269.3MiB / 384MiB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-broker-1",
|
||||
"CPUPerc": "0.23%",
|
||||
"MemUsage": "96.34MiB / 512MiB",
|
||||
"PIDs": "53"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-postgres-1",
|
||||
"CPUPerc": "0.71%",
|
||||
"MemUsage": "45.93MiB / 768MiB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-lb-1",
|
||||
"CPUPerc": "0.17%",
|
||||
"MemUsage": "24.96MiB / 128MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-prometheus-1",
|
||||
"CPUPerc": "0.14%",
|
||||
"MemUsage": "157.3MiB / 512MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-s3-1",
|
||||
"CPUPerc": "0.04%",
|
||||
"MemUsage": "160.2MiB / 512MiB",
|
||||
"PIDs": "25"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-valkey-1",
|
||||
"CPUPerc": "0.17%",
|
||||
"MemUsage": "13.07MiB / 128MiB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
],
|
||||
"other_projects": {},
|
||||
"sampling": "Shared Docker snapshot; no throughput claim"
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
{
|
||||
"at": "2026-11-06T23:50:46.782935+00:00",
|
||||
"runtime_sha": "bc1d7aa7713858acb5d65b61d07e69a640f1ee97",
|
||||
"platform": "Darwin",
|
||||
"architecture": "arm64",
|
||||
"host_cpus": 16,
|
||||
"host_load": [
|
||||
2.43505859375,
|
||||
2.41015625,
|
||||
2.33154296875
|
||||
],
|
||||
"docker": "16 CPUs; 8217190400 bytes; 28.3.2",
|
||||
"own_project": "mtusi2026lab6",
|
||||
"own": [
|
||||
{
|
||||
"Name": "mtusi2026lab6-worker-1",
|
||||
"CPUPerc": "0.25%",
|
||||
"MemUsage": "339.7MiB / 1GiB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-relay-1",
|
||||
"CPUPerc": "0.15%",
|
||||
"MemUsage": "118.1MiB / 192MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-2",
|
||||
"CPUPerc": "0.80%",
|
||||
"MemUsage": "250.3MiB / 512MiB",
|
||||
"PIDs": "18"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-api-1",
|
||||
"CPUPerc": "8.79%",
|
||||
"MemUsage": "255.3MiB / 512MiB",
|
||||
"PIDs": "19"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-grafana-1",
|
||||
"CPUPerc": "0.34%",
|
||||
"MemUsage": "320.6MiB / 384MiB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-loki-1",
|
||||
"CPUPerc": "0.51%",
|
||||
"MemUsage": "210.6MiB / 384MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-tempo-1",
|
||||
"CPUPerc": "0.23%",
|
||||
"MemUsage": "200.6MiB / 384MiB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-otel-collector-1",
|
||||
"CPUPerc": "0.07%",
|
||||
"MemUsage": "107.4MiB / 256MiB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-broker-1",
|
||||
"CPUPerc": "1.14%",
|
||||
"MemUsage": "87.95MiB / 512MiB",
|
||||
"PIDs": "53"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-postgres-1",
|
||||
"CPUPerc": "1.74%",
|
||||
"MemUsage": "76.54MiB / 768MiB",
|
||||
"PIDs": "29"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-lb-1",
|
||||
"CPUPerc": "0.99%",
|
||||
"MemUsage": "18.9MiB / 128MiB",
|
||||
"PIDs": "17"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-prometheus-1",
|
||||
"CPUPerc": "0.12%",
|
||||
"MemUsage": "154.4MiB / 512MiB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-s3-1",
|
||||
"CPUPerc": "1.27%",
|
||||
"MemUsage": "160.9MiB / 512MiB",
|
||||
"PIDs": "25"
|
||||
},
|
||||
{
|
||||
"Name": "mtusi2026lab6-valkey-1",
|
||||
"CPUPerc": "1.20%",
|
||||
"MemUsage": "13.05MiB / 128MiB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
],
|
||||
"other_projects": {},
|
||||
"sampling": "Shared Docker snapshot; no throughput claim"
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-06T23:54:56Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode head --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/head-1.json
|
||||
{"mode": "head", "repeat": 1, "result": "PASS", "retained": {"normal": 85, "error": 1, "slow": 1}}
|
||||
exit_code: 0 elapsed_seconds: 124.106
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:05:25Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode head --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/head-2.json
|
||||
{"mode": "head", "repeat": 2, "result": "PASS", "retained": {"normal": 76, "error": 4, "slow": 1}}
|
||||
exit_code: 0 elapsed_seconds: 124.746
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T01:09:32Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode head --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/head-3.json
|
||||
{"mode": "head", "repeat": 3, "result": "PASS", "retained": {"normal": 76, "error": 1, "slow": 1}}
|
||||
exit_code: 0 elapsed_seconds: 125.216
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"decision_at": "2026-11-07T01:05:32.987756+00:00",
|
||||
"interpretation": "Observed wall-clock gaps while monotonic elapsed remains nominal; likely host suspension or clock discontinuity. Cause not asserted as proven. Only these two completed series are repeated; raw originals preserved byte-for-byte.",
|
||||
"runs": [
|
||||
{
|
||||
"series": "on-3",
|
||||
"reason": "Wall-clock discontinuity inside the measurement/accounting window; excluded from resource/ingest comparison, HTTP PASS alone is insufficient.",
|
||||
"load_seconds": 39.970131158828735,
|
||||
"finish_to_report_seconds": 932.7143309116364,
|
||||
"max_resource_gap_seconds": 2.047463893890381,
|
||||
"files": {
|
||||
"on-3.json": "d0c4799cea3f5a5c819c175c738b6099158bc5ca9b6d2d840a89c9d73d394953",
|
||||
"on-3.txt": "64199cc40d2a81d15a219e63c04aac875929eba41110bfbf3909a4d43518e010"
|
||||
}
|
||||
},
|
||||
{
|
||||
"series": "head-3",
|
||||
"reason": "Wall-clock discontinuity inside the measurement/accounting window; excluded from resource/ingest comparison, HTTP PASS alone is insufficient.",
|
||||
"load_seconds": 938.1542620658875,
|
||||
"finish_to_report_seconds": 34.9133198261261,
|
||||
"max_resource_gap_seconds": 900.3541407585144,
|
||||
"files": {
|
||||
"head-3.json": "3dbecccf017b26bc6a256860b795f8cc9e4c47099da2d46e8303120f49686f45",
|
||||
"head-3.txt": "0b4f1607837921746893700a5897b1a1c94db115d59c750f3ec9631266e95a69"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:30:52Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode head --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/head-3.json
|
||||
{"mode": "head", "repeat": 3, "result": "PASS", "retained": {"normal": 70, "error": 1, "slow": 3}}
|
||||
exit_code: 0 elapsed_seconds: 125.007
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:13:50Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode on --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/on-3.json
|
||||
{"mode": "on", "repeat": 3, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 123.841
|
||||
@@ -0,0 +1,115 @@
|
||||
{
|
||||
"at": "2026-11-07T01:11:53.670099+00:00",
|
||||
"policy": "40s workload: wall window38..42s, finish-to-report30..45s, maximum2s-sampling gap<5s. All three conditions required independently of HTTP PASS.",
|
||||
"runs": [
|
||||
{
|
||||
"file": "off-1.json",
|
||||
"load_seconds": 39.9768009185791,
|
||||
"finish_to_report_seconds": 34.378885984420776,
|
||||
"max_resource_gap_seconds": 2.047776937484741,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "off-2.json",
|
||||
"load_seconds": 39.96762013435364,
|
||||
"finish_to_report_seconds": 35.243422985076904,
|
||||
"max_resource_gap_seconds": 2.048405885696411,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "off-3.json",
|
||||
"load_seconds": 39.97038507461548,
|
||||
"finish_to_report_seconds": 35.21114897727966,
|
||||
"max_resource_gap_seconds": 2.0483851432800293,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "on-1.json",
|
||||
"load_seconds": 39.960899114608765,
|
||||
"finish_to_report_seconds": 35.31035900115967,
|
||||
"max_resource_gap_seconds": 2.0440289974212646,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "on-2.json",
|
||||
"load_seconds": 39.967061042785645,
|
||||
"finish_to_report_seconds": 34.75409412384033,
|
||||
"max_resource_gap_seconds": 2.045710802078247,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "on-3.json",
|
||||
"load_seconds": 39.961934089660645,
|
||||
"finish_to_report_seconds": 35.3386869430542,
|
||||
"max_resource_gap_seconds": 2.0448179244995117,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "head-1.json",
|
||||
"load_seconds": 39.960408210754395,
|
||||
"finish_to_report_seconds": 34.749109983444214,
|
||||
"max_resource_gap_seconds": 2.0449328422546387,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "head-2.json",
|
||||
"load_seconds": 39.95907402038574,
|
||||
"finish_to_report_seconds": 34.537447929382324,
|
||||
"max_resource_gap_seconds": 2.0501198768615723,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "head-3.json",
|
||||
"load_seconds": 39.96534299850464,
|
||||
"finish_to_report_seconds": 35.03631401062012,
|
||||
"max_resource_gap_seconds": 2.0513830184936523,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "tail-1.json",
|
||||
"load_seconds": 39.969640254974365,
|
||||
"finish_to_report_seconds": 35.31044888496399,
|
||||
"max_resource_gap_seconds": 2.0470268726348877,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "tail-2.json",
|
||||
"load_seconds": 39.96142888069153,
|
||||
"finish_to_report_seconds": 34.63654613494873,
|
||||
"max_resource_gap_seconds": 2.0448548793792725,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "tail-3.json",
|
||||
"load_seconds": 39.968284130096436,
|
||||
"finish_to_report_seconds": 35.04265284538269,
|
||||
"max_resource_gap_seconds": 2.047273874282837,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "balanced-1.json",
|
||||
"load_seconds": 39.965638160705566,
|
||||
"finish_to_report_seconds": 34.83575201034546,
|
||||
"max_resource_gap_seconds": 2.0490190982818604,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "balanced-2.json",
|
||||
"load_seconds": 39.96572422981262,
|
||||
"finish_to_report_seconds": 35.22300386428833,
|
||||
"max_resource_gap_seconds": 2.0449130535125732,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "balanced-3.json",
|
||||
"load_seconds": 39.96644401550293,
|
||||
"finish_to_report_seconds": 35.353606939315796,
|
||||
"max_resource_gap_seconds": 2.0450639724731445,
|
||||
"result": "usable"
|
||||
}
|
||||
],
|
||||
"excluded": [
|
||||
"invalid-time-gap/on-3.json",
|
||||
"invalid-time-gap/head-3.json"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,115 @@
|
||||
{
|
||||
"at": "2026-11-07T01:11:53.670099+00:00",
|
||||
"policy": "40s workload: wall window38..42s, finish-to-report30..45s, maximum2s-sampling gap<5s. All three conditions required independently of HTTP PASS.",
|
||||
"runs": [
|
||||
{
|
||||
"file": "off-1.json",
|
||||
"load_seconds": 39.9768009185791,
|
||||
"finish_to_report_seconds": 34.378885984420776,
|
||||
"max_resource_gap_seconds": 2.047776937484741,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "off-2.json",
|
||||
"load_seconds": 39.96762013435364,
|
||||
"finish_to_report_seconds": 35.243422985076904,
|
||||
"max_resource_gap_seconds": 2.048405885696411,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "off-3.json",
|
||||
"load_seconds": 39.97038507461548,
|
||||
"finish_to_report_seconds": 35.21114897727966,
|
||||
"max_resource_gap_seconds": 2.0483851432800293,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "on-1.json",
|
||||
"load_seconds": 39.960899114608765,
|
||||
"finish_to_report_seconds": 35.31035900115967,
|
||||
"max_resource_gap_seconds": 2.0440289974212646,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "on-2.json",
|
||||
"load_seconds": 39.967061042785645,
|
||||
"finish_to_report_seconds": 34.75409412384033,
|
||||
"max_resource_gap_seconds": 2.045710802078247,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "on-3.json",
|
||||
"load_seconds": 39.961934089660645,
|
||||
"finish_to_report_seconds": 35.3386869430542,
|
||||
"max_resource_gap_seconds": 2.0448179244995117,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "head-1.json",
|
||||
"load_seconds": 39.960408210754395,
|
||||
"finish_to_report_seconds": 34.749109983444214,
|
||||
"max_resource_gap_seconds": 2.0449328422546387,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "head-2.json",
|
||||
"load_seconds": 39.95907402038574,
|
||||
"finish_to_report_seconds": 34.537447929382324,
|
||||
"max_resource_gap_seconds": 2.0501198768615723,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "head-3.json",
|
||||
"load_seconds": 39.96534299850464,
|
||||
"finish_to_report_seconds": 35.03631401062012,
|
||||
"max_resource_gap_seconds": 2.0513830184936523,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "tail-1.json",
|
||||
"load_seconds": 39.969640254974365,
|
||||
"finish_to_report_seconds": 35.31044888496399,
|
||||
"max_resource_gap_seconds": 2.0470268726348877,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "tail-2.json",
|
||||
"load_seconds": 39.96142888069153,
|
||||
"finish_to_report_seconds": 34.63654613494873,
|
||||
"max_resource_gap_seconds": 2.0448548793792725,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "tail-3.json",
|
||||
"load_seconds": 39.968284130096436,
|
||||
"finish_to_report_seconds": 35.04265284538269,
|
||||
"max_resource_gap_seconds": 2.047273874282837,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "balanced-1.json",
|
||||
"load_seconds": 39.965638160705566,
|
||||
"finish_to_report_seconds": 34.83575201034546,
|
||||
"max_resource_gap_seconds": 2.0490190982818604,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "balanced-2.json",
|
||||
"load_seconds": 39.96572422981262,
|
||||
"finish_to_report_seconds": 35.22300386428833,
|
||||
"max_resource_gap_seconds": 2.0449130535125732,
|
||||
"result": "usable"
|
||||
},
|
||||
{
|
||||
"file": "balanced-3.json",
|
||||
"load_seconds": 39.96644401550293,
|
||||
"finish_to_report_seconds": 35.353606939315796,
|
||||
"max_resource_gap_seconds": 2.0450639724731445,
|
||||
"result": "usable"
|
||||
}
|
||||
],
|
||||
"excluded": [
|
||||
"invalid-time-gap/on-3.json",
|
||||
"invalid-time-gap/head-3.json"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
# Повтор трека после позднего переноса
|
||||
|
||||
Приложение и измеритель: `bc1d7aa7713858acb5d65b61d07e69a640f1ee97`. Образ/лимиты: `../runtime.json`. Причина повторения — изменён путь auth/cache, используемый GET users/me; исторические серии `evidence/track5` на bc0b162 остаются неизменными и не выдаются за результат нового кода.
|
||||
|
||||
Гипотеза: head 10% уменьшает ingest, теряя редкие события; tail сохраняет ошибки и slow, сокращая запись в Tempo, но сохраняет расходы SDK. Balanced: tail 5% обычных трасс, все ERROR и latency >=300 ms; период метрик 30 s вместо 10 s.
|
||||
|
||||
Команда каждой серии: `scripts/lab6.sh python3 scripts/obs_cost.py --mode MODE --repeat N --duration 40 --rate 20 --output evidence/forward-published/cost/MODE-N.json`. Порядок: off/on/head/tail/balanced, три круга. 20 прогревочных GET, 35 s ожидания экспорта; 40 s open-loop, 800 arrivals с периодом 50 ms, 64 клиентских потока. 768 обычных запросов, 16 контролируемых 503, 16 задержек 350 ms. Затем 35 s drain. finally возвращает обычный Collector, SDK on/ratio1/interval10s и отключает fault hook. Все серии идут последовательно, отдельные проверки и другие нагрузки этого агента одновременно не запускаются.
|
||||
|
||||
Latency измерена клиентом, schedule_lag записан отдельно. CPU — delta cgroup total_usage / время внешних Docker samples; RAM — usage−inactive_file. Это не SDK-метрики. Группы application (API, relay, worker), observability, business stores/LB раздельны. Окно CPU охватывает нагрузку, не последнюю работу tail sampler во время drain; RAM — пик наблюдаемых samples, не непрерывный максимум.
|
||||
|
||||
Ingest: Tempo/Loki received bytes и Prometheus appended samples; разные единицы не складываются. Окно включает drain и постоянный фон self-scrapes. SDK ended сравнивается с Collector accepted, export failures учитываются отдельно; для tail accepted−sent включает намеренную фильтрацию. Отбор server traces по уникальному префиксу каждой серии и категории, search limit5000 >800. Для head неизвестное число несозданных DB spans не оценивается вымышленным числом; наблюдается доля server roots. Off — отсутствие SDK, а не ошибка доставки.
|
||||
|
||||
Фон Docker/host записан в conditions-start и conditions-round-N, inventory дополнительно есть в каждом raw JSON. Host не эксклюзивен, поэтому малые различия latency/CPU не доказывают статистическую значимость. Три повтора дают наблюдаемый диапазон, не доверительный интервал. Workload HTTP/DB, worker ожидает заданий; массовая обработка изображений отдельно проверена в png-boundary.json. Tail decision_wait10s не доказывает полноту цепочек с очень поздними retry. Успешные завершённые серии сохраняются для финального аудита без автоматического повторного прогона.
|
||||
|
||||
Сводка: `python3 scripts/obs_cost_summary.py evidence/forward-published/cost`. Графики: `uv run --no-project --with matplotlib==3.11.2 python tools/plot_observability.py evidence/forward-published/cost`.
|
||||
|
||||
## Пригодность серий и повтор двух прогонов
|
||||
|
||||
После двух первых кругов обнаружен разрыв настенных часов относительно монотонного таймера (вероятен сон host, причина не утверждается доказанной). В первоначальном on-3 окно от конца нагрузки до отчёта стало 932,714 s, в head-3 сама нагрузка по wall clock — 938,154 s, максимальный разрыв внешних samples — 900,354 s. HTTP PASS этих прогонов не означает пригодность CPU/ingest. Оба оригинала сохранены без изменения в invalid-time-gap, SHA-256 и решение — invalid-series.json; они исключены из сравнения. Остальные завершённые серии не перезапускаются.
|
||||
|
||||
Для оставшихся измерений включён ограниченный по времени `caffeinate -i -s`; повторяются только on-3 и head-3 с теми же параметрами и SHA, последовательно. Проверка пригодности всех включённых серий: finished−started 38..42 s, utc−finished 30..45 s (35 s drain плюс запросы/округление), max соседнего resource.at gap <5 s при периоде2 s. Результат — measurement-audit.json. Отдельные паузы до начала измеряемого окна не включаются в CPU/ingest и сами по себе не бракуют серию. Контекст общего host и фактический фон остаются ограничениями.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-06T23:50:47Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode off --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/off-1.json
|
||||
{"mode": "off", "repeat": 1, "result": "PASS", "retained": {"normal": 0, "error": 0, "slow": 0}}
|
||||
exit_code: 0 elapsed_seconds: 125.202
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:01:16Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode off --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/off-2.json
|
||||
{"mode": "off", "repeat": 2, "result": "PASS", "retained": {"normal": 0, "error": 0, "slow": 0}}
|
||||
exit_code: 0 elapsed_seconds: 125.016
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:11:45Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode off --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/off-3.json
|
||||
{"mode": "off", "repeat": 3, "result": "PASS", "retained": {"normal": 0, "error": 0, "slow": 0}}
|
||||
exit_code: 0 elapsed_seconds: 125.038
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-06T23:52:52Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode on --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/on-1.json
|
||||
{"mode": "on", "repeat": 1, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 124.436
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:03:21Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode on --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/on-2.json
|
||||
{"mode": "on", "repeat": 2, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 123.922
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T01:07:28Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode on --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/on-3.json
|
||||
{"mode": "on", "repeat": 3, "result": "PASS", "retained": {"normal": 768, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 124.718
|
||||
@@ -0,0 +1,29 @@
|
||||
# Результаты повторного трека
|
||||
|
||||
Приложение и измеритель: bc1d7aa7713858acb5d65b61d07e69a640f1ee97. Все 15 пригодных серий завершены: 12000 запросов, 0 неожиданных ответов. Два первоначальных прогона (ещё 1600 запросов) исключены по разрывам времени, сохранены в invalid-time-gap и повторены; подробности в method.md и invalid-series.json. Все включённые серии прошли measurement-audit.json. Во всех сериях SDK ended = Collector accepted, exporter failures = 0.
|
||||
|
||||
| Режим | normal p95 ms: median [min;max] | общий p99 ms | CPU app / obs cores | RAM app / obs MiB, median peak |
|
||||
|---|---:|---:|---:|---:|
|
||||
| off | 11.583 [11.572; 11.856] | 358.73 | 0.1404 / 0.0256 | 655.5 / 1013.3 |
|
||||
| on | 12.709 [12.554; 12.993] | 359.47 | 0.1726 / 0.0287 | 706.8 / 1058.5 |
|
||||
| head | 12.556 [12.354; 12.652] | 358.37 | 0.1657 / 0.0261 | 723.8 / 1076.1 |
|
||||
| tail | 12.781 [12.555; 12.920] | 359.11 | 0.1717 / 0.0272 | 721.0 / 1021.2 |
|
||||
| balanced | 12.895 [12.833; 13.031] | 359.75 | 0.1749 / 0.0258 | 707.1 / 1027.9 |
|
||||
|
||||
| Режим | Tempo KiB | Loki KiB | Prom samples | SDK / sent spans, median | normal /2304 | error /48 | slow /48 |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| off | 0.0 | 0.0 | 33118 | 0 / 0 | 0 | 0 | 0 |
|
||||
| on | 1662.2 | 288.3 | 36284 | 5489 / 5489 | 2304 | 48 | 48 |
|
||||
| head | 176.2 | 288.4 | 37234 | 543 / 543 | 237 | 6 | 3 |
|
||||
| tail | 251.8 | 289.7 | 36657 | 5492 / 699 | 235 | 48 | 48 |
|
||||
| balanced | 130.2 | 289.8 | 34459 | 5486 / 352 | 101 | 48 | 48 |
|
||||
|
||||
Balanced сохраняет 48/48 errors и 48/48 slow, медианный Tempo ingest ниже on на 92.2%. Head пропустил 42 ошибок и 45 slow. Это отбор sampling, а не сбой доставки.
|
||||
|
||||
Фактическая скорость 20.012–20.020 requests/s. Максимальный schedule lag 61.055 ms. Внешних samples на серию: 21–21.
|
||||
|
||||
Рекомендация в пределах этого workload: balanced сокращает запись трасс и сохраняет контрольные ERROR/slow, но не устраняет цену SDK. Все spans создаются до tail-решения. Малые различия latency/CPU не доказывают статистическую значимость; ограничения общего host, HTTP/DB workload и окна CPU приведены в method.md. Поздние spans за пределами decision window отдельно не доказаны.
|
||||
|
||||
Проверенная конфигурация: `OTEL_COLLECTOR_CONFIG=collector-balanced.yaml OTEL_METRIC_EXPORT_INTERVAL=30000 scripts/lab6.sh docker compose up -d --no-deps --wait otel-collector api relay worker`. Экспериментальные fault hooks по умолчанию выключены. После серий восстановлен default on/100%, рекомендация не включается скрыто.
|
||||
|
||||
comparison.json рассчитан из 15 raw JSON; comparison.png/svg показывают median, отдельные повторы и min/max. Старые evidence/track5 не изменены и не смешаны с этим набором.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-06T23:57:01Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode tail --repeat 1 --duration 40 --rate 20 --output evidence/forward-published/cost/tail-1.json
|
||||
{"mode": "tail", "repeat": 1, "result": "PASS", "retained": {"normal": 82, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 125.775
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:07:30Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode tail --repeat 2 --duration 40 --rate 20 --output evidence/forward-published/cost/tail-2.json
|
||||
{"mode": "tail", "repeat": 2, "result": "PASS", "retained": {"normal": 65, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 125.706
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,5 @@
|
||||
date: 2026-11-07T00:47:55Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_cost.py --mode tail --repeat 3 --duration 40 --rate 20 --output evidence/forward-published/cost/tail-3.json
|
||||
{"mode": "tail", "repeat": 3, "result": "PASS", "retained": {"normal": 88, "error": 16, "slow": 16}}
|
||||
exit_code: 0 elapsed_seconds: 127.417
|
||||
@@ -0,0 +1,230 @@
|
||||
date: 2026-11-06T23:49:55Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_check.py dashboards --strict
|
||||
{
|
||||
"checked_at": "2026-11-06T23:49:56+00:00",
|
||||
"targets": [
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "RPS бизнес-запросов",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Доля 5xx (2m)",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "p95 latency (5m)",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Доля не-5xx (окно 1h)",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "RPS по маршруту и классу статуса",
|
||||
"datasource": "prometheus",
|
||||
"series": 11,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Доля ошибок",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Доля ошибок",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Latency p50 / p95 / p99",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Latency p50 / p95 / p99",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Latency p50 / p95 / p99",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "p95 по маршруту",
|
||||
"datasource": "prometheus",
|
||||
"series": 9,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "RPS по реплике",
|
||||
"datasource": "prometheus",
|
||||
"series": 4,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Активные запросы",
|
||||
"datasource": "prometheus",
|
||||
"series": 6,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Логи API (ошибки и предупреждения)",
|
||||
"datasource": "loki",
|
||||
"lines": 5
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-api.json",
|
||||
"panel": "Трассы API с ошибкой",
|
||||
"datasource": "tempo",
|
||||
"traces": 5
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Jobs по исходу",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Длительность обработки job",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Длительность обработки job",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Длительность обработки job",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Логи worker",
|
||||
"datasource": "loki",
|
||||
"lines": 5
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Трассы worker",
|
||||
"datasource": "tempo",
|
||||
"traces": 5
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Durable queue depth (includes retry)",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Oldest durable job age",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "DB pool used / waiting / capacity",
|
||||
"datasource": "prometheus",
|
||||
"series": 9,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Worker busy / capacity",
|
||||
"datasource": "prometheus",
|
||||
"series": 2,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Process CPU cores",
|
||||
"datasource": "prometheus",
|
||||
"series": 3,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Process resident memory",
|
||||
"datasource": "prometheus",
|
||||
"series": 3,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Time to ready p50 / p95 / p99",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Time to ready p50 / p95 / p99",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Time to ready p50 / p95 / p99",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "Ready within 30s, SLO 95% / 1h",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
},
|
||||
{
|
||||
"dashboard": "messenger-pipeline.json",
|
||||
"panel": "API availability, SLO 99% / 1h",
|
||||
"datasource": "prometheus",
|
||||
"series": 1,
|
||||
"missing_metrics": []
|
||||
}
|
||||
],
|
||||
"problems": []
|
||||
}
|
||||
exit_code: 0 elapsed_seconds: 0.112
|
||||
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"at": "2026-11-07T01:11:54.405624+00:00",
|
||||
"process_settings": [
|
||||
{
|
||||
"name": "/mtusi2026lab6-api-1",
|
||||
"settings": {
|
||||
"OTEL_SDK_DISABLED": "false",
|
||||
"OTEL_TRACE_RATIO": "1",
|
||||
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
|
||||
"COST_EXPERIMENT": "false"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab6-api-2",
|
||||
"settings": {
|
||||
"OTEL_SDK_DISABLED": "false",
|
||||
"OTEL_TRACE_RATIO": "1",
|
||||
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
|
||||
"COST_EXPERIMENT": "false"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab6-relay-1",
|
||||
"settings": {
|
||||
"OTEL_SDK_DISABLED": "false",
|
||||
"OTEL_TRACE_RATIO": "1",
|
||||
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
|
||||
"COST_EXPERIMENT": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab6-worker-1",
|
||||
"settings": {
|
||||
"OTEL_SDK_DISABLED": "false",
|
||||
"OTEL_TRACE_RATIO": "1",
|
||||
"OTEL_METRIC_EXPORT_INTERVAL": "10000",
|
||||
"COST_EXPERIMENT": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"collector_command": [
|
||||
"--config=/etc/otelcol/collector.yaml"
|
||||
],
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
date: 2026-11-06T23:44:50Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ .venv/bin/python -c 'import sqlalchemy,alembic; from sqlalchemy import select,literal; from sqlalchemy.dialects import postgresql; print("SQLAlchemy",sqlalchemy.__version__,"Alembic",alembic.__version__); print(select(literal(1)).with_for_update(read=True,key_share=True).compile(dialect=postgresql.dialect()))'
|
||||
SQLAlchemy 2.0.45 Alembic 1.16.5
|
||||
SELECT %(param_1)s AS anon_1 FOR KEY SHARE
|
||||
exit_code: 0 elapsed_seconds: 0.182
|
||||
@@ -0,0 +1,8 @@
|
||||
date: 2026-11-06T23:45:11Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18086"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.876}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
exit_code: 0 elapsed_seconds: 16.433
|
||||
@@ -0,0 +1,14 @@
|
||||
date: 2026-11-07T01:12:37Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ make check validate lint
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
.venv/bin/python scripts/validate_openapi.py
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
.venv/bin/ruff check src student-tests scripts/obs_cost_summary.py scripts/obs_cost.py scripts/obs_incidents.py scripts/obs_recovery.py scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
|
||||
All checks passed!
|
||||
.venv/bin/ruff format --check src student-tests scripts/obs_cost_summary.py scripts/obs_cost.py scripts/obs_incidents.py scripts/obs_recovery.py scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
|
||||
216 files already formatted
|
||||
.venv/bin/ty check src student-tests alembic
|
||||
All checks passed!
|
||||
exit_code: 0 elapsed_seconds: 0.763
|
||||
@@ -0,0 +1,6 @@
|
||||
date: 2026-11-07T01:11:54Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ make test
|
||||
python3 tests/smoke.py --base-url "http://127.0.0.1:18086"
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
exit_code: 0 elapsed_seconds: 1.813
|
||||
@@ -0,0 +1,22 @@
|
||||
[
|
||||
{
|
||||
"severity": "minor",
|
||||
"file": "scripts/limits.py:33; .github/workflows/application.yml:85",
|
||||
"problem": "Inherited service allowlists omit OTel/helper services. Runtime limits checked independently in evidence/grade3/runtime.json; helper audit remains incompatible. Late-forward audit: evidence/forward-published/runtime.json confirms current container limits independently."
|
||||
},
|
||||
{
|
||||
"severity": "minor",
|
||||
"file": "infra/prometheus/rules/messenger.alerts.yml",
|
||||
"problem": "API error-rate SLI excludes HAProxy-generated 503 and can miss the first burst of a new counter series. Full PostgreSQL pause did not fire (database-incident.txt); sustained SQL-lock incident is verified separately. Late-forward SQL-lock alert firing/resolved: evidence/forward-published/incident-database.txt. No alert rules changed."
|
||||
},
|
||||
{
|
||||
"severity": "minor",
|
||||
"file": "infra/otel/collector.yaml; compose.yaml",
|
||||
"problem": "Collector reached 252.7 MiB / 256 MiB (98.7%) during storage outage without observed OOM; memory headroom is small (storage-incident.txt). Configuration unchanged, evidence/forward-published/preserved-mechanisms.json; no claim of a new storage-outage run."
|
||||
},
|
||||
{
|
||||
"severity": "minor",
|
||||
"file": "evidence/track5/comparison.svg; evidence/track5/*.txt; openspec/specs/lab6-observability-cost/spec.md; evidence/forward-published/cost/comparison.svg; evidence/forward-published/*.txt",
|
||||
"problem": "git show --check reports trailing whitespace in generated SVG/raw command output and a final blank line in generated OpenSpec. This cosmetic check did not pass; make lint passed. Raw output preserved without reformatting. Current evidence has the same cosmetic class; actual result retained in evidence/forward-published/whitespace-check.txt. No source formatting failure."
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,52 @@
|
||||
date: 2026-11-06T23:41:45Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ sh -c '
|
||||
set -eu
|
||||
export API_IMAGE=mtusi-messenger:mtusi2026lab6
|
||||
dc() { docker compose -p hlpipe-lab6-forward-lb -f student-tests/tools/lb_errors/compose.yaml "$@"; }
|
||||
trap "dc down" EXIT
|
||||
dc up -d --wait api lb
|
||||
dc run --rm probe 502
|
||||
dc run --rm probe 504
|
||||
dc stop api
|
||||
dc run --rm probe 503
|
||||
'
|
||||
Network hlpipe-lab6-forward-lb_default Creating
|
||||
Network hlpipe-lab6-forward-lb_default Created
|
||||
Container hlpipe-lab6-forward-lb-api-1 Creating
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Creating
|
||||
Container hlpipe-lab6-forward-lb-api-1 Created
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Created
|
||||
Container hlpipe-lab6-forward-lb-api-1 Starting
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Starting
|
||||
Container hlpipe-lab6-forward-lb-api-1 Started
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Started
|
||||
Container hlpipe-lab6-forward-lb-api-1 Waiting
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Waiting
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Healthy
|
||||
Container hlpipe-lab6-forward-lb-api-1 Healthy
|
||||
{"status": 502, "request_id": "a\"b", "json_valid": true, "echo_matches": true, "seconds": 0.001}
|
||||
{"status": 502, "request_id": "a\\b", "json_valid": true, "echo_matches": true, "seconds": 0.001}
|
||||
{"status": 502, "request_id": "a\tb", "json_valid": true, "echo_matches": true, "seconds": 0.001}
|
||||
{"status": 502, "request_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "json_valid": true, "echo_matches": true, "seconds": 0.001}
|
||||
{"status": 504, "request_id": "a\"b", "json_valid": true, "echo_matches": true, "seconds": 10.012}
|
||||
{"status": 504, "request_id": "a\\b", "json_valid": true, "echo_matches": true, "seconds": 10.006}
|
||||
{"status": 504, "request_id": "a\tb", "json_valid": true, "echo_matches": true, "seconds": 10.012}
|
||||
{"status": 504, "request_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "json_valid": true, "echo_matches": true, "seconds": 10.005}
|
||||
Container hlpipe-lab6-forward-lb-api-1 Stopping
|
||||
Container hlpipe-lab6-forward-lb-api-1 Stopped
|
||||
{"status": 503, "request_id": "a\"b", "json_valid": true, "echo_matches": true, "seconds": 0.004}
|
||||
{"status": 503, "request_id": "a\\b", "json_valid": true, "echo_matches": true, "seconds": 0.0}
|
||||
{"status": 503, "request_id": "a\tb", "json_valid": true, "echo_matches": true, "seconds": 0.0}
|
||||
{"status": 503, "request_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "json_valid": true, "echo_matches": true, "seconds": 0.0}
|
||||
Container hlpipe-lab6-forward-lb-api-1 Stopping
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Stopping
|
||||
Container hlpipe-lab6-forward-lb-api-1 Stopped
|
||||
Container hlpipe-lab6-forward-lb-api-1 Removing
|
||||
Container hlpipe-lab6-forward-lb-api-1 Removed
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Stopped
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Removing
|
||||
Container hlpipe-lab6-forward-lb-lb-1 Removed
|
||||
Network hlpipe-lab6-forward-lb_default Removing
|
||||
Network hlpipe-lab6-forward-lb_default Removed
|
||||
exit_code: 0 elapsed_seconds: 56.915
|
||||
@@ -0,0 +1,570 @@
|
||||
date: 2026-11-06T23:46:44Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_incidents.py chain
|
||||
{
|
||||
"started": "2026-11-06T23:46:44+00:00",
|
||||
"finished": "2026-11-06T23:46:51+00:00",
|
||||
"result": "PASS",
|
||||
"scenario": "chain",
|
||||
"observations": {
|
||||
"upload_request_id": "353d6004-0fce-4307-b74f-3e1330e099f1",
|
||||
"trace_id": "2a788dd8fe3bdba6c2b00ff3c6578b34",
|
||||
"retry": false,
|
||||
"outcomes": [
|
||||
"done"
|
||||
],
|
||||
"spans": [
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "POST /api/v1/chats/{chat_id}/attachments",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "1KnDTB7ilNc=",
|
||||
"parent_span_id": null,
|
||||
"duration_ms": 12.268,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "kOTXX/pL+PI=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.111,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "N7KwpItUcOg=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.063,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "ExH/3uFnT+c=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.049,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "6r9mViaW30k=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.059,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "VV9EDONJaKo=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.176,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "LGXb7KbUBKk=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.054,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "cY2vJIoHh+I=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.058,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "4WNnApqEnW4=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.046,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "afET0pqdh0w=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.042,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "qqkEFWjNMRc=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.044,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "B2NutixhjjM=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.185,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "iXFIfBn80Z4=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.222,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "ClGR/8bvx7s=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.091,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "KHHQiGtugf8=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.055,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "h6ioIL28mmE=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.062,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "d1s9gYOAjTY=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.044,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "Z73gIsACvyk=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.226,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "ZU4nVi/O270=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.155,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "2xriCNx45Ss=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.142,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "7clq4wwABWo=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.117,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "HCWi0TAraeg=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.192,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "eqo0RZhx2r4=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 5.315,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-relay",
|
||||
"name": "attachments.attachment.process.v1 create",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "DdVW16xtPjM=",
|
||||
"parent_span_id": "1KnDTB7ilNc=",
|
||||
"duration_ms": 0.033,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-relay",
|
||||
"name": "attachments.attachment.process.v1 publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "T6EWXaDDQRY=",
|
||||
"parent_span_id": "DdVW16xtPjM=",
|
||||
"duration_ms": 4.356,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "attachments.attachment.process.v1 process",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "IJrEIXz5vQU=",
|
||||
"parent_span_id": "DdVW16xtPjM=",
|
||||
"duration_ms": 42.854,
|
||||
"attempt": "1"
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "6yaGj8OjHpo=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.284,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "wvYv52Ceymw=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.195,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "IXF/7Dnea8I=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.17,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "ZFDgbFUJ38s=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.164,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "IiJRka3nsOk=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.55,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "Ql3IoVPxbuk=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.385,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "dZVxc0YqKd4=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.207,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "z1F9TtIQdcA=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.185,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "eMgOi4KYRuM=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.571,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "2qFq/3351v8=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.173,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "f3lKJAgUqPM=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.142,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "LN76EV7HKBo=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.133,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "jylzdd/SomQ=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.132,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "ZrNx+53YHDg=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.282,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "C3gPIERIt/0=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.45,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "o//TrE5/tR8=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.095,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "eU7D/wTzm3s=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.082,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "hbXwj1bqqr0=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.066,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "7WW7O9uzEag=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.081,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "R5CuUJ4DKx4=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.139,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "BscD3XrQ6IM=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.236,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "yJ5825EbSlY=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.177,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "ctFlHkHK3w4=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.131,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "miaP4WbLSsw=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 0.2,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 get_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "51Ks1BHIgsY=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 3.52,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 copy_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "xKXCi8QQTjI=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 6.112,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "7yfvSxRV3Io=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 3.501,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "qW0rRHnhoqo=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 3.845,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "8LvmSmoKWfM=",
|
||||
"parent_span_id": "IJrEIXz5vQU=",
|
||||
"duration_ms": 3.201,
|
||||
"attempt": null
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
exit_code: 0 elapsed_seconds: 7.279
|
||||
@@ -0,0 +1,730 @@
|
||||
date: 2026-11-06T23:48:04Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_incidents.py database
|
||||
{
|
||||
"started": "2026-11-06T23:48:04+00:00",
|
||||
"finished": "2026-11-06T23:49:55+00:00",
|
||||
"result": "PASS",
|
||||
"scenario": "database",
|
||||
"observations": {
|
||||
"paused": "2026-11-06T23:48:16+00:00",
|
||||
"restored": "2026-11-06T23:48:59+00:00",
|
||||
"requests": [
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-10",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-11",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-12",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-13",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-14",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-15",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-16",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-17",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-18",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-19",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-20",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-21",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-22",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-23",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-24",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-25",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-26",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-27",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-28",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-29",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-30",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-31",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-32",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-33",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-34",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-35",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-36",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-37",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-38",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-39",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-40",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-41",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-42",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-43",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-44",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-45",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-46",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-47",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-48",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-49",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-50",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-51",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-52",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-53",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-54",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-55",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-56",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-57",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-58",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-59",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-60",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-61",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-62",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-63",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-64",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-65",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-66",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-67",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-68",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-69",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-70",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-71",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-72",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-73",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-74",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-75",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-76",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-77",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-78",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-79",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-80",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-81",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-82",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-83",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-84",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-85",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-86",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-87",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-88",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-89",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-90",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-91",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-92",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-93",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-94",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-95",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-96",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-97",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-98",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-99",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-100",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-101",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-102",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-103",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-104",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-105",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-106",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-107",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-108",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-109",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-110",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-111",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-112",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-113",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-114",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-115",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-116",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-117",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-118",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-119",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-120",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-121",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-122",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-123",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-124",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-125",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-126",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-127",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-128",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-129",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-130",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-131",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-132",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-133",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-134",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-135",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-136",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-137",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-138",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-139",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-140",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-141",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-142",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-143",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-144",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-145",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-146",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-147",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-148",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-149",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-150",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-151",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-152",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-153",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-154",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-155",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-156",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-157",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-158",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-159",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-160",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-161",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-162",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-163",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-164",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-165",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-166",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-167",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-168",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-169",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-170",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-171",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-172",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-173",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-174",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-175",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-176",
|
||||
"status": 503
|
||||
},
|
||||
{
|
||||
"request_id": "db-incident-8b5642602908-177",
|
||||
"status": 503
|
||||
}
|
||||
],
|
||||
"signals": {
|
||||
"error_ratio": 0.9181136211080942,
|
||||
"pool_waiting": 0.0
|
||||
},
|
||||
"firing": {
|
||||
"utc": "2026-11-06T23:48:59+00:00",
|
||||
"state": "firing",
|
||||
"alerts": [
|
||||
{
|
||||
"labels": {
|
||||
"alertname": "MessengerHighErrorRate",
|
||||
"severity": "warning"
|
||||
},
|
||||
"annotations": {
|
||||
"description": "Availability target is 99% over 1 hour; health and client 4xx are excluded from failures.",
|
||||
"runbook": "infra/prometheus/runbooks.md#messengerhigherrorrate",
|
||||
"summary": "More than 5% business HTTP 5xx over 1 minute"
|
||||
},
|
||||
"state": "firing",
|
||||
"activeAt": "2026-11-06T23:48:43.389340978Z",
|
||||
"value": "9.181136211080942e-01"
|
||||
}
|
||||
]
|
||||
},
|
||||
"resolved": {
|
||||
"utc": "2026-11-06T23:49:54+00:00",
|
||||
"state": "resolved",
|
||||
"alerts": []
|
||||
},
|
||||
"dependency_traces": [
|
||||
{
|
||||
"trace_id": "1cee5df1144b8c4eb2b801d449bfc0d4",
|
||||
"db_errors": [
|
||||
{
|
||||
"name": "SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_ERROR",
|
||||
"duration_ms": 2002.866
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
exit_code: 0 elapsed_seconds: 111.140
|
||||
@@ -0,0 +1,881 @@
|
||||
date: 2026-11-06T23:46:51Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_incidents.py retry
|
||||
{
|
||||
"started": "2026-11-06T23:46:51+00:00",
|
||||
"finished": "2026-11-06T23:47:04+00:00",
|
||||
"result": "PASS",
|
||||
"scenario": "retry",
|
||||
"observations": {
|
||||
"upload_request_id": "6d25dbc3-0394-4733-8ecb-63c1ac721847",
|
||||
"trace_id": "933831a64efb8a5deb62b2669afc9698",
|
||||
"retry": true,
|
||||
"outcomes": [
|
||||
"retry_scheduled",
|
||||
"done"
|
||||
],
|
||||
"spans": [
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "POST /api/v1/chats/{chat_id}/attachments",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "Debvq4Digx0=",
|
||||
"parent_span_id": null,
|
||||
"duration_ms": 21.315,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "Waullyq3N34=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.148,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "K0kakII6xo4=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.094,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "AiKsaFLbKb8=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.088,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "VrrcbEJGSCc=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.076,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "AEIu7yg0sn0=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.247,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "Wng5k5TYN4E=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.118,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "sQsK2QPQPJU=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.106,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "7asDsVS6+xs=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.089,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "EdJsPxFHnjM=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.092,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "PcoY45akFEQ=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.072,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "CdbrJRYuFnE=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.438,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "DBBcI9Ulv/M=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.317,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "VtOmph6FOBI=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.104,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "DPXHG5HHGa8=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.068,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "1BvLOYIb1uQ=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.074,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "WUjMiYKCT8Q=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.069,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "hyJX1jww6uc=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.305,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "48fAH8bMuSA=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.294,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "bT2xInlvctI=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.284,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "k/ENR5nG9/g=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.153,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "t3n5uTl2PlM=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.362,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "0U2pHU/RDjU=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 5.47,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-relay",
|
||||
"name": "attachments.attachment.process.v1 create",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "bwRa62vgs2k=",
|
||||
"parent_span_id": "Debvq4Digx0=",
|
||||
"duration_ms": 0.008,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-relay",
|
||||
"name": "attachments.attachment.process.v1 publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "KBBn1geg3IA=",
|
||||
"parent_span_id": "bwRa62vgs2k=",
|
||||
"duration_ms": 1.818,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "attachments.attachment.process.v1 process",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"status": "STATUS_CODE_ERROR",
|
||||
"span_id": "BfeHo65w9oE=",
|
||||
"parent_span_id": "bwRa62vgs2k=",
|
||||
"duration_ms": 1042.638,
|
||||
"attempt": "1"
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "bukxkY2R92E=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.116,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "select",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "FcFtes5C2S0=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.122,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "select",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "VnuhpEWn90A=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.109,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "show",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "c4vU6fUOroM=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.087,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "show",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "vs4kP7fYreM=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.082,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "QRTMZ3eRZuQ=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.087,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "QDbAA2U75Jw=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.086,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "i6m5Do6EEvo=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.562,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "dFJa0q3l5c4=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.545,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "H3ogXCCzG0A=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.211,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "RaAOGgl3WQk=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.167,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "/r2s8Kv3oxI=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.365,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "kAk9rtMbpVM=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.088,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "9IR9tpNfKLc=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.167,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "M9S+7N7xuvQ=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.072,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "AFBHKda3lQU=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.086,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "fmP3rOGz/6Y=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.149,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "4bmV8TnBwkU=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.228,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "VlQ2TBIQJ18=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.284,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "g+FdHSn48nA=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.229,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "IcDc7gRMQrI=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.205,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "pUxoqQhJ7e8=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.223,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "PGfPW8e8aUc=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.599,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "O8F+/YbnorM=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.438,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "G3uEKRm668s=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.282,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "N9g279eP5uk=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.596,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "AnDJGXjW+eE=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 1.173,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "UkN0hQ86NWs=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.192,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "7uLkyTzXADQ=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.141,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "8oCbHKZgLgA=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.129,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "rEsJdw/1gEA=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.107,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "6rP7QLkb1I8=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.342,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "fXmqpv3ea4A=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.243,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "4n80obWEpfo=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.153,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "SELECT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "w060gr0ejnc=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.126,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "BK1ZrQHnpFA=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.291,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "10uHmCMpWZg=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.08,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "FeXJXVagnQk=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.07,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "Zy5J9SneT4I=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.061,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "OgHgDqHQkAA=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.069,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "UGk4+2mR1SY=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.147,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "JHlQA/pOcPQ=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.204,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 get_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_ERROR",
|
||||
"span_id": "+ZFBPxYYZB8=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 979.107,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-relay",
|
||||
"name": "attachments.attachment.process.v1 create",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "ElpYDj40+k0=",
|
||||
"parent_span_id": "BfeHo65w9oE=",
|
||||
"duration_ms": 0.02,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-relay",
|
||||
"name": "attachments.attachment.process.v1 publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "orqs4nj6zdU=",
|
||||
"parent_span_id": "ElpYDj40+k0=",
|
||||
"duration_ms": 3.277,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "attachments.attachment.process.v1 process",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "8xpUCowkZ+s=",
|
||||
"parent_span_id": "ElpYDj40+k0=",
|
||||
"duration_ms": 2490.326,
|
||||
"attempt": "2"
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 get_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "zhEmRuqzYl4=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 1755.064,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 copy_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "uvNwJ0jDn4g=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 6.867,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "gkl/eAs8x4o=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 3.521,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "wCX2paxCccQ=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 2.763,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "S3 put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "8UCt7bEeO+Q=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 2.431,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "VMB1TYgfTdk=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.134,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": ";",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "UHP5U7dMydE=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.063,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "ROLLBACK;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "GSoLKxyZ0WA=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.055,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "BEGIN;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "m+xW8xOWVyQ=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.06,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "c+H4LJWBCek=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.19,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "AYRYePBZsqE=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.343,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "UPDATE",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "XUj8IqOu4fY=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.34,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "INSERT",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "f1ZCF+vhHKg=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.106,
|
||||
"attempt": null
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "COMMIT;",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": "STATUS_CODE_UNSET",
|
||||
"span_id": "EGs4gQZnt0g=",
|
||||
"parent_span_id": "8xpUCowkZ+s=",
|
||||
"duration_ms": 0.219,
|
||||
"attempt": null
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
exit_code: 0 elapsed_seconds: 13.425
|
||||
@@ -0,0 +1,45 @@
|
||||
date: 2026-11-06T23:47:04Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ python3 scripts/obs_incidents.py worker
|
||||
{
|
||||
"started": "2026-11-06T23:47:04+00:00",
|
||||
"finished": "2026-11-06T23:48:04+00:00",
|
||||
"result": "PASS",
|
||||
"scenario": "worker",
|
||||
"observations": {
|
||||
"stopped": "2026-11-06T23:47:05+00:00",
|
||||
"restored": "2026-11-06T23:47:52+00:00",
|
||||
"jobs": 8,
|
||||
"backlog": {
|
||||
"depth": 8.0,
|
||||
"age_seconds": 34.470043
|
||||
},
|
||||
"firing": {
|
||||
"utc": "2026-11-06T23:47:49+00:00",
|
||||
"state": "firing",
|
||||
"alerts": [
|
||||
{
|
||||
"labels": {
|
||||
"alertname": "MessengerQueueLag",
|
||||
"severity": "warning"
|
||||
},
|
||||
"annotations": {
|
||||
"description": "Target is 95% ready within 30 seconds over 1 hour; depth includes retry and leased work.",
|
||||
"runbook": "infra/prometheus/runbooks.md#messengerqueuelag",
|
||||
"summary": "Durable attachment backlog has waited more than 15 seconds"
|
||||
},
|
||||
"state": "firing",
|
||||
"activeAt": "2026-11-06T23:47:33.389340978Z",
|
||||
"value": "3.4470043e+01"
|
||||
}
|
||||
]
|
||||
},
|
||||
"resolved": {
|
||||
"utc": "2026-11-06T23:48:04+00:00",
|
||||
"state": "resolved",
|
||||
"alerts": []
|
||||
},
|
||||
"depth_after": 0.0
|
||||
}
|
||||
}
|
||||
exit_code: 0 elapsed_seconds: 59.808
|
||||
@@ -0,0 +1,154 @@
|
||||
date: 2026-11-06T23:37:24Z commit: bc1d7aa7713858acb5d65b61d07e69a640f1ee97
|
||||
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab6", "LB_HTTP_PORT": "18086", "LB_STATS_PORT": "26085", "BASE_URL": "http://127.0.0.1:18086", "NO_PROXY": "localhost,127.0.0.1,::1"}
|
||||
$ .venv/bin/python student-tests/tools/forward_network.py
|
||||
{
|
||||
"checks": [
|
||||
{
|
||||
"case": "non_ascii_basic",
|
||||
"raw_hex": "42617369632080",
|
||||
"status": 401,
|
||||
"instance": "88b81a690916"
|
||||
},
|
||||
{
|
||||
"case": "non_ascii_basic",
|
||||
"raw_hex": "426173696320ff",
|
||||
"status": 401,
|
||||
"instance": "cbf0e0c086a8"
|
||||
},
|
||||
{
|
||||
"case": "non_ascii_basic",
|
||||
"raw_hex": "426173696320c3a9",
|
||||
"status": 401,
|
||||
"instance": "88b81a690916"
|
||||
},
|
||||
{
|
||||
"case": "non_ascii_basic",
|
||||
"raw_hex": "42617369632059577870593255366332566a636d5630ff",
|
||||
"status": 401,
|
||||
"instance": "cbf0e0c086a8"
|
||||
},
|
||||
{
|
||||
"case": "nul_basic_display_title",
|
||||
"position": "'\\x00'",
|
||||
"status": [
|
||||
401,
|
||||
400,
|
||||
201,
|
||||
400
|
||||
],
|
||||
"instance": "88b81a690916"
|
||||
},
|
||||
{
|
||||
"case": "nul_basic_display_title",
|
||||
"position": "'\\x00abc'",
|
||||
"status": [
|
||||
401,
|
||||
400,
|
||||
201,
|
||||
400
|
||||
],
|
||||
"instance": "cbf0e0c086a8"
|
||||
},
|
||||
{
|
||||
"case": "nul_basic_display_title",
|
||||
"position": "'abc\\x00def'",
|
||||
"status": [
|
||||
401,
|
||||
400,
|
||||
201,
|
||||
400
|
||||
],
|
||||
"instance": "88b81a690916"
|
||||
},
|
||||
{
|
||||
"case": "nul_basic_display_title",
|
||||
"position": "'abc\\x00'",
|
||||
"status": [
|
||||
401,
|
||||
400,
|
||||
201,
|
||||
400
|
||||
],
|
||||
"instance": "cbf0e0c086a8"
|
||||
},
|
||||
{
|
||||
"case": "nul_message_corrected_retry",
|
||||
"position": "'\\x00'",
|
||||
"instances": [
|
||||
"cbf0e0c086a8",
|
||||
"88b81a690916"
|
||||
],
|
||||
"status": [
|
||||
400,
|
||||
201,
|
||||
201
|
||||
],
|
||||
"copies": 1
|
||||
},
|
||||
{
|
||||
"case": "nul_message_corrected_retry",
|
||||
"position": "'\\x00abc'",
|
||||
"instances": [
|
||||
"88b81a690916",
|
||||
"cbf0e0c086a8"
|
||||
],
|
||||
"status": [
|
||||
400,
|
||||
201,
|
||||
201
|
||||
],
|
||||
"copies": 1
|
||||
},
|
||||
{
|
||||
"case": "nul_message_corrected_retry",
|
||||
"position": "'abc\\x00def'",
|
||||
"instances": [
|
||||
"cbf0e0c086a8",
|
||||
"88b81a690916"
|
||||
],
|
||||
"status": [
|
||||
400,
|
||||
201,
|
||||
201
|
||||
],
|
||||
"copies": 1
|
||||
},
|
||||
{
|
||||
"case": "nul_message_corrected_retry",
|
||||
"position": "'abc\\x00'",
|
||||
"instances": [
|
||||
"88b81a690916",
|
||||
"cbf0e0c086a8"
|
||||
],
|
||||
"status": [
|
||||
400,
|
||||
201,
|
||||
201
|
||||
],
|
||||
"copies": 1
|
||||
},
|
||||
{
|
||||
"case": "nul_access",
|
||||
"status": [
|
||||
404,
|
||||
404
|
||||
]
|
||||
},
|
||||
{
|
||||
"case": "logout_across_replicas",
|
||||
"status": [
|
||||
204,
|
||||
401,
|
||||
401,
|
||||
401,
|
||||
401
|
||||
]
|
||||
}
|
||||
],
|
||||
"instances": [
|
||||
"88b81a690916",
|
||||
"cbf0e0c086a8"
|
||||
],
|
||||
"pass": true
|
||||
}
|
||||
exit_code: 0 elapsed_seconds: 0.326
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user