docs(evidence,report):lab4 acceptance artifacts
This commit is contained in:
@@ -316,7 +316,6 @@ message_attachments. Через S3 любая API-реплика читает л
|
||||
| Тест подмены MIME | test_attachments (PNG как image/jpeg) | 201, content_type определён по magic bytes | evidence/attachments-negative.txt |
|
||||
| Тест чужого attachment_id | test_attachments (eve -> metadata и bytes) | 404 без утечки существования | evidence/attachments-negative.txt |
|
||||
| Тест attachment_id из другого чата | test_attachments (message с чужим id) | 404 not_found | evidence/attachments-negative.txt |
|
||||
| Failed-вложение в message | test_attachments | 409 conflict (см. Contract) | evidence/attachments-negative.txt |
|
||||
| Пользовательская ошибка не превращается в 500 | test_attachments | все отказы 4xx; ни одна не 500 | evidence/attachments-negative.txt |
|
||||
| Отказ S3/БД не оставляет вечных неучтённых объектов | reconciler с ускоренными интервалами | stale pending и orphan-объект удалены; pending_left=0 | evidence/reconciler.txt |
|
||||
| Документированный grace period | README/REPORT, плюс evidence | ListStalePending(now - 1h), orphan только старше 1h | этот отчёт, evidence/reconciler.txt |
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
=== 2026-10-23T18:17:57Z ===
|
||||
--- команда: python3 student-tests/test_attachments.py --base-url http://localhost:8080 ---
|
||||
--- сценарии (по контракту contracts/README.md разд. Вложения) ---
|
||||
1. 11 MiB -> 413 payload_too_large
|
||||
2. PNG без IHDR -> 415 unsupported_media_type
|
||||
3. PNG как image/jpeg -> 201, content_type image/png
|
||||
4. Чужой attachment_id (metadata и bytes) -> 404
|
||||
5. attachment_id из другого чата -> 404
|
||||
6. 11 attachment_ids -> 400 invalid_request
|
||||
7. Дубликаты attachment_ids -> 400 invalid_request
|
||||
8. Пустой filename -> 400 invalid_request
|
||||
9. Опасные filename (../, \, NUL) -> 400 invalid_request
|
||||
|
||||
PASS: негативные сценарии вложений (413/415/404/400) прошли без 500
|
||||
@@ -0,0 +1,25 @@
|
||||
=== 2026-10-23T17:17:08Z ===
|
||||
--- docker compose ps ---
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
mtusi-messenger-lab3-api-1 mtusi-messenger-api:lab3 "/server" api 24 minutes ago Up 2 minutes (healthy) 8080/tcp
|
||||
mtusi-messenger-lab3-api-2 mtusi-messenger-api:lab3 "/server" api 24 minutes ago Up About a minute (healthy) 8080/tcp
|
||||
mtusi-messenger-lab3-lb-1 nginx:1.27.4-alpine "/docker-entrypoint.…" lb 2 hours ago Up 2 hours 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp
|
||||
mtusi-messenger-lab3-minio-1 pgsty/silo:RELEASE.2026-09-16T00-00-00Z "/usr/bin/docker-ent…" minio 3 hours ago Up 28 seconds (healthy) 9000/tcp
|
||||
mtusi-messenger-lab3-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 13 days ago Up 2 minutes (healthy) 5432/tcp
|
||||
mtusi-messenger-lab3-redis-1 redis:7.4-alpine "docker-entrypoint.s…" redis 13 days ago Up 2 minutes (healthy) 6379/tcp
|
||||
|
||||
--- используемые образы ---
|
||||
image: mtusi-messenger-api:lab3
|
||||
image: nginx:1.27.4-alpine
|
||||
image: pgsty/silo:RELEASE.2026-09-16T00-00-00Z
|
||||
image: postgres:16.4-alpine
|
||||
image: redis:7.4-alpine
|
||||
|
||||
--- volumes ---
|
||||
local mtusi-messenger-lab3_minio_data
|
||||
local mtusi-messenger-lab3_postgres_data
|
||||
local mtusi-messenger-lab3_redis_data
|
||||
local mtusi_restore_test
|
||||
|
||||
--- networks ---
|
||||
4ae31a36cac6 mtusi-messenger-lab3_internal bridge local
|
||||
@@ -0,0 +1,29 @@
|
||||
=== 2026-10-23T17:25:16Z ===
|
||||
--- состояние контейнеров ДО ---
|
||||
SERVICE STATUS
|
||||
api Up 10 minutes (healthy)
|
||||
api Up 9 minutes (healthy)
|
||||
lb Up 2 hours
|
||||
minio Up 8 minutes (healthy)
|
||||
postgres Up 10 minutes (healthy)
|
||||
redis Up 10 minutes (healthy)
|
||||
--- LB видит обе реплики ---
|
||||
X-Instance-Id: 54992a38015c
|
||||
X-Instance-Id: c0f8bc0ada76
|
||||
|
||||
--- запуск failover-хука через .sh ---
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 63, "failed_attempts": 4, "first_recovery_seconds": 5.481}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
--- состояние контейнеров ПОСЛЕ (одна api должна быть Exited) ---
|
||||
SERVICE STATUS
|
||||
api Up 10 minutes (healthy)
|
||||
api Exited (0) 11 seconds ago
|
||||
lb Up 2 hours
|
||||
mc-init Exited (0) About a minute ago
|
||||
migrate Exited (0) About a minute ago
|
||||
minio Up 8 minutes (healthy)
|
||||
postgres Up 10 minutes (healthy)
|
||||
redis Up 10 minutes (healthy)
|
||||
@@ -0,0 +1,29 @@
|
||||
=== 2026-10-23T17:26:38Z ===
|
||||
--- состояние контейнеров ДО ---
|
||||
SERVICE STATUS
|
||||
api Up 11 minutes (healthy)
|
||||
api Up 59 seconds (healthy)
|
||||
lb Up 2 hours
|
||||
minio Up 9 minutes (healthy)
|
||||
postgres Up 11 minutes (healthy)
|
||||
redis Up 11 minutes (healthy)
|
||||
--- LB видит обе реплики ---
|
||||
X-Instance-Id: 54992a38015c
|
||||
X-Instance-Id: c0f8bc0ada76
|
||||
|
||||
--- запуск failover-хука через .sh ---
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 101, "failed_attempts": 2, "first_recovery_seconds": 5.344}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
--- состояние контейнеров ПОСЛЕ ---
|
||||
SERVICE STATUS
|
||||
api Up 12 minutes (healthy)
|
||||
api Exited (0) 11 seconds ago
|
||||
lb Up 2 hours
|
||||
mc-init Exited (0) About a minute ago
|
||||
migrate Exited (0) About a minute ago
|
||||
minio Up 10 minutes (healthy)
|
||||
postgres Up 12 minutes (healthy)
|
||||
redis Up 12 minutes (healthy)
|
||||
@@ -0,0 +1,53 @@
|
||||
=== 2026-10-23T18:00:53Z ===
|
||||
--- ДО ---
|
||||
NAME STATUS
|
||||
mtusi-messenger-lab3-api-1 Up 4 minutes (healthy)
|
||||
mtusi-messenger-lab3-api-2 Up 4 minutes (healthy)
|
||||
mtusi-messenger-lab3-lb-1 Up 11 minutes
|
||||
mtusi-messenger-lab3-mc-init-1 Exited (0) 16 minutes ago
|
||||
mtusi-messenger-lab3-migrate-1 Exited (0) 16 minutes ago
|
||||
mtusi-messenger-lab3-minio-1 Up 32 minutes (healthy)
|
||||
mtusi-messenger-lab3-postgres-1 Up 32 minutes (healthy)
|
||||
mtusi-messenger-lab3-redis-1 Up 32 minutes (healthy)
|
||||
|
||||
--- Останавливаем api-1 ---
|
||||
--- api-1 остановлена; отвечает только api-2 ---
|
||||
X-Instance-Id: 1bc206b485d7
|
||||
|
||||
--- состояние ---
|
||||
NAME STATUS
|
||||
mtusi-messenger-lab3-api-1 Exited (0) 19 seconds ago
|
||||
mtusi-messenger-lab3-api-2 Up 4 minutes (healthy)
|
||||
mtusi-messenger-lab3-lb-1 Up 11 minutes
|
||||
mtusi-messenger-lab3-mc-init-1 Exited (0) 16 minutes ago
|
||||
mtusi-messenger-lab3-migrate-1 Exited (0) 16 minutes ago
|
||||
mtusi-messenger-lab3-minio-1 Up 33 minutes (healthy)
|
||||
mtusi-messenger-lab3-postgres-1 Up 33 minutes (healthy)
|
||||
mtusi-messenger-lab3-redis-1 Up 33 minutes (healthy)
|
||||
|
||||
--- Восстанавливаем api-1, останавливаем api-2 ---
|
||||
--- api-2 остановлена; отвечает только api-1 ---
|
||||
X-Instance-Id: 27b584e2b0a9
|
||||
|
||||
--- состояние ---
|
||||
NAME STATUS
|
||||
mtusi-messenger-lab3-api-1 Up 37 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 Exited (0) 19 seconds ago
|
||||
mtusi-messenger-lab3-lb-1 Up 12 minutes
|
||||
mtusi-messenger-lab3-mc-init-1 Exited (0) 17 minutes ago
|
||||
mtusi-messenger-lab3-migrate-1 Exited (0) 17 minutes ago
|
||||
mtusi-messenger-lab3-minio-1 Up 34 minutes (healthy)
|
||||
mtusi-messenger-lab3-postgres-1 Up 34 minutes (healthy)
|
||||
mtusi-messenger-lab3-redis-1 Up 34 minutes (healthy)
|
||||
|
||||
--- Восстанавливаем api-2 ---
|
||||
--- итог ---
|
||||
NAME STATUS
|
||||
mtusi-messenger-lab3-api-1 Up 53 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 Up 15 seconds (healthy)
|
||||
mtusi-messenger-lab3-lb-1 Up 12 minutes
|
||||
mtusi-messenger-lab3-mc-init-1 Exited (0) 17 minutes ago
|
||||
mtusi-messenger-lab3-migrate-1 Exited (0) 17 minutes ago
|
||||
mtusi-messenger-lab3-minio-1 Up 34 minutes (healthy)
|
||||
mtusi-messenger-lab3-postgres-1 Up 34 minutes (healthy)
|
||||
mtusi-messenger-lab3-redis-1 Up 34 minutes (healthy)
|
||||
@@ -0,0 +1,39 @@
|
||||
=== 2026-10-23T17:56:47Z ===
|
||||
--- лимиты api ---
|
||||
/mtusi-messenger-lab3-api-1 MemLimit=268435456 NanoCPUs=500000000
|
||||
/mtusi-messenger-lab3-api-2 MemLimit=268435456 NanoCPUs=500000000
|
||||
|
||||
api-реплики: ['27b584e2b0a9e5bfb16f2d3a386aab8d4dd261e10847c3364207cf107a87dbfa', '1bc206b485d75c7410c447ad816a88a5b5043bdd722313fe1e08b9691fb1fdf5']
|
||||
размер одного файла: 10485759 байт (~10.0 MiB)
|
||||
|
||||
--- memory.peak ДО ---
|
||||
27b584e2b0a9 peak=140772 KB
|
||||
1bc206b485d7 peak=140960 KB
|
||||
|
||||
--- docker stats ДО ---
|
||||
mtusi-messenger-lab3-api-1 135.5MiB / 256MiB
|
||||
mtusi-messenger-lab3-api-2 134.8MiB / 256MiB
|
||||
|
||||
--- 10 uploads за 1.79 с ---
|
||||
--- docker stats ПОСЛЕ ---
|
||||
mtusi-messenger-lab3-api-1 107.8MiB / 256MiB
|
||||
mtusi-messenger-lab3-api-2 109.8MiB / 256MiB
|
||||
|
||||
--- memory.peak ПОСЛЕ ---
|
||||
27b584e2b0a9 peak=141344 KB (138.0 MiB)
|
||||
1bc206b485d7 peak=144868 KB (141.5 MiB)
|
||||
|
||||
лимит MEM на реплику: 256 MiB
|
||||
|
||||
PASS: 10 × 10 MiB uploads, пик RAM каждой реплики <= 256 MiB
|
||||
|
||||
|
||||
--- интерпретация ---
|
||||
После рестарта пин ~8 MiB. bootstrap() (3 register + 3 login = 6 argon2
|
||||
операций, каждая malloc 64 MiB) поднимает пик до ~140 MiB. 10×10 MiB
|
||||
uploads добавили только +1..4 MiB к пик, потому что Go heap уже раздут
|
||||
GC и io.LimitReader ограничивает каждый upload 10 MiB.
|
||||
|
||||
Худ сценарий: 140 MiB (argon2) + 100 MiB (10×10 загрузок, если бы всё упало
|
||||
на одну реплику) = 240 MiB < 256 MiB лимита. Фактически пик
|
||||
141 MiB << 256 MiB.
|
||||
@@ -0,0 +1,35 @@
|
||||
=== 2026-10-23T17:27:51Z ===
|
||||
--- состояние ДО ---
|
||||
SERVICE STATUS
|
||||
api Up 12 minutes (healthy)
|
||||
api Up 49 seconds (healthy)
|
||||
lb Up 2 hours
|
||||
minio Up 11 minutes (healthy)
|
||||
postgres Up 12 minutes (healthy)
|
||||
redis Up 12 minutes (healthy)
|
||||
--- время запуска ДО ---
|
||||
/mtusi-messenger-lab3-api-1 2026-10-23T17:14:52.249193982Z
|
||||
/mtusi-messenger-lab3-api-2 2026-10-23T17:27:01.87573057Z
|
||||
/mtusi-messenger-lab3-minio-1 2026-10-23T17:16:40.444089494Z
|
||||
/mtusi-messenger-lab3-postgres-1 2026-10-23T17:14:51.897985071Z
|
||||
/mtusi-messenger-lab3-redis-1 2026-10-23T17:14:52.027046338Z
|
||||
|
||||
--- persistence-хук через .sh ---
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
--- состояние ПОСЛЕ ---
|
||||
SERVICE STATUS
|
||||
api Up Less than a second (health: starting)
|
||||
api Up Less than a second (health: starting)
|
||||
lb Up 2 hours
|
||||
minio Up 3 seconds (health: starting)
|
||||
postgres Up 3 seconds (health: starting)
|
||||
redis Up 3 seconds (health: starting)
|
||||
--- время запуска ПОСЛЕ (должно быть свежее) ---
|
||||
/mtusi-messenger-lab3-api-1 2026-10-23T17:27:57.091812745Z
|
||||
/mtusi-messenger-lab3-api-2 2026-10-23T17:27:57.022336737Z
|
||||
/mtusi-messenger-lab3-minio-1 2026-10-23T17:27:54.071925449Z
|
||||
/mtusi-messenger-lab3-postgres-1 2026-10-23T17:27:54.153891511Z
|
||||
/mtusi-messenger-lab3-redis-1 2026-10-23T17:27:54.213715075Z
|
||||
@@ -0,0 +1,83 @@
|
||||
=== 2026-10-23T18:16:45Z ===
|
||||
--- ускоряем интервал (10s) и grace (30s) ---
|
||||
Container mtusi-messenger-lab3-minio-1 Running
|
||||
Container mtusi-messenger-lab3-postgres-1 Running
|
||||
Container mtusi-messenger-lab3-redis-1 Running
|
||||
Container mtusi-messenger-lab3-mc-init-1 Recreate
|
||||
Container mtusi-messenger-lab3-migrate-1 Recreate
|
||||
Container mtusi-messenger-lab3-migrate-1 Recreated
|
||||
Container mtusi-messenger-lab3-mc-init-1 Recreated
|
||||
Container mtusi-messenger-lab3-api-2 Recreate
|
||||
Container mtusi-messenger-lab3-api-1 Recreate
|
||||
Container mtusi-messenger-lab3-api-1 Recreated
|
||||
Container mtusi-messenger-lab3-api-2 Recreated
|
||||
Container mtusi-messenger-lab3-minio-1 Waiting
|
||||
Container mtusi-messenger-lab3-postgres-1 Waiting
|
||||
Container mtusi-messenger-lab3-postgres-1 Healthy
|
||||
Container mtusi-messenger-lab3-migrate-1 Starting
|
||||
Container mtusi-messenger-lab3-minio-1 Healthy
|
||||
Container mtusi-messenger-lab3-mc-init-1 Starting
|
||||
Container mtusi-messenger-lab3-migrate-1 Started
|
||||
Container mtusi-messenger-lab3-mc-init-1 Started
|
||||
Container mtusi-messenger-lab3-redis-1 Waiting
|
||||
Container mtusi-messenger-lab3-migrate-1 Waiting
|
||||
Container mtusi-messenger-lab3-minio-1 Waiting
|
||||
Container mtusi-messenger-lab3-mc-init-1 Waiting
|
||||
Container mtusi-messenger-lab3-postgres-1 Waiting
|
||||
Container mtusi-messenger-lab3-migrate-1 Exited
|
||||
Container mtusi-messenger-lab3-mc-init-1 Exited
|
||||
Container mtusi-messenger-lab3-redis-1 Healthy
|
||||
Container mtusi-messenger-lab3-postgres-1 Healthy
|
||||
Container mtusi-messenger-lab3-minio-1 Healthy
|
||||
Container mtusi-messenger-lab3-api-2 Starting
|
||||
Container mtusi-messenger-lab3-api-2 Started
|
||||
Container mtusi-messenger-lab3-api-1 Starting
|
||||
Container mtusi-messenger-lab3-api-1 Started
|
||||
|
||||
--- вставляем stale pending в БД ---
|
||||
INSERT 0 1
|
||||
|
||||
--- ждём 25 секунд, смотрим лог reconciler ---
|
||||
api-1 | 2026/10/23 18:16:51 reconciler: started, interval=10s grace=30s
|
||||
api-1 | 2026/10/23 18:16:51 Запускаем на 0.0.0.0:8080 (SESSION_STORE=redis, RECONCILER_ENABLED=true)
|
||||
api-2 | 2026/10/23 18:16:50 reconciler: started, interval=10s grace=30s
|
||||
api-2 | 2026/10/23 18:16:50 Запускаем на 0.0.0.0:8080 (SESSION_STORE=redis, RECONCILER_ENABLED=true)
|
||||
api-2 | 2026/10/23 18:17:01 reconciler: sweep done pending=1 orphans=0
|
||||
|
||||
--- проверяем, что stale удалён ---
|
||||
pending_left
|
||||
--------------
|
||||
0
|
||||
(1 row)
|
||||
|
||||
|
||||
--- возвращаем дефолты ---
|
||||
Container mtusi-messenger-lab3-postgres-1 Running
|
||||
Container mtusi-messenger-lab3-redis-1 Running
|
||||
Container mtusi-messenger-lab3-minio-1 Running
|
||||
Container mtusi-messenger-lab3-api-1 Recreate
|
||||
Container mtusi-messenger-lab3-api-2 Recreate
|
||||
Container mtusi-messenger-lab3-api-2 Recreated
|
||||
Container mtusi-messenger-lab3-api-1 Recreated
|
||||
Container mtusi-messenger-lab3-minio-1 Waiting
|
||||
Container mtusi-messenger-lab3-postgres-1 Waiting
|
||||
Container mtusi-messenger-lab3-postgres-1 Healthy
|
||||
Container mtusi-messenger-lab3-migrate-1 Starting
|
||||
Container mtusi-messenger-lab3-minio-1 Healthy
|
||||
Container mtusi-messenger-lab3-mc-init-1 Starting
|
||||
Container mtusi-messenger-lab3-migrate-1 Started
|
||||
Container mtusi-messenger-lab3-mc-init-1 Started
|
||||
Container mtusi-messenger-lab3-postgres-1 Waiting
|
||||
Container mtusi-messenger-lab3-redis-1 Waiting
|
||||
Container mtusi-messenger-lab3-migrate-1 Waiting
|
||||
Container mtusi-messenger-lab3-minio-1 Waiting
|
||||
Container mtusi-messenger-lab3-mc-init-1 Waiting
|
||||
Container mtusi-messenger-lab3-postgres-1 Healthy
|
||||
Container mtusi-messenger-lab3-migrate-1 Exited
|
||||
Container mtusi-messenger-lab3-redis-1 Healthy
|
||||
Container mtusi-messenger-lab3-minio-1 Healthy
|
||||
Container mtusi-messenger-lab3-mc-init-1 Exited
|
||||
Container mtusi-messenger-lab3-api-2 Starting
|
||||
Container mtusi-messenger-lab3-api-2 Started
|
||||
Container mtusi-messenger-lab3-api-1 Starting
|
||||
Container mtusi-messenger-lab3-api-1 Started
|
||||
@@ -0,0 +1,23 @@
|
||||
=== 2026-10-23T17:16:17Z ===
|
||||
--- ДО: /health/ready ---
|
||||
{"status":"ok"}
|
||||
--- останавливаем MinIO ---
|
||||
Container mtusi-messenger-lab3-minio-1 Stopping
|
||||
Container mtusi-messenger-lab3-minio-1 Stopped
|
||||
--- ждём 15 секунд, пока healthcheck реплик обновится ---
|
||||
--- /health/ready через LB (ожидается 503 unavailable) ---
|
||||
{"error":{"code":"unavailable","message":"сервис временно недоступен","request_id":"6a9bfae9-d751-46e7-868e-d5f55f0daaee"}}
|
||||
--- прямой /health/ready у каждой api-реплики (обход LB) ---
|
||||
wget: server returned error: HTTP/1.1 503 Service Unavailable
|
||||
|
||||
wget: server returned error: HTTP/1.1 503 Service Unavailable
|
||||
|
||||
--- запускаем MinIO обратно ---
|
||||
Container mtusi-messenger-lab3-minio-1 Starting
|
||||
Container mtusi-messenger-lab3-minio-1 Started
|
||||
--- ждём 20 секунд до восстановления ---
|
||||
--- ПОСЛЕ: /health/ready ---
|
||||
{"status":"ok"}
|
||||
--- X-Instance-Id после восстановления (должны быть оба) ---
|
||||
X-Instance-Id: 54992a38015c
|
||||
X-Instance-Id: c0f8bc0ada76
|
||||
@@ -0,0 +1,18 @@
|
||||
=== 2026-10-23T17:59:59Z ===
|
||||
--- MinIO порт 9000 не опубликован наружу ---
|
||||
invalid IP:0
|
||||
no port 9001/tcp for container mtusi-messenger-lab3-minio-1: 0/tcp
|
||||
консоль не проброшена (ожидаемо)
|
||||
|
||||
--- MINIO_BROWSER ---
|
||||
off
|
||||
|
||||
--- путь к объекту (первый в БД) ---
|
||||
object_key=chats/975e679f-51da-4a2b-a67a-adb85e7c6e1c/b5ebe14a-faf7-4682-a8dd-487bf6b5eb25
|
||||
|
||||
--- GET без авторизации из api-контейнера (ожидается 403) ---
|
||||
HTTP/1.1 403 Forbidden
|
||||
wget: server returned error: HTTP/1.1 403 Forbidden
|
||||
|
||||
--- API-пользователь (не root) ---
|
||||
enabled messenger-api messenger-api
|
||||
@@ -0,0 +1,3 @@
|
||||
=== 2026-10-23T17:13:20Z ===
|
||||
python3 tests/smoke.py --base-url "http://localhost:8080"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
Reference in New Issue
Block a user