Track 2: query plan and recovery
Hypothesis: at 300k messages the history index is worth keeping even under a tight memory budget, while the membership index is the cheaper one to remove. Recovery time should grow linearly with message count, so the dump-loss window can be estimated from a few points.
Dataset
One PostgreSQL 16.10 database, messenger_research:
- 5,000 chats, 2,000 users, 72,300 memberships.
- 100k / 200k / 300k messages, timestamps spread over 90 days.
- Chat sizes are deliberately skewed (
random()^3): the largest chat holds about 6% of all messages, most chats stay small. - Probe user 1 belongs to the ten largest chats and every 100th chat, so the chat-list and cross-chat feed queries always return rows.
- The app database
messengeris not touched.
Schema is applied from app/internal/database/migrations/0001_init.up.sql.
The two compared indexes are:
chat_members_user_idx (user_id, chat_id)messages_chat_created_id_idx (chat_id, created_at, id)
Index experiment
bench_indexes.sh rebuilds the dataset at 300k, warms each query, then reports
the median of five EXPLAIN (ANALYZE, BUFFERS) runs. The comparison states have
exactly one of the two indexes; the script asserts that state and restores both
indexes at the end.
| state | user_chats ms | chat_messages ms | user_feed ms |
|---|---|---|---|
| both indexes | 1.38 | 0.43 | 35.40 |
only messages_chat_created_id_idx |
7.36 | 0.47 | 35.22 |
only chat_members_user_idx |
1.40 | 57.63 | 80.40 |
Query shapes:
user_chats:chats JOIN chat_members WHERE user_id = $1 ORDER BY created_at, id LIMIT 50chat_messages: keyset page of one chat on(created_at, id)user_feed:messages JOIN chat_members WHERE user_id = $1plus a keyset cursor, ordered globally by(created_at, id)
Index sizes after REINDEX (300k messages, both present):
chat_members_user_idx: about 3.5 MBmessages_chat_created_id_idx: about 17 MBmessagestable including indexes: about 75 MB
Write cost (pgbench, 4 clients, 10 s, message-like inserts on a clone table): 2,719 tps with the messages index, 2,640 tps without it. The difference is small and within run noise; the read benefit below is much larger.
Recovery experiment
bench_recovery.sh recreates the database at each size from the migration,
seeds it, checks that both indexes exist, then runs pg_dump -Fc and
pg_restore into a fresh database. It verifies message and membership counts,
an MD5 fingerprint of every message row, and both indexes in the restored copy.
| messages | dump s | restore s | dump MiB | DB MiB | lost msgs at 878 RPS |
|---|---|---|---|---|---|
| 100,000 | 1.02 | 1.44 | 4.9 | 45.2 | 896 |
| 200,000 | 1.92 | 2.36 | 9.3 | 70.8 | 1,686 |
| 300,000 | 2.66 | 3.28 | 13.8 | 95.0 | 2,335 |
Linear fits (N = messages in millions):
- dump:
t = 0.23 + 8.20 * Ns, R² = 1.00 - restore:
t = 0.52 + 9.20 * Ns, R² = 1.00 - dump + restore:
t = 0.75 + 17.4 * Ns, R² = 1.00
The loss column uses the closed-loop 878 RPS already measured in
REPORT.md. pg_dump takes a consistent snapshot and does not block writers;
the estimate means "writes committed after that snapshot are lost if the dump
is the recovery point". Total outage including an offline restore is
dump + restore times the write rate (about 2.2k, 3.7k and 5.2k messages at
the three sizes).
Decision
Keep messages_chat_created_id_idx. Without it a chat page goes from 0.43 ms
to 57.6 ms and the cross-chat feed from 35.4 ms to 80.4 ms. If exactly one of
the two indexes must be removed, remove chat_members_user_idx: it costs about
3.5 MB and saves about 6 ms on the chat list, while the history index costs
about 17 MB and prevents a two-order-of-magnitude regression on history reads.
If the budget allows 21 MB for both, keep both.
Reproduce
Requires Docker Compose and the repository .env (cp .env.example .env,
then change POSTGRES_PASSWORD). Only the postgres service is needed.
docker compose --env-file .env up -d --wait postgres
./research/bench_indexes.sh
./research/bench_recovery.sh
python3 research/analyze.py | tee research/results/recovery.md
Outputs: research/results/indexes.md, research/results/recovery.csv,
research/results/recovery.md.
Limits
- Synthetic skewed data, not a production trace.
- Single PostgreSQL node, warm cache, no concurrent writes during the query benchmark.
- One dump/restore run per size; timings are indicative, not a capacity plan.
- The 878 RPS figure comes from the closed-loop client in
REPORT.md; it is not a maximum throughput claim.