javohir.dev
KOENUZ
All projectsLive

LiveCommerce

A flash-sale platform for live broadcasts: 1,000 purchase attempts for 100 items, zero oversells

Open live service: live.javohir.devGitHub code

What this project proves

  • Correctness: 1,000 attempts for 100 items, repeated three times, with 0 oversells, 0 payment/sales mismatches and 0 HTTP errors; five invariants checked after every run
  • Fault tolerance: stopping Kafka or the payment database for real lost 0 events and charged nobody twice (chaos tests)
  • Measurement-driven speed-ups: sell-out 1 min 27 s → about 23 s, order creation p95 5.72 s → 2.68 s; the connection-pool hypothesis was rejected by measurement
  • Closing a production business risk with evidence: one account could lock up all stock with unpaid orders; a partial unique index fixed it (from 15 → 0 to 15 → 5)
  • Edge protection: rate limits on login, sign-up and orders, with the client IP taken from the trusted proxy chain so a forged X-Forwarded-For cannot dodge the 429 (verified in production)
  • Operations: SHA-tagged deploys, health-gated rollback, a two-way drill in production, and a JWT key that keeps tokens valid across restarts
Role
Solo: architecture, seven services, frontend, tests, load testing, deployment and operations (127 commits)
Period
2026.09 – 2026.10
Stack
  • Kotlin
  • Spring Boot
  • Spring Cloud Gateway
  • Spring Data JPA
  • PostgreSQL
  • Apache Kafka
  • WebSocket (STOMP)
  • Flyway
  • Toss Payments
  • OpenTelemetry
  • Jaeger
  • Next.js
  • TypeScript
  • Tailwind CSS
  • JUnit 5
  • k6
  • Docker
  • Caddy

Key numbers

ItemValueTypeEvidence
Oversells (100 items, 1,000 attempts, all 3 runs)0MeasuredView evidence
Sold = charged100 / 100MeasuredView evidence
Order creation p952.68 sMeasuredView evidence
Payment submission p950.63 sMeasuredView evidence
Sell-out time23 sMeasuredView evidence
Stock update delivery (500 viewers)99.2 %MeasuredView evidence
Viewer-count messages reduced98 %MeasuredView evidence
Chaos tests: lost events / double charges0MeasuredView evidence

The problem

A limited sale during a live broadcast packs orders into a few seconds, and stock, payment and delivery live in different services. In a flash sale, selling out is normal, so charging first turns every sell-out into a refund and a PG fee, and one stopped service can lose events or double-charge.

Architecture

api-gateway (Spring Cloud Gateway) verifies JWTs, rewrites identity headers, and handles rate limiting and the WebSocket proxy. The Saga orchestrator in order-service holds stock for ten minutes before taking payment (AWAITING_STOCK → AWAITING_PAYMENT → PAYMENT_CONFIRMING → CONFIRMING_STOCK → COMPLETED, with a five-minute payment window). inventory (hold, confirm, release, expiry sweep), payment (Toss confirm and cancel with an Idempotency-Key), live (STOMP chat, live stock, viewer count), auth (RS256 JWT, JWKS, refresh token rotation) and delivery (shipping, HMAC courier webhooks) each own a PostgreSQL database and read one another's data only through APIs or Kafka events. Events leave through each service's transactional outbox together with a traceparent, so the whole Saga stays one Jaeger trace across Kafka.

Data model

Key constraints and indexes

  • database per service: PostgreSQL 5개, 서비스 간 FK 없음, order_id / member_id UUID와 Kafka 이벤트로만 연결 (Flyway 마이그레이션 21개)
  • orders.uq_orders_one_open_per_member_product: 열린 상태 partial UNIQUE (V5)
  • orders.version (@Version), 인덱스 (status, updated_at), partial payment_deadline WHERE status='AWAITING_PAYMENT'
  • product_stock.quantity_available CHECK (>= 0): 코드가 틀려도 DB가 초과 판매를 막음, unit_price는 서버 측
  • inventory_reservations.order_id UNIQUE: 중복 이벤트가 재고를 두 번 줄이지 않음, partial index expires_at WHERE status='HELD'
  • payments.order_id UNIQUE: 주문 하나에 결제 하나
  • tracking_events.event_id UNIQUE: 택배 웹훅 재전송도 한 번만 적용
  • refresh_tokens.token_hash UNIQUE, family_id 인덱스: 재사용된 토큰은 가족 전체를 폐기
  • outbox_events.trace_parent (W3C), partial index created_at WHERE status='PENDING'

Key design decisions

Hold stock first, then charge (hold-then-pay)

Context
Selling out is normal in a flash sale, so charging first turns every sell-out into a refund and a PG fee.
Decision
Hold stock for ten minutes, open a five-minute payment window, and confirm the hold once payment is confirmed.
Trade-off
Stock stays locked for ten minutes, which made a one-open-order-per-buyer rule necessary.

Orchestration, not choreography

Context
The state machine, recovery and admin retries need one home.
Decision
order-service coordinates the Saga, and a recovery scheduler resumes stuck steps every five seconds.
Trade-off
order-service becomes the central point.

Transactional outbox, PUBLISHED only after the broker acks

Context
Events must survive Kafka being down.
Decision
State changes and events are written in one transaction; the relay sends in parallel and marks only the leading run of acknowledged events as PUBLISHED.
Trade-off
Duplicates can happen, so every consumer is idempotent.

Toss called outside the transaction, result written with the outbox

Context
A slow external API must not hold a database connection.
Decision
With TransactionTemplate, Toss confirmation is called outside, and the result plus outbox event are written in one transaction.
Trade-off
If the result is not recorded, the recovery scheduler retries with the same Idempotency-Key.

Reject sold-out early with a lock-free scalar read

Context
Even after sell-out every request queued on FOR UPDATE, giving a p95 of 5.8 s.
Decision
Read one number before locking and reject at zero. The entity is not read first, because Hibernate's first-level cache would return a stale copy at the next FOR UPDATE and open the door to oversells.
Trade-off
A buyer may see sold-out while stock is being returned, but oversells remain impossible.

One open order: a partial unique index

Context
One account could lock up all stock without paying, and the guarantee had to hold under concurrent requests too.
Decision
A partial unique index on (member_id, product_id) for open statuses (V5); a violation returns 409 OPEN_ORDER_EXISTS with the existing orderId.
Trade-off
Buyers must pay or cancel the old order before a new one, and the UI takes them back to it.

Rate limits in gateway memory, orders keyed by user

Context
Keeping it simple without Redis, and in Korea many buyers share one CGNAT IP.
Decision
Login 10/min and sign-up 5/hour by IP, orders 20/min by user; the IP is the right-most public address in the XFF chain.
Trade-off
It is only accurate with one gateway, and per-IP login limits do not stop attacks on one account from many IPs.

JWT key in .env, kid as an RFC 7638 thumbprint

Context
Tokens must stay valid across restarts and deploys and match the gateway's JWKS cache.
Decision
JWT_SIGNING_KEY is loaded from the server's .env (chmod 600) and the same key always yields the same kid; without the key, compose refuses to start.
Trade-off
Rotating the key briefly rejects older tokens.

Before and after

Beforesell-out 1 min 27 s · order p95 5.72 sView evidence
Aftersell-out ~23 s · order p95 2.68 sView evidence
What changed
Four causes found and fixed: requests waiting on FOR UPDATE even after sell-out → a scalar read before the lock; one partition and one consumer per topic (stock-hold p95 17.8 s) → six partitions and six threads (keyed by orderId); an outbox relay sending one event at a time every 500 ms → parallel batches every 100 ms; the viewer count sent to all 500 viewers on every join (O(n²)) → at most once per second and only on change. Payment submission p95 0.71 → 0.63 s, stock hold p95 17.77 → 4.80 s, completion p95 33.55 → 11.04 s, viewer-count messages 454,583 → 8,151. The 'before' latencies are truncated because k6 stopped waiting for slow orders after 15–20 s, so the real gap is larger.

Incident write-up

Symptom
In production on 2026-10-09, lc-hoard-check.sh created unpaid orders back to back from one test account: 2×201 and stock 15 → 0. A single user could shut everyone else out of the broadcast's whole stock for the ten-minute hold.
Root cause
The Saga opened a separate hold for every POST /api/orders, and no rule anywhere said one buyer, one open order.
Fix
Migration V5: a partial unique index on (member_id, product_id) for the open statuses (AWAITING_STOCK, AWAITING_PAYMENT, PAYMENT_CONFIRMING, CONFIRMING_STOCK). The controller checks first; under a race, the DataIntegrityViolationException is identified by the index name and answered with 409 OPEN_ORDER_EXISTS plus the existing orderId, and the frontend resumes that order.
Prevention
On a local Postgres, the second of two concurrent transactions waited 1.5 s and was rejected; a production re-check gave 1×201 + 409 with stock stopping at 5. An integration test asserts exactly one 201 out of ten concurrent requests, and k6 flash-sale.js now gives each VU its own user (USERS ≥ VUS is checked).

Tests and quality

Tests
46
Coverage
—
CI
—
Test tools
JUnit 5 (통합 24, 카오스 3, gateway 단위 14, SigningKeys 5), 실행 중인 전체 스택에 실제 HTTP, k6 3종

Deployment and operations

Deployment
docker-compose.prod.yml on an iwinv VPS: 16 containers (7 Spring services, Next.js, 5 PostgreSQL, Kafka KRaft, Jaeger, Caddy edge). No container publishes a port; only edge joins the external proxy network, sending /api/* and /ws/* to the gateway and everything else to the frontend. JVMs run as USER app (non-root) with per-service memory limits (Spring 512 MB, PostgreSQL 256 MB, Kafka 768 MB). PAYMENTS_FAKE_ENABLED=false. deploy.sh: conflict-marker check → git pull --ff-only → build 8 SHA-tagged images → up -d → check every service's /actuator/health and the frontend from inside edge, 60 × 3 s → on success record APP_TAG and .deploy-history. Measured on 2026-10-10: deploy 2 min 38 s (116 s build), about 27 s of downtime.
Backups
The server-wide cron backup (daily at 03:00, kept 7 days) shows all five livecommerce PostgreSQL databases dumped in its log; a restore test has not been run yet.
Rollback
rollback.sh <sha> checks that all 8 images exist, then runs up -d --no-deps --force-recreate without building. Flyway migrations only go forward (V5 only adds an index, so the old code still runs). In the production drill a rollback took 48 s (about 36 s down) and rolling forward again 39 s (about 26 s).
Monitoring
lc-diagnose.sh checks in one command container state, restarts/OOM, memory, 24-hour ERROR count, public endpoint latency, orders by status, orders stuck over 15 minutes, outbox PENDING, stock, database sizes and Kafka lag. On 2026-10-09: 9 days of uptime, 0 restarts, 0 OOM, 0 ERRORs, Kafka lag 0, PENDING 0. Full Saga traces in Jaeger (OpenTelemetry, 20% sampling).

Screenshots

LiveCommerce screens at a glance
Live room: viewer count, chat, remaining stock and the 10-minute timer (video simulated)
On the first order the address (postcode search) is captured and saved as the default
Payment complete: the Saga steps hold → payment → payment confirmation → order completed
My orders: an order in delivery with its courier webhook history, a pending one with 'continue payment' (409 → existing order), and one cancelled by a card decline with its held stock returned
Operations dashboard refreshed every 5 s; it caught a real fault: inventory event delivery failing with Topic inventory-events not present in metadata (H4, H5)
Saga inspector: order created +0 ms → stock held +180 ms → payment confirmed +55 s → stock confirmed +56 s, with retry and cancel
Jaeger: POST /api/orders across 4 services, 11 spans, 213.68 ms; the outbox traceparent keeps the async part after Kafka in the same trace

Limits and retrospective

Current limits

  • Load numbers come from one laptop (±25% variance) and are not production throughput; next: a separate load machine and a production-like environment
  • flash-sale.js has not been re-measured since V5, so the before/after table uses the old script; next: re-measure a median of three with the new script
  • live-service keeps viewers and sessions in memory, so it runs as one instance and viewers reconnect on restart; next: Redis pub/sub or a broker relay
  • The rate limiter lives in gateway memory, so more gateways multiply the limit and a restart resets it; login limits are per IP and do not stop credential stuffing from many IPs. Next: a Redis limiter and per-account limits
  • The JWT key lives in env, not KMS, with no rotation; next: rotation with two keys in the JWKS and a secret manager
  • Deploys cause about 26–36 s of downtime (single instance, no rolling or blue-green), during which the gateway returns 500 instead of 503, so clients cannot tell they should retry
  • outbox_events and refresh_tokens are never pruned and grow over time; next: a retention job for PUBLISHED and expired rows
  • The 8 GB VPS is shared with other projects and uses 813 MB of swap, so latency can rise under memory pressure
  • No CI (GitHub Actions billing), alerting, backup restore drill or frontend automated tests yet
  • Payments use Toss test keys and the broadcast video is simulated

Key code

Current status and next steps

Three runs of 1,000 purchase attempts for 100 items (200 VUs) each ended with 0 oversells, 0 mismatches between payments and units sold, and 0 HTTP errors, with five invariants checked through the admin API after every run. Measuring the bottlenecks cut the sell-out time from 1 min 27 s to about 23 s, and a connection-pool hypothesis was rejected by measurement. In production I closed a hole where one account could hold the entire stock (15 → 0) with two unpaid orders, using a partial unique index (15 → 5); fixed a JWT key problem where, after an auth restart, the same token got different answers from different services (401/200) (0eeaf2c); and added limits to login and ordering, which had none (085ded0). I also tracked down an outbox transaction holding locks for minutes when the broker was unreachable (881afda), consumers missing new partitions after a partition increase, a k6 invariant that showed 101 sold because a hold from a previous run expired, and 14 failing integration tests caused by another project's container occupying port 8080.

LiveCommerce | javohir.dev