All projectsLive

FlashDrop

First-come coupons with zero over-issue: four concurrency strategies implemented and measured with k6

What this project proves

  • Proving the problem with numbers: the no-lock version issued 300 coupons instead of 50 and remaining_quantity stopped at 46, not 0
  • Three correct solutions: the other strategies let exactly 50 of 300 requests through, with exactly 50 rows in the database
  • Explaining the result: I found why the optimistic lock beat FOR UPDATE under this load; FOR UPDATE queues even the 250 requests that will be rejected
  • Measurement discipline: warm-up, median of three, runs partly refused by the TCP backlog marked invalid automatically; when re-measuring overturned the conclusions I fixed the README
  • Operations: SHA-tagged deploys and rollbacks, daily backups with restore tests, and a deploy guard added after conflict markers reached production
Role
Solo: design, backend, tests, load testing, frontend, deployment and operations (21 commits)
Period
2026.09 – 2026.10
Stack
  • Java
  • Spring Boot
  • Spring Data JPA
  • PostgreSQL
  • Redis
  • Lua
  • Flyway
  • Bucket4j
  • JUnit 5
  • Testcontainers
  • k6
  • Docker
  • Caddy
  • Cloudflare

Key numbers

ItemValueTypeEvidence
Redis Lua throughput873 req/sMeasuredView evidence
Redis Lua p95178 msMeasuredView evidence
SELECT FOR UPDATE throughput188 req/sMeasuredView evidence
SELECT FOR UPDATE p951,465 msMeasuredView evidence
Optimistic lock throughput331 req/sMeasuredView evidence
Optimistic lock p95831 msMeasuredView evidence
No lock: coupons issued out of 50300MeasuredView evidence
Rollback command time1.2 sMeasuredView evidence
Database restore from backup (11/11 tables)0.22 sMeasuredView evidence

The problem

Three hundred concurrent requests hit 50 coupons, and not a single extra coupon may be issued. A plain read, check, write flow issues 300 coupons instead of 50 because of lost updates. I implemented the same business rule four ways to measure the balance between correctness, throughput and latency under real load.

Architecture

Caddy serves the static frontend and forwards /api/* to Spring Boot. Requests first pass a Bucket4j rate limiter whose state lives in Redis. Each strategy has its own endpoint, entity and table. The Redis strategy checks the remaining count and the set of users who already claimed inside one Lua script, and writes to PostgreSQL in the background with @Async.

Data model

Key constraints and indexes

  • campaign_*.code VARCHAR(50) NOT NULL UNIQUE
  • claim_*: UNIQUE (campaign_id, user_id): 중복 발급을 DB에서 차단하고 existsByCampaignIdAndUserId 조회를 빠르게 함
  • campaign_optimistic.version BIGINT NOT NULL DEFAULT 0: @Version 낙관적 락
  • V9 CHECK (remaining_quantity >= 0): for_update, optimistic, redis에만 적용 (no_lock은 oversell이 보이도록 일부러 제외)
  • claim_redis.id UUID: 앱에서 생성해 DB 기록을 기다리지 않고 응답
  • Redis 키 (claim.lua): stock(DECR), claimed set(SADD). 결과 1 발급, 0 품절, -1 캠페인 없음, -2 이미 받음

Key design decisions

Redis: DECR + SADD in Lua, database write via @Async

Context
FOR UPDATE queued every request behind one hot row (p95 1,465 ms).
Decision
SISMEMBER → GET → DECR + SADD run in one claim.lua, so the hot path has neither Postgres nor a check-then-act gap.
Trade-off
If the process dies between Redis and the database, the coupon can be missing from the database.

Optimistic retry split into two beans

Context
The @Transactional proxy cannot intercept self-invocation, so with both in one class no transaction is actually applied.
Decision
OptimisticClaimService (retries, max 20) → OptimisticClaimAttempt (@Transactional, @Version)
Trade-off
One piece of logic is split across two classes.

Separate tables and endpoints per strategy

Context
k6 results must not mix with one another.
Decision
Each of the four strategies has its own campaign / claim tables, entities and endpoint.
Trade-off
Code and migrations are duplicated (8 tables).

The no-lock version kept on purpose

Context
To show the problem in numbers.
Decision
Kept an endpoint where the lost update happens as is, without a CHECK constraint on its table.
Trade-off
A real product would never ship such an endpoint.

Idempotency and the stock floor in the database: UNIQUE(campaign_id, user_id), CHECK (remaining_quantity >= 0)

Context
Even buggy application logic must not issue twice or push stock below zero.
Decision
UNIQUE on every claim table and the V9 CHECK on the three correct strategies; production rejected an update setting remaining_quantity = -1.
Trade-off
The Redis strategy checks in two places, Redis and the database.

Rate-limit key is the userId

Context
It is the only identifier in the project.
Decision
A Bucket4j + Redis (Lettuce) token bucket keyed by userId, so the limit is shared across instances.
Trade-off
The client can change it, so production needs it combined with the IP.

Before and after

Before1 cold run: Redis 240.8 rps · p95 582 ms
Afterwarm-up + median of 3: Redis 873 rps · p95 178 msView evidence
What changed
The first table came from a single cold run before the JIT and the connection pool had warmed up. Adding warm-up, three runs, a median and a validity rule (success + sold-out < 300 means invalid) to scripts/bench.ps1 and re-measuring overturned two conclusions: no-lock was not 'slowest' but 'fast and wrong' (80.4 → 307 rps), and the optimistic lock was not 'slowest' but 1.76x faster than FOR UPDATE (114.1 → 331 rps). The reason is that findByCodeForUpdate takes the lock before checking the remaining count. The fix (a plain SELECT to check sell-out before locking) is not applied yet.

Incident write-up

Symptom
After a deploy on 2026-10-09 the flashdrop.javohir.dev UI broke and the page showed a raw <<<<<<< HEAD marker.
Root cause
The local repository was behind GitHub, so the push was rejected; the git pull merge conflicted in four files and the markers were committed unresolved. Caddy served the frontend straight from the git working directory, so the pull went live immediately, and a backend rollback could not undo it.
Fix
Restored frontend/index.html from the last working commit, rewrote the READMEs, removed accidentally committed IDE build output (bin/), added it to .gitignore and redeployed (7ac0754).
Prevention
deploy.sh now uses git grep to scan upstream for conflict markers and stops the deploy if it finds any, and merges only with --ff-only (a2dd204). A hidden side effect turned up too: while fixing the conflict, the READMEs (ko/en/uz) had reverted to the old cold-run table; I caught it when comparing the portfolio text with the repo and restored the re-measured results (a29c567).

Tests and quality

Tests
4
Coverage
—
CI
—
Test tools
JUnit 5, Testcontainers (PostgreSQL 17, Redis 7), virtual threads, k6, bench.ps1

Deployment and operations

Deployment
iwinv VPS on Docker Compose: app (Spring Boot, non-root, 1 GB memory limit), web (Caddy: static console + /api/* proxy), PostgreSQL 17, Redis 7 and seed (demo campaigns). Database and Redis ports are not exposed. deploy.sh: conflict-marker check → git merge --ff-only → build the image with a SHA tag too → update app and web, recording the SHA in .deploy-history. HikariCP 50, Tomcat accept-count 500.
Backups
Daily pg_dump by cron at 03:00 KST, kept for 7 days, covering all 9 Postgres databases on the server. The 2026-10-09 03:00 run was confirmed. Restore tested: FlashDrop 0.22 s (11/11 tables), LedgerCore 0.66 s (14/14 tables)
Rollback
rollback.sh points latest at the previous SHA image and recreates the app without building. In production the command took real 0m1.201s; seen from outside, it returned 502 from 03:32:00 to 03:32:07 and then 200 again.
Monitoring
pg_stat_statements enabled; the database is connected to PG Insight over the monitoring network

Screenshots

Limits and retrospective

Current limits

  • A gap between Redis and Postgres: a coupon issued in Redis is missing from Postgres if the app dies before the @Async write. @Async errors are swallowed (no handler), a full executor queue (500) throws TaskRejectedException after the DECR, campaign_redis.remaining_quantity is not synced after init, and there is no reconciliation job
  • The FOR UPDATE path is not optimal: it locks before checking stock, so requests that will be rejected also queue; a plain SELECT before the lock is not added yet
  • The rate-limit key is weak: the client can change the userId, so production needs it combined with the IP
  • Few tests: only four concurrency tests, with no unit or HTTP tests. NoLockConcurrencyTest is red on purpose, so gradlew test never goes fully green
  • A single instance: the rollback command takes 1.2 s, but there are about 8 s of 502s while the JVM starts. There is no health-gated switchover, deploys do not wait for health, and backups share a disk with the database
  • Measurement environment: numbers come from one laptop (Windows + Docker Desktop, i7-8565U), and the OS listen backlog refused requests in some runs
  • Console counters are per browser session and reset on refresh; there is no shared metrics endpoint

If I built it again

  • Background writes through Redis Streams or an outbox from day one: ACK only after the database commit, re-read lost messages, plus a reconciliation job
  • CI from day one (tests, conflict check, a small k6 smoke), the frontend inside the image, and blue-green deploys gated on health checks

Key code

Current status and next steps

The no-lock version issued 300 coupons instead of 50 (remaining_quantity stayed at 46, not 0), while the other three let exactly 50 of 300 requests through, leaving exactly 50 rows in the database. Redis beat FOR UPDATE by 4.6x on throughput and 8.2x on p95. Figures are the median of three valid runs after one warm-up; runs where success + sold-out added up to fewer than 300 (some requests refused by the TCP backlog) were marked invalid automatically and rerun. The rate limit let 6 of 12 requests from one userId through and answered 6 with 429 (5 tokens + greedy refill).

FlashDrop | javohir.dev