FlashDrop
First-come coupons with zero over-issue: four concurrency strategies implemented and measured with k6
What this project proves
- Proving the problem with numbers: the no-lock version issued 300 coupons instead of 50 and remaining_quantity stopped at 46, not 0
- Three correct solutions: the other strategies let exactly 50 of 300 requests through, with exactly 50 rows in the database
- Explaining the result: I found why the optimistic lock beat FOR UPDATE under this load; FOR UPDATE queues even the 250 requests that will be rejected
- Measurement discipline: warm-up, median of three, runs partly refused by the TCP backlog marked invalid automatically; when re-measuring overturned the conclusions I fixed the README
- Operations: SHA-tagged deploys and rollbacks, daily backups with restore tests, and a deploy guard added after conflict markers reached production
- Role
- Solo: design, backend, tests, load testing, frontend, deployment and operations (21 commits)
- Period
- 2026.09 – 2026.10
- Stack
- Java
- Spring Boot
- Spring Data JPA
- PostgreSQL
- Redis
- Lua
- Flyway
- Bucket4j
- JUnit 5
- Testcontainers
- k6
- Docker
- Caddy
- Cloudflare
Key numbers
| Item | Value | Type | Evidence |
|---|---|---|---|
| Redis Lua throughput | 873 req/s | Measured | View evidence |
| Redis Lua p95 | 178 ms | Measured | View evidence |
| SELECT FOR UPDATE throughput | 188 req/s | Measured | View evidence |
| SELECT FOR UPDATE p95 | 1,465 ms | Measured | View evidence |
| Optimistic lock throughput | 331 req/s | Measured | View evidence |
| Optimistic lock p95 | 831 ms | Measured | View evidence |
| No lock: coupons issued out of 50 | 300 | Measured | View evidence |
| Rollback command time | 1.2 s | Measured | View evidence |
| Database restore from backup (11/11 tables) | 0.22 s | Measured | View evidence |
The problem
Three hundred concurrent requests hit 50 coupons, and not a single extra coupon may be issued. A plain read, check, write flow issues 300 coupons instead of 50 because of lost updates. I implemented the same business rule four ways to measure the balance between correctness, throughput and latency under real load.
Architecture
Caddy serves the static frontend and forwards /api/* to Spring Boot. Requests first pass a Bucket4j rate limiter whose state lives in Redis. Each strategy has its own endpoint, entity and table. The Redis strategy checks the remaining count and the set of users who already claimed inside one Lua script, and writes to PostgreSQL in the background with @Async.
Data model
Key constraints and indexes
- campaign_*.code VARCHAR(50) NOT NULL UNIQUE
- claim_*: UNIQUE (campaign_id, user_id): 중복 발급을 DB에서 차단하고 existsByCampaignIdAndUserId 조회를 빠르게 함
- campaign_optimistic.version BIGINT NOT NULL DEFAULT 0: @Version 낙관적 락
- V9 CHECK (remaining_quantity >= 0): for_update, optimistic, redis에만 적용 (no_lock은 oversell이 보이도록 일부러 제외)
- claim_redis.id UUID: 앱에서 생성해 DB 기록을 기다리지 않고 응답
- Redis 키 (claim.lua): stock(DECR), claimed set(SADD). 결과 1 발급, 0 품절, -1 캠페인 없음, -2 이미 받음
Key design decisions
Redis: DECR + SADD in Lua, database write via @Async
- Context
- FOR UPDATE queued every request behind one hot row (p95 1,465 ms).
- Decision
- SISMEMBER → GET → DECR + SADD run in one claim.lua, so the hot path has neither Postgres nor a check-then-act gap.
- Trade-off
- If the process dies between Redis and the database, the coupon can be missing from the database.
Optimistic retry split into two beans
- Context
- The @Transactional proxy cannot intercept self-invocation, so with both in one class no transaction is actually applied.
- Decision
- OptimisticClaimService (retries, max 20) → OptimisticClaimAttempt (@Transactional, @Version)
- Trade-off
- One piece of logic is split across two classes.
Separate tables and endpoints per strategy
- Context
- k6 results must not mix with one another.
- Decision
- Each of the four strategies has its own campaign / claim tables, entities and endpoint.
- Trade-off
- Code and migrations are duplicated (8 tables).
The no-lock version kept on purpose
- Context
- To show the problem in numbers.
- Decision
- Kept an endpoint where the lost update happens as is, without a CHECK constraint on its table.
- Trade-off
- A real product would never ship such an endpoint.
Idempotency and the stock floor in the database: UNIQUE(campaign_id, user_id), CHECK (remaining_quantity >= 0)
- Context
- Even buggy application logic must not issue twice or push stock below zero.
- Decision
- UNIQUE on every claim table and the V9 CHECK on the three correct strategies; production rejected an update setting remaining_quantity = -1.
- Trade-off
- The Redis strategy checks in two places, Redis and the database.
Rate-limit key is the userId
- Context
- It is the only identifier in the project.
- Decision
- A Bucket4j + Redis (Lettuce) token bucket keyed by userId, so the limit is shared across instances.
- Trade-off
- The client can change it, so production needs it combined with the IP.
Before and after
- What changed
- The first table came from a single cold run before the JIT and the connection pool had warmed up. Adding warm-up, three runs, a median and a validity rule (success + sold-out < 300 means invalid) to scripts/bench.ps1 and re-measuring overturned two conclusions: no-lock was not 'slowest' but 'fast and wrong' (80.4 → 307 rps), and the optimistic lock was not 'slowest' but 1.76x faster than FOR UPDATE (114.1 → 331 rps). The reason is that findByCodeForUpdate takes the lock before checking the remaining count. The fix (a plain SELECT to check sell-out before locking) is not applied yet.
Incident write-up
- Symptom
- After a deploy on 2026-10-09 the flashdrop.javohir.dev UI broke and the page showed a raw <<<<<<< HEAD marker.
- Root cause
- The local repository was behind GitHub, so the push was rejected; the git pull merge conflicted in four files and the markers were committed unresolved. Caddy served the frontend straight from the git working directory, so the pull went live immediately, and a backend rollback could not undo it.
- Fix
- Restored frontend/index.html from the last working commit, rewrote the READMEs, removed accidentally committed IDE build output (bin/), added it to .gitignore and redeployed (7ac0754).
- Prevention
- deploy.sh now uses git grep to scan upstream for conflict markers and stops the deploy if it finds any, and merges only with --ff-only (a2dd204). A hidden side effect turned up too: while fixing the conflict, the READMEs (ko/en/uz) had reverted to the old cold-run table; I caught it when comparing the portfolio text with the repo and restored the re-measured results (a29c567).
Tests and quality
- Tests
- 4
- Coverage
- —
- CI
- —
- Test tools
- JUnit 5, Testcontainers (PostgreSQL 17, Redis 7), virtual threads, k6, bench.ps1
Deployment and operations
- Deployment
- iwinv VPS on Docker Compose: app (Spring Boot, non-root, 1 GB memory limit), web (Caddy: static console + /api/* proxy), PostgreSQL 17, Redis 7 and seed (demo campaigns). Database and Redis ports are not exposed. deploy.sh: conflict-marker check → git merge --ff-only → build the image with a SHA tag too → update app and web, recording the SHA in .deploy-history. HikariCP 50, Tomcat accept-count 500.
- Backups
- Daily pg_dump by cron at 03:00 KST, kept for 7 days, covering all 9 Postgres databases on the server. The 2026-10-09 03:00 run was confirmed. Restore tested: FlashDrop 0.22 s (11/11 tables), LedgerCore 0.66 s (14/14 tables)
- Rollback
- rollback.sh points latest at the previous SHA image and recreates the app without building. In production the command took real 0m1.201s; seen from outside, it returned 502 from 03:32:00 to 03:32:07 and then 200 again.
- Monitoring
- pg_stat_statements enabled; the database is connected to PG Insight over the monitoring network
Screenshots
Limits and retrospective
Current limits
- A gap between Redis and Postgres: a coupon issued in Redis is missing from Postgres if the app dies before the @Async write. @Async errors are swallowed (no handler), a full executor queue (500) throws TaskRejectedException after the DECR, campaign_redis.remaining_quantity is not synced after init, and there is no reconciliation job
- The FOR UPDATE path is not optimal: it locks before checking stock, so requests that will be rejected also queue; a plain SELECT before the lock is not added yet
- The rate-limit key is weak: the client can change the userId, so production needs it combined with the IP
- Few tests: only four concurrency tests, with no unit or HTTP tests. NoLockConcurrencyTest is red on purpose, so gradlew test never goes fully green
- A single instance: the rollback command takes 1.2 s, but there are about 8 s of 502s while the JVM starts. There is no health-gated switchover, deploys do not wait for health, and backups share a disk with the database
- Measurement environment: numbers come from one laptop (Windows + Docker Desktop, i7-8565U), and the OS listen backlog refused requests in some runs
- Console counters are per browser session and reset on refresh; there is no shared metrics endpoint
If I built it again
- Background writes through Redis Streams or an outbox from day one: ACK only after the database commit, re-read lost messages, plus a reconciliation job
- CI from day one (tests, conflict check, a small k6 smoke), the frontend inside the image, and blue-green deploys gated on health checks
Key code
src/main/java/com/flashdrop/strategy/nolock/NoLockClaimService.javaA deliberate lost update: read, check, write without locks
src/main/java/com/flashdrop/strategy/forupdate/ForUpdateCampaignRepository.java@Lock(PESSIMISTIC_WRITE) findByCodeForUpdate
src/main/java/com/flashdrop/strategy/optimistic/OptimisticClaimAttempt.javaThe transaction moved into its own bean to avoid self-invocation
src/main/resources/scripts/claim.luaDuplicate check, remaining count and decrement in one atomic operation
src/main/java/com/flashdrop/strategy/redis/RedisClaimService.javaMaps Lua result codes to business statuses and hands the database write to the background
src/main/java/com/flashdrop/strategy/redis/RedisClaimWriteBackService.javaWrites to Postgres later with @Async
src/main/java/com/flashdrop/ratelimit/RateLimitInterceptor.javaBucket4j + Redis token bucket with 429 and Retry-After
scripts/bench.ps1Warm-up, three runs, median, rerunning invalid runs
scripts/deploy.sh, scripts/rollback.shConflict guard, SHA-tagged images, rollback
Current status and next steps
The no-lock version issued 300 coupons instead of 50 (remaining_quantity stayed at 46, not 0), while the other three let exactly 50 of 300 requests through, leaving exactly 50 rows in the database. Redis beat FOR UPDATE by 4.6x on throughput and 8.2x on p95. Figures are the median of three valid runs after one warm-up; runs where success + sold-out added up to fewer than 300 (some requests refused by the TCP backlog) were marked invalid automatically and rerun. The rate limit let 6 of 12 requests from one userId through and answered 6 with 429 (5 tokens + greedy refill).