07 / WORK / CASE STUDY
Large-scale Django Microservices Commerce Platform
Thirty-three Django microservices behind six business domains — e-commerce, multi-tenant store POS, B2B purchasing, sales CRM and commissions, courses and designer co-creation. In production.
- TYPE
- BACKEND
- SCOPE
- Internal · in production
- STACK
- DJANGO · DRF · CELERY · MYSQL · REDIS · NGINX
- LINKS
- Internal project (private)
01 / PROBLEM
One retailer runs an online mall, chain stores, B2B wholesale and a field sales team at once — four channels creating orders against the same inventory, the same members, the same payment rails. Separate systems per channel end in oversold stock, split identities and reconciliation chaos. This platform carries it on domain-split microservices: one Django service and one database per business domain, integrated across domains by retrying REST calls and Celery tasks; payments, e-invoicing and logistics live in dedicated services so nothing else touches third-party credentials.
02 / CONSTRAINTS
- Company-internal; the codebase is private. Every scale figure here re-runs as a command inside the repo.
- Bare-metal with no cloud budget: no Kubernetes — health checks, three-stage dependency-ordered startup and Celery worker management all fall to a hand-built orchestration script.
- Tenant isolation is non-negotiable: store POS runs one database per tenant, and the tenant header is only ever a claim — authority comes from the signed JWT.
- The system is live: payment- and order-path fixes must land without interrupting transactions.
03 / ARCHITECTURE
A many-service architecture — grown domain by domain, not carved from a monolith: Django 4.2 LTS + DRF across 33 services, 807 data models, 729 ViewSet registrations plus 2,203 custom actions, ~1.53M lines of Python excluding migrations. The case for services is release cadence and risk: a payment service tolerates change very differently from a content service, and independent deploy boundaries let high-risk domains ship in small, separate steps.
Auth is asymmetric RS256 JWT: one issuing service holds the private key, everything else verifies with the public key and never calls back to the user store; permission codes are baked into the token at issue time. Each service owns a MySQL database; three Redis instances split cache, sessions-plus-locks, and Celery queues. Cross-service references store bare primary keys — no database foreign keys — with Snowflake IDs serialized as strings for JS precision. One Celery worker per service, five with Beat schedules; payments, e-invoicing, convenience-store and home logistics, FCM push and social login concentrate in dedicated services. Beyond CI, a weekly four-layer regression runs: health gate → a 218-endpoint read-only smoke baseline → pytest → cross-service E2E.
04 / RESPONSIBILITIES
Lead development: on the project since its start in January 2025, sole main contributor since September 2025 — 731 of 825 commits (89%, both git identities mine). Scope spans service architecture and new domains (multi-tenant POS, B2B, sales CRM and commission attribution, courses, designer co-creation), platform-wide security/performance/correctness audits, the regression infrastructure and deployment operations.
05 / CHALLENGES → SOLUTIONS
CHALLENGE
An audit found the payment-callback signature check implemented off-spec — every legitimate callback was rejected, so payments succeeded while orders never updated; and success only touched the e-commerce order, with no sync path at all for B2B and POS.
SOLUTION
Rewrote the signature algorithm to spec and built a single post-payment sync entry point fanning out by order origin — e-commerce, B2B, POS, subscription — with idempotent branches so duplicate callbacks self-heal instead of double-booking. Verified the loop against real sandbox callbacks before shipping.
CHALLENGE
Async tasks in several services registered cleanly, logged nothing — and never ran: tasks were routed to the queues declared in TASK_ROUTES while workers listened on differently named ones, messages piling up silently with no consumer. The same mismatch surfaced independently in four services — alerts never sent, the reward chain severed, image variants stalled.
SOLUTION
Diffed TASK_ROUTES against each startup script’s -Q flags, unified on a per-service queue-name convention and fixed the orchestration script — then proved it live: alert mail actually arriving, the reward chain passing an idempotent E2E.
CHALLENGE
Past 30 services, hot-path decay had no owner: login measured 4.08 seconds (a synchronous remote permission lookup on every login), and the inventory stats endpoint hard-hung for 20 seconds on a re-entrant threading.Lock deadlock.
SOLUTION
Built a curl baseline first, then fixed 28 services in waves with re-measurement after each: login went 4.08s → 0.55–0.75s (~5–7×), the stats endpoint from a 20-second hang to 0.014s (Lock → RLock), zero new 5xx throughout — and settled CONN_MAX_AGE=0 as the platform-wide rule under gevent workers.
06 / SYSTEM FACTS
33
microservices, one DB each
807
data models
16,433
backend test functions
89%
of commits mine (731/825)
07 / LESSONS
A queue name is a contract, and contracts kept in sync by hand always break: the same routes-vs-worker mismatch recurred independently in four services because the name lived in two files. The real fix is not fixing it four times — it is generating worker queue flags from each service’s own config. One source of truth.
Performance needs an owned baseline, because decay is silent: a 4-second login and a 20-second hang did not happen overnight — they accrued through unmeasured routine until a dedicated audit made them visible. Next time the hot-path baseline starts at ten services, not thirty.
NEXT
Multi-tenant Retail POS System