Capstone 4: Testing, CI/CD and Going Live

The first 45 articles finished BeeOrder: it takes orders, deducts stock, and receives payment callbacks. This article does exactly one thing — turn "it runs on my machine" into "anyone can run it somewhere else, and we can back out when it breaks." Delivery, in substance, is writing three kinds of hidden precondition down as explicit assets: a reproducible build (the same code produces the same artifact on any machine), verifiable quality (the mistakes worth blocking get blocked by a machine, not by somebody remembering), and a reversible runtime (when the new version comes in, the old one drains before it leaves). As long as one of the three still depends on "I remembered," the incident will come from that one.

Six words, one line each (they appear everywhere below):

• Artifact: the output of one build, never modified afterwards — a jar or an image tag. What you deploy is an artifact, never source
• Image tag: the artifact's fingerprint. Tag with the commit sha and you can answer the lethal question "which build is production running right now?"
• Readiness probe: answers "may I send it traffic?" A dependency not ready should pull traffic, not kill the process
• Liveness probe: answers "is this process salvageable?" It should judge the process alone; wire it wrong and a database blip restarts every instance
• Graceful shutdown: stop accepting new requests, let in-flight requests finish, then exit
• Canary release: hand 5% of traffic to the new version first, widen only once it looks healthy

Those five gates map onto the article's spine: quality gates (Sections 2-3), build gates (Sections 4-5), runtime gates (Sections 6-10), exit discipline (Section 8), rollout cadence (Section 12). Every failed release can be pinned to exactly one spot on this tree — which makes it both the starting point of triage and the index of the error table at the end.

This timeline is the single most important picture in the article; every section later dissects one of its frames. Memorise three conclusions first: only the tag CI verified may ship (frame 1), an old instance leaves in two steps — pulled from the pool, then drained — not one (frames 4-5), and a hard kill is the fallback when the grace period expires, never the normal method (frame 7).

After this article you should be able to answer three questions:

1. "It works on my machine" is missing which three kinds of precondition, and which stage supplies each one?
2. Why may readiness depend on the database while liveness absolutely may not? What concrete incident does the reversal cause?
3. After one docker compose up -d, in what order should you confirm what, during those 30 seconds?

"It works on my machine" is the most expensive sentence in the history of shipping software. What it really means is that my machine carries a pile of hidden conditions — versions, time zones, memory, configuration — that happen to add up to a working state, and none of those conditions were ever written down. Delivery is precisely the act of making all of those hidden conditions explicit.

Three real disasters, every one of which I have seen with my own eyes:

What these three share is not "a wrong parameter" but an environment that was never version-controlled. Spread the hidden preconditions out and there are only three families, each with its own place to be written down and its own verifier:

More tests is not the goal. The point of testing is to cover a specific risk at a specific cost. Split tests into three layers, give each layer one class of problem, and you avoid the embarrassment of "hundreds of tests and the production incident still slipped through."

Start with the happy path: enough stock, correct amount, order persisted.

The most typical behavior of a payment gateway is to call back more than once. When the same trade_no arrives twice, the system may book it only once.

Once the contract is frozen, the front end can build in parallel. @WebMvcTest loads only the web layer, starts no database, and tests the HTTP face alone.

SQL, unique constraints and transaction rollback can only be tested against a real database. Testcontainers' trick is declaring a throwaway real MySQL in code, so the test machine needs nothing preinstalled.

The number that fools reviewers most reliably is "82% line coverage". Both tests below make that line count as covered; only the second actually blocks a regression.

A unit test can never catch oversell — a mock does not run concurrently. Concurrency problems must be verified with concurrency; this is the one point in this article on which there is no compromise.

CI's value fits in one sentence: expose mistakes before the merge, not after go-live. Manual checks will always miss; a machine will not, as long as the rules are right.

Here is a ready-to-use GitHub Actions pipeline with three stages: build-and-test → package-and-push → deploy. The stages are chained with needs, so a failure upstream stops everything downstream — fail fast.

The pipeline's last step produces an image rather than a jar, because an image seals "application + JDK + time zone + startup flags" together, while a jar only seals the application. This is #37's recipe in its final delivery form:

Three hygiene problems of the build stage hide in the trade-offs of this section — whether .dockerignore has to be written first, why the builder's ~/.m2 re-downloads every time, and why an alpine/musl base intermittently fails name resolution. Pick build and you get them frame by frame:

The order of those four COPY lines follows Docker's layer-cache rule: the most stable layer gets copied first and the most volatile one stays outermost, so a rebuild only invalidates the top few layers. This figure peels the image from bottom to top — click each layer to see what update forces it to rebuild:

The skeleton can also be generated directly: tick the items this version needs and every line comes annotated with the precondition it satisfies (the shutdown signal and memory awareness are precisely what make Sections 8 and 9 work inside a container):

That ./scripts/wait-healthy.sh app line is the only place in the whole pipeline that separates "the container exists" from "the service works". Do not sleep 30; poll:

A pipeline is not isolated; it has to mesh with the branch model. BeeOrder uses the simplest trunk-based variant: main protected + PRs must be green + releases triggered by tags.

The choice you really have to make before shipping is "what shape is the deliverable". These three are not alternatives on one axis: jar and war are packaging formats; an image is a runtime environment.

**A war that wants to boot with java -jar and deploy into an external Tomcat has two hard prerequisites**; miss either and you get a 404:

This also explains the most common delivery-stage "it packaged fine but will not start": launching a repackaged fat jar with java -cp app.jar com.beeorder.BeeOrderApplication, and getting class-not-found — the jars in BOOT-INF/lib are not on the system classpath, and only JarLauncher's custom class loader knows they exist. The lab below reproduces exactly that:

A development Compose only needs "it starts." A production Compose must answer four questions: how to restart, how to keep data, how to cap resources, and how to collect logs.

A MySQL backup is one cron entry and one command:

Frames 4, 5 and 6 of the animation in Section 0 all belong here. Graceful shutdown is a three-leg relay, not a setting: the orchestrator pulls the instance from the pool, the app waits for in-flight requests to finish, and the signal genuinely reaches the JVM. Break any leg and the symptom is identical — "not one shutdown line in the log, and requests got severed".

The three config blocks map to the three legs, and none is optional:

Both grace periods must line up — this is the most common mismatch. Spring is configured for a 25-second drain while docker stop waits only 10 seconds before SIGKILL, so the last 15 seconds of requests are severed anyway.

Misconfigured and aligned side by side is the whole content of this comparison figure — on the left, the fifteen seconds that get severed; on the right, what the same numbers look like once they agree:

The verification method is crude but it works: run docker compose stop --timeout 40 app in one terminal and docker compose logs -f app in another. It only counts if you see Commencing graceful shutdown and Waiting for requests to complete. This lab walks the whole signal chain and all three shutdown postures:

The signal leg inside an image is where things most often break. dockerimg's signal argument demonstrates exactly how the shell form of ENTRYPOINT parks /bin/sh on PID 1 and swallows SIGTERM along the way:

Chain the three legs together and you get the full shutdown timeline below — from readiness turning red to the process actually exiting, every beat shows who waits for whom, and which beat leaves which symptom when it breaks:

Rather than reading this chain twice, walk it in the inner-kernel console — these commands play the probe division of labour through to hard-kill versus graceful side by side, and afterwards every line of the config looks familiar:

Who supplies the probes config from the last section? Actuator. During delivery it is the app's one and only operations interface, and it is also the item security audits love to name.

Probes, grace periods, image tags — delivery terminology all sounds alike, yet the difference is entirely "which question does it answer", and one miswire is one incident. Time for a terminology match: click a term on the left, then the question it actually answers on the right:

Port 8080 inside the container is never exposed to the public. Nginx handles TLS, compression, real-client-IP forwarding and the upload size cap — none of which the application should do for itself.

On the application side you must turn on server.forward-headers-strategy=framework, and the reason is plain: by default Spring Boot does not trust these proxy headers. Without it, redirect URLs the app generates come out as http:// instead of https://, request.getRemoteAddr() reports Nginx's internal address, and absolute-URL assembly goes wrong.

Nine out of ten go-live incidents are "one thing that should have been checked got forgotten in the heat of the moment." A checklist's job is to let the list replace your memory.

On a single-host Compose setup, the most practical choice is image-tag switching. Record the previous tag; when something breaks, switch back.

During a canary, old and new versions run side by side, and the failure you actually get is not a 500 but "the same request landed on two different versions" — so the precondition for canary-ing is that the idempotency key, the state machine and the schema can all hold both versions at once. This lab shows the 5% instance coexisting and being switched back:

wrk is right for quickly getting a "good enough number"; JMeter is right for complex scenarios (parameterization, ramp-up). Which one matters less than the rule that you define the target before you test.

The acceptance targets for BeeOrder's checkout endpoint:

A typical checkout load test (wrk):

Bottleneck hunting follows one fixed path, from the outside in:

• Application: is the thread pool saturated (Tomcat max-threads), is there lock contention
• JVM: is GC frequent (jstat -gcutil), is the heap large enough
• Database: are new slow queries in the log, does EXPLAIN use the index
• Connection pool: is HikariCP's active connection count pinned at maximumPoolSize

The "slow request and timeout" path is the one most often misattributed to the application while it is really being dragged down by something downstream. This lab lines all five outcomes (success, validation failure, business error, database failure, slow request) up on the same pipeline; watch where the queue forms and which layer times out first:

"How much do I switch over" and "how does the old process leave" are the only two dials in a rollout, and they are the scene of every release incident. This sandbox puts both dials on one panel, and each cell reports the same six numbers for one load test: new-version error rate, P99, in-flight requests severed, corrupted rows created, and time to roll back.

Every "error text" column below can be copied and searched verbatim — do not paraphrase it, do not abbreviate it. The annoying part of the delivery stage is that many failures never surface as a Java exception: the symptom is a log that stops mid-line, or a port that refuses you.

The ClassNotFoundException in row 1 deserves a scene of its own — what makes it confusing is that the build was green and the jar is right there in your hand, yet the message reads as if the class was never packaged. Do not look at the answer yet; click the frame you believe is the culprit:

One warm-up first, testing the decision card in Section 9 and the probe configuration in Section 8:

Then a combined question tying Sections 4, 6 and 15 together:

Goal: confirm with your own eyes that graceful shutdown works — not that it is "configured", but that it is visible in the log. You need one machine with Docker and a BeeOrder image that starts.

Step 1, bring the stack up with Section 7's Compose file and confirm it is healthy:

Step 2, generate some in-flight traffic, then stop:

Step 3, check that these three lines appear in order (if any is missing, go back to Section 8):

Goal: use one deliberate database outage to watch readiness go red while liveness stays green.

If liveness also went red in step 3, you pointed it at the aggregate /actuator/health. Go back to Section 9's group block.

Write a plan no longer than one page that answers these five questions, each with a concrete "how I will verify it":

1. How long does rollback take, and who presses the button? (Hint: switch the tag only, target under 60s.)
2. During the drill you deliberately inject "a slow query on the new build at 5% traffic" — how do you spot it within five minutes? (Hint: the first two cells of Section 14's sandbox, and a version tag on every metric.)
3. During the canary, one payment callback hits the old version and then the new one. Is idempotency still intact, and on what basis? (Hint: #45 Section 5's unique-index backstop.)
4. Is the DDL forward-compatible? When you roll the image back, does the database come with it? (Hint: checklist item 13 — add before you remove, split across two releases.)
5. Which three numbers are your only evidence for "widen the canary"? (Suggested: error rate, P99, pool active connections — the same three as Section 12's table.)

The acceptance criterion is a single sentence: the drill must actually press the rollback button, not "I know how". A rollback plan that has never been executed is not a rollback plan.

• Delivery = writing the hidden preconditions down: build premises into the Dockerfile, runtime premises into Compose, process premises into branch protection and needs
• Layer tests by risk, not by count: units for branches, slices for contracts, integration for SQL and concurrency; oversell can only be caught concurrently, and coverage never substitutes for assertion strength
• What you ship is an image tag, not a jar and not source: no CI-verified sha, no deployment; rollback is switching the tag
• Graceful shutdown is a three-leg relay: pull traffic → drain → exit, both grace periods aligned, and the ENTRYPOINT in exec form
• Liveness carries no dependencies; readiness does: reversed, one database blip becomes a site-wide restart storm
• A rollout has exactly two dials: the traffic percentage sets your blast radius, the exit method sets how much corrupted data you create — configure both

Looking back, BeeOrder's go-live is not the credit of any single article:

• In #1, installing the JDK felt like installing software; it was really about the "bytecode + virtual machine" abstraction
• Learning IoC/AOP in #6-#14 is what lets #45's checkout service gain transaction power from one annotation
• The configuration and REST contract from #21-#25 are the real groundwork for #44's API contract
• The connection pool in #28, transaction internals in #31 and caching in #32 all resurface in Section 13's load test and Section 11's checklist
• The testing and deployment of #33-#37, plus this article's CI/CD, upgrade "it runs" into "I dare ship it"

One last question from #45 closes the series: checkout and stock deduction must share one transaction (both succeed or both roll back), writing the stock ledger shares it too (it is part of the books), but sending notifications and adding points must commit independently — a failed notification must not roll back the order. That boundary judgement runs through every trade-off from #31's transaction propagation, through #45's idempotency design, to this article's canary and rollback.