Most security tooling fails for an unglamorous reason: someone flipped a switch to “block” before the switch had earned it. The evidence on this is unusually concrete, because the best-documented scanner rollout in the industry — the one behind Google’s Tricorder-derived adoption figures — treats a blocking decision as the output of measurement rather than an act of faith. This walkthrough turns that evidence into a sequence you can run on a real pipeline, one rule at a time. The premise of DevSecOps is that checks live inside delivery and belong to the delivery team; the premise of this guide is narrower: a check may only refuse work once the numbers say it should.
What a gate owes its users
Before the steps, the model. Every control in a pipeline occupies a position, carries a confidence level, imposes a cost, and takes an action — block, warn, or record. A gate that blocks is spending the team’s trust every time it fires, which makes it the most expensive action available and the one that most deserves calibration. The field’s shared definition of DevSecOps leaves room for judgement here on purpose: security woven through delivery, owned by delivery, with no separate phase at the end — but nothing in that sentence licenses blocking on vibes. Calibration is what separates the two.
Step one: write the policy before touching the tool
Three documents precede any installation: the gate policy (what refuses a merge, and under what conditions), a store for findings in SARIF 2.1.0 so history survives tool changes, and an identity model for the pipeline itself. The policy needs exactly one hard sentence — a merge is refused only when a finding is severe, fixable, newly introduced by the change under review, and known with high confidence — plus one soft one: everything else becomes tracked work, owned by a named person, expiring within ninety days. Baseline debt predating the programme never blocks. If you have ever turned a content checklist into a repeatable template rather than an act of improvisation — the logic behind standardized content templates — you already understand why the paper comes first: process beats intention, every time it is tested.
Step two: run report-only and sample thirty findings
Set the rule to record. Then pull a sample of thirty findings it produced and read every one, scoring each as true positive, false positive, or “technically true but useless”. Thirty is not superstition; it is the sample size the field’s reference deployment used per rule before trusting it. While you read, note how many findings generated review comments, because the target there is under ten percent — a rule that interrupts reviewers constantly is a rule that will be overridden constantly once it can block. Confirm too that the rule scores changed lines only. Whole-repository scanning buries reviewers in their ancestors’ mistakes, and buried reviewers stop reading.
Step three: compute the effective false-positive rate
From the sample, effective false-positive rate is the share of findings a developer would act on versus dismiss. The bar for blocking is close to zero — not “low”, near zero — because a blocker’s errors are paid for in bypass culture. A worked instance makes the arithmetic honest: one audit found a rule — call it X-17 — whose effective false-positive rate came out at 43.6 percent. Not slightly noisy — worse than a coin flip. The correct response was demotion to report-only, which is exactly what happened. Publishing that number to the team, with the demotion attached, does more for gate credibility than any policy memo, because it proves the rules are subject to the same evidence as the code.
Step four: every exception gets an owner and a fuse
Exceptions are not failures of the gate; they are the gate’s relief valve, and relief valves need maintenance. Each waiver carries a named owner and an expiry — the common template ceiling is ninety days — and the register gets reviewed on a schedule. Expired exceptions should fail loudly. The health metric worth tracking is how many exceptions are open past expiry; a number that only grows means the policy’s soft sentence is being ignored, and the hard sentence will follow it into disuse. This is the same discipline as any quality process: evaluating signals for sustainable growth works because someone reviews the numbers on a cadence rather than when the mood strikes.
Step five: when a DevSecOps rule starts blocking
Flip the rule to block only after all four conditions hold: near-zero effective false-positive rate, review-comment share under ten percent, changed-lines-only scoring, and the thirty-finding sample read. Then watch one number monthly: the override rate. A healthy ceiling is under five percent per month. Overrides are votes of no confidence, and the evidence is blunt about what a surplus means — one team logged 96 blocks out of 1,240 pull requests, with 22 overridden, for a 22.9 percent override rate. The team’s conclusion was not “developers are careless”; it was that the gate had lost authority and needed recalibration. Automation earns the same scrutiny in any pipeline: writers who structure posts with AI assistance still review every output, because unreviewed automation degrades quality quietly and then suddenly.
Reading a real DevSecOps gate’s numbers
The same audit that produced the override figure checked what happened after the gate: 412 deploys occurred, 398 arrived with verifiable receipts — a provenance coverage of roughly 96.6 percent. Nine defects escaped to production that quarter, and post-incident review judged three of them catchable by the pipeline’s existing checks. Three out of nine is the honest middle of this discipline: gates do not catch everything, but a third of escapes is a rate worth having, and the two numbers together — coverage and escapes — tell you where to spend the next month. Fix time matters as well: track the median and ninetieth percentile for new findings, because a gate that finds issues nobody fixes has moved the problem, not solved it.
Traps that swallow well-meaning calibrations
Teams that follow the numbers still trip over the same handful of mistakes, so check yours against the list.
- Blocking everything above medium severity. It feels rigorous; in practice it trains the whole organisation to hunt for bypasses, and the bypass habit outlives the policy that caused it.
- Standing credentials in CI. Long-lived cloud keys in pipeline variables are an incident on a timer; OIDC or workload identity removes the timer.
- Fork pull requests receiving secrets, or untrusted code executing under pull_request_target. Either one turns your own CI into the attacker’s code runner.
- Wildcard trust policies in cloud roles — “anyone in this org may assume this role” is a boundary that stops being one the day an account is compromised.
- Gate rules read from the pull request’s own commit instead of the base commit, letting a hostile change rewrite its own referee.
- Buying the tool suite before the policy, the owners, and the SARIF store exist. The purchase is the least of the work.
None of these is exotic. All of them have public incident reports attached, which is the only comfort worth offering: you are not the first team to make each mistake, and the reports are specific enough to learn from second-hand.
One more trap deserves its own paragraph because it hides inside success. A gate that works accumulates authority, and authority attracts requests: block the deprecated library, block the unapproved licence, block the build from that one team. Each request sounds reasonable. Each one widens the policy past what the evidence supports, until the override rate tells you — as it told the team with the 22.9 percent figure — that the gate has become a suggestion box. Hold the hard sentence to its four conditions, and route everything else through the exception register instead.
The maintenance contract
Calibration is not an event; it is a monthly habit with a fixed agenda: override rate, effective false-positive rates, exception register, escape count, fix-time percentiles. Retire noisy rules; promote quiet accurate ones; re-sample after major tool upgrades. For the wider system this ritual sits inside — the placed-controls model, the standards from SSDF to SLSA, the metrics and their thresholds — the reference is the DevSecOps field guide at xOps School, which is where the figures cited here were drawn from. Run the sequence on one rule, publish the numbers honestly, and the gate you switch on will be the rare kind: one that people trust before it ever blocks them.