What pull-request review cannot prove, and why nobody notices

An accessibility fix went up as a pull request. Eleven added lines, one file. Lint passed. Unit tests passed. An AI code reviewer read it, summed it up correctly, and said it had no actionable comments. The fix also broke every filter menu on the page. Pick an option with a mouse, and the menu closes and nothing gets filtered.

This is pull request 50 on ShipGuarde's public demo app. It is a demo, and the bug was planted on purpose, so read it as a worked example. But nothing about it is exotic. It is the blur-before-click race, one of the oldest traps in front-end work, and it got through for a reason that applies to any team: half of the bug was never in the diff.

Eleven lines, every check green#

The pull request does one thing. Escape and a click outside already closed an open filter menu. But a keyboard user who tabbed past the last option was left with the menu open behind them. So the change listens for focus leaving the menu:

wrap.addEventListener("focusout", (e) => {
  if (!wrap.contains(e.relatedTarget)) closeAllDropdowns();
});

If focus moved to something inside the menu, stay open. If it moved anywhere else, close. It is a sensible fix, written the way most of us would write it. Here is everything that checked it:

• Lint passed.
• Unit tests passed. One of them is called, literally, "a group filter keeps only matching rows".
• An AI code reviewer passed it. Its walkthrough described the change accurately. It rated the review effort trivial, about five minutes, put the merge risk at minimal, and reported that no actionable comments were generated. Its check went green.
• The author's test plan had five ticked boxes: lint, tests, tab from the Status search box through the options and Clear (menu stays open), tab out past Clear (menu closes), and Escape and outside click still close it.

Every item on that list is true. The reviewer was right about the diff: these eleven lines do what the description says. The tests were right about the filter logic, which filters correctly. The author tested the keyboard, which is what the change was about. And the page was broken.

What the browser saw#

One more check ran on that pull request. It opened the PR's preview deployment in a real browser and followed a test written as one plain-English instruction. It was saved five weeks earlier and set to run on pull requests. No selectors, no recorded steps:

Click the button labelled "Status" in the filter bar. In the menu that opens, click the option labelled "Failed". Confirm the table shows only rows with a Failed badge and the count reads "5 of 20 runs".

It failed at step 3. The report on the pull request, word for word:

Failed at step 3: the Status dropdown closed, but the 'Failed' option was not selected and the table was not filtered (all 20 runs of various statuses remain visible) (same result twice). Remaining steps not run.

It tried twice, got the same result both times, and stopped. The verdict was Do not ship, and on GitHub its check was the only red one.

I did not take the agent's word for it. I opened the same preview in headless Chromium, then in WebKit, clicked Status, clicked Failed, and counted rows: 20 before, 20 after, and the menu gone. On the production build, which does not have the change, the same two clicks leave the menu open and the table at 5 rows. Then I did it with the keyboard on the preview: tab to Failed, press Space, and it filters to 5. The keyboard path works. Only the mouse is broken, and not just in Status: the Priority and Owner menus fail the same way.

Half the bug is not in the diff#

Here is what happens when you click Failed, event by event. I logged it in the browser rather than reasoning it out.

• When the menu opens, the code puts focus in its search box.
• You press the mouse on the Failed option. Focus leaves the search box, so focusout fires.
• relatedTarget, the element receiving focus, is null. You pressed on a label, and a label does not take focus.
• !wrap.contains(null) is true, so the menu closes and is removed from the page.
• You release the mouse. The option you pressed no longer exists, so no click lands on it, no checkbox changes, and the table stays at 20 rows.

Why does nothing take focus? Each option has a real checkbox inside it, kept for keyboard and screen-reader users. But it is drawn invisible and set to ignore the pointer, so the mouse always hits the label around it:

/* styles.css */
.ms-option input[type="checkbox"] {
  position: absolute;
  opacity: 0;
  width: 1px;
  height: 1px;
  margin: 0;
  pointer-events: none;
}

That rule went in five weeks before the pull request, in a commit titled "style: custom check indicator in filter menus". It was a good change too. The pull request did not touch that file.

So the bug is the product of two correct changes, in two files, five weeks apart. The diff contains one of them. It shows a focusout handler that is right. It does not show the CSS rule that decides where a mouse click lands. A reviewer of those eleven lines, human or model, only sees the other half if it goes looking for it. Even then it is reasoning about what a browser will do, not watching one do it.

Suspecting is not proving#

Reading code can get close. ShipGuarde runs its own code-review agent next to the browser check, and on this pull request it did flag a focusout race that could close the menu before a click lands. It hedged, as a careful reader should: a click "may be lost or the UX is broken".

That is the most reading can honestly give you: a well-founded guess about what a browser will do. Some of those guesses are right. Some are not. A reviewer that calls every focusout handler a possible race soon becomes a reviewer people skim past.

The difference is in what a check can do with its opinion. A comment is advice. A gate needs evidence. On this pull request the verdict names its evidence in the headline: "Status dropdown filter flow failed in Visual QA due to premature focusout closure". The code finding pointed at the cause. The browser proved the effect.

An earlier demo pull request, number 42, shows the same split from the other side. It added a soft, nearly invisible shimmer over the same filter bar. That time the AI code reviewer did spot the cause. It left an inline comment rated major, explained that the overlay covers the search box and the filter controls, and suggested pointer-events: none. Its check still went green, because a comment is not a block. ShipGuarde opened the preview, tried to open the Status filter with clicks, hover and the keyboard, got nothing back, and said do not ship.

Two pull requests. In one the reviewer found nothing to flag, in the other it found the exact line. Both times its check was green, and both times the only red check was the one that clicked.

Why nobody notices#

The second half of the title is the part that costs money. Look at what else ran on pull request 50:

• Console and network: no console or network errors.
• Performance: 100 out of 100.
• Lint: no issues on the changed file.

Nothing throws. Nothing logs. Nothing is slow. There is no signal for any dashboard to pick up. From the user's side, they open Status, click Failed, the menu closes, and the table looks the same. It looks like a misclick. And if someone does report it, the developer who checks with the keyboard, the way the change was tested, will find that it works.

Every check on that list looked at its own slice and reported honestly on it. The bug lives between the slices: in the finished page, after the CSS applies and the events fire in their real order. Only a browser goes there.

What it takes to see it#

I am the founder of ShipGuarde, so weigh this part with that in mind. But the recipe does not need any product. It needs three things.

• A real browser on the real build of the pull request. Not the diff and not a static render. The preview deployment, with the real CSS, the real scripts and the events in their real order.
• A user action, checked by its outcome. "Pick Failed and the count reads 5 of 20" is a better test than "the menu closes on focusout", because it does not care how the menu is built. It only cares whether the filter works.
• A run on every pull request, including the ones that look unrelated. This change was about the keyboard. The breakage was on the mouse. The author's test plan, reasonably, checked the keyboard.

If you already have an end-to-end suite, that gives you the honest question to ask of it: does any test pick an option in that filter with a pointer, on every pull request? If it does, it would likely have caught this too. If your tests follow what each change was about, it would not have, because this change was about the keyboard.

This is what ShipGuarde does on each pull request. It reviews the diff, drives the preview in a real browser, and gives one call: ship or do not ship. The test that caught pull request 50 is the plain-English instruction quoted above, saved once.

The takeaway#

Code review is good at code. On pull request 50 an AI reviewer described eleven lines correctly and found nothing wrong with them, and it was right. Nothing was wrong with them. The bug was in how those lines met a CSS rule from five weeks before, in a browser, under a mouse.

Reading can suspect a bug like that. Only running the page can prove it. If every check on your pull requests reads, then some of your green builds look like this one, and nothing will tell you which.