There is a quiet kind of dishonesty that lives in almost every test suite, and most of us never notice it because it shows up wearing the color green.

It looks like this. You have a feature that depends on something heavy — a GPU, a compiled model, a paid API key, a service that isn't always running. So you write a test, and at the top of it you write something sensible: if the heavy thing isn't here, skip this test. The suite runs. Everything is green. You ship.

The problem is that a skipped test and a passing test look almost identical from a distance, and "green" gets read by tired humans as "it works." On a day this week, that gap turned into a rule we now enforce in code. This is a walkthrough of that rule — small enough to copy into your own project by the end of the article.

The setup: optional dependencies are normal

NAOMS runs language models. For a long stretch, the model layer was an optional build feature — you could compile the daemon with the LLM support turned on, or without it, depending on what you were doing. That's a completely reasonable design. Not every developer on every machine needs to pull in a multi-gigabyte model just to run the unit tests for, say, the consent engine.

So the tests for the LLM path were written defensively. Roughly:

if (!llmIsBuilt()) {
  // model support not compiled in — skip
  return;
}
// ... the real assertions ...

Read that on its own and it seems thoughtful. It's trying to be a good citizen: don't fail the build on a machine that legitimately can't run this. The intention is kind. The result is a trap.

Why the skip is a lie

Here is the failure mode, step by step:

1. Someone changes the build so the LLM feature silently stops compiling in — a flag gets dropped, a dependency moves, a refactor reorganizes the feature gates.
2. llmIsBuilt() now returns false. Not because the developer chose to skip the LLM path on purpose, but because of a regression nobody noticed.
3. Every LLM test hits the early return and skips.
4. The suite reports all green.
5. The LLM path is now completely untested — and the dashboard says it's fine.

The test was supposed to answer one question: does the model path work? Instead it answered a different question — is the model path present? — and then reported the answer to the second question as if it were the answer to the first. That substitution is the lie. The test didn't fail when the thing it guards broke. It went quiet, and quiet read as healthy.

This is the same shape as a smoke detector with a dead battery. It isn't beeping. The absence of the alarm feels like safety. It is the opposite of safety, because the one signal you were counting on has been switched off without telling you.

The rule: fail loudly, don't skip silently

The fix landed this week in a single commit with a title that is itself the whole lesson: tests MUST fail when the LLM isn't built — no silent skips.

The change inverts the default. Instead of "if the model isn't built, skip," the rule becomes "the model is supposed to be built in this suite; if it isn't, that's a failure, not a free pass."

In rough shape, the defensive skip:

if (!llmIsBuilt()) return;   // silently green

becomes an assertion:

// In the suite that is SUPPOSED to exercise the model,
// a missing model is a broken environment, not an excuse.
assert(llmIsBuilt(),
  "LLM feature not compiled in — this suite must run against a real model. " +
  "Build with the llm feature, or move this test to the no-LLM tier.");

Two things changed, and only two:

• The default flipped from permissive to strict. Absence of the dependency is now a red signal in the suite that claims to cover it.
• The decision became explicit. If you genuinely want a tier of tests that runs without the model, you now have to say so on purpose by putting them in a different suite — not by letting a runtime skip make the choice for you, invisibly, at the worst possible moment.

That's it. The crypto is nothing. The discipline is everything.

Why this is worth a whole article

Because the temptation to write the silent skip is universal, and it always comes dressed as good manners. Be polite to the machine that can't run this. Don't break the build for someone else's missing setup. Those instincts are good — but they belong in how you organize your suites, not in a runtime if that papers over regressions.

The honest version of "be polite about optional dependencies" is:

• Tier your tests explicitly. A "no-LLM" tier that never touches the model is fine — it just has to be a named, intentional tier, not an accident.
• In the tier that claims to test the dependency, assert the dependency is present. If it's missing, that suite has nothing true to say, so it must go red and tell you why.
• Make the skip a decision a human typed, not a branch a regression took.

There was a companion beat the same week that makes the stakes concrete. A three-agent audit of the in-process model path filed its findings: 3 critical, 3 high, and 4 medium risks. Those risks are exactly the kind of thing a silently-skipping suite would have let sail past as green. You cannot audit a path your tests politely declined to run.

Try it in your own project

You don't need our stack to apply this. Wherever you have a skipIf(...) or an early return guarding a feature behind an optional dependency, ask one question:

If this dependency disappeared because of a bug, would my suite tell me — or would it just go quiet?

If the answer is "go quiet," do this:

1. Decide which suite is supposed to have the dependency. In that suite, replace the silent skip with an assertion that the dependency is present, with a message that says what to build or set.
2. If you also need a path that runs without the dependency, give it its own named suite — and make sure something, somewhere, runs the with-dependency suite for real (a CI lane, a nightly, a pre-release gate).
3. Read your green dashboard once more, and this time count the skips. Every skip is a test that has stopped answering its own question.

A green checkmark should mean "I ran the real thing and it worked." The moment it can also mean "I didn't run, and I'm not going to mention it," the checkmark has stopped being evidence and started being decoration.

Related: Any Test That Passes Over a Gap Is a Lie · The Honest Audit.

Tests aren't allowed to lie. The fix is one assertion and one honest default.

Written by AI agents from real project logs; owned and edited by Mujo.