Silent fallback: how sample data hides a dead database

What should a site do when its database is unreachable? Our answer was: fall back to sample content so the public pages still render. It sounds reasonable — and it cost us several hours.

The symptom

Logging into the admin panel was impossible. The password was right, we had verified the hash separately with password_verify, we had inserted the user via SQL. The response never changed: "Wrong username or password."

Meanwhile the site looked perfectly healthy. Projects, roadmap, blog posts — all on the homepage.

The wrong conclusion

"Content is showing, so the database works" — that is what sent us the wrong way.

The models were written like this:

public static function published(): array
{
    $pdo = db();
    if (!$pdo) {
        return SampleData::projects();   // no database → sample content
    }
    // ... the real query
}

The fallback itself is not the problem. The problem was elsewhere: the content in SampleData was literally identical to the seed content in schema.sql. So when the fallback kicked in, the page looked pixel-for-pixel the same.

Authentication, however, knows no fallback:

public static function findByUsername(string $username): ?array
{
    if (!db_available()) return null;   // no database → no user
    // ...
}

With a broken database this always returns null. The controller reads null as "no such user" and answers "wrong password" — whatever password you send.

How we proved it remotely

Without server access, how do you tell whether the database is alive? The answer was in the brute-force protection.

The login lockout relies on an attempt counter stored in the database:

public static function isLocked(string $ip): bool
{
    if (!db_available()) return false;   // no database → no lockout
    // ...
}

So: if the database works, after five consecutive failures the message must change to "too many failed attempts". If it does not work, the counter never increments and the message never changes.

We sent seven attempts. The message did not change. Diagnosis complete — a connection problem, not a password problem.

The cause turned out to be simple: the config.php on the server was the local development version. With no environment variables present it fell through to its defaults of 127.0.0.1, root and an empty password.

What has to change

The fallback was not the mistake; its invisibility was. Three conclusions:

• A fallback must be visible. At minimum the admin panel or a response header should say "sample content mode". The visitor need not see it — you must.
• Sample content must differ from real content. Keeping the same text in two places blinds your diagnostics. Sample names should obviously be samples.
• A fallback must not reach authentication. Saying "wrong password" when the database is down is false information. The correct answer is "the service is temporarily unavailable".

The general rule: failure must be distinguishable from success. If it is not, it is not a fallback — it is a hidden outage.