Why False Positives Kill Incident Response (And How to Fix Flaps)

The Psychology of Alert Noise in Distributed Teams

In modern systems engineering, your monitoring software is your team's nervous system. When that system cries wolf multiple times a week, human psychology kicks in: engineers create Slack mute rules, defer investigating alerts, and assume the problem is an upstream glitch.

The cost of a false positive is not just the 15 minutes spent checking Grafana; it is the erosion of vigilance that leaves real outages unaddressed until angry customers tweet at leadership.

“Every notification that does not require an immediate, human-driven operational action is not an alert—it is log spam.”
— Google Site Reliability Engineering Handbook

Implementing Exponential Flap Damping

Flapping occurs when an endpoint rapidly alternates between UP and DOWN states due to packet loss, garbage collection pauses, or rate limits. Naive monitors send 20 alerts in 10 minutes.

StatusEagle uses an exponential decay penalty system: each transition adds penalty points. While penalty remains above threshold, alerts are suppressed into a single aggregated digest.

flap-detector.ts

// Exponential decay penalty for state oscillation
export class FlapDampingMonitor {
  private penalty = 0;
  private readonly decayFactor = 0.85; // Halves every 4 evaluation cycles
  private readonly flapThreshold = 100;

  recordStateChange(isDown: boolean) {
    this.penalty = (this.penalty * this.decayFactor) + 35;
    const isFlapping = this.penalty >= this.flapThreshold;

    return {
      isFlapping,
      currentPenalty: Math.round(this.penalty),
      suppressNotification: isFlapping
    };
  }
}

Four Rules for Calibrating Incident Urgency

To achieve a true 'Zero False Positive' posture, classify incoming telemetry into distinct escalation tracks:

SEV-1 (Wake-Up): Confirmed 100% outage across 2+ distinct geographic regions for > 90 seconds.
SEV-2 (High): Latency p95 exceeds 2500ms for 5 consecutive minutes; alert Slack but don't dial phones.
SEV-3 (Informational): Single region latency spike; auto-trigger routing diagnostics and archive in audit log.
SEV-4 (Maintenance): SSL expiring in 30 days; post as low-priority GitHub issue or Jira task.

Was this article helpful?

Your feedback helps us refine our engineering deep-dives.

Written by Marcus Thorne

Principal Infrastructure Architect

@mthorne_infra

Focuses on TCP/TLS telemetry, DNS latency anomalies, and automated incident triage pipelines across multi-cloud deployments.

The Anatomy of a Modern Multi-Region Uptime Probe

Building Customer Trust During Outages: Status Page Playbook