What is flaky test rate, and how do you measure it in Metabase?

Flaky test rate is the share of tests — or of runs — that both pass and fail against the same code. Flaky tests erode the one thing a test suite is for: a trustworthy verdict. They drag down first-attempt build success rate, stretch pipeline duration with reruns, and train developers to ignore red builds. Measure the rate in Metabase from per-test results exported from Jenkins, CircleCI, Buildkite, or Azure DevOps Pipelines.

What does flaky test rate measure?

It measures how much of your suite's signal is noise. Two definitions are in common use, and it matters which one you pick:

• Strict (same-commit): a test that produced both a pass and a fail against an identical commit_sha is nondeterministic — no further evidence needed. This requires retry or repeated-run data, which is exactly what CI retries produce.
• Loose (trailing-window proxy): a test whose failure rate over the last few weeks sits strictly between 0% and 100%. No retry data needed, but it overcounts: a genuinely broken test that got fixed mid-window looks flaky under this definition. Use it as a candidate list, not a verdict.

Whichever definition feeds the list, prioritize by cost, not count: a flaky five-second unit test is an annoyance, while a flaky twelve-minute integration test that fails weekly burns hours of rerun time and blocks merges. Track newly-flaky tests per week too — a suite where the flaky set is churning is a suite where reliability debt is being created faster than it's paid down, which eventually shows up across your software delivery analytics as slower lead time for changes and noisier DORA metrics.

What data does it need?

• A test_results table with one row per test execution: build_id, commit_sha, suite, test_name,result, run_at, duration_seconds,attempt.
• The commit_sha is what makes the strict definition possible — without it you can't tell a flaky test from a broken-then-fixed one.
• The attempt number, so retried executions are recorded rather than overwritten — auto-retry that discards first attempts destroys the evidence.

SQL patterns

WITH per_commit AS (
  SELECT
    suite,
    test_name,
    commit_sha,
    COUNT(*) FILTER (WHERE result = 'pass') AS passes,
    COUNT(*) FILTER (WHERE result = 'fail') AS failures
  FROM test_results
  WHERE run_at >= CURRENT_DATE - INTERVAL '14 days'
  GROUP BY suite, test_name, commit_sha
)
SELECT
  suite,
  test_name,
  COUNT(*) AS flaky_commits,
  SUM(failures) AS flaky_failures
FROM per_commit
WHERE passes > 0
  AND failures > 0
GROUP BY suite, test_name
ORDER BY flaky_failures DESC;

WITH flaky AS (
  SELECT DISTINCT suite, test_name
  FROM test_results
  GROUP BY suite, test_name, commit_sha
  HAVING COUNT(*) FILTER (WHERE result = 'pass') > 0
    AND COUNT(*) FILTER (WHERE result = 'fail') > 0
),
failed_builds AS (
  SELECT
    t.build_id,
    date_trunc('week', MIN(t.run_at)) AS week,
    COUNT(*) FILTER (WHERE t.result = 'fail') AS failed_tests,
    COUNT(*) FILTER (
      WHERE t.result = 'fail'
        AND (t.suite, t.test_name) IN (SELECT suite, test_name FROM flaky)
    ) AS flaky_failed_tests
  FROM test_results t
  GROUP BY t.build_id
  HAVING COUNT(*) FILTER (WHERE t.result = 'fail') > 0
)
SELECT
  week,
  COUNT(*) AS failed_builds,
  ROUND(
    100.0 * COUNT(*) FILTER (WHERE failed_tests = flaky_failed_tests)
    / NULLIF(COUNT(*), 0), 1
  ) AS flaky_only_failures_pct
FROM failed_builds
GROUP BY week
ORDER BY week;

WITH flaky AS (
  SELECT DISTINCT suite, test_name
  FROM test_results
  WHERE run_at >= CURRENT_DATE - INTERVAL '30 days'
  GROUP BY suite, test_name, commit_sha
  HAVING COUNT(*) FILTER (WHERE result = 'pass') > 0
    AND COUNT(*) FILTER (WHERE result = 'fail') > 0
)
SELECT
  t.suite,
  t.test_name,
  COUNT(*) FILTER (WHERE t.result = 'fail') AS failures,
  ROUND(
    COUNT(*) FILTER (WHERE t.result = 'fail')
    * (percentile_cont(0.5) WITHIN GROUP (
        ORDER BY t.duration_seconds
      ))::numeric / 60, 1
  ) AS est_minutes_wasted
FROM test_results t
JOIN flaky f
  ON f.suite = t.suite AND f.test_name = t.test_name
WHERE t.run_at >= CURRENT_DATE - INTERVAL '30 days'
GROUP BY t.suite, t.test_name
ORDER BY est_minutes_wasted DESC
LIMIT 20;

Pitfalls

Where this metric applies

• Jenkins + Metabase — per-test results from JUnit reports
• CircleCI + Metabase — test results with timings per workflow
• Buildkite + Metabase — test executions and retries per build
• Azure DevOps Pipelines + Metabase— test runs and outcomes per pipeline
• GitHub Actions + Metabase — test reports from workflow artifacts
• GitLab CI + Metabase — JUnit test reports per pipeline

Related

Metrics

Dashboards

FAQ