48.4k Log in Get started
Metric

What is data freshness, and how do you monitor it in Metabase?

Data freshness is how old the newest successfully delivered data is, per destination or table. It's the metric that catches what failure counts can't: pipelines that stopped running quietly, syncs that succeed while moving nothing, and dashboards silently reporting last Tuesday. Measure it in Metabase from the sync logs and load audits your customer data stack already produces (Segment, RudderStack, Hightouch, and warehouse loaders).

TL;DR — Freshness = time since the last successfuldelivery, per destination. Set an SLO per destination based on the decision the data feeds, and alert on breaches — it's the metric that notices when a pipeline goes silent instead of failing loudly.

How is data freshness defined?

For each destination (or table): now minus the timestamp of the last successful delivery. Two refinements make it trustworthy:

  • Successful matters — a run that errored or synced zero rows doesn't reset the clock.
  • Measure both hops — event time to warehouse (load lag), and warehouse to destination (sync lag). A fresh warehouse feeding a stale audience sync still burns money.

Freshness SLOs

A freshness number without a threshold is trivia. Give every destination an SLO derived from the decision it feeds:

  • Hours — suppression lists, audience syncs feeding live ad targeting, lifecycle-messaging triggers.
  • Daily — spend and performance dashboards, executive reporting, most joins with revenue data.
  • Weekly — cohort analyses, historical models, anything read monthly.

Store the SLOs in a small table (destination, freshness_slo_hours) so the breach query stays declarative — and so changing an SLO is a data change, not a dashboard edit.

What data does freshness need?

  • A sync run log with run_finished_at, status, and destination — from audit tables, the tool's API, or your loader.
  • A loaded_at timestamp your pipeline stamps on every batch, for the warehouse hop.
  • The event timestamp inside the data, to separate load lag from sync lag.
  • The SLO table above, one row per destination.

SQL patterns

Hours since last successful sync, by destinationPostgreSQL
SELECT
  destination,
  MAX(run_finished_at) FILTER (WHERE status = 'success' AND COALESCE(rows_synced, 0) > 0)
    AS last_successful_sync,
  ROUND(EXTRACT(EPOCH FROM (
    NOW() - MAX(run_finished_at) FILTER (WHERE status = 'success' AND COALESCE(rows_synced, 0) > 0)
  )) / 3600, 1) AS hours_since_success
FROM cdp_sync_runs
GROUP BY destination
ORDER BY hours_since_success DESC;
Destinations breaching their SLOPostgreSQL
SELECT
  s.destination,
  s.freshness_slo_hours,
  ROUND(EXTRACT(EPOCH FROM (
    NOW() - MAX(r.run_finished_at) FILTER (WHERE r.status = 'success' AND COALESCE(r.rows_synced, 0) > 0)
  )) / 3600, 1) AS hours_since_success
FROM sync_slos s
LEFT JOIN cdp_sync_runs r ON r.destination = s.destination
GROUP BY s.destination, s.freshness_slo_hours
HAVING EXTRACT(EPOCH FROM (
    NOW() - MAX(r.run_finished_at) FILTER (WHERE r.status = 'success' AND COALESCE(r.rows_synced, 0) > 0)
  )) / 3600 > s.freshness_slo_hours
ORDER BY hours_since_success DESC;
Load lag: event time vs. loaded time per source tablePostgreSQL
SELECT
  source_table,
  MAX(loaded_at)                          AS last_loaded_at,
  MAX(event_timestamp)                    AS newest_event,
  ROUND(EXTRACT(EPOCH FROM (
    MAX(loaded_at) - MAX(event_timestamp)
  )) / 60, 1) AS load_lag_minutes
FROM warehouse_load_audit
GROUP BY source_table
ORDER BY last_loaded_at ASC;

Pitfalls

Resetting the clock on failed or empty runs.→ A run that errors — or "succeeds" while syncing zero rows — hasn't delivered anything. Filter to successful runs with rows moved, or freshness reads healthy while the pipeline delivers nothing.
One global freshness threshold.→ Daily is generous for a spend dashboard and catastrophic for a suppression list. Per-destination SLOs are the difference between a useful alert and a muted channel.
Confusing event time with load time.→ An event created Monday and loaded Friday looks fresh byloaded_at and stale by event time. Track the gap explicitly — it's where backfills and timezone bugs hide.
Monitoring freshness from the tool's UI only.→ The status page shows one tool. Landing every pipeline's run log in the warehouse puts all of them on one Metabase dashboard someone actually watches.

Where this metric applies

Analytics

Marketing analyticsMarketing & growth integrations

Dashboards & metrics

CDP sync healthSync success rateActivation rateChurn rate

FAQ

What's the difference between freshness and latency?
Freshness is a state: how old the newest successfully landed data is, right now. Latency is a property of each run: how long data took to move. A pipeline can have excellent latency and terrible freshness — fast runs that stopped being scheduled three days ago. Dashboards should alert on freshness; latency explains why.
How fresh does marketing data need to be?
Match the SLO to the decision the data feeds. Budget-pacing dashboards read yesterday's spend, so daily is fine. Audience syncs feeding live ad targeting or lifecycle sends go stale in hours — a suppression list that's two days old means messaging people who already unsubscribed. Set the SLO per destination, not globally.
Why did freshness break without any failed runs?
Because the runs stopped happening: a paused schedule, an expired trigger, a deploy that dropped a cron entry. Zero failures and zero successes look identical to a success-rate metric — which is why freshness and sync success rate belong on the same dashboard. One catches loud failures, the other catches silence.
Where does the freshness data itself come from?
Three places, in preference order: the pipeline's own audit tables in your warehouse (Hightouch writes hightouch_audit.sync_runs natively), the tool's API sync metadata (Segment's Public API, RudderStack's monitoring surface), or a lightweight loaded_at column your loader stamps on every batch.