TECHNICAL CASE STUDY · Observability · Operations

Fleet observability

One view across many failure modes.

Implemented components · no current fleet-health claim

FROM THE IMPLEMENTATION

Require a check with a meaningful signal

JavaScript · SELECTED SOURCE
function checksHaveKnownSignal(checks) {
  const signalStates = ['ok', 'healthy', 'degraded', 'syncing', 'warming_up', 'down', 'error'];
  if (!checks || typeof checks !== 'object') return false;
  const entries = Object.entries(checks);
  if (entries.length === 0) return false;
  return entries.some(([, value]) => {
    if (!value || typeof value !== 'object') return false;
    return signalStates.includes(String(value.status || value.state || '').toLowerCase());
  });
}
tf-hetzner/projects/monitoring/admin-api/server.js

Selected complete helper used by health normalization. Empty or entirely unknown dependency checks cannot support an overall healthy claim. This helper establishes that a signal exists; the surrounding logic interprets its health state.

THE ENGINEERING PROBLEM

What needed to work

Infrastructure telemetry often answers whether a process exists without answering whether the service is useful. A reachable indexer can serve stale data; a healthy database does not guarantee a healthy application. I designed a shared observation plane connecting service status, chain progress, resource signals and diagnostic context.

Constraints and requirements

  • The estate spans applications, node integrations, databases and edge services. Each component has its own definition of readiness, so the monitoring interface must normalize useful status without erasing the dependency checks behind it.
  • Public health information and raw operational telemetry have different exposure needs. A dashboard needs a bounded read surface; metrics backends and unrestricted query interfaces should retain their own access controls.

ARCHITECTURE & DESIGN

How the system fits together

  1. Service health + metrics + logs
  2. Exporters + semantic probes
  3. Prometheus / Loki
  4. Grafana + admin UI
  5. Alerts + operator diagnosis

Prometheus collects host, container, application and database metrics. Loki and Promtail provide centralized logs. Grafana dashboards connect service health, chain lag, edge behavior, backup freshness and resource pressure instead of isolating each signal in an unrelated tool.

A custom administration API polls semantic health from services and supplies the dashboard. Application adapters expose named dependency checks and states such as healthy, degraded, syncing and down. The edge separates public health/read routes from protected monitoring and operational surfaces.

A separate region-loss watchdog is designed to run outside the primary region with a small standard-library dependency footprint. Its state machine alerts after all monitored routes are unreachable and reports recovery when one responds. It checks reachability, including HTTP error responses, rather than semantic application health or automatic regional failover.

Decisions and tradeoffs

Separate reachability from semantic readiness.

An HTTP success can carry a down status, an error check or stale data. Qualification needs to inspect those values and compare progress with its source.

Tradeoff Service-specific adapters and freshness rules take more implementation work than a universal port check, but expose the failures that users actually feel.

Design the monitoring interface together with its access boundary.

The public dashboard can consume scoped status without exposing every metrics series, log query or administrative operation on the same edge route.

Tradeoff Route policy and backend scraping need coordinated configuration. Keeping them explicit makes a useful public view compatible with protected diagnostics.

PLANNING & EXECUTION

From requirements to working behavior

  1. I built the shared stack configuration, service dashboards, administration API and conformance checks. The implementation connects health, metrics, reports and logs through common service expectations while preserving project-specific producers and operational vocabulary.
  2. Backup alerts distinguish stale snapshots from missing timestamp series and preserve the identity of each backup plane. A separate missing-series alert covers an entire timestamp plane disappearing, which the staleness rule alone cannot see. The recovery material and running service have separate freshness signals.
  3. The supporting backup workflow attempts service restart after upload failure and releases its lock through a separate cleanup boundary. Retention pruning runs after successful upload and counts completed generations, so partial uploads cannot displace a good backup. This establishes workflow safeguards, not a measured restore time or a completed disaster-recovery drill.

VALIDATION & RECOVERY

How the behavior is checked

  • Maintained tests reject HTTP-200 responses with down semantics, error checks, stale timestamps, excessive index lag and unreachable telemetry. Additional suites check alert rules, dashboard configuration, service registration and the protected/public edge policy. These establish implemented checks, not current fleet-wide conformance.

Implementation references

Repository-relative references identify the implementation behind this case study. Selected code excerpts are also available in the portfolio.

  • tf-hetzner/projects/monitoring/README.md
  • tf-hetzner/projects/monitoring/admin-api/
  • tf-hetzner/projects/monitoring/alert-rules.yml
  • tf-hetzner/projects/monitoring/tests/test_data_floor.py
  • tf-hetzner/projects/monitoring/tests/test_caddy_access_policy.py
  • tf-hetzner/projects/monitoring/dr-watchdog/dr_watchdog.py: Watchdog
  • tf-hetzner/projects/monitoring/dr-watchdog/test_dr_watchdog.py
  • tf-hetzner/ansible/playbooks/backup-service-volume.yml
  • tf-hetzner/ansible/playbooks/postgres-backup.yml