TECHNICAL CASE STUDY · Reliability · Public infrastructure

Public RPC & node operations

Health means moving forward.

Operated public infrastructure · retirement scheduled November 21, 2026

FROM THE IMPLEMENTATION

Detect stalled chain progress

Python · SELECTED SOURCE
def is_stalled(state: WatchdogState, now: float, window_s: int) -> bool:
    return (
        now - state["eth_advance_at"] >= window_s
        and now - state["boundary_advance_at"] >= window_s
    )
tf-pulsechain/ansible/playbooks/common/nodes/rpc/stall_watchdog.py

Both progress clocks must expire before the watchdog declares a stall. A single delayed signal does not immediately drain a node. Reformatted source excerpt.

THE WORK OF RUNNING THE PLATFORM

Networking, availability and operations

Availability depends on coordinating edge routing, origin capacity, client behavior and repeatable operational changes.

The request path

Public edge and origin pools

Terraform defines proxied Cloudflare entry points, regional origin pools, HTTPS health monitors and least-outstanding-request steering. Multiple origins provide replacement capacity during maintenance or failure. Cookie affinity balances connection continuity against redistribution; this is a configured resilience design, not a guarantee that every failure preserves service.

Restricted origin and local services

Nginx admits Cloudflare traffic at the origin and proxies JSON-RPC to the local execution service. PulseChain also exposes the selected beacon route; Ethereum uses its execution-only public branch. Host firewall tasks, TLS configuration and separate administrative listeners keep public access distinct from client control.

An intentionally simpler current proxy

The later shared profile uses plain loopback proxying and an origin request budget. Its deployment removes the older JavaScript classifier, execution cache and drift-sampler services. Those retained files document earlier engineering; they do not establish current cached RPC or active proxy-level semantic guards.

Redundancy and failover

Prove a peer serves before stopping a node

The deployment gate first requires the target’s local maintenance response, then checks public health and execution synchronization through a serving peer. PulseChain additionally requires a synced, non-optimistic beacon peer with its execution dependency online. It reconfirms local drain before proceeding.

Distinguish progress, consistency and backfill

The implemented watchdog compares execution-head and finalized-boundary progress, persists observation clocks and checks beacon consistency where selected. Historical backfill has a separate alert path: a slow history job need not remove useful current-head service. Observation gaps reset clocks instead of inventing evidence across downtime.

Keep maintenance ownership explicit

Watchdog code writes a dedicated ownership-marked sentinel and removes only verified self-owned state. Operator maintenance stays separate. The later simple proxy no longer visibly checks that sentinel, so end-to-end automatic watchdog draining is unverified for that profile; operator-driven peer drain remains a separately implemented deployment gate.

Abuse prevention and resource boundaries

Budget traffic at edge and origin

Cloudflare rules describe per-client edge request budgets. The current origin profile adds a shared JSON-RPC rate zone keyed by Cloudflare’s client-identity header, returning throttling responses when exceeded. Trusting that header depends on the Cloudflare-only admission boundary; an unrestricted direct origin would undermine the intended attribution.

Match expensive-call policy to the actual client

A method allowlist controls the public client surface, and upgrade qualification checks supported methods and boundary errors, including Ethereum log-range limits. Nginx also bounds request size and upstream waits. Those controls do not bound the cost of one trace or large request. Retired proxy range and concurrency guards are not claimed as active protection.

Contain resource pressure

The execution memory policy requires systemd cgroup v2, a host-capacity preflight, a pressure watermark, a hard ceiling and no swap. Post-reboot checks verify the actual parent/child hierarchy and Go heap setting. The design preserves headroom for consensus and host services rather than treating total RAM as an unlimited execution-client budget.

Deployment and upgrade execution

  1. Select the exact client revision and image, validate source configuration and qualify the affected execution or beacon behavior in the appropriate development lane. A passing syntax check establishes neither chain correctness nor production readiness.
  2. Run the reviewed Ansible playbook serially. Pause conflicting maintenance writers, enter maintenance and verify public peer service before changing the selected origin. The remaining origins carry traffic while that node is unavailable.
  3. Preserve existing chain data. The Ethereum upgrade path checks graceful shutdown and image identity before creating or restoring its checkpoint; it rejects an OOM exit or unsupported stop signal rather than assuming a clean storage transition.
  4. After restart, verify the memory hierarchy, expected chain identity, synchronization and advancing head; compare a fixed-height block hash with the chosen reference where required. Clear maintenance only after readiness, then retain the timed post-change endpoint observations.

Ongoing upkeep

Maintain the serving boundary

Source-controlled upkeep includes refreshing Cloudflare admission ranges, deploying Nginx changes, checking CORS behavior and retaining health, synchronization and runtime-identity evidence. A routing change needs public-path verification because a healthy direct origin does not prove the load balancer selects it.

Separate data recovery from a fresh installation

Persistent chain data and upgrade checkpoints have explicit identities and recovery paths. Published bootstrap snapshots are a separate operations concern; their availability and restore fitness need their own checks. A retained dataset or snapshot workflow does not establish a completed restore or recovery-time promise.

Failure scenarios and operational responses

One origin enters maintenance or fails its edge health monitor.

Pool selection can route to remaining healthy origins; the serial deployment gate requires evidence that a public peer actually serves before continuing.

A client answers HTTP requests but stops advancing, or historical sync stalls.

Progress and backfill observations distinguish the cases. The watchdog algorithm is implemented; automatic draining in the simplified profile needs fresh integration verification.

An upgrade stops uncleanly or a supposedly ready node reports the wrong chain.

The checkpoint/readiness gates fail closed and preserve maintenance. Recovery uses the reviewed data-preserving path rather than an unplanned reset.

THE ENGINEERING PROBLEM

What needed to work

A public RPC can answer a cheap request while its execution client has stopped advancing. Treating HTTP success as readiness keeps a bad origin in rotation and sends users into timeouts. I operated public node services and built operational safeguards around upstream clients, with progress-aware watchdog policy and source-controlled maintenance workflows. Their connection to traffic routing depends on the health profile.

Constraints and requirements

  • PulseChain needs execution and beacon services together; the public Ethereum service is execution-only. Fleet configuration and checks must preserve that distinction rather than infer one universal node layout.
  • Deployment maintenance belongs to the operator. An automatic recovery process must never clear that decision, and a missing signal must not become proof that a healthy node should be removed.

ARCHITECTURE & DESIGN

How the system fits together

  1. Public client requests
  2. Cloudflare load balancing
  3. Nginx routing
  4. Execution RPC / Pulse beacon
  5. Progress health feeds traffic eligibility

Terraform describes Cloudflare DNS, load balancers, origin pools and monitors. Ansible converges Linux hosts, containerized clients, the reverse proxy and systemd timers. The control plane changes configuration through source rather than ad hoc host edits.

The watchdog combines execution-head observations with a separately refreshed finalized boundary and persists state between timer invocations. It records stall maintenance in an ownership-marked sentinel. Traffic removal requires a health/proxy profile that consumes that signal; the later simple health path does not establish that connection by itself.

Decisions and tradeoffs

Require both progress clocks to expire before declaring a head stall.

One delayed observation should not remove a node that is still doing useful work. Either advancing signal resets the relevant no-progress clock.

Tradeoff Conservative detection favors avoiding a false drain over reacting instantly. Thresholds express an operational choice rather than a promise of immediate failover.

Separate watchdog maintenance from operator maintenance.

The recovery path verifies ownership before removing its own sentinel. A restart or a later healthy sample does not grant authority to undo another action.

Tradeoff Additional persistent state and transitions require more testing, but make automatic recovery explainable and preserve deliberate maintenance.

PLANNING & EXECUTION

From requirements to working behavior

  1. I implemented the timer installation alongside the existing RPC deployment tasks. The deployment sequence takes one node through maintenance and verification at a time, with other replicas available to serve. Actual traffic eligibility and progress are checked for the selected deployed profile.
  2. Historical synchronization uses a separate backfill signal and alert path. A stalled backfill is an alert condition, distinct from the head-stall maintenance decision. Observation gaps restart the clocks instead of inventing evidence across downtime.

VALIDATION & RECOVERY

How the behavior is checked

  • Maintained tests exercise progress thresholds, state loading, maintenance ownership, recovery transitions, beacon consistency and backfill behavior. Lint, formatting and strict types support those behavioral checks; actual deployment adds chain comparison and a timed public-endpoint watch.

Implementation references

Repository-relative references identify the implementation behind this case study. Selected code excerpts are also available in the portfolio.

  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/stall_watchdog.py
  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/stall_watchdog_install.yml
  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/beacon_boundary_refresh.py
  • tf-pulsechain/ansible/tests/test_stall_watchdog.py
  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/public_rpc_pulsechain/server.conf.j9
  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/public_rpc_ethereum/server.conf.j9
  • tf-pulsechain/ansible/tests/test_simple_rpc_config.py
  • tf-pulsechain/cloudflare-rpc-pulsechain.tf
  • tf-pulsechain/cloudflare-rate-limits.tf
  • tf-pulsechain/ansible/playbooks/common/nginx/nginx.conf.j9
  • tf-pulsechain/ansible/playbooks/common/nginx/nginx_install.yml
  • tf-pulsechain/ansible/playbooks/common/nginx/update-cfips.j9
  • tf-pulsechain/ansible/playbooks/common/pulse-health/verify_public_drain.yml
  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/public_rpc_pulsechain/execution_memory_verify.yml
  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/public_rpc_ethereum/upgrade_checkpoint.yml
  • tf-pulsechain/ansible/playbooks/common/nodes/rpc/public_rpc_ethereum/verify_ready.yml
  • tf-pulsechain/scripts/rpc_qualification_upgrade.py