TECHNICAL CASE STUDY · Historical work · Streaming data

Firehose & Substreams

Composable blockchain data pipelines.

Historical implementation · removed from later platform scope

FROM THE IMPLEMENTATION

Turn a typed stream message into database changes

Rust · SELECTED SOURCE
fn set_dashboard_metrics_row(tables: &mut Tables, metrics: &dex::DashboardMetrics) {
    if metrics.block_number == 0 {
        return;
    }
    tables
        .create_row("dashboard_metrics", "latest")
        .set("block_number", metrics.block_number)
        .set("timestamp", metrics.timestamp)
        .set("volume_24h_usd", numeric_or_zero(&metrics.volume_24h_usd))
        .set("tvl_usd", numeric_or_zero(&metrics.tvl_usd))
        .set("tx_count_24h", metrics.tx_count_24h)
        .set("active_pools", metrics.active_pools)
        .set("plsx_burned_24h", numeric_or_zero(&metrics.plsx_burned_24h))
        .set("gas_gwei", numeric_or_zero(&metrics.gas_gwei));
}
tf-hetzner/projects/dextrace/substreams/dextrace/src/lib.rs

Selected complete helper from the historical Substreams implementation. The default protobuf message is skipped; real messages become a named database row with numeric formatting handled at the sink boundary. This is implementation evidence, not a claim that the displayed metrics are live or financially complete.

THE ENGINEERING PROBLEM

What needed to work

Blockchain analytics repeatedly needs the same extraction work: discover pools, track reserve changes, identify events and attach token metadata. Pricing logic changes more frequently. In this historical project I built Rust/WASM processing around upstream Firehose and Substreams so applications could reuse extraction without coupling every consumer to one pricing implementation.

Constraints and requirements

  • Raw streaming data, reusable indexed stores, token pricing and an application database have different responsibilities. A useful boundary lets the application evolve while preserving the upstream data model and the provenance of derived values.
  • This was an implemented indexing direction within a broader analytics project. Retained roadmaps include unfinished features, and Firehose/Substreams were removed from the later platform scope. Historical implementation and a completed product are distinct.

ARCHITECTURE & DESIGN

How the system fits together

  1. Firehose block stream
  2. Rust / WASM modules
  3. DEX stores + pricing modules
  4. Database-change output
  5. PostgreSQL sink

The stable stores package contains pool-token mappings, reserve state, factory registries, token metadata and swap or liquidity events. A separate token-prices package imports those stores and produces liquidity-weighted, multi-hop token-price output.

The application module combines pool and event data with imported prices. Its database output uses the upstream Substreams database-change format so an existing SQL sink can materialize records in PostgreSQL instead of requiring a custom transport.

Decisions and tradeoffs

Split durable extraction from faster-changing valuation logic.

Pool identities and event history are reusable inputs. Changing price derivation should not require redesigning the store that every consumer imports.

Tradeoff Package versions and imports become explicit integration work. In exchange, each stage has a smaller responsibility and a clearer reason to change.

Emit an established database-change representation.

Using the existing SQL-sink boundary makes the handoff to PostgreSQL inspectable and avoids embedding database connectivity inside the WASM module.

Tradeoff Database schema and sink operations remain a separate delivery concern. Producing a valid stream does not establish that every derived metric is complete.

PLANNING & EXECUTION

From requirements to working behavior

  1. I implemented the source modules, manifests and build paths for the split packages, along with the application module that joins indexed data into database changes. The recorded workflow generates bindings, compiles WASM and packages explicit module versions before bounded stream or sink runs.
  2. Pricing output includes derivation information intended for auditability, while the application source contains event recognition and aggregation logic. I keep these implemented components separate from the retained roadmap for richer analytics.

VALIDATION & RECOVERY

How the behavior is checked

  • The maintained Rust tests exercise event-topic classification, amount conversion and aggregation behavior, including swap-volume calculations. Module manifests and protobuf definitions make inputs and outputs reviewable. Sustained stream processing, database reconciliation and frontend behavior require their own integration evidence.

Implementation references

Repository-relative references identify the implementation behind this case study. Selected code excerpts are also available in the portfolio.

  • tf-hetzner/projects/dextrace/substreams/pulsechain/README.md
  • tf-hetzner/projects/dextrace/substreams/pulsechain/src/stores.rs
  • tf-hetzner/projects/dextrace/substreams/pulsechain/src/pricing.rs
  • tf-hetzner/projects/dextrace/substreams/dextrace/src/lib.rs
  • tf-hetzner/projects/dextrace/substreams/dextrace/substreams.yaml