Skip to content

How Stratum works

Stratum turns raw external data into trustworthy quantitative signals. It does this with a medallion architecture — three refinement layers — plus end-to-end provenance so every number can be traced back to the data that produced it.

This page is a public overview. It deliberately omits deployment, account, and security implementation detail.

Stratum ingests public macro, market, regulatory and news data on schedules matched to each publisher’s release cadence:

DomainSources
Macro & ratesFRED economic series (Federal Reserve, BLS, BEA, Census, Treasury), U.S. Treasury auction results
MarketsYahoo Finance prices for indices, sector ETFs, rates, credit, commodities, crypto and AI-related equities
Positioning & flowsCFTC Commitments of Traders, FINRA short interest, FINRA margin statistics, SEC EDGAR 13F institutional holdings
Energy & riskU.S. EIA energy data, FEMA disaster declarations
News & researchCentral-bank and financial news feeds (RSS/Atom) and subscribed research newsletters

Monthly and quarterly FRED series are only fetched when the publisher has released new data. Data published with a delay, such as FINRA’s monthly margin statistics, only enters a signal once it would actually have been available, which avoids look-ahead bias in backtests.

Source responses are stored exactly as received (JSON, XML, or CSV depending on the source), alongside metadata: when it was fetched, the HTTP status, the source version, and a run identifier. Nothing is interpreted at this stage — bronze is the audit trail.

Bronze is cleaned, typed, and deduplicated into columnar tables (Parquet, with the highest-volume sources on Apache Iceberg). Every silver table is registered in the data catalog, so it can be queried directly with standard SQL.

Silver tables are joined across sources and scored into domain signals — the actual product. Gold is where, say, Treasury yields, credit spreads and margin debt become a single interpretable indicator. See Gold signals for the full list.

Gold tables are Apache Iceberg v2 tables written with an idempotent MERGE INTO: re-running a job upserts rather than appends, writes are ACID, and every change is a snapshot you can query back in time.

Why Iceberg for signals? A signal that feeds a model or a report has to be correct and stay correct when pipelines are re-run. Iceberg gives gold that:

  • Safe re-runs and backfills. Each signal row has a stable key, so a re-run or a backfilled source updates the row instead of adding a duplicate. You never have to dedupe downstream.
  • Consistent reads. Writes are atomic. A query running while a table refreshes sees either the previous version or the new one, never a mix.
  • Recent history. Every change is a snapshot, so you can query a table as it was shortly before a refresh. Snapshots are kept for 5 days, then expired by the weekly maintenance job. This is not an archive; the lineage record is the long-term trace.
  • Open and portable. The data is Parquet files with an open table format on top, registered in the Glue catalog. Any Iceberg-capable engine, such as Athena or Spark, reads it with standard SQL. There is no proprietary database to license or export from.
Public sources ──▶ Bronze (raw) ──▶ Silver (normalized) ──▶ Gold (signals)
audit trail SQL-queryable Iceberg, the product

Gold has no fixed schedule. When a source’s ingestion succeeds, it rebuilds that source’s silver table and then re-runs every gold signal that reads it. A signal built from several sources is refreshed after each of them, so it is never more than one ingestion behind. A gold job whose input doesn’t exist yet exits cleanly instead of failing, which keeps partial environments and backfills safe.

Every ETL run appends a lineage record signed with an AWS KMS key: the inputs it read, the outputs it wrote, record counts in and out, the schema hash and the run identifier. Records form one hash chain per source and layer: each names the hash of the record before it, so editing, deleting or reordering history breaks the chain.

  • Verified daily, in public. A verifier checks every chain each day and publishes a signed anchor, the head of every chain, at www.stratumtrace.io/transparency/, where your browser checks the signature.
  • Write-once archive. Every record is also copied to S3 Object Lock storage, which can’t be altered or deleted during its retention period.
  • Audit bundles. Any record can be exported, with the rest of its chain and the anchor covering it, as a zip that verifies offline. The bundle includes its own verifier, a single Python file: python verify_bundle.py verify-bundle bundle.zip.

On the roadmap: tier-based access controls, trusted third-party timestamps, and the code version in every record.

Each transformation is deterministic and idempotent — re-running a stage on the same input produces the same output, with no duplicate records. That property is what lets the platform be re-run, backfilled, and validated with confidence.

Stratum is defined entirely as infrastructure-as-code and deployed through a CI/CD pipeline across isolated AWS accounts, with security guardrails enforced at the organization level. Ingestion runs on serverless compute; transformations run as managed Spark jobs; signals are queryable through a managed SQL engine.

The pages alongside this one go deeper on the signals themselves (Gold signals), the validation methodology (Research layer), and how data from different sources is combined (Cross-source joins).