How Stratum works
Stratum turns raw external data into trustworthy quantitative signals. It does this with a medallion architecture — three refinement layers — plus end-to-end provenance so every number can be traced back to the data that produced it.
This page is a public overview. It deliberately omits deployment, account, and security implementation detail.
Data sources
Section titled “Data sources”Stratum ingests public macro, market, regulatory and news data on schedules matched to each publisher’s release cadence:
| Domain | Sources |
|---|---|
| Macro & rates | FRED economic series (Federal Reserve, BLS, BEA, Census, Treasury), U.S. Treasury auction results |
| Markets | Yahoo Finance prices for indices, sector ETFs, rates, credit, commodities, crypto and AI-related equities |
| Positioning & flows | CFTC Commitments of Traders, FINRA short interest, FINRA margin statistics, SEC EDGAR 13F institutional holdings |
| Energy & risk | U.S. EIA energy data, FEMA disaster declarations |
| News & research | Central-bank and financial news feeds (RSS/Atom) and subscribed research newsletters |
Monthly and quarterly FRED series are only fetched when the publisher has released new data. Data published with a delay, such as FINRA’s monthly margin statistics, only enters a signal once it would actually have been available, which avoids look-ahead bias in backtests.
The three layers
Section titled “The three layers”Bronze — raw
Section titled “Bronze — raw”Source responses are stored exactly as received (JSON, XML, or CSV depending on the source), alongside metadata: when it was fetched, the HTTP status, the source version, and a run identifier. Nothing is interpreted at this stage — bronze is the audit trail.
Silver — normalized
Section titled “Silver — normalized”Bronze is cleaned, typed, and deduplicated into columnar tables (Parquet, with the highest-volume sources on Apache Iceberg). Every silver table is registered in the data catalog, so it can be queried directly with standard SQL.
Gold — signals
Section titled “Gold — signals”Silver tables are joined across sources and scored into domain signals — the actual product. Gold is where, say, Treasury yields, credit spreads and margin debt become a single interpretable indicator. See Gold signals for the full list.
Gold tables are Apache Iceberg v2 tables written with an idempotent
MERGE INTO: re-running a job upserts rather than appends, writes are ACID, and
every change is a snapshot you can query back in time.
Why Iceberg for signals? A signal that feeds a model or a report has to be correct and stay correct when pipelines are re-run. Iceberg gives gold that:
- Safe re-runs and backfills. Each signal row has a stable key, so a re-run or a backfilled source updates the row instead of adding a duplicate. You never have to dedupe downstream.
- Consistent reads. Writes are atomic. A query running while a table refreshes sees either the previous version or the new one, never a mix.
- Recent history. Every change is a snapshot, so you can query a table as it was shortly before a refresh. Snapshots are kept for 5 days, then expired by the weekly maintenance job. This is not an archive; the lineage record is the long-term trace.
- Open and portable. The data is Parquet files with an open table format on top, registered in the Glue catalog. Any Iceberg-capable engine, such as Athena or Spark, reads it with standard SQL. There is no proprietary database to license or export from.
Public sources ──▶ Bronze (raw) ──▶ Silver (normalized) ──▶ Gold (signals) audit trail SQL-queryable Iceberg, the productEvent-driven refresh
Section titled “Event-driven refresh”Gold has no fixed schedule. When a source’s ingestion succeeds, it rebuilds that source’s silver table and then re-runs every gold signal that reads it. A signal built from several sources is refreshed after each of them, so it is never more than one ingestion behind. A gold job whose input doesn’t exist yet exits cleanly instead of failing, which keeps partial environments and backfills safe.
Provenance by design
Section titled “Provenance by design”Every ETL run appends a lineage record signed with an AWS KMS key: the inputs it read, the outputs it wrote, record counts in and out, the schema hash and the run identifier. Records form one hash chain per source and layer: each names the hash of the record before it, so editing, deleting or reordering history breaks the chain.
- Verified daily, in public. A verifier checks every chain each day and publishes a signed anchor, the head of every chain, at www.stratumtrace.io/transparency/, where your browser checks the signature.
- Write-once archive. Every record is also copied to S3 Object Lock storage, which can’t be altered or deleted during its retention period.
- Audit bundles. Any record can be exported, with the rest of its chain and
the anchor covering it, as a zip that verifies offline. The bundle includes
its own verifier, a single Python file:
python verify_bundle.py verify-bundle bundle.zip.
On the roadmap: tier-based access controls, trusted third-party timestamps, and the code version in every record.
Reproducibility & idempotency
Section titled “Reproducibility & idempotency”Each transformation is deterministic and idempotent — re-running a stage on the same input produces the same output, with no duplicate records. That property is what lets the platform be re-run, backfilled, and validated with confidence.
Built on AWS
Section titled “Built on AWS”Stratum is defined entirely as infrastructure-as-code and deployed through a CI/CD pipeline across isolated AWS accounts, with security guardrails enforced at the organization level. Ingestion runs on serverless compute; transformations run as managed Spark jobs; signals are queryable through a managed SQL engine.
What’s next
Section titled “What’s next”The pages alongside this one go deeper on the signals themselves (Gold signals), the validation methodology (Research layer), and how data from different sources is combined (Cross-source joins).