Not investment advice. The interpretations below describe how each
signal is constructed and how comparable measures have behaved historically.
They are for information only, not a recommendation to buy or sell any
security, and past behaviour does not guarantee future results.
Gold signals are pre-computed, opinionated quantitative datasets built from one or more silver sources. They are designed for direct consumption in quant notebooks, backtesting engines, and the external API.
All gold signals are:
Apache Iceberg tables (v2, Parquet-backed) written with an idempotent
MERGE INTO on a record_pk surrogate key — re-running a gold ETL upserts
rather than appending, giving ACID writes and snapshot/time-travel
Stored in the Iceberg warehouse at s3://{gold-bucket}/iceberg/{signal_name}/
Self-registered in the Glue Catalog (stratum_gold_{env}) by the Iceberg
write (no separate catalog-registration step)
Kept compact and bounded by a weekly maintenance job (compaction + 5-day
snapshot expiry) — see etl/maintenance/iceberg_maintenance.py
Lineage-tracked: each ETL run appends a KMS-signed record (inputs, outputs,
record counts, schema hash) to a hash chain that is verified daily, archived
to WORM storage and exportable as an offline-verifiable audit bundle
Checked daily for freshness, emptiness and row-count drops by the
data-quality monitor, which alerts on breach
Access today is IAM plus Athena workgroups. Lake Formation tier-gating is
built but not yet enabled
The record_pk is md5(date) for single-row-per-date signals and
md5(entity|date) for entity-keyed signals (ticker, contract, cusip, …).
The legacy domain=/signal= Parquet layout is retired; the per-signal
Output path rows below now read iceberg/{signal_name}/.
Exception: the email newsletter_inbox gold table (in the email
pipeline’s own Glue database, eu-west-2) remains Hive/Parquet with partition
projection — it has an array<embedding> column and a distinct
catalog/digest path. See
its section.
Gold has no schedule of its own. Each source has a refresh state machine, stratum-{env}-refresh-{source}. An EventBridge rule starts it when that source’s ingestion ends SUCCEEDED. The refresh rebuilds the source’s silver Parquet, then starts every gold signal that reads it (infrastructure/constructs/source_refresh.py). A signal with several sources re-runs after each one.
credit_leverage still treats margin debt as optional: in an environment with no margin silver, the margin z-score is 0 and the composite rests on spreads and auctions.
A gold job whose required silver input does not exist logs a warning and exits successfully instead of failing (lib/glue_job.require_inputs). Each refresh, silver and gold failure ends its execution FAILED after the SNS notification. The stack refuses to synthesize if any gold signal has no refresh to start it.
Corporate spreads (FRED) define the output calendar (daily business days)
FINRA margin data (monthly, dated month end) enters on the first trading day at least 21 days after month end, when FINRA has published it, then is forward-filled to daily. Joining on the month-end date instead would leak about three weeks of future data into backtests, and drop months ending on a weekend
Treasury auction data is joined on auction date; between auctions the last value carries forward
All z-scored over trailing 252 days before composite aggregation
Runs expanding-window walk-forward validation on every gold signal column against SPY forward returns (5d, 20d, 60d horizons). Answers: “does this signal actually predict anything, and is it still working?”
signal_health = “active” → signal is currently predictive, safe to use in strategies
signal_health = “weakening” → historically worked but losing edge; monitor closely
signal_health = “dead” → no evidence of predictive power; do not trade on this signal
Regime-conditional IC reveals whether a signal only works in specific environments (e.g., credit_stress_score may be highly predictive in hiking cycles but useless during holds)
13F data has a 45-day reporting lag. By the time a filing is public, the trade is 1.5-4.5 months old.
The signal’s value comes from (a) identifying persistent trends that continue past the disclosure date, and (b) detecting herd behavior early in the filing window when only some managers have reported.
Not all position changes are directional bets — some are index rebalancing, tax-loss harvesting, or fund inflows/outflows.
positioning_zscore_26w > 2.0 → “extreme_long” — trade is maximally crowded on the long side. Historically mean-reverting. Contrarian sell signal.
positioning_zscore_26w < -2.0 → “extreme_short” — specs are max short. Mean-reverting. Contrarian buy signal.
spec_commercial_divergence — when specs and commercials disagree strongly, commercials (who have physical exposure and structural information edge) tend to be right over 2-8 week horizons.
positioning_momentum = “reversing_to_long/short” — early detection of a positioning unwind. The weekly change direction has flipped vs the 4-week trend.
crowding_score — high values mean the top 4 traders dominate one side, making the market vulnerable to forced unwinds.
Per-ticker technical indicators computed from OHLCV silver data. This is a price-analytics table, not a strategy signal — its columns are reusable features that downstream signals (or notebooks) can consume.
config/yahoo_tickers.json, 59 tickers: broad indices, rates, credit, commodities, crypto, the sector SPDRs, and since 2026-10-07 the inputs of the planned AI bubble index (each tagged “AI bubble index: ” in its description):
Role
Tickers
Megacap concentration / hyperscalers
META, TSLA (plus the existing AAPL, MSFT, NVDA, AMZN, GOOGL)
Breadth (equal weight vs cap weight)
RSP vs SPY, QQQE vs QQQ
AI semiconductors and supply chain
SMH, AVGO, AMD, MU, TSM, ASML
Debt-funded / leveraged AI compute
ORCL, CRWV, NBIS
Data-center power
VRT, CEG
Valuation and speculative froth
PLTR, AIQ, ARKK
Leveraged retail flows
SOXL, TQQQ (volume and relative_volume)
Tech volatility
^VXN
What reads the file. The Fargate ingestion task reads s3://{bronze_bucket}/config/yahoo_tickers.json at run time, not the repo copy. After a change, upload the file there; nothing deploys it. Silver and gold are Iceberg and register new tickers’ partitions by themselves. The partition_projection ticker enum that stateless_stack.py still passes for yahoo_prices is dead config: GoldEtlWorkflow never reads it.
How gold refreshes. When stratum-{env}-yahoo-ingestion ends SUCCEEDED, the source refresh stratum-{env}-refresh-yahoo starts gold yahoo_prices, etf_rotation and risk_composite (see How gold signals run). To refresh by hand: aws stepfunctions start-execution --state-machine-arn <stratum-{env}-gold-yahoo-prices> --input '{}'.
The daily run fetches only PERIOD=10d, so new tickers also need a one-off history backfill before their 200-day and 1-year features mean anything: aws stepfunctions start-execution --state-machine-arn <stratum-{env}-yahoo-ingestion> --input '{"period": "5y"}'.
log_return is the primary input for portfolio-level signal aggregation (additive across time, symmetric for long/short).
cycle_score → +1 suggests peak greed (mean-reversion candidate); → −1 suggests deep fear (accumulation candidate). Built from price-vs-200dma, RSI, vol regime, and volume-price trend, each z-scored against its own 1-year history.
relative_volume > 2.0 with positive daily_return_pct flags institutional accumulation (used by etf_rotation signal).
stratum_gold_email_{env}.newsletter_inbox (note: separate Glue catalog DB from stratum_gold_{env}; eu-west-2, like the rest of the lake, since 2026-10-04)
Source
silver_email ⟕ DynamoDB enrichment cache ({project}-{env}-email-enrichment). Silver itself is fed by SES inbound mail and the hourly RSS / Atom poller (Fed/ECB/BoE press, FT, CNBC, Bloomberg, etc. — see docs/tech/ingestion-scheduling.md#rss-ingestion-phase-2). RSS items land with alias='rss' and source_type='rss'.
Frequency
Daily, 00:30 UTC, partitioned by alias, received_year, received_month
ETL
etl/external/email_gold.py
Enrichment
lambda/email_enricher/handler.py — Bedrock (Nova Micro by default) + Titan Text Embeddings v2
Each inbound newsletter (or, in upcoming phases, RSS / Reuters item) is parsed by an LLM into a typed feature row suitable for ML or signal generation: sentiment, ticker mentions, named entities, claim type, time horizon, plus a 1024-dim embedding for clustering and similarity search. The regex-based v1 (topic_tags, is_newsletter_likely, body_summary_preview) is fully replaced — see git history if you need to reference it.
enrichment_status = 'ok' is the default filter for any analytics query. pending means the silver row hasn’t been enriched yet (possible during catch-up); other statuses indicate a model or parse failure.
tickers are the primary join key to yahoo_prices and any market-data table. They’re uppercased, deduped, shape-checked and filtered against NOT_TICKERS at validation time. Since v4 a ticker can also be mapped from a named company (e.g. “Carnival” → CCL), so treat them as high-precision but not infallible. Non-US symbols carry an exchange suffix (SU.PA), which won’t join to US-only market data.
Repaired rows (since 2026-10-06): when the model omitted summary_short, gold’s value is the first sentence of summary_long; a flag the model left null is stored as false; invalid numerical claims are dropped. Which repairs a row got is recorded on its DynamoDB cache record (repairs), not in gold. See architecture.md § “Validator repairs”.
is_actionable=true AND noise_score < 0.3 narrows to “this is a real analyst view, not a news aggregator”. Useful as an ML label.
embedding + cosine similarity is how the upcoming digest builder will dedupe stories across newsletters/RSS — see Phase 3.
Sentiment vs. confidence: a bullish view at confidence=0.4 is hedged; a neutral view at confidence=0.9 is a confident “no edge here”. Don’t collapse them.
ENRICHMENT_VERSION (in lib/email_enrichment_schema.py) is the contract version between the prompt + schema and what’s persisted. Bumping it makes the pending-list Lambda re-enrich rows below the new version on the next scheduled run (within the 7-day lookback window). For older rows, run scripts/backfill_email_enrichment.py --since YYYY-MM-DD, which uses the same “is this done?” rule and paces to the region’s quota.
Version
Date
Change
4
2026-10-05
Tickers may be mapped from named public companies, ETFs, indices and crypto when the model is certain, not only copied when written as symbols. tickers and entities became required. The validator drops malformed symbols and known non-tickers (NOT_TICKERS: legal suffixes, fiat currencies, indicators, institutions). Re-enriched over 30 days so the anomaly baseline stays comparable
The newsletter_inbox table feeds the daily digest builder — see
docs/finance/digest.md. Phase 3 reads the rows in a window anchored on
gold builds (not a rolling last 24 hours) where
enrichment_status IN ('ok', 'truncated'), clusters by embedding similarity,
scores against a watchlist, and delivers a Markdown briefing daily at 07:00 UTC.