How it works and its sources

How it works and its sources for sigstar. Collect market, on-chain, news and macro data, and explain, forecast and alert on crypto price moves from every…

collect (dlt)         organise (dbt)                analyse (Python)          outputs
public APIs ──► sigstar_raw.* ──► sigstar_staging.* ──► sigstar.* marts ──► explain, relationships,
chainstar warehouse (read-only ATTACH) ──────────┘     features_daily      anomalies, forecasts,
                                                                            alerts ──► brief (MD + HTML)

collect: dlt pipelines load each source into DuckDB (schema sigstar_raw) with merge loads on each table's key. Every stream (one market on one venue at one interval, one news query, one series) keeps its own cursor in dlt's state, so runs are incremental, and adding an asset backfills only that asset. A stream that fails (a busy API, a venue not offered in your country) is reported and skipped; the others still load. Only closed candles are stored.
organise: a bundled dbt project (dbt-duckdb) builds staging views and marts, then runs its data tests. A source never collected yields an empty table, so the marts always build.
analyse: pandas and statsmodels read the marts read-only.
polite HTTP: one client for every source: identifying User-Agent, per-host pacing at or below each published limit, robots.txt obeyed (RFC 9309, with Crawl-delay), retries with backoff and Retry-After, an on-disk cache. Details per source in SOURCES.md.

Files sigstar writes next to sigstar.yml: warehouse.duckdb (all data), .sigstar/ (dlt state, dbt profile and logs, HTTP cache), reports/, alerts.jsonl, alerts.csv and alerts-skipped.jsonl (conditions not evaluated because on-chain coverage was not comparable).

Sources

Angle	Source	Default
Prices (OHLCV, daily + hourly)	Coinbase Exchange, Kraken, Binance (data-api.binance.vision)	on
Derivatives (funding, open interest)	Binance USDⓈ-M futures; OKX	Binance on, OKX off (its terms)
Sentiment	Crypto Fear & Greed Index (alternative.me)	on
News volume and tone, headlines	GDELT DOC 2.0	on
Headlines with snippets	Bitcoin Magazine RSS	on
Macro	FRED: DFF, DGS10, DTWEXBGS	on when `SIGSTAR_FRED_API_KEY` is set
On-chain activity, DEX volume, USD prices from DEX swaps	chainstar warehouse (DuckDB)	when configured

Several exchanges license their market data for personal use only: keep reports and data to yourself, or use commercial mode (below). Which news feeds were left out and why, rate limits, and the attribution each source asks for are in SOURCES.md. Reports carry the required credits.