How it works and its sources

How it works and its sources for sigstar. Collect market, on-chain, news and macro data, and explain, forecast and alert on crypto price moves from every…

collect (dlt)         organise (dbt)                analyse (Python)          outputs
public APIs ──► sigstar_raw.* ──► sigstar_staging.* ──► sigstar.* marts ──► explain, relationships,
chainstar warehouse (read-only ATTACH) ──────────┘     features_daily      anomalies, forecasts,
                                                                            alerts ──► brief (MD + HTML)

• collect: dlt pipelines load each source into DuckDB (schema sigstar_raw) with merge loads on each table's key. Every stream (one market on one venue at one interval, one news query, one series) keeps its own cursor in dlt's state, so runs are incremental, and adding an asset backfills only that asset. A stream that fails (a busy API, a venue not offered in your country) is reported and skipped; the others still load. Only closed candles are stored.
• organise: a bundled dbt project (dbt-duckdb) builds staging views and marts, then runs its data tests. A source never collected yields an empty table, so the marts always build.
• analyse: pandas and statsmodels read the marts read-only.
• polite HTTP: one client for every source: identifying User-Agent, per-host pacing at or below each published limit, robots.txt obeyed (RFC 9309, with Crawl-delay), retries with backoff and Retry-After, an on-disk cache. Details per source in SOURCES.md.

Files sigstar writes next to sigstar.yml: warehouse.duckdb (all data), .sigstar/ (dlt state, dbt profile and logs, HTTP cache), reports/, alerts.jsonl and alerts.csv.

Sources

Several exchanges license their market data for personal use only: keep reports and data to yourself. Which news feeds were left out and why, rate limits, and the attribution each source asks for are in SOURCES.md. Reports carry the required credits.