Chains and targets

Chains and targets for chainstar. Raw blockchain data to clean dimensional tables in your own warehouse, from one command.

chainstar run checks the identity before extracting anything and stops if the endpoint serves a different chain (a testnet, devnet or another EVM chain).

Bitcoin. For each block chainstar calls getblockhash and then getblock with verbosity 3, which includes the spent output (prevout: value, address, script type) for every input. That is what lets chainstar compute fees and name senders without one extra call per input. Bitcoin Core returns it only for blocks it still has undo data for, so a pruned node fails for older blocks, with an error that says so. Amounts arrive as BTC with 8 decimals and are converted to whole satoshis exactly.

Solana. START:END counts slots. Not every slot has a block: for each chunk chainstar asks getBlocks which slots have a finalized block, then fetches only those with getBlock (jsonParsed, full transaction details, maxSupportedTransactionVersion 1, commitment finalized). Skipped slots produce no rows and cost no failed requests. The end of the range must not be past the latest finalized slot, which chainstar run checks. Version-1 transactions are live on mainnet: an endpoint or setting that accepts only version 0 refuses every block that contains one.

Both extractors send one request at a time, honour --requests-per-second, make up to three attempts per request with a growing delay between them, and ask for gzip (it shrank a Bitcoin block about 2.5 times and a Solana block about 7 times). Tested in October 2026 against https://bitcoin-rpc.publicnode.com (Bitcoin Core 29.3) and https://solana-rpc.publicnode.com (solana-core 4.3.0); https://api.mainnet-beta.solana.com answered the same identity and slot calls. Public endpoints are fine for trying chainstar on a few blocks; a Bitcoin block with input values is about 15 MB of JSON (6 MB compressed) and a Solana block about 4 to 8 MB. These publicnode endpoints are the defaults when no RPC URL is set (see Public endpoints).

Targets

--target (or CHAINSTAR_TARGET, or target: in chainstar.yml) says where the tables go. Every target works with run, sync (its progress record, reorganisation rollbacks and incremental builds), status and export. Passwords, tokens and keys never go in the target or in chainstar.yml: each comes from an environment variable or the warehouse's own standard mechanism, never appears in dbt's profile (dbt reads it from a DBT_ENV_SECRET_* variable, which dbt keeps out of its logs), and chainstar removes it from every error.

ClickHouse, BigQuery and Snowflake need their extra (chainstar[clickhouse], chainstar[bigquery], chainstar[snowflake]; see Install); the Docker image has them all. chainstar doctor --target ... checks the target without changing it: it connects (ClickHouse and Snowflake report their versions), lists a lake's files, checks the MotherDuck token, and names the extra to install when the dbt adapter is missing.

What "tested" means above: live is the full suite (load, build and its 93 dbt tests, two syncs equal to a full rebuild, a reorganisation rolled back, exports) against the real thing in a throwaway container, and on ClickHouse the star schema equals DuckDB's row for row and type for type. Unit checks what chainstar sends: the SQL, its parameters and their types, the Parquet files given to the bulk loader, and the dbt profile. For BigQuery and Snowflake the bundled dbt project is also compiled offline, for a first build and an incremental one, and every model and test must parse as that warehouse's SQL. On the BigQuery emulator (ghcr.io/goccy/bigquery-emulator:0.8.1), which uses BigQuery's own SQL front end, the 30 models build from the compiled SQL and the star schema equals DuckDB's; the emulator cannot run dbt-bigquery (its create schema job fails) or Parquet load jobs (it crashes), so neither was tested, and a bug in it (SELECT x, x AS y ... UNION ALL SELECT 2, 3 returns 2 for y) blanks the Bitcoin and Solana fees, which that test leaves out.

Every target gets the same schemas: chainstar_raw (the extracted columns, as loaded: cryo's for Ethereum and Base, and bitcoin_* and solana_* tables), chainstar_staging (dbt views and the chains seed) and chainstar (the star schema). chainstar sync adds a fourth, chainstar_state, for its progress record. ClickHouse calls them databases and BigQuery datasets.

DuckDB: --target duckdb:path/to/file.duckdb. The file is created if it does not exist. Avoid naming it chainstar.duckdb: DuckDB names the database after the file, so chainstar.fct_transfers would become ambiguous and you would have to write chainstar.chainstar.fct_transfers.

Postgres: --target postgres:postgresql://user@host:5432/dbname. Put the password in the PGPASSWORD environment variable rather than the URL, so it stays out of your shell history. chainstar loads raw tables through DuckDB's postgres extension, which DuckDB downloads from its extension repository the first time it is used. The database user needs permission to create schemas.

MotherDuck: --target motherduck:my_db (or duckdb:md:my_db), with your token in motherduck_token. It runs exactly as a DuckDB file does, through DuckDB's md: paths and its motherduck extension, which DuckDB downloads the first time (the Docker image includes it). Untested against MotherDuck itself (no token was available), so these are open: whether the database must exist first, and whether MotherDuck's transactions cover a sync batch as DuckDB's do.

Parquet data lake: --target parquet:./lake on local disk, or parquet:s3://bucket/prefix on S3-compatible storage. The star schema and the progress record are written in the Hive layout that DuckDB, Spark, Athena and Trino read:

lake/chainstar/fct_transfers/chain=base/date_day=2026-10-21/data_0.parquet
lake/chainstar/fct_fees/chain=base/date_day=2026-10-21/data_0.parquet
lake/chainstar/fct_swaps/chain=base/date_day=2026-10-21/data_0.parquet   (and fct_token_prices)
lake/chainstar/dim_wallets/chain=base/data_0.parquet        (and dim_tokens, dim_contracts, dim_pools)
lake/chainstar/dim_dates/data_0.parquet
lake/chainstar_state/sync_progress/data_0.parquet
lake/_chainstar/published.parquet                           (what was published)

The partition columns are in the folder names only, as those engines expect, so read the files with Hive partitioning, for example in DuckDB select * from read_parquet('lake/chainstar/fct_transfers/**/*.parquet', hive_partitioning = true).

How it is built: in a local DuckDB file, the engine (lake-engine.duckdb in the current folder; choose another with ?engine=PATH), exactly as a DuckDB target, then published to the lake after each successful build. dbt-duckdb can write Parquet directly (its external materialization), but it rewrites each table in full every time and cannot build incrementally; building in the engine keeps incremental builds, the transactional progress record and reorganisation rollbacks the same as on DuckDB, and the lake only ever receives tables that were built and tested. Publishing is incremental too: each partition's row count and a hash of its rows are kept in _chainstar/published.parquet, and only partitions whose rows changed are rewritten. A partition that no longer exists (after a rollback, for example) is deleted on local disk; on object storage, which DuckDB writes but cannot delete from, its file is replaced by an empty one, which readers see as no rows. The manifest is written last, so a publish that stops half-way is completed by the next one. Each file is replaced in one upload, but a reader that lists the lake while a publish runs can see some partitions new and others not yet. Keep the engine file with the data folder: it holds the raw tables and the progress that later syncs continue from. status and export read the lake itself, so they work on any machine that can reach it.

For S3, set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (and AWS_SESSION_TOKEN for temporary keys); without them chainstar uses the AWS credential chain (shared config and credentials files, SSO, instance and container roles). The region comes from ?region=, AWS_REGION or AWS_DEFAULT_REGION. For other S3-compatible storage name its endpoint; http:// turns TLS off, and requests use path-style URLs unless url_style=vhost:

--target "parquet:s3://my-bucket/chainstar?region=eu-west-1"                                   # AWS S3
--target "parquet:s3://my-bucket/chainstar?endpoint=https://ACCOUNT_ID.r2.cloudflarestorage.com&region=auto"   # Cloudflare R2
--target "parquet:s3://my-bucket/chainstar?endpoint=http://127.0.0.1:9000&region=us-east-1"    # MinIO

ClickHouse: --target clickhouse:http://user@host:8123, or https:// for TLS (port 8443 by default), with the password in CLICKHOUSE_PASSWORD. chainstar's schemas become ClickHouse databases, so the user needs to create databases and tables. Raw tables are MergeTree tables ordered by chain and block, with nullable columns. dbt-clickhouse builds the star schema with delete+insert (its delete_insert), which deletes with lightweight DELETE, so the user must be allowed to set allow_nondeterministic_mutations; dbt sessions also set join_use_nulls and cast_keep_nullable, which the models need. ClickHouse has no transaction spanning several statements, so a sync batch is not atomic there: the raw rows are replaced first and the progress row is written last, as a new version of the row (the progress table has a _version column, and readers take the newest). If a sync stops between the two, the next one finds the old progress and loads the same blocks again, replacing them rather than adding them twice (the tests simulate this). A rollback deletes raw and fact rows before it records the new progress, so if it stops half-way the hash check fails again and it repeats.

BigQuery: --target bigquery:my-project, optionally ?location=EU for where new datasets are made. Credentials are Google's Application Default Credentials: the key file named by GOOGLE_APPLICATION_CREDENTIALS, gcloud auth application-default login, or the metadata server on Google Cloud. chainstar's schemas become datasets of the project; the identity needs to create datasets and run load and query jobs. Raw rows go up as Parquet load jobs and the raw tables are clustered by chain and block; the star schema uses dbt-bigquery's merge on each table's key, which replaces the same rows delete+insert would. BigQuery's multi-statement transactions need sessions, which chainstar does not use, so progress is versioned as on ClickHouse. Every sync runs DML and load jobs, which BigQuery bills or counts against quotas. Query results for export arrive as Arrow; whole-number BIGNUMERIC columns come back as 38-digit decimals.

Snowflake: --target "snowflake:user@account/DATABASE?warehouse=WH&role=ROLE". The account is your account identifier (for example myorg-myaccount). The password comes from SNOWFLAKE_PASSWORD; for key-pair authentication set SNOWFLAKE_PRIVATE_KEY_PATH (and SNOWFLAKE_PRIVATE_KEY_PASSPHRASE if the key is encrypted) instead; for single sign-on add &authenticator=externalbrowser. The database must exist; chainstar's schemas are created in it, in upper case, as Snowflake stores unquoted names. Raw rows are uploaded to each table's stage with PUT and loaded with COPY INTO ... FILE_FORMAT = (TYPE = PARQUET) MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE, each file under a new name and with FORCE = TRUE, so a repeated load is never skipped as already loaded, and inside the batch's transaction together with its progress, as on DuckDB and Postgres.