Keeping a warehouse in sync

Keeping a warehouse in sync for chainstar. Raw blockchain data to clean dimensional tables in your own warehouse, from one command.

chainstar run fetches a range you name. chainstar sync keeps a warehouse current: each time it runs, it continues from where the last sync stopped, up to the newest block that is safe to take, and exits; with --every, it keeps going until you stop it.

Your wallet, kept current

chainstar wallet 0xYourAddress --chain base --recent 1000 --sync

The first run fetches the latest 1,000 blocks and writes base-0xyouraddress.xlsx, as chainstar wallet always does. Run the same command again tomorrow or next month and it fetches only the blocks since then, and rewrites the spreadsheet; --recent matters only the first time. To have it update by itself while the terminal stays open, add --every 1h; Ctrl-C stops it. To run it on a schedule, see Running it as a service. Each wallet keeps its own folder and warehouse, as before, and costs the same per block as chainstar wallet.

A warehouse, kept current

chainstar sync --chain base --from-recent 1000      # first sync: the latest 1,000 blocks
chainstar sync --chain base                         # later: continue where it stopped
chainstar sync --chain ethereum --chain base --every 5m

A chain's first sync into a warehouse needs a starting point: --from BLOCK, or --from-recent N for the latest N blocks (Solana: slots). After that, every sync resumes from the warehouse's own record of its progress, so the same command works every time; --from and --from-recent are then ignored, with a note. Several chains are synced one after the other, and the star schema is built once at the end. Without --chain, sync takes the chains listed under chains: in chainstar.yml, or else every chain already synced into the target.

Each chain needs an endpoint for that chain, so with several give each its own URL in CHAINSTAR_RPC_URL_<CHAIN>, for example CHAINSTAR_RPC_URL_BASE and CHAINSTAR_RPC_URL_SOLANA; chainstar uses it before CHAINSTAR_RPC_URL, and the public endpoint when neither is set. --rpc-url works for a single chain.

A real first sync of Base on the public endpoint, 21 October 2026:

$ chainstar sync --chain base --from-recent 10
2026-10-21T17:28:58Z all      start    chainstar 0.1.0.dev0; target duckdb:warehouse.duckdb; data data
2026-10-21T17:28:58Z base     rpc      https://base-rpc.publicnode.com (free shared public endpoint)
2026-10-21T17:28:58Z base     note     no RPC URL set, so chainstar uses a free shared public endpoint (5 requests/s, fine for trying small ranges); for more, use your own with --rpc-url or CHAINSTAR_RPC_URL
2026-10-21T17:28:59Z base     head     chain id 8453, latest block 52,907,195; safe head 52,907,136 (60 confirmations)
2026-10-21T17:30:53Z base     extract  52,907,127-52,907,136: erc20_transfers 6,509, blocks 10, transactions 2,133  9.1s
2026-10-21T17:30:53Z base     decode   52,907,127-52,907,136: tokens 175, fetched 175  105.5s
2026-10-21T17:30:54Z base     load     52,907,127-52,907,136: 8,827 raw rows; synced to 52,907,136  0.2s
2026-10-21T17:31:02Z all      model    dbt build (full refresh: first sync or rollback of base): 23 models, 54 tests passed  8.2s

and, a few minutes later, the same command without --from-recent continues from the next block:

$ chainstar sync --chain base
2026-10-21T17:31:49Z all      start    chainstar 0.1.0.dev0; target duckdb:warehouse.duckdb; data data
2026-10-21T17:31:49Z base     rpc      https://base-rpc.publicnode.com (free shared public endpoint)
2026-10-21T17:31:49Z base     note     no RPC URL set, so chainstar uses a free shared public endpoint (5 requests/s, fine for trying small ranges); for more, use your own with --rpc-url or CHAINSTAR_RPC_URL
2026-10-21T17:31:49Z base     head     chain id 8453, latest block 52,907,281; safe head 52,907,222 (60 confirmations)
2026-10-21T17:31:50Z base     check    block 52,907,136 still has hash 0x2e6f0107...c0b1bc; continuing from 52,907,137
2026-10-21T17:37:07Z base     extract  52,907,137-52,907,222: erc20_transfers 26,352, blocks 86, transactions 18,783  70.0s
2026-10-21T17:37:07Z base     decode   52,907,137-52,907,222: tokens 586, fetched 411  247.2s
2026-10-21T17:37:07Z base     load     52,907,137-52,907,222: 45,807 raw rows; synced to 52,907,222  0.5s
2026-10-21T17:37:14Z all      model    dbt build (incremental): 23 models, 54 tests passed  6.6s

chainstar sync options, besides --target, --data-dir, --requests-per-second, --chunk-size and --config, which work as for run:

What one sync does, per chain:

1. Head. Asks the node for its latest block and works out the safe head (below).
2. Check. Asks the chain for the hash of the last block it synced and compares it with the one it stored. If they differ, see Reorganisations.
3. Extract and load, in batches of ten chunks: 1,000 blocks with the default chunk size, 100 on public endpoints, whose chunks are 10. Each batch is loaded in one transaction together with the progress it makes, so a crash or a power cut never leaves the warehouse claiming blocks it does not hold, or holding raw rows its record does not cover. Only the batch's blocks are replaced in the raw tables; nothing else is reloaded.
4. Model. After all chains, dbt builds the star schema incrementally: each table processes only rows it does not have yet (see Incremental tables).

Stopping is safe at any point. The first Ctrl-C or SIGTERM (what docker stop and systemctl stop send) stops at the next safe point: between batches, or on Ethereum and Base between a batch's extraction and its token-metadata step, in which case that batch is not loaded and the next sync loads it, reusing the chunk files already saved. The dbt build is skipped too; the next sync does it. So stopping waits for at most one extraction stage, which on a public endpoint can take a minute or two. A second Ctrl-C or SIGTERM stops at once, which is also safe: chunk files appear only when complete, and an unfinished transaction is rolled back.

Can a sync keep up?

Only if your endpoint answers faster than the chain makes blocks. Measured on the free public endpoints on 21 October 2026:

• Solana makes about 2.5 blocks a second. At the public endpoint's 5 requests per second, chainstar fetched 99 slots in 87.5 seconds, about 1.1 a second, so a sync --every few minutes falls further behind each time. To follow Solana's head, use your own RPC and raise --requests-per-second.
• Base makes one block every 2 seconds. On the public endpoint 86 blocks took 70 seconds to extract, faster than the chain, plus 247 seconds reading names of 411 new tokens; that second part shrinks as tokens repeat, because metadata is fetched once.
• Ethereum makes one block every 12 seconds, well within the public endpoint's pace.
• Bitcoin makes one block about every 10 minutes; one block takes seconds.

chainstar status shows the lag; --max-lag turns a growing lag into a failing health check.

How far behind the head: confirmations

A block's confirmations are the blocks built on top of it, plus one, so the newest block has one. Sync takes a block only once it has enough, so the short reorganisations every chain has never reach the warehouse:

Set them per chain in chainstar.yml, or with --confirmations for every chain in a command:

confirmations:
  ethereum: finalized
  base: 120

Reorganisations

A reorganisation deeper than the confirmation depth is rare but possible, so before each sync chainstar checks that the last synced block still has the hash it had. If not, it:

1. compares the blocks it stored, newest first, with the chain until one still matches (blocks link to their parents by hash, so everything before a matching block is unchanged); it checks at most 128 and otherwise stops and asks you to look;
2. deletes everything after that block: its chunk files (whole files, so the rollback may start a few blocks earlier), its raw rows, and its rows in fct_transfers and fct_fees;
3. syncs that range again from the chain as it is now, and rebuilds the star schema in full so the dimensions are recounted.

The log says what happened; from the tests' simulated reorganisation:

ethereum reorg    warning: block 26,239,739 now has hash 0xabababab...ababab, not 0x5f3cfd2d...77e8c1 as synced: the chain reorganised deeper than 1 confirmation
ethereum rollback warning: rolled back from block 26,239,739: deleted 3 chunk files, 47 raw rows and 0 fact rows; syncing them again, then rebuilding the star schema in full

This is tested with a simulated reorganisation on DuckDB and Postgres: afterwards the tables equal a full rebuild from the new chain, and the block before the fork is untouched. Files are deleted before the database, so if the process stops half-way the hash check fails again next time and the rollback repeats.

Incremental tables

The star-schema tables are dbt incremental models. Each keeps, per chain, the highest block it already holds, and a sync builds only rows above that: new transfers and fees are appended, and wallets, tokens and contracts seen again are updated (first seen kept, last seen and counts moved on). Both warehouses use dbt's delete+insert strategy on each table's unique key, which dbt-duckdb and dbt-postgres both support on every version chainstar allows (merge needs DuckDB 1.4 or Postgres 15). dim_dates, a few hundred rows a year, is rebuilt each time from a one-row-per-chain summary (chainstar_staging.int_chain_coverage) instead of from every block.

The tables are rebuilt in full instead whenever older blocks may have changed: on a chain's first sync (a run may have loaded older blocks), after a rollback, with --full-refresh, and on every chainstar run, which therefore behaves as before. An incremental build after two syncs gives exactly the rows a full rebuild gives, on DuckDB and on Postgres, for all four chains; the tests check it table by table.

On Postgres, sync also indexes the raw tables by chain and block, and the incremental tables by key and by chain and block, so a sync reads only its own blocks. DuckDB needs no indexes: it skips data by its min/max statistics.

Two things a full rebuild would recompute but an incremental build leaves as they were: an ERC-20 token's name, symbol or decimals fetched again later (only if you delete data/<chain>/erc20_metadata/) reach older rows' amount only with --full-refresh; the same holds for a Solana mint whose decimals were unknown in its first transfers.

The progress record

Sync keeps its progress in the warehouse, in chainstar_state.sync_progress, one row per chain, written in the same transaction as the data it describes:

Times are UTC. Block numbers on Solana are slots.

Checking on it: chainstar status

$ chainstar status --max-lag 10m
chainstar status: duckdb:warehouse.duckdb at 2026-10-21 17:37:16 UTC

base
  synced to    block 52,907,222, 2026-10-21 17:29:51 UTC (first synced block 52,907,127)
  head         52,907,444; safe head 52,907,385 (60 confirmations)
  lag          222 blocks, 7m 25s
  last run     2026-10-21 17:37:07 UTC, ok
  star schema  built to block 52,907,222
  rows         fct_transfers 32,861, fct_fees 20,916, dim_wallets 19,673, dim_tokens 586, dim_contracts 586

healthy: every chain is within 10m

--json prints the same as JSON for scripts. --max-lag DURATION makes it a health check: it exits with status 1 when any chain's newest synced block is older than that. The lag in time is measured from that block's own time, so it works without the node and includes the confirmation depth: about 2 minutes on Base, an hour on Bitcoin. Allow for that and for the sync interval, for example --max-lag 15m for Base synced every 5 minutes, and --chain bitcoin --max-lag 3h. The lag in blocks needs the node; when it cannot be reached, status says so and still reports the rest.

Running it as a service

Ready-to-use files are in examples/:

• cron (examples/cron/crontab): chainstar sync every 5 minutes. Runs never overlap: a sync refuses to start while another uses the same data folder.

• systemd (examples/systemd/): a service that runs one sync, and a timer that starts it 5 minutes after the previous run finished. Logs go to the journal (journalctl -u chainstar-sync).

• Docker, one container that keeps syncing:

  docker run -d --name chainstar-sync --restart unless-stopped --stop-timeout 180 \
    -v "$PWD:/work" -e CHAINSTAR_RPC_URL_BASE \
    ghcr.io/bit-primitive/chainstar sync --chain base --from-recent 100 --every 5m
  docker logs -f chainstar-sync
  docker exec chainstar-sync chainstar status
  

  --stop-timeout gives the current extraction stage time to finish on docker stop (the default is 10 seconds, after which Docker kills it, which is safe but wastes the stage's work).

• Docker Compose with Postgres (examples/docker-compose.yml): Postgres plus a sync container every 5 minutes, with chainstar status --max-lag 15m as its health check. docker compose up -d in that folder; put a password and RPC URLs in a .env file next to it.

Run the first sync by hand, or keep --from-recent in the command as the Docker examples do: it applies only until the first sync has happened.

Logs

One line per step, with the block range, row counts and time taken:

2026-10-21T17:30:53Z base     extract  52,907,127-52,907,136: erc20_transfers 6,509, blocks 10, transactions 2,133  9.1s

Steps: head, check, extract, decode, load, model, export (wallets), reorg, rollback, wait, stop. Warnings and errors say so after the step name. With --json-logs each line is a JSON object with time, level, chain, stage, message, seconds where timed, and the step's numbers (first_block, last_block, rows, head, safe_head, ...), for log shippers. RPC URLs are redacted in both.

Good to know

• One sync per data folder. A second one refuses to start. Do not point syncs with different data folders, or on different machines, at the same warehouse and chain.
• sync and run on the same chain and warehouse work together when they share a data folder: run reloads every chunk on disk and rebuilds in full. With different data folders, run would replace the rows sync recorded; keep them apart.
• Errors do not stop the others. A chain that fails (node down, rate limited) is logged and recorded in last_error; the other chains are synced, the exit status is 1, and with --every the next round tries again.
• Wallets on Ethereum and Base store blocks only where the wallet has transfers, so their checkpoint hash is asked of the node at the end of each batch rather than taken from the data.
• dbt tests run on every build, over whole tables, as with run; on a large warehouse they take a growing share of each sync.
• A DuckDB file has one writer. chainstar status waits up to 15 seconds while a sync step holds it; other tools that open the file can make a sync fail with a hint, as for run. For a warehouse others query while it syncs, use Postgres.
• Public endpoints are fine for --every 5m on one or two chains. A Bitcoin block is about 6 MB to download from them, so a long Bitcoin catch-up wants your own node.