Skip to main content

Warp Syncer

Warp Syncer is a service for syncing with Arweave nodes. It downloads all transaction from Arweave, filters for SmartWeave contracts, parses and saves interactions to the database.


We use Make for build orchestration. To generate binaries simply use make in the root directory

Local development

Setting up a local dev environment for the first time:

# Start all needed services
docker-compose up -d

# Dump the database (only schema is needed) and save it locally
pg_dump -h ... -U ... ... | gzip > gateway_dump.gz
gunzip gateway_dump.gz
psql -U postgres -h -p 7654 -d warp < gateway_dump

# Build and launch syncer
make run


To speed up the build we're using a vendor dir during the build. To make it easier we added commands to the Makefile

# Build container and tag it
make docker-build

# Run container (builds it before)
make docker-run


Syncer has a default configuration that works in local development (src/utils/config). To overwrite it you can a json configuration file or ENV variables:

Environment VariableDescription
SYNCER_DBHOSTPostgres DB host address
SYNCER_DBSSLMODEIs SSL enabled to connect the database. Possible values: enabled or disableds
SYNCER_DBPINGTIMEOUTHow much time can PING to the db take before it's considered a failure
SYNCER_RESTLISTENADDRESSREST API address. API used for monitoring etc.
SYNCER_STOPTIMEOUTMaximum time Syncer will be closing before stop is forced.
SYNCER_LOGLEVELLog level. Possible values: trace, debug, info, warn, error
SYNCER_ARREQUESTTIMEOUTTime limit for requests. The timeout includes connection time, any redirects, and reading the response body
SYNCER_ARCHECKPEERTIMEOUTMiminum time a peer needs to answer in order to be considered responsive. This should be much smaller than request timeout
SYNCER_ARDIALERTIMEOUTMaximum amount of time a dial will wait for a connect to complete.
SYNCER_ARDIALERKEEPALIVEInterval between keep-alive probes for an active network connection.
SYNCER_ARIDLECONNTIMEOUTMaximum amount of time an idle (keep-alive) connection will remain idle before closing itself.
SYNCER_ARTLSHANDSHAKETIMEOUTMaximum amount of time waiting to wait for a TLS handshake
SYNCER_ARLIMITERINTERVALTime in which max num of requests is enforced.
SYNCER_ARLIMITERBURSTSIZEMax num requests to particular peer per interval
SYNCER_ARLIMITERDECREASEFACTORLimit is a float numbef = max frequency per second. Whenever a HTTP 429 Too Many Requests is received we multiply limit by this factor. This way even if the limit is set too high eventually it'll get small enough.
SYNCER_ARLIMITERDECREASEINTERVALHow often limiters get decreased. This timeout won't allow sudden burst to decrease the limit too much
SYNCER_LISTENERQUEUESIZENumber of blocks cached before listener stops downloading new ones
SYNCER_LISTENERREQUIREDCONFIRMATIONBLOCKSMinimum amount of confirmations (blocks on top of the given block) that are required to consider a given block as confirmed (i.e. not being a fork)
SYNCER_LISTENERNETWORKINFONODEURLURL of the node we're using to get the current block height. It's the Warp's Gateway URL to avoid race conditions
SYNCER_LISTENERPERIODTime between requests to the Warp's Gateway for network info
SYNCER_LISTENERNUMWORKERSNumber of workers that download the transactions
SYNCER_PEERMONITORMAXTIMEBLACKLISTEDMaximum time a peer is blacklisted. Even after this duration is over it may take some time for the peer to be re-checked
SYNCER_PEERMONITORMAXPEERSREMOVEDFROMBLACKLISTMaximum number of peers that can be removed from the blacklist Peers that are blacklisted longer than PeerMonitorMaxTimeBlacklisted will get eventually re-used. To avoid spikes we can only remove at most this many peers from the blacklist in one try
SYNCER_PEERMONITORPERIODTime between sending monitoring requests to peers. Peers are downloaded from the arweave API and checked in parallel by couple of workers
SYNCER_PEERMONITORMAXPEERSMax number of peers that can be used for retrying requests
SYNCER_PEERMONITORNUMWORKERSNumber of workers that check peers in parallel
SYNCER_STOREBATCHSIZENum of Interactions that are stored in the Store before being inserted into the database in one db transaction and batch.
SYNCER_STOREMAXTIMEINQUEUEMaximal time interactions wait to be inserted. After this time insert is forced. Example values: 400ms, 1s, 1m
SYNCER_STOREMAXBACKOFFINTERVALMax time between failed retries to save data.


Syncing interactions from Arweave

sync command starts periodically monitoring the network for new blocks. Once a block is verified it downloads all its transactions, checks them and parses Smartweave interactions. Those interactions get inserted to the database. It's possible interactions from many blocks get inserted in one DB transaction, but it is not possible that interactions from one block get divided into different DB transactions.

Monitoring Peers

Syncer periodically asks for peers. It gets NetworkInfo from each peer and sorts by block height and duration of the requests. Later whenever a request to fails it is retried with one of the peers.

If a peer fails to respond it gets blacklisted. Blacklisted peers don't get monitored and aren't used for retrying requests. Occasionally oldest peers get removed from the blacklist and may get re-checked.

Arweave REST API client

Syncer uses its own REST API client, heavily inspired by goar ( It's a complete rewrite that doesn't have all the API calls added, but has:

  • configurable rate limiting per each peer
  • retrying requests with different peers