Skip to main content

Overview

This tutorial walks through how an application can leverage common streaming data patterns to ingest Stellar network transaction data using a few select packages from the Stellar Go Repo github.com/stellar/go collectively known as the 'Ingestion' SDK:

The Ingestion SDK packages​

  • github.com/stellar/go/amount utility package to convert prices from network transaction operations to string
  • github.com/stellar/go/historyarchive github.com/stellar/go/support/datastore github.com/stellar/go/support/storage utility package with convenient wrappers for accessing history archives, and avoid low-level http aspects
  • github.com/stellar/go/ingest provides parsing functionality over the network ledger data, converts to more developer-centric LedgerTransaction model
  • github.com/stellar/go/ingest/ledgerbackend provides the captive core backend implementation
  • github.com/stellar/go/network provides convenient pre-configured settings for Testnet and Mainnet networks
  • github.com/stellar/go/xdr a complete Golang binding to the Stellar network data model

Ingestion project setup​

Project requirements​

To build an example streaming network ingestion pipeline from live Stellar network transaction data, you'll need:

  • A developer workstation with Go programming language runtime installed
  • An IDE to edit Go code, VSCode is good if one is needed
  • A newly initialized, empty Go project folder. mkdir pipeline; cd pipeline; go mod init example/pipeline
  • stellar-core must be installed on your workstation and available on your o/s PATH

The Stellar network data model is defined in an IDL format expressed in XDR encoding. Our example application is only interested in a small subset of the overall transaction data model related to buying and selling of assets, i.e. a payment, and defines its own data model internally:

::AppPayment
   Timestamp: uint
   BuyerAccountId: string
   SellerAccountId: string
   AssetCode: string
   Amount: string
}

The example application will run a network ingestion pipeline to derive a smaller ApplicationPayment model from the Stellar network transaction data model as 'source of origin' and thus enable the application to avoid large compute resources that would have been required for maintaining storage of the full Stellar network data model.

The ingestion pipeline will perform three distinct stream processor roles:

Inbound Adapter​

Acts as the 'source of origin' for the pipeline. Retrieves LedgerCloseMeta generated from a Stellar network using captive core. LedgerCloseMeta is the top-level aggregate in the Stellar data model of which all Stellar network transaction data is nested within. Publishes the LedgerCloseMeta onto the pipeline.

Transformer​

Subscribes to receive LedgerCloseMeta from the pipeline. Uses the Go SDK package github.com/stellar/go/xdr to parse the nested network data model for payment operations and convert those into a new instance of application data model ApplicationPayment instances. Publishes ApplicationPayment to the pipeline.

Outbound Adapter​

Acts as the termination of the pipeline, it subscribes to receive ApplicationPayment and publishes the data off the pipeline and to an external data store, a ZeroMQ Publisher Socket, which is essentially a message broker.

Summary​

Refer to Ingestion Pipeline Sample Application for complete code demonstrating usage of the 'ingestion' SDK packages to create these adapters and transformers and run a live pipeline against the Stellar network.