1.x to 2.x Migration Guide

Here you will find the required steps to migrate an existing single-tenant (1.x) Stellar Disbursement Platform application (SDP) to a multi-tenant (2.x+) version.

Why Migrate?

Starting with version 2.x, the SDP provides a multi-tenant architecture, where multiple organizations can manage disbursements under a unified infrastructure while maintaining separate datasets and funds management.

New features and fixes will only be published to the multi-tenant version.

The multi-tenant version also suits single-tenant scenarios, through a simplified configuration that will be described later in this document.

caution

This guide was prepared and tested with 1.1.7 -> 2.1.0 code bases. If you're in a later version, it's possible that the new database migrations cause an error when following these steps.

You have the option of using the 2.1.0 version to perform this migration, and then upgrade to the latest version after the migration is completed.

Preparing for the Migrations

Halt the Single-Tenant Version 🚧

Before proceeding with the migration, ensure that the single-tenant version of the SDP is not running. This is crucial to prevent any data inconsistencies or conflicts during the migration process.

Double-Spending Prevention 🚨

To avoid double-spending, ensure that no payment is in the PENDING state, otherwise this could result in having the same payment submitted independently by both single-tenant and multi-tenant instances. You can do that by checking the payments table for any payment in the PENDING state:

SELECT * FROM public.payments WHERE status = ANY(array['PENDING']::payment_status[]);

Similarly, check the submitter_transactions table for any transaction in the PENDING state:

SELECT * FROM public.submitter_transactions WHERE status = ANY(array['PROCESSING']::transaction_status[]);

If there are any payments in the PENDING state or transactions in the PROCESSING state, you should wait for them to be processed and transitioned to either SUCCESS or FAILED before proceeding with the migration.

Remove Channel Accounts 🧹

In version 2.x, the channel accounts are encrypted using the CHANNEL_ACCOUNT_ENCRYPTION_PASSPHRASE, which is a different env from the one used in the single-tenant version (DISTRIBUTION_SEED). To avoid any issues, let's remove all existing channel accounts in the single-tenant version priot to the migration:

bash

./stellar-disbursement-platform channel-accounts delete --delete-all-accounts

Database Backup & Setup 💾

Backup your single-tenant database before proceeding with the migration. At this point, the single-tenant instance should be halted, and there shouldn't be any PENDING payments nor PROCESSING submitter_transactions.

It's also recommended that you create a new database for the multi-tenant version to avoid any data loss. Use the dump from the single-tenant database to populate the initial state of this multi-tenant one. From this point onwards, anytime we refer to the database, we'll be referring to the new multi-tenant database that was created from a dump of the single-tenant database.

Here's how you can do the backup and restore:

bash

pg_dump --dbname=$singleTenantDB > sdp-singleTenant.sql
createdb $multiTenantDB
psql --dbname=$multiTenantDB < sdp-singleTenant.sql

Changes Explained

Among the changes introduced in the multi-tenant version, the most significant ones are:

Infrastructure: the multi-tenant version includes an event-broker (Kafka) as an additional infrastructure component that is highly-recommended for multi-tenant setups, especially when high throughput is required.
Environment Variables: the multi-tenant version introduces new environment variables to configure the event broker, as well as additional variables relevant for multi-tenancy.
Seggregation of Funds: the multi-tenant version introduces the concept of a distribution account for each tenant, which is used to submit transactions on behalf of the tenant. There's also the HOST distribution account, that's used to fund the tenant distribution accounts and the TSS channel accounts.
CLI Commands: some CLI commands have been revised to be tenant-aware, while others have been included to support new multi-tenant features.
Database Structure: the database structure has been revised to accommodate multi-tenancy and the tables are now distributed across multiple schemas, providing isolation between tenants.

Infrastructure

For the infrastructure setup, SDP Multi-tenant offers flexible operational modes.

Event Broker vs Scheduled Jobs

Administrators can choose between using an event broker for event-driven operations, or scheduled jobs for periodic operations. Event brokers are recommended for multi-tenant setups, as they provide a scalable and reliable way to handle events, while scheduled jobs are recommended for local setups or single-tenant SDPs. Also, event-brokers provide faster communication between services.

Anchor Platform Version

While the single-tenant used stellar/anchor-platform:2.1.3, the multi-tenant version requires stellar/anchor-platform:2.6.0 or later.

Environment Variables

Below are the environment variables that have been added or modified in the multi-tenant version.

General environment variables:

ADMIN_ACCOUNT: The username of the admin account used to authenticate HTTP requests to the Admin server. The Admin-targeted requests should add the "Authorization" header, formatted as Base64-encoded "ADMIN_ACCOUNT:ADMIN_API_KEY".
ADMIN_API_KEY: The api key of the admin accountused to authenticate HTTP requests to the Admin server. The Admin-targeted requests should add the "Authorization" header, formatted as Base64-encoded "ADMIN_ACCOUNT:ADMIN_API_KEY".
ADMIN_PORT: the port of the Admin server used to create and manage tenants. Default is 8003.
INSTANCE_NAME: the name of the SDP instance to be displayed in the stellar.toml file. Example: "SDP Testnet".
SINGLE_TENANT_MODE: When set to "true", it enables the single-tenant mode, which is useful for local development or single-tenant setups. In addition to set it to true, you'll need to configure the default tenant by calling the POST /tenants/default-tenant request.
TENANT_XLM_BOOTSTRAP_AMOUNT: The amount of XLM that the HOST Stellar account will deposit deposited to the tenant distribution account for tenant bootstrap.

Stellar accounts configuration environment variables:

CHANNEL_ACCOUNT_ENCRYPTION_PASSPHRASE: A Stellar-compliant ed25519 private key used to encrypt/decrypt the channel accounts' private keys. When not set, it will default to the value of the 'distribution-seed' option, which was used in the single-tenant version. Attention, when migrating from the single-tenant, setting this config to something different from the old DISTRIBUTION_SEED will prevent the code from being able to decrypt the channel accounts.
DISTRIBUTION_ACCOUNT_ENCRYPTION_PASSPHRASE: A Stellar-compliant ed25519 private key used to encrypt/decrypt the in-memory distribution accounts' private keys.

Event broker configuration environment variables:

BROKER_URLS: A comma-separated list of the message broker URLs.
CONSUMER_GROUP_ID: Specifies a group ID for the broker consumers.
EVENT_BROKER_TYPE: Specifies the type of event broker to be used. Options: "KAFKA", "NONE". Defaults to "Kafka".
KAFKA_SECURITY_PROTOCOL: Defines the security protocol for Kafka. Options: PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, SSL.
KAFKA_SASL_USERNAME: Specifies the Kafka SASL Username, required when the kafka security protocol is set to either SASL_PLAINTEXT or SASL_SSL.
KAFKA_SASL_PASSWORD: Specifies the Kafka SASL Password, required when the kafka security protocol is set to either SASL_PLAINTEXT or SASL_SSL.
KAFKA_SSL_ACCESS_KEY: The Kafka Access Key (keystore) in PEM format, required when the kafka security protocol is set to SSL.
KAFKA_SSL_ACCESS_CERTIFICATE: The Kafka SSL Access Certificate in PEM format that matches with the Kafka Access Key, required when the kafka security protocol is set to SSL.

Scheduler environment variables:

ENABLE_SCHEDULER: Default "false". This enables scheduled jobs that replace the operations of a broker for synchronizing payments between SDP and TSS as well as submitting receiver invitations. It should be set to "true" when the event broker is disabled.
SCHEDULER_PAYMENT_JOB_SECONDS: Interval in seconds for the job that synchronizes payments between SDP and TSS. Minimum is 5s.
SCHEDULER_RECEIVER_INVITATION_JOB_SECONDS: Interval in seconds for the job that submits receiver invitations. Minimum is 5s.

On the Anchor Platform side, we must set the following envs:

SEP10_HOME_DOMAINS="localhost:8000, *.stellar.local:8000": a comma-separated list of home domains used for SEP-10. This should contain the domains of the SDP tenant instances.
SEP10_HOME_DOMAIN="": this should be an empty string for the Multi-tenant version.
SEP10_WEB_AUTH_DOMAIN=<localhost:8080>: the home domain of the anchor platform instance.

Seggregation of Funds

In the multi-tenant version, tenants funds are isolated from each other unless the tenant is created with "distribution_account_type": "DISTRIBUTION_ACCOUNT.STELLAR.ENV". For more information on the distribution account types, please refer to the Tenant Provisioning section.

The channel accounts on the other hand are shared between tenants, and they are created by the HOST distribution account, set in the DISTRIBUTION_SEED environment variable. The channel accounts are encrypted using the CHANNEL_ACCOUNT_ENCRYPTION_PASSPHRASE environment variable, or the DISTRIBUTION_SEED if the former is not set. In the single-tenant version, the channel accounts were encrypted by the DISTRIBUTION_SEED.

CLI Commands

The following CLI commands were updated to become tenant-aware:

Database Migrations & Population

The single-tenant commands for DB migration and pre-population used to be:

bash

./stellar-disbursement-platform db migrate up           # ⚠️ DECOMISSIONED in 2.x!
./stellar-disbursement-platform db auth migrate up      # ⚠️ 2.x REQUIRES A (NEW) TENANT-AWARE FLAG!
./stellar-disbursement-platform db setup-for-network    # ⚠️ 2.x REQUIRES A (NEW) TENANT-AWARE FLAG!

The new multi-tenant commands are:

bash

./stellar-disbursement-platform db admin migrate up
./stellar-disbursement-platform db tss migrate up
./stellar-disbursement-platform db auth migrate up --all
./stellar-disbursement-platform db sdp migrate up --all
./stellar-disbursement-platform db setup-for-network --all

Please notice that some commands require a tenant-aware flag, that can be either:

--all: to execute these migrations in all tenants.
--tenant-id: to specify the tenant ID to execute the migrations.

Channel Accounts

The ensure command was updated from using the --num-channel-accounts-ensure flag to using a positional argument:

bash

./stellar-disbursement-platform channel-accounts ensure --num-channel-accounts-ensure 1 # OLD WAY
./stellar-disbursement-platform channel-accounts ensure 1                               # NEW WAY

Database Structure

In the updated version, the database structure has been revised to accommodate multi-tenancy. As a result, the tables are now distributed across multiple schemas, and new tables have been introduced to support the multi-tenant architecture. The following schemas are used in the multi-tenant version:

admin: it houses tables associated with tenant administration. It serves as the central point for managing tenant-related information and to resolve tenant schema names based on tenant IDs. None of these tables existed in the single-tenant version.
sdp_<tenant_name>: are per-tenant schemas that are prefixed with sdp_. For example, for tenants BlueCorp and RedCorp, the provisioned schemas would be named sdp_bluecorp and sdp_redcorp, respectively. These schemas contain all necessary tables for the SDP operation tailored to each tenant, including per-tenant user authentication.
tss: is a schema dedicated to the Transaction Submitter Service (TSS). The TSS tables do not belong to any tenant, although each TSS transaction contains a column that signals which tenant it belongs to.

These changes allow for the isolation of tenant data per schema, ensuring that each tenant's data is kept separate from other tenants.

Step-by-Step Migration Guide

tip

The project has a script that performs these steps, as well as a CI that ensures these steps are working. You can use them as a reference.

Using the new database created from the single-tenant database dump as a starting point (as described in the Database Backup & Setup section), we can now proceed with the migration steps below.

Deploy the New Version

To transaction to a multi-tenant setup, deploy the latest version of the SDP 2+ version. If you're using helm charts, you can get the helm chart from the SDP Helm Chart repository.

Executing Initial Database Migrations

Following the deployment of the multi-tenant SDP, the next step is to perform the initial database migrations. It does not include the tenant-specific tables yet, they will be created later.

Migration Commands:

bash

./stellar-disbursement-platform db admin migrate up
./stellar-disbursement-platform db tss migrate up

These commands will create the tenant admin tables on the admin schema and the Transaction Submitter Service tables on the tss schema, respectively.

New Tenant Provisioning Process

After successfully applying the database migrations, the next step is to provision a new tenant. This is achieved by making an HTTP request to the Admin API.

Be aware that it will provision the tenants with all the per-tenant migrations included.

Starting the SDP API server

To facilitate tenant provisioning, initiate the SDP Server using the command:

bash

./stellar-disbursement-platform serve

Posting to the Admin API

API port: The Admin API is accessible on port 8003 by default. This port setting can be adjusted by altering the ADMIN_PORT environment variable.
Authentication: Admin API employs Basic Authentication for securing access. To authenticate, populate the request "Authorization" header with "Authorization: Basic ${Base64(ADMIN_ACCOUNT:ADMIN_API_KEY)}".

Here's a shell script that can be used to create a tenant through the Admin API:

bash

ADMIN_ACCOUNT="SDP-admin"
ADMIN_API_KEY="api_key_1234567890"
basicAuthCredentials=$(echo -n "$ADMIN_ACCOUNT:$ADMIN_API_KEY" | base64)
AuthHeader="Authorization: Basic $basicAuthCredentials"

tenant="bluecorp"
baseURL="http://$tenant.stellar.local:8000"
sdpUIBaseURL="http://$tenant.stellar.local:3000"
ownerEmail="owner@$tenant.org"
# NOTE: please refer to the distribution_account_type values in the API reference
distributionAccountType="DISTRIBUTION_ACCOUNT.STELLAR.DB_VAULT"

curl -X POST http://localhost:8003/tenants \
     -H "Content-Type: application/json" \
     -H "$AuthHeader" \
     -d '{
            "name": "'"$tenant"'",
            "organization_name": "Blue Corp",
            "base_url": "'"$baseURL"'",
            "sdp_ui_base_url": "'"$sdpUIBaseURL"'",
            "owner_email": "'"$ownerEmail"'",
            "owner_first_name": "john",
            "owner_last_name": "doe",
            "distribution_account_type": "'"$distributionAccountType"'"
     }'

In the SDP Multi-tenant, certain configurations previously managed through environment variables in the single-tenant setup are now stored within the tenants table in the admin schema. This change allows each tenant to have its own custom configuration.

The field name is important because it's the tenant identifier. The other fields can be set to the same values used on the environment variables used on the non-tenant version.

Importing your data

Now that we've provisioned a new tenant, we can import our data into it. We need to copy our old tables' data to the new tenant schema.

Please notice that the following tables don't need to be copied:

gorp_migrations
auth_migrations
countries
organizations

To help with the import process, we added a function to the admin schema that will copy the data from the v1 tables located in the public schema to the new tenant schema. The function is called import_tenant_data_from_v1_to_v2 and it receives the tenant name as a parameter. Tenant name needs to be the same as the $name used in the tenant creation API call.

    SELECT admin.migrate_tenant_data_from_v1_to_v2('tenant_name')

This concludes the migration of the SDP data to the multi-tenant version. The next step is to drop the old tables that are no longer needed.

Drop Old Tables

Now, the only missing step is to drop the single-tenant tables that should not be in the multi-tenant database:

BEGIN TRANSACTION;

DROP TABLE public.messages CASCADE;
DROP TABLE public.payments CASCADE;
DROP TABLE public.disbursements CASCADE;
DROP TABLE public.receiver_verifications CASCADE;
DROP TABLE public.receiver_wallets CASCADE;
DROP TABLE public.auth_user_password_reset CASCADE;
DROP TABLE public.auth_user_mfa_codes CASCADE;
DROP TABLE public.receivers CASCADE;
DROP TABLE public.auth_users CASCADE;
DROP TABLE public.wallets_assets CASCADE;
DROP TABLE public.assets CASCADE;
DROP TABLE public.wallets CASCADE;
DROP TABLE public.organizations CASCADE;
DROP TABLE public.gorp_migrations CASCADE;
DROP TABLE public.auth_migrations CASCADE;
DROP TABLE public.countries CASCADE;
DROP TABLE public.submitter_transactions CASCADE;
DROP TABLE public.channel_accounts CASCADE;

COMMIT;

Conclusion 🎉

This should conclude the data migration from the single-tenant version to the multi-tenant version of the SDP. Please, make sure to run an e2e test to ensure that everything is working as expected.

Why Migrate?​

Preparing for the Migrations​

Halt the Single-Tenant Version 🚧​

Double-Spending Prevention 🚨​

Remove Channel Accounts 🧹​

Database Backup & Setup 💾​

Changes Explained​

Infrastructure​

Event Broker vs Scheduled Jobs​

Anchor Platform Version​

Environment Variables​

Seggregation of Funds​

CLI Commands​

Database Migrations & Population​

Channel Accounts​

Database Structure​

Step-by-Step Migration Guide​

Deploy the New Version​

Executing Initial Database Migrations​

New Tenant Provisioning Process​

Starting the SDP API server​

Posting to the Admin API​

Importing your data​

Drop Old Tables​

Conclusion 🎉​