How should we set up testnet infrastructure?

[theoreticalbts]

This isn't exactly a tinman issue, it's more of a discussion of the specific deployment practice that should be used for a particular testnet. But it's related and there's no better place to put this, so I'm putting it here.

A testnet may have nodes with the following functions:

• (a) A fastgen node to handle state initialization by producing blocks with timestamps in the past.
• (b) One or more witness nodes to produce blocks.
• (c) One or more seed nodes which are the default connection points for new p2p peers.
• (d) If API's are desired, one or more API nodes with the relevant plugin(s) enabled.
• (e) If the testnet is to be long-lived, it should have several nodes for reliability and memory [1].
• (f) If transactions are to be created over time while the testnet is live, a node with the private keys of the relevant testnet account(s), which runs a script that generates transactions, signs them with the private key, and submits them to a testnet API endpoint. TN: Gatling - Support price feeds #13
• (g) If transactions are to be "ported" over time from the mainnet while the testnet is live, a node with the private keys of the relevant testnet account(s), which runs a script that queries the mainnet API, generates transactions based on the results of those queries, signs them with the private key, and submits them to a testnet API endpoint. Populate transactions #2
• (h) The parts of Steemit's stack built on the blockchain -- condenser, jussi, yo, hivemind, etc.
• (i) Third-party services, such as steemd.com, busy, etc.

In particular testnets, some of these may be omitted, or mutiple roles may be combined on a single machine.

For the currently existing testnet (pythagoras) and the previous testnet (halloween), I manually did the following steps:

• Create 6 AWS instances using the dev AWS account
• Compile the develop branch of steemd and make install somewhere in /opt on one node
• tar | xz the resulting installation
• nc the tarball to the other nodes
• Run sha256sum on each of the 6 instances and manually compare the output value to ensure the tarball transferred correctly
• Untar the tarball
• Use nano to edit the config file for each node, each node's config file will be slightly different
• All 6 nodes have the 5 IP addresses of the 5 other nodes listed as seed nodes
• Each witness node's config file has 1/3 of the witnesses enabled with corresponding private keys
• Each seed node will do double duty as API node, so have some plugins and API's enabled on seed nodes
• Connect my dev machine as a temporary fastgen node with debug_node_plugin enabled, run tinman submit as specified in README.md
• Once all the actions occur, the witness nodes should spontaneously start producing blocks
• Once this happens, Ctrl+C to quit the fastgen node

This handles (a)-(e) although I'm sure the AWS / Docker experts reading this are cringing about how simple and old-fashioned my sysadmin techniques are. I didn't address (f)-(i). A node of type (f) will need to be created for price feeds #13, and type (g) will be needed for porting transactions #2.

Note that (f) and (g) don't need to run steemd at all, they just run Python scripts and talk to the API, so the smallest possible instance size will probably be fine for these nodes. @mvandeberg is of the opinion that every one of (a)-(i) should be crammed into a single machine, I say the 6 nodes I created (3 witness nodes and 3 seed / API nodes) is the minimum amount we should have for a long-lived, publically accessible, type (b) testnet.

I only have the most hazy idea of what's required for (h) and (i).

[1] Currently steemd does not save blocks after last_irreversible_block on disk. So for example, if you have a single witness, then shut it down, upgrade and restart it, it may start producing a fork! This may be avoided by splitting witness duty among at least 3 nodes, at most one of which is down at a time, allowing a --required-participation=60 flag to forbid producing a minority fork before it's caught up to the present.

[mvandeberg]

I think it makes the most sense to have 3 types of nodes for the testnet.

1. master seed node, also called the fastgen node. This node can handle porting transactions and all "special" test net functions.
2. slave seed node, a normal consensus only seed node.
3. steemd full node, a full node for powering the steemit stack.

For the time being, these nodes will need to be differentiated by build targets/config files/bash scripts handled through docker. Eventually, when the modular blockchain begins to take shape I want to eliminate the idea of a low memory node as a build option and have it be handled entirely via plugins. This will reduce the complexity of our docker image significantly.

[mvandeberg]

The reason I believe the scripts that handle porting transactions should live on the same machine as the master node is that it is easiest to do via the cli wallet (which is in the docker image) and is one less IP/domain that we need to track. It can interact safely through api.steemit.com and localhost. We can then put a domain in front of the master/slave seed node cluster.

The master node does not need to list seed nodes but can self reference seed.testnet.steemit.com. We will have second ELB group that is also behind seed.testnet.steemit.com that we can bring up when the master node is "live" and producing testnet blocks. Once this boost strapping is done we will have 4-6 seed nodes all behind seed.testnet.steemit.com.

[theoreticalbts]

master seed node, also called the fastgen node. This node can handle porting transactions and all "special" test net functions

The fastgen node uses the debug plugin and is temporary. The included transactions generated start out with witnesses controlled by the fastgen node, which creates blocks in the past. At some point the fastgen node submits transactions which change the block signing keys of witnesses away from the fastgen-controlled keys. If you have a normal node connected with the appropriate witnesses enabled, it will pick up block production duties, allowing the fastgen node to be deleted.

Porting transactions is a client function. It doesn't require running a node at all, it will simply be a script that connects to mainnet and testnet API providing nodes.

it is easiest to do via the cli wallet (which is in the docker image) and is one less IP/domain that we need to track

Not really. There are two reasons for this:

• The cli_wallet doesn't have a wrapper for every operation, especially the new operations involving SMT's.
• The cli_wallet has a longstanding issue with hierarchical threshold multisig accounts.

The easiest thing to do is construct the transactions as JSON, have the transaction construction script figure out who should sign and what their key is, and then submit the transaction to a simple C++ utility, sign_transaction, that inputs JSON-format transaction+key on stdin and outputs signatures on stdout.

The hierarchical threshold multisig issue is relevant because currently the generated testnet transactions create an account called porter and add it to the authority of every testnet account imported from the snapshot. Thus, you need a wallet that can properly handle hierarchical threshold multisig. You may have a bad time using the cli_wallet, but the tooling provided by tinman makes it easy: All the porting script needs to do is include a little extra JSON to indicate that each ported transaction needs to be signed by porter's active key, and tinman keysub / tinman submit will take care of the actual signing.