Performing chain upgrades
When consensus-breaking changes are made to the Penumbra protocol, node operators must coordinate upgrading to the new version of the software at the same time. Penumbra uses a governance proposal for scheduling upgrades at a specific block height.
Upgrade process abstractly
At a high level, the upgrade process consists of the following steps:
- Governance proposal submitted, specifying explicit chain height
n
for halt to occur. - Governance proposal passes.
- Chain reaches specified height
n-1
, nodes stop generating blocks. - Manual upgrade is performed on each validator and fullnode:
- Prepare migration directory via
pd export
. - Install the new version of pd.
- Apply changes to node state via
pd migrate
. - Copy a few files and directories around, clean up CometBFT state.
- Restart node.
After the node is restarted on the new version, it should be able to talk to the network again. Once enough validators with sufficient stake weight have upgraded, the network will resume generating blocks.
Genesis time
In order for the chain to start again after the upgrade, all nodes must be using the same genesis information,
including the timestamp for the genesis event. While the pd migrate
command will create a new genesis.json
file,
it cannot know the correct genesis start time to use without the operator supplying the --genesis-start
flag.
The community may choose to specify a start time within the upgrade proposal. If so, all operators must use that value
when performing the migration, as described below. Otherwise, validators must coordinate out of band to agree
on a genesis start time.
Leveraging the governance proposal is the recommended way to solve this problem. If the genesis start time is a value
in the future, then after the upgrade is performed, the node will start, but not process blocks. It will wait
until the --genesis-start
time is reached, at which point it will resume processing blocks. In this way,
the community of validators can coordinate resumption of chain activity, even when operators perform migrate their ndoes
at slightly different times.
Testnet 71 -> 72
For the most recent upgrade on the Penumbra testnet, use this value for genesis time: 2024-04-23T00:00:00.000000000Z
.
See an example below for how to supply this value when performing the migration.
Performing a chain upgrade
The following steps assume that your node uses the default directory of ~/.penumbra/testnet_data/node0/
to store state for both pd
and cometbft
. If your instance is using a different directory, update the paths accordingly.
- Stop both
pd
andcometbft
. Depending on how you run Penumbra, this could meansudo systemctl stop penumbra cometbft
. - Back up the existing node state, as a precaution:
tar -cf ~/.penumbra/testnet_data/node0-state-backup-71.tar ~/.penumbra/testnet_data/node0
- Download the latest version of
pd
and install it. Runpd --version
and confirm you seev0.73.1
before proceeding. - Prepare an export directory:
pd export --home ~/.penumbra/testnet_data/node0/pd --export-directory ~/.penumbra/testnet_data/node0/pd-exported-state-71
- Apply the migration:
pd migrate --genesis-start "2024-04-23T00:00:00.000000000Z" --target-directory ~/.penumbra/testnet_data/node0/pd-exported-state-71/
You must use that precise genesis time, otherwise your node will not be able to reach consensus with the rest of the network. - Move the migrated state into place:
rm -r ~/.penumbra/testnet_data/node0/pd/rocksdb && mv ~/.penumbra/testnet_data/node0/pd-exported-state-71/rocksdb ~/.penumbra/testnet_data/node0/pd/
- Copy the new genesis into place:
cp ~/.penumbra/testnet_data/node0/pd-exported-state-71/genesis.json ~/.penumbra/testnet_data/node0/cometbft/config/genesis.json
- Copy the new signing state into place:
cp ~/.penumbra/testnet_data/node0/pd-exported-state-71/priv_validator_state.json ~/.penumbra/testnet_data/node0/cometbft/data/priv_validator_state.json
- Clean up the old CometBFT state:
find ~/.penumbra/testnet_data/node0/cometbft/data/ -mindepth 1 -maxdepth 1 -type d -and -not -name tx_index.db -exec rm -r {} +
Finally, restart the node, e.g. sudo systemctl restart penumbra cometbft
. Check the logs, and you should see the chain progressing
past the halt height n
.
If you want to host a snapshot for this migration, copy the file
~/.penumbra/testnet_data/node0/pd-migrated-state-v0.73.1.tar.gz
to the appropriate hosting environment,
and inform the users of your validator.