Troubleshoot validator problems
Use this guide to solve the most common validator problems.
Validator has 0 voting power
If your validator has 0 voting power, your validator has become auto-unbonded. On the mainnet, validators unbond when they do not vote on
9500 of the last
10000 blocks (
50 of the last
100 blocks on the testnet). Because blocks are proposed every ~5 seconds, a validator that is unresponsive for ~13 hours (~4 minutes on testnet) become unbonded. This problem usually happens when your Terrad process crashes.
To return the voting power back to your validator:
If Terrad is not running, restart it:_1terrad start
Wait for your full node to reach the latest block, and run:_1terrad tx slashing unjail <terra> --chain-id=<chain_id> --from=<from>
<terra>is the address of your validator account.
<name>is the name of the validator account. To find this information, run
terrad keys list.
If you don't wait for Terrad to sync before running
unjail, an error message will inform you that your validator is still jailed.
- Check your validator again to see if your voting power is back:
If your voting power is less than it was previously, you may have been slashed for downtime.
Terrad crashes because of too many open files
The default number of files Linux can open per process is
1024. Terrad is known to open more than this amount, causing the process to crash.
Increase the number of open files allowed by running
ulimit -n 4096.
Restart the process with
If you are using
systemdor another process manager to launch Terrad, you might need to configure them. The following sample
systemdfile fixes the problem:_16# /etc/systemd/system/terrad.service_16[Unit]_16Description=Terra Columbus Node_16After=network.target_16_16[Service]_16Type=simple_16User=ubuntu_16WorkingDirectory=/home/ubuntu_16ExecStart=/home/ubuntu/go/bin/terrad start_16Restart=on-failure_16RestartSec=3_16LimitNOFILE=4096_16_16[Install]_16WantedBy=multi-user.target
Terrad crashes because of memory fragmentation
As described in this issue, huge memory allocation can cause memory fragmentation issue. Temporal solution is just using small wasm cache size like 50~100MB.
_1contract-memory-cache-size = 100
_1write-vm-memory-cache-size = 100
The validator is not active
The validator is jailed. To solve this problem,
unjailthe validator by running:
terrad tx slashing unjail <terra> --chain-id=<chain_id> --from=<from>
The validator is not in the active validator set. Only the top 130 validators are in this set. To fix this problem, increase your total stake to be larger than the 130th validator.