Troubleshooting

Systematic troubleshooting resolves most node issues. Check logs first, verify network connectivity second, review resource usage third.

Initial Diagnostics

# Service status
sudo systemctl status selendra
sudo journalctl -u selendra -n 100

# Network health
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method":"system_health"}' \
  http://localhost:9933

# System resources
top  # CPU/memory
df -h  # Disk space
iostat -x 1 5  # I/O performance

Node Won't Start

Sync Problems

Stuck Syncing

Symptoms: Block height not increasing. Check sync status:

curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method":"system_syncState"}' \
  http://localhost:9933

Common causes and solutions:

Low peer count (< 10):

# Check firewall
sudo ufw allow 30333/tcp
sudo ufw allow 30333/udp

Slow storage (< 100 MB/s):

# Test I/O performance
sudo dd if=/dev/zero of=/var/lib/selendra/test bs=1M count=1024

Solution: Upgrade to NVMe SSD.

Corrupted sync state:

sudo systemctl stop selendra
sudo rm -rf /var/lib/selendra/chains/selendra/db
sudo systemctl start selendra

Slow Sync Speed

Expected: >10 blocks/second. If slower, optimize:

# In systemd service, increase cache
--state-cache-size 8
--db-cache 4096

Sync Restarting

Cause: Database corruption from unclean shutdown or hardware issues.

Solutions:

• Always use sudo systemctl stop selendra (never kill -9)
• Check filesystem: sudo e2fsck -f /dev/nvme0n1p1
• Test RAM: Install memtest86+ and run from GRUB

Connection Issues

No Peers (< 5 connections)

Diagnostics:

# Check ports
sudo ss -tulpn | grep selendra

# Test external access
curl -v telnet://PUBLIC_IP:30333

Solutions:

• Firewall: sudo ufw allow 30333/tcp; sudo ufw allow 30333/udp
• Router NAT: Forward port 30333 to node IP
• ISP blocking: Contact ISP or use VPN for P2P traffic
• Bootnode issues: Add custom bootnodes with --bootnodes /ip4/IP/tcp/30333/p2p/PEER_ID

Peer Count Drops Randomly

Causes:

• Network instability: Test with ping -c 100 8.8.8.8 | grep loss (>5% loss problematic)
• Resource exhaustion: Check file descriptors with lsof -p $(pidof selendra) | wc -l

Solution: Increase limits in systemd service:

[Service]
LimitNOFILE=65536

Validator Issues

Not Producing Blocks

Check session keys registered:

curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method":"author_hasSessionKeys", "params":["SESSION_KEYS_HEX"]}' \
  http://localhost:9933

Should return true. Verify on-chain at portal.selendra.org → Network → Staking → Accounts.

Common causes:

• Keys not set: Generate with author_rotateKeys and register via portal
• Time desync: Check with timedatectl, sync with sudo ntpdate -s time.google.com
• High latency/CPU: Reduce system load or upgrade hardware

Session Keys Lost

After restart showing "Session keys mismatch":

sudo systemctl stop selendra
sudo rm -rf /var/lib/selendra/chains/selendra/keystore
sudo systemctl start selendra

# Generate new keys
curl -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method":"author_rotateKeys"}' \
  http://localhost:9933

# Update on-chain via portal

Getting Slashed

Check status: Portal → Network → Staking → Accounts shows "slashed" indicator.

Causes on Selendra:

• Extended downtime (multiple days for non-reserved validators)
• Equivocation (running two nodes with same keys - NEVER do this)

Prevention:

• Monitor uptime continuously
• Backup keys before maintenance
• Never run duplicate validators
• Test recovery procedures

Performance Issues

Temporary swap (if OOM):

sudo fallocate -l 8G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

RPC Issues

RPC Not Responding

Enable RPC in systemd service:

--rpc-port 9933
--rpc-external  # For remote access
--rpc-cors all

Open firewall: sudo ufw allow 9933/tcp

RPC Timeouts

Solutions:

• Increase timeout: --rpc-timeout 60
• Rate limit clients or reduce query complexity
• Use dedicated RPC node (not validator) for heavy loads

Emergency Procedures

Validator Offline

Immediate actions:

1. Restart service: sudo systemctl restart selendra
2. Announce in Telegram: "Validator [name] investigating issues"
3. Assess downtime:
   • <1 hour: Monitor and document
   • 1-4 hours: Coordinate with other validators

   • 4 hours: Consider failover (slash risk after multiple days)

Database Corruption Recovery

Rapid recovery (4-8 hours to sync):

sudo systemctl stop selendra
sudo rm -rf /var/lib/selendra/chains/selendra/db

# Restore keys if needed
sudo cp -r ~/keystore-backup /var/lib/selendra/chains/selendra/keystore

# Restart and monitor
sudo systemctl start selendra
sudo journalctl -u selendra -f | grep "bps"

Hardware Failure

Quick provisioning:

1. Deploy cloud server (fastest option)
2. Install Selendra binary
3. Restore validator keys from secure backup
4. Wait for sync (~8 hours)
5. Verify operation before announcing recovery

Prevention: Maintain cold standby backup validator.

Getting Help

Information to Include

# System info
uname -a; free -h; df -h

# Selendra version
selendra --version

# Recent logs
sudo journalctl -u selendra -n 200 > logs.txt

# Network status
curl -s -H "Content-Type: application/json" \
  -d '{"id":1, "jsonrpc":"2.0", "method":"system_health"}' \
  http://localhost:9933

# Configuration
sudo systemctl cat selendra

Support Channels

Validator emergencies: Direct Telegram contact. Multiple team members monitoring 24/7.

Related Documentation

Run Full Node | Run Validator | Monitoring | Maintenance

Community

Telegram: t.me/selendranetwork | X: @selendranetwork | GitHub: github.com/selendra/selendra

Validator emergencies receive priority attention.