fdctl Command Line Interface
The Firedancer binary fdctl contains many subcommands which can be run from the command line.
run
Runs the validator. This command does not exit until the validator does, an error in running the validator will be propagated to the exit code of the process. The command can be run with the capabilities listed below but it is suggested to run it as sudo. The command writes an abbreviated log output to stderr and nothing will be written to stdout.
| Arguments | Description |
|---|---|
--config | Path to a configuation TOML file to run the validator with |
Capabilities
| Capability | Reason |
|---|---|
CAP_NET_RAW | call socket(2) to bind to a raw socket for use by XDP |
CAP_SYS_ADMIN | call bpf(2) with the BPF_OBJ_GET command to initialize XDP |
CAP_SYS_ADMIN | call unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces |
CAP_SETUID | call setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID |
CAP_SETGID | call setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit |
CAP_SYS_RESOURCE | call setpriority(2) to increase thread priorities. Not required if the process already has a nice value of -19 |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_NOFILE` to allow more open files for Agave. Not reqired if the resource limit is already high enough |
CAP_NET_BIND_SERVICE | call bind(2) to bind to a privileged port for serving metrics. Only required if the bind port is below 1024 |
$ fdctl run --config config.toml
Log at "/tmp/fd-0.0.0_755725_fd_fd_2024_07_01_03_24_27_470038344_GMT+00"
NOTICE 07-01 03:24:27.471949 755725 f0 main src/disco/topo/fd_topo.c(445):
SUMMARY
Total Tiles: 16
Total Memory Locked: 28145897472 bytes (26 GiB + 218 MiB + 20 KiB)
Required Gigantic Pages: 26
Required Huge Pages: 109
Required Normal Pages: 36
Required Gigantic Pages (NUMA node 0): 26
Required Huge Pages (NUMA node 0): 109
[...]
NOTICE 07-01 03:24:28.772209 755745 16 metric:0 src/disco/topo/fd_topo_run.c(30): booting tile metric:0 pid:755742 tid:755745
NOTICE 07-01 03:24:28.773617 755747 13 shred:0 src/disco/topo/fd_topo_run.c(30): booting tile shred:0 pid:755740 tid:755747
NOTICE 07-01 03:24:28.773798 755748 9 pack:0 src/disco/topo/fd_topo_run.c(30): booting tile pack:0 pid:755739 tid:755748
[...]monitor
Monitors a validator that is running locally on this machine. This is a low level performance monitor mostly useful for diagnosing throughput issues. The monitor takes over the controlling terminal and refreshes it many times a second with up to date information. You can exit the monitor by sending Ctrl+C or SIGINT.
| Arguments | Description |
|---|---|
--config | Path to a configuation TOML file to run the monitor with. This must be the same configuration file the validator was started with |
Capabilities
| Capability | Reason |
|---|---|
CAP_SYS_ADMIN | call unshare(2) with CLONE_NEWUSER to sandbox the process in a user namespace. Only required on kernels which restrict unprivileged user namespaces |
CAP_SETUID | call setresuid(2) to switch uid to the sandbox user. Not required if the UID is already the same as the sandbox UID |
CAP_SETGID | call setresgid(2) to switch gid to the sandbox user. Not required if the GID is already the same as the sandbox GID |
CAP_SYS_RESOURCE | call rlimit(2) to increase RLIMIT_MEMLOCK so all memory can be locked with mlock(2). Not required if the process already has a high enough limit |
$ fdctl monitor --config config.toml
snapshot for 2024-07-01 04:31:34.669413032 GMT+00
tile | pid | stale | heart | sig | in backp | backp cnt | % hkeep | % backp | % wait | % ovrnp | % ovrnr | % filt1 | % filt2 | % finish
---------+---------+------------+-------+------------+----------+---------------------+----------+----------+----------+----------+----------+----------+----------+----------
net | 763643 | - | - | run( run) | -( -) | 0( +0) | 42.300 | 0.000 | 57.700 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
quic | 763638 | - | - | run( run) | -( -) | 0( +0) | 0.331 | 0.000 | 99.669 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763641 | - | - | run( run) | -( -) | 0( +0) | 0.490 | 0.000 | 99.509 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763639 | - | - | run( run) | -( -) | 0( +0) | 0.488 | 0.000 | 99.417 | 0.000 | 0.000 | 0.000 | 0.000 | 0.095
verify | 763640 | - | - | run( run) | -( -) | 0( +0) | 0.483 | 0.000 | 99.516 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763644 | - | - | run( run) | -( -) | 0( +0) | 0.490 | 0.000 | 99.510 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
verify | 763642 | - | - | run( run) | -( -) | 0( +0) | 0.489 | 0.000 | 99.511 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
dedup | 763634 | - | - | run( run) | -( -) | 0( +0) | 0.049 | 0.000 | 99.951 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
pack | 763635 | - | - | run( run) | -( -) | 0( +0) | 0.060 | 0.000 | 99.937 | 0.000 | 0.000 | 0.000 | 0.000 | 0.003
bank | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.287 | 0.000 | 99.584 | 0.000 | 0.000 | 0.000 | 0.000 | 0.129
bank | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.286 | 0.000 | 99.714 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
poh | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.463 | 0.000 | 99.517 | 0.000 | 0.000 | 0.000 | 0.018 | 0.002
shred | 763637 | - | - | run( run) | -( -) | 0( +0) | 0.404 | 0.000 | 99.519 | 0.000 | 0.000 | 0.000 | 0.000 | 0.077
store | 763629 | - | - | run( run) | -( -) | 0( +0) | 0.293 | 0.000 | 99.447 | 0.000 | 0.000 | 0.000 | 0.000 | 0.260
sign | 763636 | - | - | run( run) | -( -) | 0( +0) | 0.239 | 0.000 | 99.729 | 0.000 | 0.000 | 0.000 | 0.000 | 0.032
metric | 763630 | - | - | run( run) | -( -) | 0( +0) | 100.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
link | tot TPS | tot bps | uniq TPS | uniq bps | ha tr% | uniq bw% | filt tr% | filt bw% | ovrnp cnt | ovrnr cnt | slow cnt | tx seq
------------------+----------+----------+----------+----------+----------+----------+----------+----------+---------------------+---------------------+---------------------+-------------------
quic->net | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
shred->net | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
net->quic | 15.4 | 44.4K | 15.4 | 44.4K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
sign->quic | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 15.4 | 39.3K | 15.4 | 39.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
quic->verify | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 16( +0)
verify->dedup | 15.4 | 43.3K | 15.4 | 43.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 16( +1)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0)
verify->dedup | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 15( +0)
dedup->pack | 15.4 | 43.3K | 15.4 | 43.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 77( +1)
poh->pack | 15.4 | 39.3K | 15.4 | 39.3K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
poh->pack | 15.4 | 5907.3 | 15.4 | 5907.3 | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
pack->bank | 30.8 | 519.8K | 30.8 | 519.8K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 306( +3)
pack->bank | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 306( +3)
bank->poh | 30.8 | 529.7K | 30.8 | 529.7K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 229( +2)
bank->poh | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 0( +0)
poh->poh | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
pack->poh | 46.2 | 519.8K | 46.2 | 519.8K | 100.000 | 100.000 | 100.000 | 100.000 | 0( +0) | 0( +0) | 0( +0) | 306( +3)
net->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
poh->shred | 969.2 | 597.5K | 969.2 | 597.5K | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 4930( +63)
poh->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
poh->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
sign->shred | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
shred->store | 30.8 | 10.3M | 30.8 | 10.3M | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 1( +0) | 154( +2)
quic->sign | 0.0 | 0.0 | 0.0 | 0.0 | 0.000 | 0.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)
shred->sign | 15.4 | 3938.2 | 15.4 | 3938.2 | 100.000 | 100.000 | 0.000 | 0.000 | 0( +0) | 0( +0) | 0( +0) | 0( +0)configure
Configures the operating system so that it can run Firedancer. See the guide for more information. There are three possible stages to each configure command:
hugetlbfsReserves huge and gigantic pages for use by Firedancer and mounts huge page filesystems for then under a path in the configuration TOML file.sysctlSet required kernel parameters.ethtool-channelsConfigures the number of channels on the network device.ethtool-groDisables generic receive offload (GRO) on the network device.
toml
[hugetlbfs]
mount_path = "/mnt/.fd"
[layout]
net_tile_count = 2
[tiles]
[net]
interface = "ens3f0"configure init <stage>...
Prepare the operating system environment to run Firedancer. This will reserve and mount the huge page filesystems, set the kernel parameters, and configure the number of combined channels on the network device.
Capabilities
| Capability | Reason |
|---|---|
root | increase /proc/sys/vm/nr_hugepages and mount hugetblfs filesystems. Only applies for the hugetlbfs stage |
root | increase network device channels with ethtool --set-channels. Only applies for the ethtool-channels stage |
root | disable network device generic-receive-offload (gro) with ethtool --set-offload generic-receive-offload off. Only applies for the ethtool-gro stage |
CAP_SYS_ADMIN | set kernel parameters in /proc/sys. Only applies for the sysctl stage |
$ fdctl configure init all
NOTICE hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
NOTICE hugetlbfs ... configuring
NOTICE RUN: `mkdir -p /mnt/.fd/.huge`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=241172480`
NOTICE RUN: `mkdir -p /mnt/.fd/.gigantic`
NOTICE RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=27917287424`
NOTICE sysctl ... unconfigured ... kernel parameter `/proc/sys/vm/max_map_count` is too low (got 65536 but expected at least 1000000)
NOTICE RUN: `echo "1000000" > /proc/sys/vm/max_map_count`
NOTICE RUN: `echo "1024000" > /proc/sys/fs/file-max`
NOTICE RUN: `echo "1024000" > /proc/sys/fs/nr_open`
NOTICE RUN: `echo "2" > /proc/sys/net/ipv4/conf/lo/rp_filter`
NOTICE RUN: `echo "1" > /proc/sys/net/ipv4/conf/lo/accept_local`
WARNING kernel parameter `/proc/sys/net/core/bpf_jit_enable` is too low (got 0 but expected at least 1). Proceeding but performance may be reduced.
WARNING kernel parameter `/proc/sys/kernel/numa_balancing` is set to 1, not the expected value of 0. Proceeding but performance may be reduced.
NOTICE ethtool-channels ... unconfigured ... device `ens3f0` does not have right number of channels (got 1 but expected 2)
NOTICE ethtool-channels ... configuring
NOTICE ethtool-channels ... RUN: `ethtool --set-channels ens3f0 combined 2`
NOTICE ethtool-gro ... unconfigured ... device `ens3f0` has generic-receive-offload enabled. Should be disabled
NOTICE ethtool-gro ... configuring
NOTICE ethtool-gro ... RUN: `ethtool --offload ens3f0 generic-receive-offload off`configure check <stage>...
Check if the operating system environment is properly configured. Exits with a non-zero exit code if it is not, after printing relevant diagnostics to stderr.
$ fdctl configure check all
WARNING hugetlbfs ... not configured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
WARNING sysctl ... not configured ... kernel parameter `/proc/sys/vm/max_map_count` is too low (got 65536 but expected at least 1000000)
WARNING ethtool-channels ... not configured ... device `ens3f0` does not have right number of channels (got 1 but expected 2)
WARNING ethtool-gro ... not configured ... device `ens3f0` has generic-receive-offload enabled. Should be disabled
ERR failed to configure some stagesconfigure fini <stage>...
Remove any Firedancer specific operating system configuration still lingering. This only unmounts the hugetlbfs stages and returns the reserved huge and gigantic pages to the kernel pool. It will not reduce sysctls that were earlier increased, or change the network channel count back as we no longer know what the original value was.
Capabilities
| Capability | Reason |
|---|---|
root | remove directories from /mnt, unmount hugetlbfs. Only applies for the hugetlbfs stage |
$ fdctl configure fini all
NOTICE ethtool ... finishing
NOTICE sysctl ... finishing
NOTICE hugetlbfs ... finishing
NOTICE RUN: `umount /mnt/.fd/.huge`
NOTICE RUN: `rmdir /mnt/.fd/.huge`
NOTICE RUN: `umount /mnt/.fd/.gigantic`
NOTICE RUN: `rmdir /mnt/.fd/.gigantic`
NOTICE RUN: `rmdir /mnt/.fd/.normal`
NOTICE RUN: `rmdir /mnt/.fd`version
Prints the current version of the validator to the standard output and exits. The command writes diagnostic messages from logs to stderr.
sh
$ fdctl version
0.101.11814keys
keys pubkey <PATH>
Prints the base58 encoding of the public key in the file at <PATH> to the standard output and exits. The file at <PATH> should be an Agave style identity.json key file. The command writes diagnostic messages from logs to stderr.
sh
$ fdctl keys pubkey ~/.firedancer/fd1/identity.json
Fe4StcZSQ228dKK2hni7aCP7ZprNhj8QKWzFe5usGFYFkeys new <identity|vote>
Creates a new keypair from the kernel random number generator and writes it to the identity key path, or vote key path. The key path is retrieved from the configuration TOML file
| Arguments | Description |
|---|---|
--config | Path to a configuation TOML file which determines where the key is written. Either [consensus.identity_path] or [consensus.vote_account_path] for identity or vote arguments respectively |
toml
[consensus]
identity_path = "/home/{user}/.fd/keys/identity.json"$ fdctl keys new vote --config config.toml
NOTICE successfully created keypair in `/home/fd/.fd/keys/identity.json`mem
Prints information about the memory requirements and the tile configuration and layout of the validator to stdout before exiting. The command writes diagnostic messsages from logs to stderr.
Firedancer preallocates and locks all memory it needs from huge and gigantic page mounts before booting, and the hugetlbfs stage of fdctl configure will reserve the memory described here for exclusive use by Firedancer.
| Arguments | Description |
|---|---|
--config | Path to a configuation TOML file to print memory usage information with |
sh
$ fdctl mem --config config.toml
SUMMARY
Total Tiles: 17
Total Memory Locked: 27088932864 bytes (25 GiB + 234 MiB + 20 KiB)
Required Gigantic Pages: 25
Required Huge Pages: 117
Required Normal Pages: 27
Required Gigantic Pages (NUMA node 0): 25
Required Huge Pages (NUMA node 0): 117
WORKSPACES
0 ( 1 GiB): net_quic page_cnt=1 page_sz=gigantic numa_idx=0 footprint=68173824 loose=1005563904
1 ( 1 GiB): net_shred page_cnt=1 page_sz=gigantic numa_idx=0 footprint=68173824 loose=1005563904
[...]
LINKS
0 ( 32 MiB): net_quic kind_id=0 wksp_id=0 depth=16384 mtu=2048 burst=1
1 ( 32 MiB): net_shred kind_id=0 wksp_id=1 depth=16384 mtu=2048 burst=1
[...]
TILES
0 ( 3 GiB): net kind_id=0 wksp_id=18 cpu_idx=1 out_link=-1 in=[-2, -3] out=[ 0, 1]
1 ( 3 GiB): quic kind_id=0 wksp_id=19 cpu_idx=2 out_link=4 in=[ 0, -21] out=[ 2, 20]
[...]