Getting Started

Frankendancer

This guide details building and running the Frankendancer validator which is a hybrid of Firedancer and Agave code running side by side. Frankendancer replaces the Agave networking stack and block production components to perform better while leader. It is not yet possible to run a full Firedancer validator, which is in heavy development.

Hardware Requirements

Because Firedancer currently depends on the Agave validator, the hardware requirements are at least what's recommended for that validator. Firedancer hopes to reduce these over time.

Minimum

12-Core CPU @ >2.5GHz
64GB RAM
512GB SSD

Recommended

32-Core CPU @ >3GHz with AVX512 support
128GB RAM with ECC memory
1TB NVMe SSD with separate disk for OS
1 Gigabit/s Network Bandwidth

Installing

Prerequisites

Firedancer must be built from source and currently only supports building and running on Linux. Firedancer requires a recent Linux kernel, at least v4.18. This corresponds to Ubuntu 20.04, Fedora 29, Debian 11, or RHEL 8.

GCC version 8.5 or higher. Only GCC version 11, 12, and 13 are supported and tested by the Firedancer developers.
rustup
clang, git, and make

NOTE

Firedancer currently builds the Agave validator as a dependency, which requires a full Rust toolchain. Once Firedancer is able to stand alone, this will no longer be required.

Other dependencies of the Firedancer build can be installed with a convenience script. First, clone the source code with:

$ git clone --recurse-submodules https://github.com/firedancer-io/firedancer.git
$ cd firedancer
$ git checkout v0.1 # Latest Frankendancer Testnet release

Then you can run the deps.sh script to install system packages and compile library dependencies. System packages will be installed via. the package manager on your system, while library dependencies will be compiled and output placed under ./opt.

$ ./deps.sh

Releases

Firedancer does not produce pre-built binaries and you must build from source, but Firedancer releases are made available as tags. The following naming convention is used,

main This should not be used. The main branch is bleeding edge and includes all Firedancer development and changes that could break Frankendancer.
v0.xxx.yyyyy Official Frankendancer releases.

The Frankendancer versoning has three components,

Major version is always 0. The first full Firedancer release will be 1.x
Minor version increments by 100 for each new Frankendancer release. The minor version will then increment by 1 for new minor versions within this release.
The patch number encodes the Agave validator version. An Agave version of v1.17.14 is represented as 11714.

================= main branch =================
   \                             \
    \ v0.100.11814                \ v0.200.11901
     \                             \
      \ v0.100.11815                \ v0.201.11902
       \
        \ v0.101.11815

Building

Once dependencies are installed, you can build Firedancer. Because Firedancer depends on the Agave validator, this will also build some Agave components.

$ make -j fdctl solana

You will need around 32GiB of available memory to build Firedancer. If you run out of memory compiling, make can return a variety of errors.

TIP

The Firedancer production validator is built as a single binary fdctl short for Firedancer control. You can start, stop, and monitor the Firedancer instance from this one program. The solana CLI binary can be built with make as well for convenience so you can run RPC commands like solana transfer.

Firedancer automatically detects the hardware it is being built on and enables architecture specific instructions for maximum performance if possible. This means binaries built on one machine may not be able to run on another.

If you wish to target a lower machine architecture you can compile for a specific target by setting the MACHINE environment variable to one of the targets under config/.

$ MACHINE=linux_gcc_x86_64 make -j fdctl solana

The default target is native, and compiled binaries will be placed in ./build/native/gcc/bin.

Running

Configuration

Firedancer has many configuration options which are discussed later. For now, we override only the essential options needed to start the validator on Testnet.

config.toml

toml

user = "firedancer"

[gossip]
    entrypoints = [
      "entrypoint.testnet.solana.com:8001",
      "entrypoint2.testnet.solana.com:8001",
      "entrypoint3.testnet.solana.com:8001",
    ]

[consensus]
    identity_path = "/home/firedancer/validator-keypair.json"
    vote_account_path = "/home/firedancer/vote-keypair.json"

    known_validators = [
        "5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8on",
        "dDzy5SR3AXdYWVqbDEkVFdvSPCtS9ihF5kJkHCtXoFs",
        "Ft5fbkqNa76vnsjYNwjDZUXoTWpP7VYm3mtsaQckQADN",
        "eoKpUABi59aT4rR9HGS3LcMecfut9x7zJyodWWP43YQ",
        "9QxCLckBiJc783jnMvXZubK4wH86Eqqvashtrwvcsgkv",
    ]

[rpc]
    port = 8899
    full_api = true
    private = true

[reporting]
    solana_metrics_config = "host=https://metrics.solana.com:8086,db=tds,u=testnet_write,p=c4fa841aa918bf8274e3e2a44d77568d9861b3ea"

This configuration will cause Firedancer to run as the user firedancer on the local machine. The identity_path and vote_account_path should be Agave style keys, which can be generated with the Solana Labs cli.

This will put the ledger in /home/firedancer/.firedancer/fd1/ledger. To customize this path, refer to the configuration guide.

LEDGER

The Firedancer blockstore in the ledger directory is compatible with the one for the Agave validator, and it is possible to switch between validator clients while keeping the ledger directory in place.

Additionally, this configuration enables the full RPC API at port 8899. Although the port will not be published to other validators in gossip, use a firewall to restrict access to this port for maximum security.

Currently, testnet is the only live cluster that Firedancer can be run against and trying to start against devnet or mainnet-beta entrypoints will result in an error.

The Firedancer client can report diagnostic metrics similar to an Agave client. It is recommended to set the [reporting.solana_metrics_config] in the config file to the appropriate value for the cluster. The options for the different clusters are listed in the default.toml file in the reporting section.

Permissions

There are two users involved in running Firedancer. The user that you launch fdctl with, and the user Firedancer switches to after it has started. The requirements for these users are very different:

The user Firedancer starts as is not specified in configuration, but is simply the user that launches the process. For most commands, including fdctl run and configure it needs to be root or have various capabilities described below to setup kernel bypass networking. It is recommended to simply use the root user when launching.
The user Firedancer switches to after it has booted up and performed privileged initialization. This is given by the user option in your configuration TOML file. Firedancer requires nothing from this user and it should be as minimally permissioned as possible. It should never be root or another superuser, and the user should not be present in the sudoers file or have any other privileges.

Only the fdctl run and monitor commands will switch to the non-privileged user, and other commands will run as the startup user until they complete. Most commands can be started with capabilities rather than as the root user, although this isn't recommended. If you are an advanced operator, you can see which capabilities are required for a command by running it unprivileged:

$ ./build/native/gcc/bin/fdctl run
WARNING run ... process requires capability `CAP_NET_RAW` to call `socket(2)` to bind to a raw socket for use by XDP
WARNING run ... process requires capability `CAP_SYS_ADMIN` to call `bpf(2)` with the `BPF_OBJ_GET` command to initialize XDP
WARNING run ... process requires capability `CAP_SYS_ADMIN` to call `unshare(2)` with `CLONE_NEWUSER` to sandbox the process in a user namespace
WARNING run ... process requires capability `CAP_SETUID` to call `setresuid(2)` to switch uid to the sandbox user
WARNING run ... process requires capability `CAP_SETGID` to call `setresgid(2)` to switch gid to the sandbox user
ERR     insufficient permissions to execute command `run`. It is recommended to start Firedancer as the root user, but you can also start it with the missing capabilities listed above. The program only needs to start with elevated permissions to do privileged operations at boot, and will immediately drop permissions and switch to the user specified in your configuration file once they are complete. Firedancer will not execute outside of the boot process as root, and will refuse to start if it cannot drop privileges. Firedancer needs to be started privileged to configure high performance networking with XDP.

For additional layers of defense against local privilege escalation, it is not suggested to setcap(8) the fdctl binary as this can create a larger attack surface.

Initialization

The validator uses some Linux features that must be enabled and configured before it can be started correctly. It is possible for advanced operators to do this configuration manually, but fdctl provides a command to check and automate this step.

WARNING

Running any fdctl configure command may make permanent changes to your system. You should be careful before running these commands on a production host.

The initialization steps are described in detail later. But plowing ahead at the moment:

$ sudo ./build/native/gcc/bin/fdctl configure init all --config ~/config.toml

You will be told what steps are performed:

NOTICE  hugetlbfs ... unconfigured ... mounts `/mnt/.fd/.huge` and `/mnt/.fd/.gigantic` do not exist
NOTICE  hugetlbfs ... configuring
NOTICE  RUN: `mkdir -p /mnt/.fd/.huge`
NOTICE  RUN: `mount -t hugetlbfs none /mnt/.fd/.huge -o pagesize=2097152,min_size=241172480`
NOTICE  RUN: `mkdir -p /mnt/.fd/.gigantic`
NOTICE  RUN: `mount -t hugetlbfs none /mnt/.fd/.gigantic -o pagesize=1073741824,min_size=27917287424`
NOTICE  sysctl ... already valid
NOTICE  ethtool-channels ... already valid
NOTICE  ethtool-gro ... already valid

It is strongly suggested to run the configure command when the system boots, and it needs to be run each time the system is rebooted.

Running

Finally, we can run Firedancer:

$ sudo ./build/native/gcc/bin/fdctl run --config ~/config.toml

Firedancer logs selected output to stderr and a more detailed log to a local file. Every tile in Firedancer runs in a separate process for security isolation, so you will see a complete process tree get launched.

$ pstree 1741904 -as
systemd --switched-root --system --deserialize 17
  └─sudo ./build/native/gcc/bin/fdctl run --config ~/config.toml
      └─fdctl run --config ~/config.toml
          └─fdctl run --config ~/config.toml
              ├─fdctl run-agave --config-fd 0
              │   └─35*[{fdctl}]
              ├─fdctl run1 net 0 --pipe-fd 7 --config-fd 0
              ├─fdctl run1 quic 0 --pipe-fd 8 --config-fd 0
              ├─fdctl run1 verify 0 --pipe-fd 9 --config-fd 0
              ├─fdctl run1 verify 1 --pipe-fd 10 --config-fd 0
              ├─fdctl run1 verify 2 --pipe-fd 11 --config-fd 0
              ├─fdctl run1 verify 3 --pipe-fd 12 --config-fd 0
              ├─fdctl run1 verify 4 --pipe-fd 13 --config-fd 0
              ├─fdctl run1 pack 0 --pipe-fd 15 --config-fd 0
              ├─fdctl run1 dedup 0 --pipe-fd 14 --config-fd 0
              ├─fdctl run1 shred 0 --pipe-fd 16 --config-fd 0
              ├─fdctl run1 metric 0 --pipe-fd 18 --config-fd 0
              └─fdctl run1 sign 0 --pipe-fd 17 --config-fd 0

If any of the processes dies or is killed it will bring all of the others down with it.

Networking

Firedancer uses AF_XDP, a Linux API for high performance networking. For more background see the kernel documentation.

Although AF_XDP works with any ethernet network interface, results may vary across drivers. Popular well tested drivers include:

ixgbe — Intel X540
i40e — Intel X710 series
ice — Intel E800 series

Firedancer installs an XDP program on the network interface [tiles.net.interface] and lo while it is running. This program redirects traffic on ports that Firedancer is listening on via AF_XDP. Traffic targetting any other applications (e.g. an SSH or HTTP server running on the system) passes through as usual. The XDP program is unloaded when the Firedancer process exits.

AF_XDP requires CAP_SYS_ADMIN and CAP_NET_RAW privileges. This is one of the reasons why Firedancer requires root permissions on Linux.

WARNING

Packets received and sent via AF_XDP will not appear under standard network monitoring tools like tcpdump.

Getting Started ​

Frankendancer ​

Hardware Requirements ​

Installing ​

Prerequisites ​

Releases ​

Building ​

Running ​

Configuration ​

Permissions ​

Initialization ​

Running ​

Networking ​

Getting Started

Frankendancer

Hardware Requirements

Installing

Prerequisites

Releases

Building

Running

Configuration

Permissions

Initialization

Running

Networking