Metrics
Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via. a Prometheus HTTP endpoint:
toml
[tiles.metric]
prometheus_listen_port = 7999sh
$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...WARNING
Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.
There are three metric types reported by Firedancer, following the Prometheus data model:
counter— A cumulative metric repesenting a monotonically increasing counter.gauge— A single numerical value that can go arbitrarily up or down.histogram— Samples observations like packet sizes and counts them in buckets.
All Links
There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).
sh
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0These link related metrics have four labels, which are to identify the link the metric is for:
kind— The name of the tile consuming from the link.kind_id— The tile index of the tile which is consuming from the link.link_kind— The name of the link being consumed.link_kind_id— The link index of the link which is being consumed.
| Metric | Type | Description |
|---|---|---|
| link_slow_count | counter | The number of times the consumer was detected as rate limiting consumer by the producer. |
| link_consumed_count | counter | The number of times the link reader has consumed a fragment. |
| link_consumed_size_bytes | counter | The total number of bytes read by the link consumer. |
| link_filtered_count | counter | The number of fragments that were filtered and not consumed. |
| link_filtered_size_bytes | counter | The total number of bytes read by the link consumer that were filtered. |
| link_overrun_polling_count | counter | The number of times the link has been overrun while polling. |
| link_overrun_polling_frag_count | counter | The number of fragments the link has not processed because it was overrun while polling. |
| link_overrun_reading_count | counter | The number of input overruns detected while reading metadata by the consumer. |
| link_overrun_reading_frag_count | counter | The number of fragments the link has not processed because it was overrun while reading. |
All Tiles
Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:
sh
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268
# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:
kind— The tile name the metric is being reported for.kind_id— The tile index of the tile which is reporting the metric.
| Metric | Type | Description |
|---|---|---|
| tile_pid | gauge | The process ID of the tile. |
| tile_tid | gauge | The thread ID of the tile. Always the same as the Pid in production, but might be different in development. |
| tile_context_switch_involuntary_count | counter | The number of involuntary context switches. |
| tile_context_switch_voluntary_count | counter | The number of voluntary context switches. |
| tile_status | gauge | The current status of the tile. 0 is booting, 1 is running. |
| tile_heartbeat | gauge | The last UNIX timestamp in nanoseconds that the tile heartbeated. |
| tile_in_backpressure | gauge | Whether the tile is currently backpressured or not, either 1 or 0. |
| tile_backpressure_count | counter | Number of times the times the tile has had to wait for one of more consumers to catch up to resume publishing. |
| tile_regime_duration_nanos_caught_up_housekeeping | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Housekeeping) |
| tile_regime_duration_nanos_processing_housekeeping | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Housekeeping) |
| tile_regime_duration_nanos_backpressure_housekeeping | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Backpressure + Housekeeping) |
| tile_regime_duration_nanos_caught_up_prefrag | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Prefrag) |
| tile_regime_duration_nanos_processing_prefrag | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Prefrag) |
| tile_regime_duration_nanos_backpressure_prefrag | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Backpressure + Prefrag) |
| tile_regime_duration_nanos_caught_up_postfrag | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Caught up + Postfrag) |
| tile_regime_duration_nanos_processing_postfrag | counter | Mutually exclusive and exhaustive duration of time the tile spent in each of the regimes. (Processing + Postfrag) |
Net Tile
| Metric | Type | Description |
|---|---|---|
| net_received_packets | counter | Number of IP packets received. |
| net_received_bytes | counter | Total bytes received (including IP, UDP headers). |
| net_sent_packets | counter | Number of IP packets sent. |
| net_sent_bytes | counter | Total bytes sent (including IP, UDP headers). |
| net_xdp_rx_dropped_ring_full | counter | Number of packets dropped because the RX completion queue was empty. This is only reported for net tile 0, since the measurement is across all RX queues. |
| net_xdp_rx_dropped_other | counter | Number of packets dropped for other reasons. This is only reported for net tile 0, since the measurement is across all RX queues. |
| net_tx_dropped | counter | Number of packets dropped because the TX submission queue was empty. This is reported for all net tiles. |
Quic Tile
| Metric | Type | Description |
|---|---|---|
| quic_txns_overrun | counter | Count of txns overrun before reassembled (too small txn_reassembly_count). |
| quic_txn_reasms_started | counter | Count of fragmented txn receive ops started. |
| quic_txn_reasms_active | gauge | Number of fragmented txn receive ops currently active. |
| quic_frags_ok | counter | Count of txn frags received |
| quic_frags_gap | counter | Count of txn frags dropped due to data gap |
| quic_frags_dup | counter | Count of txn frags dropped due to dup (stream already completed) |
| quic_txns_received_udp | counter | Count of txns received via TPU. (TPU/UDP) |
| quic_txns_received_quic_fast | counter | Count of txns received via TPU. (TPU/QUIC unfragmented) |
| quic_txns_received_quic_frag | counter | Count of txns received via TPU. (TPU/QUIC fragmented) |
| quic_txns_abandoned | counter | Count of txns abandoned because a conn was lost. |
| quic_quic_packet_too_small | counter | Count of packets received on the QUIC port that were too small to be a valid IP packet. |
| quic_quic_txn_too_small | counter | Count of txns received via QUIC dropped because they were too small. |
| quic_quic_txn_too_large | counter | Count of txns received via QUIC dropped because they were too large. |
| quic_non_quic_packet_too_small | counter | Count of packets received on the non-QUIC port that were too small to be a valid IP packet. |
| quic_non_quic_packet_too_large | counter | Count of packets received on the non-QUIC port that were too large to be a valid transaction. |
| quic_received_packets | counter | Number of IP packets received. |
| quic_received_bytes | counter | Total bytes received (including IP, UDP, QUIC headers). |
| quic_sent_packets | counter | Number of IP packets sent. |
| quic_sent_bytes | counter | Total bytes sent (including IP, UDP, QUIC headers). |
| quic_connections_active | gauge | The number of currently active QUIC connections. |
| quic_connections_created | counter | The total number of connections that have been created. |
| quic_connections_closed | counter | Number of connections gracefully closed. |
| quic_connections_aborted | counter | Number of connections aborted. |
| quic_connections_timed_out | counter | Number of connections timed out. |
| quic_connections_retried | counter | Number of connections established with retry. |
| quic_connection_error_no_slots | counter | Number of connections that failed to create due to lack of slots. |
| quic_connection_error_retry_fail | counter | Number of connections that failed during retry (e.g. invalid token). |
| quic_pkt_crypto_failed | counter | Number of packets that failed decryption. |
| quic_pkt_no_conn | counter | Number of packets with an unknown connection ID. |
| quic_pkt_tx_alloc_fail | counter | Number of packets failed to send because of metadata alloc fail. |
| quic_handshakes_created | counter | Number of handshake flows created. |
| quic_handshake_error_alloc_fail | counter | Number of handshakes dropped due to alloc fail. |
| quic_stream_received_events | counter | Number of stream RX events. |
| quic_stream_received_bytes | counter | Total stream payload bytes received. |
| quic_received_frames_unknown | counter | Number of QUIC frames received. (Unknown frame type) |
| quic_received_frames_ack | counter | Number of QUIC frames received. (ACK frame) |
| quic_received_frames_reset_stream | counter | Number of QUIC frames received. (RESET_STREAM frame) |
| quic_received_frames_stop_sending | counter | Number of QUIC frames received. (STOP_SENDING frame) |
| quic_received_frames_crypto | counter | Number of QUIC frames received. (CRYPTO frame) |
| quic_received_frames_new_token | counter | Number of QUIC frames received. (NEW_TOKEN frame) |
| quic_received_frames_stream | counter | Number of QUIC frames received. (STREAM frame) |
| quic_received_frames_max_data | counter | Number of QUIC frames received. (MAX_DATA frame) |
| quic_received_frames_max_stream_data | counter | Number of QUIC frames received. (MAX_STREAM_DATA frame) |
| quic_received_frames_max_streams | counter | Number of QUIC frames received. (MAX_STREAMS frame) |
| quic_received_frames_data_blocked | counter | Number of QUIC frames received. (DATA_BLOCKED frame) |
| quic_received_frames_stream_data_blocked | counter | Number of QUIC frames received. (STREAM_DATA_BLOCKED frame) |
| quic_received_frames_streams_blocked | counter | Number of QUIC frames received. (STREAMS_BLOCKED(bidi) frame) |
| quic_received_frames_new_conn_id | counter | Number of QUIC frames received. (NEW_CONN_ID frame) |
| quic_received_frames_retire_conn_id | counter | Number of QUIC frames received. (RETIRE_CONN_ID frame) |
| quic_received_frames_path_challenge | counter | Number of QUIC frames received. (PATH_CHALLENGE frame) |
| quic_received_frames_path_response | counter | Number of QUIC frames received. (PATH_RESPONSE frame) |
| quic_received_frames_conn_close_quic | counter | Number of QUIC frames received. (CONN_CLOSE(transport) frame) |
| quic_received_frames_conn_close_app | counter | Number of QUIC frames received. (CONN_CLOSE(app) frame) |
| quic_received_frames_handshake_done | counter | Number of QUIC frames received. (HANDSHAKE_DONE frame) |
| quic_received_frames_ping | counter | Number of QUIC frames received. (PING frame) |
| quic_received_frames_padding | counter | Number of QUIC frames received. (PADDING frame) |
| quic_ack_tx_noop | counter | ACK events (non-ACK-eliciting packet) |
| quic_ack_tx_new | counter | ACK events (new ACK range) |
| quic_ack_tx_merged | counter | ACK events (merged into existing ACK range) |
| quic_ack_tx_drop | counter | ACK events (out of buffers) |
| quic_ack_tx_cancel | counter | ACK events (ACK suppressed by handler) |
| quic_service_duration_seconds | histogram | Duration spent in service |
| quic_receive_duration_seconds | histogram | Duration spent receiving packets |
| quic_frame_fail_parse | counter | Number of QUIC frames failed to parse. |
Verify Tile
| Metric | Type | Description |
|---|---|---|
| verify_transaction_parse_failure | counter | Count of transactions that failed to parse |
| verify_transaction_dedup_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
| verify_transaction_verify_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
Dedup Tile
| Metric | Type | Description |
|---|---|---|
| dedup_transaction_dedup_failure | counter | Count of transactions that failed to deduplicate in the dedup stage |
| dedup_gossiped_votes_received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path |
Resolv Tile
| Metric | Type | Description |
|---|---|---|
| resolv_no_bank_drop | counter | Count of transactions dropped because the bank was not available |
| resolv_lut_resolved_invalid_lookup_index | counter | Count of address lookup tables resolved (The transaction referenced an index in a LUT that didn't exist) |
| resolv_lut_resolved_account_uninitialized | counter | Count of address lookup tables resolved (The account referenced as a LUT hasn't been initialized) |
| resolv_lut_resolved_invalid_account_data | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be parsed) |
| resolv_lut_resolved_invalid_account_owner | counter | Count of address lookup tables resolved (The account referenced as a LUT wasn't owned by the ALUT program ID) |
| resolv_lut_resolved_account_not_found | counter | Count of address lookup tables resolved (The account referenced as a LUT couldn't be found) |
| resolv_lut_resolved_success | counter | Count of address lookup tables resolved (Resolved successfully) |
| resolv_blockhash_expired | counter | Count of transactions that failed to resolve because the blockhash was expired |
| resolv_blockhash_unknown | counter | Count of transactions with an unknown blockhash. These may be very recent, very old, nonces, or bogus. |
Pack Tile
| Metric | Type | Description |
|---|---|---|
| pack_schedule_microblock_duration_seconds | histogram | Duration of scheduling one microblock |
| pack_no_sched_microblock_duration_seconds | histogram | Duration of discovering that there are no schedulable transactions |
| pack_insert_transaction_duration_seconds | histogram | Duration of inserting one transaction into the pool of available transactions |
| pack_complete_microblock_duration_seconds | histogram | Duration of the computation associated with marking one microblock as complete |
| pack_total_transactions_per_microblock_count | histogram | Count of transactions in a scheduled microblock, including both votes and non-votes |
| pack_votes_per_microblock_count | histogram | Count of simple vote transactions in a scheduled microblock |
| pack_normal_transaction_received | counter | Count of transactions received via the normal TPU path |
| pack_transaction_inserted_bundle_blacklist | counter | Result of inserting a transaction into the pack object (Transaction uses an account on the bundle blacklist) |
| pack_transaction_inserted_write_sysvar | counter | Result of inserting a transaction into the pack object (Transaction tries to write to a sysvar) |
| pack_transaction_inserted_estimation_fail | counter | Result of inserting a transaction into the pack object (Estimating compute cost and/or fee failed) |
| pack_transaction_inserted_duplicate_account | counter | Result of inserting a transaction into the pack object (Transaction included an account address twice) |
| pack_transaction_inserted_too_many_accounts | counter | Result of inserting a transaction into the pack object (Transaction tried to load too many accounts) |
| pack_transaction_inserted_too_large | counter | Result of inserting a transaction into the pack object (Transaction requests too many CUs) |
| pack_transaction_inserted_expired | counter | Result of inserting a transaction into the pack object (Transaction already expired) |
| pack_transaction_inserted_addr_lut | counter | Result of inserting a transaction into the pack object (Transaction loaded accounts from a lookup table) |
| pack_transaction_inserted_unaffordable | counter | Result of inserting a transaction into the pack object (Fee payer's balance below transaction fee) |
| pack_transaction_inserted_duplicate | counter | Result of inserting a transaction into the pack object (Pack aware of transaction with same signature) |
| pack_transaction_inserted_priority | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and other competing transactions) |
| pack_transaction_inserted_nonvote_add | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote added to pending transactions) |
| pack_transaction_inserted_vote_add | counter | Result of inserting a transaction into the pack object (Simple vote transaction was added to pending transactions) |
| pack_transaction_inserted_nonvote_replace | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote replaced a lower priority transaction) |
| pack_transaction_inserted_vote_replace | counter | Result of inserting a transaction into the pack object (Simple vote transaction replaced a lower priority transaction) |
| pack_metric_timing_no_txn_no_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had no transactions available, and wasn't leader) |
| pack_metric_timing_txn_no_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had transactions available, but wasn't leader or had hit a limit) |
| pack_metric_timing_no_txn_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had no transactions available, had banks but wasn't leader?) |
| pack_metric_timing_txn_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had transactions available, had banks but wasn't leader?) |
| pack_metric_timing_no_txn_no_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had no transactions available, and was leader but had no available banks) |
| pack_metric_timing_txn_no_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had transactions available, was leader, but had no available banks) |
| pack_metric_timing_no_txn_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had available banks but no transactions) |
| pack_metric_timing_txn_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had banks and transactions available but couldn't schedule anything non-conflicting) |
| pack_metric_timing_no_txn_no_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_txn_no_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_no_txn_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_txn_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_no_txn_no_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy?) |
| pack_metric_timing_txn_no_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy?) |
| pack_metric_timing_no_txn_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock and now has no transactions) |
| pack_metric_timing_txn_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock) |
| pack_transaction_dropped_from_extra | counter | Transactions dropped from the extra transaction storage because it was full |
| pack_transaction_inserted_to_extra | counter | Transactions inserted into the extra transaction storage because pack's primary storage was full |
| pack_transaction_inserted_from_extra | counter | Transactions pulled from the extra transaction storage and inserted into pack's primary storage |
| pack_transaction_expired | counter | Transactions deleted from pack because their TTL expired |
| pack_available_transactions | gauge | The total number of pending transactions in pack's pool that are available to be scheduled |
| pack_available_vote_transactions | gauge | The number of pending simple vote transactions in pack's pool that are available to be scheduled |
| pack_pending_transactions_heap_size | gauge | The maximum number of pending transactions that pack can consider. This value is fixed at Firedancer startup but is a useful reference for AvailableTransactions and AvailableVoteTransactions. |
| pack_conflicting_transactions | gauge | The number of available transactions that are temporarily not being considered due to account lock conflicts with many higher paying transactions |
| pack_smallest_pending_transaction | gauge | A lower bound on the smallest non-vote transaction (in cost units) that is immediately available for scheduling |
| pack_microblock_per_block_limit | counter | The number of times pack did not pack a microblock because the limit on microblocks/block had been reached |
| pack_data_per_block_limit | counter | The number of times pack did not pack a microblock because it reached reached the data per block limit at the start of trying to schedule a microblock |
| pack_transaction_schedule_taken | counter | Result of trying to consider a transaction for scheduling (Pack included the transaction in the microblock) |
| pack_transaction_schedule_cu_limit | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block CU limit) |
| pack_transaction_schedule_fast_path | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the fast bitvector check) |
| pack_transaction_schedule_byte_limit | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block data size limit) |
| pack_transaction_schedule_write_cost | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) |
| pack_transaction_schedule_slow_path | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the full slow check) |
| pack_cus_consumed_in_block | gauge | The number of cost units consumed in the current block, or 0 if pack is not currently packing a block |
| pack_cus_scheduled | histogram | The number of cost units scheduled for each block pack produced. This can be higher than the block limit because of returned CUs. |
| pack_cus_rebated | histogram | The number of compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses. |
| pack_cus_net | histogram | The net number of cost units (scheduled - rebated) in each block pack produced. |
| pack_delete_missed | counter | Count of attempts to delete a transaction that wasn't found |
| pack_delete_hit | counter | Count of attempts to delete a transaction that was found and deleted |
Bank Tile
| Metric | Type | Description |
|---|---|---|
| bank_transaction_sanitize_failure | counter | Number of transactions that failed to sanitize. |
| bank_transaction_not_executed_failure | counter | Number of transactions that did not execute. This is different than transactions which fail to execute, which make it onto the chain. |
| bank_precompile_verify_failure | counter | Number of transactions that failed precompile verification and thus will not execute. |
| bank_slot_acquire_success | counter | Result of acquiring a slot. (Success) |
| bank_slot_acquire_too_high | counter | Result of acquiring a slot. (Too high) |
| bank_slot_acquire_too_low | counter | Result of acquiring a slot. (Too low) |
| bank_transaction_load_address_tables_success | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (Success) |
| bank_transaction_load_address_tables_slot_hashes_sysvar_not_found | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The slot hashes syvar could not be found.) |
| bank_transaction_load_address_tables_account_not_found | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account storing the address lookup table was deactivated or could not be found.) |
| bank_transaction_load_address_tables_invalid_account_owner | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account that owns the referenced lookup table is not the address lookup table program.) |
| bank_transaction_load_address_tables_invalid_account_data | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The data for the referenced address lookup table is malformed.) |
| bank_transaction_load_address_tables_invalid_index | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The referenced index in the address lookup table does not exist.) |
| bank_transaction_load_success | counter | Result of loading a transaction. (Success) |
| bank_transaction_load_account_in_use | counter | Result of loading a transaction. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_transaction_load_account_loaded_twice | counter | Result of loading a transaction. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_transaction_load_account_not_found | counter | Result of loading a transaction. (Attempt to debit an account but found no record of a prior credit.) |
| bank_transaction_load_program_account_not_found | counter | Result of loading a transaction. (Attempt to load a program that does not exist.) |
| bank_transaction_load_insufficient_funds_for_fee | counter | Result of loading a transaction. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_transaction_load_invalid_account_for_fee | counter | Result of loading a transaction. (This account may not be used to pay transaction fees.) |
| bank_transaction_load_already_processed | counter | Result of loading a transaction. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_transaction_load_blockhash_not_found | counter | Result of loading a transaction. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_transaction_load_instruction_error | counter | Result of loading a transaction. (An error occurred while processing an instruction.) |
| bank_transaction_load_call_chain_too_deep | counter | Result of loading a transaction. (Loader call chain is too deep.) |
| bank_transaction_load_missing_signature_for_fee | counter | Result of loading a transaction. (Transaction requires a fee but has no signature present.) |
| bank_transaction_load_invalid_account_index | counter | Result of loading a transaction. (Transaction contains an invalid account reference.) |
| bank_transaction_load_signature_failure | counter | Result of loading a transaction. (Transaction did not pass signature verification.) |
| bank_transaction_load_invalid_program_for_execution | counter | Result of loading a transaction. (This program may not be used for executing instructions.) |
| bank_transaction_load_sanitize_failure | counter | Result of loading a transaction. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_transaction_load_cluster_maintenance | counter | Result of loading a transaction. (Transactions are currently disabled due to cluster maintenance.) |
| bank_transaction_load_account_borrow_outstanding | counter | Result of loading a transaction. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_transaction_load_would_exceed_max_block_cost_limit | counter | Result of loading a transaction. (Transaction would exceed max Block Cost Limit.) |
| bank_transaction_load_unsupported_version | counter | Result of loading a transaction. (Transaction version is unsupported.) |
| bank_transaction_load_invalid_writable_account | counter | Result of loading a transaction. (Transaction loads a writable account that cannot be written.) |
| bank_transaction_load_would_exceed_max_account_cost_limit | counter | Result of loading a transaction. (Transaction would exceed max account limit within the block.) |
| bank_transaction_load_would_exceed_account_data_block_limit | counter | Result of loading a transaction. (Transaction would exceed account data limit within the block.) |
| bank_transaction_load_too_many_account_locks | counter | Result of loading a transaction. (Transaction locked too many accounts.) |
| bank_transaction_load_address_lookup_table_not_found | counter | Result of loading a transaction. (Address lookup table not found.) |
| bank_transaction_load_invalid_address_lookup_table_owner | counter | Result of loading a transaction. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_transaction_load_invalid_address_lookup_table_data | counter | Result of loading a transaction. (Attempted to lookup addresses from an invalid account.) |
| bank_transaction_load_invalid_address_lookup_table_index | counter | Result of loading a transaction. (Address table lookup uses an invalid index.) |
| bank_transaction_load_invalid_rent_paying_account | counter | Result of loading a transaction. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_transaction_load_would_exceed_max_vote_cost_limit | counter | Result of loading a transaction. (Transaction would exceed max Vote Cost Limit.) |
| bank_transaction_load_would_exceed_account_data_total_limit | counter | Result of loading a transaction. (Transaction would exceed total account data limit.) |
| bank_transaction_load_duplicate_instruction | counter | Result of loading a transaction. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_transaction_load_insufficient_funds_for_rent | counter | Result of loading a transaction. (Transaction results in an account with insufficient funds for rent.) |
| bank_transaction_load_max_loaded_accounts_data_size_exceeded | counter | Result of loading a transaction. (Transaction exceeded max loaded accounts data size cap.) |
| bank_transaction_load_invalid_loaded_accounts_data_size_limit | counter | Result of loading a transaction. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_transaction_load_resanitization_needed | counter | Result of loading a transaction. (Sanitized transaction differed before/after feature activiation. Needs to be resanitized.) |
| bank_transaction_load_program_execution_temporarily_restricted | counter | Result of loading a transaction. (Program execution is temporarily restricted on an account.) |
| bank_transaction_load_unbalanced_transaction | counter | Result of loading a transaction. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_transaction_load_program_cache_hit_max_limit | counter | Result of loading a transaction. (The total program cache size hit the maximum allowed limit.) |
| bank_transaction_executing_success | counter | Result of executing a transaction. Could be a failure or success. (Success) |
| bank_transaction_executing_account_in_use | counter | Result of executing a transaction. Could be a failure or success. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_transaction_executing_account_loaded_twice | counter | Result of executing a transaction. Could be a failure or success. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_transaction_executing_account_not_found | counter | Result of executing a transaction. Could be a failure or success. (Attempt to debit an account but found no record of a prior credit.) |
| bank_transaction_executing_program_account_not_found | counter | Result of executing a transaction. Could be a failure or success. (Attempt to load a program that does not exist.) |
| bank_transaction_executing_insufficient_funds_for_fee | counter | Result of executing a transaction. Could be a failure or success. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_transaction_executing_invalid_account_for_fee | counter | Result of executing a transaction. Could be a failure or success. (This account may not be used to pay transaction fees.) |
| bank_transaction_executing_already_processed | counter | Result of executing a transaction. Could be a failure or success. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_transaction_executing_blockhash_not_found | counter | Result of executing a transaction. Could be a failure or success. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_transaction_executing_instruction_error | counter | Result of executing a transaction. Could be a failure or success. (An error occurred while processing an instruction.) |
| bank_transaction_executing_call_chain_too_deep | counter | Result of executing a transaction. Could be a failure or success. (Loader call chain is too deep.) |
| bank_transaction_executing_missing_signature_for_fee | counter | Result of executing a transaction. Could be a failure or success. (Transaction requires a fee but has no signature present.) |
| bank_transaction_executing_invalid_account_index | counter | Result of executing a transaction. Could be a failure or success. (Transaction contains an invalid account reference.) |
| bank_transaction_executing_signature_failure | counter | Result of executing a transaction. Could be a failure or success. (Transaction did not pass signature verification.) |
| bank_transaction_executing_invalid_program_for_execution | counter | Result of executing a transaction. Could be a failure or success. (This program may not be used for executing instructions.) |
| bank_transaction_executing_sanitize_failure | counter | Result of executing a transaction. Could be a failure or success. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_transaction_executing_cluster_maintenance | counter | Result of executing a transaction. Could be a failure or success. (Transactions are currently disabled due to cluster maintenance.) |
| bank_transaction_executing_account_borrow_outstanding | counter | Result of executing a transaction. Could be a failure or success. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_transaction_executing_would_exceed_max_block_cost_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed max Block Cost Limit.) |
| bank_transaction_executing_unsupported_version | counter | Result of executing a transaction. Could be a failure or success. (Transaction version is unsupported.) |
| bank_transaction_executing_invalid_writable_account | counter | Result of executing a transaction. Could be a failure or success. (Transaction loads a writable account that cannot be written.) |
| bank_transaction_executing_would_exceed_max_account_cost_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed max account limit within the block.) |
| bank_transaction_executing_would_exceed_account_data_block_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed account data limit within the block.) |
| bank_transaction_executing_too_many_account_locks | counter | Result of executing a transaction. Could be a failure or success. (Transaction locked too many accounts.) |
| bank_transaction_executing_address_lookup_table_not_found | counter | Result of executing a transaction. Could be a failure or success. (Address lookup table not found.) |
| bank_transaction_executing_invalid_address_lookup_table_owner | counter | Result of executing a transaction. Could be a failure or success. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_transaction_executing_invalid_address_lookup_table_data | counter | Result of executing a transaction. Could be a failure or success. (Attempted to lookup addresses from an invalid account.) |
| bank_transaction_executing_invalid_address_lookup_table_index | counter | Result of executing a transaction. Could be a failure or success. (Address table lookup uses an invalid index.) |
| bank_transaction_executing_invalid_rent_paying_account | counter | Result of executing a transaction. Could be a failure or success. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_transaction_executing_would_exceed_max_vote_cost_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed max Vote Cost Limit.) |
| bank_transaction_executing_would_exceed_account_data_total_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed total account data limit.) |
| bank_transaction_executing_duplicate_instruction | counter | Result of executing a transaction. Could be a failure or success. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_transaction_executing_insufficient_funds_for_rent | counter | Result of executing a transaction. Could be a failure or success. (Transaction results in an account with insufficient funds for rent.) |
| bank_transaction_executing_max_loaded_accounts_data_size_exceeded | counter | Result of executing a transaction. Could be a failure or success. (Transaction exceeded max loaded accounts data size cap.) |
| bank_transaction_executing_invalid_loaded_accounts_data_size_limit | counter | Result of executing a transaction. Could be a failure or success. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_transaction_executing_resanitization_needed | counter | Result of executing a transaction. Could be a failure or success. (Sanitized transaction differed before/after feature activiation. Needs to be resanitized.) |
| bank_transaction_executing_program_execution_temporarily_restricted | counter | Result of executing a transaction. Could be a failure or success. (Program execution is temporarily restricted on an account.) |
| bank_transaction_executing_unbalanced_transaction | counter | Result of executing a transaction. Could be a failure or success. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_transaction_executing_program_cache_hit_max_limit | counter | Result of executing a transaction. Could be a failure or success. (The total program cache size hit the maximum allowed limit.) |
| bank_transaction_executed_success | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Success) |
| bank_transaction_executed_account_in_use | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_transaction_executed_account_loaded_twice | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_transaction_executed_account_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempt to debit an account but found no record of a prior credit.) |
| bank_transaction_executed_program_account_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempt to load a program that does not exist.) |
| bank_transaction_executed_insufficient_funds_for_fee | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_transaction_executed_invalid_account_for_fee | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (This account may not be used to pay transaction fees.) |
| bank_transaction_executed_already_processed | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_transaction_executed_blockhash_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_transaction_executed_instruction_error | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (An error occurred while processing an instruction.) |
| bank_transaction_executed_call_chain_too_deep | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Loader call chain is too deep.) |
| bank_transaction_executed_missing_signature_for_fee | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction requires a fee but has no signature present.) |
| bank_transaction_executed_invalid_account_index | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction contains an invalid account reference.) |
| bank_transaction_executed_signature_failure | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction did not pass signature verification.) |
| bank_transaction_executed_invalid_program_for_execution | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (This program may not be used for executing instructions.) |
| bank_transaction_executed_sanitize_failure | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_transaction_executed_cluster_maintenance | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transactions are currently disabled due to cluster maintenance.) |
| bank_transaction_executed_account_borrow_outstanding | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_transaction_executed_would_exceed_max_block_cost_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed max Block Cost Limit.) |
| bank_transaction_executed_unsupported_version | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction version is unsupported.) |
| bank_transaction_executed_invalid_writable_account | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction loads a writable account that cannot be written.) |
| bank_transaction_executed_would_exceed_max_account_cost_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed max account limit within the block.) |
| bank_transaction_executed_would_exceed_account_data_block_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed account data limit within the block.) |
| bank_transaction_executed_too_many_account_locks | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction locked too many accounts.) |
| bank_transaction_executed_address_lookup_table_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Address lookup table not found.) |
| bank_transaction_executed_invalid_address_lookup_table_owner | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_transaction_executed_invalid_address_lookup_table_data | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempted to lookup addresses from an invalid account.) |
| bank_transaction_executed_invalid_address_lookup_table_index | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Address table lookup uses an invalid index.) |
| bank_transaction_executed_invalid_rent_paying_account | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_transaction_executed_would_exceed_max_vote_cost_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed max Vote Cost Limit.) |
| bank_transaction_executed_would_exceed_account_data_total_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed total account data limit.) |
| bank_transaction_executed_duplicate_instruction | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_transaction_executed_insufficient_funds_for_rent | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction results in an account with insufficient funds for rent.) |
| bank_transaction_executed_max_loaded_accounts_data_size_exceeded | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction exceeded max loaded accounts data size cap.) |
| bank_transaction_executed_invalid_loaded_accounts_data_size_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_transaction_executed_resanitization_needed | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Sanitized transaction differed before/after feature activiation. Needs to be resanitized.) |
| bank_transaction_executed_program_execution_temporarily_restricted | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Program execution is temporarily restricted on an account.) |
| bank_transaction_executed_unbalanced_transaction | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_transaction_executed_program_cache_hit_max_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The total program cache size hit the maximum allowed limit.) |
| bank_cost_model_undercount | counter | Count of transactions that used more CUs than the cost model should have permitted them to |
Poh Tile
| Metric | Type | Description |
|---|---|---|
| poh_begin_leader_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the bank. |
| poh_first_microblock_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the first microblock. |
| poh_slot_done_delay_seconds | histogram | Delay between when we become leader in a slot and when we finish the slot. |
Shred Tile
| Metric | Type | Description |
|---|---|---|
| shred_cluster_contact_info_cnt | histogram | Number of contact infos in the cluster contact info message |
| shred_microblocks_abandoned | counter | The number of microblocks that were abandoned because we switched slots without finishing the current slot |
| shred_batch_sz | histogram | The size (in bytes) of each microblock batch that is shredded |
| shred_batch_microblock_cnt | histogram | The number of microblocks in each microblock batch that is shredded |
| shred_shredding_duration_seconds | histogram | Duration of producing one FEC set from the shredder |
| shred_add_shred_duration_seconds | histogram | Duration of verifying and processing one shred received from the network |
| shred_shred_processed_bad_slot | counter | The result of processing a thread from the network (Shred was for a slot for which we don't know the leader) |
| shred_shred_processed_parse_failed | counter | The result of processing a thread from the network (Shred parsing failed) |
| shred_shred_processed_rejected | counter | The result of processing a thread from the network (Shred was invalid for one of many reasons) |
| shred_shred_processed_ignored | counter | The result of processing a thread from the network (Shred was ignored because we had already received or reconstructed it) |
| shred_shred_processed_okay | counter | The result of processing a thread from the network (Shred accepted to an incomplete FEC set) |
| shred_shred_processed_completes | counter | The result of processing a thread from the network (Shred accepted and resulted in a valid, complete FEC set) |
| shred_fec_set_spilled | counter | The number of FEC sets that were spilled because they didn't complete in time and we needed space |
| shred_shred_rejected_initial | counter | The number shreds that were rejected before any resources were allocated for the FEC set |
| shred_fec_rejected_fatal | counter | The number of FEC sets that were rejected for reasons that cause the whole FEC set to become invalid |
Store Tile
| Metric | Type | Description |
|---|---|---|
| store_transactions_inserted | counter | Count of transactions produced while we were leader in the shreds that have been inserted so far |