Metrics
Overview
Firedancer maintains many internal performance counters for use by developers and monitoring tools, and exposes them via. a Prometheus HTTP endpoint:
toml
[tiles.metric]
prometheus_listen_port = 7999sh
$ curl http://localhost:7999/metrics
# HELP tile_pid The process ID of the tile.
# TYPE tile_pid gauge
tile_pid{kind="net",kind_id="0"} 1527373
tile_pid{kind="quic",kind_id="0"} 1527370
tile_pid{kind="quic",kind_id="1"} 1527371
tile_pid{kind="verify",kind_id="0"} 1527369
tile_pid{kind="verify",kind_id="1"} 1527374
tile_pid{kind="dedup",kind_id="0"} 1527365
...WARNING
Metrics are currently only provided for developer and diagnostic use, and the endpoint or data provided may break or change in incompatible ways at any time.
There are three metric types reported by Firedancer, following the Prometheus data model:
counter— A cumulative metric repesenting a monotonically increasing counter.gauge— A single numerical value that can go arbitrarily up or down.histogram— Samples observations like packet sizes and counts them in buckets.
All Links
There are some metrics reported for links, which are data queues between tiles. Links are single-producer multi-consumer, so the same link (identified by a link_kind and link_kind_id) will potentially have multiple consumers reporting the metric, one per consumer (identified by a kind and kind_id).
sh
# HELP link_published_size_bytes The total number of bytes read by the link consumer.
# TYPE link_published_size_bytes counter
link_published_size_bytes{kind="net",kind_id="0",link_kind="quic_net",link_kind_id="0"} 0
link_published_size_bytes{kind="net",kind_id="0",link_kind="shred_net",link_kind_id="0"} 0These link related metrics have four labels, which are to identify the link the metric is for:
kind— The name of the tile consuming from the link.kind_id— The tile index of the tile which is consuming from the link.link_kind— The name of the link being consumed.link_kind_id— The link index of the link which is being consumed.
| Metric | Type | Description |
|---|---|---|
| link_slow_count | counter | The number of times the consumer was detected as rate limiting consumer by the producer. |
| link_published_count | counter | The number of times the link has consumed and published a fragment. |
| link_published_size_bytes | counter | The total number of bytes read by the link consumer. |
| link_filtered_count | counter | The number of consumed fragments that were filtered. |
| link_filtered_size_bytes | counter | The total number of bytes read by the link consumer that were filtered. |
| link_overrun_polling_count | counter | The number of times the link has been overrun while polling. |
| link_overrun_polling_frag_count | counter | The number of fragments the link has not processed because it was overrun while polling. |
| link_overrun_reading_count | counter | The number of input overruns detected while reading metadata by the consumer. |
All Tiles
Most performance counters are local to a tile and are not aggregated. For example if you have two QUIC tiles (two CPU cores assigned to serving incoming QUIC connections) each QUIC counter will appear twice:
sh
# HELP quic_connections_created The total number of connections that have been created.
# TYPE quic_connections_created counter
quic_connections_created{kind="quic",kind_id="0"} 42145
quic_connections_created{kind="quic",kind_id="1"} 38268
# HELP quic_connections_aborted Number of connections aborted.
# TYPE quic_connections_aborted counter
quic_connections_aborted{kind="quic",kind_id="0"} 14
quic_connections_aborted{kind="quic",kind_id="1"} 23All tile related metrics have just two labels, and these are to identify which tile (and which tile index, for tiles of the same kind) the metric is for:
kind— The tile name the metric is being reported for.kind_id— The tile index of the tile which is reporting the metric.
| Metric | Type | Description |
|---|---|---|
| tile_pid | gauge | The process ID of the tile. |
| tile_tid | gauge | The thread ID of the tile. Always the same as the Pid in production, but might be different in development. |
| stem_in_backpressure | gauge | Whether the tile is currently backpressured or not, either 1 or 0. |
| stem_backpressure_count | counter | Number of times the times the tile has had to wait for one of more consumers to catch up to resume publishing. |
| stem_loop_housekeeping_duration_seconds | histogram | Duration of one iteration of the run loop which did housekeeping. The various loop durations are mutually exclusive and exhaustive, so the sum of time across all of them is roughly the total running time of the tile. Loop durations are per iteration of the run loop and non-blocking, so for example each 'caught up' sample does not represent the time we waited for new input data, but rather how long each iteration of the spin loop waiting for the data took. |
| stem_loop_backpressure_duration_seconds | histogram | Duration of one iteration of the run loop which terminated because we were backpressured by a consumer. |
| stem_loop_caught_up_duration_seconds | histogram | Duration of one iteration of the run loop which terminated because there was no new data to process. |
| stem_loop_overrun_polling_duration_seconds | histogram | Duration of one iteration of the run loop which terminated because we were overrun while polling. |
| stem_loop_overrun_reading_duration_seconds | histogram | Duration of one iteration of the run loop which terminated because we were overrun while reading. |
| stem_loop_filter_before_fragment_duration_seconds | histogram | Duration of one iteration of the run loop which terminated because we filtered the fragment before reading it. |
| stem_loop_filter_after_fragment_duration_seconds | histogram | Duration of one iteration of the run loop which terminated because we filtered the fragment after reading it. |
| stem_loop_finish_duration_seconds | histogram | Duration of one iteration of the run loop which received, did not filter, and processed the fragment. |
| stem_fragment_filtered_size_bytes | histogram | Size of each fragment that was filtered and not processed by the tile. |
| stem_fragment_handled_size_bytes | histogram | Size of each fragment that was processed (not filtered) by the tile. |
Net Tile
| Metric | Type | Description |
|---|---|---|
| net_tile_received_packets | counter | Number of IP packets received. |
| net_tile_received_bytes | counter | Total bytes received (including IP, UDP headers). |
| net_tile_sent_packets | counter | Number of IP packets sent. |
| net_tile_sent_bytes | counter | Total bytes sent (including IP, UDP headers). |
Quic Tile
| Metric | Type | Description |
|---|---|---|
| quic_tile_non_quic_reassembly_append_success | counter | Result of fragment reassembly for a non-QUIC UDP transaction. (Success) |
| quic_tile_non_quic_reassembly_append_error_oversize | counter | Result of fragment reassembly for a non-QUIC UDP transaction. (Oversize message) |
| quic_tile_non_quic_reassembly_append_error_skip | counter | Result of fragment reassembly for a non-QUIC UDP transaction. (Out-of-order data within QUIC stream) |
| quic_tile_non_quic_reassembly_append_error_state | counter | Result of fragment reassembly for a non-QUIC UDP transaction. (Unexpected slot state) |
| quic_tile_non_quic_reassembly_publish_success | counter | Result of publishing reassmbled fragment for a non-QUIC UDP transaction. (Success) |
| quic_tile_non_quic_reassembly_publish_error_oversize | counter | Result of publishing reassmbled fragment for a non-QUIC UDP transaction. (Oversize message) |
| quic_tile_non_quic_reassembly_publish_error_skip | counter | Result of publishing reassmbled fragment for a non-QUIC UDP transaction. (Out-of-order data within QUIC stream) |
| quic_tile_non_quic_reassembly_publish_error_state | counter | Result of publishing reassmbled fragment for a non-QUIC UDP transaction. (Unexpected slot state) |
| quic_tile_reassembly_append_success | counter | Result of fragment reassembly for a QUIC transaction. (Success) |
| quic_tile_reassembly_append_error_oversize | counter | Result of fragment reassembly for a QUIC transaction. (Oversize message) |
| quic_tile_reassembly_append_error_skip | counter | Result of fragment reassembly for a QUIC transaction. (Out-of-order data within QUIC stream) |
| quic_tile_reassembly_append_error_state | counter | Result of fragment reassembly for a QUIC transaction. (Unexpected slot state) |
| quic_tile_reassembly_publish_success | counter | Result of publishing reassmbled fragment for a QUIC transaction. (Success) |
| quic_tile_reassembly_publish_error_oversize | counter | Result of publishing reassmbled fragment for a QUIC transaction. (Oversize message) |
| quic_tile_reassembly_publish_error_skip | counter | Result of publishing reassmbled fragment for a QUIC transaction. (Out-of-order data within QUIC stream) |
| quic_tile_reassembly_publish_error_state | counter | Result of publishing reassmbled fragment for a QUIC transaction. (Unexpected slot state) |
| quic_tile_reassembly_notify_aborted | counter | Reassembly slot was aborted before it was notified. |
| quic_tile_reassembly_notify_clobbered | counter | Reassembly slot was clobbered before it was notified. |
| quic_tile_quic_packet_too_small | counter | Count of packets received on the QUIC port that were too small to be a valid IP packet. |
| quic_tile_non_quic_packet_too_small | counter | Count of packets received on the non-QUIC port that were too small to be a valid IP packet. |
| quic_tile_non_quic_packet_too_large | counter | Count of packets received on the non-QUIC port that were too large to be a valid transaction. |
| quic_received_packets | counter | Number of IP packets received. |
| quic_received_bytes | counter | Total bytes received (including IP, UDP, QUIC headers). |
| quic_sent_packets | counter | Number of IP packets sent. |
| quic_sent_bytes | counter | Total bytes sent (including IP, UDP, QUIC headers). |
| quic_connections_active | gauge | The number of currently active QUIC connections. |
| quic_connections_created | counter | The total number of connections that have been created. |
| quic_connections_closed | counter | Number of connections gracefully closed. |
| quic_connections_aborted | counter | Number of connections aborted. |
| quic_connections_retried | counter | Number of connections established with retry. |
| quic_connection_error_no_slots | counter | Number of connections that failed to create due to lack of slots. |
| quic_connection_error_tls_fail | counter | Number of connections that aborted due to TLS failure. |
| quic_connection_error_retry_fail | counter | Number of connections that failed during retry (e.g. invalid token). |
| quic_handshakes_created | counter | Number of handshake flows created. |
| quic_handshake_error_alloc_fail | counter | Number of handshakes dropped due to alloc fail. |
| quic_stream_opened_bidi_client | counter | Number of streams opened. (Bidirectional client) |
| quic_stream_opened_bidi_server | counter | Number of streams opened. (Bidirectional server) |
| quic_stream_opened_uni_client | counter | Number of streams opened. (Unidirectional client) |
| quic_stream_opened_uni_server | counter | Number of streams opened. (Unidirectional server) |
| quic_stream_closed_bidi_client | counter | Number of streams closed. (Bidirectional client) |
| quic_stream_closed_bidi_server | counter | Number of streams closed. (Bidirectional server) |
| quic_stream_closed_uni_client | counter | Number of streams closed. (Unidirectional client) |
| quic_stream_closed_uni_server | counter | Number of streams closed. (Unidirectional server) |
| quic_stream_active_bidi_client | gauge | Number of active streams. (Bidirectional client) |
| quic_stream_active_bidi_server | gauge | Number of active streams. (Bidirectional server) |
| quic_stream_active_uni_client | gauge | Number of active streams. (Unidirectional client) |
| quic_stream_active_uni_server | gauge | Number of active streams. (Unidirectional server) |
| quic_stream_received_events | counter | Number of stream RX events. |
| quic_stream_received_bytes | counter | Total stream payload bytes received. |
Verify Tile
| Metric | Type | Description |
|---|---|---|
| verify_transaction_parse_failure | counter | Count of transactions that failed to parse |
| verify_transaction_dedup_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
| verify_transaction_verify_failure | counter | Count of transactions that failed to deduplicate in the verify stage |
Dedup Tile
| Metric | Type | Description |
|---|---|---|
| dedup_gossiped_votes_received | counter | Count of simple vote transactions received over gossip instead of via the normal TPU path |
Pack Tile
| Metric | Type | Description |
|---|---|---|
| pack_schedule_microblock_duration_seconds | histogram | Duration of scheduling one microblock |
| pack_insert_transaction_duration_seconds | histogram | Duration of inserting one transaction into the pool of available transactions |
| pack_total_transactions_per_microblock_count | histogram | Count of transactions in a scheduled microblock, including both votes and non-votes |
| pack_votes_per_microblock_count | histogram | Count of simple vote transactions in a scheduled microblock |
| pack_normal_transaction_received | counter | Count of transactions received via the normal TPU path |
| pack_transaction_inserted_write_sysvar | counter | Result of inserting a transaction into the pack object (Transaction tries to write to a sysvar) |
| pack_transaction_inserted_estimation_fail | counter | Result of inserting a transaction into the pack object (Estimating compute cost and/or fee failed) |
| pack_transaction_inserted_duplicate_account | counter | Result of inserting a transaction into the pack object (Transaction included an account address twice) |
| pack_transaction_inserted_too_many_accounts | counter | Result of inserting a transaction into the pack object (Transaction tried to load too many accounts) |
| pack_transaction_inserted_too_large | counter | Result of inserting a transaction into the pack object (Transaction requests too many CUs) |
| pack_transaction_inserted_expired | counter | Result of inserting a transaction into the pack object (Transaction already expired) |
| pack_transaction_inserted_addr_lut | counter | Result of inserting a transaction into the pack object (Transaction loaded accounts from a lookup table) |
| pack_transaction_inserted_unaffordable | counter | Result of inserting a transaction into the pack object (Fee payer's balance below transaction fee) |
| pack_transaction_inserted_duplicate | counter | Result of inserting a transaction into the pack object (Pack aware of transaction with same signature) |
| pack_transaction_inserted_priority | counter | Result of inserting a transaction into the pack object (Transaction's fee was too low given its compute unit requirement and other competing transactions) |
| pack_transaction_inserted_nonvote_add | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote added to pending transactions) |
| pack_transaction_inserted_vote_add | counter | Result of inserting a transaction into the pack object (Simple vote transaction was added to pending transactions) |
| pack_transaction_inserted_nonvote_replace | counter | Result of inserting a transaction into the pack object (Transaction that was not a simple vote replaced a lower priority transaction) |
| pack_transaction_inserted_vote_replace | counter | Result of inserting a transaction into the pack object (Simple vote transaction replaced a lower priority transaction) |
| pack_metric_timing_no_txn_no_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had no transactions available, and wasn't leader) |
| pack_metric_timing_txn_no_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had transactions available, but wasn't leader or had hit a limit) |
| pack_metric_timing_no_txn_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had no transactions available, had banks but wasn't leader?) |
| pack_metric_timing_txn_bank_no_leader_no_microblock | counter | Time in nanos spent in each state (Pack had transactions available, had banks but wasn't leader?) |
| pack_metric_timing_no_txn_no_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had no transactions available, and was leader but had no available banks) |
| pack_metric_timing_txn_no_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had transactions available, was leader, but had no available banks) |
| pack_metric_timing_no_txn_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had available banks but no transactions) |
| pack_metric_timing_txn_bank_leader_no_microblock | counter | Time in nanos spent in each state (Pack had banks and transactions available but couldn't schedule anything non-conflicting) |
| pack_metric_timing_no_txn_no_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_txn_no_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_no_txn_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_txn_bank_no_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock while not leader?) |
| pack_metric_timing_no_txn_no_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy?) |
| pack_metric_timing_txn_no_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock but all banks were busy?) |
| pack_metric_timing_no_txn_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock and now has no transactions) |
| pack_metric_timing_txn_bank_leader_microblock | counter | Time in nanos spent in each state (Pack scheduled a non-empty microblock) |
| pack_transaction_dropped_from_extra | counter | Transactions dropped from the extra transaction storage because it was full |
| pack_transaction_inserted_to_extra | counter | Transactions inserted into the extra transaction storage because pack's primary storage was full |
| pack_transaction_inserted_from_extra | counter | Transactions pulled from the extra transaction storage and inserted into pack's primary storage |
| pack_transaction_expired | counter | Transactions deleted from pack because their TTL expired |
| pack_available_transactions | gauge | The total number of pending transactions in pack's pool that are available to be scheduled |
| pack_available_vote_transactions | gauge | The number of pending simple vote transactions in pack's pool that are available to be scheduled |
| pack_pending_transactions_heap_size | gauge | The maximum number of pending transactions that pack can consider. This value is fixed at Firedancer startup but is a useful reference for AvailableTransactions and AvailableVoteTransactions. |
| pack_microblock_per_block_limit | counter | The number of times pack did not pack a microblock because the limit on microblocks/block had been reached |
| pack_data_per_block_limit | counter | The number of times pack did not pack a microblock because it reached reached the data per block limit at the start of trying to schedule a microblock |
| pack_transaction_schedule_taken | counter | Result of trying to consider a transaction for scheduling (Pack included the transaction in the microblock) |
| pack_transaction_schedule_cu_limit | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block CU limit) |
| pack_transaction_schedule_fast_path | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the fast bitvector check) |
| pack_transaction_schedule_byte_limit | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have exceeded the block data size limit) |
| pack_transaction_schedule_write_cost | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because it would have caused a writable account to exceed the per-account block write cost limit) |
| pack_transaction_schedule_slow_path | counter | Result of trying to consider a transaction for scheduling (Pack skipped the transaction because of account conflicts using the full slow check) |
| pack_cus_consumed_in_block | gauge | The number of cost units consumed in the current block, or 0 if pack is not currently packing a block |
| pack_cus_scheduled | histogram | The number of cost units scheduled for each block pack produced. This can be higher than the block limit because of returned CUs. |
| pack_cus_rebated | histogram | The number of compute units rebated for each block pack produced. Compute units are rebated when a transaction fails prior to execution or requests more compute units than it uses. |
| pack_cus_net | histogram | The net number of cost units (scheduled - rebated) in each block pack produced. |
| pack_cost_model_undercount | counter | Count of transactions that used more CUs than the cost model should have permitted them to |
| pack_delete_missed | counter | Count of attempts to delete a transaction that wasn't found |
| pack_delete_hit | counter | Count of attempts to delete a transaction that was found and deleted |
Bank Tile
| Metric | Type | Description |
|---|---|---|
| bank_tile_transaction_sanitize_failure | counter | Number of transactions that failed to sanitize. |
| bank_tile_transaction_not_executed_failure | counter | Number of transactions that did not execute. This is different than transactions which fail to execute, which make it onto the chain. |
| bank_tile_precompile_verify_failure | counter | Number of transactions that failed precompile verification and thus will not execute. |
| bank_tile_slot_acquire_success | counter | Result of acquiring a slot. (Success) |
| bank_tile_slot_acquire_too_high | counter | Result of acquiring a slot. (Too high) |
| bank_tile_slot_acquire_too_low | counter | Result of acquiring a slot. (Too low) |
| bank_tile_transaction_load_address_tables_success | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (Success) |
| bank_tile_transaction_load_address_tables_slot_hashes_sysvar_not_found | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The slot hashes syvar could not be found.) |
| bank_tile_transaction_load_address_tables_account_not_found | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account storing the address lookup table was deactivated or could not be found.) |
| bank_tile_transaction_load_address_tables_invalid_account_owner | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The account that owns the referenced lookup table is not the address lookup table program.) |
| bank_tile_transaction_load_address_tables_invalid_account_data | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The data for the referenced address lookup table is malformed.) |
| bank_tile_transaction_load_address_tables_invalid_index | counter | Result of loading address lookup tables for a transaction. If there are multiple errors for the transaction, only the first one is reported. (The referenced index in the address lookup table does not exist.) |
| bank_tile_transaction_load_success | counter | Result of loading a transaction. (Success) |
| bank_tile_transaction_load_account_in_use | counter | Result of loading a transaction. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_tile_transaction_load_account_loaded_twice | counter | Result of loading a transaction. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_tile_transaction_load_account_not_found | counter | Result of loading a transaction. (Attempt to debit an account but found no record of a prior credit.) |
| bank_tile_transaction_load_program_account_not_found | counter | Result of loading a transaction. (Attempt to load a program that does not exist.) |
| bank_tile_transaction_load_insufficient_funds_for_fee | counter | Result of loading a transaction. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_tile_transaction_load_invalid_account_for_fee | counter | Result of loading a transaction. (This account may not be used to pay transaction fees.) |
| bank_tile_transaction_load_already_processed | counter | Result of loading a transaction. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_tile_transaction_load_blockhash_not_found | counter | Result of loading a transaction. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_tile_transaction_load_instruction_error | counter | Result of loading a transaction. (An error occurred while processing an instruction.) |
| bank_tile_transaction_load_call_chain_too_deep | counter | Result of loading a transaction. (Loader call chain is too deep.) |
| bank_tile_transaction_load_missing_signature_for_fee | counter | Result of loading a transaction. (Transaction requires a fee but has no signature present.) |
| bank_tile_transaction_load_invalid_account_index | counter | Result of loading a transaction. (Transaction contains an invalid account reference.) |
| bank_tile_transaction_load_signature_failure | counter | Result of loading a transaction. (Transaction did not pass signature verification.) |
| bank_tile_transaction_load_invalid_program_for_execution | counter | Result of loading a transaction. (This program may not be used for executing instructions.) |
| bank_tile_transaction_load_sanitize_failure | counter | Result of loading a transaction. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_tile_transaction_load_cluster_maintenance | counter | Result of loading a transaction. (Transactions are currently disabled due to cluster maintenance.) |
| bank_tile_transaction_load_account_borrow_outstanding | counter | Result of loading a transaction. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_tile_transaction_load_would_exceed_max_block_cost_limit | counter | Result of loading a transaction. (Transaction would exceed max Block Cost Limit.) |
| bank_tile_transaction_load_unsupported_version | counter | Result of loading a transaction. (Transaction version is unsupported.) |
| bank_tile_transaction_load_invalid_writable_account | counter | Result of loading a transaction. (Transaction loads a writable account that cannot be written.) |
| bank_tile_transaction_load_would_exceed_max_account_cost_limit | counter | Result of loading a transaction. (Transaction would exceed max account limit within the block.) |
| bank_tile_transaction_load_would_exceed_account_data_block_limit | counter | Result of loading a transaction. (Transaction would exceed account data limit within the block.) |
| bank_tile_transaction_load_too_many_account_locks | counter | Result of loading a transaction. (Transaction locked too many accounts.) |
| bank_tile_transaction_load_address_lookup_table_not_found | counter | Result of loading a transaction. (Address lookup table not found.) |
| bank_tile_transaction_load_invalid_address_lookup_table_owner | counter | Result of loading a transaction. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_tile_transaction_load_invalid_address_lookup_table_data | counter | Result of loading a transaction. (Attempted to lookup addresses from an invalid account.) |
| bank_tile_transaction_load_invalid_address_lookup_table_index | counter | Result of loading a transaction. (Address table lookup uses an invalid index.) |
| bank_tile_transaction_load_invalid_rent_paying_account | counter | Result of loading a transaction. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_tile_transaction_load_would_exceed_max_vote_cost_limit | counter | Result of loading a transaction. (Transaction would exceed max Vote Cost Limit.) |
| bank_tile_transaction_load_would_exceed_account_data_total_limit | counter | Result of loading a transaction. (Transaction would exceed total account data limit.) |
| bank_tile_transaction_load_duplicate_instruction | counter | Result of loading a transaction. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_tile_transaction_load_insufficient_funds_for_rent | counter | Result of loading a transaction. (Transaction results in an account with insufficient funds for rent.) |
| bank_tile_transaction_load_max_loaded_accounts_data_size_exceeded | counter | Result of loading a transaction. (Transaction exceeded max loaded accounts data size cap.) |
| bank_tile_transaction_load_invalid_loaded_accounts_data_size_limit | counter | Result of loading a transaction. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_tile_transaction_load_resanitization_needed | counter | Result of loading a transaction. (Sanitized transaction differed before/after feature activiation. Needs to be resanitized.) |
| bank_tile_transaction_load_program_execution_temporarily_restricted | counter | Result of loading a transaction. (Program execution is temporarily restricted on an account.) |
| bank_tile_transaction_load_unbalanced_transaction | counter | Result of loading a transaction. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_tile_transaction_load_program_cache_hit_max_limit | counter | Result of loading a transaction. (The total program cache size hit the maximum allowed limit.) |
| bank_tile_transaction_executing_success | counter | Result of executing a transaction. Could be a failure or success. (Success) |
| bank_tile_transaction_executing_account_in_use | counter | Result of executing a transaction. Could be a failure or success. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_tile_transaction_executing_account_loaded_twice | counter | Result of executing a transaction. Could be a failure or success. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_tile_transaction_executing_account_not_found | counter | Result of executing a transaction. Could be a failure or success. (Attempt to debit an account but found no record of a prior credit.) |
| bank_tile_transaction_executing_program_account_not_found | counter | Result of executing a transaction. Could be a failure or success. (Attempt to load a program that does not exist.) |
| bank_tile_transaction_executing_insufficient_funds_for_fee | counter | Result of executing a transaction. Could be a failure or success. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_tile_transaction_executing_invalid_account_for_fee | counter | Result of executing a transaction. Could be a failure or success. (This account may not be used to pay transaction fees.) |
| bank_tile_transaction_executing_already_processed | counter | Result of executing a transaction. Could be a failure or success. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_tile_transaction_executing_blockhash_not_found | counter | Result of executing a transaction. Could be a failure or success. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_tile_transaction_executing_instruction_error | counter | Result of executing a transaction. Could be a failure or success. (An error occurred while processing an instruction.) |
| bank_tile_transaction_executing_call_chain_too_deep | counter | Result of executing a transaction. Could be a failure or success. (Loader call chain is too deep.) |
| bank_tile_transaction_executing_missing_signature_for_fee | counter | Result of executing a transaction. Could be a failure or success. (Transaction requires a fee but has no signature present.) |
| bank_tile_transaction_executing_invalid_account_index | counter | Result of executing a transaction. Could be a failure or success. (Transaction contains an invalid account reference.) |
| bank_tile_transaction_executing_signature_failure | counter | Result of executing a transaction. Could be a failure or success. (Transaction did not pass signature verification.) |
| bank_tile_transaction_executing_invalid_program_for_execution | counter | Result of executing a transaction. Could be a failure or success. (This program may not be used for executing instructions.) |
| bank_tile_transaction_executing_sanitize_failure | counter | Result of executing a transaction. Could be a failure or success. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_tile_transaction_executing_cluster_maintenance | counter | Result of executing a transaction. Could be a failure or success. (Transactions are currently disabled due to cluster maintenance.) |
| bank_tile_transaction_executing_account_borrow_outstanding | counter | Result of executing a transaction. Could be a failure or success. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_tile_transaction_executing_would_exceed_max_block_cost_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed max Block Cost Limit.) |
| bank_tile_transaction_executing_unsupported_version | counter | Result of executing a transaction. Could be a failure or success. (Transaction version is unsupported.) |
| bank_tile_transaction_executing_invalid_writable_account | counter | Result of executing a transaction. Could be a failure or success. (Transaction loads a writable account that cannot be written.) |
| bank_tile_transaction_executing_would_exceed_max_account_cost_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed max account limit within the block.) |
| bank_tile_transaction_executing_would_exceed_account_data_block_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed account data limit within the block.) |
| bank_tile_transaction_executing_too_many_account_locks | counter | Result of executing a transaction. Could be a failure or success. (Transaction locked too many accounts.) |
| bank_tile_transaction_executing_address_lookup_table_not_found | counter | Result of executing a transaction. Could be a failure or success. (Address lookup table not found.) |
| bank_tile_transaction_executing_invalid_address_lookup_table_owner | counter | Result of executing a transaction. Could be a failure or success. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_tile_transaction_executing_invalid_address_lookup_table_data | counter | Result of executing a transaction. Could be a failure or success. (Attempted to lookup addresses from an invalid account.) |
| bank_tile_transaction_executing_invalid_address_lookup_table_index | counter | Result of executing a transaction. Could be a failure or success. (Address table lookup uses an invalid index.) |
| bank_tile_transaction_executing_invalid_rent_paying_account | counter | Result of executing a transaction. Could be a failure or success. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_tile_transaction_executing_would_exceed_max_vote_cost_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed max Vote Cost Limit.) |
| bank_tile_transaction_executing_would_exceed_account_data_total_limit | counter | Result of executing a transaction. Could be a failure or success. (Transaction would exceed total account data limit.) |
| bank_tile_transaction_executing_duplicate_instruction | counter | Result of executing a transaction. Could be a failure or success. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_tile_transaction_executing_insufficient_funds_for_rent | counter | Result of executing a transaction. Could be a failure or success. (Transaction results in an account with insufficient funds for rent.) |
| bank_tile_transaction_executing_max_loaded_accounts_data_size_exceeded | counter | Result of executing a transaction. Could be a failure or success. (Transaction exceeded max loaded accounts data size cap.) |
| bank_tile_transaction_executing_invalid_loaded_accounts_data_size_limit | counter | Result of executing a transaction. Could be a failure or success. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_tile_transaction_executing_resanitization_needed | counter | Result of executing a transaction. Could be a failure or success. (Sanitized transaction differed before/after feature activiation. Needs to be resanitized.) |
| bank_tile_transaction_executing_program_execution_temporarily_restricted | counter | Result of executing a transaction. Could be a failure or success. (Program execution is temporarily restricted on an account.) |
| bank_tile_transaction_executing_unbalanced_transaction | counter | Result of executing a transaction. Could be a failure or success. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_tile_transaction_executing_program_cache_hit_max_limit | counter | Result of executing a transaction. Could be a failure or success. (The total program cache size hit the maximum allowed limit.) |
| bank_tile_transaction_executed_success | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Success) |
| bank_tile_transaction_executed_account_in_use | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (An account is already being processed in another transaction in a way that does not support parallelism.) |
| bank_tile_transaction_executed_account_loaded_twice | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (A Pubkey appears twice in the transaction's account_keys. Instructions can reference Pubkeys more than once but the message must contain a list with no duplicate keys.) |
| bank_tile_transaction_executed_account_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempt to debit an account but found no record of a prior credit.) |
| bank_tile_transaction_executed_program_account_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempt to load a program that does not exist.) |
| bank_tile_transaction_executed_insufficient_funds_for_fee | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The fee payer Pubkey does not have sufficient balance to pay the fee to schedule the transaction.) |
| bank_tile_transaction_executed_invalid_account_for_fee | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (This account may not be used to pay transaction fees.) |
| bank_tile_transaction_executed_already_processed | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The bank has seen this transaction before. This can occur under normal operation when a UDP packet is duplicated, as a user error from a client not updating its recent_blockhash, or as a double-spend attack.) |
| bank_tile_transaction_executed_blockhash_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The bank has not seen the given recent_blockhash or the transaction is too old and the recent_blockhash has been discarded.) |
| bank_tile_transaction_executed_instruction_error | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (An error occurred while processing an instruction.) |
| bank_tile_transaction_executed_call_chain_too_deep | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Loader call chain is too deep.) |
| bank_tile_transaction_executed_missing_signature_for_fee | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction requires a fee but has no signature present.) |
| bank_tile_transaction_executed_invalid_account_index | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction contains an invalid account reference.) |
| bank_tile_transaction_executed_signature_failure | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction did not pass signature verification.) |
| bank_tile_transaction_executed_invalid_program_for_execution | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (This program may not be used for executing instructions.) |
| bank_tile_transaction_executed_sanitize_failure | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction failed to sanitize accounts offsets correctly implies that account locks are not taken for this TX, and should not be unlocked.) |
| bank_tile_transaction_executed_cluster_maintenance | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transactions are currently disabled due to cluster maintenance.) |
| bank_tile_transaction_executed_account_borrow_outstanding | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction processing left an account with an outstanding borrowed reference.) |
| bank_tile_transaction_executed_would_exceed_max_block_cost_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed max Block Cost Limit.) |
| bank_tile_transaction_executed_unsupported_version | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction version is unsupported.) |
| bank_tile_transaction_executed_invalid_writable_account | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction loads a writable account that cannot be written.) |
| bank_tile_transaction_executed_would_exceed_max_account_cost_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed max account limit within the block.) |
| bank_tile_transaction_executed_would_exceed_account_data_block_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed account data limit within the block.) |
| bank_tile_transaction_executed_too_many_account_locks | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction locked too many accounts.) |
| bank_tile_transaction_executed_address_lookup_table_not_found | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Address lookup table not found.) |
| bank_tile_transaction_executed_invalid_address_lookup_table_owner | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempted to lookup addresses from an account owned by the wrong program.) |
| bank_tile_transaction_executed_invalid_address_lookup_table_data | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Attempted to lookup addresses from an invalid account.) |
| bank_tile_transaction_executed_invalid_address_lookup_table_index | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Address table lookup uses an invalid index.) |
| bank_tile_transaction_executed_invalid_rent_paying_account | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction leaves an account with a lower balance than rent-exempt minimum.) |
| bank_tile_transaction_executed_would_exceed_max_vote_cost_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed max Vote Cost Limit.) |
| bank_tile_transaction_executed_would_exceed_account_data_total_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction would exceed total account data limit.) |
| bank_tile_transaction_executed_duplicate_instruction | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction contains a duplicate instruction that is not allowed.) |
| bank_tile_transaction_executed_insufficient_funds_for_rent | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction results in an account with insufficient funds for rent.) |
| bank_tile_transaction_executed_max_loaded_accounts_data_size_exceeded | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Transaction exceeded max loaded accounts data size cap.) |
| bank_tile_transaction_executed_invalid_loaded_accounts_data_size_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (LoadedAccountsDataSizeLimit set for transaction must be greater than 0.) |
| bank_tile_transaction_executed_resanitization_needed | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Sanitized transaction differed before/after feature activiation. Needs to be resanitized.) |
| bank_tile_transaction_executed_program_execution_temporarily_restricted | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (Program execution is temporarily restricted on an account.) |
| bank_tile_transaction_executed_unbalanced_transaction | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The total balance before the transaction does not equal the total balance after the transaction.) |
| bank_tile_transaction_executed_program_cache_hit_max_limit | counter | When a transaction executes (makes it onto the chain), result of executing a transaction. The transaction can still fail. (The total program cache size hit the maximum allowed limit.) |
Poh Tile
| Metric | Type | Description |
|---|---|---|
| poh_tile_begin_leader_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the bank. |
| poh_tile_first_microblock_delay_seconds | histogram | Delay between when we become leader in a slot and when we receive the first microblock. |
| poh_tile_slot_done_delay_seconds | histogram | Delay between when we become leader in a slot and when we finish the slot. |
Store Tile
| Metric | Type | Description |
|---|---|---|
| store_tile_transactions_inserted | counter | Count of transactions produced while we were leader in the shreds that have been inserted so far |
Shred Tile
| Metric | Type | Description |
|---|---|---|
| shred_cluster_contact_info_cnt | histogram | Number of contact infos in the cluster contact info message |
| shred_microblocks_abandoned | counter | The number of microblocks that were abandoned because we switched slots without finishing the current slot |
| shred_batch_sz | histogram | The size (in bytes) of each microblock batch that is shredded |
| shred_batch_microblock_cnt | histogram | The number of microblocks in each microblock batch that is shredded |
| shred_shredding_duration_seconds | histogram | Duration of producing one FEC set from the shredder |
| shred_add_shred_duration_seconds | histogram | Duration of verifying and processing one shred received from the network |
| shred_shred_processed_bad_slot | counter | The result of processing a thread from the network (Shred was for a slot for which we don't know the leader) |
| shred_shred_processed_parse_failed | counter | The result of processing a thread from the network (Shred parsing failed) |
| shred_shred_processed_rejected | counter | The result of processing a thread from the network (Shred was invalid for one of many reasons) |
| shred_shred_processed_ignored | counter | The result of processing a thread from the network (Shred was ignored because we had already received or reconstructed it) |
| shred_shred_processed_okay | counter | The result of processing a thread from the network (Shred accepted to an incomplete FEC set) |
| shred_shred_processed_completes | counter | The result of processing a thread from the network (Shred accepted and resulted in a valid, complete FEC set) |
| shred_fec_set_spilled | counter | The number of FEC sets that were spilled because they didn't complete in time and we needed space |
| shred_shred_rejected_initial | counter | The number shreds that were rejected before any resources were allocated for the FEC set |
| shred_fec_rejected_fatal | counter | The number of FEC sets that were rejected for reasons that cause the whole FEC set to become invalid |