Skip to main content

Use metrics to monitor node performance

To enable the Prometheus monitoring and alerting service to access Hyperledger Besu metrics, use the --metrics-enabled option. Use Grafana to visualize the collected data. See the sample Besu Full Grafana dashboard.

The Besu example networks have monitoring with Prometheus and Grafana configured.

Use Prometheus to monitor the number of blocks your Besu node is behind the chain head, and to alert you that your node is not keeping up with the chain head.

This recording shows examples of monitoring Hyperledger Besu.

Install Prometheus

To use Prometheus with Besu, install the Prometheus main component. On MacOS, install with Homebrew:

brew install prometheus
tip

You can also install:

  • Exporters that send system metrics to Prometheus to monitor non-Besu-specific items such as disk and CPU usage.
  • Other Prometheus components, such as the Alert Manager. Additional configuration is not required for these components because Prometheus handles and analyzes data directly from the feed.

Set up and run Prometheus with Besu

To configure Prometheus and run with Besu:

  1. Configure Prometheus to poll Besu. For example, add the following YAML fragment to the scrape_configs block of the prometheus.yml file:
- job_name: besu
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
static_configs:
- targets:
- localhost:9545

Prometheus requires 3 MB of space per node per hour for metrics, with a scrape_interval of 15 seconds.

  1. Start Besu with the --metrics-enabled option. To start a single node for testing with metrics enabled, run the following command:
besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled

To specify the host and port on which Prometheus accesses Besu, use the --metrics-host and --metrics-port options. The default host and port are 127.0.0.1 (localhost) and 9545.

danger

To avoid DNS rebinding attacks, if running Prometheus on a different host than your Besu node (any host other than localhost), add the hostname that Prometheus uses to --host-allowlist.

For example, if Prometheus is configured to get metrics from http://besu.local:8008/metrics, then besu.local has to be in --host-allowlist.

  1. In another terminal, run Prometheus specifying the prometheus.yml file:

    prometheus --config.file=prometheus.yml
  2. View the Prometheus graphical interface.

    tip

    Use a log ingestion tool, such as Logstash, to parse the logs and alert you to configured anomalies.

Run Prometheus with Besu in push mode

The --metrics-enabled option enables Prometheus polling of Besu, but sometimes metrics are hard to poll (for example, when running inside Docker containers with varying IP addresses). To enable Besu to push metrics to a Prometheus push gateway, use the --metrics-push-enabled option.

To configure Prometheus and run with Besu pushing to a push gateway:

  1. Configure Prometheus to read from a push gateway. For example, add the following YAML fragment to the scrape_configs block of the prometheus.yml file:

    - job_name: push-gateway
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
    - localhost:9091
  2. Start the push gateway. You can deploy the push gateway using the Docker image:

    docker pull prom/pushgateway
    docker run -d -p 9091:9091 prom/pushgateway
  3. Start Besu specifying the --metrics-push-enabled option and port of the push gateway:

besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-push-enabled --metrics-push-port=9091 --metrics-push-host=127.0.0.1
  1. In another terminal, run Prometheus specifying the prometheus.yml file:

    prometheus --config.file=prometheus.yml
  2. View the Prometheus graphical interface.

View Prometheus graphical interface

  1. Open a Web browser to http://localhost:9090 to view the Prometheus graphical interface.

  2. Choose Graph from the menu bar and click the Console tab below.

  3. From the Insert metric at cursor drop-down, select a metric such as besu_blockchain_difficulty_total or ethereum_blockchain_height and click Execute. The values display.

  4. Click the Graph tab to view the data as a time-based graph. The query string displays below the graph. For example, {ethereum_blockchain_height{instance="localhost:9545",job="prometheus"}.

Metrics list

The following table lists available metrics. Each metric starts with a metric category prefix. Metrics specific to Besu use the besu_ prefix, followed by another metric category. Metric categories can be enabled using the metrics-category command line option. If a metric has a JSON-RPC equivalent, it is included in the definition column.

NameMetric typeDefinition
besu_blockchain_chain_head_gas_limitGaugeBlock gas limit of the current chain head block
besu_blockchain_chain_head_gas_usedGaugeGas used by the current chain head block
besu_blockchain_chain_head_ommer_countGaugeNumber of uncles in the current chain head block (JSON-RPC equivalent: eth_getUncleCountByBlockHash or eth_getUncleCountByBlockNumber)
besu_blockchain_chain_head_timestampGaugeTimestamp from the current chain head
besu_blockchain_chain_head_transaction_countGaugeNumber of transactions in the current chain head block (JSON-RPC equivalent: eth_getBlockTransactionCountByHash or eth_getBlockTransactionCountByNumber)
besu_blockchain_difficulty_totalGaugeDifficulty of the chain head (JSON-RPC equivalent: difficulty of admin_peers)
besu_executors_ethscheduler_computation_active_threads_currentGaugeCurrent number of threads executing computation tasks
besu_executors_ethscheduler_computation_completed_tasks_totalGaugeTotal number of computation tasks executed
besu_executors_ethscheduler_computation_pool_size_currentGaugeCurrent number of threads in the computation thread pool
besu_executors_ethscheduler_computation_queue_length_currentGaugeCurrent number of computation tasks awaiting execution
besu_executors_ethscheduler_computation_rejected_tasks_totalCounterTotal number of tasks rejected by this computation executor
besu_executors_ethscheduler_computation_submitted_tasks_totalGaugeTotal number of computation tasks submitted
besu_executors_ethscheduler_timer_active_threads_currentGaugeCurrent number of threads executing timer tasks
besu_executors_ethscheduler_timer_completed_tasks_totalGaugeTotal number of timer tasks executed
besu_executors_ethscheduler_timer_pool_size_currentGaugeCurrent number of threads in the timer thread pool
besu_executors_ethscheduler_timer_queue_length_currentGaugeCurrent number of timer tasks awaiting execution
besu_executors_ethscheduler_timer_rejected_tasks_totalCounterTotal number of tasks rejected by this timer executor
besu_executors_ethscheduler_timer_submitted_tasks_totalGaugeTotal number of timer tasks submitted
besu_executors_ethscheduler_workers_active_threads_currentGaugeCurrent number of threads executing worker tasks
besu_executors_ethscheduler_workers_completed_tasks_totalGaugeTotal number of worker tasks executed
besu_executors_ethscheduler_workers_pool_size_currentGaugeCurrent number of threads in the worker thread pool
besu_executors_ethscheduler_workers_queue_length_currentGaugeCurrent number of worker tasks awaiting execution
besu_executors_ethscheduler_workers_rejected_tasks_totalCounterTotal number of tasks rejected by this worker executor
besu_executors_ethscheduler_workers_submitted_tasks_totalGaugeTotal number of worker tasks submitted
besu_network_discovery_inflight_interactions_currentGaugeCurrent number of inflight discovery interactions
besu_network_discovery_interaction_countCounterTotal number of discovery interactions initiated
besu_network_discovery_interaction_retry_countCounterTotal number of interaction retries performed
besu_network_discovery_messages_inboundCounterTotal number of P2P discovery messages received
besu_network_discovery_messages_outboundCounterTotal number of P2P discovery messages sent
besu_network_netty_boss_pending_tasksGaugeNumber of pending tasks in Netty boss event loop
besu_network_netty_workers_pending_tasksGaugeNumber of pending tasks in Netty workers event loop
besu_network_p2p_messages_inboundCounterTotal number of P2P messages received
besu_network_vertx_eventloop_pending_tasksGaugeNumber of pending tasks in Vertx event loop
besu_network_vertx_worker_pool_completed_totalCounterTotal number of tasks completed by Vertx worker pool
besu_network_vertx_worker_pool_rejected_totalCounterTotal number of tasks rejected by Vertx worker pool
besu_network_vertx_worker_pool_submitted_totalCounterTotal number of tasks submitted to Vertx worker pool
besu_peers_connected_totalCounterTotal number of peers connected
besu_peers_disconnected_totalCounterTotal number of peers disconnected
besu_peers_pending_peer_requests_currentGaugeCurrent number of peer requests pending because peers are busy
besu_pruner_mark_time_durationGaugeCumulative number of seconds spent marking the state trie across all pruning cycles
besu_pruner_mark_operations_totalCounterTotal number of mark operations performed
besu_pruner_marked_nodes_totalCounterTotal number of nodes marked as in use
besu_pruner_sweep_operations_totalCounterTotal number of sweep operations performed
besu_pruner_swept_nodes_totalCounterTotal number of unused nodes removed
besu_stratum_connectionsCounterNumber of connections over time
besu_stratum_difficultyGaugeCurrent mining difficulty
besu_stratum_disconnectionsCounterNumber of disconnections over time
besu_stratum_minersGaugeNumber of connected miners
besu_synchronizer_chain_download_pipeline_processed_totalCounterNumber of entries processed by each chain download pipeline stage
besu_synchronizer_chain_download_pipeline_restartsCounterNumber of times chain download pipeline has been restarted
besu_synchronizer_fast_sync_pivot_block_currentGaugeThe current fast sync pivot block
besu_synchronizer_fast_sync_pivot_block_selected_countCounterNumber of times a fast sync pivot block has been selected
besu_synchronizer_fast_sync_validation_modeCounterNumber of blocks validated using light vs full validation during fast sync
besu_synchronizer_in_syncGaugeWhether or not the local node has caught up to the best known peer (1 or 0)
besu_synchronizer_taskSummaryInternal processing tasks
besu_synchronizer_world_state_completed_requests_totalCounterTotal number of node data requests completed as part of fast sync world state download
besu_synchronizer_world_state_existing_nodes_totalCounterTotal number of node data requests completed using existing data
besu_synchronizer_world_state_inflight_requests_currentGaugeNumber of in progress requests for world state data
besu_synchronizer_world_state_node_requests_since_last_progress_currentGaugeNumber of world state requests made since the last time new data was returned
besu_synchronizer_world_state_pending_requests_cache_sizeGaugePending request cache size for fast sync world state download
besu_synchronizer_world_state_pending_requests_currentGaugeNumber of pending requests for fast sync world state download
besu_synchronizer_world_state_pipeline_processed_totalCounterNumber of entries processed by each world state download pipeline stage
besu_synchronizer_world_state_retried_requests_totalCounterTotal number of node data requests repeated as part of fast sync world state download
besu_transaction_pool_pending_transactions_messages_skipped_totalCounterTotal number of pending transactions messages skipped by the processor
besu_transaction_pool_transactionsGaugeCurrent size of the transaction pool (JSON-RPC equivalent: result number of txpool_besuTransactions)
besu_transaction_pool_transactions_added_totalCounterCount of transactions added to the transaction pool
besu_transaction_pool_transactions_messages_skipped_totalCounterTotal number of transactions messages skipped by the processor.
ethereum_best_known_block_numberGaugeEstimated highest block available (JSON-RPC equivalent: highestBlock of eth_syncing, or eth_blockNumber if not syncing)
ethereum_blockchain_heightGaugeCurrent height of the canonical chain (JSON-RPC equivalent: eth_blockNumber)
ethereum_peer_countGaugeCurrent number of peers connected (JSON-RPC equivalent: net_peerCount)
ethereum_peer_limitGaugeMaximum number of peers this node allows to connect
jvm_buffer_pool_capacity_bytesGaugeBytes capacity of a given JVM buffer pool
jvm_buffer_pool_used_buffersGaugeUsed buffers of a given JVM buffer pool
jvm_buffer_pool_used_bytesGaugeUsed bytes of a given JVM buffer pool
jvm_classes_loadedGaugeCurrent number of classes loaded in the JVM
jvm_classes_loaded_totalCounterTotal number of classes loaded since the JVM started execution
jvm_classes_unloaded_totalCounterTotal number of classes unloaded since the JVM started execution
jvm_gc_collection_secondsSummarySeconds spent in a given JVM garbage collector
jvm_memory_bytes_committedGaugeCommitted bytes of a given JVM memory area
jvm_memory_bytes_initGaugeInitial bytes of a given JVM memory area
jvm_memory_bytes_maxGaugeMaximum bytes of a given JVM memory area
jvm_memory_bytes_usedGaugeUsed bytes of a given JVM memory area
jvm_memory_pool_bytes_committedGaugeCommitted bytes of a given JVM memory pool
jvm_memory_pool_bytes_initGaugeInitial bytes of a given JVM memory pool
jvm_memory_pool_bytes_maxGaugeMaximum bytes of a given JVM memory pool
jvm_memory_pool_bytes_usedGaugeUsed bytes of a given JVM memory pool
jvm_threads_currentGaugeCurrent thread count of a JVM
jvm_threads_daemonGaugeDaemon thread count of a JVM
jvm_threads_deadlockedGaugeCycles of JVM threads in deadlock waiting to acquire object monitors or ownable synchronizers
jvm_threads_deadlocked_monitorGaugeCycles of JVM threads in deadlock waiting to acquire object monitors
jvm_threads_peakGaugePeak thread count of a JVM
jvm_threads_started_totalCounterStarted thread count of a JVM
jvm_threads_stateGaugeCurrent count of threads by state
process_cpu_seconds_totalCounterTotal user and system CPU time spent in seconds
process_max_fdsGaugeMaximum number of open file descriptors
process_open_fdsGaugeNumber of open file descriptors
process_start_time_secondsGaugeStart time of the process since Unix epoch in seconds
info
  • The ethereum_best_known_block_number metric always has a value. When the eth_syncing JSON-RPC method returns false, the current chain height displays.
  • Although the ethereum_peer_limit metric does not have a JSON-RPC equivalent, the max peers command line option sets the maximum number of P2P connections that can be established.