Use metrics to monitor node performance

To enable the Prometheus monitoring and alerting service to access Hyperledger Besu metrics, use the --metrics-enabled option. Use Grafana to visualize the collected data. See the sample Besu Grafana dashboard.

The Besu example networks have monitoring with Prometheus and Grafana configured.

Install Prometheus

To use Prometheus with Besu, install the Prometheus main component. On MacOS, install with Homebrew:

brew install prometheus

Set up and run Prometheus with Besu

To configure Prometheus and run with Besu:

1. Configure Prometheus to poll Besu. For example, add the following YAML fragment to the scrape_configs block of the prometheus.yml file:

   Example

   Fragment to insert in prometheus.ymlFull prometheus.yml example

     - job_name: besu
       scrape_interval: 15s
       scrape_timeout: 10s
       metrics_path: /metrics
       scheme: http
       static_configs:
       - targets:
         - localhost:9545
   

   global:
     scrape_interval: 15s
   
   scrape_configs:
     - job_name: "prometheus"
       static_configs:
       - targets: ["localhost:9090"]
     - job_name: besu
       scrape_interval: 15s
       scrape_timeout: 10s
       metrics_path: /metrics
       scheme: http
       static_configs:
       - targets:
         - localhost:9545
   

   Prometheus requires 3 MB of space per node per hour for metrics, with a scrape_interval of 15 seconds.

2. Start Besu with the --metrics-enabled option. To start a single node for testing with metrics enabled, run the following command:

   SyntaxExample

   besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled
   

   besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled
   

   To specify the host and port on which Prometheus accesses Besu, use the --metrics-host and --metrics-port options. The default host and port are 127.0.0.1 (localhost) and 9545.

   Important

   To avoid DNS rebinding attacks, if running Prometheus on a different host than your Besu node (any host other than localhost), add the hostname that Prometheus uses to --host-allowlist.

   For example, if Prometheus is configured to get metrics from http://besu.local:8008/metrics, then besu.local has to be in --host-allowlist.

3. In another terminal, run Prometheus specifying the prometheus.yml file:

   prometheus --config.file=prometheus.yml
   

4. View the Prometheus graphical interface.

   Tip

   Use a log ingestion tool, such as Logstash, to parse the logs and alert you to configured anomalies.

Run Prometheus with Besu in push mode

The --metrics-enabled option enables Prometheus polling of Besu, but sometimes metrics are hard to poll (for example, when running inside Docker containers with varying IP addresses). To enable Besu to push metrics to a Prometheus Pushgateway, use the --metrics-push-enabled option.

To configure Prometheus and run with Besu pushing to a push gateway:

1. Configure Prometheus to read from a push gateway. For example, add the following YAML fragment to the scrape_configs block of the prometheus.yml file:

   - job_name: push-gateway
     metrics_path: /metrics
     scheme: http
     static_configs:
     - targets:
       - localhost:9091
   

2. Start the push gateway. You can deploy the push gateway using the Docker image:

   docker pull prom/pushgateway
   docker run -d -p 9091:9091 prom/pushgateway
   

3. Start Besu specifying the --metrics-push-enabled option and port of the push gateway:

   SyntaxExample

   besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-push-enabled --metrics-push-port=9091 --metrics-push-host=127.0.0.1
   

   besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-push-enabled --metrics-push-port=9091 --metrics-push-host=127.0.0.1
   

4. In another terminal, run Prometheus specifying the prometheus.yml file:

   prometheus --config.file=prometheus.yml
   

5. View the Prometheus graphical interface.

View Prometheus graphical interface

1. Open a Web browser to http://localhost:9090 to view the Prometheus graphical interface.

2. Choose Graph from the menu bar and click the Console tab below.

3. From the Insert metric at cursor drop-down, select a metric such as besu_blockchain_difficulty_total or ethereum_blockchain_height and click Execute. The values display.

4. Click the Graph tab to view the data as a time-based graph. The query string displays below the graph. For example, {ethereum_blockchain_height{instance="localhost:9545",job="prometheus"}.

Metrics list

The following table lists available metrics. Each metric starts with a metric category prefix. Metrics specific to Besu use the besu_ prefix, followed by another metric category. Metric categories can be enabled using the metrics-category command line option. If a metric has a JSON-RPC equivalent, it is included in the definition column.