Updated 2019-10-03

Monitoring Hyperledger Besu

Monitor Node Performance and Connectivity Using the JSON-RPC API

You can monitor node performance using the debug_metrics JSON-RPC API method.

Monitor Node Performance Using Prometheus

Use the --metrics-enabled option to enable the Prometheus monitoring and alerting service to access Besu metrics. You can also visualize the collected data using Grafana. A sample Besu Grafana dashboard is provided.

To specify the host and port on which Prometheus accesses Besu, use the --metrics-host and --metrics-port options. The default host and port are 127.0.0.1 and 9545.

Prometheus requires 3MB of space per node per hour for metrics, with a scrape_interval of 15s.

To use Prometheus with Besu, install the prometheus main component. On MacOS, install with Homebrew:

brew install prometheus

Tip

You can also install:

  • Exporters and send system metrics to Prometheus to monitor non-Pantheon specific items such as disk usage and CPU usage.
  • Other Prometheus components such as the Alert Manager. Additional configuration is not required for these components because Prometheus handles and analyzes data directly from the feed.

Setting up and Running Prometheus with Besu

To configure Prometheus and run with Besu:

  1. Configure Prometheus to poll Besu. For example, add the following yaml fragment to the scrape_configs block of the prometheus.yml file:

      job_name: besu-dev
      scrape_interval: 15s
      scrape_timeout: 10s
      metrics_path: /metrics
      scheme: http
      static_configs:
      - targets:
        - localhost:9545
    
  2. Start Besu with the --metrics-enabled option. To start a single node for testing with metrics enabled:

    besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73
    --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled
    
  3. In another terminal, run Prometheus specifying the prometheus.yml file:

    prometheus --config.file=config.yml 
    
  4. Open a web browser to http://localhost:9090 to view the Prometheus graphical interface.

  5. Choose Graph from the menu bar and click the Console tab below.

  6. From the Insert metric at cursor drop-down, select a metric such as besu_blockchain_difficulty_total or ethereum_blockchain_height and click Execute. The values are displayed.

    Standard Ethereum metrics are prefixed with ethereum_ and listed in the table below. Metrics specific to Besu are prefixed with besu_.

    Name Metric Type Definition JSON-RPC Equivalent
    ethereum_blockchain_height Gauge Current height of the canonical chain eth_blockNumber
    ethereum_best_known_block_number Gauge Estimated highest block available highestBlock of eth_syncing or eth_blockNumber, if not syncing
    ethereum_peer_count Gauge Current number of peers connected net_peerCount
    ethereum_peer_limit Gauge Maximum number of peers this node allows to connect No equivalent

    Important

    • The ethereum_best_known_block_number metric always has a value. When the eth_syncing JSON-RPC method returns false, the current chain height is used.
    • Although the ethereum_peer_limit metric does not have a JSON-RPC equivalent, the max peers command line option sets the maximum number of P2P connections that can be established.

    Click the Graph tab to view the data as a time-based graph. The query string is displayed below the graph. For example: {ethereum_blockchain_height{instance="localhost:9545",job="prometheus"}

Tip

Use a log ingestion tool such as Logstash to parse the logs and alert you to configured anomalies.

Running Prometheus with Besu in Push Mode

The --metrics-enabled option enables Prometheus polling Besu but sometimes metrics are hard to poll (for example, when running inside Docker containers with varying IP addresses). The --metrics-push-enabled option enables Besu to push metrics to a Prometheus Pushgateway.

To configure Prometheus and run with Besu pushing to a push gateway:

  1. Configure Prometheus to read from a push gateway. For example, add the following yaml fragment to the scrape_configs block of the prometheus.yml file:

     - job_name: push-gateway
       metrics_path: /metrics
       scheme: http
       static_configs:
       - targets:
         - localhost:9091
    
  2. Start the push gateway. The push gateway can be deployed using the Docker image:

    docker pull prom/pushgateway
    docker run -d -p 9091:9091 prom/pushgateway
    
  3. Start Besu specifying the --metrics-push-enabled option and port of the push gateway:

    besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-push-enabled --metrics-push-port=9091 --metrics-push-host=127.0.0.1
    
  4. In another terminal, run Prometheus specifying the prometheus.yml file:

    prometheus --config.file=config.yml 
    
  5. View the Prometheus graphical interface as described in Setting up and Running Prometheus with Besu.

Monitor Node Performance and Connectivity Using the JSON-RPC API

You can monitor node performance using the debug_metrics JSON-RPC API method.