Use OpenTelemetry

You can use the OpenTelemetry monitoring and tracing service to gather node metrics and traces. To enable OpenTelemetry to access Hyperledger Besu, use the --metrics-enabled and --metrics-protocol=opentelemetry options. Use Splunk to visualize the collected data. A Besu Sync example is available.

Install OpenTelemetry Collector

Download and install the OpenTelemetry Collector.

Setting up and running OpenTelemetry with Besu

1. Configure OpenTelemetry to accept data from Besu. For example, use the following configuration for your otel-collector-config.yml file, and send data to Splunk and Splunk APM:

   otel-collector-config.yml

   receivers:
       otlp:
           protocols:
               grpc:
               http:
   
   exporters:
       splunk_hec/traces:
           # Splunk HTTP Event Collector token.
           token: "11111111-1111-1111-1111-1111111111113"
           # URL to a Splunk instance to send data to.
           endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"
           # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
           source: "besu:traces"
           # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
           sourcetype: "otlp"
           # Splunk index, optional name of the Splunk index targeted.
           index: "traces"
           # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
           max_connections: 20
           # Whether to disable gzip compression over HTTP. Defaults to false.
           disable_compression: false
           # HTTP timeout when sending data. Defaults to 10s.
           timeout: 10s
           # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
           # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
           insecure_skip_verify: true
       splunk_hec/metrics:
           # Splunk HTTP Event Collector token.
           token: "11111111-1111-1111-1111-1111111111113"
           # URL to a Splunk instance to send data to.
           endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"
           # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
           source: "besu:metrics"
           # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
           sourcetype: "prometheus"
           # Splunk index, optional name of the Splunk index targeted.
           index: "metrics"
           # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.
           max_connections: 20
           # Whether to disable gzip compression over HTTP. Defaults to false.
           disable_compression: false
           # HTTP timeout when sending data. Defaults to 10s.
           timeout: 10s
           # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.
           # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.
           insecure_skip_verify: true
       # Traces
       sapm:
           access_token: "${SPLUNK_ACCESS_TOKEN}"
           endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace"
       # Metrics + Events
       signalfx:
           access_token: "${SPLUNK_ACCESS_TOKEN}"
           realm: "${SPLUNK_REALM}"
   
   processors:
       batch:
   
   extensions:
       health_check:
       pprof:
       zpages:
   
   service:
       extensions: [pprof, zpages, health_check]
       pipelines:
           traces:
               receivers: [otlp]
               exporters: [splunk_hec/traces, sapm]
               processors: [batch]
           metrics:
               receivers: [otlp]
               exporters: [splunk_hec/metrics, signalfx]
               processors: [batch]
   

   It is easiest to run the OpenTelemetry collector with Docker with the following command:

   SyntaxExample

   docker run -d \
     -v ./otel-collector-config.yml:/etc/otel/config.yaml \
     -e SPLUNK_ACCESS_TOKEN=<access token> \
     -e SPLUNK_REALM=<realm> \
     -p 4317:4317 \
     otel/opentelemetry-collector-contrib:latest
   

   docker run -d \
     -v ./otel-collector-config.yml:/etc/otel/config.yaml \
     -e SPLUNK_ACCESS_TOKEN=abcdefg654 \
     -e SPLUNK_REALM=us1 \
     -p 4317:4317 \
     otel/opentelemetry-collector-contrib:latest
   

   You can also refer to this Docker-compose example.

2. Start Besu with the --metrics-enabled and --metrics-protocol=opentelemetry options. For example, run the following command to start a single node:

   SyntaxExample

   OTEL_EXPORTER_OTLP_ENDPOINT=https://<host>:<port> besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry
   

   OTEL_EXPORTER_OTLP_ENDPOINT=https://localhost:4317 besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry
   

   The OpenTelemetry SDK mandates how to configure the OpenTelemetry gRPC client, so data flows to the collector from Besu.

   You can use the following environment variables:

   Name	Description	Required
   OTEL_EXPORTER_OTLP_ENDPOINT	OpenTelemetry Collector endpoint, of the form https://host:port. The default value is https://localhost:4317	Yes
   OTEL_EXPORTER_OTLP_INSECURE	Whether to allow insecure connections for OpenTelemetry data. False by default.	No