Hyperledger Fabric is an o&M focused blockchain. Fabric has included features for o&M of peer and Orderer nodes since version 1.4. This tutorial describes how to configure o&M management services for Fabric network nodes and how to use Prometheus and statsD/Graphite to visually monitor the real-time performance metrics of each node in a Hyperledger Fabric network.

Java development relevant tutorial: Fabric block chain, rounding | Fabric block chain Node. JS development steps

1. Configure o&M services for Hyperledger Fabric nodes

Hyperledger Fabric 1.4 provides the following features for o&M apis of peer and Orderer nodes:

  • Log level management: / logSpec
  • Node health check: /healthz
  • Run monitoring metrics: /metrics

Configuring o&M services for Fabric blockchain nodes isn’t rocket science, but it’s not easy if you miss some details.

First, modify core.yaml to configure the o&M service of the peer node, mainly including the configuration of listening address and TLS (we temporarily disable this part).

Open core.yaml with an editor:

$ vi ~/fabric-samples/config/core.yaml
Copy the code

The following figure shows the default value of listenAddress for the O&M service on the peer node:

Now let’s start the BYFN network and try to access the log level API.

$ cd ~/fabric-samples/first-network
$ ./byfn.sh -m up
$ docker exec -it cli bash

# curl peer0.org1.example.com:9443/logspec
Copy the code

Unfortunately, the curl command returns the following error:

curl: (7) Failed to connect to peer0.org1.example.com port 9443: Connection refused
Copy the code

Let’s see why.

First check our operations service configuration and open the core.yaml file:

# vi /etc/hyperledger/fabric/core.yaml
Copy the code

As you may recall, we used 127.0.0.1 as the value of listenAddress, which means that the operations API cannot be accessed externally.

Let’s do what’s necessary inside the peer container to check again. We will use wget instead of curl this time because curl is not installed in the container.

$ docker exec -it peer0.org1.example.com bash

# wget peer0.org1.example.com:9443/logspec
Copy the code

As expected, the error message appears again:

Connecting to peer0.org1.example.com (peer0.org1.example.com)... failed: Connection refusedCopy the code

But if you connect with 127.0.0.1 it will work:

# wget 127.0.0.1:9443 / logspec
Copy the code

The results are as follows:

This indicates that we need to set up an o&M listening address.

To do this, modify the Docker-compose file to specify the CORE_OPERATIONS_LISTENADDRESS environment variable for each peer.

$ vi ~/fabric-samples/first-network/base/docker-compose-base.yaml
Copy the code

Refer to the following figure for modification:

Now let’s try to access the/logSpec API again, and don’t forget to restart the BYFN network.

$ docker exec -it cli bash

# curl peer0.org1.example.com:9443/logspec
Copy the code

The following output is displayed:

Similarly, we can check node health:

# curl peer1.org1.example.com:9443/healthz
Copy the code

The following output is displayed:

Finally, let’s enable the Prometheus indicator by setting ORE_METRICS_PROVIDER= Prometheus for each peer in docker-comement-base. yaml and modify core.yaml to declare the indicator provider as Prometheus:

If the configuration is correct, the current values of a set of monitoring indicators are displayed after you run the following command in the CLI container:

2. Monitor the running indicators of the Hyperledger Fabric network using Prometheus

First download the latest version of Prometheus and unzip it:

$curl - LO, https://github.com/prometheus/prometheus/releases/download/v2.7.1/prometheus-2.7.1.linux-amd64.tar.gz \ $tar - XVZF Prometheus - 2.7.1. Linux - amd64. Tar. GzCopy the code

The Prometheus. Yml file can be found in the Prometheus folder. We need to modify this file to capture Hyperledger Fabric running indicator data by adding job_name and targets to scrape_configs.

Now we can run Prometheus with Docker:

$ sudo docker run -d --name prometheus-server -p 9090:9090 \
  --restart always \
  -v /home/mccdev/prometheus/prometheus/prometheus.yml:/prometheus.yml \
  prom/prometheus \
  --config.file=/prometheus.yml
Copy the code

Note that since Docker always starts the container on a private network, we need to add the Prometheus container to the Fabric. Run the following command to view the fabric network:

$ docker inspect peer1.org1.example.com
Copy the code

Connect Prometheus to the Fabric network:

$ sudo docker network connect net_byfn 5b8cbf9d8fa6
Copy the code

Statistics can be accessed at http://localhost:9090/

To check that the peer’s running metrics are successfully scraped, type scrape_samples_dignity to see the result table, which should be non-empty.

3. Monitor the operating indicators of Hyperledger Fabric network with StatsD/Graphite

Now we switch to StatsD to visually monitor the performance metrics of the Fabric network, Prometheus using pull and StatsD using push.

First of all, we need to obtain the Docker image of Graphite and StatsD from here.

Then, start graphite container:

$ docker run -d\
 --name graphite\
 --restart=always\
 -p 80:80\
 -p 2003-2004:2003-2004\
 -p 2023-2024:2023-2024\
 -p 8125:8125/udp\
 -p 8126:8126\
 graphiteapp/graphite-statsd
Copy the code

Now it’s time to modify docker-copose-base.yaml. Each peer should set the following environment variables:

CORE_OPERATIONS_LISTENADDRESS
CORE_METRICS_PROVIDER
CORE_METRICS_STATSD_ADDRESS
CORE_METRICS_STATSD_NETWORK
CORE_METRICS_STATSD_PREFIX
Copy the code

We should also make sure to specify port 8125. The following is an example of the peer configuration:

peer0.org1.example.com: container_name: peer0.org1.example.com extends: file: peer-base.yaml service: peer-base environment: - CORE_PEER_ID=peer0.org1.example.com - CORE_PEER_ADDRESS=peer0.org1.example.com:7051 - CORE_PEER_GOSSIP_BOOTSTRAP=peer1.org1.example.com:7051 - CORE_PEER_GOSSIP_EXTERNALENDPOINT=peer0.org1.example.com:7051 -  CORE_PEER_LOCALMSPID=Org1MSP# metrics- CORE_OPERATIONS_LISTENADDRESS=peer0.org1.example.com:8125 - CORE_METRICS_PROVIDER=statsd - CORE_METRICS_STATSD_ADDRESS=graphite:8125 - CORE_METRICS_STATSD_NETWORK=udp - CORE_METRICS_STATSD_PREFIX=PEER_01 volumes: - /var/run/:/host/var/run/ - .. /crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/msp:/etc/hyperledger/fabric/msp - .. /crypto-config/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls:/etc/hyperledger/fabric/tls - peer0.org1.example.com:/var/hyperledger/production ports: - 7051:7051 - 7053:7053 - 8125:8125Copy the code

Note the ports used: for peer0, the port should be 8125:8125, for Peer1, 9125:8125, and so on.

Orderers should be configured as follows:

- ORDERER_OPERATIONS_LISTENADDRESS=orderer.example.com:8125
- ORDERER_METRICS_PROVIDER=statsd
- ORDERER_METRICS_STATSD_ADDRESS=graphite:8125
- ORDERER_METRICS_STATSD_NETWORK=udp
- ORDERER_METRICS_STATSD_PREFIX=ORDERER

ports:
   - 7125:8125
Copy the code

Now let’s start the BYFN network.

Remember to connect Graphite container to BYFN network, otherwise you will see the following error:

Error: error getting endorser client for channel: endorser client failed to 
connect to peer0.org1.example.com:7051: failed to create new connection: 
context deadline exceeded
Copy the code

Peer container log will show that Graphite was not found in the network:

Error: failed to initialize operations subystems: dial udp: lookup graphite 
on 127.0.0.11:53: no such host
Copy the code

Use the following command to connect the Graphite container to the fabric network:

$ sudo docker network connect net_byfn graphite
Copy the code

If all is well, you should be able to see the performance indicator at the following address:

http://localhost/metrics/index.json
Copy the code

The results are as follows:

Visit the following address to view the visualized data:

http://localhost/
Copy the code

The result page is as follows:

4. Tutorial summary

In this tutorial, you learned how to configure the O&M service API for hyperledger Fabric’s nodes and how to visually monitor hyperledger Fabric using Prometheus or StatsD. If you want to continue your in-depth study of Hyperledger Fabric blockchain code and application development, you can check out these two online interactive tutorials:

  • Hyperledger Fabric Blockchain Java development in detail
  • Hyperledger Fabric blockchain Node.js development in detail

Operation and Maintenance Service and Visual Monitoring of Hyperledger Fabric