Preface: Letters from Readers is a question and answer column created by the old HBase store. It aims to help more people solve hBase-related problems that they often encounter in work. The old shop will try to help you solve these problems or help you to send out distress stickers, the old shop hopes that this will be a small platform for mutual help. Have a problem, please directly in the old shop backstage message, there is a good solution, please don’t be mean, sincerely welcome you can actively explore solutions in the message area, boldly express their views, perhaps you help others to solve the problem today, is you may encounter tomorrow the answer.
From: Liu * Gang
Small apes questions
During the restart of the HBase cluster, all RS nodes are successfully started, but HMaser cannot be started. The error log is as follows:
unexpected error, closing socket connection and attempting reconnect java.io.IOException: Packet len4745468 is out of range! at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at Org. Apache. Zookeeper. ClientCnxn $SendThread. Run (1081) ClientCnxn. Java: the 2020-04-02 22:31:08, 673 ERROR [hadoop01:16000.activeMasterManager] zookeeper.RecoverableZooKeeper: ZooKeeper getChildren failed after 4 attempts the 22:31:08 2020-04-02, 674 FATAL [hadoop01:16000 activeMasterManager] master.HMaster: Failed to become active master org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/region-in-transition at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.getChildren(RecoverableZooKeeper.java:295) at org.apache.hadoop.hbase.zookeeper.ZKUtil.listChildrenNoWatch(ZKUtil.java:513) at org.apache.hadoop.hbase.master.AssignmentManager.processDeadServersAndRegionsInTransition(AssignmentManager.java:519) at org.apache.hadoop.hbase.master.AssignmentManager.joinCluster(AssignmentManager.java:494) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:748) at org.apache.hadoop.hbase.master.HMaster.access$500(HMaster.java:184) at org.apache.hadoop.hbase.master.HMaster$1.run(HMaster.java:1729) at java.lang.Thread.run(Thread.java:748)Copy the codeThe ape analysis
- HBase version: Apache 1.2.1
- Cluster size: 120,000 + Region
See the error log, seemed to see only the ZK, log keywords is [ZooKeeper. GetChildren | Packet | out of range | ConnectionLoss for/hbase/region – in – the transition]. As we know, the HBase Master needs to do a lot of initialization work when it restarts. It needs to interact with the ZK data node, such as registering, modifying, and obtaining metadata or node status. ZooKeeper Failed to become active master when getChildren (region-in-transition) exceeded the Packet range.
So what is a Packet? Little Ape asked Baidu, Baidu replied:
In ZooKeeper, Packet is a minimal communication protocol unit, namely Packet. Pakcet is used for network transmission between client and server. Any object to be transmitted needs to be packaged as a Packet object.
That is to read the ZK node packet length is limited, this time we must first go to the Internet to find the zK parameters can be adjusted. And it did: Jute. Maxbuffer, feeling lucky. To paraphrase this parameter from the official website:
(Java system property: jute.maxbuffer) This option can only be set as a Java system property. There is no zookeeper prefix on it. It specifies the maximum size of the data that can be stored in a znode. The default is 0xfffff, or just under 1M. If this option is changed, the system property must be set on all servers and clients otherwise problems will arise. This is really a sanity check. ZooKeeper is designed to store data on the order of kilobytes in size.
Translation:
(Java system properties: jute.maxBuffer) This option can only be set to Java system properties. There is no Zookeeper prefix on it. It specifies the maximum size of data that can be stored in zNode. The default is 0xFFfff, or less than 1M. If you change this option, system properties must be set on all servers and clients, or problems will occur. This is really a health check. ZooKeeper is designed to store data in kilobytes.
There’s another way to say it:
Note that this parameter does not take effect on both the Server and Client. In fact, Zookeeper controls the size of the data read from the Server (outgoingBuffer) after the client is set. On the server side, it controls the size of the data written from the Client (incomingBuffer).
The relevant codes are as follows:
protected final ByteBuffer lenBuffer = ByteBuffer.allocateDirect(4); protected ByteBuffer incomingBuffer = lenBuffer; protected void readLength() throws IOException { int len = incomingBuffer.getInt(); if (len < 0 || len >= ClientCnxn.packetLen) { throw new IOException("Packet len" + len + " is out of range!" ); } incomingBuffer = ByteBuffer.allocate(len); } public static final int packetLen = Integer.getInteger("jute.maxbuffer", 4096 * 1024);Copy the codeSo why is it reading such a big package? Based on the aforementioned keyword /hbase/region-in-transition and the size of regions to be allocated (120,000 +), we guess that there are too many regions. As a result, the /hbase/region-in-transition node is too large, and the HMaster fails to read data from the node. We also found a related issue in the HBase Jira library: Cluster with too many regions always withstand some master failover scenarios issues.apache.org/jira/browse…
Most of us are not the first to get our shoes wet. Maybe the problem you solve for someone today is the answer you may encounter tomorrow. This is also the original intention of the old store to set up the q&A column “Letters from Readers” — in order to better spread and share knowledge!
Little ape to answer
This problem is not limited to /region-in-transition nodes, of course. Nodes such as /unssigned can also have this problem. The solution is as follows: Solution 1: Clear historical junk data on zK nodes
This scheme aims to reduce the data size of ZK nodes, whether it can be reduced below the red line.
Scheme 2: Increase the parameter jute. Maxbuffer
$vim $ZOOKEEPER_HOME/bin/ zkcli. sh $djute. maxbuffer=<buffer_size> parameter "$JAVA" "-Dzookeeper.log.dir=${ZOO_LOG_DIR}" "-Dzookeeper.root.logger=${ZOO_LOG4J_PROP}" "-Djute.maxbuffer=1073741824" \ -cp "$CLASSPATH" $$JVMFLAGS CLIENT_JVMFLAGS \ org apache. The zookeeper. ZooKeeperMain "$@ # $vim set the Server side $ZOOKEEPER_HOME/conf/zoo. CFG # Add jute.maxbuffer=<buffer_size> jute. maxBuffer =1073741824Copy the codeIncreasing this parameter may be risky, as mentioned above zK is designed to store data in kilobytes.
Scenario 3: Use hierarchy (from community comment section)
In this solution, the ·/hbase/region-in-transition directory is fragmented using the prefix of the region ID. Area, for example, 1234567890 abcdef will be located in/hbase/region – in – the transition / 1234/1234567890 abcdef. Therefore, we have to iterate to get the full list.
reference
- Issues.apache.org/jira/browse…
- Cloud.tencent.com/developer/a…
- Yuzhouwan.com/posts/31915…
Reprint please indicate the source! Welcome to follow my wechat official account [HBase Work Notes]