background

OpenTSDB is a distributed, scalable, and open source time series database implemented in Java and based on HBase column storage and query.

  • Liverpoolfc.tv: opentsdb.net/

  • Official documentation: opentsdb.net/docs/build/…

  • GitHub Release: github.com/OpenTSDB/op…

System environment

Install on CentOS7. The virtual host information is as follows:

[root@hadoop6 local]# uname -aLinux hadoop6 3.10.0-1127. El7 x86_64#1 SMP Tue Mar 31 23:36:51 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@hadoop6 local]# cat /proc/versionLinux Version 3.10.0-1127.el7.x86_64 ([email protected]) (GCC version 4.8.5 20150623 (Red Hat 4.8.5-39)  (GCC) )#1 SMP Tue Mar 31 23:36:51 UTC 2020
[root@hadoop6 local]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
Copy the code

Install the prerequisite

OpenTSDB relies on HBase because it is based on HBase column storage. I use the existing big data environment here, and the components involved here are:

  • Hadoop3.3.0
  • HBase2.3.0
  • ZooKeeper

Configure and start Hadoop

[root@hadoop6 ~]# cd /usr/local/hadoop

# start the Hadoop
[root@hadoop6 hadoop]# ./sbin/start-all.sh

# JPS You can see five service processes in addition to Hadoop: NameNode, SecondaryNameNode, ResourceManager, NodeManager, and DataNode
[root@hadoop6 hadoop]# jps
5696 NameNode
6193 SecondaryNameNode
7208 Jps
6541 ResourceManager
6717 NodeManager
5934 DataNode
Copy the code

Visit http://hadoop6:50070/ to go to the Hadoop background management page.

Configure and start HBase

[root@hadoop6 ~]# cd /usr/local/hbase
[root@hadoop6 hbase]# vi ./conf/hbase-site.xml

# start HBase
[root@hadoop6 hbase]# ./bin/start-hbase.sh

# JPS can see that in addition to Hadoop's five services, there are three more h-starting processes: HRegionServer, HMaster, and HQuorumPeer
[root@hadoop6 hbase]# jps
5696 NameNode
68673 HRegionServer
6193 SecondaryNameNode
68277 HQuorumPeer
68429 HMaster
6541 ResourceManager
6717 NodeManager
68988 Jps
5934 DataNode
Copy the code

Visit http://hadoop6:16010/ to go to the HBase background management page.

Finally, verify ZooKeeper

# error
[root@hadoop6 hbase]# telnet localhost 2181-bash: Telnet: the command was not foundCheck whether telnet-server is installed
[root@hadoop6 hbase]# rpm -qa telnet-server

Install telnet-server if not installed
[root@hadoop6 hbase]# yum install telnet-server

Install Telnet again
[root@hadoop6 hbase]# yum install telnet

The ZooKeeper connection was successfully tested
[root@hadoop6 hbase]# telnet localhost 2181
Trying ::1%1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.
Copy the code

At this point, the dependency environment is set up

Note: Remember to turn off the firewall or allow the corresponding port.

# disable firewall
[root@hadoop6 ~]# systemctl stop firewalld.service
Copy the code

Download and install to start OpenTSDB

[root@hadoop6 local]Wget # https://github.com/OpenTSDB/opentsdb/releases/download/v2.4.1/opentsdb-2.4.1-1-20210902183110-root.noarch.rpm
[root@hadoop6 local]# yum localinstall opentsdb - against 2.4.1-1-20210902183110 - root. Noarch. RPM
Copy the code

Note: The installation directory is described as follows:

- /etc/opentsdb -configuration files, you know, this is a key place, eg: ZK configuration - / TMP/opentsdb-temporary cache files - /usr/share/opentsdb-application files, -/usr/share/opentsdb /bin - The "TSDB" startup script that launches a TSD or command line tools -/usr/share/opentsdb/lib-java JAR library files, Dependent libraries - / usr/share/opentsdb/plugins - the Location for the plugin files and dependencies, Plug-in - / usr/share/opentsdb/static - static files for the GUI, Web page static resource - / usr/share/opentsdb/tools - Scripts and other tools, tools - / var/log/opentsdb - Logs, logCopy the code

Create tables in HBase

SQL > create table
[root@hadoop6 local]# env COMPRESSION=NONE HBASE_HOME=/usr/local/hbase /usr/share/opentsdb/tools/create_table.sh

[root@hadoop6 local]# hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/ hadoop/share/hadoop/common/lib/slf4j - log4j12-1.7.25. Jar! /org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found bindingin [jar:file:/usr/local/ hbase/lib/client - facing - thirdparty/slf4j - log4j12-1.7.30. Jar! /org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell
Use "help" to get list of supported commands.
Use "exit"To quit this interactive shell. For Reference, both please visit: http://hbase.apache.org/2.0/book.html#shellVersion 2.3.0 re0e1382705c59d3fb3ad8f5bff720a9dc7120fb8, Mon Jul 6 22:27:43 UTC 2020 Took 0.0017 secondsThe # list command verifies that the table was created successfullyHbase (main):001:0> list TABLE TSDB tsdb-meta tsdb-tree tsdb-uid 4 row(s) Took 0.5124 seconds => ["tsdb"."tsdb-meta"."tsdb-tree"."tsdb-uid"]
hbase(main):002:0> 
Copy the code

You can also view the newly created four tables on the HBase background management page.

Start the OpenTSDB service

All set, let’s start the OpenTSDB service.

# clearly, opentsdb not installed to the system service, refer to https://github.com/OpenTSDB/opentsdb/blob/master/build-aux/rpm/systemd/opentsdb.service for configuration
[root@hadoop6 ~]# systemctl start opentsdb
Failed to start opentsdb.service: Unit not found.

Start the Opentsdb service directly to the installed directory
[root@hadoop6 ~]# /usr/share/opentsdb/etc/init.d/opentsdb startStarting opentsdb: [confirm]After all, opentsdb is a Java service. JPS can see the Main and TSDMain processes
[root@hadoop6 opentsdb]# jps
37584 Main
27761 HRegionServer
27525 HMaster
25174 DataNode
85207 TSDMain
25479 SecondaryNameNode
26153 NodeManager
27369 HQuorumPeer
24942 NameNode
25902 ResourceManager
11246 Jps
Copy the code

Verify the OpenTSDB service status

[root@hadoop6 opentsdb]# /usr/share/opentsdb/etc/init.d/opentsdb statusOpentsdb is dead, but the PID file exists [root@hadoop6 opentsdb]# tailf /var/log/opentsdb/opentsdb-hadoop6-opentsdb.err /usr/bin/tsdb: line 117:exec: Java: not found [root@hadoop6 opentsdb]# java -version
java version "1.8.0 comes with _261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b12, mixed mode)
[root@hadoop6 opentsdb]# echo $JAVA_HOME
/usr/local/jdk
[root@hadoop6 opentsdb]# ln -s /usr/local/jdk/bin/java /usr/bin/java
[root@hadoop6 opentsdb]# /usr/share/opentsdb/etc/init.d/opentsdb startStarting opentsdb: [confirm] [root@hadoop6 opentsdb]# /usr/share/opentsdb/etc/init.d/opentsdb statusOpentsdb (PID 85207) is running...Copy the code

Call it a day ~~

Visit: http://hadoop6:4242/, default to the chart page.

You can also view key indicator statistics, run logs, and version information.

A small summary

In general, using OpenTSDB as a storage architecture for timing data involves too many components, which causes the problem that the overall architecture is relatively fragile. In addition, complex operation and maintenance requires the maintenance of a set of basic components of big data, especially in distributed deployment, such natural defects (dependence on Hadoop and HBase) are exposed. Of course, it is good to study the source code and experience the design principle of the sequential database.

If you have any questions or any bugs are found, please feel free to contact me.

Your comments and suggestions are welcome!