Technical background

Bigtable originated in three old papers from Google.

Designed to

In order to solve the real-time read and write problem of massive structured data in big data environment. To compensate for the lack of real-time storage in the Hadoop ecosystem.

Design idea

distributed

Technical nature

  • Concept: Distributed column store noSQL database

  • Explanation:

Column storage: The underlying data file storage format is column storage

Nosql: Non-relational, can store structured and semi-structured data

Database: Meets the characteristics of a database

Core features

Large tables: Built for storing massive amounts of data, with hundreds of millions of rows and millions of columns

Distributed:

1. Distributed memory: Data is stored in the memory for real-time read and write

2. Distributed external storage: Data that cannot be stored in the MEMORY is written to the HDFS

Multiple versions: a column cluster allows the number of versions to be set, that is, to allow a number to store multiple versions, through “ns: TB cf: C ts” only one data defect

No:

RDBMS: a relational database, such as “mysql, oracle, postgresql, essentially”

2. Nosql: non-relational database, for example, hbase, Redis,mongodb

Random real-time read and write

1. Data is stored in memory preferentially and has a high hit ratio

2. The data with a medium hit ratio is stored in the cache

3. Data with low hit ratio is stored in HDFS

Dependent on HDFS: Massive data is stored in HDFS, which ensures data security and data persistence. Join operations are not supported, but row-level transactions are supported. Complex Join operations require integration. For null columns, there is no storage footprint, so the table design can be very sparseCopy the code

The cluster character

Physical architecture

client

Type:

  • Bin /hbase shell: does not support SQL statements. You can use help ‘command’,exit,

  • The Java API:

Hue: hbase provides the ThriftServer server for connecting to the Hue client. The ThriftServer implements the access and execution of the entire process

SQL on hbase: Hbase for lower-layer MapReduce operations

zookeeper

Concept: The ZK in hbase is called the true · primary node

Responsibilities:

  • The primary node of the secondary cluster is selected to implement the HA mechanism

  • Provides access points for the cluster and stores addressing points for all regions

  • Stores almost all metadata in hbase, such as node information, active and standby nodes of master, Nodes of RegionServer, and table status

  • Fault tolerance is implemented. If the RegionServer registered node disappears, the RegionServer is considered faulty and the HMaster needs to start the recovery program

  • Regionserver address of Region [Rootregion] that stores Meta tables

  • Responsible for flush, splitWAL status, etc

  • Hbase Schema is stored, including tables and column families of each table

Content:

Each node is registered with ZooKeeper

Zookeeper stores region information of meta tables. Clients can request clusters only after accessing ZooKeeper

HDFS:

Responsibility: Store HFiles, wal

Hmaster:

Concept: Secondary primary node

Responsibilities:

  • Responsible for managing all slave node information,

  • Allocate regions to RegionServer and implement load balancing on RegionServer.

  • Migrate region data, for example, in split scenarios

  • Modify meta tables and metadata. Meta tables are not equal to metadata and can be regarded as part of metadata. Meta tables store region information, and metadata includes RegionServer, Region, table, and namespace information

  • Receives DDL requests from clients to modify metadata and storage in ZooKeeper

  • Zookeeper monitors the status of each RegionServer. If the RegionServer is faulty, the process for restoring the RegionServer is started

  • Discover the failed Region Server and reassign regions on it

  • Garbage collection on HDFS

  • Process schema update requests

RegionServer:

Responsibilities:

  • Responsible for receiving and processing all client data read/write requests

  • Manages all regions on the node and processes I/O requests to these regions

  • Responsible for recording WAL

  • Read cache: memcache, accounting for 40% of the heap memory, and flush the cache to HDFS

  • compression

  • Process Region fragments

Supplement:

  • Regionserver is a slave node with multiple regionServers

  • Regionserver Manages many different table partitions in a RegionServer

  • Data from the META table is also stored in regionServer

Hlog:

Concept: Pre-write logs, similar to edits files in HDFS and binlog files in mysql.

Supplement:

Each Region Server maintains one Hlog, not one Hlog for each Region. In this way, logs of different regions (from different tables) are mixed together. In this way, continuously adding a single file reduces the number of disk addressing and improves the table write performance. If a Region Server is offline, the logs on the RegionServer need to be split and distributed to other Region Servers for recovery. The HLog File is a normal Hadoop Sequence File.

Region:

Concept:

The data storage unit in regionServer is not region-level

Supplement:

  • A table splits data into multiple regions based on rowkeys to hash the table’s data.

  • By default, a table is created with only one region

  • The sum of ranges for different regions of the same table is from minus infinity to infinity

  • A Region can be managed by only one RegionServer

  • Region is the smallest load balancing unit in hbase and is the basis for implementing distribution

  • Unbalanced region allocation may cause data skew and hotspot problems

  • When reading or writing data, the user determines the partition from which the data is read and written according to the RowKey

  • Each partition is a directory

  • Number of Stores /2 = Number of regions

Store

Meaning: When designing a column cluster, put columns with the same I/O attributes in the same column cluster, because the underlying storage is a column cluster storage that occupies a store; Put it in a storeFile to provide fast read and write operations

Concept:

Region is a partition of a table, and Store is a partition of a column cluster. Machine memory x 40% /128M= Number of stores

MemStore

Function:

Supplement:

Belongs to the memory storage area in the Store. Within each store there is a memStore to store the data that has just been written. Write memory, 40% of the heap,

StoreFile /HFile

Function:

Explanation:

1. If the memstore meets the requirements, the memstore is flushed. The data file that is flushed is called storeFile and stored in HDFS is called HFile2. There may be multiple storeFiles in a store, but eventually they will be optimized and merged into one file [binary file]. 3. Storefile is the encapsulation of Hfile, or it can be regarded as the same file with different names

Supplement:

HFile is divided into six parts:

1.Data Block – Holds the Data in the table. This part can be compressed

2.Meta Block segment (Optional) – Saves user-defined KV pairs that can be compressed.

3.File Info – Hfile meta information is not compressed. Users can also add their own meta information in this section.

4.Data Block Index – The Index of a Data Block. The key of each index is the key of the first record of the block being indexed.

5.Meta Block Index (Optional) – Index of the Meta Block.

6.Trailer — This section is fixed length. The offset of each segment is saved

The logical storage

NameSpace

Namespace, physical existence. By default, there are two namespaces default and hbase. Each table must have its own namespace. If no namespace is specified, the namespace belongs to default

Table

Mysql > select table_name from table_name where namespace:table_name = ‘default’

Rowkey

The rowkey, which is used to uniquely specify a row of data (note that it is not a single row), exists at the time the table is created and does not need to be created again.

The design of rowkeys affects node data allocation of underlying regions, causing hotspot problems and affecting read and write performance.

The rowkey is the only index in hbase. Hbase does not support custom indexes. There is no other index except the Rowkey.

ColumnFamily

Column cluster, column grouping. A column cluster has multiple columns, and each column must belong to a column cluster. You can specify the version number of all columns in this column cluster by column cluster. Generally, fields with the same IO attributes are placed in the same column cluster. You must specify at least one column cluster when creating a table. It is recommended that a table contain only one or two column clusters. If the number of column clusters exceeds two, the READ and write performance of hbase deteriorates.

Column

Column, column label. If there are multiple columns in a column cluster, a column must belong to a column cluster, and the number of columns in two column clusters need not be equal. Two rows running under the same column cluster have different columns, that is, semi-structured. A unique column is specified by cf: c.

Value

Value: Values in each column. Hbase does not have data types but is stored in bytes, allowing storing values of multiple versions

Version

Version number, belongs to the column of an attribute of the clusters, the default version is 1, you can configure multiple versions to store multiple versions of the data query, by default only show the latest version of each column data if each store data more than version number, the old version will be overwritten version is designed to store, the default display the latest version is to read

TimeStamp

Timestamp is an identifier. The default value is Greenwich Mean time (GMT) at which data is inserted. A query requires a timestamp to uniquely identify a data item

Column storage

Concept:

Unlike the row storage format of AN RDBMS, NOSQL uses a column storage format.

Advantages:

Row storage operations are row – level, and column storage operations are column – level. So more granular row stores have columns that are the same in a row (structured), and column stores have columns that are different in each row (semi-structured). Therefore, a wide range of data formats are supported. Semi-structured data supports structured data, while structured data does not support semi-structured data. The performance is optimized and the READ/write speed is faster

DDL

Namespace list_namespace create_namespace drop_namespace describe_namespace list_namespace_tables 2. DDL Remarks: Only the administrator admin can perform this operation. List create: When creating a table, specify the table name and column cluster name. Describe /desc DROP: If hbase tables are in enable state, table structures cannot be modified or deleted. Disable disable enableCopy the code

DML

DML PUT: No update or delete operations are performed in hbase. All updates are inserted. Scan: used to query one or more rows or scan all tablesCopy the code

Hot issues and data skew

concept

Hot: A node receives a large number of requests from clients, resulting in a heavy load on the node and a low load on other nodes

The phenomenon of

Data skew: A large number of computing tasks are concentrated on a node, resulting in high load on this node and low load on other nodes

The difference between

  • Hot issues: At the request level, hot issues may cause data skew

  • Data skew: At the computing level, data skew may not be caused by hot issues

why

Hotspot problems occur in many scenarios. The causes of hotspot problems in hbase are as follows:

1. A large number of Rowkeys requested by the client are located in a region 2. There is only one partition in the table and all requests are concentrated on this region

To solve

1. Specify the region range of the table partition to avoid unbalanced partition ranges. 2. You need to create a pre-partition or multiple partitions based on the number of partitions

pre-split

Background: When creating a table, there is only one partition by default. In this case, it is easy to handle server hotspot problems

Concept: When creating a table, design a table based on rowkeys. A table has multiple partitions at the beginning. Partitions need to be designed based on the prefix of rowkeys or the complete Rowkeys

Meaning:

1. Improve data read and write efficiency

2. Load balancing to prevent data skew

3. Facilitate region scheduling for cluster Dr

4. Optimize the number of Maps

Method:

1. Specify the scope of partition the create ‘ns1: t1’, ‘f1’ and SPLITS = > [‘ 10 ‘, ’20’, ’30’, ’40’]

Create ‘ns1: T2 ‘, ‘f1’, SPLITS_FILE => ‘/export/datas/ Montes.txt ‘

Admin. createTable(HtableDescriptor desc, Byte splitKeys)

Design rules for RowKey

Why: Why is rowKey design important?

  • All data is stored by reading and writing partitions based on rowkeys, that is, partitions are divided based on rowkeys.

  • Rowkey is a unique index in hbase.

  • Rowkey is prefix matched when queried.

Consequences: What are the consequences of a poorly designed Rowkey?

  • It can cause hot issues

  • A full table scan is caused

Design rules:

Rule of uniqueness: Each rowkey must uniquely mark a row, similar to the uniqueness of a primary key

Hashing principle:

Build a random hash of rowkeys that do not allow rowkeys to be contiguous to avoid hot issues.

What if the rowKey prefix is a fixed field and causes a sequence?

  1. Encoding, using md5/ CRC32 etc to encode fixed fields as rowkeys

  2. Reversed, rowkeys are in alphabetical order for ASCII characters, fixed fields are not contiguous after reversed

  3. Random number, adding a random number to a fixed field is not recommended because it is not easy to maintain

Business principle: Design rowkeys according to business dimensions, such as those that are often used as query criteria

Composition principle: Design a Rowkey by combining business dimensions

Length rule: Generally, the rowkey length should not exceed 100 bytes. You can compress the rowkey length in hexadecimal or 30 bits

JavaAPI

Introduction: In hbase, hbase shell is generally used to encapsulate scripts and perform DDL operations. In hbase shell xxx.query, exit is added at the end of script operations. Javaapi is used to perform DML operations. HBaseConfiguration; 1.HBaseConfiguration; 2. Create an hbase initialization object. 2. Hbase administrator object. Requires DDL operations require object 1. 2 tableExists disableTable 3. 4 deleteTable deleteNamespace 5. ListNamespaceDescriptors 3.HTableDescriptor; AddFamily 2. CreateTable 3. GetTableName 4. Table Name Object 1.getNameAsString 5.HColumnDescriptor; SetMaxVersions 2.setBlockCacheEnabled 3. 6.NamespaceDescriptor; CreateNamespace 2. Create 7.Get; Get command to obtain data 8. A result represents all data of a rowkey. A cell is a column of data. Table object 1.get 2.put 3.delete 4.getScanner 11. 1.addColumn 12.Delete; 1.addColumn 13.ResultScanner; A collection of result objectsCopy the code

read

1. The driver class, set the parameters in the encapsulates the Mr Programs that call methods TableMapReduceUtil. Inittablemapper (input and the parameters of the map); 2. Mapper class, need to inherit TableMapper class, default input input key and value ImmutableBytesWritable and Result implementation: the bottom through tableInpputFormat implementation; Sqoop uses dbinputFormat to implement < explain > because Mr Reads hbase data and does not aggregate data, reduce is not requiredCopy the code

write

1. The driver class, set the parameters in the encapsulates the Mr Programs that call methods TableMapReduceUtil. Inittablereduce (reduce and the output parameters); 2. Reducer class, need to inherit TableReducer class, default output value is put object implementation: the bottom through TableOutputFormat implementation; Sqoop uses dboutputFormat to implement (explain) the need to build put objects in reduce classes to store data, so reduce is neededCopy the code

The Filter Filter

Background:

  • In hbase, all data is stored in alphabetical order based on the ASCII code table of rowkey

  • In hbase, rowkey queries match the prefix

  • The rowKey is the only index in hbase. All data can be read or written by rowkey query or full table scan

  • Hbase has no underlying data type. Data is stored in byte arrays.

  • Store each column of data in the keyValue pair format. Key = NS + TB + CF + C +ts, value=value

  • Generally used with scan

    Shell filter types: QualifierFilter (>=, ‘binary:xyz’) 3. ROWPREFIXFILTER, Rowkey Prefix filter

    Javaapi filter types: first class: comparison filter; FamilyFilter 3.QualifierFilter 4.ValueFilter 2. Encapsulated with more, a good way to implement 1. PrexfixFilter: rowkey prefix filter 2. SingleColumnValueFilter/singleColumnValueExcludeFilter: Single value filter 3. MultipleColumnPrefixFilter: filter column 4. PageFilter: paging filters, generally in the work, you must specify the corresponding starting position, the general collocation while the startrow to use The third category: A combined filter is a combination of filters, a FilterList

    How to use filters: 1. Specify “{COLUMNS attribute}” in shell, for example: {COLUMNS => “}; {while the STARTROW = > ‘closed interval, STOPROW = >’ open interval ‘} {ROWPREFIXFILTER = > ‘, the FILTER = > ‘}

Hbase:ns/Hbase:meta

Hbase: Namespace table. This table records information about all namespaces in hbase. When a user makes a request, the user can check whether the namespace accessed by the user exists

“Hbase: Meta” table, metadata table in hbase table, meta table records:

The RowKey in the Meta table contains regionInfo and Server. The RowKey of regionInfo records the region name, startKey, and StopKey. The RowKey containing the Server records the address and port of the RegionServer of the Region.

The meta table records the region names of all tables in hbase except its own. Region name consists of the following parameters: table name +region start range + Region time + Region UNIQUE ID.

Supplement:

1. How does the client find the region of the corresponding region in a data read/write request?

A: Zk records the Region information of the Meta table, including the region name, startkey, and Stopkey, and the node address and port of the region. The REGION information can be used to map meta table data found on the RegionServer. The Meta table records region information of all hbase tables except the hbase table. The Region information includes the region name, startkey, and stopkey, and the node address and port of the region. Locate the region on the RegionServer based on the information

Hbase storage principles

Overview of read and write requests:

  • When the client requests the meta table for the first time, the region information is not recorded locally. Therefore, the client requests the REGION information of the META table first.

  • Then, regionServer requests meta table data based on Region information to obtain region information corresponding to the RowKey in the table.

  • The client sends a read/write request to RegionServer based on the Region information corresponding to the RowKey in the returned table. The client caches the local address and Region information for future use and directly requests RegionServer to obtain table data.

  • Regionserver operates region on request and determines which store to read and write from according to the cluster of columns.

Read requests:

The request preferentially reads the data in memory, that is, memStore. If the data is not in memory, read the cache, and then read HDFS

Write requests:

  • The request is first written to memStore. When memStore meets the requirements, flush is triggered to convert memStore data into StoreFile files

  • The HDFS client encapsulated in hbase generates files in hfile format and stores them in HDFS

Hbase read/write request process

Write request process

  • The client submits a write command to obtain region information and request the RegionServer based on the RowKey to perform operations on the RowKey

  • The request is submitted to the corresponding RegionServer, which operates on the name of the region to which the request is written

  • Determine which store in the region to write to based on the column cluster in the request

  • Write data to memstore as keyValue

  • When memStore reaches a certain condition, flush is triggered, and data in memory is written to HDFS.

  • When a storefile in the HDFS reaches a certain condition, it triggers a compaction to merge and sort the storefile

  • When the size of the entire region reaches a certain threshold, a split is triggered. One region is split into two regions, and the original region is destroyed

Read Request Process

  • The client requests ZK to obtain the address of the meta table from ZK

  • The client reads the data of the meta table and obtains all the reigon information of the table according to the table name in the read command.

  • Find the Region to be read and the Corresponding RegionServer address

  • If a ROWkey is specified in a GET request, region information corresponding to the Rowkey is returned

  • If a SCAN request is made, all region information corresponding to the table is returned

  • Request the corresponding RegionServer based on the returned Reigon address,

  • Regionserver Finds the region based on the region name and reads the region

  • If a column is specified in the request, the store corresponding to the column cluster is read; if no column is specified in the request, the data of all stores is read

  • When the request reads store, it preferentially reads store in memory, namely memstore. If it is not found in memStore, if cache is enabled, it searches for memcache; if cache is not enabled, it directly searches for StoreFile in HDFS; if it is not found in HDFS, data does not exist. Return the data directly. If it exists in the HDFS, return the data. If caching is enabled, add the data to memcache for future use

    Query memory: Yes: Returned No: Cache: Yes: Returned No: HDFS Yes: Returned and add cache No: Returned

Explanation:

  1. After practical analysis, the newly written data is the highest hit ratio in the request access, so it is saved in memory

  2. According to the LRU least recently used algorithm, the data with high access frequency is added to the cache to improve the hit ratio of cache access

The LSM – tree model

Concepts: Log-struct-merge model tree, wal, flush, split, etc.

Function: the random disordered data into ordered data, through the ordered algorithm to speed up the data read, because the need to sort when writing, all sacrifice a certain efficiency of writing data, use memory to achieve data read and write storage

Main steps:

  • Data is now saved to a write-ahead log and then written to memory on request

  • Sort the data in memory

  • Write the data in memory to a file, forming an ordered data file

  • Merge: Merge ordered data files to create a new ordered file

  • Delete and update: In the model design, there is no delete and update per se, it is simulated by insertion, and the actual physical deletion is usually done at the time of file merge

WAL

  • Concept: write ahead log, write-ahead-log

  • Responsibility: When a write request is generated, the write request is submitted to RegionServer, which records the write operation in WAL (Hlog) and then writes it to MemStore

  • Storage: Stores WAL in HDFS. Each RegionServer maintains a COPY of WAL.

  • Scenario: Used for data recovery. Hfile is restored by HDFS and Memstore is restored by WAL

Supplement:

  • Oldwals: Hlog log files that are no longer needed

Flush

  • Concept: Brush

  • Run the following command to sort data in memStore by rowkey lexicographical order, and then write the data in MemStore to HDFS for persistence to generate an ordered StoreFile

  • Scenario: The MemStore memory is used and the memory capacity is limited. To improve the hit ratio of new data, old data needs to be imported to the HDFS for safe storage

Compaction

  • Concept: Merge

  • Run the following command to merge multiple storeFile files in the HDFS to create a unified ordered file

  • Scenario: To provide fast reading of files, merge multiple storeFile files into one overall ordered storeFile file, because reading multiple data sources is not as fast as reading one data source

  • Process:

    Minor compaction: Compacts a few files generated earlier in the HDFS from a consolidated compaction. Major compaction: compacts all files from a consolidated compaction that results in the deletion or expiration of data

Split

  • Concept: split.

  • Function: If a region has too much data, it is split into two regions.

  • Scene: In this case, the RegionServer divides the region into two partitions. The Master allocates the two partitions to different RegionServers. After the regionServer successfully allocates the two partitions, the old Region is offline.

The data load

Concept: Data can be imported to hbase in two modes.

  • Using put objects

For example, hbase shell, Java API, and Mr Program encapsulation. Disadvantages: Data flows pass through WAL, then memory, and finally HDFS. When massive data is imported, the I/OS of region and HDFS are too high, which increases the load on the server and affects other applications.

  • Bulkload

    Principle: Hbase underlying storage is the Hfile file on the HDFS, and data is associated with the Meta table. Therefore, data can be converted to hfile locally, uploaded to the HDFS, and added to the META table.

    Scenario: This mode is applicable to hbase scenarios where a large amount of data is imported in batches and stable performance is required

    Disadvantages:

    Data is read from HDFS for the first time and not stored in memStore, so it will be slow for the first time. However, if the amount of data is very large, the final data is read from HDFS for the first time in both methods, so there is no difference. Data is directly transferred to HDFS without WAL. There is no way to recover the data. We need to convert it again

Implementation:

Application implementation:

  1. Responsible for converting ordinary files into Hfile files

  2. Load converted hfile files into hbase tables

Hbase implementation:

1.ImportTSV is a command delivered in the hbase-server.jar package. It can import data into an hbase table in put mode and convert a common file into an hfile file in two ways

2.com pleteBulkload. The above importtsv command converts a common file into an hfile file, but it is not loaded into the hbase table. Completebulkload is responsible for loading the hfile file into the hbase table

< command > : 1. Yarn jar /export/ Servers/hbase-1.2.0-cDH5.14.0 /lib/hbase-server-1.2.0-cdh5.14.0.jar importtsv -Dimporttsv.columns=a,b,c <tablename> <inputdir> -Dimporttsv.separator=',' -Dimporttsv.bulk.output=/bulkload/output2 2. Yarn JAR export/ Servers/hbase-1.2.0-cDH5.14.0 /lib/hbase-server-1.2.0-cdh5.14.0.jar CompleteBulkload /bulkload/output2 Mrhbase < Explanation > : - Dimporttsv. The columns of the specified file column map the columns of the cluster and column Dimporttsv. Columns = HBASE_ROW_KEY, info: name, info: the age, the info: sex mrhbase/bulkload/input Separator Specifies the column separators in the file to be read. By default, the columns are separated by tabs. - dimporttsv.bulk. output Specifies the HDFS location of the generated HFILECopy the code

SQL on Hbase

Background:

As a NOSQL database, hbase has its own API implementation and does not support THE SQL language, which is not conducive to development and data analysis personnel. SQL on hbase solves this problem by using SQL or JDBC operations on hbase

Principle:

  • Based on Java API/MapReduce implementation

  • Implemented based on hbase shell

  • The bypass is implemented through HDFS

Implementation:

Hive integration hbase:

Principle: HQL statements in Hive are used to convert the underlying operations to MapReduce operations. Mapreduce operations are also performed on hbase tables: Create a table associated with hbase in Hive and perform operations on hbase tables in Hive

Phoenix Third-party tools:

Principle: A secondary index is built based on hbase and directly invokes the API implementation of hbase. Therefore, hbase integration and performance are optimized

Sqoop third-party tools:

Principle: The mapReduce program is also used to import data from a relational database to HDFS. Then, the importtsv and CompleteBulkload commands are used to import data from HDFS. Sqoop can import but cannot export data. Because semi-structured data supports structured data storage, structured data does not support semi-structured data storage

Supplement:

  • If an hbase table already exists, only external tables can be created in Hive. Keys are used to represent rowkeys

  • If the hbase table does not exist, the first column of the Hive table is used as the hbase rowkey by default

  • Hbase is associated with Hive. When loading data from associated tables in Hive, load cannot be used because mapReduce is not used in the load command. The load command uses the PUT command of HDFS, and only the INSERT command can be used

Secondary indexes

  • Concept: Build an index based on the RowKey, called a secondary index.

  • Meaning: RowKey is a unique index and rowkey is prefix matched. If we do not know the prefix, but we know some rowkey fields, can we only scan all rowkey fields? Secondary indexes solve the rowKey unique index problem.

  • Solution: Build a secondary index.

  • Method:

    Create an index table, use the query conditions in the original table as rowkeys in the index table, and use rowkeys in the original table as values in the index table. If the prefix of the original table is not specified, the system queries the index table based on the query condition, finds the rowkey of the original table, and then searches the original table based on the obtained Rowkey

coprocessor

  • Background: Create a secondary index, because the index table and the original table are two different tables, how to ensure the data synchronization of the two tables?

  • Synchronous mode: Manually insert the index table as well as the original table

  • Disadvantages: Poor performance, cumbersome operation, double the number of requests, high load on the server

  • Build coprocessors, build triggers similar to those in mysql

  • Rely on third-party tools to implement secondary indexing: solr, ES build index type rich, automatic synchronization,Phoenix build index SQL

  • Concept: Coprocessor, which helps developers build custom secondary indexes

  • Essence: User-defined a program that implements interfaces provided by hbase

Classification:

  1. Observer: The observer coprocessor, similar to a listener or trigger, is used for secondary index synchronization. It listens to the original table to see if the client inserts a piece of data into the original table. The coprocessor automatically inserts a piece of data into the index table

  2. Endpoint: a coprocessor, similar to a stored procedure or a method in Java, that is used to perform distributed aggregate statistics. It listens to a table and calculates the number of rowkeys for each region in the table. When called by the client, the number of rowkeys for each region is returned

  • How can I quickly count the number of rowkeys in a table

  • By writing the MapReduce method, the slowest and stupidest way

  • Run the count command provided by hbase

  • Develop endpoint co-processing, the fastest and most efficient way

Hbase optimization

Concept: hbase optimization is implemented using flush, compaction, split, and column cluster attributes

Flush

  • Description: It is used to write data in memStore to HDFS and convert it into storeFile, freeing up memory for storing new data

  • Automatic trigger rules:

    By default, this is triggered when a single memstore reaches 128 MB of storage.

    By default, when the memstore of the entire region reaches 128 MB x 4, the value is triggered.

    By default, memStore usage in the entire RegionServer reaches 40% of the heap memory.

Disadvantages:

Automatic trigger: The disk I/O load is high and services are affected. Therefore, automatic trigger is disabled and manually triggered periodically when services are light

Manual trigger:

Turn off auto trigger: Turn up all auto trigger conditions, periodically triggering manually with the Flush command before the auto trigger conditions are reached

Conclusion:

Turn up the automatic trigger condition to turn off automatic trigger flush and optimize by manually triggering flush during low traffic

Compaction

  • Meaning: It is used to merge multiple storeFile files into one storefile, which becomes an overall orderly file and speeds up the reading efficiency

Automatic trigger rules:

Minor compaction: Compacts a small number of storefiles into a single file. This occurs when a smart compaction algorithm compacts the first three files to a single file. A massive merge that merges all storeFile files into one file triggers a compaction every seven days by defaultCopy the code

  • Disadvantages:

    Automatic trigger: The automatic trigger is disabled, which causes the disk I/O load to be heavy and affects services. In general, automatic trigger is disabled. Manual trigger is periodically triggered when services are light based on the actual situation

  • Conclusion:

Disable automatic triggering major compaction and manually trigger major_compact when operations are low

Split

  • Description: Split a region into two regions to share the load of the region

  • Automatic trigger rules:

    Rules prior to version 0.94: If a store in the entire region is 10GB in size, Math.min(getDesiredMaxFileSize(),initialSize * tableRegionsCount * tableRegionsCount * tableRegionsCount); If the number of regions ranges from 0 to 100, min (10g, 256 MB * Number of regions to the third power)

  • Disadvantages:

    Automatic trigger: The disk I/O load is too high and services are affected. The automatic trigger is usually disabled. Manually triggered periodically when services are light. Hbase. Client. The keyvalue. Maxsize = 100 gb, is unlikely to be the value, according to the amount of data of cluster region, to manually trigger on a regular basis

  • Conclusion:

Turn off automatic triggering and trigger split manually when traffic is low

Column attribute clusters

  • BLOOMFILTER: indicates the BLOOMFILTER

    This command is used to determine whether desired data exists in the StoreFile based on the index when the storeFile is retrieved. If not, skip this command.

    Level: None This function is disabled.

    Row checks if there is a required Rowkey and skips if there is none.

    Rowcol checks if rowkey and CF :c are required and skips if not.

  • Versions: MaxVersions, the maximum number of Versions, the maximum number of Versions that can be stored in a column family.

  • TTL: indicates the version survival time, which is calculated from the timestamp of the version. Once this time is reached, the data is expired and hbase automatically clears data.

  • MIN_VERSIONS: indicates the minimum version of the data in a column family. After the TTL is reached, the minimum version is reserved.

  • BLOCKCACHE: specifies whether to read the data of this column cluster to the memcache cache. This function is enabled by default. You are advised to disable the column cluster that is not frequently read.

  • IN_MEMORY: IN_MEMORY is the highest level of cache in BLOCKCACHE, which is disabled by default. IN_MEMORY is the highest level of cache, and LRU cleaning is performed periodically. The lower the level of cache, IN_MEMORY will be cleaned first

  • BLOCKSIZE: size of storeFile blocks. The default value is 64 KB. The smaller data blocks are, the more data blocks are, the more indexes are maintained, the more memory is occupied, and the faster I read and the slower I write

  • COMPRESSION: Hbase COMPRESSION is actually the COMPRESSION of Hadoop and the storage of StoreFile COMPRESSION

Comparison between Hbase and RDBMS

Hbase

Application Programming Interface (API) and MapReduce are used to access HBase tables. Data is column-oriented, that is, each column is a continuous unit and the total amount of data does not depend on a specific machine. Depending on the number of machines, HBase does not support ACID (Atomicity, Consistency, Isolation, and Healing) for both structured and unstructured data. HBase does not support transactions and does not support JoinsCopy the code

rdbms

Support scaling up using SQL queries row-oriented, that is, each row is a contiguous cell and the total amount of data depends on the server configuration. ACID support is suitable for structured data. Traditional relational databases are generally centralized with transaction support and Join supportCopy the code

Add, delete, modify, and check Hbase

  • Run the following command: Hbase does not update or delete operations, but only add and query operations. Update is to insert new data into memory, and delete is to logically delete data, and then overwrite old version data and physically delete data in subsequent merge

  • Meaning: For fast return processing of read and write requests, data is returned as soon as it is written

  • The differences between Get and Scan:

    The get command is the fastest query method in hbase because rowkey is specified, but only one data can be queried. The scan command can return multiple pieces of data, which can be used together with filters. Put and delete return no results.

The difference between hbase/hive/HDFS/RDBMS

hbase

  • Concept: real-time read and write database

  • Purpose: Store data

  • Scenario: Real-time read/write files that implement a large amount of data. For example, search engine is generally used in high-performance computing (HPC) architectures to store semi-structured or structured data

hive

  • Concept: Structured data warehouse

  • Purpose: Store data

  • Scenario: A tool that can only store structured data and is used to build a data warehouse

hdfs

  • Concept: File storage system

  • Purpose: Store data

  • Scenario: It is used to store structured, semi-structured, and unstructured data. It is suitable for storing and reading data multiple times at a time, for example, a web disk for data archiving

rdbms

  • Concept: Relational database

  • Purpose: Store data

The hard just Presto | Presto principle & tuning & interview & practical comprehensive upgrade edition

The hard just Apache Iceberg | technology research and application in the practice of companies big summary”

The hard just ClickHouse | 40000 words long ClickHouse & based on practice & tuning all perspective”

The Boy hard just | SQL data warehouse of the Gospel of the data warehouse system modeling & implementation & notice little summary”

The hard just Hive | small tuning interview, a summary of 40000 words

The hard just user picture (a) | tag system user portrait under construction of small guide

The hard just user portrait (2) | portrait of users based on the large data to build a small encyclopedia