technology

Technology: dataneclues/thrift

hive –service metastore

/ usr/Java/jdk1.8.0 _131 / bin/Java - Dproc_jar - Dproc_metastore - XX: + UseParallelGC -agentlib:jdwp=transport=dt_socket,server=y,address=8000,suspend=y -Dlog4j.configurationFile=hive-log4j2.properties . - Djava. Util. Logging. Config file = / soft/hive - 3.1.2 / conf/parquet - logging. The properties - Dyarn. The dir = / soft/hadoop - 3.1.3 / logs - Dyarn. Log. The file = hadoop. The log - Dyarn. Home. The dir = / soft/hadoop - 3.1.3 - Dyarn. Root. The logger = INFO, the console - Djava. If the path = / soft/hadoop - 3.1.3 / lib/native - Xmx256m - Dhadoop. The dir = / soft/hadoop - 3.1.3 / logs - Dhadoop. Log. The file = hadoop. The log - Dhadoop. Home. The dir = / soft/hadoop - 3.1.3 - Dhadoop. Id. STR = root - Dhadoop. Root. The logger = INFO, the console  -Dhadoop.policy.file=hadoop-policy.xml -Dhadoop.security.logger=INFO,NullAppender org.apache.hadoop.util.RunJar / soft/hive - 3.1.2 / lib/hive - metastore - 3.1.2. Jar. Org. Apache hadoop. Hive. Metastore. HiveMetaStoreCopy the code

Program entrance

Org, apache hadoop. Hive. Metastore. HiveMetaStore:

1. Load the configuration and parse command line parameters. Use the apache-commons-CLI package to parse command line parameters

final Configuration conf = MetastoreConf.newMetastoreConf();
HiveMetastoreCli cli = new HiveMetastoreCli(conf);
cli.parse(args);
Copy the code

2. Run startMetaStore to start the Thrift Server

Lock startLock = new ReentrantLock();
Condition startCondition = startLock.newCondition();
AtomicBoolean startedServing = new AtomicBoolean();
// When metaStore is successfully started, start the transaction related processing thread CompactorThread
startMetaStoreThreads(conf, startLock, startCondition, startedServing);
startMetaStore(cli.getPort(), HadoopThriftAuthBridge.getBridge(), conf, startLock,
  startCondition, startedServing);
Copy the code

Starting thrift Server Determines whether to enable Kerberos to start servers of the corresponding type. The following describes how to start servers without SSL:

// serverSocket: new TServerSocket(new InetSocketAddress(9083));
TThreadPoolServer.Args args = new TThreadPoolServer.Args(serverSocket)
//HMSHandler baseHandler = new HiveMetaStore.HMSHandler("new db based metaserver", conf, false); 
//IHMSHandler handler = newRetryingHMSHandler(baseHandler, conf);
//processor = new TSetIpAddressProcessor<>(handler);
  .processor(processor)
  .transportFactory(transFactory)
  .protocolFactory(protocolFactory)
  .inputProtocolFactory(inputProtoFactory)
  .minWorkerThreads(minWorkerThreads)
  .maxWorkerThreads(maxWorkerThreads);

TServer tServer = new TThreadPoolServer(args);
Copy the code

The core processing logic is in the handler, which is decorated with a retry mechanism. The core HMSHandler is responsible for handling the client request after calling the initialization method

synchronized (HMSHandler.class) {
    if (currentUrl == null| |! currentUrl.equals(MetaStoreInit.getConnectionURL(conf))) {// Create a default database default, default catalog
      createDefaultDB();
      // Create the default role admincreateDefaultRoles(); addAdminUsers(); currentUrl = MetaStoreInit.getConnectionURL(conf); }}Copy the code

When createDefaultDB is created, ObjectStore of RawStoreProxy is initialized. ObjectStore is the database operation interface used by MetaStore to operate mysql database

LOG.info("ObjectStore, initialize called");
prop = dsProps;
// Obtain the persistent managed object of Datanucleus
pm = getPersistenceManager();
try {
  String productName = MetaStoreDirectSql.getProductName(pm);
  sqlGenerator = new SQLGenerator(DatabaseProduct.determineDatabaseProduct(productName), conf);
} catch (SQLException e) {
  LOG.error("error trying to figure out the database product", e);
  throw newRuntimeException(e); } isInitialized = pm ! =null;
if (isInitialized) {
  dbType = determineDatabaseProduct();
  expressionProxy = createExpressionProxy(conf);
  if (MetastoreConf.getBoolVar(getConf(), ConfVars.TRY_DIRECT_SQL)) {
    String schema = prop.getProperty("javax.jdo.mapping.Schema");
    schema = org.apache.commons.lang.StringUtils.defaultIfBlank(schema, null);
    //As of now, only the partition retrieval is done this way to improve job startup time;
    directSql = new MetaStoreDirectSql(pm, conf, schema);
  }
}
LOG.debug("RawStore: {}, with PersistenceManager: {}" +
        " created in the thread with id: {}".this, pm, Thread.currentThread().getId());
Copy the code

3. Communicate with the client

The RetryingHMSHandler will be called and the corresponding interface method of hivemetaStore.hmshandler will be called. Such as hiveclient – show the database – > HiveMetaStore. HMMSHandler. Get_databases – > ObejctStore. GetAllDatabase hiveclient – create table test(id int,name string); -> Verify whether the database exists, verify the field, verify whether the table exists, create table, create table_ds, create table_pri and a series of operations