Online application upgrade, need to supplement the missing data correlation, you write a program to deal with it?


Customer information synchronization, because it is online sensitive fields are encrypted processing, so you need to write a program decryption processing?


I remember years ago, I used to do this kind of thing, code this kind of code. Today, I’m going to go back and do a quick share of these things, just in case you’re having this problem, and put it into practice, it might be more productive.


01. Said




In fact, the diagram above has simplified the solution of the problem, so let me explain it again.


The first step is to write a Java Main function that receives the parameters required by the application execution, such as database connection URL, user name, and password. Time to update data and so on.


Then it is necessary to provide a trigger shell script, which is specialized in the industry. Considering the convenience of operation and maintenance colleagues to operate online, it can be run with one key.


After thinking clearly, in fact, it is not difficult to write data processing programs.


The most basic and simplest way to write a data handler can be JDBC; Slightly awesome can use MyBatis, the program to write deployment online only need to compile the class file; In addition, the program needs to introduce the database related class library (JAR package) and the use of the tripartite class library; Occasionally, I will receive a demand scenario that combines source data srcData to process online data.


Therefore, the package published online is roughly as follows:


Classes are used to store compiled data handlers. Lib is used to store jar packages related to databases and three-party class libraries. Notifydataupdate. sh is used to trigger the scripts of data handlers.Copy the code

02. Implementation

The above has said, write data processing procedures are not difficult, this pediatric things, is not the focus of our share today (do not understand, ask Baidu, Baidu will tell you a lot of).

Let’s focus on how to write a shell script that triggers a data handler.

#! /usr/bin/ksh
LANG=zh_CN
export LANG
echo "-- -- -- -- -- -- -- -- -- --" `date` "-- -- -- -- -- -- -- --"
cd /usr/app/p2pDataUpdate

# do business
APP_BASE=/usr/app/p2pDataUpdate
LIB_BASE=$APP_BASE/lib

CLASSPATH=./:./classes
CLASSPATH=$CLASSPATH:$LIB_BASE/db2jcc_license_cu.jar
CLASSPATH=$CLASSPATH:$LIB_BASE/db2jcc.jar

export CLASSPATH

java P2PDataUpdate jdbc:db2://localhost:50000/dev username userpwd /usr/app/p2pDataUpdate/srcData.txtCopy the code



3. Write the last

– Shared a simple data processing tool, if we add a crontab timing shell script, then we can solve the business data synchronization scenario (which is also used a lot). If you don’t know how to work with Crontab, please refer back to our previously shared article on Online application monitoring.


In order to make it easier for you to play with the tool above, let’s finally share some experiences.


Lesson 1: How to remove whitespace from shell scripts?

sed -i 's/\r$//' XXXX.shCopy the code

Lesson 2: How do I pass the current date into a shell script?

LD=`perl -MPOSIX -le 'print strftime "%Y-%m-%d", localtime(time()-86400); '`echo LD=${LD}java P2PDataUpdate jdbc:db2://localhost:50000/dev username userpwd  ${LD}Copy the code

Lesson 3: How do I change the Time zone of Linux?

cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtimeCopy the code

Lesson 4: If the data creation time intime field has an index, pay attention to the following concatenation will be indexed

intime>='the 2012-04-01-00.00.00.000000' and intime<='the 2012-04-30-23.59.59.999999'Copy the code


Hopefully, today’s simple sharing will help you be more productive and make more time for coffee.


That’s all for today’s sharing. If you feel a little bit interesting, please follow the wechat public account “Yizhuxiaozhuan” for more wonderful sharing.