Source side configuration:
1.1 Install OGG software.
Ogg software does not require version 12
Configuration MGR
PORT 7810
DYNAMICPORTLIST 7811-7914
AUTORESTART REPLICAT dpe*, WAITMINUTES 1, RETRIES 5
AUTORESTART REPLICAT ext*, WAITMINUTES 1, RETRIES 5
PURGEOLDEXTRACTS /home/ogg/kafka_ogg/dirdat/kf*,USECHECKPOINTS, minkeephours 6
Configure the Ext extraction process parameters:
EXTRACT extkaf
–setenv (NLS_LANG=”AMERICAN_AMERICA.AL32UTF8″)
The setenv (ORACLE_HOME = / u01 / app/oracle/product / 11.2.0 / dbhome_1)
userid goldengate@ogg ,password Golden_1230
–getupdatebefores
GETTRUNCATES
REPORTCOUNT EVERY 15 MINUTES, RATE
DISCARDFILE ./dirrpt/extkaf.dsc,APPEND,MEGABYTES 1024
–THREADOPTIONS MAXCOMMITPROPAGATIONDELAY 90000 IOLATENCY 90000
DBOPTIONS ALLOWUNUSEDCOLUMN
–WARNLONGTRANS 2h,CHECKINTERVAL 3m
EXTTRAIL ./dirdat/kf
— TRANLOGOPTIONS CONVERTUCS2CLOBS
TRANLOGOPTIONS EXCLUDEUSER goldengate
TRANLOGOPTIONS DBLOGREADER
— TRANLOGOPTIONS _noReadAhead Any
–DYNAMICRESOLUTION
Table schema1.tablename1;
Table schema1.tablename2;
Table schema1.tablename3 ;
Table4schema1.tablename4;
Configure the delivery process:
extract dpekaf
Rmthost 172.31.31.10, mgrport 7810
passthru
numfiles 500
rmttrail /home/ogg/kafka_ogg/dirdat/kf
Table schema1.tablename1;
Table schema1.tablename2;
Table schema1.tablename3 ;
Table4schema1.tablename4;
Goldengate for big data
Target terminal configuration MGR:
PORT 7810
DYNAMICPORTLIST 7811-7914
AUTORESTART REPLICAT rep*, WAITMINUTES 1, RETRIES 5
PURGEOLDEXTRACTS /home/ogg/kafka_ogg/dirdat/kf*,USECHECKPOINTS, minkeephours 6
The process of configuring the Replicat game agent:
Storage process 1:
replicat repykaf1
– the setenv (JAVA_HOME = / / home/ogg jdk1.8.0 _111)
– the setenv (JRE_HOME = / / home/ogg jdk1.8.0 _111 / jre)
–setenv (PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH)
–setenv (CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib)
–setenv (LD_LIBRARY_PATH=$JAVA_HOMEwww.walajiao.com/jre/lib/amd64/server:/home/ogg/kafka_ogg/lib)
–getenv (JAVA_HOME)
–getenv (JRE_HOME)
–getenv (CLASSPATH)
–getenv (LD_LIBRARY_PATH)
–getenv (PATH)
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka1.props
GETTRUNCATES
REPORTCOUNT EVERY 1 MINUTES ,RATE
GROUPTRANSOPS 1000
Table schema1.tablename1;
Storage process 2:
replicat repykaf2
– the setenv (JAVA_HOME = / / home/ogg jdk1.8.0 _111)
– the setenv (JRE_HOME = / / home/ogg jdk1.8.0 _111 / jre)
–setenv (PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH)
–setenv (CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib)
–setenv (LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:/home/ogg/kafka_ogg/lib)
–getenv (JAVA_HOME)
–getenv (JRE_HOME)
–getenv (CLASSPATH)
–getenv (LD_LIBRARY_PATH)
–getenv (PATH)
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka2.props
GETTRUNCATES
REPORTCOUNT EVERY 1 MINUTES ,RATE
GROUPTRANSOPS 1000
Table schema1.tablename2;
Storage process 3:
replicat repykaf3
– the setenv (JAVA_HOME = / / home/ogg jdk1.8.0 _111)
– the setenv (JRE_HOME = / / home/ogg jdk1.8.0 _111 / jre)
–setenv (PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH)
–setenv (CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib)
–setenv (LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:/home/ogg/kafka_ogg/lib)
–getenv (JAVA_HOME)
–getenv (JRE_HOME)
–getenv (CLASSPATH)
–getenv (LD_LIBRARY_PATH)
–getenv (PATH)
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka3.props
GETTRUNCATES
REPORTCOUNT EVERY 1 MINUTES ,RATE
GROUPTRANSOPS 1000
Table schema1.tablename3;
Storage process 4:
replicat repykaf4
– the setenv (JAVA_HOME = / / home/ogg jdk1.8.0 _111)
– the setenv (JRE_HOME = / / home/ogg jdk1.8.0 _111 / jre)
–setenv (PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH)
–setenv (CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib)
–setenv (LD_LIBRARY_PATH=$JAVA_HOME/jre/lib/amd64/server:/home/ogg/kafka_ogg/lib)
–getenv (JAVA_HOME)
–getenv (JRE_HOME)
–getenv (CLASSPATH)
–getenv (LD_LIBRARY_PATH)
–getenv (PATH)
TARGETDB LIBFILE libggjava.so SET property=dirprm/kafka4.props
GETTRUNCATES
REPORTCOUNT EVERY 1 MINUTES ,RATE
GROUPTRANSOPS 1000
Table schema1.tablename4;
Configure the parameters file to go to Kafka
My OGG unzip directory is kafka_ogg
In the OGG unzip directory is the: AdapterExamples folder
cp /home/ogg/kafka_ogg/AdapterExamples/big-data/kafka/* /home/ogg/kafka_ogg/dirprm/
Editor:
Vi custom_kafka_producer.properties
Servers = IP: port, IP: port example
The bootstrap. The servers = 172.31.31.10: $6667172.31…. 31.11: $6667172.31…. 31.12: $6667172.31…. 31.13:6667
acks=1
reconnect.backoff.ms=1000
compression.type=gzip
value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
100KB per partition
batch.size=102400
linger.ms=10000
max.request.size=10240000
send.buffer.bytes=10240000
The editor
Kafka1.props
gg.handlerlist = kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
gg.handler.kafkahandler.topicMappingTemplate=topic-name
Here is the topic name you created.
gg.handler.kafkahandler.format=json
gg.handler.kafkahandler.BlockingSend =false
gg.handler.kafkahandler.includeTokens=false
gg.handler.kafkahandler.mode=tx
goldengate.userexit.timestamp=utc
goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE
gg.log=log4j
gg.log.level=INFO
gg.report.time=30sec
Gg. Classpath = dirprm / : / home/ogg/kafka_ogg ggjava/resources/lib / : / usr/HDP / / kafka/libs / 2.4.0.0-169
javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=ggjava/ggjava.jar
goldengate.userexit.utf8mode=true
gg.handler.kafkahandler.keyMappingTemplate=HH
gg.handler.kafkahandler.format.includePrimaryKeys=true
Kafka2.props
Kafka3.props
Kafka4.props