This is the sixth day of my participation in the August More text Challenge. For details, see:August is more challenging

preface

What is Oracle RAC?

  • Oracle Real Application Clusters (RAC) allow customers to run a single Oracle database across multiple servers to maximize availability and achieve horizontal scalability while accessing shared storage.
  • A user session connected to an Oracle RAC instance can fail over during an interruption and safely replay changes without making any changes to the end user application, thereby hiding the impact of the interruption from the end user.

  • Oracle RAC runs on top of clusters and provides the highest level of availability, scalability, and low-cost computing power for Oracle databases.
  • If a node in the cluster fails, Oracle will continue to run on the remaining nodes.
  • Oracle’s main innovation is a technology called cache merging.
  • Cache merging allows nodes in a cluster to efficiently synchronize their memory caches through high-speed cluster interconnection, thus minimizing disk I/O.
  • The most important advantage of caching is that it enables disks of all nodes in the cluster to share access to all data without having to partition data between nodes.

Features:

  • Consists of multiple interconnected computers using shared storage.
  • There is no user awareness and they appear to be a server to the end user and the application.
  • High availability: As long as one node is alive, services can be provided properly, avoiding single points of failure.
  • High-performance, multi-node load balancing.
  • Nodes can be easily added and deleted to meet system requirements.

So, how do you deploy an RAC database environment?

The general steps are as follows:

  • Install two Linux oss of Redhat7.3 with at least 2 GB physical memory.
  • Network configuration (Dual NICS, prepare IP: Public IP, Virtual IP, Private IP, Scan IP)
  • Storage configuration (6 5G shared disks are used as ASM disks and 50 GB is reserved in the root directory for grid and Oracle installation)
  • Preinstallation preparations (modified system parameters in /etc/sysctl.conf, disabled firewall selinux, synchronized NTPD clock periodically, and installed yum source configuration

User groups and users, directory, new environment variable configuration, and user resource limitations/etc/security/limits the conf configuration, / etc/PAM. D/login to revise the pam_limits. So, etc.)

  • Installing the Grid Software
  • Install the Oracle software and build a database
  • Modify the database memory configuration, keep the password valid, enable archiving, assign periodic deletion script for archiving, assign RMAN backup mechanism)

​​

The host configuration for this article isRedhat 7.3 x86_64.The memory of 2 g.Hard disk 100 g ，The double card ，Iscsi shared storage disk 5G x 6

node The host version The host name The instance name Oracle version Public IP Private IP Virtual IP Scan IP
A node Redhat 7.3 rac01 orcl1 11.2.0.4 192.168.56.10 172.0.0.1 192.168.56.20 192.168.56.110
Two nodes Redhat 7.3 rac02 orcl2 11.2.0.4 192.168.56.11 172.0.0.2 192.168.56.21 192.168.56.110

1. Configure before installation

Note: in the following headings (RAC01 & RAC02) means that both node 1 and node 2 need to execute, (RAC01) means that only node 1 needs to execute.

1 Linux Host installation (RAC01 & RAC02)

  • Centos, Redhat, and Oracle Linux can be installed on the Linux server.
  • Download: RedHat developers.redhat.com/products/rh…
  • OracleLinux download: yum.oracle.com/oracle-linu…
  • Centos download: vault.centos.org/

  • For details about how to install a Linux OS, see:
  • How can Windows hosts play with virtual Machine Linux installations, just read this article
  • Here are three steps to installing Linux on MacOS

2 configure yum and install rac01& RAC02 dependency packages

  • Linux remote connection tool
  • This article will use XShell and Xftp tools. The installation package can be downloaded from the official website or obtained by private bloggers.
  • Other tools are available, such as putty, SecureCRT, etc.

Note: The ISO file of the system image must be mounted in advance.

  • Parallels Desktop Mounting a Linux host image:

  • If VMware Workstation mounts a Linux image:

  • Install dependency packages:
## Mount the image source
mount /dev/cdrom /mnt
## Configure yum source
cat <<EOF>>/etc/yum.repos.d/local.repo
[local]
name=local
baseurl=file:///mnt
gpgcheck=0
enabled=1
EOF
Install dependencies
yum groupinstall -y "Server with GUI"yum install -y bc \ binutils \ compat-libcap1 \ compat-libstdc++-33 \ gcc \ gcc-c++ \ elfutils-libelf \ elfutils-libelf-devel \ glibc \ glibc-devel \ ksh \ libaio \ libaio-devel \ libgcc \ libstdc++ \ libstdc++-devel \ libxcb \ libX11 \ libXau \ libXi \ libXtst \ libXrender \ libXrender-devel \ make \ net-tools \ nfs-utils \ smartmontools \ sysstat \ e2fsprogs \ e2fsprogs-libs \ fontconfig-devel \ expect \ unzip \ openssh-clients \ readline* \  tigervnc* \ psmisc --skip-brokenCompat-libstdc ++-33-3.2.3-72.el7.x86_64.rpmRPM -e ksh-20120801-142.el7.x86_64 RPM -ivh compat-libstdc++ -33-3.3-72.el7.x86_64 RPM -ivh PDKSH - 5.2.14-37. El5. X86_64. RPM## Check the installation of dependency packages
rpm -q bc binutils compat-libcap1 compat-libstdc++-33 gcc gcc-c++ elfutils-libelf elfutils-libelf-devel glibc glibc-devel ksh libaio libaio-devel libgcc libstdc++ libstdc++-devel libxcb libX11 libXau libXi libXtst libXrender libXrender-devel make net-tools nfs-utils smartmontools sysstat e2fsprogs e2fsprogs-libs fontconfig-devel expect unzip openssh-clients readline | grep "not installed"
Copy the code

3 Network Configuration

  • Rac01:
Configure the Public IP addressNmcli connection modify eth0 ipv4.addresses 192.168.56.10/24 ipv4.gateway 192.168.56.1 ipv4.method manual autoconnect yes## Configure the Private IP addressNmcli connection modify eth1 ipv4. Addresses 172.0.0.1/24 ipv4. Method manual autoconnect yes# # to take effect
nmcli connection up eth0
nmcli connection up eth1
Copy the code
  • Rac02:
Configure the Public IP addressNmcli connection modify eth0 ipv4.addresses 192.168.56.11/24 ipv4.gateway 192.168.56.1 ipv4.method manual autoconnect yes## Configure the Private IP addressNmcli connection modify eth1 ipv4. Addresses 172.0.0.2/24 ipv4. Method manual autoconnect yes# # to take effect
nmcli connection up eth0
nmcli connection up eth1
Copy the code

4 Storage Configuration (RAC01 & RAC02)

For details about how to configure ISCSI shared storage, see:This section describes how to configure an ISCSI shared storage device in Windows

##iscsi identifies shared storage
yum install -y iscsi-initiator-utils*
Targetname, 10.211.55.18 is the IP address of the iscsi shared storage deviceIscsiadm -m discovery -t st -p 10.211.55.18## Connect to shared storageIscsiadm - m node iqn.2008-08.com.starwindsoftware:10.211.55.18-lucifer - p - T 10.211.55.18 -l LSBLKInstall multipath # #
yum install -y device-mapper*
mpathconf --enable --with_multipathd y
## View the scsi_ID of the shared disk
/usr/lib/udev/scsi_id -g -u /dev/sdb
/usr/lib/udev/scsi_id -g -u /dev/sdc
/usr/lib/udev/scsi_id -g -u /dev/sdd
/usr/lib/udev/scsi_id -g -u /dev/sde
/usr/lib/udev/scsi_id -g -u /dev/sdf
/usr/lib/udev/scsi_id -g -u /dev/sdg
Configure multipath, wwiD as scsi_ID, alias can be customized, here configure 3 OCR disks, 3 DATA disks
cat <<EOF>/etc/multipath.conf defaults { user_friendly_names yes } blacklist { devnode "^sda" } multipaths { multipath { wwid "27e2b3ddbd14752bb" alias ocr_1 } multipath { wwid "27e2b3ddb87ff88ee" alias ocr_2 } multipath { wwid "27e2b3ddb39fd2463" alias ocr_3 } multipath { wwid "2852b96c1283206bf6" alias data_1 } multipath { wwid "2852b96c12e8449cb9" alias data_2 } multipath { wwid "2852b96c12fc938e95" alias data_3 } } EOF
## Activate multipath
multipath -F
multipath -v2
multipath -ll
## Configure UDEV to bind disks
for i in ocr_* data_*; do
	printf "%s %s\n" "$i" "$(udevadm info --query=all --name=/dev/mapper/"$i" | grep -i dm_uuid)" >>/dev/mapper/udev_info
done
while read -r line; do
	dm_uuid=$(echo "$line" | awk -F'=' '{print $2}')
	disk_name=$(echo "$line" | awk '{print $1}')
	echo "KERNEL==\"dm-*\",ENV{DM_UUID}==\"${dm_uuid}\",SYMLINK+=\"asm_${disk_name}\",OWNER=\"grid\",GROUP=\"asmadmin\",MODE=\"0660\"" >>/etc/udev/rules.d/99-oracle-asmdevices.rules
done </dev/mapper/udev_info
# # overloading udev
udevadm control --reload-rules
udevadm trigger --type=devices
Copy the code

5 Hosts file configuration (RAC01&RAC02)

cat <<EOF>>/etc/hosts #Public IP 192.168.56.10 rac01 192.168.56.11 rac02 #Private IP 172.0.0.1 rac01-priv 172.0.0.2 rac02-priv Rac01-vip 192.168.56.21 RAC02-VIP #Scan IP 192.168.56.110 rac-scan EOF
Copy the code

6 Firewall Configuration (RAC01 & RAC02)

systemctl stop firewalld.service
systemctl disable firewalld.service
systemctl status firewalld.service
Copy the code

7 SELinux Configuration (RAC01 & RAC02)

## Take effect after restart
sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 0
## Check after restart
getenforce
Copy the code

8 Time Synchronization Configuration (RAC01 & RAC02)

To disable the chronyd # #
yum install -y chrony
timedatectl set-timezone Asia/Shanghai
systemctl stop chronyd.service
systemctl disable chronyd.service
To disable the NTPD # #
yum install -y ntpdate
##10.211.55.200 is the IP address of the time server. It synchronizes the system time at 12:00 every day
cat <<EOF>>/var/spool/cron/root 00 12 * * * /usr/sbin/ntpdate -u 10.211.55.200&& /usr/sbin/hwclock -w EOF
## View scheduled tasks
crontab -l
## Execute manually/usr/sbin/ntpdate -u 10.211.55.200&& /usr/sbin/hwclock -wCopy the code

9 Close the Transparency page and NUMA (RAC01 & RAC02)

## Close the transparent page and NUMA and restart it to take effect
sed -i 's/quiet/quiet transparent_hugepage=never numa=off/' /etc/default/grub
grub2-mkconfig -o /boot/grub2/grub.cfg
## Check whether it takes effect after the restart
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /proc/cmdline
Copy the code

10 Avahi – Daemon configuration (RAC01 & RAC02)

yum install -y avahi*
systemctl stop avahi-daemon.socket
systemctl stop avahi-daemon.service
pgrep -f avahi-daemon | awk '{print "kill -9 "$2}'
# # configuration NOZEROCONF = yes
cat <<EOF>>/etc/sysconfig/network
NOZEROCONF=yes
EOF
Copy the code

11 System Parameter Configuration (RAC01 & RAC02)

## Configure parameter file
memTotal=$(grep MemTotal /proc/meminfo | awk '{print $2}')
totalMemory=$((memTotal / 2048))
shmall=$((memTotal / 4))
if [ $shmall -lt 2097152 ]; then
	shmall=2097152
fi
shmmax=$((memTotal * 1024 - 1))
if [ "$shmmax" -lt 4294967295 ]; then
	shmmax=4294967295
fi
cat <<EOF>>/etc/sysctl.conf
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = $shmall
kernel.shmmax = $shmmax
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth1.rp_filter = 2
EOF
# # to take effect
sysctl -p
Copy the code

12 System Resource Limitation Configuration (RAC01 & RAC02)

# # configuration limits. Conf
cat <<EOF>>/etc/security/limits.conf oracle soft nofile 1024 oracle hard nofile 65536 oracle soft stack 10240 oracle hard stack 32768 oracle soft nproc 2047 oracle hard nproc 16384 oracle hard memlock 134217728 oracle soft memlock 134217728 grid soft nofile 1024 grid hard nofile 65536 grid soft stack 10240 grid hard stack 32768 grid soft nproc 2047 grid hard nproc 16384 EOF
# # configure PAM. D/login
cat <<EOF>>/etc/pam.d/login session required pam_limits.so session required /lib64/security/pam_limits.so EOF
Copy the code

13 User and Group, Directory Creation (RAC01 & RAC02)

# # group created/usr/sbin/groupadd -g 54321 oinstall /usr/sbin/groupadd -g 54322 dba /usr/sbin/groupadd -g 54323 oper /usr/sbin/groupadd  -g 54324 backupdba /usr/sbin/groupadd -g 54325 dgdba /usr/sbin/groupadd -g 54326 kmdba /usr/sbin/groupadd -g 54327 asmdba /usr/sbin/groupadd -g 54328 asmoper /usr/sbin/groupadd -g 54329 asmadmin /usr/sbin/groupadd -g 54330 racdba## User creation
/usr/sbin/useradd -u 11012 -g oinstall -G asmadmin,asmdba,asmoper,dba,racdba,oper grid
/usr/sbin/useradd -u 54321 -g oinstall -G asmdba,dba,backupdba,dgdba,kmdba,racdba,oper oracle
Change user password to oracle
echo "oracle" |passwd oracle --stdin
echo "oracle" |passwd grid --stdin
## Create software directoryThe mkdir -p/u01 / app / 11.2.0 / grid mkdir -p/u01 / app/grid mkdir -p/u01 / app/oracle/product / 11.2.0 / db mkdir -p /u01/app/oraInventory mkdir -p /backup mkdir -p /home/oracle/scripts chown -R oracle:oinstall /backup chown -R oracle:oinstall /home/oracle/scripts chown -R grid:oinstall /u01 chown -R grid:oinstall /u01/app/grid chown -R Grid :oinstall/U01 /app/11.2.0/grid chown -r Grid :oinstall/U01 /app/oraInventory chown -r Oracle :oinstall/U01 /app/oracle  chmod -R 775 /u01Copy the code

14 Setting Environment Variables (RAC01 & RAC02)

  • The grid users:
cat <<EOF>>/home/grid/.bash_profile ################OracleBegin######################### umask 022 export TMP=/tmp export TMPDIR=\$TMP export NLS_LANG=AMERICAN_AMERICA.AL32UTF8 export ORACLE_BASE=/u01/app/grid export ORACLE_HOME=/ U01 /app/11.2.0/grid export ORACLE_TERM=xterm export TNS_ADMIN=\$ORACLE_HOME/network/admin export LD_LIBRARY_PATH=\$ORACLE_HOME/lib:/lib:/usr/lib export ORACLE_SID=+ASM1 export PATH=/usr/sbin:\$PATH export PATH=\$ORACLE_HOME/bin:\$ORACLE_HOME/OPatch:\$PATH alias sas='sqlplus / as sysasm' export PS1="[\`whoami\`@\`hostname\`:"'\$PWD]\$ ' EOF
Copy the code

Note: ORACLE_SID is different for each node (+ASM1/+ASM2).

  • The oracle user:
cat <<EOF>>/home/oracle/.bash_profile
################OracleBegin#########################
umask 022
export TMP=/tmp
export TMPDIR=\$TMP
export NLS_LANG=AMERICAN_AMERICA.AL32UTF8
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=\$ORACLE_BASE/product/11.2.0/db
export ORACLE_HOSTNAME=rac01
export ORACLE_TERM=xterm
export TNS_ADMIN=\$ORACLE_HOME/network/admin
export LD_LIBRARY_PATH=\$ORACLE_HOME/lib:/lib:/usr/lib
export ORACLE_SID=orcl1
export PATH=/usr/sbin:\$PATH
export PATH=\$ORACLE_HOME/bin:\$ORACLE_HOME/OPatch:\$PATH
alias sas='sqlplus / as sysdba'
export PS1="[\`whoami\`@\`hostname\`:"'\$PWD]\$ '
################OracleEnd#########################
EOF
Copy the code

Note: ORACLE_HOSTNAME (RAC01 / RAC02) and ORACLE_SID (ORCL1 / ORCL2) are different for each node.

15 Uploading and Decompressing Installation Media (RAC01)

## Create a directory for storing installation media
mkdir /soft
Upload the installation media to the /soft directory
p13390677_112040_Linux-x86-64_1of7.zip
p13390677_112040_Linux-x86-64_2of7.zip
p13390677_112040_Linux-x86-64_3of7.zip
## Decompress the installation media
cd /soft
unzip -q p13390677_112040_Linux-x86-64_1of7.zip
unzip -q p13390677_112040_Linux-x86-64_2of7.zip
unzip -q p13390677_112040_Linux-x86-64_3of7.zip
## Authorization directory
chown -R oracle:oinstall /soft/database
chown -R grid:oinstall /soft/grid
## install cvuqdisk (rac01&rac02)
cd/ soft/grid/RPM RPM - the ivh cvuqdisk 1.0.9-1 RPM## Transfer to node 2 installationRPM rac2:/ TMP RPM -ivh/TMP /cvuqdisk-1.0.9-1.rpmCopy the code

Note: Only need to upload the decompression at node one

At this point, the preparations are complete.

Install Grid software (RAC01)

  • Configure the VNC GUI for the grid user
Switch to user grid as user root
su - grid
## Run vncserver and enter the password as prompted
vncserver
## On the VNC client, enter 192.168.56.10:1, enter the password you just entered to connect.
Copy the code

  • Right click to open terminal:

  • Start installation:
## Apply environment variables
source ~/.bash_profile
Go to the installation directory
cd /soft/grid
## Execute setup program to start installation, add jar package to prevent popover does not show the problem. / runInstaller jreLoc/etc/alternatives/jre_1. 8.0Copy the code

  • Skip version updates:

  • Select cluster installation mode:

  • Custom mode installation:

  • Language selection:

  • Change the cluster name and scan name:

Note: The SCAN name must be the same as the scan name configured in /etc/hosts.

  • Configure mutual trust between Grid user nodes:

Note: Click Add to Add node 2, set pubile Hostname to RAC02, virtual Hostname to RAC02-VIP, enter password: Oracle, and click Setup to start trust.

  • Click Test to Test trust:

  • Check whether the network information is correct:

  • Select THE ASM installation mode:

  • To fill in OCR Adjudication Disk information:

Note: The number of disks in the OCR mode External, Normal, and High is 1, 3, and 5.

  • Enter the password of user SYS for the ASM instance:

  • Not using IPMI:

  • Select user group, default:

  • Select the grid installation directory, default:

  • Check before installation:

  • Start installation:

  • Execute the root script :(rac01&rac02)

Note: When Linux7 is installed with 11204 version, there is a BUG when executing root.sh. You need to install patch 18370031 before executing root.sh to fix the BUG.

  • 18370031 patch installation :(rac01&rac02)
## Upload patch package
p18370031_112040_Linux-x86-64.zip
Decompress the patch package
cd /soft
unzip -q p18370031_112040_Linux-x86-64.zip
## Authorize the patch pack
chown -R grid:oinstall /soft/18370031
## Start patch installation, both nodes need to be executed
opatch napply -oh $ORACLE_HOME -local /soft/18370031 -silent
Copy the code

  • Start root script :(rac01&rac02)
## Execute as user root/ u01 / app/oraInventory/orainstRoot. Sh/u01 / app / 11.2.0 / grid/root. ShCopy the code

Execution process is too long, do not record;

  • Continue installation:

If DNS resolution is not configured, ignore it.

  • Grid cluster software is successfully installed:

  • Add an ASM DATA disk:

1. Create silently

asmca -silent -sysAsmPassword oracle -asmsnmpPassword oracle -oui_internal -configureASM -diskString '/dev/asm*' -diskGroupName DATA -diskList /dev/asm_data_1,/dev/asm_data_2,/dev/asm_data_3 -redundancy NORMAL -au_size 1
Copy the code

2. Graphical creation

asmca
Copy the code

  • Click Create to create DATA:

  • Click OK to create DATA:

You are advised to restart the two hosts and check whether the Grid cluster is running properly after the restart.

Install the Oracle software

  • Configure the VNC GUI for the Oracle user
Switch to user grid as user root
su - oracle
## Run vncserver and enter the password as prompted
vncserver
## On the VNC client, enter 192.168.56.10:2 and enter the password you just entered to connect.
Copy the code

  • Right click to open terminal:

  • Start installation:
## Apply environment variables
source ~/.bash_profile
Go to the installation directory
cd /soft/database
## Execute setup program to start installation, add jar package to prevent popover does not show the problem. / runInstaller jreLoc/etc/alternatives/jre_1. 8.0Copy the code

  • Oracle update email not accepted:

  • Skip software updates:

  • Choose to install only Oracle software:

  • Configure oracle user trust:

Enter your password, click Setup to start, and click Test when successful.

  • Language selection:

  • Select enterprise edition installation:

  • Check the user group, default:

  • Check before installation:

  • Start installing the Oracle software:

  • Error solved:

Ins_emagent. mk: Ins_emagent. mk: ins_emagent.mk: ins_emagent.

Execute the following command for both nodes, then click continue :(rac01&rac02)

sed -i 's/^\(\s*\$(MK_EMAGENT_NMECTL)\)\s*$/\1 -lnnz11/g' "$ORACLE_HOME/sysman/lib/ins_emagent.mk
Copy the code
  • Run the root.sh script :(rac01&rac02)

  • Click Next, the installation is successful:

At this point, the Oracle software has been successfully installed.

Note: If you need to install PSU patches, you are advised to do so before building the library.

Install the patch syntax as follows :(rac01&rac02)

opatch auto /soft/31718723 -oh $ORACLE_HOME
Copy the code

Create a database instance

dbca
Copy the code
  • Select rac mode:

  • Select Create database:

  • Select a custom template:

  • Enter the database instance ID:

  • To disable EM and retain automated system jobs:

  • Enter the password of SYS. Remember:

  • Select the ASM disk for storing DATA: DATA

  • Close the flashback area, and the database can be opened at any time:

  • Select a custom component, default:
  • Configure initialization parameters:

Note: If you are using automanaged memory, 70% physical memory is recommended, provided /etc/shm is the same size as physical memory.

  • The default block is 8K, and the number of processes is increased to 1500:

  • Select database character set, default AL32UTF8:

  • Start building repository:

  • The database was created successfully:

At this point, the database instance is created.

Database optimization configuration (RAC01)

1 Enable the database archiving mode

## Close the database instance
srvctl stop database -d orcl
## Enable a single node to mount mode
srvctl start instance -d orcl -i orcl1 -o mount
## Open archive
alter database archivelog;
## Set the archive path
ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='LOCATION=+DATA';
## Restart database instance
srvctl stop instance -d orcl -i orcl1
srvctl start database -d orcl
## Check the archive
archive log list
Copy the code

2 Configure periodically deleting an archive scheduled task

## Enter the oracle user
su - oracle
## Write script
{
	echo '#! /bin/bash'
    echo 'source ~/.bash_profile'
    echo 'deltime=`date +"20%y%m%d%H%M%S"`'
    echo "rman target / nocatalog msglog /home/oracle/scripts/del_arch_\${deltime}.log<<EOF"
    echo 'crosscheck archivelog all; '
    echo "delete noprompt archivelog until time 'sysdate-7';"
    echo "delete noprompt force archivelog until time 'SYSDATE-10';"
    echo 'EOF'
} >>/home/oracle/scripts/del_arch.sh
## Write scheduled task
cat <<EOF>>/var/spool/cron/oracle
12 00 * * * /home/oracle/scripts/del_arch.sh
EOF
## Execute tests manually
/home/oracle/scripts/del_arch.sh
Copy the code

3 Set the password to never expire

ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;
Copy the code
For details, please refer to:Oracle sets user passwords to never expire
After you’ve seen how to install Oracle RAC step by step, have you ever thought about scripting to automate the installation? If so, check out the column:Oracle script one-click installation

That’s the end of this sharing

If you think the article is helpful to you, like, collect, follow, comment, one key four support, your support is the biggest motivation for my creation.

Technical communication can pay attention to the public number: Lucifer think twice before you act