Linux command pickups – Commonly used development aid class commands

Original: Coding diary (wechat official ID: Codelogs), welcome to share, reprint please reserve the source.

Introduction to the

There are a lot of commands in Linux, such as curl, mysql, SSH, vim, etc. This article gives a general introduction to them.

Use curl to debug an interface

Curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl curl

# send HTTP request, default GET method
curl 'http://localhost/user/get'

Curl curl curl curl curl curl curl curl curl curl curl
# If content-type is not specified, the default is Application /x-www-form-urlencoded
curl 'http://localhost/user/add' -d 'id=32&name=zhangsan'
# --data-urlencode, urlencode the request body before sending
curl 'http://localhost/user/add' --data-urlencode 'id=32&name=zhangsan'

# add the request header with -h, specifying that the content-type is application/json
curl 'http://localhost/user/add' -H 'Content-Type: application/json; charset=UTF-8' -d '{"id":32,"name":"zhangsan"}'

Using -x, you can specify the request method
curl -X DELETE 'http://localhost/user/delete'
curl -X PUT 'http://localhost/user/modify' -d 'id=32&name=zhangsan'



# -g, you can concatenate the request body to the URL, so the following two calls are equivalent
curl -G 'http://localhost/user/add' -d 'id=32&name=zhangsan'
curl 'http://localhost/user/add? id=32&name=zhangsan'

-i displays both the response header and the content. -i displays only the response header
curl -i 'http://localhost/user/add? id=32&name=zhangsan'

Curl = curl = curl = curl = curl
curl -v http://www.baidu.com
-s displays no progress information and error information. -s displays error information
curl -sS http://www.baidu.com

$curl = $curl = $curl = $curl
$ time curl "http://www.baidu.com" -o /dev/null -s
...
real    0m0.066s
user    0m0.010s
sys     0m0.000s
-w prints the detailed time of the request, including the time of DNS search, the time of establishing the connection, the time of receiving the first byte of the response from the server, and the time of obtaining all the response results. Note that the time is cumulative, that is, from the moment when the command is executed
$ curl -w " time_namelookup:%{time_namelookup}s\n time_connect:%{time_connect}s\n time_starttransfer:%{time_starttransfer}s\n time_total:%{time_total}s\n speed_download:%{speed_download}\n http_code:%{http_code}" "http://www.baidu.com"-o /dev/null -s time_namelookup:0.009906s time_connect:0.035826s time_startTransfer :0.062370s time_total:0.062406s Speed_download: 38403.000 http_code: 200Copy the code

Curl can also be used to upload and download files.

-# display the progress bar. -o Download the response body as a file
curl -LO -# http://localhost/user/avatar/xxxxx.png  
# wget is also often used to download files
wget http://localhost/user/avatar/xxxxx.png  

-f is similar to the form in HTML. The default content-type is multipart/form-data, which is a common way of uploading files on the Web
curl http://localhost/user/avatar/upload -F 'id=32' -F 'avatar=@./xxxxx.png'

# upload file, this is also a way to upload a file, the body of the request is directly the file content, the entire servercurl http://localhost/user/avatar? id=32 --data-binary'@./xxxxx.png' -#


# curl can also access the FTP server, as follows:
# Download FTP fileThe curl -u username, password, ftp://localhost/user/avatar/xxxxx.png - OUpload files to FTPThe curl -u username: password ftp://localhost/user/avatar/ - t. / XXXXX. PNGCopy the code

Curl curl curl curl curl curl curl curl curl curl curl curl

Curl: (60) SSL certificate problem: Unable to get local issuer certificate
You can ignore HTTPS certificate verification with -k
curl -k https://www.ywnds.com/

Curl will block for a long time. You need to specify a timeout for curl
curl --connect-timeout 2 --max-time 2 http://www.baidu.com

Binary output can mess up your terminal. Use "--output -" to tell
Use --compressed, gzip -d
curl -H 'Accept-Encoding:gzip' http://www.baidu.com/ --compressed
curl -H 'Accept-Encoding:gzip' http://www.baidu.com/ | gzip -d

# Some websites still use GBK coding, which will lead to curl result garbled, or even directly reported Failed writing body error, in this case, we need to use iconv to convert coding
curl -i http://www.xxxxx.com | iconv -f GBK
Curl curl curl curl curl curl curl curl curl curl curl curl curl
curl -w '\n' http://localhost/health
Copy the code

Curl can define almost every aspect of an HTTP request, so I often use curl as a tool to communicate the details of the interface with my colleagues. I have also seen many API documents that use curl directly to define HTTP requests.

Use the mysql command to execute SQL

The mysql command is simple to use, as follows:

mysql -h localhost -P 3306 -u root -psecret test_db
Copy the code

After executing the above command, you will enter the SQL interactive interface, and then you can execute the SQL statement. Pay attention to the way of specifying the password, there is no space between -p and secret.

Other useful options are as follows:

options	role
-A	Do not load table names, column names and other metadata, when the network is not good can speed up the command start speed, the price is that the Tab key cannot automatically complete table names and column names
–default-character-set utf8mb4	Specifies a character set to avoid garbled characters in some environments
-e	Specify the SQL to be executed and exit the command interface
-N	The column name is not displayed in the query result
-t	The query results are displayed in tables
-B	The query results are not displayed in tables, and fields are separated by tabs
-X	The query results are output in XML format
-U	Perform queries and updates safely, such as if you forgot to limit large tables
–pager	Specifies the result paging display, default is standard output,--pager=lessRepresents the use of less as a paging display

Pager can be set to any Linux command to process query results, as follows:

$ mysql -h localhost -P 3306 -u root -psecret test_db
If pager is not set, the query result will be output directly
mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00'; +-------+------------------------------------+---------------------+---------------------+-------+----------+ | id | log_info | create_time | update_time | seq | add_time | + -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- + | | 7706 DOwhV4Odbuf0wVwyZxrJFbBOEGF0fhf7Ab | 2021-04-18 17:22:48 | 2022-03-11 00:25:49 | 7706 | 0 | | 8155 | abc | 2021-04-18 16:55:42 | 2022-07-18 21:28:29 | 8155 | 0 | | 14535 | uAXFeCJ3jtxhiQJS26q43MarWL5M | 2021-04-18 17:06:54 | 2022-06-26 21:19:42 | 14535 | 0 | | 15303 | abc | 2021-04-18 17:30:18 | 2022-05-04 00:38:15 | 15303 | 0 | | 31933 | cLfKXMYkaIta16kzmyWKFC | 2021-04-18 17:06:19 | 2022-09-14 20:14:10 | 31933 | 0 | | 32503 | Rb6yMJ6fvViF28YWN9GA | The 2021-04-18 17:33:59 | 2022-03-14 17:38:04 | 32503 | | 0 +-------+------------------------------------+---------------------+---------------------+-------+----------+ 6 rowsin set (0.29 sec)

# set pager to cat -n and add line number to output
mysql> pager cat -n
PAGER set to 'cat -n'
mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00'; 1 +-------+------------------------------------+---------------------+---------------------+-------+----------+ 2 | id |  log_info | create_time | update_time | seq | add_time | 3 + -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- - | | 7706 + 4  DOwhV4Odbuf0wVwyZxrJFbBOEGF0fhf7Ab | 2021-04-18 17:22:48 | 2022-03-11 00:25:49 | 7706 | 0 | 5 | 8155 | abc | 2021-04-18  16:55:42 | 2022-07-18 21:28:29 | 8155 | 0 | 6 | 14535 | uAXFeCJ3jtxhiQJS26q43MarWL5M | 2021-04-18 17:06:54 | 2022-06-26  21:19:42 | 14535 | 0 | 7 | 15303 | abc | 2021-04-18 17:30:18 | 2022-05-04 00:38:15 | 15303 | 0 | 8 | 31933 | cLfKXMYkaIta16kzmyWKFC | 2021-04-18 17:06:19 | 2022-09-14 20:14:10 | 31933 | 0 | 9 | 32503 | Rb6yMJ6fvViF28YWN9GA | The 2021-04-18 17:33:59 | 2022-03-14 17:38:04 | 32503 | | 0 to 10 +-------+------------------------------------+---------------------+---------------------+-------+----------+ 6 rowsin set (0.29 sec)

# set pager to grep ABC and only look at lines containing ABC
mysql> pager grep abc
PAGER set to 'grep abc'
mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00';
|  8155 | abc                                | 2021-04-18 16:55:42 | 2022-07-18 21:28:29 |  8155 |        0 |
| 15303 | abc                                | 2021-04-18 17:30:18 | 2022-05-04 00:38:15 | 15303 |        0 |
6 rows in set (0.29 sec)

Pager = less pager = less pager = less pager
mysql> pager less -iSX
PAGER set to 'less -iSX'
mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00'; +-------+------------------------------------+---------------------+---------------------+-------+----------+ | id | log_info | create_time | update_time | seq | add_time | + -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - + -- -- -- -- -- - + -- -- -- -- -- -- -- -- -- -- + | | 7706 DOwhV4Odbuf0wVwyZxrJFbBOEGF0fhf7Ab | 2021-04-18 17:22:48 | 2022-03-11 00:25:49 | 7706 | 0 | | 8155 | abc | 2021-04-18 16:55:42 | 2022-07-18 21:28:29 | 8155 | 0 | | 14535 | uAXFeCJ3jtxhiQJS26q43MarWL5M | 2021-04-18 17:06:54 | 2022-06-26 21:19:42 | 14535 | 0 | | 15303 | abc | 2021-04-18 17:30:18 | 2022-05-04 00:38:15 | 15303 | 0 | | 31933 | cLfKXMYkaIta16kzmyWKFC | 2021-04-18 17:06:19 | 2022-09-14 20:14:10 | 31933 | 0 | | 32503 | Rb6yMJ6fvViF28YWN9GA | The 2021-04-18 17:33:59 | 2022-03-14 17:38:04 | 32503 | | 0 +-------+------------------------------------+---------------------+---------------------+-------+----------+ (END)Set pager to md5sum - to generate MD5 for query results
Compare md5 to see if the current time period has changed
mysql> pager md5sum -
PAGER set to 'md5sum -'
mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00';
40ed2c49a72ff68ba11b97d843bb4da2  -
6 rows in set (0.29 sec)

mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00';
40ed2c49a72ff68ba11b97d843bb4da2  -
6 rows in set (0.31 sec)

# use cat to save the first query result to a file, then use ICdiff to compare whether the query result changes
mysql> pager cat > result.txt
PAGER set to 'cat > result.txt'
mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00';
6 rows in set(0.31 SEC) mysql> pager icdiff result. TXT /dev/stdin pagerset to 'icdiff result.txt /dev/stdin'
mysql> select * from app_log where create_time >= '2021-04-18' and create_time < 'the 2021-04-18 17:35:00';
Copy the code

To restore the Pager, just typenopagerPress Enter.

In addition, the related mysql command line options can be permanently configured in the mysql. CNF configuration file, so as to avoid the need to carry their own common options in the command line, as follows:

$ sudo vim /etc/mysql/conf.d/mysql.cnf
[mysql]
default-character-set=utf8mb4
pager=less -iSFX
safe-updates=TRUE
Copy the code

Use SSH to manage hosts

Logging In to a Remote Host

Generally, we use SSH to log in and manage remote hosts, such as installing software and troubleshooting. Common uses are as follows:

Log in to host 192.168.0.1 as user work and enter the password of user work$SSH [email protected]# -p Specifies the port. Sometimes the host does not use the default port 22 for security reasons$SSH [email protected] -p 8022Run the command directly on the remote host
$ ssh [email protected] -e 'ping www.baidu.com'PING www.a.shifen.com (112.80.248.76) 56(84) bytes of data.64 bytes from 112.80.248.76 (112.80.248.76): Icmp_seq =1 TTL =54 time=19.1 ms 64 bytes from 112.80.248.76 (112.80.248.76): icmp_seq=2 TTL =54 time=19.9 msCopy the code

Uploading and downloading files

In addition to logging in to the host remotely, we often need to upload and download files as follows:

SSH SSH SSH SSH SSH SSH SSH SSH SSH SSH
Log /home/work/logs/app.log on 192.168.0.1$SCP [email protected]: / home/work/logs/app. Enter the/logs //logs/app.log to /home/work/logs/ 192.168.0.1$SCP. / logs/app. The log [email protected]: / home/work/logs /The SCP source file and the target file can be on a remote host.$SCP [email protected]: / home/work/logs/app. The log [email protected]: / home/work/logs/app. The logYou can actually upload and download using SSH
The pv command is used to check the progress, but the cat command can also be used$SSH [email protected]'cat /home/work/logs/app.log' | pv > ./logs/app.log

# use SSH to upload files, where the pv command is used to check the progress$pv. / logs/app. The log | SSH to [email protected]'cat > /home/work/logs/app.log'

Compare remote files with local file contents$SSH [email protected]'cat /path/to/remotefile'| icdiff/path/to/localfile -Copy the code

SSHFS Mounts a remote directory

If you feel that uploading and downloading is a hassle, you can use SSHFS to mount directories from the remote host directly to the local machine, so that you can access remote files as if they were local files.

SSHFS is a user-mode file system based on FUSE and SSH. You can use SSHFS wherever you can use SSH. As a user-mode file system, SSHFS does not require special permissions as long as the network is connected.

# SSHFS mount directory
$ sshfs [email protected]:/home/work/logs /home/work/logs

# list files
$ ls /home/work/logs
app_2022-01-23T12:05:51.log  app_2022-01-23T12:05:53.log  app_2022-01-23T12:05:57.log  app_2022-01-23T12:06:00.log

Use SSH to list remote directories with the same result as above$SSH [email protected]'ls /home/work/logs'
app_2022-01-23T12:05:51.log
app_2022-01-23T12:05:53.log
app_2022-01-23T12:05:57.log
app_2022-01-23T12:06:00.log

Copy the code

You may wonder, why the above result a newline while another line, this is because the ls will detect whether object is the output device, if it is terminal display will rank, otherwise the landscape, not letter you can see for ls | cat.

In fact a lot of order will have this kind of behavior, such as curl default does not display the schedule, but if the object is not the output device, such as curl | cat, will show the progress information.

Password-free login

Once you use SSH in this way, you will find it troublesome to enter a password every time. Fortunately, SSH provides a private key authentication mechanism, so we can generate a pair of public and private keys to avoid entering a password every time.

SSH key: id_ras; public key: id_ras.pub; public key: id_ras
ssh-keygen -o -t rsa -b 4096 -C "[email protected]"

Use ssh-copy-id to upload the public key to the target hostSSH - copy - id - I ~ /. SSH/id_rsa pub [email protected]# if ssh-copy-id is not available, the public key can be uploaded in this wayThe cat ~ /. SSH/id_rsa. Pub | SSH [email protected]'cat >> ~/.ssh/authorized_keys'

# test whether the publickey is effective if Authentication succeeded (publickey) is seen. ", indicating successSSH - vT [email protected] < / dev/nullCopy the code

Github/Gitee/Gitlab/github/ gitlab

1. Use ssh-kengen to generate a key pair.
2. Copy the public key to the SSH key management page on Github and save it.
3. The test configuration succeeded. Procedure

Port forwarding

Sometimes, two hosts A and C cannot communicate with each other, but A and B can communicate with each other, and B and C can communicate with each other. In this case, we can use host B to perform port forwarding so that host A can access the network service on host C.

There are many software that can implement port forwarding, such as Iptables, Socat, PortMap, rinetd, etc. SSH is also a common one. The usage is as follows:

SSH local port forwarding: SSH local port forwarding: SSH local port forwarding: SSH local port forwarding: SSH local port forwarding: SSH local port forwarding: SSH local port forwarding: SSH local port forwarding: SSH local port forwarding: SSH local port forwarding
ssh -fgN -L2001:host_c:80 work@host_b
SSH remote port forwarding is performed on machine B, port 22333 is enabled on machine A (enabled by SSHD service on machine A), and the traffic on port 22333 of machine A is transferred to port 80 of machine C through machine B (transferred in the middle)
ssh -fgN -R 22333:host_c:80 work@host_a

SSH remote dynamic port forwarding, executed on machine B, will enable socks proxy service on port 2222 on machine A, and the proxy data will be sent to machine B, which in turn forwards to any network service that can be accessed by machine B
ssh -fgN -R 2222 work@host_a
The socks proxy service is enabled on port 2222 on machine A. The proxy data is sent to machine B, which in turn forwards the data to any network service that machine B can access
ssh -Nfg -D 2222 work@host_b

# Machine A logs in to machine C through the local SOCKS proxy
ssh -o ProxyCommand='ncat --proxy 127.0.0.1:2222 --proxy-type socks5 %h %p' work@host_c
# Machine A logs in to machine C using SSH from machine B
ssh -t work@host_b 'ssh work@host_c'
Copy the code

Why are there two forwarding modes: local port forwarding and remote port forwarding?

This depends on the direction of network connection allowed. For example, in some cases, B can connect to A, but A cannot connect to B. In this case, remote port forwarding is required!

conclusion

In addition to these common commands, there are many other types of utility commands in Linux, such as:

• Vim, emacs: For editing text files, you can actually configure them as idea-like ides, see Spacevim, spacemacs.
• Openssl: used to generate key files and certificates and perform encryption and decryption operations.
• Ffmpeg: can process audio and video files, such as MP4 to FLV, MP4 to GIF, recording, clipping, watching live and so on.
• Convert: The image format can be converted, such as PNG to JPEG.
• Jp2a: Pictures can be turned into character paintings.
• Wkhtmltopdf, xhtml2PDF can convert HTML to PDF, and Pandoc can convert various document formats.

Here is not an introduction, after the opportunity to take out a separate introduction.

Linux command gap-Getting Started Linux command gap-text Processing Linux command gap-Software Resource Observation Linux command gap-Hardware resource observation Linux command gap-Profiling tools Linux command gap-dynamic tracking tools Linux command gap-Understanding system load Linux command gap-network packet capture tool Linux command gap-view system information

Content of the past

Awk is really a magic tool for Linux text command tips (top) Linux text command tips (bottom) character encoding solution