Linux Three Musketeers Awk command Name
Awk pattern scanning and processing language
Copy the codeCommand role
Processing of text and data
Detailed instructions
Awk is a programming language for text and data processing under Linux/Unix. Data can come from standard input (STDIN), one or more files, or the output of other commands. It is used on the command line, but more as a script. Awk has a lot of built-in functionality, such as arrays, functions, and so on. This is what it has in common with C. Flexibility is awK’s biggest advantage.
Syntax format
awk [options] 'scripts' var=value filename
Copy the codeCommonly used parameters
- -f Specifies a delimiter (can be a string or regular expression)
- -f Reads the awk command from the script file
- -v var=value Assigns a value variable to pass the external variable to AWK
Basic script structure
awk ‘BEGIN{ print “start” } pattern{ commands } END{ print “end” }’ filename
An AWK script usually consists of a BEGIN statement, a pattern matching statement, and an END statement, all of which are optional
Working principle:
The first step is to execute the BEGIN statement. The second step is to read a line from the file or standard input. Then the pattern statement is executed to scan the file line by line until all files are readCopy the codeExamples show:
echo "hello " | awk 'BEGIN{ print "welcome" } END{ print "2017-08-08" }' welcome 2017-08-08 echo -e "hello" | awk 'the BEGIN {print "welcome"} {print} END {print "2017-08-08"}' welcome hello # 2017-08-08 without the print parameter is the default print only the current line echo | awk '{ a="hello"; b="nihao"; c="mingongge"; print a,b,c; } 'hello nihao mingongge # use print comma-delimited, printed in the blank space boundary echo | awk' {a = "MGG"; b="mingg"; c="mingongge"; print a" is "b" or "c; }' MGG is mingg or mingongge #awk print statementCopy the codeAwk variables
Built-in variables
FS # input field separator (-f) default space RS # input record separator (-f) default space RS # input record separator (-f) Default space ORS # output record separator, default newlineCopy the codeExternal variables
[mingongge@ ~]#a=100
[mingongge@ ~]#b=100
[mingongge@ ~]#echo |awk '{print v1*v2 }' v1=$a v2=$b
10000
Copy the codeAwk operation and judgment
Arithmetic operator
[mingongge@ ~]#awk 'BEGIN{a="b"; print a,a++,a--,++a; }' b 0 1 1 [mingongge@ ~]#awk 'BEGIN{a="0"; print a,a++,a--,++a; }' 0 0 1 1 [mingongge@ ~]#awk 'BEGIN{a="0"; print a,a++,--a,++a; }' 0 0 0 1 # As in other programming languages, all are used as arithmetic operators to operate, the operands are automatically converted to numeric values, and all non-numeric values are changed to 0Copy the codeThe assignment operator
= += -= *= /= %= ^= **= Regular operator
~! ~ matches regular expression/does not match regular expression logical operators
| | && operator logic or logic and relations
< <= > >=! = = other operators
$field references space string linker? : ternary operator ln whether there is a key value in the arrayCopy the codeAwk regular
^ Start of line locator $End of line locator. Matches any single character * Matches zero or more leading characters (including carriage return) + matches one or more leading characters? Match zero or one leading characters in [] to match any one character within a specified character groups / ^ [ab] [^] match is not in any one character within a specified character groups () subexpression \ | or escape characters ~,! ~ Matching or unmatching conditional statements x{m} x characters repeated m times x{m,} x characters repeated at least m times x{m, n} x characters repeated at least m times but not more than n times (parameter -posix or --re-interval must be specified)Copy the codeLinux swordsmen Sed command description
Sed is a powerful streaming text editor
Detailed instructions
Sed is a stream editor and a great tool for text processing. With more powerful processing with the re, the current line is stored in a temporary buffer called “pattern space,” and the contents of the buffer are then processed with the sed command, printed to the screen, and the next line is processed.
The command format
sed [options] 'command' file(s)
sed [options] -f scriptfile file(s)
Copy the codeCommonly used parameters
-e # Process the input text file with the specified instruction -n # cancel the default output (only the changed lines will be printed if used with the p command) -h # help -v # Display version informationCopy the codeCommon commands
A # insert text below current line I # insert text above current line C # change selected line to new text D # delete, Delete the selected line D # Delete the first line of the template block S # replace the specified character H # copy the contents of the template block to the memory buffer H # Append the contents of the template block to the memory buffer G # Obtain the contents of the memory buffer and replace the text in the current template block G # obtain the contents of the memory buffer N # Reads the next input line, processes the new line with the next command instead of using the first command n # appends the next input line to the template block and inserts a new line between the two, Change the current line number p # Print a matching line P #(uppercase) Print the first line of the template Q # exit Sed B #lable branch to the marked part of the script, branch to the end of the script if none exists r #file Read the line from file t #label if branch, Starting at the last line, if the condition is satisfied or the T, T command, it will branch to the labeled command, or at the end of the script, the T #label error branch, starting at the last line, if an error occurs or the T, T command, it will branch to the labeled command, ** w #file Write and append the first line of the template block to the end of the file **! # indicates that the following command takes effect on all unselected lines ** = # prints the current line number ** # # extends the comment to the next line break **Copy the codeSed replacement command
G # said inline fully replace replaced with s command USES the (global) p # said printing line w # said to write a template file x # said swap block of text in a text and buffer y # said translate one character for another character (but not for the regular expression) 1 # substring matching tag & The string tag has been matchedCopy the codeSed to regular
^ # Match the beginning of a line $# Match the end of a line. # Match any character that is not a newline * # Match zero or more characters [] # match a character in the specified range [^] # Match a character in the specified range (..) # match substring & # Save the search character to replace other characters < # match word start > # match word end x{m} # repeat character x, m times x{m,} # repeat character x, m times x{m,n} # repeat character x, m times at least, m times but not more than n timesCopy the codeSed common Example 1
Echo "hello world" | sed 's / / - / 1 g' hello world # global replace - since the first space, just only a space in the textCopy the code2. Delete operations
Sed '/^$/d' filename 'sed '2d' filename' Sed '$d' filename # sed '/^test/'d filename # delete the last line starting with testCopy the code3. Match the replacement
Echo "hello world" | sed 's/w + / [&] / g' [hello] [world] echo "hello world" | sed 's/w + / "&"/g' "hello," "world" # w + match each word, & echo string means to match to the AAA BBB | sed 's/([a-z] +) ([a-z] +) / [2] [1] /' (BBB) (AAA) # substring matching replacementCopy the code4. Scope of selection
Sed -n '/= 0/,/ Max /p' SVNserve.conf #min-encryption =0 #max-encryption = 256 # All lines in the range =0 to Max will be printedCopy the code5. Sed multi-point editing (-e)
[root@centos001 ~]#cat -n test 1 This is a test file 2 Welcome 3 to 4 here 5 Hello WORLD 6 7 Linux Centos6.8 8 redhat Sed -e '2,6d' -e 's/ Linux centos6.8/Linux centos6.8/ 'test this is a test file Linux centos6.8 Sed --expression=' /Linux centos6.8/Linux centos6.8/ '--expression=' /to/ to/' test** This is a test file Welcome TO Here Hello WORLD Linux CenTOs6.8 RedhatCopy the code6. Read and write
[root@centos001 ~]#cat test1 welcom to here [root@centos001 ~]#sed '/here/r test1' test this is a test file welcome to Sed -n '/centos6.8/w test2' test [root@centos001 ~]#cat test2 Linux centos6.8 # write all rows matching test file to Centos6.8 to test2 file, file may not existCopy the codeAppend and insert
[root@centos001 ~]#sed '/^l/a2017-08-08' test2 Linux centos6.8 2017-08-08 [root@centos001 ~]#sed '/^l/i2017-08-08' #sed '/^l/i2017-08-08 Test2 2017-08-08 Linux centos6.8 # insert 2017-08-08 ####### to match a line starting with l ################ [root@centos001 ~]#sed I '/^l/i2017-08-08' test2 [root@centos001 ~]#cat test2 2017-08-08 Linux centos6.8Copy the code8. Other command examples
[root@centos001 ~]#cat -n test2 1 2017-08-08 2 Linux centos6.8 3 08 4 5 test [root@centos001 ~]#**sed '/08/{n; s/l/L/; }' test2 2017-08-08 Linux centos6.8 08 test # [root@centos001 ~]#sed '1,4y/8/9/' test2 2017-09-09 Linux centos6.9 09 test # replace all numbers 8 with 9 [root@centos001 ~]#**sed '1q' test2** 2017-08-08 #Copy the code9. Print odd or even lines
[root@centos001 ~]#sed -n 'p; n' test2 20170808 08 [root@centos001 ~]#sed -n 'n; P 'test2 Linux centos6.8 test [root@centos001 ~]#sed -n '1~2p' test2 20170808 08 [root@centos001 ~]#sed -n '2~2p' test2 Linux centos6.8 testCopy the codePrint the next line of the matching string
[root@centos001 ~]#sed -n '/linux/{n; p}' test2 08 [root@centos001 ~]#awk '/linux/{getline; print}' test2 08Copy the codeLinux three Musketeers Grep Command Description Command name
grep
Command role
Text lookup or search tool
Detailed instructions
It can also be used with regular expressions to search text and print out matched lines. It can also be used to filter and search specific strings, which is very flexible
Commonly used parameters
-a # Do not ignore binary data -a # display the line that matches the template and the content after that line -b # Display the line that matches the template and the content before that line -b # Display the line that matches the template, -c # Count the columns that match the template. -C # Display the columns that match the template. -D # This parameter must be used when specifying that you are looking for a directory rather than a file. Otherwise grep returns the information and stops the action. -e # Specifies a string as a template style to find the contents of the file. -e # Uses the template style as an extended ordinary representation, meaning that using the extended regular expression -f # specifies a template file whose contents have one or more template styles, Ask grep to find the contents of the file that match the template conditions, formatted as the template style for each column. -f # Treat the template style as a list of fixed strings. -g # Treat the template style as a normal representation to use -h # Before displaying the column that matches the template style, -h # Before displaying the column that matches the template style, -l # List the names of files that match the specified template. -l # List the names of files that do not match the specified template. -n # -q # Show no information -r /-r # This parameter has the same effect as specifying the -d recurse parameter -s # show no error information -v # reverse lookup -v # show version information -w # show only full-word columns -x # show only full-column columns -y -o # Prints only a portion of the regular expression matched in the file. ^ # Matches lines starting with XX. $# matches lines ending with XXCopy the code1. Grep “file” file_1 file_2 file_3
2. Run the -v command to grep -v “file” file_name
Grep “file” file_name –color=auto
4, use regular expression -e option:
grep -E "[1-9]+"
egrep "[1-9]+"
Copy the code-o: output only the matched part of the file.
echo this is a test line. | grep -o -E "[a-z]+."
line.
echo this is a test line. | egrep -o "[a-z]+."
line.
Copy the codeThe number of lines in the statistics file or text containing the matching string -c option:
grep -c "text" file_name
2
Copy the code7, the output matching string lines include the -n option: grep “text” -n file_name or cat file_name | grep “text” – n
Grep “text” -n file_1 file_2
Grep -l “text” file1 file2 file3…
10. Grep Recursive search File Run the grep “text”. -r -n command to recursively search for text in multi-level directories
Ignore character case in matching styles:
echo "hello world" | grep -i "HELLO"
hello
Copy the codeThe -e option specifies multiple matching styles:
echo this is a text line | grep -e "is" -e "line" -o
is
line
Copy the code13. You can also use the -f option to match multiple styles, writing out the characters to be matched line by line in the style file.
cat patfile
aaa
bbb
echo aaa bbb ccc ddd eee | grep -f patfile -o
Copy the codeInclude or exclude specified files in grep search results: only recursively search character “main()” in.php and.html files in directories.
grep "main()" . -r --include *.{php,html}
Copy the code15. Exclude all README files from search results
-r –exclude “README” 16. Exclude all files in the Filelist file from the search result
grep "main()" . -r --exclude-from filelist
Copy the codeThis section describes the Grep command in Linux
Grep '^J' testfile # Display lines starting with J grep '70$' testfile # Display lines ending with 70 grep -v "834" testfile # show all not including 834 rows grep ': 12 / testfile # show: 12 / line grep' : 498 - testfile # shows: of the 498 - line grep '[a-z] [a-z] {4} : [[: space:]] [a-z]' Testfile # displays a line like this, One uppercase letter + four lowercase letters + space + one uppercase letter grep '[a-z]{1,}[[:space:]][Kk]' testfile # grep -n '[0-9]{6,}$' testfile # And print the line number grep -i "Lincoln" testfile # To display Lincoln's line, case insensitiveCopy the codeSed Example
Sed '1,3d' testfile # sed '1 ',3d' testfile # sed 'n '5,10p' testfile # print out 5-10 lines of the file Sed '/Lane/d' testfile # delete Lane rows sed-ne '/[1-9]{5}:1[12]/p' testfile # print lines with 5 digits +:11/12 sed 's/^Fred/***&/' testfile *** sed -e 's/.*Jose.*/ Jose HAS RETIRE/g' testfile # Replace Jose HAS RETIRE with Jose HAS RETIRE -n '/^Popeye/p' Testfile | sed 's / [0-9] {1} / [0-9] {1} / [0-9] {1} / 11/14/46 /' # will begin with Popeye line printing, Then replace "number/number/number" with 11/14/46 ##pattern{n} for n times. ##pattern{n,} Matches the pattern at least n times. ## sed '/^$/d' testfile # sed 's/.$//g' # End line sed 's / ^ [] [] * / / g' # sed to delete rows first Spaces' s / /. / [] [] [] * / g '# delete a period followed by two or more Spaces, use a space instead of sed' s / ^. / / g '# sed to delete the first characters' s/COL/(... Sed 's/^////g' # sed -n '3,/245700/'p testfile # sed -n '2,26! 'p testfile # Prints lines of files (except 2-26)Copy the codeThis section describes Awk instances
Awk -f: '/^Dan/{print $2}' datafile # '/^[CE]/{print $1}' datafile # print awk -- F: '{if(length($1) == 4) print $1}' awk -- F: Awk -f: '/^Vinh/{print "a"$5}' 2.txt Awk -f: '($5 == 68900) {print $1}' 2. TXT: awk -f: '($5 == 68900) {print $1}' 2. '{if(length($1) == 11) print $1}' 2. TXT awk -f: '$1~/Tommy Savage/ {print $5}' 2. TXT awk -f: '$1~/Tommy Savage/ {print $5}' '($1 = = "Tommy" Savage ") {print $5}' 2. TXT # print with: space and the first listed as Tommy the fifth column content of Savage ll | awk 'BEGIN {size = 0; } {size=size+$5; } END{print "[END]size is ",size}' awk 'BEGIN{size=0; } {size=size+$5; } END{print "[END]size is ",size/1024/1024,"M"}' awk 'BEGIN{a=10; a+=10; Print a} '# 20 a + 10 is equivalent to a = a + 10 echo | awk' BEGIN {a = "100 testaaa"} a ~ / test / {print "ok"} '# if there is a regular matching test characters, Ok awk 'BEGIN{a="b"; print a=="b"?" ok":"err"}' ok awk 'BEGIN{a="b"; print a=="c"?" Ok ":"err"}' err # ternary operator? Awk '/root/{print $0}' passwd # match all lines containing root awk -f: '$5~/root/{print $0}' passwd Line matching the fifth field is the root of the ifconfig eth0 | awk 'BEGIN {FS = "[[: space:]] +} NR = = {print $4}' 2 # print IP address awk '{print toupper ($0)}' test. TXT #toupper is a built-in awK function that converts lowercase letters to uppercaseCopy the codeLinux three swordsmen Awk, Sed, Grep command details, here is the introduction to the end.
