Pipe related commands

The target

• cut
• sort
• wc
• uniq
• tee
• tr
• split
• awk
• sed
• grep

• The preparatory work

  zhangsan 68 99 26
  lisi 98 66 96
  wangwu 38 33 86
  zhaoliu 78 44 36
  maq 88 22 66
  zhouba 98 44 46
  Copy the code

• The above is the score sheet information

• Using commas, the first column is your name, the second column is your Chinese score, the third column is your math score, and the fourth column is your English score

The preparatory work

vim 1.txt

111:aaa:bbb:ccc
222:ddd:eee:fff
333:ggg:hhh
444:iii
Copy the code

1 cut

1.1 the target

• cutFrom command results according to conditionsextractCorresponding to the content

1.2 implementation

Step 1: extract the 5th character from the first 2 lines of the 1.txt file

The command	meaning
Cut action file	Intercepts content from the specified file

• parameter

parameter	English	meaning
-c	characters	Select content by character

head -2 1.txt | cut -c 5
Copy the code

Step 2: intercept the first two lines in 1.txt file with “:” to separate the content of the first 1,2 paragraphs

parameter	English	meaning
-d 'separator'	delimiter	Specifying delimiters
-f n1,n2	fields	After the segmentation shows the number of paragraphs of content, use.segmentation

Scope of control

The scope of	meaning
n	Only the NTH item is displayed
n-	Display the NTH entry all the way to the end of the line
n-m	Display entries from NTH to MTH (including m)

The head - 2 1. TXT | the cut - d ':' - 1, 2 fCopy the code

head -2 1.txt | cut -d ':' -f 1-2
Copy the code

1.3 summary

• throughCut Action target fileCorresponding content can be extracted according to conditions

• The preparatory work

  vim score.txt

  zhangsan 68 99 26
  lisi 98 66 96
  wangwu 38 33 86
  zhaoliu 78 44 36
  maq 88 22 66
  zhouba 98 44 46
  Copy the code

2 sort

2.1 the target

• Sort sorts the content of a text file in behavioral units.

2.2 the path

• Step 1: Sort the string
• Step 2: Reorder
• Step 3: Sort by value
• Step 4: Rank your grades

2.3 implementation

Step 1: Sort the string

[root@node01 tmp]# cat 2.txt
banana
apple
pear
orange
pear

[root@node01 tmp]# sort 2.txt 
apple
banana
orange
pear
pear
Copy the code

Step 2: Reorder

parameter	English	meaning
-u	unique	unrepetitive

It simply removes duplicate lines from the output line.

[root@node01 tmp]# sort -u 2.txt 
apple
banana
orange
pear
Copy the code

Step 3: Sort by value

parameter	English	meaning
-n	numeric-sort	Sort by numerical size
-r	reverse	Reverse the order

• To prepare data

  [root@node01 tmp]# cat 3.txt 
  1
  3
  5
  7
  11
  2
  4
  6
  10
  8
  9
  Copy the code

• Sort by string by default

  [root@node01 tmp]# sort 2.txt 
  1
  10
  11
  2
  3
  4
  5
  6
  7
  8
  9
  Copy the code

• ascending

  [root@node01 tmp]# sort -n 2.txt
  1
  2
  3
  4
  5
  6
  7
  8
  9
  10
  11
  Copy the code

• Reverse order

  [root@node01 tmp]# sort -n -r 2.txt
  11
  10
  9
  8
  7
  6
  5
  4
  3
  2
  1
  Copy the code

• merge

  [root@node01 tmp]# sort -nr 2.txt  
  11
  10
  9
  8
  7
  6
  5
  4
  3
  2
  1
  Copy the code

Step 4: Rank your grades

parameter	English	meaning
-t	field-separator	Specify field delimiters
-k	key	Sort by that column

‘ ‘

Sort -t ',' -k2nr score.txt 'sort -t ',' -k2nr score.txtCopy the code

3 wc command

3.1 the target

• Displays information about the number of bytes, words, and lines of the specified file.

3.2 the path

• Step 1: Display the number of bytes, words, and lines of the specified file.
• Step 2: Display only the number of lines of the file
• Step 3: count the number of lines words bytes of multiple files
• Step 4: Check/etcHow many subcontents are in the directory

3.3 implementation

Step 1: Show the designationfile Bytes, words, linesInformation.

The command	meaning
Wc filename	According to specifiedfile Bytes, words, linesinformation

[root@hadoop01 export]# cat 4.txt
111
222 bbb
333 aaa bbb 
444 aaa bbb ccc
555 aaa bbb ccc ddd
666 aaa bbb ccc ddd eee

[root@hadoop01 export]# wc 4.txt 
 6 21 85 4.txt
Copy the code

Step 2: Display only the number of lines of the file

parameter	English	meaning
-c	bytes	The number of bytes
-w	words	Number of words
-l	lines	The number of rows

[root@hadoop01 export]# wc 4.txt 
 6 21 85 3.txt
Copy the code

Step 3: count the number of lines words bytes of multiple files

[root@hadoop01 export]# wc 1.txt 2.txt 3.txt 4 4 52 1.txt 11 11 24 2.txt 6 21 85 3.txt 21 36 161 total amount [root@hadoop01 Export # wc *.txt 4 4 52 1. TXT 11 11 24 2. TXT 6 21 85 3Copy the code

Step 4: Check/etcHow many subcontents are in the directory

[root@hadoop01 export]# ls /etc | wc -w
240
Copy the code

3.4 summary

• throughWc fileYou canstatisticalOf the fileNumber of bytes, words, lines.

4 uniq

The uniq command is used to check and delete repeated lines in a text file. It is generally used together with the sort command.

4.1 the target

• The uniq command is used to check and delete repeated lines in a text file. It is generally used together with the sort command.

4.2 the path

• Step 1: Implement de-duplication effect
• The second step: not only to the weight, but also statistics of the number of occurrences

4.3 implementation

Step 1: Implement de-duplication effect

The command	English	meaning
Uniq [parameter] file	The only unique	Remove duplicate rows

[root@hadoop01 export]# cat 5.txt Zhang SAN 98 Li Si 100 Wang Wu 90 Zhao Liu 95 Ma Qi 70 Li Si 100 Wang Wu 90 Zhao Liu 95 MA Qi 70 # sort [root@hadoop01 Export] # cat 5. TXT | sort li si 100 70 ma qi and li si 100 ma fifty and fifty and 90 90 70 98 zhang zhao six 95 zhao 95 # to heavy [root @ hadoop01 export] # cat 5. TXT | sort | uniq seven 70 fifty and 90 zhang SAN li si 100 ma zhao 6 95 98Copy the code

The second step: not only to the weight, but also statistics of the number of occurrences

parameter	English	meaning
-c	count	Count the number of occurrences in each line

[root @ hadoop01 export] # cat 5. TXT | sort | uniq -c 2 li si seven 2 fifty and 90 1 70 100 2 ma zhang zhao 6 95 98Copy the code

4.4 summary

• throughUniq [options] fileCan be completed to repeat and count times

5 tee

5.1 the target

• throughteeCan command resultsThrough the pipeOutput to theMultiple filesIn the

5.2 implementation

The command	meaning
Command result | 1 file 2 3 tee file	throughteeCan command resultsThrough the pipeOutput to theMultiple filesIn the

• Place the results of de-statistics in a.tuck, B.tuck, and C.tuck files

  cat 5.txt | sort | uniq -c | tee a.txt b.txt c.txt
  Copy the code

5.3 summary

• throughteeCan command resultsThrough the pipeOutput to theMultiple filesIn the

6 tr

6.1 the target

• throughtrCommand is used toreplace 或 deleteCharacters in a file.

6.2 the path

• Step 1: Implement the substitution effect
• Step 2: Implement the delete effect
• Step 3: Complete the word count case

6.3 implementation

Step 1: Implement the substitution effect

The command	English	meaning
Command result is replaced | tr characters New characters	translate	Implement the substitution effect

# replace lowercase I with uppercase I # convert itheima to uppercase # Convert HELLO to lowercaseCopy the code

# will be lowercase I replace uppercase I echo "itheima | tr" 'I' 'I' # put itheima converted to uppercase echo "itheima" | tr '[a-z]' '[a-z] # convert HELLO to lowercase echo "HELLO" |tr '[A-Z]' '[a-z]'Copy the code

Step 2: Implement the delete effect

The command	English	meaning
Command result | tr -d deleted characters	delete	Deletes the specified character

• Requirement: Delete the number in abc1d4e5f

echo 'abc1d4e5f' | tr -d '[0-9]'
Copy the code

Step 3: Word count

The preparatory work

[root@hadoop01 export]# cat words.txt 
hello,world,hadoop
hive,sqoop,flume,hello
kitty,tom,jerry,world
hadoop
Copy the code

One replaces nu with a newline

2 the sorting

3 to heavy

4 count

[root @ # statistics the number of occurrences of each word hadoop01 export] # cat words. TXT | tr ', ' '\ n' | sort | uniq -c 1 flume hadoop 2 hello 1 hive jerry 1 kitty 1 sqoop 1 tom 2 worldCopy the code

• The preparatory work

  Cat -n /etc/*. Conf >> /export/v.txt add the command result to /export/v.txtCopy the code

7 split

7.1 the target

• throughsplitCommand willA large file Cut intoA number ofSmall file

7.2 the path

• Step 1: Cut the large file into smaller files by byte
• Step 2: Divide the large file into smaller files by the number of lines

7.3 implementation

Step 1: PressbyteCut large files into smaller ones

The command	English	meaning
Split-b 10k file	byte	Cut large files into pieces10KBThe small file

Step 2: PressThe number of rowsCut large files into smaller ones

The command	English	meaning
Split-l 1000 file	lines	Cut large files into piecesLine 1000The small file

7.4 summary

• Split a large file into smaller files with the split option filename command

• Preparation 1:

  vim score.txt

  zhangsan 68 99 26
  lisi 98 66 96
  wangwu 38 33 86
  zhaoliu 78 44 36
  maq 88 22 66
  zhouba 98 44 46
  Copy the code

8 awk

8.1 the target

• throughawkimplementationFuzzy query.Extract fields as needed, can also be carried outjudgeAnd simpleoperationAnd so on.

Step 8.2

• Step 1: Fuzzy query

• Step 2: Specify the separator and display the content according to the subscript

• Step 3: Specify a separator for the output field

• Step 4: Call the functions provided by AWK

• Step 5: Use the if statement to determine whether $4 passes

• Step 6: Sum the contents of a segment

8.3 implementation

Step 1: Search for Zhangsan and Lisi’s results

The command	meaning
awk ‘/zhangsan|lisi/’ score.txt	Fuzzy query

Step 2: Specify the separator and display the content according to the subscript

The command	meaning
awk -F ‘,’ ‘{print $1,$2, $3}’ 1.txt	TXT file and print the first paragraph, second paragraph, and third paragraph according to commas

options

options	English	meaning
-F ','	field-separator	useThe specified charactersegmentation
$+ digital		To obtainHow many piececontent
$0		To obtainThe current linecontent
NF	field	Represents the number of fields in the current row
$NF		Represents the last field
$(NF-1)		Represents the penultimate field
NR		The r is dealing with which row

Step 3: Specify a separator to display content by subscript

The command	meaning
awk -F ‘ ‘ ‘{OFS=”===”}{print $1,$2, $3}’ 1.txt	TXT file and print the first paragraph, second paragraph, and third paragraph according to commas

options

options	English	meaning
OFS = "character"	output field separator	Segment split string when output outward

Step 4: Call the functions provided by AWK

The command	meaning
awk -F ‘,’ ‘{print toupper($2)}’ 1.txt	TXT file and print the first paragraph, second paragraph, and third paragraph according to commas

Common functions are as follows:

The function name	meaning	role
toupper()	upper	The character is converted to uppercase
tolower()	lower	Characters are converted to lowercase
length()	length	Return character length

Step 5: Query the information of students who pass the if statement

The command	meaning
awk -F ‘,’ ‘{if( $4>60) print$1, $4 }’ score.txt	If you pass, you show it $1,$4
awk -F ‘,’ ‘{if( $4>60) print$1, $4. pass the exam. else print$1, $4, “fail “}’ score.txt	Show name, $4, pass or not

options

parameter	meaning
if( $0 ~ “aa”) print$0	If the line contains “AA “, the line is printed
if( $1 ~ “aa”) print$0	If the first paragraph of ** contains “aa”, print that line
if( $1 == “lisi”) print$0	ifThe first paragraph is equal to“Lisi “, just print this line

Step 6: Find the average score of the subject

The command	meaning
Awk ‘BEGIN{initialize operation}{each line is executed} END{operation at END}’ filename	BEGIN{this contains the statement before execution} {here are the statements to execute when processing each line} END {this is the statement to execute after all the lines are processed}

awk -F ',' 'BEGIN{}{total=total+$4}END{print total, NR, (total/NR)}' score.txt
Copy the code

• The preparatory work

  vim 1.txt

  aaa java root
  bbb hello
  ccc rt
  ddd root nologin
  eee rtt
  fff ROOT nologin
  ggg rttt
  Copy the code

9 sed

9.1 the target

• You can use SED to implement filtering and replacement.

9.2 the path

• Step 1: implement the query function
• Step 2: Implement the delete function
• Step 3: Implement the modification function
• Step 4: Implement replacement functionality
• Step 5: Operate on the original file
• Step 6: Comprehensive practice

9.3 implementation

Step 1: implement the query function

The command	meaning
Sed optional target file	To the target fileFilter query 或 replace

Optional parameters

optional	English	meaning
p	print	print
$		That’s the last line
-n		Only the processed results are displayed
-e	expression	Process according to the expression

• Exercise 1: List rows 1 to 5 in 1.txt

Sed-n-e '1,5p' 1.txtCopy the code

• Exercise 2 List all the data in 01.txt

sed -n -e '1,$p' 1.txt 
Copy the code

• Exercise 3 List all data in 01.txt and display line numbers

optional	meaning
=	Prints the current line number

Sed -n -e '1,$=' -e '1,$p' 1.txt sed -n -e '1,$=' -e '1,$p' 1.txtCopy the code

• Exercise 4: Find 01. TXT containing the root line

The answer:

sed -n -e '/root/p' 1.txt
Copy the code

• Exercise 5 List the contents in 01.txt that contain root, which is case insensitive, and display the line number

optional	English	meaning
I	ignore	Ignore case

The answer:

nl 1.txt | sed -n -e '/root/Ip'

nl 01.txt | grep -i root

cat -n 01.txt | grep -i root
Copy the code

• Exercise 6 Find the letters in 1. TXTrIt is followed by rows of multiple t’s and displays the line number

optional	English	meaning
-r	regexp-extended	Identification of a regular

The answer:

nl 01.txt | sed -nr -e '/r+t/p'
Copy the code

or

sed -nr -e '/r+t/p' -e '/r+t/=' 01.txt
Copy the code

Step 2: Implement the delete function

• Exercise 1 Delete the first three lines in 01.txt and display the line number

optional	English	meaning
d	delete	Delete specified content

The answer:

Nl 01. TXT | sed -e '1, 3 d'Copy the code

• Exercise 2 Retain the first 4 lines in 1.txt and display the line numbers

The answer:

Nl 01. TXT | sed -e '5, $d' nl 1. TXT | sed - n - e '1, 4 p'Copy the code

Step 3: Implement the modification function

• Exercise 1: Add aaAAA after the second line of 01.txt and display the line number

parameter	English	meaning
i	insert	The targetIn front of theInsert content
a	append	The targetbehindAdditional content

The answer:

nl 01.txt | sed -e '2a aaaaa'
Copy the code

• Exercise 2 add BBBBB before line 1 of 1.txt and display line number

The answer:

nl 01.txt | sed -e '1i bbbbb'
Copy the code

Step 4: Implement replacement functionality

• Exercise 1 Replace nologin in 1.txt with Huawei and display the line number

	English	meaning
s/oldString/newString/	replace	replace

The answer:

nl 1.txt | sed -e 's/nologin/huawei/'
Copy the code

• Exercise 2 replace lines 1,2 in 01.txt with aaa and display line numbers

options	English
2c New String	replace	Replace the selected line with a new string

The answer:

Nl passwd | sed -e '1, 2 c aaa'Copy the code

Step 5: Operate on the original file

• Exercise 1 Replace Nologin with Huawei in 01.txt

parameter	English	meaning
-i	in-place	Replace the original file

The answer:

sed -i -e 's/nologin/huawei/' 01.txt
Copy the code

• Exercise 2 is replaced with aaaaaa at lines 2 and 3 in 01.txt

The answer:

Sed - i-e '2,3c aaa' 01.txtCopy the code

Note: Before the operation, it is best to back up the data, put the operation wrong, the data cannot be recovered!

• Exercise 3 Delete the first two lines in 01.txt and delete the data in the original file

The answer:

Sed -i -e '1,2d' 01.txt nl passwd view the dataCopy the code

Step 6: Comprehensive practice

• Exercise 1 Obtaining an IP Address

The answer:

ifconfig eth0 | grep "inet addr" | sed -e 's/^.*inet addr://' | sed -e 's/Bcast:.*$//' 
Copy the code

• Exercise 2 extracts data from 1.txt, matches content containing root, and replaces nologin with itheima

The answer:

Nl 01. TXT | grep 'root' | sed -e 's/nologin itheima/' or nl 01. TXT | sed - n - e'/root/p | sed -e 's/nologin itheima/' Or nl 01. TXT | sed - n - e '/ root / {s/nologin/itheima/p}' # show only replace the contentCopy the code

• Exercise 3 extract the data from 1.txt, delete the first 2 lines, replace Nologin with itheima, and display the line number

The answer:

Nl 01. TXT | sed -e '1, 2 d | sed -e' s/nologin itheima/'Copy the code