Due to the limitation of network transmission, we often need to cut large files in Linux system. In this way, a large file is cut into multiple small files, which can be transferred and merged after the transfer.

File split – split it is convenient to split large files using the split command on Linux

Command syntax

-a: # Specifies the length of the suffix of the output file name (default: 2: AA, AB...). -d: # Specifies the output filename suffix with a number instead of -l: # Line splitting mode (specifies the number of lines to be cut into a small file; The default number of lines is 1000) -b: # Binary split mode (supported in k/m) -c: # file size segmentation model (as far as possible to maintain the integrity of each row) cutting the split [-] a [-d] [-l < number > lines] [- byte < > b] [-c < > byte] [to cut the file] [output filename]Copy the code

Using the instance

SQL /data/users_ # split file $split -l 300000 users. SQL /data/users_ # split $split -l 300000 users. SQL /data/users_ # split $split -l 300000 users -d -b 100m users.sql /data/users_Copy the code

Help information

$split --help Usage: split [OPTION]... [FILE [PREFIX]] Output pieces of FILE to PREFIXaa, PREFIXab, ... ; default size is 1000 lines, and default PREFIX is 'x'. With no FILE, or when FILE is -, read standard input. Mandatory arguments to long options are mandatory for short options too. -a, --suffix-length=N Generate suffixes of length N (default 2) -- Additional -suffix= suffix append an Additional SUFFIX to file names -b, --bytes=SIZE Put SIZE bytes per output file The SIZE of bytes per output file -c, --line-bytes=SIZE put at most SIZE bytes of records per output file -d use numeric suffixes starting at 0, Numeric suffixes[=FROM] same as-d, but allow setting the start value-e, --elide-empty-files do not generate empty output files with '-n' --filter=COMMAND write to shell COMMAND; File name is $file -l, --lines=NUMBER put NUMBER lines/records per output file -n, --number=CHUNKS generate CHUNKS output files; Separator = separator instead of newline as the record separator; '\0' (zero) specifies the NUL character-u, --unbuffered immediately Copy input to output with '-n r/... --verbose print a diagnostic just before each Output file is opened --help display this help and exit --version output version information and exit The SIZE argument is an integer and optional unit (example: 10K is 10*1024). Units are K,M,G,T,P,E,Z,Y (powers of 1024) or KB,MB,... (powers of 1000). CHUNKS may be: N split into N files based on size of input K/N output Kth of N to stdout l/N split into N files without splitting lines/records l/K/N output Kth of N to stdout without splitting lines/records r/N like 'l' but use round robin distribution r/K/N likewise but only output Kth of N to stdout GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Full documentation at: <http://www.gnu.org/software/coreutils/split> or available locally via: info '(coreutils) split invocation'Copy the code

File merge – cat On Linux it is also convenient to merge multiple small files using the cat command

Command syntax

- n: # show line Numbers - e: # $character as the end of each line - t: # show the TAB character (^ I) cat [-n] [e] [-t] [output filename]Copy the code

Example ‘# merge file $cat /data/users_* > users.sql’ help information

$cat --h Usage: cat [OPTION]... [FILE]... Concatenate FILE(s) to standard output. With no FILE, or when FILE is -, read standard input. -A, --show-all equivalent to -vET -b, --number-nonblank number nonempty output lines, overrides -n -e equivalent to -vE -E, --show-ends display $ at end of each line -n, --number number all output lines -s, --squeeze-blank suppress repeated empty output lines -t equivalent to -vT -T, --show-tabs display TAB characters as ^I -u (ignored) -v, --show-nonprinting use ^ and M- notation, except for LFD and TAB --help display this help and exit --version output version information and exit Examples: cat f - g Output f's contents, then standard input, then g's contents. cat Copy standard input to standard output. GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Full documentation at: <http://www.gnu.org/software/coreutils/cat> or available locally via: info '(coreutils) cat invocation'Copy the code