A nodeJS stream is a stream.
What is flow? In everyday life, there is a flow of water. It is easy to think of a faucet, and the water flowing from the faucet is orderly and directed (flowing from high to low). Our flows in NodeJS are the same, they are ordered and directed. Flows in NodeJS are readable, or writable, or read-writable. And the stream inherits EventEmitter. Therefore, all streams are real columns of EventEmitter.
Stream is a basic concept in Node.js, similar to EventEmitter, which focuses on event-driven data processing in IO pipes. Like an array or map, a Stream is a collection of data, but it represents data that is not necessarily in memory.
Node.js streams are of the following types:
- Readable Stream: a Readable Stream, a producer of data, (e.g. Fs.createreadstream ())
- Writable Stream: a Writable Stream, a consumer of data, (e.g. Fs.createwritestream ())
- Duplex Stream: indicates a Duplex Stream that can be read or written
- A Transform Stream of data
Stream itself provides a set of interface specifications, many node.js built-in modules follow this specification, such as the famous FS module, which uses the Stream interface to read and write files. Similarly, each HTTP request is a readable stream, and the HTTP response is a writable stream.
The main benefit of streams in nodeJS is that large files are not read into memory all at once. Only one block of data from the data source is read at a time. This chunk of data can then be processed immediately in subsequent processes (once the data is processed it goes into the garbage collection mechanism). Instead of waiting for all the data.
To understand this, let’s look at a simple stream column:
1. Create a large file with the following code:
const fs = require('fs'); const file = fs.createWriteStream('./big.txt'); For (let I = 0; i <= 5000000; I++) {file.write(' I'm FFZ, I'm going to test a big file, do you see how big I am? '); } file.end();Copy the codeI created a new app.js file in my project file, and put the above code into app.js. You can see that after 5 million loops, 5 million data were written to big.txt, so it will generate a big.txt file in the file directory, as follows:
ReadFile Reads the file:
Let’s read the file using readFile (readFile is read into memory at once).
Let’s change the app.js code to the following:
const fs = require('fs'); const Koa = require('koa'); const app = new Koa(); app.use(async(ctx, next) => { const res = ctx.res; fs.readFile('./big.txt', (err, data) => { if (err) { throw err; } else { res.end(data); }})}); app.listen(3001, () => { console.log('listening on 3001'); });Copy the codeAfter we run Node app.js, we can check the memory usage of this code (16MB) as follows:
But when we run http://localhost:3001/, we find that the memory footprint (368MB) is as follows:
ReadFile will read the entire contents of the big.txt file into the Buffer format and then write it to the return object. This is very inefficient, and if the file is larger than 1G or 2G, the memory will be blocked. Or the server will simply crash.
This can be avoided by using the createReadStream method in Node. Let’s change the app.js code to something like this:
const fs = require('fs');
const Koa = require('koa');
const app = new Koa();
app.use(async(ctx, next) => {
const res = ctx.res;
const file = fs.createReadStream('./big.txt');
file.pipe(res);
});
app.listen(3001, () => {
console.log('listening on 3001');
});
Copy the codeThen we continue to look at memory usage as follows:
As you can see, our footprint is only 18.6 MB. In other words, createReadStream does not read large files into memory all at once. Only one block of data from the data source is read at a time. That’s the beauty of flow. Let’s see indecency separately.
Fs.createreadstream () readable stream
The basic usage method is as follows:
const fs = require('fs'); Const rs = fs.createreadstream ('./big.txt', {flags: 'r', // The file operates in the same way as readFile, which is readable by default. 'utF-8 ', // autoClose: true, // Whether to close the file descriptor for internal use in the operating system start: 0, // end: 5, // highWaterMark: 1 // Number of reads per time});Copy the codeFs.createreadstream has the following listening events: see nodejs.cn/api/stream. Here is a screenshot for a brief look, as follows:
With these listener methods, we can first look at a complete real column as follows:
const fs = require('fs'); Const file = fs.createreadstream ('./msg. TXT ', {flags: 'r', // The file operates in the same way as readFile, which is readable by default. 'utF-8 ', // autoClose: true, // Whether to close the file descriptor for internal use in the operating system start: 0, // end: 5, // highWaterMark: 1 // Number of reads per time}); File.on ('open', () => {console.log(' start reading file '); }); File. On (' data ', (data) = > {the console. The log (' read data: '); console.log(data); }); File.on ('end', () => {console.log(' all files have been read '); }); File.on ('close', () => {console.log(' file is closed '); }); Check the file. The 'error', (err) = > {the console. The log (' failed to read file '); });Copy the codeExecute as shown below:
As can be seen from the above figure, the file is opened first, the open event is executed, and then the data event is continuously triggered. When the data event is finished, the end event is triggered, and then the file is closed, and the close event is triggered.
Pause () method:
If we want to pause the data event while reading, we can use the ReadStream object’s pause method to pause the data event. The following code:
File. On (' data ', (data) = > {the console. The log (' read data: '); console.log(data); file.pause(); });Copy the codePause () was paused above, and if we want to re-read now, we need to use resume(), as shown below:
setTimeout(() => {
file.resume();
}, 100);
Copy the code
Writing in this way will not end, if you put the timer in on can solve the problem
For other events, such as readable events, you can see the official documentation (nodejs.cn/api/stream….). There will be no more analysis here.
Fs.createwritestream () writable stream
The following code is shown:
const fs = require('fs'); Const file = fs.createWritestream ('./1.txt', {flags: 'w', // The file operates the same way as writeFile, which is readable by default. 'utF-8 ', // autoClose: true, // whether to close the file descriptor used in the operating system start: 0, // highWaterMark: 1 // number of writes per time}); Let the f1 = file. Write (' 1 ', 'utf-8, () = > {the console. The log (' write successful 1111'); }); F1 = file. Write (' 2 ', 'utf-8, () = > {the console. The log (' write successful 2222'); }); F1 = file. Write (' 3 ', 'utf-8, () = > {the console. The log (' write successful 3333'); }); // mark the end of the file file.end(); / / handle events file. On (' finish ', () = > {the console. The log (' write complete '); }); file.on('error', (err) => { console.log(err); });Copy the codeA 1.txt file with 123 contents will be generated in the root directory of my project.
Please see the official website (nodejs.cn/api/fs.html…) for details.
Flow pipe (pipe).
We need to put the data that we read in the readable stream above into the writable stream and write it to the file. We can operate with the following code:
const fs = require('fs'); Hello world const MSG = fs.createreadstream ('./msg. TXT ', {highWaterMark: 5}); Const f1 = fs.createWritestream ('./1.txt', {encoding: 'utF-8 ', highWaterMark: 1}); // Write to 1.txt const f1 = fs.createWritestream ('./1.txt', {encoding: 'utf-8', highWaterMark: 1}); MSG. On ('data', (chunk) => {f1.write(chunk, 'utF-8 ', () => {console.log(' write succeeded '); }); });Copy the codeBut to implement the above mechanism, we can use the pipe mechanism, which provides a mechanism for an output stream to an input stream. Typically we use it to get data from one stream and pass it to another stream. As shown below:
With the above code, we can change to the following code:
const fs = require('fs'); Hello world const MSG = fs.createreadstream ('./msg. TXT ', {highWaterMark: 5}); Const f1 = fs.createWritestream ('./1.txt', {encoding: 'utF-8 ', highWaterMark: 1}); // Write to 1.txt const f1 = fs.createWritestream ('./1.txt', {encoding: 'utf-8', highWaterMark: 1}); const res = msg.pipe(f1); console.log(res);Copy the codeAfter printing res above, we can view the basic information in the command line as follows:
