Here to record the technical points encountered in the work, as well as some of their own thoughts on life, published on Wednesday or Friday.

cover

Streams in Node

Streams are one of the best features of Node. It helps us do a lot of things during development. Such as streaming through large amounts of data, or helping us separate applications.

What about Streams? There are basically the following points:

  • Processing large amounts of data
  • Using the pipe method
  • Transformation flows
  • Reading and writing flow
  • Decoupling the I/O

Processing unlimited amounts of data

Using the Data event, we can process a small file with very little memory.

We can actually handle an unlimited amount of data, for example, we can read bytes from a pseudo-random number generator.

const rs = fs.createReadStream('/dev/urandom')
let size = 0
rs.on('data'.(data) = > {
  size += data.length
  console.log('File size:', size)
})

Copy the code

The results are as follows:

Even though the program keeps running and the number of files is unlimited, it doesn’t crash.

Scalability is one of the characteristics of streams, and most programs written using streams can scale any input very well.

Flow Mode vs. Pull-base stream

A stream in a Node can be in flow mode or pull-based mode. When we add data to a flow, it goes into flow mode, which means that whenever there is data, the Data event is called.

In the sample code above, the readStream is not in flow mode when it is created, and we place it in flow mode through the data event.

If we want to stop it, we can call pause(), the pause method for the readable stream. If we want to restart it, we can call resume().

However, the flow mode can also be problematic because in some cases, even if the flow is paused, it can be flooded with incoming data, and the incoming flow may not accept pause() control.

Another way to extract data from a stream is to wait for a readable event and then keep calling the stream’s read method until null (the stream terminator entity) is returned. In this way, we can extract data from the stream and stop extracting it if necessary.

In other words, we don’t need to tell the flow to pause and then continue; We can start and stop it as needed.

Here’s an example:

const fs = require('fs')
const rs = fs.createReadStream(__filename)
rs.on('readable'.() = > {
  let data = rs.read()
  while(data ! = =null) {
    console.log('Data read:', data)
    data = rs.read()
  }
})

rs.on('end'.() = > {
  console.log('No more data')})Copy the code

In this example, we will use readable events to determine whether there is data, rather than calling data events directly.

Of course, a better way to extract data from a stream is to pipe our data into the stream we create. This way the problem of managing memory can be handled internally.

Understand the events of the stream

All streams inherit from the EventEmitter class with a different set of events. It is useful to know some of the events we often use when working with flows.

First, the data event. Triggered when new data is read from a readable stream. Data as the first argument to the event handler. It is important to note that, unlike other event handlers, attaching data listeners has side effects. When the first data listener is connected, our stream will be unpaused.

Second, the end event. Triggered when there is no data in the readable stream.

Third, the Finish event. Emitted when the writable stream ends and all pending writes have completed.

The close event. Usually emitted when the stream is completely closed, and the stream does not necessarily fire an event.

Fifth, the Puse event. Used to pause a readable stream. We can ignore this method for the most part.

Resume event. Restart a readable stream.

Pipe method

The pipe method is used to connect two streams together. We often use the shell scripts | pipe symbol to implement this function. In this way, we can connect multiple pipes together to process data more easily.

The Streams API also provides pipe methods. Each readable method has a PIPE method.

Let’s use an example to get a feel for it:

const zlib = require('zlib')
const map =  require('tar-map-stream')
const decompress = zlib.createGunzip()
const whoami = process.env.USER ||process.env.USERNAME

const convert = map((header) = >{
  header.uname = whoami
  header.mtime = new Date()
  header.name = header.name.replace('the node - v0.1.100'.'edon - v0.0.0')
  return header
})
const compress = zlib.createGzip()

process.stdin.pipe(decompress)
  .pipe(convert)
  .pipe(compress)
  .pipe(process.stdout)
Copy the code

Then execute:

The curl https://nodejs.org/dist/v0.1.100/node-v0.100.tar.gz | node read. Js > edon. Tar. GzCopy the code

The PIPE method binds the data listener to the source of the Streams stream, and then directs the received data to the target Streams.

When we string streams together through pipes, we are actually telling Node to parse the data with those streams.

Using a pipe pipe to process data is relatively safer than using the data method because it is free to handle backpressure, a concept we can understand as memory management.

For example, when a stream that generates data quickly might be overwhelmed by a slower write stream, a buffer stress strategy is used to prevent memory from filling up and the process from crashing. The ducting method provides this back pressure.

The code above, we through | pipe symbol will request data diversion to our index. The js script.

The process.stdin standard I/O is then passed down layer by layer through the PIPE method and finally stored in the edon.tar.gz file by redirection >.

The whole process is as follows:

curl—> process.stdin—> decompress —> content—> compress—> process.stdout—> endon.tar.gz

The last

  • “JavaScript Advanced Programming”
  • Reply “vue-router” or “router” to receive VueRouter source analysis document.
  • Reply “vuex” or “vuex” to receive a vuex source analysis document.

Full text, if you like.

Please “like” and “watching”, preferably add a “follow”, or share in moments.