Here to record the technical points encountered in the work, as well as some of their own thoughts on life, published on Wednesday or Friday.

cover

Streams in Node

Streams are one of the best features of Node. It helps us do a lot of things during development. Such as streaming through large amounts of data, or helping us separate applications.

What about Streams? There are basically the following points:

  • Processing large amounts of data
  • Using the pipe method
  • Transformation flows
  • Reading and writing flow
  • Decoupling the I/O

Keep the pipeline flow alive

Typically, when the original flow is piped to the target flow, the target flow terminates with the original flow.

Sometimes we want to do something extra after the original stream ends. To do this, let’s look at an example:

Name it anything, start.js

const net = require('net')
const fs = require('fs')

net
  .createServer((socket) = > {
    const content = fs.createReadStream(__filename)
    content.pipe(socket)
    content.on('end'.() = > {
      socket.end('\n========Footer=====\n')
    })
  })
  .listen(4000)

Copy the code

At this point we execute

node start.js
Copy the code

Then we send a request:

curl http://localhost:4000
Copy the code

You will find an error message

This is because when the Content readable stream ends, the socket stream connected to it ends. At this point, it is impossible to add content behind the socket.

We can modify our code as follows:

const net = require('net')
const fs = require('fs')

net
  .createServer((socket) = > {
    const content = fs.createReadStream(__filename)
    content.pipe(socket, { end: false })
    content.on('end'.() = > {
      socket.end('=========Footer========')
    })
  })
  .listen(4000)
Copy the code

This time we add a second parameter end: false to content.pipe. This tells the pipe method to avoid terminating the target stream at the end of the source stream, so that our code does not report errors.

Accordingly, we can receive the message returned:

Pipeline flow in production

The PIPE method is a very important feature of Streams streams. It allows us to combine multiple streams into one line of code.

As part of Node’s core, it is useful in cases where process runtime is not important. For example, we often use cli tools.

But the bad part is its error handling. If there is an error in one of the pipe flows, it often cancels the pipe connection and destroys the remaining flows. In this way, they do not leak resources, but may cause memory leaks.

Here’s another example:

const http = require('http')
const fs = require('fs')

const server = http.createServer((req, res) = > {
  fs.createReadStream('veryBigData.file').pipe(res)
})

server.listen(4000)
Copy the code

The service is piping data back to the user, so there is a high risk of memory overhead and file-related information leakage.

If the HTTP response is closed before the file is fully transferred to the user, some information about the file will definitely be leaked, and the file stream will also have some memory overhead, and the file stream will remain in memory because we did not close it.

So we need some error-handling mechanism to destroy the flow in our pipeline when appropriate.

This needs to mention another module -pump. Pump is designed to address these issues.

Pump (pump)

Let’s try another example:

const fs = require('fs')
const http = require('http')
const pump = require('pump')

const server = http.createServer((req, res) = > {
  const stream = fs.createReadStream('veryBigData.file')
  pump(stream, res, done)
})

function done(err) {
  if (err) {
    return console.log('File diversion error ----', err)
  }

  console.log('File diversion successful')
}

server.listen(4000)

Copy the code

At this point, run the file, make the request, and we’ll see that it’s reporting an error.

Each flow passed to the pump method is passed to the next flow. If the last pass was a function, pump will execute the method after all the streams have completed.

Pump has some additional methods built in. Such as closing, error handling, and ways to close another stream without affecting other streams.

If one of the streams closes, the other streams are destroyed and the callback function passed to pump is called.

Of course, we can handle these errors manually or destroy the stream when the data is closed, for example:


const server = http.createServer((req, res) = > {
  const stream = fs.createReadStream('veryBigData.file')
  stream.pipe(res)
  res.on('close'.() = >{
    stream.destory()
  })
})

server.listen(4000)
Copy the code

But in general, pump is more convenient and safer.

pumpify

When writing pipes, especially as a separate module. We might want to export these methods to external users. What to do?

As we said earlier, a pipeline consists of a series of transport streams. We write data to the first stream in the pipe, and then the data travels through it until we write to the last stream.

Let’s look at another example:

const {createGzip} = require('zlib')
const {crateCipher, createCipheriv} = require('crypto')
const pumpify = require('pumpify')
const base64 = require('base64-encode-stream')

function pipeline(){
  const stream1 = createGzip()
  const stream2 = createCipheriv()
  const stream3 = base64()

  return pumpify(stream1,stream2,stream3)
}
Copy the code

It’s kind of like Promisify.

The last

  • “JavaScript Advanced Programming”
  • Reply “vue-router” or “router” to receive VueRouter source analysis document.
  • Reply “vuex” or “vuex” to receive a vuex source analysis document.

Full text, if you like.

Please “like” and “watching”, preferably add a “follow”, or share in moments.