The original link
About the author
Published June 21, 2018
This guide is intended for front-end developers who are familiar with Javascript but not yet familiar with Node.js. I’m not focusing on the language itself here — Node.js uses the V8 engine, so it’s the same interpreter as Google Chrome, as you probably already know (but it also runs on different VMS, see
node-chakracore)
directory
- The Node version
- Don’t need the Babel
- Callback style
- Event loop
- Event emitter
- flow
- Module system
- The environment variable
- The integrated use of
- conclusion
We work with Node.js all the time, even if you’re a front-end developer — NPM scripts, Webpack configurations, gulp tasks, application packaging or running tests, etc. Even if you don’t really need to understand these tasks in depth, you can sometimes get confused and code in very strange ways because some of the core concepts of Node.js are missing. Once you get familiar with Node.js, you can also automate some of the things you would otherwise need to do manually, allowing you to look at server-side code more confidently and write more complex scripts.
The Node version
The biggest difference between Node.js and client code is that you can decide based on the environment you’re running in, and you know exactly what features it supports — you can choose which version to use based on your requirements and the servers available.
Node.js has a public release schedule that tells us that odd-numbered versions are not supported long term. The current VERSION of LTS (Long-term Support) will be actively developed until April 2019 and then maintained through updates to critical code by December 31, 2019. New versions of Node.js are being actively developed, bringing many new features as well as security and performance improvements. This may be a good reason to use the current active version. However, no one is really forcing you, and if you don’t want to do this, you can just use the old version and wait until you feel the time is right to update.
Node.js is widely used in modern front-end toolchains – it’s hard to imagine a modern project without using Node tools for any processing. As a result, you’re probably already familiar with NVM (Node Version Manager), which allows you to install several node versions at the same time, choosing the right version for each project. The reason for using this tool is that different projects often use different versions of Node, and you don’t want to keep them in sync forever; you just want to preserve the environment in which you write and test them. There are many such tools available in other languages, such as Virtualenv for Python, Rbenv for Ruby, and so on.
Don’t need the Babel
Since you are free to choose any Node.js version, you will most likely use the LTS version. This version, as of 8.11.3 at the time of writing, supports almost all ECMAScript 2015 specifications except tail-recursion.
This means we don’t need Babel unless you run into a very old version of Node.js, need to convert JSX, or need some other cutting-edge converter. In practice, Babel isn’t that important, so you can run the same code as you write, without any compiler — a client genius we’ve forgotten.
We also don’t need Webpack or Browserify, so we don’t have tools to reload our code — if you’re developing something like a Web server, you can use Nodemon to reload your application after the file changes.
And since we don’t ship the code anywhere, there’s no need to shrink it — save a step: you just use the code exactly as it is, which is amazing!
Callback style
Previously, asynchronous functions in Node.js accepted callbacks with signatures (ERR, data), where the first argument represented an error message – if it was null, it was all correct, otherwise you had to deal with errors. These handlers are called when the operation is done and we get the response. For example, let’s read a file:
const fs = require('fs');
fs.readFile('myFile.js', (err, file) => {
if (err) {
console.error('There was an error reading file :(');
// process is a global object in Node
// https://nodejs.org/api/process.html#process_process_exit_code
process.exit(1);
}
// do something with file content
});
Copy the codeAs we quickly discovered, this style was difficult to write readable and maintainable code, and even made for callback hell. Later, a new native asynchronous processing method, the Promise, was introduced. It is standardized in ECMAScript 2015 (it is a global object for browsers and Node.js runtimes). Recently async/await has been standardized in ECMAScript 2017. Node.js 7.6+ supports this specification, so you can use it in the LTS version.
With Promise, we avoided “callback hell.” However, the problem we have now is that the old code and many of the built-in modules still use callbacks. It’s not too hard to convert them to promises — to illustrate, we’ll convert fs.readfile to Promise:
const fs = require('fs'); function readFile(... arguments) { return new Promise((resolve, reject) => { fs.readFile(... arguments, (err, data) => { if (err) { reject(err); } else { resolve(data); }}); }); }Copy the codeThis pattern can be easily extended to any function, and the built-in utils module has a special function – utils.promisify. Examples from official documentation:
const util = require('util');
const fs = require('fs');
const stat = util.promisify(fs.stat);
stat('.').then((stats) => {
// Do something with stats
}).catch((error) => {
// Handle the error.
});
Copy the codeThe Node.js core team understood that we needed to migrate from the old style, and they tried to introduce a version of the promisified file system module built in – there is already the Promisified file system module, although it is still experimental as of this writing.
You’ll still encounter a lot of old-school, call-back Node.js code, wrapped in utils.promisify for consistency.
Event loop
The event loop is almost the same as in the browser environment, with some extensions. However, since this topic is a bit more advanced, I’ll cover it in full, not just the differences (I’ll highlight this section so you know what’s unique to Node.js).
Event loops in Node.js
JavaScript is built with asynchronous behavior in mind, so we usually don’t do everything at once. In the following ways, events are not executed in direct order:
microtasks
For example, deal with Promises immediately, such as promise.resolve. This means that the code will be executed in the same event loop, but only after all synchronized code has been executed.
process.nextTick
This is a Node.js specific method that does not exist in any browser (and process objects). It behaves like a microtask, but with priority. This means that it will execute immediately after all synchronized code, even if other microtasks have been introduced before — which is dangerous and can lead to an infinite loop. Nomenclature is incorrect because it is executed in the same event loop, not in its next tick. However, it may remain the same for compatibility reasons.
setImmediate
While it does exist in some browsers, it does not behave consistently across all browsers, so you need to be very careful when using it in browsers. It is similar to the setTimeout (0) code, but sometimes takes precedence over it. The naming here isn’t the best either – we’re talking about the next iteration of the event loop, and it’s not really immidiate.
setTimeout/setInterval
Timers are represented identically in Nodes and browsers. One important thing about timers is that the delay we provide does not mean that the callback will be executed after this time. What it really means is that once the main thread has completed all operations (including microtasks) and there are no other timers of higher priority, Node.js will execute the callback after that time.
Let’s look at this example:
I’ll show you the correct output after the script executes below, but if you’d like, try doing it yourself (as a “JavaScript interpreter”) :
const fs = require('fs'); console.log('beginning of the program'); const promise = new Promise(resolve => { // function, passed to the Promise constructor // is executed synchronously! console.log('I am in the promise function! '); resolve('resolved message'); }); promise.then(() => { console.log('I am in the first resolved promise'); }).then(() => { console.log('I am in the second resolved promise'); }); process.nextTick(() => { console.log('I am in the process next tick now'); }); fs.readFile('index.html', () => { console.log('=================='); setTimeout(() => { console.log('I am in the callback from setTimeout with 0ms delay'); }, 0); setImmediate(() => { console.log('I am from setImmediate callback'); }); }); setTimeout(() => { console.log('I am in the callback from setTimeout with 0ms delay'); }, 0); setImmediate(() => { console.log('I am from setImmediate callback'); });Copy the codeThe correct execution sequence is as follows:
node event-loop.js
beginning of the program
I am in the promise function!
I am in the process next tick now
I am in the first resolved promise
I am in the second resolved promise
I am in the callback from setTimeout with 0ms delay
I am from setImmediate callback
==================
I am from setImmediate callback
I am in the callback from setTimeout with 0ms delay
Copy the codeYou can get more information about event loops and process.nexttick in the official Node.js documentation.
Event emitter
Many of the core modules in Node.js send or receive different events. It has an implementation of EventEmitter, which is a publish-subscribe model. This is very similar to the browser DOM event, with slightly different syntax, and the best way to understand it is to implement it yourself:
class EventEmitter { constructor() { this.events = {}; } checkExistence(event) { if (! this.events[event]) { this.events[event] = []; } } once(event, cb) { this.checkExistence(event); const cbWithRemove = (... args) => { cb(... args); this.off(event, cbWithRemove); }; this.events[event].push(cbWithRemove); } on(event, cb) { this.checkExistence(event); this.events[event].push(cb); } off(event, cb) { this.checkExistence(event); this.events[event] = this.events[event].filter( registeredCallback => registeredCallback ! == cb ); } emit(event, ... args) { this.checkExistence(event); this.events[event].forEach(cb => cb(... args)); }}Copy the codeThe code above only shows the mode itself, not the exact functionality – please don’t use it in your code!
This is all the basic code we need! It allows you to subscribe to events, unsubscribe later, and distribute different events. For example, responder bodies, request bodies, and streams – they all actually extend or implement EventEmitter!
Because it is such a simple concept, it is used in many NPM packages. So if you want to use the same event emitters in your browser, you can always use them.
flow
“Streams is one of the best and most misunderstood concepts in Node.js.”
Dominic Tarr
Streams allows you to process data in chunks, rather than just complete operations such as reading files. To understand what they do, let’s look at a simple example: Suppose we want to return an arbitrary size request file to the user. Our code might look something like this:
function (req, res) { const filename = req.url.slice(1); fs.readFile(filename, (err, data) => { if (err) { res.statusCode = 500; res.end('Something went wrong'); } else { res.end(data); }}); }Copy the codeThis code works, especially on a locally developed machine, but it can also fail – do you see the problem? If the file is too big, we have problems reading the file, we put everything into memory, if there is not enough memory space, this will not work properly. If we have a lot of concurrent requests, this code won’t work either – we have to keep the data objects in memory until we’ve sent everything.
However, we don’t need the file at all – we just need to return it from the file system, we don’t view the contents ourselves, so we can read a part of it and immediately return it to the client to free up our memory, repeating the process until we’re done sending the entire file. This is a quick introduction to Streams – we have a mechanism for receiving data in chunks, and we decide what to do with it. For example, we can also do this:
function (req, res) {
const filename = req.url.slice(1);
const filestream = fs.createReadStream(filename, { encoding: 'utf-8' });
let result = '';
filestream.on('data', chunk => {
result += chunk;
});
filestream.on('end', () => {
res.end(result);
});
// if file does not exist, error callback will be called
filestream.on('error', () => {
res.statusCode = 500;
res.end('Something went wrong');
});
}
Copy the codeHere we create a stream to read the file — this stream executes the EventEmitter class, and on the Data event we receive the next block, and on the End event we get a signal that the stream has ended, and then we read the entire file. This implementation is the same as the previous one – we wait for the entire file to be read and then return it in response. Also, it has the same problem: we keep the entire file in memory and then send it back. We can solve this problem if we know that the response object itself implements a writable stream, we can write information to the stream without keeping it in memory:
Function (req, res) {const filename = req.uarl.slice (1); const filestream = fs.createReadStream(filename, { encoding: 'utf-8' }); filestream.on('data', chunk => { res.write(chunk); }); filestream.on('end', () => { res.end(); }); // if file does not exist, error callback will be called filestream.on('error', () => { res.statusCode = 500; res.end('Something went wrong'); }); }Copy the codeThe response body implements writable flow,
fs.createReadStreamCreate readable streams, as well as bidirectional and transform streams. The differences between them and how they work are beyond the scope of this tutorial, but it’s useful to know they exist.
We no longer need the result variable, just write the read block to the response body immediately, without keeping it in memory! This means that we can even read large files without worrying about high concurrent requests – because the files are not stored in memory, it will not exceed the amount that memory can carry. But there is a problem. In our solution, we read the file from one stream (file system reads the file) and write it to another (network request), and the two things have different latencies. The emphasis here is really different. After a while, our response flow will be overwhelmed because it is much slower. This problem is a description of back pressure, and Node has a solution: Each readable stream has a pipe method that redirects all data to a given stream relative to its load: if it is busy, it will pause the original stream and resume it. Using this approach, we can simplify the code to:
function (req, res) {
const filename = req.url.slice(1);
const filestream = fs.createReadStream(filename, { encoding: 'utf-8' });
filestream.pipe(res);
// if file does not exist, error callback will be called
filestream.on('error', () => {
res.statusCode = 500;
res.end('Something went wrong');
});
}
Copy the codeStreams has changed several times over the course of Node’s history, so be careful when reading old manuals and check the official documentation often!
Module system
Node.js uses the CommonJS module. You’ve probably used – every time you use require to get a module in a WebPack configuration, you’re actually using the CommonJS module; It is also used every time module.exports is declared. However, you might also see something like exports.some = {} without module, and in this section we’ll look at exactly how that works.
First, we’ll talk about commonJS modules, which typically have.js extensions instead of.esm /.mjs files (ECMAScript modules) that allow you to use the import/export syntax. Also, it’s important to understand that WebPack and Browserify (as well as other packaging tools) use their own require functions, so don’t get confused – they won’t be covered here, just understand that they are different things (even if they behave very similarly).
So where do we actually get these “global” objects like Module, Requier, and exports? In fact, node.js adds it at runtime – instead of just executing the given javascript file, it actually includes it in a function that has all these variables:
function (exports, require, module, __filename, __dirname) {
// your module
}
Copy the codeYou can view the package by executing the following code snippet on the command line:
1node -e "console.log(require('module').wrapper)"
These are variables that are injected into the module and can be used as “global” variables, even if they are not truly global. I strongly encourage you to study them, especially module variables. You can call console.log (module) in a javascript file to compare the results printed from the main file with those printed from the required file.
Next, let’s take a look at the exports object – here’s a small example that shows some warnings related to it:
exports.name = 'our name';
// this works
exports = { name: 'our name' };
// this doesn't work
module.exports = { name: 'our name' };
// this works!
Copy the codeThe example above may leave you wondering why this is the case. The answer is the nature of the exports object – it is just an argument passed to a function, so in the case of specifying a new object to it, we just overwrite the variable and the old reference is gone. Although it’s not completely gone – module.exports is the same object – so they’re actually the same reference to a single object:
module.exports === exports;
// true
Copy the codeThe last part is require – which is a function that gets the module name and returns the module’s exports object. How exactly does it parse modules? There is a very simple rule:
- Retrieves core modules by name
- If the path starts with
. /或../At the beginning, it tries to parse the file - If you can’t find the file, try to find contains in it
index.jsDirectory of files - if
pathDon’t to. /或../At the beginning, please go tonode_modules /And check the folder/file:- In the folder where we ran the script
- One step up, until we get there
/ node_modules
There are other places, mostly for compatibility, where you can provide a lookup path by specifying the variable NODE_PATH, which may be useful. If you want to see the exact order in which node_modules are parsed, simply print the module object in the script and look for the Paths property. After I did this, I printed the following:
➜ TMP node test. Js Module {id: '. ', exports: {}, the parent: null, filename: '/ Users/seva zaikov had his typist/TMP/test. The js', the loaded: false, children: [], paths: [ '/Users/seva.zaikov/tmp/node_modules', '/Users/seva.zaikov/node_modules', '/Users/node_modules', '/node_modules' ] }Copy the codeAnother interesting thing about require is that after the first require call module is cached, it will not be executed again, We will only return the cached export object – this means you can do some logic and ensure it will only be executed once after the first require call (this is not quite true – you can remove the module ID from require.cache and reload the module if you need it again)
The environment variable
As described in twelve-factor applications, it is a good practice to store configuration in environment variables. You can set variables for a shell session:
export MY_VARIABLE="some variable value"
Node is a cross-platform engine, and ideally your application should run on any platform (for example, the development environment). You choose a production environment to run your code in, usually some Linux distribution. My examples cover MacOS/Linux only, not Windows. The syntax for environment variables in Windows is different, you can use things like cross-env, but you should keep this in mind in other cases as well.
You can add the following line of code to the bash/ZSH configuration file to set up in any new terminal session. However, you usually only provide specific variables for these instances when the application is running:
APP_DB_URI="....." SECRET_KEY="secret key value" node server.js
You can use the process.env object to access these variables in node.js applications:
const CONFIG = {
db: process.env.APP_DB_URI,
secret: process.env.SECRET_KEY
}
Copy the codeThe integrated use of
In the following example, we will create a simple HTTP service that will return a file named after the url/ string. If the file does Not exist, we return a 404 Not Found error, and if the user tries to cheat by using relative or nested paths, we return a 403 error. We’ve used some of these functions before, but haven’t really documented them – this time it will contain a lot of information:
// we require only built-in modules, so Node.js // does not traverse our node_modules folders // https://nodejs.org/api/http.html#http_http_createserver_options_requestlistener const { createServer } = require("http"); const fs = require("fs"); const url = require("url"); const path = require("path"); // we pass the folder name with files as an environment variable // so we can use a different folder locally const FOLDER_NAME = process.env.FOLDER_NAME; const PORT = process.env.PORT || 8080; const server = createServer((req, res) => { // req.url contains full url, with querystring // we ignored it before, but here we want to ensure // that we only get pathname, without querystring // https://nodejs.org/api/http.html#http_message_url const parsedURL = url.parse(req.url); // we don't need the first / symbol const pathname = parsedURL.pathname.slice(1); // in order to return a response, we have to call res.end() // https://nodejs.org/api/http.html#http_response_end_data_encoding_callback // // > The method, response.end(), MUST be called on each response. // if we don't call it, the connection won't close and a requester // will wait for it until the timeout // // by default, we return a response with [code 200](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes) // in case something went wrong, we are supposed to return // a correct status code, using the res.statusCode = ... property: // https://nodejs.org/api/http.html#http_response_statuscode if (pathname.startsWith(".")) { res.statusCode = 403; res.end("Relative paths are not allowed"); } else if (pathname.includes("/")) { res.statusCode = 403; res.end("Nested paths are not allowed"); } else { // https://nodejs.org/en/docs/guides/working-with-different-filesystems/ // in order to stay cross-platform, we can't just create a path on our own // we have to use the platform-specific separator as a delimiter // path.join() does exactly that for us: // https://nodejs.org/api/path.html#path_path_join_paths const filePath = path.join(__dirname, FOLDER_NAME, pathname); const fileStream = fs.createReadStream(filePath); fileStream.pipe(res); fileStream.on("error", e => { // we handle only non-existant files, but there are plenty // of possible error codes. you can get all common codes from the docs: // https://nodejs.org/api/errors.html#errors_common_system_errors if (e.code === "ENOENT") { res.statusCode = 404; res.end("This file does not exist."); } else { res.statusCode = 500; res.end("Internal server error"); }}); }}); server.listen(PORT, () => { console.log(application is listening at the port ${PORT}); });Copy the codeconclusion
In this tutorial, we cover a number of basic Node.js principles. We didn’t delve into specific apis, and we did miss something. However, this guide should be a good starting point to give you confidence in reading apis, editing existing code, or creating new scripts. You can now understand the errors, the interfaces used by the built-in modules, and what you can expect from typical Node.js objects and interfaces.
Next time, we’ll delve into the Web service using Node.js, the Node.js REPL, how to write CLI applications, and how to write small scripts using Node.js. You can subscribe to be notified of these new articles.
Related articles
Node.js REPL depth
June 5, 2018 » Don’t use acronyms
June 3, 2018 » Unit testing