This article is participating in node.js advanced technology essay, click to see details.
Node.js works as a single process in the CPU with single-threaded and non-blocking IO. With this single-threaded and single-process approach, there is a limit to what can be done, even if the server’s performance is strong and resources are used efficiently. Node.js is designed to build distributed applications with multiple nodes, hence the name.
Workload is one of the main reasons we start scaling applications, including availability and fault tolerance. Extension can be done in a number of ways. The simplest way is to clone multiple node.js instances, which can be cloned using the Clustermoudle provided by Node.js.
Before we start processing requests using resource-intensive Node.js servers, let’s take a look at how the Cluster module works.
How does a Cluster work?
There are two types of processes running in Cluster mode: the Master process and the Worker process. The Master process is responsible for receiving all requests and deciding which Worker process should handle them. Worker processes can be thought of as normal Node.js singleton processing requests.
How does the Master process allocate requests?
-
The first method is round-robin: the Master process listens on the port and distributes the newly received requests to the Worker process in a circular allocation mode. Of course, the Master process has some built-in processing to avoid Worker overload. This is the way it is handled on most platforms except Windows.
-
The second way is to use socket: the Master process creates the socket and passes the request to the corresponding Worker process for processing.
In theory, the second approach should provide the best performance. However, due to the uncertainty of allocation scheduling of operating system itself, requests are often distributed unevenly. For example, there are eight Node.js instances at the same time, but 70% of requests are allocated to two of the process instances.
Create a simple Node.js service
Let’s create a simple Node.js service to handle requests:
/*** server.js ***/
const http = require(" HTTP ");// Get the process ID
const processId = process.pid;
// Create the HTTP service and process the request
const server = http.createServer((req, res) = > {
// Simulate CPU operation
for (let index = 0; index < 1e7; index++);
res.end(`Process handled by pid: ${processId}`);
});
// Listen on port 8080
server.listen(8080.() = > {
console.log(`Server Started in process ${processId}`);
});
Copy the codeThe response is as follows:
Load test the Node.js service
We will use the ApacheBench work for testing, or you can choose a benchmark testing tool, such as Autocannon, to your liking.
We will load test our Node.js service with 500 concurrent requests within 10s.
➜ test_app ab -c 500 -t 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Finished 3502 requests
Server Software:
Server Hostname: localhost
Server Port: 8080
Document Path: /
Document Length: 29 bytes
Concurrency Level: 500
Time taken for tests: 11.342 seconds
Complete requests: 3502
Failed requests: 0
Total transferred: 416104 bytes
HTML transferred: 116029 bytes
Requests per second: 308.76 [#/sec] (mean)
Time per request: 1619.385 [ms] (mean)
Time per request: 3.239 [ms] (mean.across all concurrent requests)
Transfer rate: 35.83 [Kbytes/sec] received
Connection Times (ms)
min mean[+ / -sd] median max
Connect: 0 6 3.7 5 17
Processing: 21 1411 193.9 1412 2750
Waiting: 4 742 395.9 746 1424
Total: 21 1417 192.9 1420 2750
Percentage of the requests served within a certain time (ms)
50% 1420
66% 1422
75% 1438
80% 1438
90% 1624
95% 1624
98% 1624
99% 1625
100% 2750 (longest request)
Copy the codeFrom the above test, 3,502 Requests were processed, with a throughput rate of 308req/s and an average Time per request of 1,619 ms.
The results of this load test were good and should be sufficient to support most small to medium sized site applications. But we are not using CPU resources as much as we can, and most of the available CPU resources are unused.
Using Cluster Mode
Now let’s use Cluster mode to upgrade our Node.js service
/** cluster.js **/
const os = require(" OS ");const cluster = require(" cluster ");if (cluster.isMaster) {
const number_of_cpus = os.cpus().length;
console.log(`Master ${process.pid} is running`);
console.log(`Forking Server for ${number_of_cpus} CPUs\n`);
// Create a woker process based on the CPU kernel format
for (let index = 0; index < number_of_cpus; index++) {
cluster.fork();
}
// One worker process exits, and logs are printedCluster. On (" exit ",(worker, code, signal) = > {
console.log(`\nWorker ${worker.process.pid} died\n`);
});
} else {
require(". / server "); }Copy the codeMost cpus today have at least a dual-core processor. The processor of my personal computer is the 8th generation I7, with 8 cores, and the resources of the remaining 7 cores are idle.
When we run cluster.js, the server responds as follows:
If you run multiple Node.js instances in Cluster mode, this will take full advantage of CPU/ server power. The request is assigned by the Master process to one of the eight Worker processes for processing.
Load test node.js services in Cluster mode
➜ test_app ab -c 500 -t 10 http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Finished 20374 requests
Server Software:
Server Hostname: localhost
Server Port: 8080
Document Path: /
Document Length: 29 bytes
Concurrency Level: 500
Time taken for tests: 10.000 seconds
Complete requests: 20374
Failed requests: 0
Total transferred: 2118896 bytes
HTML transferred: 590846 bytes
Requests per second: 2037.39 [#/sec] (mean)
Time per request: 245.412 [ms] (mean)
Time per request: 0.491 [ms] (mean.across all concurrent requests)
Transfer rate: 206.92 [Kbytes/sec] received
Connection Times (ms)
min mean[+ / -sd] median max
Connect: 0 0 1.3 0 12
Processing: 6 242 15.6 241 369
Waiting: 6 242 15.5 241 368
Total: 18 242 15.5 241 371
Percentage of the requests served within a certain time (ms)
50% 241
66% 244
75% 246
80% 247
90% 251
95% 259
98% 283
99% 290
100% 371 (longest request)
Copy the codeOur Requests per second increased from 308per/s to 2,037 per/s, a nearly six-fold increase. The average Time per request dropped from 1619ms to 245ms. We received a total of 3,502 requests before, and now we have 20,374 requests (a 5.8-fold increase).
As you can see, we didn’t change the sever. Js code, just adding a few lines of cluster.js10 resulted in a huge performance improvement.
Availability and zero down time
When we have a server instance and it crashes, the instance must be restarted, which can cause downtime. Even if the process is automated with tools such as PM2, there are delays and no single request can be processed during that time.
Mock service crash
/*** server.js ***/
const http = require(" HTTP ");// Get the node.js process ID
const processId = process.pid;
// Create an HTTP service
const server = http.createServer((req, res) = > {
// Simulate CPU operation
for (let index = 0; index < 1e7; index++);
res.end(`Process handled by pid: ${processId}`);
});
// Listen on port 80
server.listen(8080.() = > {
console.log(`Server Started in process ${processId}`);
});
// ⚠️ Note: The code below is for process crash testing only, do not use in production environments
setTimeout(() = > {
process.exit(1);
}, Math.random() * 10000);
Copy the codeIf we add the last three lines of code to sever. Js, we can restart the server and see that all processes are crashing. Since no Worker process is available at the end, the main process still exists and the entire service may crash.
➜ test_app node cluster.js
Master 63104 is running
Forking Server for 8 CPUs
Server Started in process 63111
Server Started in process 63118
Server Started in process 63112
Server Started in process 63130
Server Started in process 63119
Server Started in process 63137
Server Started in process 63142
Server Started in process 63146
Worker 63142 died
Worker 63112 died
Worker 63111 died
Worker 63146 died
Worker 63119 died
Worker 63130 died
Worker 63118 died
Worker 63137Died ➜ test_appCopy the codeZero down time for processing
When we have multiple server instances, we can easily improve server availability.
Let’s open our cluster.js file and add some code:
/** cluster.js **/
const os = require(" OS ");const cluster = require(" cluster ");if (cluster.isMaster) {
const number_of_cpus = os.cpus().length;
console.log(`Master ${process.pid} is running`);
console.log(`Forking Server for ${number_of_cpus} CPUs\n`);
// Create a woker process based on the CPU kernel format
for (let index = 0; index < number_of_cpus; index++) { cluster.fork(); } cluster. On (" exit ",(worker, code, signal) = > {
/** * Check whether the worker process exits and is not killed by the Master process. The fork process * /
if(code ! = =0 && !worker.exitedAfterDisconnect) {
console.log(`Worker ${worker.process.pid} died`); cluster.fork(); }}); }else {
require(". / server "); }Copy the codeLoad test the node.js service that has been restarted
Let’s retest our cluster.js for load testing, and note that we have modified server.js to crash irregularly. The test results are as follows:
➜ test_app ab -c 500 -t 10 -r http://localhost:8080/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking localhost (be patient)
Completed 5000 requests
Completed 10000 requests
Completed 15000 requests
Completed 20000 requests
Finished 20200 requests
Server Software:
Server Hostname: localhost
Server Port: 8080
Document Path: /
Document Length: 29 bytes
Concurrency Level: 500
Time taken for tests: 10.000 seconds
Complete requests: 20200
Failed requests: 12
(Connect: 0, Receive: 4, Length: 4, Exceptions: 4)
Total transferred: 2100488 bytes
HTML transferred: 585713 bytes
Requests per second: 2019.91 [#/sec] (mean)
Time per request: 247.536 [ms] (mean)
Time per request: 0.495 [ms] (mean.across all concurrent requests)
Transfer rate: 205.12 [Kbytes/sec] received
Connection Times (ms)
min mean[+ / -sd] median max
Connect: 0 0 1.5 0 13
Processing: 13 243 15.7 241 364
Waiting: 0 243 16.0 241 363
Total: 22 243 15.5 241 370
Percentage of the requests served within a certain time (ms)
50% 241
66% 245
75% 248
80% 250
90% 258
95% 265
98% 273
99% 287
100% 370 (longest request)
➜ test_app
Copy the codeIn this load test, you can see that our throughput is 2,019 per/s. Of the total 20,200 requests, only 12 failed, giving the server an uptime of 99.941%.
Make NodeJs Handle 5X Request with 99.9% uptime Adding 10 lines of Code by Biplap Bhattarai