In Linux system, socket multiplexing can be realized by select, poll, epoll and other methods. These different methods have advantages, disadvantages and applicable scenarios, which are not the focus of this article. If you are interested, you can search for them yourself. However, in high concurrency scenarios, epoll performance is the highest. Nginx has heard that epoll is used in the underlying Nginx.
This article is mainly about how to use epoll, not the principle analysis. This article is not the most complete, nor is it the most in-depth, but it is definitely a can let ordinary people understand, after reading their own use of epoll write a tcpServer article. Just cut the crap and get to work
First of all, epoll is a system call provided by Linux, which means that epoll is not available on Windows, and the corresponding code is not compiled. If anyone knows how to compile in Windows, please comment.
tool
The development environment used in this article
- System:
Debian GNU/Linux 10 (buster) - The Linux kernel:
4.19.0-14 - GCC version:
8.3.0
Prepare knowledge
Epoll is a feature provided by the Linux kernel that provides system calls externally and provides functionality to users through three functions in C/C++
-
Epoll_create (int __size) creates an epoll. The _size parameter is useless after the linux2.6 kernel, but to be greater than 0, it is usually 1. The epoll function returns the file descriptor for the created epoll, or -1 on failure.
-
Epoll_ctl (nt __epfd, int __op, int __fd,struct epoll_event *__event) operates on the existing epoll,epfdepoll file descriptor; Op operation mode, including add, delete, modify and so on; _fd is the descriptor of the object to operate on, or the connection if you operate on a TCP connection. _event Indicates the response event of epoll. When an event occurs on a TCP connection managed by epoll, the _event object is used to transmit the event. Therefore, when adding a connection, wrap the connection as an epoll_event object
- Op type
- EPOLL_CTL_ADD Adds a descriptor
- EPOLL_CTL_DEL Deletes a descriptor
- EPOLL_CTL_MOD Modifies a descriptor
-
Epoll_wait (int __epfd, struct epoll_event *__events,int __maxEvents,int __timeout) This function is called when a response event occurs in a connection managed by epoll. Epfdepoll file descriptor; An array of concatenations that __events can operate on; __maxEvents The maximum number of events that can be processed; __timeout time (milliseconds). If -1, it will not return until an operable event occurs, because C++ does not support multiple return values. Like Go, it will return all events and the number (╥╯^╰╥).
Common types in events:
- EPOLLIN: indicates that the corresponding file descriptor is readable (the SOCKET is normally closed)
- EPOLLOUT: indicates that the corresponding file descriptor can be written
- EPOLLPRI: indicates that the corresponding file descriptor has urgent data to read (indicates that out-of-band data is coming)
- EPOLLERR: indicates an error in the corresponding file descriptor (default registration)
- EPOLLHUP: indicates that the corresponding file descriptor is hung up (default registration)
- EPOLLET: Set EPOLL to Edge Triggered mode, as opposed to Level Triggered
- EPOLLONESHOT: monitors only one event. If you want to continue monitoring the socket after this event, you need to add the socket to the EPOLL queue again
Isn’t it surprising that epoll has only three functions? I was surprised when I saw it for the first time that three functions could do something so complicated. But think about it, too, to simplify complex things, to reflect the strength of god
Two modes of epoll
Epoll events are Triggered (LT) and Edge Triggered (ET).
-
LT(level triggered) is the default working mode and supports both block and non-block sockets. In this way, the kernel tells you if a file descriptor is ready, and then you can IO the ready FD. If you do nothing, the kernel will continue to inform you, so this mode is less likely to cause errors.
-
Edge-triggered (ET) is high-speed and supports only no-block sockets. In this mode, the kernel tells you via epoll when a descriptor is never ready to go to ready. It then assumes that you know the file descriptor is ready and doesn’t send any more ready notifications for that file descriptor until the next time new data comes in.
Method to set the socket to non-blocking mode
int flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
Copy the codeRequired header file #include
Epoll principle
Here is a simple illustration of how epoll works
There is nothing too low-level involved here, because I have not studied too low-level, dare not talk about. To admit that you know what you know, and admit what you don’t know
Epoll can be thought of as a container provided by the operating system that manages someepoll_event(I drew it as a one-way linked list in the figure, but actually I used a red-black tree, because it is too troublesome to draw a tree), this event is added by us, and the event type to be responded to is set in the event. When epoll detects that a specific event has a corresponding event, it will passepoll_wait()Notice.
Simple epoll implementation
#include <iostream>// Console output
#include <sys/socket.h>/ / create a socket
#include <netinet/in.h>//socket addr
#include <sys/epoll.h>//epoll
#include <unistd.h>/ / close function
#include <fcntl.h>// Set non-blocking
using namespace std;
int main(a) {
const int EVENTS_SIZE = 20;
// Read the socket array
char buff[1024];
// Create a TCP socket
int socketFd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
// Set the address and port that the socket listens to
sockaddr_in sockAddr{};
sockAddr.sin_port = htons(8088);
sockAddr.sin_family = AF_INET;
sockAddr.sin_addr.s_addr = htons(INADDR_ANY);
// Bind the socket to the address
if (bind(socketFd, (sockaddr *) &sockAddr, sizeof(sockAddr)) == - 1) {
cout << "bind error" << endl;
return - 1;
}
// Start listening on the socket.
// The process can call ACCEPT to accept an incoming request
// The second argument, the length of the request queue
if (listen(socketFd, 10) = =- 1) {
cout << "listen error" << endl;
return - 1;
}
// create an epoll,size is not used, usually 1 is good
int eFd = epoll_create(1);
Wrap the socket as an epoll_event object
// add to epoll
epoll_event epev{};
epev.events = EPOLLIN;// Events that can be responded to
epev.data.fd = socketFd;// The file descriptor of the socket
epoll_ctl(eFd, EPOLL_CTL_ADD, socketFd, &epev);// Add to epoll
// An array of callback events that are returned when there are response events in epoll
epoll_event events[EVENTS_SIZE];
// The entire epoll_wait process is processed in an infinite loop
while (true) {
// This function blocks until timeout or a response event occurs
int eNum = epoll_wait(eFd, events, EVENTS_SIZE, - 1);
if (eNum == - 1) {
cout << "epoll_wait" << endl;
return - 1;
}
// Walk through all the events
for (int i = 0; i < eNum; i++) {
// Check whether the socket is readable this time.
if (events[i].data.fd == socketFd) {
if (events[i].events & EPOLLIN) {
sockaddr_in cli_addr{};
socklen_t length = sizeof(cli_addr);
// Accept connections from sockets
int fd = accept(socketFd, (sockaddr *) &cli_addr, &length);
if (fd > 0) {
// Set response events, set readable and edge (ET) modes
// Many people register writable events (EPOLLOUT), as explained below
epev.events = EPOLLIN | EPOLLET;
epev.data.fd = fd;
// Set the connection to non-blocking mode
int flags = fcntl(fd, F_GETFL, 0);
if (flags < 0) {
cout << "set no block error, fd:" << fd << endl;
continue;
}
if (fcntl(fd, F_SETFL, flags | O_NONBLOCK) < 0) {
cout << "set no block error, fd:" << fd << endl;
continue;
}
// Add a new connection to epoll
epoll_ctl(eFd, EPOLL_CTL_ADD, fd, &epev);
cout << "client on line fd:"<< fd << endl; }}}else {// Is not a response event to the socket
// Determine if there is a disconnect or connection error
// Because the connection is disconnected and an error occurs, the 'EPOLLIN' event is also responded to
if (events[i].events & EPOLLERR || events[i].events & EPOLLHUP) {
// Delete the connection from epoll when an error occurs
// The first is the descriptor of the epoll to operate on
// All event parameters are null
epoll_ctl(eFd, EPOLL_CTL_DEL, events[i].data.fd, nullptr);
cout << "client out fd:" << events[i].data.fd << endl;
close(events[i].data.fd);
} else if (events[i].events & EPOLLIN) {// If it is a readable event
In Windows, recv() is used to read data from the socket
int len = read(events[i].data.fd, buff, sizeof(buff));
If there is an error in reading data, close and remove the connection from epoll
if (len == - 1) {
epoll_ctl(eFd, EPOLL_CTL_DEL, events[i].data.fd, nullptr);
cout << "client out fd:" << events[i].data.fd << endl;
close(events[i].data.fd);
} else {
// Print the read data
cout << buff << endl;
// Send data to the client
char a[] = "123456";
// In Windows, send() is used to write data to a socket
write(events[i].data.fd, a, sizeof(a));
}
}
}
}
}
}
Copy the codeIn the comments, just explain the problem of new connection registration events. Many articles register writable events as well, like this
sockaddr_in cli_addr{};
socklen_t length = sizeof(cli_addr);
// Accept connections from sockets
int fd = accept(socketFd, (sockaddr *) &cli_addr, &length);
if (fd > 0) {
epev.events = EPOLLIN | EPOLLET | EPOLLOUT;
epev.data.fd = fd;
epoll_ctl(eFd, EPOLL_CTL_ADD, fd, &epev);
cout << "client on line fd:" << fd << endl;
}
Copy the codeHowever, after testing, it is also possible to write directly into the socket without registering writable events.
- EPOLLIN: If the state changes (from nothing, for example), it is triggered as soon as the input buffer is readable
- EPOLLOUT: If the state changes (from full to full, for example), it will be touched as soon as the output buffer is writable
If you register writable, there will be frequent callbacks, and there will be a lot of useless callbacks, resulting in performance degradation. If the write function fails to write to the socket (return value == -1), register EPOLLOUT and write data to the socket again after the writable event is received. If the write succeeds, cancel EPOLLOUT. I won’t give you an example here
Client testing
This time focus on the server implementation, the client is not written in C++, use Go to write a client (no other reason, just because Go is easy to write)
package main
import (
"fmt"
"net"
"sync"
"time"
)
const (
MAX_CONN = 10
)
func main(a) {
var wg sync.WaitGroup
wg.Add(1)
for i := 0; i < MAX_CONN; i++ {
go Conn("192.168.199.164:8088", i)
time.Sleep(time.Millisecond * 100)
}
wg.Wait()
}
func Conn(addr string, id int) {
conn, err := net.Dial("tcp", addr)
iferr ! =nil {
fmt.Println(err)
return
}
fmt.Println("connect ", id)
go func(a) {
buf := make([]byte.1024)
for {
n, err := conn.Read(buf)
iferr ! =nil {
break
}
fmt.Println(id, "read: ".string(buf[:n]))
}
}()
time.Sleep(time.Second * 1)
for {
_, err := conn.Write([]byte("hello"))
iferr ! =nil {
break
}
time.Sleep(time.Second * 10)}}Copy the codeThis is just a test, it’s written rough, but it doesn’t affect use
The server prints information
The client prints information
conclusion
- Epoll is one of the socket multiplexing technologies, along with SELECT and poll
- Epoll can only be used on Linux.
- Epoll events have both Level Triggered (LT) and Edge Triggered (ET) models, with LT being the default mode and ET being the high performance mode
In addition, I use an object-oriented way to encapsulate an Epoll tcpServer code a bit too much, I will not post here, has been uploaded to github code cloud
Welcome to give a star ヾ(O ◕∀◕) Blue ヾ