Golang source code analysis of NET/HTTP (two) parsing source code
This is the second installment in the NET/HTTP theme of Golang source code analysis. We’ve looked at the basic framework of TCP communication in the previous chapter, and now we’re moving on to real source code analysis
The following source code is based on GO1.14.13 Windows/AMD64 analysis
Give me an example
Let’s write a C/S example
Service listens on port 1234, sets the route to /hello and the handler to HelloServer
//service.go
func main(a) {
http.HandleFunc("/hello",HelloServer)
// Listen for TCP addresses
// Handler is usually set to nil and DefaultServeMux is used (default router)
http.ListenAndServe(": 1234".nil)}// handle the function
func HelloServer(resp http.ResponseWriter, req *http.Request) {
req.ParseForm()
fmt.Println(req.Form)
fmt.Println("path",req.URL.Path)
fmt.Println("This is xc test")
fmt.Fprintf(resp, "Hello xc!")}Copy the codeThe client sends a request every 1s and prints the return value of the server
// client.go
func main(a) {
for {
time.Sleep(time.Second)
client := &http.Client{}
req, err := http.NewRequest("GET"."http://127.0.0.1:1234/hello".nil)
iferr! =nil {
fmt.Println("NewRequest:",err)
return
}
resp,err := client.Do(req)
iferr! =nil {
fmt.Println("client.Do:",err)
}
fmt.Println("resp:",resp)
}
}
Copy the codeNow start two processes (start the server, then its client)
// Service. Go logs are printed
&{GET /hello HTTP/1.1 1 1 map[Accept-Encoding:[gzip] User-Agent:[Go-http-client/1.1]] {} <nil> 0 [] false 127.0. 01.:1234 map[] map[] <nil> map[] 127.0. 01.:58176 /hello
<nil> <nil> <nil> 0xc000184a40}
ServeHTTP 0x654780
map[]
path /hello
This is xc test
// The client. Go logs are printed
resp: &{200 OK
200
HTTP/1.1
1
1
map[Content-Length:[9] Content-Type:[text/plain; charset=utf- 8 -] Date:[Fri, 02 Apr 2021 03:09:11 GMT]]
0xc0000a0040
9
[]
false
false
map[]
0xc000184100
<nil>}Copy the codeYou can view the network status and see that the CLIENT and server have ESTABLISHED a TCP connection. Both ends are in the ESTABLISHED state for reading and writing data. LISTEN Enables the client to monitor client access.
Second, source code analysis
1, server-side source code
1.1. Http.handlefunc registers the function that handles the request into the router map
// Register the handler function with the default router DefaultServeMux. The server listens to the corresponding route and uses this handler to process the route
func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
DefaultServeMux.HandleFunc(pattern, handler)
}
Copy the codeDefaultServeMux, also known as ServeMux, can take a look at the variables stored in this route
type ServeMux struct {
mu sync.RWMutex // Lock, because requests involve concurrent processing, a locking mechanism is required
m map[string]muxEntry // Routing rules, a string corresponds to a MUX entity, where the string is the registered route expression
es []muxEntry // slice of entries sorted from longest to shortest.
hosts bool // whether any patterns contain hostnames
}
Copy the codeThe pattern route and its corresponding processor are stored in a hash table, and the processor of the corresponding route will be searched when HTTP requests are processed
func (mux *ServeMux) Handle(pattern string, handler Handler) {
/ / lock
mux.mu.Lock()
defer mux.mu.Unlock()
...
if _, exist := mux.m[pattern]; exist {
panic("http: multiple registrations for " + pattern)
}
if mux.m == nil {
mux.m = make(map[string]muxEntry)
}
// Match pattern to the function, in this case "/hello" to HelloServer
e := muxEntry{h: handler, pattern: pattern}
mux.m[pattern] = e
if pattern[len(pattern)- 1] = ='/' {
mux.es = appendSorted(mux.es, e)
}
if pattern[0] != '/' {
mux.hosts = true}}Copy the code1.2. HTTP.ListenAndServe is used to listen on TCP connections and process requests
func ListenAndServe(addr string, handler Handler) error {
server := &Server{Addr: addr, Handler: handler}
return server.ListenAndServe()
}
Copy the codeListenAndServe listens on TCP at the corresponding address and processes the corresponding client requests
func (srv *Server) ListenAndServe(a) error {
if srv.shuttingDown() {
return ErrServerClosed
}
addr := srv.Addr
if addr == "" {
addr = ":http"
}
Srver creates a socket, bind, and listen
/ listen/HTTP
ln, err := net.Listen("tcp", addr)
iferr ! =nil {
return err
}
// Listen on the port
return srv.Serve(ln)
}
Copy the codeThe net.Listen interface includes creating sockets, binding sockets to addresses, and listening ports.
func socket(ctx context.Context, net string, family, sotype, proto int, ipv6only bool, laddr, raddr sockaddr, ctrlFn func(string.string, syscall.RawConn) error) (fd *netFD, err error) {
// syscall.Socket The system call creates the Socket
s, err := sysSocket(family, sotype, proto)
iferr ! =nil {
return nil, err
}
...
// Create a descriptor for the network to complete the filling of the netFD structure
iffd, err = newFD(s, family, sotype, net); err ! =nil {
poll.CloseFunc(s)
return nil, err
}
ifladdr ! =nil && raddr == nil {
switch sotype {
/ / for TCP
case syscall.SOCK_STREAM, syscall.SOCK_SEQPACKET:
// syscal.bind() + syscall.Listen
iferr := fd.listenStream(laddr, listenerBacklog(), ctrlFn); err ! =nil {
fd.Close()
return nil, err
}
return fd, nil
// UDP processing
case syscall.SOCK_DGRAM:
iferr := fd.listenDatagram(laddr, ctrlFn); err ! =nil {
fd.Close()
return nil, err
}
return fd, nil}}iferr := fd.dial(ctx, laddr, raddr, ctrlFn); err ! =nil {
fd.Close()
return nil, err
}
return fd, nil
}
Copy the codeListen for requests from the client and wait for requests from the client using a for loop.
// Listen for client requests
func (srv *Server) Serve(l net.Listener) error{...var tempDelay time.Duration // how long to sleep on accept failure
ctx := context.WithValue(baseCtx, ServerContextKey, srv)
for {
// Receive client request rW client request information
// Accept takes a completed connection from the head of the connection queue in the Established state
rw, err := l.Accept()
iferr ! =nil {
select {
case <-srv.getDoneChan():
return ErrServerClosed
default:}// If there is a network error, wait for a delay
if ne, ok := err.(net.Error); ok && ne.Temporary() {
if tempDelay == 0 {
tempDelay = 5 * time.Millisecond
} else {
tempDelay *= 2
}
if max := 1 * time.Second; tempDelay > max {
tempDelay = max
}
srv.logf("http: Accept error: %v; retrying in %v", err, tempDelay)
time.Sleep(tempDelay)
continue
}
return err
}
...
c := srv.newConn(rw) // Create a new connection
c.setState(c.rwc, StateNew) // before Serve can return
go c.serve(connCtx) // Handle client requests}}Copy the codeHandle client requests
// Serve a new connection.
func (c *conn) serve(ctx context.Context) {
c.remoteAddr = c.rwc.RemoteAddr().String()
ctx = context.WithValue(ctx, LocalAddrContextKey, c.rwc.LocalAddr())
...
// HTTP/1.x from here on.
ctx, cancelCtx := context.WithCancel(ctx)
c.cancelCtx = cancelCtx
defer cancelCtx()
c.r = &connReader{conn: c}
c.bufr = newBufioReader(c.r)
c.bufw = newBufioWriterSize(checkConnErrorWriter{c}, 4<<10)
for {
// Loop through user requests
w, err := c.readRequest(ctx)
ifc.r.remain ! = c.server.initialReadLimitSize() {// If we read any bytes off the wire, we're active.
c.setState(c.rwc, StateActive)
}
iferr ! =nil{...// Error handling}...// Process the request
serverHandler{c.server}.ServeHTTP(w, w.req)
// Dimension correlation. }}Copy the codeLeave it to the DefaultServeMux route
func (sh serverHandler) ServeHTTP(rw ResponseWriter, req *Request) {
handler := sh.srv.Handler
if handler == nil {
handler = DefaultServeMux
}
if req.RequestURI == "*" && req.Method == "OPTIONS" {
handler = globalOptionsHandler{}
}
// This is the HelloServer registered in the example
handler.ServeHTTP(rw, req)
}
Copy the code2, the client source
Http. NewRequest NewRequest builds a request based on the method name, URL, and request body. This method is encapsulated in the format of HTTP request without elaboration
Client.do (req) processes requests
For the Do method, we expand in detail, calling the core interface, roundTrip, at each level. This interface sends HTTP requests and processes Resp
func (t *Transport) roundTrip(req *Request) (*Response, error) {
// Request a check.for{...// Get the connection to send the request in two ways (1, idle connection in the queue, 2, create a new connection)
pconn, err := t.getConn(treq, cm)
iferr ! =nil {
t.setReqCanceler(cancelKey, nil)
req.closeBody()
return nil, err
}
var resp *Response
ifpconn.alt ! =nil {
// HTTP/2 path.
t.setReqCanceler(cancelKey, nil) // not cancelable with CancelRequest
resp, err = pconn.alt.RoundTrip(req)
} else {
// Process the response
resp, err = pconn.roundTrip(treq)
}
...
// Rewind the body if we're able to.
req, err = rewindBody(req)
iferr ! =nil {
return nil, err
}
}
}
Copy the codeConnection is a relatively expensive resource. If a new connection is established before each HTTP request is sent, it may consume more time and bring large extra overhead. Allocating and reusing resources through connection pool can effectively improve the overall performance of HTTP requests. Most network library clients adopt similar strategies to reuse resources.
So there are two ways to get a connection (t.gett Conn).
The first is to select connections that are idle in the queue
A long link consists of a map[[connectMethodKey]][][]*persistConn. A connectMethodKey is a connection key that contains information such as the address and corresponds to a handle to the connection whose value persistConn is persistConn. To select a connection in the queue is to get a handle. If successful, the connection is removed from the queue and the idleLRU.
Second: If there are no connections available in the free pool, the queueForDial method is called to create a new connection
The TCP connection is created using Dial, which initiates the socket’s LISTEN (). Once the connection is successfully created, two coroutines are opened, one for the input stream writeLoop and one for the output stream readLoop. As you can see from the connection setup process, if we created a new connection for each HTTP request and started Goroutine to process the read and write data, it would take a lot of resources.
After sending the request, pconn.roundTrip handles the response in the roundTrip interface
func (pc *persistConn) roundTrip(req *transportRequest) (resp *Response, err error){... writeErrCh :=make(chan error, 1)
pc.writech <- writeRequest{req, writeErrCh, continueCh}
resc := make(chan responseAndError)
pc.reqch <- requestAndChan{
req: req.Request,
cancelKey: req.cancelKey,
ch: resc,
addedGzip: requestedGzip,
continueCh: continueCh,
callerGone: gone,
}
// loop listener
for {
testHookWaitResLoop()
select{...case re := <-resc:
...
ifre.err ! =nil {
return nil, pc.mapRoundTripError(req, startBytesWritten, re.err)
}
return re.res, nil. }}}Copy the codereference
- Design principles of GO-HTTP
- Golang HTTP transport source code