Go Library of the Day NET/HTTP (Basic and Middleware)

Introduction to the

Almost all programming languages use Hello World as an example of a starter program, and some of them start with a real life case of writing a Web server. Each programming language has a number of libraries for writing Web servers, either as standard libraries or through third-party libraries. The Go language is no exception. This article and subsequent articles explore the various Web programming frameworks in the Go language, their basic uses, reading their source code, and comparing their strengths and weaknesses. Let’s start with the Go language’s standard library, NET/HTTP. The standard library NET/HTTP makes the job of writing a Web server very simple. Let’s explore how to use the NET/HTTP library to implement some common features or modules that will help us learn about other libraries or frameworks.

Hello World

Writing a simple Web server using NET/HTTP is very simple:

package main

import (
  "fmt"
  "net/http"
)

func index(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintln(w, "Hello World")
}

func main() {
  http.HandleFunc("/", index)
  http.ListenAndServe(":8080", nil)
}

First, we register the path handler with a call to http.handlefunc (“/”, index), which sets the path/handler to index. The handler must be of type:

func (http.ResponseWriter, *http.Request)

Where *http.Request represents the HTTP Request object, which contains all the information of the Request, such as URL, header, form content, other content of the Request, etc.

Http. responseWriter is an interface type:

// net/http/server.go
type ResponseWriter interface {
  Header() Header
  Write([]byte) (int, error)
  WriteHeader(statusCode int)
}

A type that implements the responseWriter interface for sending responses to the client obviously implements the IO.Writer interface as well. So in the index handler, you can call fmt.fprintln () to write the response to the responseWriter.

Read the source code for handleFunc () in the NET/HTTP package:

func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
  DefaultServeMux.HandleFunc(pattern, handler)
}

We found that it directly calls a handleFunc () method on an object called DefaultServemux. DefaultServemux is an instance of the Servemux type:

type ServeMux struct {
  mu    sync.RWMutex
  m     map[string]muxEntry
  es    []muxEntry // slice of entries sorted from longest to shortest.
  hosts bool       // whether any patterns contain hostnames
}

var DefaultServeMux = &defaultServeMux
var defaultServeMux ServeMux

This use of providing an instance of a default type is common across libraries in the Go language, and it is convenient to use the default implementation in situations where the default parameters are sufficient. Servemux keeps the correspondence of all registered paths and handlers. The servemux.handlefunc () method is as follows:

func (mux *ServeMux) HandleFunc(pattern string, handler func(ResponseWriter, *Request)) {
  mux.Handle(pattern, HandlerFunc(handler))
}

Here we convert the handler function to the handlerFunc type, and then call the servemux.handle () method to register. Note that handlerFunc (handler) is a type conversion, not a function call. The type handlerFunc is defined as follows:

type HandlerFunc func(ResponseWriter, *Request)

func (f HandlerFunc) ServeHTTP(w ResponseWriter, r *Request) {
  f(w, r)
}

HandlerFunc actually takes the function type Func (responseWriter, *Request) as the underlying type, and defines the method serveHttp for the HandlerFunc type. Yes, the Go language allows you to define methods for (based on) the type of a function. The Serve.handle () method only accepts arguments of type Handler:

type Handler interface {
  ServeHTTP(ResponseWriter, *Request)
}

func (mux *ServeMux) Handle(pattern string, handler Handler) {
  if mux.m == nil {
    mux.m = make(map[string]muxEntry)
  }
  e := muxEntry{h: handler, pattern: pattern}
  if pattern[len(pattern)-1] == '/' {
    mux.es = appendSorted(mux.es, e)
  }
  mux.m[pattern] = e
}

Obviously HandlerFunc implements the interface Handler. The handlerFunc type is simply for the convenience of registering handlers of function type. We could of course define a type that implements the Handler interface and then register an instance of that type:

type greeting string

func (g greeting) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  fmt.Fprintln(w, g)
}

http.Handle("/greeting", greeting("Welcome, dj"))

We define a new type of greeting based on string, define a method serveHttp () for it (implementing Handler), and call the http.handle () method to register the Handler.

For the sake of distinction, we’ll call those registered with handleFunc () handlers, and those registered with Handle() handlers. From the above source code analysis, it is not difficult to see that they are essentially the same at the bottom.

After the processing logic is registered, it calls http.listenAndServe (“:8080”, nil) to listen to port 8080 on the local computer and begin processing the request. The following look at the source processing:

func ListenAndServe(addr string, handler Handler) error {
  server := &Server{Addr: addr, Handler: handler}
  return server.ListenAndServe()
}

ListenAndServe creates an object of type Server:

type Server struct {
  Addr string
  Handler Handler
  TLSConfig *tls.Config
  ReadTimeout time.Duration
  ReadHeaderTimeout time.Duration
  WriteTimeout time.Duration
  IdleTimeout time.Duration
}

Server structure have more fields, we can use these fields to adjust the parameters of the Web Server, such as the ReadTimeout/ReadHeaderTimeout/WriteTimeout/IdleTimeout is used to control the reading and writing and idle timeout. In this method, we first call the net.Listen() listening port, and then call the server. Serve() method with the returned net.Listener as an argument:

func (srv *Server) ListenAndServe() error { addr := srv.Addr ln, err := net.Listen("tcp", addr) if err ! = nil { return err } return srv.Serve(ln) }

In the server.serve () method, using an infinite for loop, the listener.accept () method is called repeatedly to Accept new connections and the new goroutine is opened to handle new connections:

func (srv *Server) Serve(l net.Listener) error { var tempDelay time.Duration // how long to sleep on accept failure for { rw, err := l.Accept() if err ! = nil { if ne, ok := err.(net.Error); ok && ne.Temporary() { if tempDelay == 0 { tempDelay = 5 * time.Millisecond } else { tempDelay *= 2 } if max := 1 * time.Second; tempDelay > max { tempDelay = max } srv.logf("http: Accept error: %v; retrying in %v", err, tempDelay) time.Sleep(tempDelay) continue } return err } tempDelay = 0 c := srv.newConn(rw) go c.serve(connCtx) } }

Here is one use of the exponential retreat strategy. If the l.acCept () call returns an error, we determine if the error is ne.temporary (). If it is a temporary error, Sleep will try again after a short period of time. Each temporary error occurs, the time of Sleep will be doubled, up to Sleep 1s. Once you have a new connection, encapsulate it as a conn object (srv.newconn (rw)), and create a goroutine to run its serve() method. The code to omit extraneous logic is as follows:

func (c *conn) serve(ctx context.Context) {
  for {
    w, err := c.readRequest(ctx)
    serverHandler{c.server}.ServeHTTP(w, w.req)
    w.finishRequest()
  }
}

The serve() method simply reads the requests sent by the client, creates a ServerHandler object and calls its serveHttp () method to handle the request, and then does some cleaning up. ServerHandler is just an intermediate helper structure with the following code:

type serverHandler struct {
  srv *Server
}

func (sh serverHandler) ServeHTTP(rw ResponseWriter, req *Request) {
  handler := sh.srv.Handler
  if handler == nil {
    handler = DefaultServeMux
  }
  handler.ServeHTTP(rw, req)
}

Retrieve the Handler from the Server object, which is the second parameter passed in the http.listenAndServe () call. In the Hello World sample code, we pass in nil. So this handler is going to take the default, defaultServemux. Call DefaultServeMux. ServeHTTP () method processes the request:

func (mux *ServeMux) ServeHTTP(w ResponseWriter, r *Request) {
  h, _ := mux.Handler(r)
  h.ServeHTTP(w, r)
}

Handler(r) looks up the Handler with the path information of the request, and then calls the Handler’s serveHttp () method to process the request:

func (mux *ServeMux) Handler(r *Request) (h Handler, pattern string) {
  host := stripHostPort(r.Host)
  return mux.handler(host, r.URL.Path)
}

func (mux *ServeMux) handler(host, path string) (h Handler, pattern string) {
  h, pattern = mux.match(path)
  return
}

func (mux *ServeMux) match(path string) (h Handler, pattern string) {
  v, ok := mux.m[path]
  if ok {
    return v.h, v.pattern
  }

  for _, e := range mux.es {
    if strings.HasPrefix(path, e.pattern) {
      return e.h, e.pattern
    }
  }
  return nil, ""
}

The code above omits a lot of irrelevant code. In the match method, it first checks to see if the path matches exactly the mux.m[path]. If it does not match exactly, the subsequent for loop matches the longest prefix of the path. Once the/root path handler is registered, all unmatched paths will eventually be handed over to the/path handler. To ensure that the longest prefix takes precedence, the path is sorted during registration. So mux.es holds a list of processes sorted by path:

func appendSorted(es []muxEntry, e muxEntry) []muxEntry {
  n := len(es)
  i := sort.Search(n, func(i int) bool {
    return len(es[i].pattern) < len(e.pattern)
  })
  if i == n {
    return append(es, e)
  }
  es = append(es, muxEntry{})
  copy(es[i+1:], es[i:])
  es[i] = e
  return es
}

Run it, type the address localhost:8080 in your browser, and you’ll see the page display Hello World. Type the URL localhost:8080/greeting, and see the page showing Welcome, DJ.

Consider: according to logic of the longest prefix, if enter localhost: 8080 / the greeting, / a/b/c/the greeting should be matching path. If you type localhost:8080/a/b/c, it should match the/path. Is that right? Answers are posted at 😀.

create`ServeMux`

Calling http.handlefunc ()/ http.handle () registers the handler/function with the default Servemux object, DefaultServemux. There is one problem with using default objects: they are uncontrollable.

On the one hand, Server parameters use default values. On the other hand, third-party libraries may also use this default object to register some processing, which is prone to conflict. What’s more, if we unwittingly call http.listenAndServe () to start the Web service, then the processing logic of the third-party library registration can be accessed through the network, which poses a great security risk. Therefore, it is recommended not to use the default object except in the sample program.

We can create a NewServeMux object using http.newServemux (), then create the custom parameters for the http.server object, and initialize the Handler field for the Server with the ServeMux object. Finally, the Server.listenAndServe () method is called to start the Web service:

func main() {
  mux := http.NewServeMux()
  mux.HandleFunc("/", index)
  mux.Handle("/greeting", greeting("Welcome to go web frameworks"))

  server := &http.Server{
    Addr:         ":8080",
    Handler:      mux,
    ReadTimeout:  20 * time.Second,
    WriteTimeout: 20 * time.Second,
  }
  server.ListenAndServe()
}

This program is basically the same as the Hello World function above, with an additional read-write timeout set.

In order to understand, I have drawn two pictures, which are not complicated to tidy up:

The middleware

Sometimes it is necessary to add some common logic to the request handling code, such as counting processing time, logging, catching outages, and so on. If you add this logic to each request handler, your code quickly becomes unmaintainable and adding new handlers becomes cumbersome. Hence the need for middleware.

Middleware is a bit like facet-oriented programming ideas, but different from the Java language. In Java, generic processing logic (also known as facets) can be inserted into the normal logical processing flow via reflection, which is rarely done in the Go language.

In Go, middleware is implemented through function closures. Functions in the Go language are the first type of values that can be passed as arguments to other functions or returned as return values from other functions. We covered the use and implementation of handlers/functions earlier. You can then use closures to encapsulate existing handlers.

First, we define a middleware type based on the function type func(http.handler) http.handler:

type Middleware func(http.Handler) http.Handler

Next we write middleware. The simplest middleware would be to print a log before and after the request:

func WithLogger(handler http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { logger.Printf("path:%s process start... \n", r.URL.Path) defer func() { logger.Printf("path:%s process end... \n", r.URL.Path) }() handler.ServeHTTP(w, r) }) }

The implementation is simple, using middleware to encapsulate the original handler object and return a new handler function. In the new handler, print the log that started processing, and then print the finished log after the function ends with the defer statement. The serveHttp () method of the original handler object is then called to execute the original processing logic.

Similarly, let’s implement a middleware that takes time for statistical processing:

func Metric(handler http.Handler) http.HandlerFunc {
  return func (w http.ResponseWriter, r *http.Request) {
    start := time.Now()
    defer func() {
      logger.Printf("path:%s elapsed:%fs\n", r.URL.Path, time.Since(start).Seconds())
    }()
    time.Sleep(1 * time.Second)
    handler.ServeHTTP(w, r)
  }
}

Metric middleware encapsulates the original processor object, records the time before execution and outputs the time after execution. To make it easier to see the results, I added a time.sleep () call to the above code.

Finally, because the processing logic of the request is written by the feature developer (and not the library author), we need to catch possible panics for the stability of the Web server. The PanicRecover middleware is as follows:

func PanicRecover(handler http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { defer func() { if err := recover(); err ! = nil { logger.Println(string(debug.Stack())) } }() handler.ServeHTTP(w, r) }) }

Recover () is called to capture panic and output stack information in case the program exits unexpectedly. In fact, there is a recover() in the conn. Serve () method, and the program generally does not exit abnormally. But custom middleware can add our own custom logic.

Now we can register the handler like this:

mux.Handle("/", PanicRecover(WithLogger(Metric(http.HandlerFunc(index)))))
mux.Handle("/greeting", PanicRecover(WithLogger(Metric(greeting("welcome, dj")))))

This is a bit cumbersome, but we can write a helper function that takes the raw processor object and mutates multiple middleware. Apply these middleware to the handler object and return a new handler object:

func applyMiddlewares(handler http.Handler, middlewares ... Middleware) http.Handler { for i := len(middlewares)-1; i >= 0; i-- { handler = middlewares[i](handler) } return handler }

Note that the order of application is from right to left, that is, combined right, the closer to the original processor the later the execution.

Using the helper function, registration can be simplified as:

middlewares := []Middleware{ PanicRecover, WithLogger, Metric, } mux.Handle("/", applyMiddlewares(http.HandlerFunc(index), middlewares...) ) mux.Handle("/greeting", applyMiddlewares(greeting("welcome, dj"), middlewares...) )

The ApplyMiddleWares () function is called once each time the registration processing logic is registered, which is still a bit tedious. We can optimize this by encapsulating our own Servemux structure, and then defining a method Use() to save the middleware, Rewrite will Handle/HandleFunc incoming HTTP. HandlerFunc/HTTP Handler processor packing ServeMux middleware and then passed on to the bottom. The Handle () method:

type MyMux struct { *http.ServeMux middlewares []Middleware } func NewMyMux() *MyMux { return &MyMux{ ServeMux: http.NewServeMux(), } } func (m *MyMux) Use(middlewares ... Middleware) { m.middlewares = append(m.middlewares, middlewares...) } func (m *MyMux) Handle(pattern string, handler http.Handler) { handler = applyMiddlewares(handler, m.middlewares...) m.ServeMux.Handle(pattern, handler) } func (m *MyMux) HandleFunc(pattern string, handler http.HandlerFunc) { newHandler := applyMiddlewares(handler, m.middlewares...) m.ServeMux.Handle(pattern, newHandler) }

To register, simply create the MyMux object and call its Use() method into the middleware to be applied:

middlewares := []Middleware{
  PanicRecover,
  WithLogger,
  Metric,
}
mux := NewMyMux()
mux.Use(middlewares...)
mux.HandleFunc("/", index)
mux.Handle("/greeting", greeting("welcome, dj"))

This approach is simple and easy to use, but it has its problems. The biggest problem is that the middleware must be set up before the Handle/HandleFunc registration can be invoked. The added middleware will not take effect on the previously registered handler/function.

To solve this problem, we can rewrite the serveHTTP method and apply the middleware after the processor has been identified. This allows subsequent middleware additions to take effect. Many third-party libraries use this approach. The default serveHttp () method for http.servemux is as follows:

func (m *ServeMux) ServeHTTP(w http.ResponseWriter, r *http.Request) {
  if r.RequestURI == "*" {
    if r.ProtoAtLeast(1, 1) {
      w.Header().Set("Connection", "close")
    }
    w.WriteHeader(http.StatusBadRequest)
    return
  }
  h, _ := m.Handler(r)
  h.ServeHTTP(w, r)
}

It is also easy to reconfigure this method to define a mymux-type serveHttp () method by simply applying the current middleware after m.Handler(r) gets the handler:

func (m *MyMux) ServeHTTP(w http.ResponseWriter, r *http.Request) { // ... H, _ := m.Handler(r) // Just add the line h = applyMiddlewares(h, m.midlewares...) h.ServeHTTP(w, r) }

As we’ll see later when we look at the source code of other Web frameworks, many do the same thing. To test recovery from an outage, write a handler that triggers panic:

func panics(w http.ResponseWriter, r *http.Request) {
  panic("not implemented")
}

mux.HandleFunc("/panic", panics)

Then select localhost:8080/ and localhost:8080/greeting, and then select localhost:8080/panic.

To consider

Consider:

The sorted list of the longest prefixes is generated in the servemux.handle () method:

func (mux *ServeMux) Handle(pattern string, handler Handler) {
  if pattern[len(pattern)-1] == '/' {
    mux.es = appendSorted(mux.es, e)
  }
}

There is an obvious limitation that the registration path must end in/for it to fire. So the localhost: 8080 / the greeting/a/b/c and localhost: 8080 / a/b/c will only match/path. If you want to let the localhost: 8080 / the greeting/a/b/c matching path/the greeting, registered path need to/the greeting / :

http.Handle("/greeting/", greeting("Welcome to go web frameworks"))

The request path /greeting will automatically redirect (301) to /greeting/.

conclusion

This article describes the basic process of using the standard library NET/HTTP to create a Web server, step by step analysis of the source code. It then shows how to use middleware to simplify common processing logic. Learning and understanding the contents of the NET/HTTP library is very helpful for learning about other Go Web frameworks. Most third-party Go Web frameworks implement their own Servemux objects based on NET/HTTP.

If you find a fun and useful Go library, please submit an issue😄 on Go Daily’s GitHub

reference

Go a library GitHub:https://github.com/darjun/go-daily-lib daily

I

My blog: https://darjun.github.io

Welcome to pay attention to my WeChat public number [GOUPUP], learn together, make progress together ~

Go Library of the Day NET/HTTP (Basic and Middleware)

Introduction to the

Hello World

createServeMux

The middleware

To consider

conclusion

reference

I

Related Posts

Run the FYNE.com website locally

Go language concurrency: declaration and use of Goroutines and Channels

Go Rip protocol simple simulation implementation

create`ServeMux`