sequence

Recently, I plan to learn golang’s standard library, and read part of the source code in detail. This directory records the learning process and experience

Go language ioutil package provides a lot of convenient IO operation tool set, this article mainly detailed analysis of ReadAll method source code implementation.

ReadAll is a common method used to read IO.Reader data at once.

The source code to achieve

1. ReadAll

Reading the source code below, we can see that ReadAll actually calls a non-exported method, which we trace step by step

// ReadAll reads from r until an error or EOF and returns the data it read. // A successful call returns err == nil, not err == EOF. Because ReadAll is // defined to read from src until EOF, It does not treat an EOF from Read // as an error to be reported. // ReadAll reads data from r until EOF or encounters an error. // Successful calls return nil err instead of EOF. // Because this function is defined to read r up to EOF, it does not treat the EOF returned by reading as an error to report. func ReadAll(r io.Reader) ([]byte, error) { return readAll(r, bytes.MinRead) }Copy the code
2. readAll

Reading this function code, you can see that ioutil.ReadAll is essentially implemented using bytes.buffer, which calls two bytes.buffer methods, one to initialize the size of the buffer and one to ReadAll IO

In addition, we can also learn the use of Panic Recover in Go. In the buffer, if sufficient memory cannot be allocated, the error panic bytes.ErrTooLarge will be directly caused. However, in this method, we expect to return this error. At this point, you can use defer Recover to recover from panic

// readAll reads from r until an error or EOF and returns the data it read // from the internal buffer allocated with a Specified capacity. // readAll reads an error or EOF from R and returns data read from the internal buffer for the specified capacity allocation. func readAll(r io.Reader, capacity int64) (b []byte, Var buf bytes. buffer // If the buffer overflows, // Return that as an error. Any other panic remains. // If buffer overflows, You get a bytes.ErrTooLarge error // If you get a bytes.ErrTooLarge error, return it, other Panic error, Still panic defer func() {e := recover() if e == nil {return} if panicErr, ok := e.(error); Ok && panicErr == bytes.ErrTooLarge {err = panicErr} else {panic(e)}}() // Check whether the value of capacity exceeds the upper limit of the int type if Int64 (int(capacity)) == Capacity {// Add capacity to the buffer buf.grow (int(capacity))} // Use buffer ReadFrom ReadFrom(r) return buf.bytes (), err}Copy the code
3. buffer

To see the implementation of the buffer function, take a look at the definition of buffer, otherwise you may be confused

Buf is written from len(buf) to off. The value before off indicates that data has been read, and off indicates the current position. Both buffer write and read methods start at this position

// A Buffer is a variable-sized buffer of bytes with Read and Write methods. // The zero value for Buffer is an empty Buffer ready to use. // A buffer is a variable size byte buffer that implements read and write methods. // A zero value of this type is an empty buffer that can be used for reading and writing. type Buffer struct { buf []byte // contents are the bytes buf[off : len(buf)] off int // read at &buf[off], write at &buf[len(buf)] lastRead readOp // last read operation, so that Unread* can work correctly. // FIXME: lastRead can fit in a single byte // memory to hold first slice; helps small buffers avoid allocation. // FIXME: it would be advisable to align Buffer to cachelines to avoid false // sharing. bootstrap [64]byte }Copy the code
4. buffer.Grow

This method is mainly used to increase the memory of the buffer, and actually calls the non-exported grow method

// Grow grows the buffer's capacity, if necessary, to guarantee space for // another n bytes. After Grow(n), at least n bytes can be written to the // buffer without another allocation. // If n is negative, // If the buffer can't Grow it will panic with ErrTooLarge. // Increase the buffer size as necessary to ensure n bytes of remaining space. // After calling Grow(n), at least n bytes of data can be written to the buffer without requiring memory. // A panic ErrTooLarge error occurs if n is less than zero or the capacity cannot be increased. func (b *Buffer) Grow(n int) { if n < 0 { panic("bytes.Buffer.Grow: negative count") } m := b.grow(n) b.buf = b.buf[0:m] }Copy the code
5. buffer.ReadFrom

Let’s look at the implementation of this method, which reads all data from IO.Reader.

Schematic diagram:

// ReadFrom reads data from r until EOF and appends it to the buffer, growing // the buffer as needed. The return value n is the number of bytes read. Any // error except io.EOF encountered during the read is also returned. If the // buffer becomes too large, ReadFrom will panic with ErrTooLarge. // ReadFrom reads data from r until it ends and writes the read data to the buffer, increasing the buffer capacity if necessary. // The return value n is the number of bytes read from r and written to B; Errors other than IO.EOF encountered while reading are returned. // If the buffer is too large, ReadFrom will cause panic with the error value ErrTooLarge. func (b *Buffer) ReadFrom(r io.Reader) (n int64, err error) { // const opInvalid = 0 // Non-read operation. B. astRead = opInvalid // If buffer is empty, reset to recover space. // If buffer is empty, reset to recover space. If b.off >= len(b.bouf) {b.reset ()} MinRead = 512 if free := cap(b.buf) - len(b.buf); Free < MinRead {// not enough space at end // insufficient space // create a newBuf newBuf := b.bof // check whether the actual free capacity is smaller than MinRead = 512 if b.off+free < MinRead { // not enough space using beginning of buffer; MakeSlice (makeSlice); makeSlice (makeSlice); makeSlice (makeSlice); makeSlice (makeSlice); makeSlice (makeSlice); makeSlice (makeSlice); makeSlice (makeSlice); makeSlice (makeSlice); NewBuf = makeSlice(2*cap(b.buuf) + MinRead)} copy(newBuf, Buf [b.off:]) // Len (b.off)-b.off equals the length of the current buF content // example: A: = make (byte [], 20) / / b: = a [10] / / len 10 / / / / cap (b) (b) / / 20 b.b uf = newBuf [: len (b.b uf) - b.o ff] / / to put off 0 b.o ff = 0} IO.EOF error; IO.EOF error; IO.EOF error; Return data length m and nil m, e := r.read (b.bouf [len(b.bouf):cap(b.bouf)]) // Only get buF with data, cap b.bouf = b.bouf [0: Len (b.buf)+m] n += int64(m) if e == IO. = nil { return n, e } } return n, nil // err is EOF, so return nil explicitly }Copy the code