HTTP method of transferring large files
First, data compression
Typically, browsers send requests with an accept-Encoding header containing a list of supported compression formats, such as gzip, Deflate, BR, etc., so that the server can select one of the compression algorithms and place it in the content-Encoding header. The original data is then compressed and sent to the browser.
Two, block transmission
The transferred files are broken down into small pieces and sent to the browser in batches. The browser can assemble and restore the files after receiving them.
In THE HTTP protocol, it is “chunked”, and the header field “Transfer-encoding :chunked” is used in the response message to indicate that the body part of the message is not sent at one time, but is divided into many chunks.
Note: The “transfer-encoding :chunked” and “Content-Length” fields are mutually exclusive. The transmission Length of a response message is either known or unknown (chunked).
Let’s take a look at the coding rules for block transfer, which is also very simple, and also uses plaintext, similar to the response header.
- Each block contains two parts, a length header and a data block;
- The length header is a line of plain text ending in CRLF (carriage return newline, i.e. \r\n), with a hexadecimal number to indicate the length;
- The data block follows the length header and also ends with CRLF, but the data does not contain CRLF;
- The end is indicated by a block of length 0, i.e. “0\r\n\r\n”.
The diagram below:
Three, scope request
Scope requests can be used when only a portion of the data is requested, not all of it. Allows clients to use special fields in the request header to indicate that only a portion of the file is fetched.
Range requests are not a required feature of the Web server, so the server must use the accept-range: bytes field in the response header to explicitly tell the client that I support Range requests. If not, send “accept-range: None” or do not send the accept-range field.
The format is bytes=x-y, where x and y are the data Range in bytes. X and y represent “offsets” and ranges must count from 0. Example: The first 10 bytes are 0-9
The Range format is flexible, the starting point x and the ending point y can be omitted:
- “0-” indicates the entire file from the beginning to the end of the document.
- “10-” starts at the 10th byte and ends at the end of the document
- “-1” is the last byte of the document
- “-10” is the countdown of 10 bytes from the end of the document
After the server receives the Range field, it needs to do four things:
- It must check if the range is legal, such as if the file is only 100 bytes long, but the request is “200-300”, which is a range out of bounds. The server returns status code 416, which means “The scope of your request is incorrect. I can’t process it. Please check again.”
- If the Range is correct, the server calculates the offset from the Range header and reads a fragment of the file, returning a status code of “206 Partical Content,” which means something like 200, but indicating that the body is only part of the original data.
- The server will add a response header field, content-range, that tells the actual offset of the fragment and the total size of the resource in the format of “bytes x-y/ Length”, which differs from the Range header with no “=” and the total length. For example, for range requests of “0-10”, the value is “bytes 0-10/100”.
- Finally, it is time to send the data, sending the fragment directly to the client using TCP, a range request is processed.
4. Multi-segment data
Use multiple “X-y” in the Range header to obtain multiple fragment data at one time.
This requires the use of a special MIME type: “multipart/byteranges”, which indicates that the boby of the message is made up of a sequence of multiple bytes, and a parameter “Boundary = XXX” is used to mark the separation between the segments.
The format of multi-segment data is similar to that of block transmission, but it needs to be marked with boundary to distinguish different segments. The figure can be used for comparison:
Each segment must start with ‘– boundary’ (followed by two ‘-‘), then mark the Type and Range of the segment with ‘content-Type’ and ‘content-range’, then end with a carriage return line feed just like a normal response header, then add the segment data, A “– boundary –” (two “–” in front and two “–” in front) was used to indicate the end of all sections.
Note that these four methods are not mutually exclusive, but can be used in combination, such as compressed and then chunked, or segmented and then chunked.
HTTP connection management
Short connection
Before sending a request, establish a connection with the server. After receiving a response packet, the server establishes a connection immediately. Because the entire connection between a client and a server is short and does not remain connected for a long time, it is called a short-lived connection.
The disadvantage of short connections is quite serious because establishing and closing a connection in TCP is an “expensive” operation. TCP requires a “three-way handshake” to establish a connection, and sends three data packets, requiring one RTT. Closing the connection is “four waves” and requires 2 RTT for 4 packets. A simple “request-response” for HTTP typically requires only four packets, or at most two RTTS if you don’t count the processing time inside the server. So the wasted time is “3÷5=60%”, which is two-thirds of the time wasted, and the transmission efficiency is shockingly low.
A long connection
The long connection is to keep the connection, using the idea of “cost sharing”, since the TCP connection and closing is very time-consuming, so the cost of this time from the original “request-reply” spread across multiple “request-reply”.
Header fields associated with the connection
Long connections are enabled by default for HTTP/1.1 connections. You can also explicitly request long connections in the header, using the Connection field and the value “keep-alive”. Regardless of whether the client asks for long connections or not, if the server supports long connections, it will always put a “Connection: keep-alive” field in the response message telling the client, “I support long connections, so use this TCP to send and receive data.”
The disadvantage of long connections is that if the TCP connection is left open for a long time, the server must keep its state in memory, which consumes the server’s resources. Therefore, long connections also need to be closed at the appropriate time, and cannot remain connected to the server forever, which can be done on both the client and the server.
On the client side, you can add a “Connection: close” field to the request header to tell the server, “Close the Connection after this communication.” When the server sees this field, it knows that the client wants to close the connection. Therefore, the server adds this field to the response packet and calls the Socket API to close the TCP connection after sending the response packet.
The server usually does not actively close the connection, but some strategies can be used. Nginx, for example, works in two ways:
- Run the keepalive_timeout command to set the timeout period for a long connection. If no data is sent or received within a period of time, disconnect the connection to prevent idle connections from occupying system resources.
- Use the Keepalive_requests directive to set the maximum number of requests that can be sent over a long connection. For example, if set to 1000, Nginx will automatically disconnect after 1000 requests have been processed on this connection.
In addition, both the client and server can attach the keep-alive: timeout=value header field to the packet to set the timeout period for the long connection. However, this field is not very binding and may not be followed by both parties of the communication, so it is not common.
Team head block
“Head-of-line blocking” has nothing to do with short and long connections, but rather because HTTP states that packets must be “one-in-one-out”, forming a “serial” queue with first-in, first-out (FIFO). If the first queue request is delayed because it is being processed too slowly, subsequent requests will have to wait as well.
Performance optimization
Because the request-reply model cannot be changed, the “queue head blocking” problem cannot be solved in HTTP/1.1, only mitigated. The remedy is “concurrent connections”, which means making multiple long connections to the same domain at the same time, using data to solve quality problems. However, HTTP recommends that clients use concurrency but not abuse it, so there is a limit on the number of concurrent requests (6 to 8).
Domain sharing: Since HTTP and browsers limit the number of concurrent connections, I open several domains, all pointing to the same server, and the actual number of connections goes up again.
HTTP redirects and jumps
Active jump: initiated by the user of the browser; Passive Redirection: an HTTP Redirection initiated by the server and beyond the control of the browser user.
The process of redirection
Using Chrome to access the URI “/18-1”, it immediately jumps to “/index.html” using 302.
As you can see, this time the “redirect” actually sends two HTTP requests, the first one returning 302, and then the second one being redirected to “/index.html”. Let’s look at the response message returned from the first request:
A new header field “Location:/index.html” appears. The Location field belongs to the response field and must appear in the response packet. But it only makes sense with the 301/302 status code, which marks the URI the server is asking for redirection, in this case to redirect the browser to “index.html”.
Note: You can safely use relative URIs when redirecting only within a station. But if you want to jump out of a station, you must use an absolute URI.
In addition, you need to be careful of performance losses when using redirects and avoid circular jumps (i.e., A->B->A)
HTTP Cookie mechanism
The Cookie mechanism of HTTP is equivalent to the server affiking a small piece of paper to each client, which writes some data that only the server can understand. When needed, the client sends these information to the server, and the server sees the Cookie and can recognize the other side.
How cookies work
- When a user accesses the server through a browser for the first time, the server does not know his identity. Therefore, create a unique identifier data in the format of “key=value” and put it in the set-cookie field, which is sent to the browser along with the response message. The server sometimes adds multiple set-cookies to the response header, storing multiple “key=value” values. But the browser doesn’t need to use multiple Cookie fields, just use one line “; “Just separate it.”
- The browser receives the response packet and sees set-cookie in it. It knows that set-cookie is the identity given by the server, so it saves the value and sends it to the server automatically in the Cookie field the next time it requests the value.
- Because there is a Cookie field in the second request, the server knows that the user is not new, has been here before, can take out the value in the Cookie, identify the user’s identity, and then provide personalized service.
Cookies are “browser-bound” and take effect only in this browser.
The attribute of the Cookie
Cookies are data that the server trusts the browser to store in the client, and these data usually record key identifying information of the user. Therefore, it is necessary to use some means outside “key=value” to protect against leakage or theft, which is the attribute of Cookie.
First, we should set the Cookie’s lifetime, its expiration date, using the Expires and max-age attributes.
- Expires is commonly known as an expiration date. It uses an absolute time, which can be understood as a deadline
- “Max-age” uses the relative time, in seconds. The browser uses the time at which the message was received and adds max-age to obtain the absolute expiration time.
Expires and max-age can be present at the same time, and both expire at the same or different time, but the browser uses max-age to calculate the expiration period.
Second, we need to set the scope of cookies so that browsers only send to specific servers and URIs. Domain and Path specify the Domain name and Path of the Cookie. Before sending the Cookie, the browser extracts the host and Path parts from the URI to compare the Cookie attributes. If the criteria are not met, the Cookie is not sent in the request header.
The last thing to consider is the security of cookies. Try not to let people outside the server see them.
- “HttpOnly” : this Cookie can only be transmitted through the browser HTTP protocol, prohibit other ways to access, the browser JS engine will disable document. Cookie and other relevant API, scripting attacks will also be out of the way.
- “SameSite” : “SameSite=Strict” ensures that cookies cannot be sent across sites following jump links, while “SameSite=Lax” allows GET/HEAD and other security methods. However, cross-site POST is prohibited.
- Secure: indicates that the Cookie can only be encrypted and transmitted using HTTPS. Plaintext HTTP forbids sending. But the Cookie itself is not encrypted, the browser is still in clear text form.
The application of the Cookie
- One of the most basic uses is identification, saving user login information, and realizing session transactions.
- Another common use is AD tracking. When you surf the Internet, you must have seen a lot of advertising pictures. Behind these pictures are advertising sites (such as Google), which will “secretly” attach cookies to you, so that when you visit other sites, other advertising can use cookies to read your identity, and then conduct behavior analysis, and then push you to advertise.
Q&A
Q: When transferring data in blocks, will it affect the processing of blocks if the data contains carriage return (\r\n)?
A: No, because there is A description of the data length before partitioning
Q: If a Range request is made on a gzip file, such as “Range: bytes=10-19”, does the Range apply to the original file or to the compressed file?
A: Look at the form of the original document. If the original file is gzip, it is correct. If the original file is text and is compressed during transmission, it is applied to the data before compression. Anyway, range is for the original file.
Q: How do you choose the connection mode to use when developing an HTTP-based client? Short connection or long connection?
A: According to the frequency of requests, short connections are used for one-time requests and long connections are used for frequent interactions with the server.
Q: How can I reduce the negative impact of long connections on the server?
A: You can set the connection number and timeout period based on the server performance to ensure that the TCP resource usage of the server is within the normal range.
Q: 301 and 302 are very similar. Try to describe the similarities and differences using your own understanding.
A: 301 is used to switch to A new address when the original address is discarded. 302 is used to switch to A new address when the original address cannot be accessed temporarily. In both cases, the browser needs to initiate A request again
Q: Can you list a few more scenarios where redirection should be used, based on your own situation?
A: Indicates that the login fails. During the Web upgrade, you can redirect to another service to display the upgrade progress
Q: What happens if the Cookie’s max-age property is set to 0?
A: Max-age =0 means that it cannot be cached but is available during the session. Cookies can be used to record user information before the browser session is closed. The cookie expires when the session ends.
Q: The benefits of cookies are clear. What do you think are the disadvantages?
A: 1. It’s not safe. If a middleman obtains a Cookie, he impersonates it as a user’s credentials. The solution is to use HTTPS for encryption. 2. There are quantity and size limits. In addition, cookies that are too large are also bad, because the transmitted data will become larger. 3. The client may not save cookies. For example, the user disables the Cookie saving function of the browser when the user sends or receives data over Telnet.