HTTP is a stateless protocol. Stateless means that no persistent link needs to be established between the client (Web browser) and the server. This means that when a client makes a Request to the server and the Web server returns a “*” response, the connection is closed and no information about the connection is retained on the server.HTTP follows the Request/Response model. The client (Web browser) sends a request to the Web server, which processes the request and returns the appropriate response. All HTTP connections are structured as a set of requests and replies. There are four stages in this process, including establishing a connection, sending request information, sending response information and closing the connection, as shown in the figure below:
The HTTP workflow described in the figure above is detailed below
The client establishes a connection with the server through TCP three-way handshake.
After the TCP connection is established, the TCP server sends an HTTP request.
After the server receives the HTTP request from the client, it returns a reply and sends data to the client
The client disconnects from the server through TCP for four times.
In today’s experiment, we will configure HFS software to capture and analyze HTTP data by simulating data transfer between two machines on a LAN.
According to the experimental environment, the steps of this experiment are as follows:
1. Configure the HFS software to obtain HTTP GET data and POST data
2. Analyzing HTTP packets
Actual combat Step 1 ConfigurationHFS software, GET HTTP GET data and POST data
In a LAN environment, we use a gadget to implement the HTTP server. Configure HFS on the server first
1 configurationHFS software
Decompress locally, go to the folder, right-click to run as administrator. The following
We first configure HFS, which can meet our experimental requirements and obtain GET and POST data packets. Click on the port in the upper left corner, enter the port, here we use 8080, as follows, and hit OK
In the virtual file system area, right-click and choose “Add Directory from Disk”, select a real directory (it must be a real directory here), and select “Real Directory” from the pop-up selection directory type. Here we use the decompressed directory on the desktop, you can see that the directory is in red
Right-click the directory, click Set User Name and Password, enter the user name and password (Demo /demo) in the dialog box that is displayed, and click OK.
Right-click the directory, click Properties, select Upload Sheet, and select anyone. Click OK, and we have the HFS tool configured and accessible from the browser on the client side.
2 GET the HTTP GETData and POST data
On the test machine, open the Wireshark tool, enter ip.addr == 10.1.1.33 as the filter criterion, and then enter the URL provided by HFS to wait for the response from the server. After success, you can see the page on the tester’s machine’s browser as follows:
At this point, we’ve got the HTTP GET method. We saved the packets obtained by Wireshark as HTTP-get.
Click login on the page, enter the user name and password (demo/demo) in the dialog box, and wait for the server to respond after confirming. Success is as follows
Next, double-click the page folder (waiting for the server to respond) and restart Wireshark.
As shown in the image above, you will see a button on the left, click on the “Upload” button, select the file, here we select “http-post.txt” on the desktop, click upload. Wait for the server to respond. The message is uploaded successfully, as follows
We save the captured packet file named HTTP-post. Mission one, that’s it.
Step 2Analyzing HTTP packets
1 The HTTP messageformat
HTTP consists of a request and a response. Therefore, there are two packet formats. The following describes the formats of HTTP request packets and HTTP response packets.
Format of the HTTP request packet
In the table above, line 1 is “request line”, lines 2, 3, and 4 are “request header”, line 5 is blank, and line 6 is “request body”. The following four parts are introduced respectively:
(1) Request line: it consists of three parts: request method, URL (see Note 1) and protocol version, separated by Spaces. Request methods include GET and POST. The format of the protocol version is HTTP/ major version. The second version number is HTTP/1.0 and HTTP/1.1.
(2) The request header contains much useful information about the client environment as well as the request body. The request header consists of keyword: value pairs. Each line is a stack. The keyword and value are separated by colons (:).
(3) Blank line. This line is very important and indispensable. Indicates the end of the request header, and the request body follows.
(4) Request body: optional part, such as GET request has no request body; POST, for example, is the request body in the form of submitting form data.
Format of the HTTP response packet
In the table above, line 1 is “status line”, lines 2, 3, and 4 are “response header”, line 5 is blank, and line 6 is “response body”. The following four parts are introduced respectively:
(1) The status line consists of three parts: protocol version, status code, and status code description, separated by Spaces. The status code is a 3-digit number. 200 299 indicates success, 300 399 indicates resource redirection, 400 499 indicates client request error, and 500 599 indicates server error (HTTP/1.1 introduces informational status codes into the protocol, ranging from 100 to 199). Here are a few common ones:
| Status code | instructions |
|---|---|
| 200 | In response to success |
| 400 | The client request has a syntax error and cannot be recognized by the server |
| 404 | The requested resource does not exist |
| 500 | Server internal error |
(3) Blank line. This line is very important and indispensable. Indicates the end of the response header
(4) The response body, the document returned by the server, the most common is HTML web pages.
2 The HTTP header fields
Both HTTP request and reply messages contain header fields. The header field is divided into four types, among which the request header field and the response header field only appear in the request message and the response message respectively. The general header field and the entity header field can appear in both kinds of messages, but the entity header field only appears when the message contains entity data. The following describes the domain name cities and functions of the four header domains.
HTTP request header fields
| Header | explain |
|---|---|
| Accept | Specifies the type of content that the client can receive |
| Accept-Charset | A set of character encodings acceptable to the browser. |
| Accept-Encoding | Specifies the type of web server content compression encoding that the browser can support. |
| Accept-Language | Browser acceptable language |
| Accept-Ranges | You can request one or more subscope fields of a web page entity |
| Authorization | HTTP authorization certificate |
| Cache-Control | Specify the caching mechanism that requests and responses follow |
| Connection | Indicates whether a persistent connection is required. (HTTP 1.1 makes persistent connections by default) |
| Cookie | When an HTTP request is sent, all cookie values stored under the domain name of the request are sent to the Web server. |
| Content-Length | The content length of the request |
| Content-Type | MIME information that corresponds to the entity being requested |
| Date | The date and time the request was sent |
| Expect | The specific server behavior requested |
| From | Email address of the user who made the request |
| Host | Specifies the domain name and port number of the requested server |
| If-Match | This is valid only if the request content matches the entity |
| If-Modified-Since | If the part of the request is modified after the specified time, the request succeeds; if it is not modified, the 304 code is returned |
| If-None-Match | If the content has not changed, the 304 code is returned with the Etag sent by the server. The Etag is compared with the Etag returned by the server to determine whether it has changed |
| If-Range | If the entity has not changed, the server sends the missing part of the client, otherwise sends the whole entity. The parameter is also Etag |
| If-Unmodified-Since | The request succeeds only if the entity has not been modified after the specified time |
| Max-Forwards | Limit the amount of time messages can be sent through proxies and gateways |
| Pragma | Used to contain implementation-specific instructions |
| Proxy-Authorization | Certificate of authorization to connect to the agent |
| Range | Only a portion of the entity is requested, specifying scope |
| Referer | The address of the previous web page, followed by the current requested web page, is the incoming path |
| TE | The client is willing to accept the transmission code and notifies the server to accept the end plus header message |
| Upgrade | Specify some transport protocol to the server for the server to convert (if supported) |
| User-Agent | User-agent contains the information about the User that sends the request |
| Via | Notification intermediate gateway or proxy server address, communication protocol |
| Warning | Warning information about message entities |
| Accept | Specifies the type of content that the client can receive |
The reply header field appears only in the reply message and is some state and requirements that the Web server provides to the browser. The following
HTTP reply header field
| Header | explain |
|---|---|
| Accept-Ranges | Indicates whether the server supports scoped requests and what type of segmented requests |
| Age | Estimated time from the original server to proxy cache formation (in seconds, non-negative) |
| Allow | A valid request for a network resource. If not allowed, 405 is returned |
| Cache-Control | Tell all caching mechanisms whether they can cache and what type |
| Content-Encoding | The type of returned content compression encoding supported by the Web server. |
| Content-Language | The language of the response body |
| Content-Length | The length of the response body |
| Content-Location | Request an alternate address for alternate resources |
| Content-MD5 | Returns the MD5 check value of the resource |
| Content-Range | The byte position of this part in the entire return body |
| Content-Type | Returns the MIME type of the content |
| Date | The time when the original server message was sent |
| ETag | The current value of the entity label of the request variable |
| Expires | The expiration date and time of the response |
| Last-Modified | The last modification time of the requested resource |
| Location | Used to redirect the recipient to the location of the non-requested URL to complete the request or to identify a new resource |
| Pragma | This includes implementing specific instructions that can be applied to any recipient on the response chain |
| Proxy-Authenticate | It indicates the authentication scheme and the parameters that can be applied to the URL of the broker |
| Retry-After | If the entity is temporarily unavailable, notify the client to try again after the specified time |
| Server | Name of the Web server software |
| Set-Cookie | Set the Http cookies |
| Trailer | Indicates that the header field exists at the end of the block transfer code |
| Transfer-Encoding | File transfer coding |
| vary | Tell the downstream proxy whether to use a cached response or request from the original server |
| Via | Tell the proxy client where the response is sent |
| Warning | Alerts entities to possible problems |
| WWW-Authenticate | Indicates the authorization scheme that the client requesting entity should use |
| refresh | Applied to redirects or a new resource is created, redirects after 5 seconds (proposed by Netscape and supported by most browsers) |
Generic header fields can be used in both request and reply messages.
HTTP generic header fields
| Header | explain |
|---|---|
| Cache-Control | Cache-control specifies the caching mechanism to be followed by requests and responses, and can have a number of values attached. |
| Connection | Indicates whether a persistent connection is required |
| Date | Indicates the time when the message is sent |
| Pragma | The Pragma header field is used to contain implementation-specific instructions. The most common is Pragma:no-cache, which is used to define the page cache |
| Trailer | Represents which header fields exist at the end of Chunked encoded entity data transmission |
| Transfer-Encoding | The WEB server indicates how it has encoded the body of the response message (not the object in the body), such as chunked, e.g. Transfer-encoding: chunked |
| Upgrade | It can specify an entirely different protocol, such as an HTTP/1.1 client can send an HTTP/1.0 request to the server that contains an Update header with a value of “HTTP/1.1” so that the client can test whether the server also uses HTTP/1.1. |
| Via | List which proxy servers the response from the client to the OCS or vice versa passes through, and what protocol (and version) they send the request with. |
| Warning | Used to warn that cache operations or transformations applied to entity data may lack semantic transparency. |
The entity header field is required only if the request and reply messages contain entity data. The entity data in the request message is some data submitted by the browser to the Web server. For example, when the browser submits a form through POST, the browser encapsulates the data in the form in the entity data part of the request message. The physical data in the reply message is the media data, such as web pages, images, and documents, that the Web server sends to the browser. The entity header field describes some properties of the entity data. The following table
HTTP entity header fields
| Header | explain |
|---|---|
| Allow | Lists the set of methods supported by the resource identified by the request URI |
| Content-Encoding | How is entity data encoded |
| Content-Language | State the natural language used for entity data |
| Content-Length | Specifies the length of entity data |
| Content-Location | Indicates the resource location of entity data |
| Content-MD5 | Provides the MD5 value of entity data to ensure the integrity of entity data |
| Content-Range | Used to specify the insertion position of a portion of the entire entity. It also indicates the length of the entire entity. When the server returns a partial response to the client, it must describe the response coverage and the entire entity length |
| Content-Type | Used to indicate the entity’s media type to the recipient, specifying the entity media type sent to the recipient by the HEAD method, or the request media type sent by the GET method |
| Expires | Specifies the validity period of entity data |
| Last-Modified | Specifies when the content saved on the server was last revised. |
3 Analyze HTTP packets from the GET method
We take HTTP-GET data packets as an example to analyze HTTP request and response data packets of Get method.
Analysis of HTTPRequest packet
We open the packet and enter the filter condition ip.addr == 10.1.1.33 as follows
The first three packets are TCP three-way handshake, and the fourth packet is the HTTP request packet sent by the client to the server. Let’s study the analysis.
The protocol before HTTP, we’re not going to talk about it this time, but if you don’t understand it, you can look at the previous experiment, and let’s look at HTTP.
Hypertext Transfer Protocol
GET/HTTP/1.1\r\n # Request line information
Expert Info (Chat/Sequence): GET/HTTP/1.1\r\n # Expert information
GET/HTTP / 1.1 \ r \ n
Severity level: Chat
Group: Sequence
Request Method: GET
Request URI: / # Request URI
The requested Version is HTTP/1.1
Host: 10.1.1.33:8080\r\n # Indicates the requested Host
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Rv :37.0) Gecko/20100101 Firefox/37.0\r\n # browser type
Accept: text/html,application/xhtml+xml,application/xml; Q = 0.9, /; Q =0.8\r\n # Type of request
Accept-Language: zh-CN,zh; Q = 0.8, en – US; Q = 0.5, en. Q =0.3\r\n # Request language
Accept-encoding: Gzip, deflate\r\n # Request Encoding format
Connection: keep-alive\r\n # Use persistent Connection
The \ r \ n # blank lines
Full request URI: 10.1.1.33:8080/ # Request URI: 10.1.1.33:8080
HTTP request 1/8
Response in frame: 2770
The Next request is frame 2775
The client sends a GEY request to the server using HTTP/1.1 to access the server at 10.1.1.33. Enter the above information into the message format as follows
Format of the HTTP request packet of the GET method
| GET | The blank space | / | The blank space | HTTP / 1.1 | \r | \n |
|---|---|---|---|---|---|---|
| Accept | : | text/html,application/xhtml+xml,application/xml | \r | \n | ||
| … | ||||||
| Connection | : | keep-alive | \r | \n | ||
| \r | \n | |||||
| Full request URI: 10.1.1.33:8080 / |
Analyze the HTTP response package
We already know from the request packet that the response packet is frame 2770, so let’s look at that
Before HTTP, we see the TCP reassembly fragments shown in the figure below. These fragments are 2270 bytes long and are segmented at the TCP layer because they exceed the maximum data fragmentation (MSS) for TCP packets. From the following information, you can see the split packet and the packet size, for example, #2767 (247), where 2767 indicates the frame number and the size is 247 bytes.
Let’s look at the specific parts of HTTP
Hypertext Transfer Protocol
HTTP/1.1 200 OK\r\n # Response line information
Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n # Expert Info
HTTP/1.1 200 OK\r\n #HTTP response message. The response code is 200
Severity level: Chat
Group: Sequence
Request Version: HTTP/1.1
Status Code: 200 #
Response Phrase: OK #
Content-type: text/ HTML \r\n # The Content Type of the response
Content-length: 2023\r\n # Packet Length
Content length: 2023
Accept-ranges: bytes\r\n # Requests supported by the server: bytes
Server: HFS 2.3 Beta \r\n # Server type
The Set – cookies: HFS_SID = 0.248448607278988; path=/; \r\n # Set Http Cookie
Cache-control: no-cache, no-store, must-revalidate, max-age=-1\r\n # Cache Control
Content-encoding: Compressed format for gzip\r\n # entity data
The \ r \ n # blank lines
HTTP response 1/8 #HTTP response
Time since Request: 0.015248000 seconds
Request in frame: 2763 # Request in frame: 2763 #
Next request in frame: 2775
Next response in frame: 2778 # Next response in frame: 2778
Content-encoded entity Body (gzip): 2023 bytes -> 4375 bytes #
Line-based text data: text/ HTML # Line-based text data
Based on the above information, you can see that the server responded to the client’s request with HTTP/1.1 200 OK. Enter the information into the message format as follows
Format of the HTTP response packet of the GET method
| HTTP / 1.1 | The blank space | 200 | The blank space | OK | \r | \n |
|---|---|---|---|---|---|---|
| Content-Type | : | text/html | \r | \n | ||
| … | ||||||
| Content-Encoding | : | gzip | \r | \n | ||
| \r | \n | |||||
| omit |
4 Analysis of theHTTP packets of the POST method
Analysis of HTTPRequest packet
The following uses HTTP-POST as an example to analyze HTTP requests and responses of the Post method. Open the packet and enter the filter condition ip.addr ==10.1.1.33. In the displayed HTTP, the Info column also contains POST, as shown below
So let’s analyze it
Hypertext Transfer Protocol #HTTP
POST/HFS2_3B287 / HTTP/1.1\r\n # Request line
Expert Info (Chat/Sequence): POST/HFS2_3B287 / HTTP/1.1\r\n # Expert Info
POST/hfs2_3b287 / HTTP / 1.1 \ r \ n
Severity level: Chat
Group: Sequence
Request Method: POST
Request URI: / hfS2_3B287 / # Request URI
Request Version: HTTP/1.1 # Request Version
Host: 10.1.1.33:8080\r\n # Specifies the Host used
User-agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Rv :37.0) Gecko/20100101 Firefox/37.0\r\n # Browser type used
Accept: text/html,application/xhtml+xml,application/xml; Q = 0.9, /; Q =0.8\r\n # The type accepted by the browser
Accept-Language: zh-CN,zh; Q = 0.8, en – US; Q = 0.5, en. Q =0.3\r\n # Desired language
Accept-encoding: gzip, deflate\r\n # Available Encoding formats, here are gzip and deflate
Referer: 10.1.1.33:8080/ hfS2_3B287 /\r\n # Initiate a request from the included URL page
Cookie: HFS_SID=0.248448607278988\r\n #Cookie message
Cookie pair: HFS_SID = 0.248448607278988
Authorization: Basic ZGVtbzpkZW1v\ R \ N # Authorization certificate information
Credentials: demo:demo # User name and password for logging in
Connection: keep-alive\r\n # Use persistent Connection
Content-Type:multipart/form-data; A boundary = — — — — — — — — — — — — — — — — — — — — — — — — — — — 54542580413055 \ r \ n # request content type
Content-length: 367\r\n # Packet Length
Content length: 367
The \ r \ n # blank lines
Full request URI: 10.1.1.33:8080/ hfS2_3B287 / # Request URI: 10.1.1.33:8080/ HFS2_3B287
HTTP request 1/6
Response in Frame: 3800 # Frame number of the Response
Next request in frame: 3802 #
This is the HTTP request package using POST method, you can see the requested connection and login user name and password, etc. Enter the above information into the message format as follows
HTTP request format of the POST method
| POST | The blank space | /hfs2_3b287/ | The blank space | HTTP / 1.1 | \r | \n |
|---|---|---|---|---|---|---|
| Accept | : | text/html,application/xhtml+xml,application/xml | \r | \n | ||
| … | ||||||
| Content-Length | : | 367 | \r | \n | ||
| \r | \n | |||||
| ignore |
Also, under HTTP, we see the following
Type Multipart/form-data is a way of uploading files. Multipart/form-data is the browser’s way of uploading files using forms. The most common scenario is when you write a message and you add an attachment to the end of the message. Attachments are usually added using a form, which is uploaded to the server in multipart/form-data format. We uploaded a file to the server in our experiment, so this is the type.
See the use in Wireshark
The following table shows the mapping between the fields in the Wireshark and Multipart/form-data: MIME Multipart Media Encapsulation: indicates data in the Multipart/form-data uploaded file.
Encapsulated MultiPart Part: Represents data from different parts of the form.
Boundary: Used to separate data from different parts of a form.
Second,
1) MIME Multipart Media Encapsulation, Type: multipart/form-data, Boundary: “—————————54542580413055”
This line indicates that the request is in multipart/form-data format and boundary is the string “———-54542580413055”.
2) About Boundary: Boundary: Used to separate the data of different parts of the form. In fact, each part of the data begins with a ‘–‘ +boundary (this is the MIME standard).
3) Encapsulated multipart Part: The description of this part of data followed by the boundary is as follows:
Content-Dispostion:form-data; name=”Filename”\r\n
Each part has at least one name and one Content section.
TXT, “This is demo for HTTP POST”. This is demo for HTTP post.
Analyzing the HTTP responsepackage
Let’s look at frame 3800 based on the actual number of frames in the response packet.
Hypertext Transfer Protocol #HTTP
HTTP/1.1 200 OK\r\n # response line
Expert Info (Chat/Sequence): HTTP/1.1 200 OK\r\n # Expert Info
HTTP/1.1 200 OK\r\n # Response message
Severity level: Chat
Group: Sequence
Request Version: HTTP/1.1 # Request Version
Status Code: 200 #
Response Phrase: OK #
Content-type: text/ HTML \r\n # # The response package is similar
Content-length: 570\r\n # Indicates the Length of the response packet
Content length: 570
Accept-ranges: bytes\r\n # Requests supported by the server: bytes
Server: HFS 2.3 Beta \r\n # Web Server type
Content-encoding: Compressed format for gzip\r\n # entity data
The \ r \ n # blank lines
HTTP response 1/6 # response
Time since Request: 0.008774000 seconds
Request in frame: 3798 # Frame number of the Request
Next Request in frame: 3802 #
Next response in frame: 3804 #
Content-encoded entity Body (gzip): 570 bytes -> 866 bytes #
Line-based text data: text/ HTML # Text content
The server sent HTTP/1.1 200 OK to the client. The server type is HFS 2.3 beta, and the data is entered into the packet format
HTTP response packet format of the POST method
| HTTP / 1.1 | The blank space | 200 | The blank space | OK | \r | \n |
|---|---|---|---|---|---|---|
| Server | : | HFS 2.3 beta | \r | \n | ||
| … | ||||||
| Content-Encoding | : | gzip | \r | \n | ||
| \r | \n | |||||
| omit |
In the experiment, we explained the main GET and POST methods, which can be considered by capturing packages, learning other methods, and improving the ability by hand.
This paper deals with an experiment: HTTP protocol for Wireshark data capture and analysis