Hey guys, this is the 13th article in the Programmer CXUAN Computer Network serialization series.

Now that we’ve covered the application, transport, network, and data link layers, it’s time to string them all together for a comprehensive review. So I’m going to use the Web page request process as an example to talk to you about how these protocols work in a computer network, how packets are sent and received, from entering the URL, to hitting the car, to finally getting the page in front of you.

First, I opened the Web Browser and typed maps.google.com into the Google Browser URL bar.

Then…

Lookup DNS cache

The browser checks for caches in four places at this stage. The first place is the browser cache, which is DNS records.

Browsers maintain DNS records for websites you visit for a fixed period of time. Therefore, it is the first place to run DNS queries. The browser first checks to see if the url has a corresponding DNS record in the browser to find the IP address of the target url.

I am a Chrome browser, so on MAC, you cannot use Chrome ://net-internals/# DNS to find the corresponding IP address, it can be found on Windows.

So how does a MAC query DNS records? You can use the nslookup command to look it up, but that’s not the point.

The Domain Name System (DNS) is a distributed database that maintains the mapping between urls and IP addresses. In the Internet, IP address is a kind of address that computers can understand, and DNS alias address is the address that humans can understand and remember. DNS is responsible for mapping the address that humans can remember into the address that computers can understand, and each URL has a unique IP address for corresponding.

For example, the official website of Google is www.google.com, and the IP address of Google is 216.58.200.228. You can access either address by typing in the URL, but the IP address is hard to remember, while google.com is simple and clear. DNS is equivalent to the family phone book we used a few years ago. For example, if you want to call CXuan, you may not remember cXuan’s phone number. In this case, you need to consult the phone book to find cXuan’s phone number.

The second area of the browser to check is the operating system cache. If the DNS record is not in the browser cache, the browser will make a system call to the operating system, which is getHostName under Windows.

On Linux and most UNIX systems, the operating system may not have DNS caching unless NSCD is installed.

NSCD is a name service caching program on Linux systems.

The third area to check is the router cache. If DNS records are not on your computer, the browser maintains DNS records with the router it is connected to.

If the router to which it is connected also has no DNS record, the browser checks whether there is a cache in the ISP. ISP cache is the cache of your local communication service provider, because THE ISP maintains its own DNS server. The essence of CACHING DNS records is to reduce the request time and achieve the effect of fast response. Once you’ve visited certain sites, your ISP may cache those pages for a quick visit next time. Are you shocked by how often you watch small movies? It would be nice to have an Internet-connected camera installed in your home.

You may be confused as to why the browser needs to check so many caches in the first place, and you may feel uncomfortable as caches may reveal our privacy, but these caches are essential for regulating network traffic and reducing data transfer times.

So, the above query process involving DNS cache is as follows.

If DNS records do not exist in any of the above four steps, then the DNS cache does not exist, and a DNS query needs to be initiated to find the IP address of the target url (in this case, maps.google.com).

Initiating DNS Query

As mentioned above, if I want to establish a connection and communicate with maps.google.com, I need to know the IP address of maps.google.com. Due to the design of DNS, the local DNS may not provide me with the correct IP address. It then needs to search multiple DNS servers across the Internet to find the correct IP address for the site.

Here is a question, why do I need to search multiple DNS servers to find the IP address of the website? Wouldn’t one server work?

The DNS is a distributed domain name server (DNS). Each DNS server only maintains a partial mapping between IP addresses and network addresses. No DNS server can maintain all mappings.

In the early design of DNS, there was only one DNS server. This server will contain all DNS mappings. This is a centralized design, which is not appropriate for today’s Internet, which has a large and growing number of hosts, and this centralized design has several problems

  • A single point of failureIf the DNS server crashes, the entire network collapses.
  • Traaffic Volume, a single DNS server has to handle all DNS queries, which can be in the millions or millions, and it is difficult for a single server to satisfy.
  • Remote Centralized databaseA single DNS server is not possiblenearbyFor all users, assuming that a DNS server in the United States cannot be close enough to be used by an Australian query, query requests will inevitably pass through slow and congested links, causing significant delays.
  • Maintenance (maintenance)Maintenance costs are high and frequent updates are required.

Therefore, it is impossible for DNS to be centrally designed in today’s network, because it has no extensible capability at all, so it adopts a distributed design, which has the following characteristics

Distributed, hierarchical databases.

First of all, the distributed design first solves the problem of DNS server scalability, so DNS uses a large number of DNS servers, their organization mode is generally hierarchical, and distributed in the world. No DNS server can have a mapping of all the hosts on the Internet. Instead, these mappings are distributed across all DNS servers.

There are roughly three types of DNS server: root DNS server, top-level Domain (TLD) DNS server, and authoritative DNS server. The hierarchical model for these servers is shown below

  • Root DNS Server, there are more than 400 root DNS servers around the world, managed by 13 different organizations. A list of root DNS servers and organizations can be found atroot-servers.org/The root DNS server provides the IP address of the TLD server.
  • Top-level domain DNS serverThere are TLD servers or server clusters for each top-level domain such as com, org, net, edu, and gov and for all national domains such as UK, FR, CA, and JP. For a list of all top-level domains, seetld-list.com/. The TDL server provides the IP address of the authoritative DNS server.
  • Authoritative DNS ServerThe organization of publicly accessible hosts on the Internet, such as Web servers and mail servers, must provide accessible DNS records that map the names of these hosts to IP addresses. An organization’s authoritative DNS server houses these DNS records.

After understanding the design concept of DNS server, we return to the steps of DNS search, DNS query methods are mainly divided into three kinds

There are three types of queries that occur in DNS lookups. By combining these queries, the optimized DNS resolution process can shorten the transfer distance. Ideally, cached record data could be used to enable the DNS domain name server to use non-recursive queries directly.

  • Recursive query: In a recursive query, a DNS client requests that the DNS server (typically a DNS recursive parser) respond to the client with the requested resource record, or return an error message if the parser cannot find the record.

  • Iterative query: In iterative query, if the DNS server queried does not match the query name, the DNS server with authority for the lower level domain name space will be returned. The DNS client will then query the reference address. This process continues to use the other DNS servers in the query chain until an error occurs or a timeout occurs.

  • Non-recursive query: This query is usually performed when a DNS resolver client queries a DNS server to obtain a record that it is authorized to access because it has authority over the record or because the record exists in its cache. The DNS server caches DNS records and returns the cached results to prevent bandwidth consumption and load on the upstream server.

The medium above responsible for initiating a DNS lookup is the DNS resolver, which is typically a DNS server maintained by an ISP and whose primary responsibility is to ask other DNS servers on the network for the correct IP address.

If you want to learn more about DNS, check out the swastika DNS protocol!

So for the domain name maps.google.com, if the ISP maintains a server with no DNS cache, it will query the DNS root server address, which will redirect it to the.com top-level domain name server. The.com top-level domain server redirects it to the Google.com authoritative server. The Google.com name server will find the IP address maps.google.com matches in its DNS records and return it to your DNS parser, which then sends it back to your browser.

It is worth noting that DNS query packets pass through many routers and devices before reaching servers such as the root domain name. Each device or router uses a routing table to determine which path is the fastest choice for packets to reach their destination. It involves routing algorithms, if friends want to know the routing algorithms, can take a look at this article www.cisco.com/c/en/us/sup…

ARP request

I’ve read a lot of articles that don’t mention this, and that’s the ARP request process.

When do YOU need to send ARP requests?

There’s actually a prerequisite here

  • If the DNS server and the host reside on the same subnet, the system performs ARP query for the DNS server according to the following ARP procedure
  • If the DNS server and our host are on different subnets, the system queries the default gateway according to the following ARP procedure

The Address Resolution Protocol(ARP) is used to map IP addresses to MAC addresses, that is, to query the MAC Address of the target IP Address.

In short, ARP is a protocol for solving the address problem. It uses the IP address as a clue to locate the MAC address of the next host that should receive the data subcontract. If the destination host is not on the same link, the system searches for the MAC address of the next-hop router.

About why the IP address and a MAC address overview you can see it on zhihu answer www.zhihu.com/question/21…

The working process of ARP is as follows

Assume that host A and HOST B are on the same link, and host A sends an IP packet to host B. The IP address of host A is 192.168.1.2, and that of host B is 192.168.1.3. Neither of them knows the MAC address of the other. Host C and host D are other hosts on the same link.

If host A wants to obtain the MAC address of host B, host A broadcasts an ARP request packet to all hosts on the Ethernet. The ARP request packet contains the MAC address of host B’s IP address that host A wants to know.

The ARP request packet sent by host A is received and resolved by all hosts or routers on the same link. Each host or router checks the information in the ARP request packet. If the destination IP address in the ARP request packet is the same as its own IP address, the host writes its MAC address into the response packet and returns host A

In this way, you can obtain a MAC address from an IP address through ARP to achieve communication on the same link.

Therefore, to send ARP broadcasts, we need to have a destination IP address and know the MAC address of the interface used for sending ARP broadcasts.

The concept of ARP caching is covered here.

Now you know that you can determine the MAC address by sending an ARP request before sending an IP packet. Does it have to broadcast -> encapsulate ARP response -> send back to the host every time?

Think about it, how does the browser do that? The browser has a built-in cache for the addresses you’ve been using most recently, so ARP is the same. The key to efficient ARP is to maintain an ARP cache (or table) on each host and router. This cache maintains the mapping between each IP address and MAC address. The MAC address obtained by ARP for the first time is mapped to an ARP cache table. The next time a packet is sent to this address, the ARP request does not need to be sent again. Instead, the MAC address in the cache table is used to send the packet. Each TIME an ARP request is sent, the mapping in the cache table is cleared.

The ARP cache reduces the use of network traffic and prevents a large number of ARP broadcasts to a certain extent.

Generally speaking, once an ARP request is sent, the same request is likely to be sent again. Therefore, THE ARP cache reduces the number of ARP packets to be sent. In addition, the sender of an ARP request can cache the MAC address of the ARP receiver. The receiver can also cache the IP and MAC addresses of ARP requesters, as shown below

However, the MAC address cache has a certain period of time. After this period, the contents of the cache will be cleared.

For an in-depth understanding of ARP, you can refer to this article by Cxuan.

ARP, the man behind the net

So, the browser queries the ARP cache first, and if the cache hits, we return the result: destination IP = MAC.

If the cache is not hit:

  • Check the routing table to see if the destination IP address is in a subnet in the local routing table. If yes, use the interface connected to that subnet, otherwise use the interface connected to the default gateway.
  • Query the MAC address of the selected network interface
  • We send a data link layer ARP request:

The hardware connected to the host and router can be classified into the following types:

Direct:

  • If we’re directly connected to the router, the router will return oneARP Reply(See below).

Hub:

  • If we connect to a hub, the hub broadcasts ARP requests to all other ports, and if the router is connected to it, it returns oneARP Reply 。

Switch:

  • If we are connected to a switch, the switch checks the local CAM/MAC table to see which port has the MAC address we are looking for. If not, the switch broadcasts the ARP request to all other ports.
  • If there is an entry in the switch’s MAC/CAM table, the switch will send an ARP request to the port that has the MAC address we want to query
  • If the router alsoThe connectionAnd within that, it will return oneARP Reply

ARP Reply:

Now that we have the IP address of the DNS server or default gateway, we can proceed with the DNS request:

  • Port 53 is used to send UDP request packets to the DNS server. If the response packets are too large, TCP is used
  • If the local /ISP DNS server does not find the result, it sends a recursive query request to the upper-level DNS server layer by layer until it finds the origin authority, and returns the result if it finds it.

(All from: github.com/skyline7548…

Encapsulating TCP packets

After obtaining the IP address of the destination server, the browser can obtain the port number (the default HTTP port number is 80, and the default HTTPS port number is 443) based on the port number in the URL, and prepare TCP packets. Packet encapsulation goes through the following layers of processing. After the data reaches the target host, the target host parses the packet. The complete request and parsing process is as follows.

There will be no more detailed introduction here. Readers can read cXuan’s article about basic KNOWLEDGE of TCP/IP in detail.

The browser establishes a TCP connection with the destination server

After going through the DNS and ARP lookups, the browser receives the IP and MAC addresses of the target server, and the browser establishes a connection to the target server to transmit the information. There are many Internet protocols that can be used here, but HTTP is the transport layer protocol used to establish the connection, which is TCP. So this step is the process by which the browser establishes a TCP connection with the target server.

A TCP connection is established through a TCP/IP three-way handshake. The three-way handshake is actually the exchange of SYN synchronization and ACK messages between the browser and the server.

Assume that the left end is the client host and the right end is the server host. Initially, both ends are in the CLOSED state.

  1. The server process is ready to receive TCP connections from outside. Then the server process is inLISTENStatus waiting for a client connection request.
  2. The client sends a connection request to the server with SYN = 1 at the beginning and an initial sequence number (seq = X). The SYN packet segment does not carry data and consumes only one sequence number. At this point, the client entersSYN-SENDState.
  3. After receiving the connection from the client, the server needs to confirm the packet segment of the client. Set both the SYN and ACK bits to 1 in the acknowledgment packet segment. The confirmation number is ACK = x + 1 and the initial number is seq = y. Note that this message segment also does not carry data, but also consumes a sequence number. At this point, the TCP server entersSyn-received (Synchronously RECEIVED)State.
  4. After receiving the response from the server, the client also needs to confirm the connection. Confirm that the ACK in the connection is set to 1, the sequence number is seq = X + 1, and the confirmation number is ACK = Y + 1. According to TCP, this packet segment may or may not carry data. If no data is carried, the sequence number of the next data packet segment is still seq = X + 1. At this point, the client entersESTABLISHEDstate
  5. After receiving the customer’s confirmation, the server also logs inESTABLISHEDState.

This completes the connection phase of the three-way handshake, and the two parties can communicate directly.

The browser sends an HTTP request to the Web server

Once the TCP connection is established, it’s time to transfer data directly and get down to business! The browser sends a GET request asking the target server to provide maps.google.com. If you fill in a form, you make a POST request. GET and POST are the two most common types of HTTP requests. It accounts for more than 90% of all HTTP requests.

In addition to the request type, HTTP requests also contain a lot of information, the most common are Host, Connection, user-Agent, accept-language, etc

First, Host represents the Host of the object. Connection: close indicates that the browser needs to tell the server that a non-persistent Connection is being used. It requires the server to close the connection after sending the responding object. User-agent: This is the request header used to tell the Web server that the browser is of type Mozilla/5.0, or Firefox. Accept-language tells the Web server that the browser wants the French version of the object if the server needs to support French types, or it will send the default version of the server. Here we introduce the main entity fields (see developer.mozilla.org/zh-CN/docs/… MDN official website

There are four types of HTTP request headers: generic headers, request headers, response headers, and entity headers.

There is a lot to be said for each of these four headers. If you want to learn more about HTTP headers, you can refer to this article on CXUAN

Understand HTTP headers in depth

The server processes the request and sends back a response

This server contains a Web server, the Apache server, that receives the request from the browser and passes it to the request handler and generates the response.

Request handler is also a program, it is generally written in.NET, PHP, Ruby and other languages, used to read the request, check the request content, cookies, if necessary update the information on the server such a program. It combines responses in specific formats such as JSON, XML, and HTML.

The server sends back an HTTP response

The server response contains the requested page and status code, content-encoding, how to Cache the page (cache-Control), cookies to set, privacy information, etc.

For example, here is a response body

For an in-depth understanding of HTTP requests and responses, see this article

By the end of this HTTP post, you’ll be fine with bickering with your interviewer

The browser displays HTML related content

The browser displays HTML content in stages. First, it renders the bare HTML skeleton. It then checks HTML tags and sends GET requests to GET other elements on the page, such as images, CSS stylesheets, JavaScript files, and so on. These static files are cached by the browser, so you don’t have to request the page again. Finally, you will see the maps.google.com display appear in your browser.

I have uploaded six PDFS by myself. After searching the public account of “Programmer Cxuan” on wechat, I reply to CXuan on the background and get all PDFS. These PDFS are as follows

Six PDF links