Accidentally drew 24 map analysis network application layer protocol!

The overall context of the article is as follows

Before with after the introduction of two articles, believe the reader to have a preliminary understanding of computer network, the following we are going to classify different protocol layer is introduced, we still adopt top-down way of introduction, the introduction of readers are more likely to accept, better absorption degree (ie, it is easier to give my thumb up, escape).

Under normal circumstances, users do not care too much about the network application program is actually according to what mechanism, but we are programmers, just use zhu Wei’s words to say: you think computer network programmers do not understand, you point to Internet users to understand? Does it smell inside?

The application layer refers to layers 5, 6, and 7 of the OSI Standard model, namely the session layer, presentation layer, and application layer.

We will use the OSI standard model when we introduce it, because it covers more layers and gives you a better understanding of the TCP/IP model.

Application Layer concept

Definition of application layer protocols

Nowadays, more and more applications use the network to communicate, such as Web browsers, remote login, E-mail, file transfer, file download, etc. The application layer protocol is the rules and standards for these activities.

The Application Layer Protocol defines how application processes on different end systems pass messages to each other. In general, the following is defined

• Type of packets exchanged: Request packets or corresponding packets
• Description of packet fields: Detailed description of each field in a packet
• Semantics of packet fields: What are the meanings of each field in a packet
• When and how does the process send messages and respond

Application layer architecture

Application Architecture refers to the structure of the Application layer. Generally speaking, there are two main architectures at the Application layer

• Client-server Architecture
• Peer-to-peer Architecture

Let’s start with the concept of client-server architecture

In client-server architecture, there is a host that is always open called a Server, which provides services from clients. Our most common server is a Web server, which serves requests from browsers.

When the Web server receives the user request through the browser, it will go through a series of processing to present the information or pages to the application through the browser. This pattern is the client-server pattern.

There are two caveats

• In client-server mode, customers usually do not communicate with each other.
• Servers usually have fixed, well-known IP addresses that provide access.

The client-server pattern typically occurs when the number of customers increases so rapidly that a single server cannot complete a large number of customer requests. To do this, you typically need a data center with a large number of hosts to keep track of all user requests.

, on the contrary, P2P is peer-to-peer system structure on the dependence of the data center is very low, because in the P2P architecture, application in direct communication between two hosts, these hosts are called peer, with a central server in the middle of the network system is different, every client is a node of peer-to-peer networks, also has the function of the server. Common APPLICATIONS of P2P architecture include file sharing, video conferencing, voip and so on.

One of the most important features of P2P networks is the scalability, because the scalability is an important goal of P2P networks: all clients can provide and obtain resources, share bandwidth, and storage space. Therefore, as more nodes join and more requests are made to the system, the capacity of the entire system increases. This is what a client-server architecture with a fixed set of servers does not have, and this is the scalability of P2P.

Process of communication

We talked about two architectures, client-server and PEER-to-peer. We all know that a computer allows multiple applications to run at the same time. These applications appear to us to be running at the same time, but how do they communicate?

In operating system terms, communicating is actually a process, not a program. A process can be thought of as a program running on an end system. When multiple processes are running on the same end system, they communicate with each other using an interprocess communication mechanism. The rules for communication between processes are determined by the operating system.

The interface between a process and a computer network

The computer is huge and multifarious, so is the computer network, the application program can not be composed of only one process, it is also a number of processes work together to negotiate the operation, however, distributed in the process between multiple end systems is how to communicate? In fact, there is a software interface between each process called a socket. A socket is the internal interface of an application program through which an application program can send or receive data and open, read, write, and close it like a file.

Through an instance to simple analogy the socket and the network process: process can analogy a house, and the sockets rather then the door of the house, when a process wants to communicate with other processes, it will push the message out, then a message transport to another through transport equipment to the house, through the door into the house for internal use.

The following diagram is a flow chart for communicating over sockets

As can be seen from the figure, Socket belongs to the internal interface of the host or server process and is controlled by the application developer. The communication between the two end systems is transmitted to the TCP buffer of the other end system through the network. Socket reads packets from the TCP buffer for internal use of the application program.

A socket is a programmable Interface for building network applications, so it is also called an Application Programming Interface (API) between an Application and a network. The application developer can control the internal details of the socket, but not the transport layer. The application developer can only choose the transport layer transport protocol and transport layer transport parameters, such as maximum cache and maximum message length.

Addressing the process

We mentioned above that web applications send messages to each other, so how do you know where you should send messages to? Is there some mechanism that lets you know where you can send? It’s like if you want to send an email, you write the content but you don’t know where to send it, so you have to have a mechanism to know the address of the other party, and this mechanism can identify the only address of the other party, which is the IP address. We’ll discuss IP addresses in more detail in a later article, but for now, IP is a 32-bit quantity that uniquely identifies the address of any host on the Internet.

Is it ok to just know the IP address? We know that a computer may run multiple network applications, so how do we determine which network application receives the incoming message? You also need to know the port number of the network application. For example, a Web application needs to be identified with port 80 and a mail server application needs to be identified with port 25.

How does an application select a transportation service

We know that applications are application-layer protocols that belong to layer 4 of the Internet, and that layer 4 protocols must help each other get the job done. Ok, so now we only have the application layer protocol, we need to send a message, how do we send a message? It’s like you know where you’re going. How do you get there? Is it walking, bus, subway or taxi?

There are many options for the application to use to send messages. We can consider data transmission reliability, throughput, timing, and security. Here are some of the things you need to consider.

• Whether data transmission is reliable

We have discussed before that packet loss exists in computer networks. The seriousness of packet loss is related to the nature of network applications. If problems occur in the process of E-mail, file transfer, remote host and Web document transfer, data loss may cause very serious consequences. In cases like online games, multiplayer video conferencing may have less impact. In view of this, the reliability of data transmission is also the first issue to be considered. Therefore, if a protocol provides such a service to ensure data delivery, it is considered to provide reliable data transfer. An application that tolerates data loss is called a Loss-tolerant application.

• throughput

In previous articles we introduced the concept of throughput, which is the rate at which the sending process can deliver bits to the receiving process during data transmission in network applications. Applications with throughput requirements are called bandwidth-sensitive applications. Bandwidth-sensitive applications have specific throughput requirements, whereas elastic applications can make more or less use of available throughput depending on the bandwidth available at the time.

• timing

What does timing mean? Timing can ensure that the two applications in the network of the transceiver will be made within the specified time, this also is the application choose transportation services need to be considered a factor, it sounds very natural, your web application to send and receive data must to be the concept of time, such as in the game, you can send a packet of data is not in the past, You’re stuck in the middle of a tower on the other side.

• security

Finally, choose a transport protocol that provides one or more security services to your application.

Transportation services that the Internet can provide

After selecting transportation services, it’s time to talk about what services the Internet can provide. In fact, the Internet provides two transport-layer protocols for applications, namely UDP and TCP. The following are some requirements for network applications to choose the appropriate transport-layer protocol.

application	Data loss	bandwidth	Time sensitive
The file transfer	Can’t lose	The elastic	Not sensitive
E-mail	Can’t lose	The elastic	Not sensitive
Web document	Can’t lose	The elastic	Not sensitive
Internet telephony/video conferencing	Tolerate loss	The elastic	Sensitive, 100 ms
Streaming storage of audio/video	Tolerate loss	The elastic	Sensitive, a few seconds
Interactive games	Tolerate loss	The elastic	Is that 100 ms
Smartphone messaging	Can’t lose	The elastic	It doesn’t matter

Let’s talk about the application scenarios of these two transportation protocols

TCP

The TCP service model has the following features

• Connection-oriented services

After application layer datagrams are sent, TCP lets the client and server exchange transport layer control information with each other. This handshake is a reminder that the client and server need to be ready to receive datagrams. After the handshake phase, a TCP Connection is established. This is a full-duplex connection, that is, both processes can simultaneously send and receive messages on this connection. After the application finishes sending the packet, the connection must be disconnected.

• Reliable data transmission

Communication processes can rely on TCP to deliver all sent data error-free and in the proper order. An application can rely on TCP to deliver the same byte stream to the receiving socket without byte loss or redundancy.

• Congestion control

TCP congestion control does not necessarily benefit the communication process directly, but it can benefit the Internet as a whole. TCP’s congestion control suppresses the sending process (client or server) when the network between the receiver and sender becomes congested, which we’ll discuss later

UDP

UDP is a lightweight transport protocol that provides minimal services. UDP is connectionless, so there is no handshake before two processes can communicate. UDP also does not guarantee whether packets are transmitted to the server, so it is like a dead end. In addition, packets arriving at the receiving process may also arrive out of order.

Below are the protocols selected by some of the applications listed in the table above

application	Application layer protocol	Support the transport agreement
E-mail	SMTP	TCP
Remote terminal access	Telnet	TCP
Web	HTTP	TCP
The file transfer	FTP	TCP
Streaming multimedia	HTTP	TCP
Internet telephone	SIP and RTP,	TCP or UDP

Application layer protocol

The following describes the important application protocols at the application layer

The WWW and HTTP

The World Wide Web (WWW) is a system that presents information on the Internet in the form of hypertext, also known as the Web. The client used to display WWW results is called a Web browser. With a browser, we don’t have to worry about which server the content we want to access is on, we just need to know the content we want to access.

The WWW defines three important concepts

• URI, which defines how and where to access information
• HTML, which defines the presentation of information
• HTTP, defines the WWW access specification

URI / URL

A Uniform Resource Identifier (URI) is a Uniform Resource Identifier (URI) that uniquely identifies resources on the Internet.

The full name of URL is Uniform Resource Locator. The Chinese name is Uniform Resource Locator, which is actually a subset of URI.

Uris include not only urls, but urNs (Uniform resource Names), and the relationship between them is as follows

Uris are no longer limited to identifying Internet resources; they can be used as identifiers for all resources.

HTML

HTML, called hypertext Markup Language (HTML), is an identifier language. It includes a series of tags. These tags unify the format of documents on the network and connect scattered Internet resources into a logical whole. HTML text is descriptive text composed of HTML commands that describe text, graphics, animations, sounds, tables, links, etc.

HTTP

The application layer Protocol of Web is HyperText Transfer Protocol (HTTP), which is the core Protocol of Web. Let’s take a look at some of the core concepts in HTTP.

The Web page

A Web Page, also known as a Web Page, is composed of objects. An object is simply a file. This file can be an HTML file, an image, a Java application, etc., all of which can be found through a URI. A Web page contains many objects. A Web page can be said to be a collection of objects.

The browser

Just as mailboxes use the email protocol SMTP, browsers are the main carriers for using HTTP. Yes, with the end of the Netscape war, browsers developed rapidly, so far there are major browsers

The Web server

The official name of the Web Server is called the Web Server. The Web Server can provide documents to Web clients such as browsers, and can also place website files for the world to browse. You can place data files for the world to download. The three most popular Web servers are Apache, Nginx and IIS.

CDN

The full name of CDN is Content Delivery Network, which applies the caching and proxy technology in HTTP protocol to respond to client requests instead of the source site. CDN is a network built on the basis of the existing network. It relies on the edge servers deployed in various places, through the central platform of load balancing, content distribution, scheduling and other functional modules, users can obtain the required content nearby, reduce network congestion, and improve user access response speed and hit ratio. The key technologies of CDN mainly include content storage and distribution.

For example, if you want to buy books on Amazon, you can only buy them through the shopping website before and then send them to your home from the United States after going through numerous customs and other checkpoints. Now, if you set up an Amazon branch base in China, you can send the books to you from China as soon as possible without sending them through the United States.

WAF

WAF is a Web Application Firewall (WAF for short). It is a product that protects Web applications by implementing a series of HTTP/HTTPS security policies. It is an application-level Firewall. Dedicated to detecting HTTP traffic, it is a security technology to protect Web applications.

WAF is usually located in front of the Web server and can prevent attacks such as SQL injection and cross-site scripting. One of the most popular open source projects is ModSecurity, which can be fully integrated into Apache or Nginx.

WebService

WebService is a Web application. WebService is a remote call technology across programming languages and operating system platforms.

WebService is an application service development specification defined by W3C. It uses client-server master-slave architecture. WSDL is usually used to define service interfaces and HTTP protocol is used to transmit XML or SOAP messages. It is a Web based (HTTP) service architecture technology that can run either on the Intranet or, with proper protection, on the extranet.

HTTP

HTTP is a convention and specification for the transfer of hypertext data, such as text, pictures, audio, and video, between two points in the computer world. HTTP is an application layer protocol. It uses TCP as the transport layer protocol because documents and data are important information that cannot be lost.

HTTP request response process

Let us through an example to explore the HTTP request and response process, we assume that the access URL for http://www.someSchool.edu/someDepartment/home.index, when we enter a URL and click enter, The following operations are performed inside the browser

• The DNS server first performs domain name mapping to find accesswww.someSchool.eduThen the HTTP client process initiates one to the server on port 80www.someSchool.edu(port 80 is the default HTTP port). There is one in both the client and server processesThe socketIt’s connected to that.
• The HTTP client sends an HTTP request packet to the server through its socket. The path is included in the textsomeDepartment/home.indexWe’ll discuss HTTP request packets in more detail later.
• The HTTP server accepts the packet through its socket, parses the request, and reads from itMemory (RAM or disk)Retrieve the object fromwww.someSchool.edu/someDepartm…HTTP response packets are sent to customers through sockets.
• The HTTP server then notifies TCP to disconnect the TCP connection. In fact, the HTTP server disconnects the TCP connection only after the client receives the response packet.
• After the HTTP client receives the response packet, the TCP connection is closed. The HTTP client extracts an HTML response file from the response, examines the HTML file, and then loops through other internal objects in the message.
• After the check is complete, the HTTP client presents the corresponding resource to the user through the display.

At this point, the whole process of typing in the url and pressing Enter is over. The above procedure describes a simple request-response process. The real request-response situation may be much more complex than the procedure described above.

HTTP Request Characteristics

From the whole process above we can conclude that HTTP packet transport has the following characteristics

• Supports client-server mode
• Simple and quick: When a client requests a service from a server, it only needs to send the request method and path. The commonly used request methods are GET, HEAD and POST. Each method specifies a different type of contact between the client and the server. Because HTTP protocol is simple, the HTTP server program size is small, so the communication speed is very fast.
• flexibleHTTP allows the transfer of data objects of any type. The type being transmitted byContent-TypeMark it.
• There is no connection: Connectionless means to limit processing to one request per connection. The server disconnects from the customer after processing the request and receiving the reply from the customer. In this way, transmission time can be saved.
• statelessHTTP is a stateless protocol. Stateless means that the protocol has no memory for transaction processing. The lack of state means that if the previous information is needed for subsequent processing, it must be retransmitted, which can result in an increase in the amount of data transferred per connection. On the other hand, the server responds faster when it doesn’t need the previous information.

Persistent and non-persistent links

The HTTP request and response process described above is a kind of non-persistent link, because TCP closes the TCP connection after transmitting packets. Each TCP connection transmits only one request packet and one response packet.

Non-persistent connections have some disadvantages.

• First, a completely new connection must be established and maintained for each requested object.
• Second, for each such connection, TCP buffers are allocated and TCP variables are held on both the client and server, placing a serious burden on the Web server. A Web server may be serving hundreds or even thousands of client requests simultaneously.

In the case of HTTP 1.1 persistent connections, the server keeps the TCP connection open after sending a response. Between the same client and server, subsequent request and response packets can be transmitted over the same connection. In general, the HTTP server should close a one-hop connection if it remains unused after a certain (configurable) interval.

HTTP Packet Format

We have described the HTTP request response process above, I believe you have a deeper understanding of HTTP, let’s take a look at HTTP message format.

The HTTP protocol mainly consists of three parts:

• Start line: Describes basic information about a request or response.
• Header fields: describes packets in more detail in key-value format.
• Entity Message: Indicates the actual data to be transmitted. It may be binary data, such as pictures and videos, rather than plain text.

The start line and Header fields become the request Header or response Header, collectively called Header. The body of the message is also called the entity, called the body. According to the HTTP protocol, each sent packet must have a Header, but the body information can be omitted. That is, the Header information is required, but the entity information can be omitted. There must also be a blank line (CRLF) between header and body. If AN HTTP request were represented in a graph, it would look something like this

If you were more specific, it would look like this

One thing to notice about this picture is that if you use GET, there’s no entity, but if you use POST, there’s entity. When a user submits a form, HTTP clients typically use the POST method; In contrast, HTML forms are typically retrieved using the GET method. The HEAD method is similar to the GET method, except that the HEAD method does not return an object.

Let’s look at HTTP response packets

You can see that the request header is the only difference between the request packet and the response packet.

Request message Request line:

GET /some/page.html HTTP / 1.1
Copy the code

Response message:

HTTP / 1.1 200 OK
Copy the code

The Cookie and Session

HTTP is a stateless protocol, that is, every time a server receives a request from a client, it is a new request, and the server does not know the historical request record of the client. The main purpose of sessions and cookies is to compensate for the stateless nature of HTTP.

What is the Session

The client requests the server, and the server allocates a memory space for the request. This object is the Session object, and the storage structure is ConcurrentHashMap. Sessions compensate for the stateless nature of HTTP by allowing the server to store records of the client’s operations during the same Session.

How does Session check whether it is the same Session

When the server receives the request for the first time, it creates a Session space (the Session object is created), generates a sessionId, and passes the ** set-cookie in the response header: JSESSIONID=XXXXXXX ** command to send a response to the client requesting to set cookies; After receiving the response, the client sets a Cookie with **JSESSIONID=XXXXXXX ** on the local client. The Cookie expires at the end of the browser session.

Next, when the client sends a request to the same website each time, the request header will carry the Cookie information (including the sessionId). Then, the server obtains the value named JSESSIONID by reading the Cookie information in the request header and obtains the sessionId of the request.

The disadvantage of the Session

The Session mechanism has A disadvantage. For example, server A stores sessions. After load balancing is performed, if the traffic volume of server A surges within A period of time, it will be forwarded to SERVER B for access, but server B does not store Session of server A, which will result in Session invalidity.

What is the Cookies

Cookie in HTTP protocol includes Web Cookie and browser Cookie, which is a small piece of data sent by the server to the Web browser. Cookies sent by the server to the browser are stored by the browser and sent to the server with the next request. Typically, it is used to determine whether two requests are coming from the same browser, for example when the user remains logged in.

HTTP Cookie mechanism is a supplement and improvement of HTTP protocol stateless

Cookies are used for three purposes

• Session management

Login, shopping cart, game score, or anything else the server should remember

• personalized

User preferences, themes, or other Settings

• tracking

Record and analyze user behavior

Cookies were once used for general client storage. While these are legal because they are the only way to store data on the client, modern storage apis are recommended today. Cookies are sent with each request, so they can degrade performance (especially for mobile data connections).

Create a Cookie

When receiving an HTTP request from a client, the server can send a set-cookie header with the response, which is usually stored by the browser, and then send the Cookie with the HTTP header to the server.

Set-cookie and Cookie headers

Set-cookie THE HTTP response header sends a Cookie from the server to the user agent. Here is an example of sending cookies

This header tells the client to store the Cookie

Now, with each new request to the server, the browser will use the Cookie header to send all previously stored cookies back to the server.

There are two types of Cookies, Session Cookies and Persistent Cookies, which are treated as Session Cookies if they do not contain an expiration date. Session cookies are stored in memory, never written to disk, and are permanently lost thereafter when the browser is closed. If a Cookie contains an expiration date, it is considered a persistent Cookie. On the expiration date specified, the Cookie is deleted from disk.

There are also Secure and HttpOnly tokens for cookies, which are described in turn

Session Cookies

The example above creates a session Cookie, which has a feature that the Cookie is deleted when the client closes because it does not specify Expires or max-age directives.

However, Web browsers may use session restore, which leaves most session cookies in a permanent state as if the browser had never been closed.

Permanent Cookies

Permanent cookies do not expire when the client closes, but expire after a specific date (Expires) or a specific length of time (max-age). For example,

Set-Cookie: id=a3fWa; Expires=Wed, 21 Oct 2015 07:28:00 GMT;
Copy the code

The Cookie debate

Although cookies can simplify users’ online activities, their use is controversial because many people consider them to be a form of infringement on users. Because of the combination of cookies and user-provided account information, Web sites can learn more about the user.

Web caching

A Web cache, also known as a proxy server, is a network entity that represents an HTTP server to meet user requirements. The Web cache has its own disk storage and holds recently requested objects in storage, as shown in the figure below

Web cache can be configured in the user’s browser, once the configuration, the user first visit, it is not the original server, need to access the proxy server to determine whether a request object, if the proxy server is not, again by the proxy server to request the original server object is returned to the client, in their own disk space to save the object at the same time.

It is important to note that the client and the initial server are built in client-server mode, whereas the proxy server can be used not only as a server but also as a client.

The proxy server is usually provided by an Internet Service Provider (ISP). Note not old color criticism… An ISP is an operator, you know.

So why do proxy servers exist? You can probably guess what it does after reading the description above.

• First, the proxy server can greatly reduce the response time to customer requests, and can give users a faster response.
• Secondly, the proxy server can reduce the traffic of an organization’s access link to the network, reduce the network bandwidth, and reduce the cost of operators.
• The proxy server can then take over the burden of the original server and improve the performance of the application.

DASH

From the above description, we know that HTTP can transfer ordinary files, audio, and video. The transmitted information is collectively called MIME type. HTTP transmits video as an object, and an object is actually a file, and a file can be represented by URL in HTTP. When the user is watching a video, the client establishes a TCP connection with the server and sends a GET request to the URL. When the server responds to the client, the client caches a certain amount of bytes of data. When the data exceeds a preset threshold, the client application starts to play the video.

One limitation of this approach is that for each client, all clients receive the same video encoding, even though each client has a different amount of bandwidth available. This results in wasted bandwidth. This is equivalent to me being a 2-megabit network and a 50-megabit fiber that can both receive the same video code and start playing video with almost the same wait time, so why would I pay for 50-megabit fiber?

In order to improve this phenomenon, DASH (Dynamic Adaptive Streaming HTTP) appears. The idea is that different networks can transmit different bits of data. DASH allows customers to play video at different encoding rates using different Internet transmission rates. For 3G users and fiber users, they naturally choose to transmit bits of data at different rates to maximize bandwidth usage.

CDN

With the increasing number of Internet users, video has gradually become the bottleneck of bit transmission and the strong demand of users. As an Internet video company, the most direct way to initially provide streaming services is to build a single large-scale data center. All videos are cached in the data center and broadcast directly from the data center to the world. But there are three problems with this approach

• If the customer is far away from the data center, then the server-to-customer branch is across many communication links and possibly through many ISPs, so how fast can your video play go?
• Each time the video data is relayed to the client, it wastes a lot of network bandwidth, and the video company pays for duplicate bandwidth
• A single point of failure occurs when a video data center breaks down or other accidents occur.

In response to being able to distribute video around the clock to users around the world, almost all major video companies use Content Distribution Networks (CDN). CDN manages servers distributed in multiple geographical locations, caches various video, audio, files, and so on on each server.

CDN content selection policy

The CDN manages servers distributed across multiple geographic locations, stores video copies on its servers, and all attempts to direct each user request to a CDN location that provides the best user experience. So how to site the server? There are actually two principles for server placement

• in-depth, its main goal is to be close to the user, and improve the delay and throughput experienced by the user by reducing the number of links and routers between the end user and the CDN cluster.
• Invite a guestThis principle invites ISPs to visit by building large clusters in a small number of (say, 10) critical locations. Invited guest design usually has lower maintenance and administration overhead than in-depth design principles.

CDN workflow

A CDN can be a private CDN(private CDN), that is, it is owned by the content provider. Another CDN is a third-party CDN(THIRD-PARTY CDN), which distributes content on behalf of multiple content providers.

Now let’s talk about the CDN workflow, as shown in the figure below

• The user wants to access the content of the specified web site

• The LDNS relays the request to the website DNS server. The WEBSITE DNS server returns to the LDNS the address of the authoritative WEBSITE CDN server

• The LDNS server sends a second request to the CDN authority to obtain the address of the CONTENT distribution server. The CDN sends the address of the CONTENT distribution server to the local DNS server

• The local DNS server sends the address of the site’s CDN content distributor to the user

• After the user knows the address of the WEB CDN content distribution server, the user directly establishes a TCP connection with the web CDN content distribution server without additional operations and sends an HTTP GET request. If DASH streams are used, blocks with different rates are selected and sent to the user according to different VERSIONS of URLS.

CDN cluster selection policy

The core of any CDN deployment is cluster Selection strategy, which is the mechanism to dynamically direct customers to a server cluster or data center in the CDN. One simple strategy is to assign customers to the closest geographically geographically. This selection strategy ignores the variation of latency and available bandwidth with Internet path time and always assigns the same cluster to specific customers. Another alternative strategy is real-time measurement, which performs periodic checks based on delay and packet loss performance between the cluster and the customer.

DNS Internet Directory Service protocol

Consider the question, how many ways can we identify ourselves? You can identify it by your ID card, you can identify it by your Social Security number, you can identify it by your driver’s license, and although we have multiple ways of identifying it, one way may be more appropriate than another in certain circumstances. Hosts on the Internet, like humans, can be identified in a variety of ways. One way to identify a host on the Internet is to use its hostname, such as www.facebook.com, www.google.com, etc. But that’s how we humans remember it, and routers don’t understand it that way. Routers like fixed-length, hierarchical IP addresses, so remember what IP is?

An IP address, in its simplest form, is a four-byte structure with a strict hierarchy. For example, an IP address such as 121.7.106.83, where each byte can be used as. Split to represent decimal numbers from 0 to 255. (The specific IP will be discussed later)

However, what routers like to parse is IP addresses, whereas what we humans can easily remember is web addresses. So how can routers parse IP addresses into familiar web addresses? That’s where DNS comes in.

The full Name of DNS is Domain Name System,DNS, it is a hierarchical DNS server (DNS server) to achieve the distributed database; It is also an application layer protocol that enables hosts to query distributed databases. A DNS server is usually a UNIX machine running BIND(Berkeley Internet Name Domain) software. The DNS protocol runs on top of UDP and uses port 53.

DNS Overview

Like HTTP, FTP, and SMTP, DNS is an application-layer protocol. DNS runs in client-server mode between communicating systems and transmits DNS packets through the following end-to-end transport protocols. But DNS is not an application that deals directly with users. DNS provides a core function for user applications and other software on the Internet.

DNS is usually used by other application-layer protocols, such as HTTP, SMTP, and FTP, which resolve the host name provided by a user to an IP address.

Here is an example to describe the DNS resolution process, which is similar to what the browser does after you type in the URL

What happened when you type in www.someschool.edu/index.html in the browser? To enable the user host to send an HTTP request packet to the Web server www.someschool.edu, perform the following operations

• DNS clients running on the same user host
• The browser extracts the hostname www.someschool.edu from the URL and passes the hostname to the client of the DNS application
• The DNS client sends a request to the DNS server containing the host name.
• The DNS client eventually receives a reply packet containing the IP address of the target host
• Once the browser receives the IP address of the target host, it can initiate a TCP connection to the HTTP server process at port 80 of that IP address.

In addition to providing IP address to host name translation, DNS provides several important services

• Host AliasingA host with a complex hostname can have one or more other aliases, such as relay1.west-coast.enterprise.com, which has both enterprise.com and relay1.west-coast.enterprise.comwww.enterprise.comThe two host aliases, in this case relay1.west-coast.enterprise.com also calledCanonical host name, and host aliases are easier to remember than canonical host names. An application can invoke DNS to get the canonical hostname corresponding to the host alias, as well as the IP address of the host.
• Mail Server AliasingSimilarly, an E-mail application can call DNS to resolve a supplied host name.
• Load distributionDNS is also used to distribute load between redundant servers. Busy sites for examplecnn.comIt is redundantly distributed on multiple servers, each running between different end systems, each with a different IP address. Because of these redundant Web servers, a collection of IP addresses is therefore associated with the same canonical host name. A DNS database stores a collection of these IP addresses. Since the client makes an HTTP request every time, DNS circulates the load among all these redundant Web servers.

DNS Overview

DNS is a complex system. We only study the main aspects of its operation here. The following is an overview of the working process of DNS

Suppose some application running on a user’s host, such as a Web browser or mail reader, needs to convert the host name to an IP address. These applications will call DNS clients and specify the hostname to be converted. The DNS on the user host sends a DNS query packet to the network through port 53 using UDP. After a period of time, the DNS on the user host receives a DNS reply packet corresponding to the host name. Thus, from the user host’s point of view, DNS is like a black box whose inner workings you cannot see. But in reality, the black box that implements DNS as a service is very complex, consisting of a large number of DNS servers distributed around the world and application layer protocols that define how DNS servers communicate with query hosts.

One of the earliest simple designs for DNS was simply to use a DNS server on the Internet. The server will contain all mappings. This is a centralized design, which is not appropriate for today’s Internet, which has a large and growing number of hosts, and this centralized design has several problems

• A single point of failureIf the DNS server crashes, the entire network collapses.
• Traaffic VolumeA single DNS server has to handle all DNS queries, which can be in the millions or millions
• Remote Centralized databaseA single DNS server is not possiblenearbyFor all users, assuming that a DNS server in the United States cannot be close enough to be used by an Australian query, query requests will inevitably pass through slow and congested links, causing significant delays.
• Maintenance (maintenance)Maintenance costs are high and frequent updates are required.

Therefore, DNS cannot be centrally designed. It has no extensible capability at all, so it adopts a distributed design. Therefore, this design has the following characteristics

Distributed, hierarchical databases

First of all, the distributed design first solves the problem of DNS server scalability, so DNS uses a large number of DNS servers, their organization mode is generally hierarchical, and distributed in the world. No DNS server can have a mapping of all the hosts on the Internet. Instead, these mappings are distributed across all DNS servers.

There are roughly three types of DNS server: root DNS server, top-level Domain (TLD) DNS server, and authoritative DNS server. The hierarchical model for these servers is shown below

Suppose a DNS client now wants to know the IP address of www.amazon.com, how does the above DNS server resolve? First, the client will associate with one of the root servers, which will return the IP address of the TLD server for the top-level domain com. The customer contacts one of these TLD servers, which returns the IP address of the authoritative server for Amazon.com. Finally, the customer contacts one of the amazom.com authoritative servers, which returns its IP address for www.amazom.com.

Let’s now discuss the domain name server hierarchy above

• Root DNS Server, there are more than 400 root DNS servers around the world, managed by 13 different organizations. A list of root DNS servers and organizations can be found atroot-servers.org/The root DNS server provides the IP address of the TLD server.
• Top-level domain DNS serverThere are TLD servers or server clusters for each top-level domain such as com, org, net, edu, and gov and for all national domains such as UK, FR, CA, and JP. For a list of all top-level domains, seetld-list.com/. The TDL server provides the IP address of the authoritative DNS server.
• Authoritative DNS ServerThe organization of publicly accessible hosts on the Internet, such as Web servers and mail servers, must provide accessible DNS records that map the names of these hosts to IP addresses. An organization’s authoritative DNS server houses these DNS records.

In addition, there is another important DNS server, which is the Local DNS server. Strictly speaking, the local DNS server does not belong to the above hierarchy, but the local DNS server is crucial. Each Internet Service Provider (ISP), such as an ISP in a residential area or an organization, has a local DNS server. When a host connects to an ISP, the ISP provides the IP address of a host, and the host has the IP address of one or more local DNS servers. By accessing network connections, users can easily determine the IP address of the DNS server. When a host sends a DNS request, the request is sent to the local DNS server, which acts as a proxy and forwards the request to the DNS server hierarchy.

DNS cache

DNS caching, sometimes called a DNS resolver cache, is a temporary database maintained by the operating system that contains recent access records for web sites and other Internet domains. That is to say, DNS cache is just a technology and means that the computer can cache the loaded resources in order to meet the fast response speed, and reference them directly when accessing them again. So how does DNS caching work?

DNS cache workflow

Before the browser makes a request to the outside world, the computer intercepts each request and looks up the domain name in the DNS cache database, which contains a list of recent domain names and the addresses THAT DNS calculated for them when it first made the request.

DNS records and packets

All DNS servers that jointly implement a DNS distributed database store Resource Records (RRS), which provide mappings from host names to IP addresses. Each DNS reply packet contains one or more resource records. RR records are used to reply to client queries.

The resource record is a 4-tuple containing the following fields

(Name, Value, Type, TTL)
Copy the code

There are different TYPES of RRS. The following is a summary table of different types of RRS

DNS RR type	explain
A record	IPv4 host record that maps domain names to IPv4 addresses
AAAA record	IPv6 host record, used to map domain names to IPv6 addresses
CNAME record	Alias record, used to map DNS domain name alias
MX records	Mail switch used to map DNS domain names to mail servers
PTR records	Pointer for reverse lookup (IP address to domain name resolution)
SRV record	SRV records for mapping available services.

DNS message

DNS has two types of packets: query packets and response packets. The two types of packets have the same format. The following is the format of DNS packets

The following describes the packet format

• The first 12 packets are header fields, that is, the header field has 12 bytes, and the first field (identifier) is a 16-bit number that identifies the query. This identifier is copied into the reply message to the query so that the customer can use it to match the sent request and the received reply. The flag field contains several flags. The flag field is 1 bit, which indicates whether the packet is a 0-query packet or a 1-response packet.

• The problem area contains information about ongoing queries. This area includes: 1) the name field contains the name of the host being queried; 2) Type field indicating the type of question being asked about the name, such as whether the host address is associated with A name (type A) or A mail server with A name (type MX).

• In the answer from the DNS server, the answer area contains the resource record for the name of the original request. The DNS RR record is a quad, and there are different types of types in the tuple. The reply area of a reply packet can contain multiple RRS. Therefore, a host name can have multiple IP addresses.

• The authority area contains records from other authority servers

• Additional areas contain other helpful records.

I will write an article about the details of specific DNS records.

P2P File Distribution

The protocols we discussed above, HTTP, SMTP, and DNS, all use client-server mode, which relies heavily on always-on infrastructure servers. P2P is a client-to-client model with minimal dependency on always-on infrastructure servers.

P2P is a computer network with a distributed architecture. In P2P systems, all computers and devices are called peers, and they exchange work with each other. Each peer in a peer-to-peer network is equal to each other. There is no privileged peer and no master administrator device on the network.

In a sense, peer-to-peer networks are the most egalitarian network in the computer world. Each peer is equal and each peer has the same rights and obligations as the other peer. A peer is both a client and a server.

Virtually every resource available in a peer-to-peer network is shared between peers without the need for any central server. Shared resources in P2P networks can be such things as processor utilization, disk storage capacity, or network bandwidth.

What is P2P used for

The main goal of P2P is to share resources and help computers and devices work together to provide specific services or perform specific tasks. As mentioned earlier, P2P is used to share various computing resources, such as network bandwidth or disk storage space. However, the most common example of peer-to-peer networking is file sharing over the Internet. Peer-to-peer networks are great for file sharing because they allow people connected to their computers to receive and send files at the same time.

BitTorrent is the main protocol used by P2P.

The role of P2P networks

P2P networks have some features that make them useful

• It is very difficult to go offline completely, and even if one of the peers goes offline, the other peers are still up and communicating. In order for P2P networks to stop working, you have to shut down all peer-to-peer networks. Peer-to-peer networks are highly scalable. Adding new peers is easy because you don’t have to do any central configuration on a central server.
• When it comes to file sharing, the larger the peer-to-peer network, the faster it is. Storing the same file on many peers in a P2P network means that when someone needs to download a file, it will be downloaded from multiple locations at the same time.

TELNET

TELNET, also known as remote login, is an application-layer protocol that enables users to control remote hosts on their local computers. For example, the picture below shows this

Host A can directly access host B through TELNET.

TELNET sends text commands to the host over a TCP connection and executes them on the host.

The following conditions must be met for remote login using TELNET

• You must know the IP address or domain name of the remote host
• You must know the login id and password

Generally, TELNET remote login uses port 23

TELNET works as follows

• The local host establishes a connection with a remote host. The connection is actually a TCP connection. The user needs to know the IP address or domain name of the specified host
• After the connection with the remote host is established, all the characters entered on the local host terminal are marked withNVT(Net Virtual Terminal)The process is actually to send a packet to the remote host.
• After receiving the data packet, the remote host sends a data packet to the local host in NVT format, including the command output and command execution result
• Finally, the local host terminal revokes the connection to the remote host, which is essentially a TCP disconnection.

SSH

TELNET has a very obvious shortcomings, that is on the host and the remote host to send data packets is clear in the process of transmission, without any security encryption, the consequences of this is likely to be Internet criminals to the packet sniffer to do bad things, for the safety of data, we usually use SSH for remote login.

SSH is an encrypted remote login system. In addition to encrypting the content of communications and being able to sniff and grab instant packets without deciphering the information contained in them, SSH has several other features

• SSH can use stronger authentication mechanisms
• SSH can forward files
• SSH supports port forwarding

Port forwarding is a method used by SSH for secure network communication. SSH uses the port forwarding technology to transmit packets of other TCP/IP protocols. In this way, SSH establishes a secure transmission channel between the client and server for other services. Port forwarding is a mechanism that forwards messages received on a specific port number to a specified IP address and port number.

FTP

File Transfer Protocol (FTP) is an application-layer Protocol. The FTP protocol consists of two parts: an FTP server and an FTP client. The FTP server is used to store files. Users can use the FTP client to access resources on the FTP server through FTP.

FTP is usually used to transfer large files over the network because of its high transmission efficiency.

By default, FTP uses TCP ports 20 and 21. 20 is used for data transmission and 21 is used for control information transmission. FTP TCP During file transfer on port 21, a TCP connection for data transfer is established each time. After the data transfer is complete, the connection for data transfer is also disconnected and the command or response process continues on the control connection.

SMTP

The Simple Mail Transfer Protocol (SMTP) is used to provide email services. SMTP also uses TCP at the transport layer.

In the early morning email, the TCP connection is directly established between the sender and receiver hosts. After composing the email, the sender saves the email to the disk, and then establishes a TCP connection with the receiving host to send the email to the disk of the receiving host. When the sender has sent the message, it is deleted from the local disk. If the receiving host fails to receive the packet due to special circumstances, the sending end will wait for a period of time and resend the packet.

While this method can ensure the integrity and validity of email, it is not suitable for today’s Internet, because in the early days of email can only be sent online, this method is obviously immature.

In view of this, the concept of mail server is proposed. Mail server constitutes the core of the whole mail system. Each recipient has a mailbox on its mail server. The mailbox of a user manages and maintains the packets sent to the user.

A typical mail delivery process starts with the sender’s user agent, travels to the sender’s mail server, then to the recipient’s mail server, where it is distributed to the recipient’s mailbox. When a user using the receiver wants to read messages from the mailbox, his mail server authenticates the user. If a message sent by a sender is not delivered correctly to the receiver’s server, the sender’s user agent stores the message in a Message queue and tries to send it again later, usually every 30 minutes. If the message fails to be sent after a certain period of time, The server deletes the message in the message queue and notifies the sender by email.

Now that you know the general process of sending mail between two mail servers, how does SMTP send mail from Alice’s mail server to Bob’s mail server? It is mainly divided into the following three stages

• Establish a connectionAt this stage, the SMTP client requests a TCP connection to port 25 of the server. Once the connection is established, the SMTP server and the client notify each other of their domain names and confirm each other’s domain names.
• Mail transfer: Once the connection is established, the mail transfer begins. SMTP relies on TCP to accurately transfer mail to the recipient’s mail server. The SMTP customer sends the source address, destination address, and content of the mail to the SMTP server. The SMTP server responds to the mail and receives the mail.
• Connection release: The SMTP client sends an exit command. The server responds to the command and closes the TCP connection.

The MIME type

In the beginning, E-mail on the Internet could only handle text format. Later, it was expanded to MIME type. MIME type (Multipurpose Internet Mail Extensions) is a type of Internet Mail extension.

It is an Internet standard that extends the E-mail standard to support many formats, such as the following

• HTML Text/HTML
• XML document.xml text/ XML
• TXT text/plain
• PNG image. PNG image/ PNG
• GIF graphics. GIF image/ GIF
• JPEG Graphics.jpeg,.jpg image/ JPEG
• AVI file. AVI Video/X-MSVideo etc.

Afterword.

This paper covers many application layer protocols, including HTTP, DNS, SMTP, FTP, TELNET and so on

We use these application layer protocols in our daily work. We are not only users, but also programmers. It is important to understand these protocols.

If the article is written well, I hope readers can like it, look at it, share it and leave a message. This will be my motivation to continue to write more articles and my motivation to increase my fans. I hope you can support it.

Reply to CXuan under my programmer’s public account of the same name to get the following PDF, which is written by myself.