Why do I need to understand HTTPS in depth

At present, more and more websites or apps have full access to HTTPS, and APPLE officials also recommend that APPS use HTTPS. Moreover, there are no operators in China who like to hijack network requests. Without HTTPS, the user experience is really ruined. Most people know that HTTPS is http-based and has encryption, but don’t know how it works. Trust me, as a developer you will always have problems with HTTPS, and it’s important to understand the protocol at its source.

What is the basis for reading this article

An understanding of the HTTP protocol is desirable, not too thorough, but the basic concepts. It would be nice to know something about TCP/IP.

What can be gained from reading this article

This article will help you understand HTTPS from top to bottom. Too many articles tell you how the agreement does it, but not why it does it, and the quest to find out if it actually does it.

Review the importance of the TCP three-way handshake from Wireshark

For those of you who are not familiar with the TCP three-way handshake, do your own research. The reason why I’m going to review the TCP three-way handshake is because HTTP links are on top of this, and any HTTP connection requires the TCP three-way handshake, and the encryption layer underneath HTTPS actually does the same thing. In addition, we can also learn about wireshark through this review. Understanding Wireshark is crucial to understanding HTTPS. TCP three-way handshake:

At this point, those of you who believe in fundamentals will remember what happened. I won’t explain too much, but go to wireshark. We will enter an arbitrary HTTP address (be careful not to visit HTTPS, there are some websites like BAT, if you type HTTP, it will be directly converted to HTTPS, so we will choose a random small website to make sure it is HTTP).

And then you can see, obviously, there are three TCP sessions, and then there’s the HTTP link.

Take a closer look at the three-way handshake:

First handshake: Host A sends packet with bit code SYN=1 and randomly generates seQ number=0 to the server. Host B knows from SYN=1 that A requires to establish online connection.

Second handshake: After receiving the request, host B confirms the online information and sends A packet with ACK number=(SEQ +1 of host A), SYN =1, ACK =1, and seQ =0 randomly

Third handshake: After receiving the packet, host A checks whether the ACK number is correct, that is, the seQ number+1 sent for the first time and the bit code ACK is 1. If yes, host A will send ack number=(SeQ +1 of host B), ACK =1. If host B receives the seQ value and ack=1, the connection is established successfully.

After the three-way handshake is complete, host A and host B begin to transmit data.

Some people might say, well, seq number isn’t a random number, and you have zeros all the time.

Click on the details:

It tells you relative numbers so you can read it. You can certainly look at the actual numbers to understand more

Check the relative number in this box to see the real SEQ number

How here is not suddenly open, of course, understand good, usually the best or tick this option so understand a little faster.

Symmetric and asymmetric encryption

This is also an important basis for understanding HTTPS. There are a lot of complicated algorithms involved here, and I’m not proficient in algorithms and cryptography, so I won’t go into too much detail here. You just have to get the general idea:

Symmetric encryption: encryption and decryption with a key, a bit is fast. The downside? Well, the obvious one is that both parties have to give each other the key to use this communication, but once the key is intercepted, it can be decode in plain text at will. It’s actually not safe enough.

Asymmetric encryption: there is a pair of keys, both public and private. The public key is distributed freely, the private key is not distributed separately. If you encrypt with one, you have to decrypt with the other. For example, if you ask the bank for a public key, the bank sends you the public key, you encrypt a message with the public key and send it to the bank, and the bank decrypts the message with the private key. This process even if someone gets the public key, but because asymmetric encryption can only be decrypted with another encryption, so it is useless to get the public key, because you can’t use the public key decryption, the private key that can be decrypted is still in the bank, the bank of course will not be transmitted, so asymmetric encryption is very safe. But this kind of asymmetric encryption is very resource-intensive and extremely slow, so use it sparingly. Do not use it indiscriminately.

HASH encryption: This is like MD5 encryption, which is irreversible.

Three principles to ensure secure communication

A. Data content encryption

That makes sense, right? Sensitive information must be encrypted, so clear text is suicide. It’s too much to explain.

B. Identity verification of communication parties

This is a lot of people don’t understand, this is what it means, logically speaking we should use asymmetric encryption should be perfect ah. But keep in mind that our packets don’t go straight from A to B. The middle has to go through countless times of router forwarding and so on. Once someone in this middle intercepts our packets and replaces them with their own packets, it is very dangerous. So we also need a mechanism to verify the identity of both parties. Make sure I’m talking to my wife and not my mother-in-law.

C. Integrity of data content

This is not easy to understand, according to the reason that TCP is able to ensure that the data is orderly and complete to the other party. But don’t forget we passed between many times a router forwarding, might have been hijacked, may after hijacked to tamper with data packets, this time we need a mechanism to protect our data has not been tampered with, can we detect even been tampered with, make sure that I wrote to my wife’s letter to complete for my wife to see, not only see half.

Design idea of HTTPS

According to the encryption algorithm and the three principles of secure communication described above, in order to ensure the security of our communication, we can imagine a process to ensure the security of communication:

  • The server generates a public key for each client and sends the public key to the client
  • The client selects an encryption algorithm, encrypts it with the public key and sends it to the server
  • The server receives the encrypted algorithm with the public key and decrypts it with its own private key, and then knows which encryption algorithm it is. I’ll be using this algorithm for the rest of my life.

So far, this idea is not perfect. The public key is useless even if it is intercepted by an intermediary, because it cannot be decrypted to determine which algorithm is used. But there is a major drawback:

Middlemen can swap the public key packet sent by the server. How can the client know whether the public key is sent by the real server or an illegal public key given by a fake middleman?

So if you look at this graph, this is basically what a man-in-the-middle attack is.

Resolve the man-in-the-middle attack problem

The solution to this problem is also crude:

  • Use an authoritative third-party authority (CA) to issue certificates to secure servers. To prove that this server is legitimate.
  • The server uses this certificate to encrypt its public key and send it to the client
  • After receiving the encrypted public key, the client uses the third-party public key to decrypt the encrypted public key returned by the server to obtain the real server public key

Problems:

Where does the client obtain the third-party public key?

  • Your operating system or browser itself carries a third-party public key from an authority

  • What if the middleman is CA certified? This is almost impossible to handle, and if it happens, all certificates under the CA are considered illegal. So CA audit is also very strict ah

CA certificate is charged ah, I do not want to pay how to do

You can make your own certificate and put the public key of the certificate in the client (for example, in the app installation directory). In this way, the APP can decrypt the certificate using its own public key instead of using the system’s. But the question is, what if someone gets your public key certificate? The digital signature authentication algorithm can ensure such problems. In fact, simply speaking, the server and the client can agree on an encryption rule in advance, so that they can know whether it is tampered. This part is not the key point, temporarily not too detailed, as long as you know there is such a thing. In fact, once you understand the whole HTTPS thing, this place will automatically make sense.

With this foundation in hand, we can finally understand the real flow of HTTPS in the way that Koban does

I’ll leave the HTTPS process at the end, because if I do it up front, I won’t understand why I did it, and I’ll soon forget it. Look at the process again with the previous foundation and you will see the light.

  • If you look at the blue part, you can see that this is a TCP connection. So HTTPS’s encryption layer is also on top of TCP.

  • The client first initiates a clientHello message. Contains a random1 number randomly generated by the client, encryption algorithms supported by the client, and SSL information.

  • After receiving the client clientHello message, the server takes out the client method sent random1 number, and takes out the client method sent to support the encryption algorithm, and then selects a encryption algorithm, and generates a random number random2, sent to the client serverHello

  • After the client authenticates the server, the server sends its public key to the client through a digital certificate

  • After receiving the certificate from the server, the client first verifies the validity of the certificate from CA. After passing the verification, it takes out the server public Key in the certificate and regenerates a random number Random3, and then generates PreMaster Key with the server public Key asymmetric encryption Random3. And send the PreMaster Key to the server, the server decrypts the PreMaster Key through the private Key to obtain Random3, at this time the client and server hold three random numbers Random1 Random2 Random3, both sides through the three random book to generate a symmetric encryption Key. The two parties use the same algorithm to generate a key based on the three random numbers. The data transmitted at the application layer is encrypted using this key.

  • Change Cipher Spec: tells the client to use this key for future communication.

Finally, asymmetric encryption is too weak for all applications data to use symmetric encryption. Symmetric encryption does not affect performance. Therefore, it can be seen that the real purpose of HTTPS is to ensure that the symmetric encryption key is not cracked, replaced, or attacked by the middle man. If the above situation occurs, the HTTPS encryption layer can also be known to avoid accidents.

Finally, we use WireShark to restore the HTTPS interaction process again

Use Github as the destination address. Here’s what it looks like.

Just look at tlSV1 and this is the encryption layer.

Let’s analyze it step by step

  • ClientHello

  • severHello

Notice that the server and the client have 2 random numbers. And the encryption algorithm has been determined.

  • severHelloDone

Serverhellodone means that all the work of the server is done.

  • Let’s look at step 4, what does the client –> server do

So you can see that there are three steps, so what do we do in each of these steps

Client Key Exchange

After the server receives the random3 encrypted information, it decrypts it with its own private key, so that the server and the client together have random 123 3 groups of random numbers, and then use these three groups of data to generate a key, which is the symmetric encryption key used in the subsequent interaction of applicationData

  • Step 5 Server-client

  • Finally, look at the sixth and final step, the transfer data layer.

New session ticket

The session ticket is the data that the server passes to the client at the last step. After receiving the encrypted data, the client can save it, so that the next time it requests HTTPS, it can send the session ticket, which can save a lot of handshake time and resource consumption. (The steps 4-5 we analyzed above are actually quite complex, especially asymmetric encryption is quite costly to the server.)

In fact, for most browsers, HTTPS connections to the same domain name, we consciously let the first HTTPS connection complete the handshake before connecting the NTH HTTPS. This saves a lot of resources because subsequent HTTPS can carry relevant information

So this ticket is actually kind of like a cookie.

During my visit to Chrome-Gitub, the browser does not use ticket technology but seession ID technology:

The sessionID function is similar to that of a ticket. However, the sessionID cannot synchronize between servers. After all, the ID is stored in the server memory, and the state machine synchronization caused by load balancing is a major problem.

conclusion

In fact, HTTPS sums up to 3 TCP handshakes and 5 TLS handshakes. Figure out what you’re doing at each step, whether you’re using symmetric or asymmetric encryption, and the whole process will be clear for sure. In the future, similar problems can be quickly positioned and optimized. Even fetching BAT’s HTTPS interface data is not difficult. If you need to leave a message below, I will teach you as soon as possible how to capture BAT HTTPS interface data and decode out plaintext.