preface

HTTPS is not a protocol, and generally speaking, if we say we use HTTPS, we mean we use TLS over HTTP. Therefore, we can further understand:

  1. TLS is not actually a protocol specifically designed to serve HTTP.
  2. TLS is a lower protocol in the application layer
  3. TLS can be applied to other application layer protocols, such as mail

The reason why I emphasize this kind of hierarchical concept is that the boundary and concept of knowledge need to be clearly demarcated. Only when encountering problems can we know where the breakthrough is. I think most articles about HTTPS will start with something about the handshake process, or the principles of cryptography, or some articles will not even cover the protocol book, but just the abstract handshake process. In my opinion, such understanding is not very structured to explain such a thing, so this is also my original intention to write this article.

To make it easier to understand, I’ll put the comparative structure in front and the details in back.

Macro look at the TLS

The above has been talking about TLS, for those who have not known HTTPS in advance, it may be a little confusing – what is TLS? If you know TLS, you probably know SSL.

  • TLS: Transport Layer Security protocol
  • SSL: Short for Secure Sockets Layer Protocol, which stands for Secure Sockets Layer Protocol

Literally, these two things are similar, and in fact, these two things can be thought of as the same concept. TLS is an upgraded version of SSL. In 1994, Netscape created SSL for HTTP in order to solve the security problems of HTTP. Later, SSL was unified and applied to a wider range of application layer protocols to further standardize SSL, also known as TLS.

TLS can be viewed as a layer protocol, so its relationship to HTTP is similar to:

Why TLS is needed

As mentioned above, in order to solve the security problems existing in HTTP, TLS protocol is introduced. We naturally have questions about what kind of security issues need to be addressed in the protocol.

To sum up the following three questions:

  1. Data Privacy Issues
  2. Data is tampered with
  3. Identity issues

Data privacy: HTTP is plaintext transmission, which means that if a middleman hijacks the request, it is easy to understand the meaning of the HTTP transmission. One might ask: why do we need to make a big deal out of the protocol when we can just do it with encrypted transmission? Here need some common sense of cryptography, if simply encrypt the data, we have three optional password scheme, symmetric encryption, asymmetric encryption, one-way hash or MAC algorithm, whether symmetric encryption and asymmetric encryption, which means that both sides need to have the decryption key, symmetric encryption with the same key, asymmetric need to have a relative key. For web client-oriented transmission, it is obviously unrealistic to implant the key in the client. Therefore, how to transfer the key safely becomes another problem in the alternative solution, which is also solved by TLS protocol handshake. As for irreversible encryption, such as the login password itself, for a man-in-the-middle attack, he does not need to know what your data is, which seems to make data privacy, but actually does not achieve security effect.

Data tampering: If HTTP protocol cannot prevent tampering. Despite the message captcha, digital signatures ensure data integrity. But cryptography alone is still hard to prevent man-in-the-middle attacks

Authentication issues: HTTP is a stateless protocol, so authentication of server identity is important, certificate authentication is a part of HTTPS secure transmission

So some security problems do exist in the HTTP, although for users, indeed there are many ways to prevent security problems, but through the analysis can be found, can’t use simple way, through a single a cryptography algorithm can solve these problems, in general, the combination of a variety of cryptography algorithm, it is a tedious work, So TLS is also to unify these tedious problems into a solution.

TLS Protocol Structure

TLS is a protocol between the application layer protocol and TCP protocol paper. TLS can be divided into two layers:

  1. TLS Handshaking Protocols
  2. TLS Record Protocol

The structure is shown as follows:

The handshake protocol can be divided into four sub-protocols:

Note: Before TLS 1.2, the change Cipher Spec Protocol is used instead of the heartbeat protocol.

The record layer protocol is a low-level protocol close to TCP. It is used to encapsulate the message defined by the handshake protocol and communicate with the underlying protocol.

Handshake subprotocol

The handshake protocol is arguably the most important subprotocol of TLS because it is responsible for negotiating a suitable cipher suite. As mentioned above, there is more than one encryption algorithm to solve a security problem, and a cipher suite is simply a collection of algorithms with different functions. Generally, two round-trip negotiations are required to complete.

In the handshake protocol, the unit is message, and the message format is shown as follows:

A message is broken into three parts:

  1. Message length: Three bytes used to define the final length of the message
  2. Message type: occupies one byte. There are different message types in the TLS handshake protocol, and each message type carries different data.
  3. Message content: Larger than one byte, this is the body content of the message

The TLS protocol has ten different message types, which are:

  1. hello_request: 0x00
  2. client_hello: 0x01
  3. server_hello: 0x02
  4. certificate: 0x0b
  5. server_key_exchange: 0x0c
  6. certificate_request: 0x0d
  7. servert_done: 0x0e
  8. certificate_verify:0x0f
  9. client_key_exchange:0x10
  10. finished: 0x14

handshake

The handshake process is usually two back and forth, mainly used to exchange different information. The handshake process of TLS1.2 can be abstracted as follows:

As you can see, after two back and forth, the client and server officially enter the data transfer. In a handshake round trip, it is not difficult to find that a request may have several different handshake protocol message types. This is because while the handshake protocol sends different messages to the recordlayer protocol, the recordlayer protocol generally sends several types as a block of data.

The * mark in the figure indicates that the message is adjusted for different cipher suites, and the [] mark indicates that the protocol is not part of the handshake protocol definition, but may also occur during the handshake. The part with lighter color belongs to a set that needs to be sent by binding, which is the link needed for two-way authentication.

Next, look closely at the meaning of each message

The Client Hello message

In the above flowchart, the client first initiates a message, which is structured as follows:

 struct {
        ProtocolVersion client_version;
        Random random;
        SessionID session_id;
        CipherSuite cipher_suites<2.2.^162 ->;
        CompressionMethod compression_methods<1.2.^8- 1>;
        select (extensions_present) {
            case false:
                struct {};
            case true:
                Extension extensions<0.2.^16- 1>;
        };
} ClientHello; 
Copy the code

This data defines six keys:

  1. Client_version: indicates the highest supported version
  2. Random: a random number generated by the client. In the FINISHED message, the PRF algorithm is used to generate primary keys and key blocks and prepare primary keys, avoiding replay attacks.
  3. Session_id: This is related to session restart. Clients that have established HTTPS connections can save the id given by the server. If this value is present, the session restart process will take place, rather than the complete process of the project
  4. Cipher_suites: password suites supported by the client
  5. Compresson_methods: Indicates compression algorithms. Generally, the compression algorithm is not enabled
  6. Extensions: Indicates supported extensions

So the message isn’t that complicated.

server_hello

The server hello message has the same structure as the client

  struct {
        ProtocolVersion server_version;
        Random random;
        SessionID session_id;
        CipherSuite cipher_suite;
        CompressionMethod compression_method;
        select (extensions_present) {
            case false:
                struct {};
            case true:
                Extension extensions<0.2.^16- 1>;
        };
} ServerHello;
Copy the code

I won’t repeat the description.

The server certificate message

This message is in the same request as the hello above. As can be seen, the certificate message will be sent after the server_hello situation is sent by the handshake protocol. However, since some cipher suites, such as DH_anon and DCDH_anon, do not send this certificate, the message is optional, but in most cases, the certificate information is used.

The message mainly carries certificate information, which is used for authentication. The message structure is as follows:

    opaque ASN1.Cert<1.2.^24- 1>;

    struct {
        ASN1.Cert certificate_list<0.2.^24- 1>;
    } Certificate;
Copy the code

You can see that there is only one certificate_list array that sends the certificate chain for certificate authentication. Details about certificates are left for later.

Server_key_exchange message

If the certificate information in the certificate message is insufficient to meet the requirements of the negotiated encryption algorithm, an additional server_KEY_exchange information is sent as a supplement. There are six cipher suites that require this message to be sent, as follows:

DHE_DSS DHE_RSA ECDHE_ECDSA ECDHE_RSA -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- / / no certificate DH_anon ECDH_anonCopy the code

Because the suite before the split line uses the temporary DH/ECDH key negotiation algorithm, the certificate does not contain dynamic DH information (DH parameters and DH public keys). Therefore, the certificate is appended to supplement the DH information after the certificate information is sent

The two types of kits under the split line are anonymous negotiation and do not have certificate information. Therefore, no certificate message will be sent. However, because additional static DH information needs to be sent, additional messages will be sent in this message.

The details of different public key cryptography algorithms are not discussed here.

Certificate_request message

As you can see, the server might send a Certificate_REQUEST message. The message is optional and also gray, I use gray to illustrate the relationship between the message and other gray information, and generally does not appear. However, in the bidirectional authentication scenario, the client is required to provide a certificate, so the server sends a message asking the client to also require an authentication certificate.

Server_hello_done message

After sending a sertificate and exchange message, the server ends with a server_HELLO_done message. The server then waits for subsequent work from the client. The message structure is simple:

 struct { } ServerHelloDone;
Copy the code

Client_certificate message

For the certificate message from the client, if the message sent from the server contains a Certificate_REQUEST message, the client sends the certificate information based on its own conditions. If no client has the appropriate certificate information, an empty array is sent.

Client_key_exchange message

This message will be sent anyway. The most important function of this message is to pass the pre_master_key, which is also known as the primary key preparation. In addition to the primary key preparation, if the client needs to send the certificate message above, it also makes supplementary information about the type DH algorithm key. But anyway, generating pre_master_key is the most important pair, generating pre_master_key is an important prerequisite for generating master_key, generating master_key after both parties get pre_master_key, after generating master_key, A [ChangeCipherSpec] message is sent to change the state. This means that the handshake process can be encrypted.

For details, see server_key_exchange on the server

[ChangeCipherSpec] agreement

For version 1.2, a protocol state transition is involved in the handshake message. After version 1.3, the protocol does not exist, but may still exist in the handshake for downward compatibility. This protocol is used to change the protocol state.

It can be understood that during the handshake, the status of the end receiving the protocol message is:

Pending Read Status Pending Read Status Pending Write Status Pending write statusCopy the code

After receiving this message, the end state is changed to:

Current Read Status Read status Current Write Status Write statusCopy the code

The message indicates that the peer end will enter the encryption phase when the handshake enters the readable and writable state. It also indicates that the handshake has been basically completed and the conditions for encrypted transmission are available.

Finished the message

After the handshake is complete, important information in the handshake will be verified. The verification parties will send their own verification information. The message structure is as follows:

    struct {
        opaque verify_data[verify_data_length];
    } Finished;

    verify_data = 
        PRF(master_secret, finished_label, Hash(handshake_messages))
        [0..verify_data_length-1];
Copy the code

The significance of this message is that all sub-messages during the handshake are not encrypted and integrity protected, and the message is easy to tamper with. To prevent messages from being tampered during the handshake, the Client and Server verify the Finished messages of each other.

After completing the Finish message, the final communication between the two parties will use master_key as the master key for symmetric encryption information.

The above is the handshake content defined in TLS1.2. Of course, there are still many details that have not been studied, such as the random number involved in the handshake and the function of session_id. But the general process has been covered, and then slowly add some details. The details require a bit of cryptography.

Scan the TLS

If you want to have a deeper understanding, you need to understand part of the cryptography related content, like the above mentioned symmetric encryption, asymmetric encryption, random number, all belong to the content of cryptography.

Common sense cryptography

Here are a few concepts of cryptography. They are:

  • The random number
  • One-way hash function (hash algorithm)
  • Message Verification code (MAC algorithm)
  • A digital signature
  • Symmetric encryption algorithm
  • Public key encryption algorithm

Random number algorithm

As an important cryptography unit, random number is widely used in cryptography. Cryptography unit means that random number will act as a factor in other algorithms. Such as generating symmetric passwords and message captcha.

Random numbers have three important properties:

  • randomness
  • unpredictability
  • irrepeatability

The three feature relationships can be expressed as:

In general, we call the random number with unrepeatability true random number, the random number with unpredictability strong pseudo-random number, and the random number pseudo-random number.

Only random number algorithms with at least strong pseudo-random number properties can be used in cryptography.

One-way hash function

A one-way hash function is a one-way hash function. The hash algorithm is also known as the MD5 algorithm. It is important to note, however, that the purpose of a one-way hash function is to ensure message integrity.

A one-way hash function has the following characteristics:

  • Different messages can reach fixed length hash values
  • Collision resistant, different messages get different hash values
  • Unidirectional, cannot get original value by hashing value
  • efficient

About strong and weak collision resistance:

  • Strong collision resistance: How hard it is to find the same message for any hash value
  • Weak collision resistance: The difficulty of finding a different message with the same hash value for a message and its hash value

It is easy to see from the features that one-way hashes can be used to compare message integrity. Because it’s small enough and fast enough, it’s useful for verifying the completeness of different content.

Pitfalls: Of course, this also exposes the limitations of one-way hash functions, which cannot prevent tampering or verify the source of information. Tampering is not prevented because attackers can easily hash different messages and masquerade them. Such as a man-in-the-middle attack.

Common hash algorithms:

  • MD5
  • SHA (secure hash algorithms, SHA – 1, SHA – 2 (256512224384), SHA – 3 (1224384), 256, 5 `

Symmetric encryption algorithm

Encryption and decryption are used with the same key encryption algorithm, called symmetric encryption algorithm. Symmetric encryption algorithm is that both parties in the transmission of data hold the same key. The biggest problem this algorithm needs to solve is the key transmission problem. Common symmetric encryption methods include AES and DES.

Public key cryptography algorithm

The public key cryptography algorithm is an asymmetric encryption algorithm. Due to the limitations of the symmetric encryption algorithm mentioned above, it is natural to think of whether there is an algorithm that uses different keys to transmit data. The decryption model of public key cryptography is also very simple:

It should be mentioned here that public key encryption does have more possibilities, but I find that many people have some misconceptions:

  1. Asymmetry is better than symmetry, or more confidential. I think this is not correct, because in terms of confidentiality, depending on the key length of the algorithm, and asymmetric encryption performance is very poor, it is difficult to say the same confidentiality, asymmetric encryption can do better than symmetric encryption.
  2. The author thinks that asymmetric encryption can solve the man in the middle attack, but asymmetric encryption cannot solve the man in the middle attack, because as long as the middleman hijacks the public key, he can replace the original public key without anyone knowing, and hide the whole thing with his fake public key and fake key.
  3. Asymmetric algorithm has more options in the transmission process, such as digital signature and key negotiation, both use asymmetric encryption asymmetric key, which is the significance of public key algorithm.

At present, the most common public key encryption algorithms are RSA algorithm and ECC algorithm.

I won’t take the time to describe the details of these two algorithms. The ECC algorithm has a shorter key and higher efficiency and confidentiality than RSA algorithm.

The disadvantage of public key encryption algorithm is the slow execution efficiency.

Message Verification Code (MAC)

Message Authentication Code is a technique for verifying the integrity and authentication. MAC algorithm for short.

The understanding of the MAC algorithm needs to be combined with the previous one-way hash algorithm. As we know, one-way hashing algorithm can guarantee message integrity.

Let’s see how the two compare.

Compared with the unidirectional hash algorithm, the MAC algorithm can generate the MAC value only when there is one more shared key. It is easy to imagine that transferring keys across the network is a problem for MAC algorithms, just as it is for symmetric encryption algorithms.

So what does the MAC algorithm solve relative to the HASH algorithm? We know that if the man in the middle attacks the HASH that the HASH algorithm generates. Since HASH and MAC algorithms do not care about message confidentiality, middlemen can modify messages and regenerate HASH values using the HASH algorithm to tamper with data. Although the MAC algorithm will suffer the same problem if the key is hijacked by the middleman, if we have a scheme to ensure that the MAC key is not hijacked, it can prevent the man in the middle attack, which is the significance of the MAC algorithm.

So, in the case of MAC algorithm key security. MAC ensures message integrity and authentication. Authentication means that only the person holding the key can generate the MAC value.

Does the MAC algorithm have its limitations, except to say that the MAC algorithm itself is not built for confidentiality, the MAC algorithm still cannot deny the problem. How to understand repudiation?

We assume that as a third party, the message authenticated by MAC algorithm is complete, but if one party does not recognize the message one day, because both sender and receiver have keys to generate MAC value, we cannot judge who generated the message from the MAC value alone. This requires the use of digital signature technology introduced later.

A digital signature

Digital signature is a technology that can authenticate sources, ensure data accuracy and prevent tampering. In terms of digital signature, there is still a need to make a comparison with one-way hashing algorithm and MAC algorithm.

The unidirectional hash algorithm can guarantee the complete accuracy of the message, meaning that there is no exception in the transmission of information. But there is no way to prevent data tampering, such as a man-in-the-middle attack, and no way to authenticate messages.

The Mac algorithm can ensure the accuracy of the message. At the same time, because there is a shared key, as long as the key is secure, the data can be tampered. However, because both parties in data transmission have the same key, the third party cannot know who sent the message, that is, the denial cannot be prevented.

Digital signature realizes message authentication mechanism by virtue of different key characteristics of public key cryptography algorithm. The principle is very simple, that is, the party that generates the message generates the signature through the key, and the party that receives the message can unlock the message through the public key and verify the correctness of the message. The reason why digital signatures can be repudiated anyway is that only the person with the public key can verify the message and only the person with the key can generate the signature. So both sides can not deny, but also to ensure the integrity of the message, the same can be tampered with anyway.

A simple process for digital signatures:

The performance of the asymmetric encryption algorithm mentioned in the signature is poor, so the performance of the preceding process is also poor, because the message length may be long. In addition, for the RSA algorithm, the longer the message to be encrypted, the longer the key public key length is. Smart students can easily imagine that using one-way hashing algorithms can generate simple hash values for messages, and one-way hashing algorithms also perform well. In this way, this problem is solved. Therefore, the general signature algorithm process is optimized as follows:

Supplement: In digital signatures, asymmetric encryption algorithms generate signatures through key encryption and verify signatures through public key decryption. When RSA is used to encrypt data, the public key is used to encrypt data transmission. The reason is that the public key is a public pair, and anyone can get it, and if you encrypt data with the key, anyone can get the public key and decrypt it.

Key negotiation algorithm

Review the previous techniques, message authentication code (MAC), symmetric encryption, and where these techniques are used, emphasizing the importance of keys. But in the process of network transmission, so that both sides can hold the same key. We’ll think about how to transmit the key.

This is the key distribution problem, to solve the key distribution problem, there are the following ways:

  1. Pre-shared keys: for example, pre-built keys needed in browsers and servers are obviously unrealistic
  2. Key center distribution: The establishment of a dedicated management key center, by it to distribute, but this storage and maintenance costs are very large.
  3. Implementation based on asymmetric algorithm: one end of the public key is responsible for generating the master key, and encrypts the master key with the public key. This prevents leaks because only the key holder can decrypt the master key. But pure this way will be attacked by the man in the middle problem, to cooperate with the certificate authentication public key legitimate.
  4. Implementation based on the DH algorithm: The DH algorithm saves various private information on both ends and transmits some public information. Both ends use the public information to generate master keys. Even if the public information is obtained, the third party cannot generate the master key through the public information, so as to achieve the effect of transmitting the key.

Here is the simple DH process, the asymmetric algorithm of key transmission is well understood, will not be described here:

In the above flow, G, P, G^AmodP, and G^BmodP are the four public numbers. But it’s hard to figure out A and B from these four numbers.

The significance of generating element G: the calculation of G is more complicated, for all values of G^XmodP (1 < x < p integer) are different, G is the element of P. And A and B randomly pick all integers between 0 and P. So A and B have enough choices, and they have enough possibilities.

Can DH algorithm prevent man-in-the-middle attack? No, the middleman can still forge A and B through intermediate tampering to achieve the effect of man-in-the-middle attack.

The DH algorithm can sign with asymmetric algorithms such as RSA to resist man-in-the-middle attacks. The DH algorithm and ECC algorithm are upgraded. The new key book negotiation algorithm is ECDH.

Certificate authentication

Certificate authentication is used to add a signature to a public key, perform key negotiation with the RSA algorithm, or encrypt data. Could have been hijacked and forged by middlemen. But if there was a technology that could tell if the public key had been forged, it could prevent man-in-the-middle attacks.

It’s easy to think of the digital signature technology we talked about earlier, because digital signatures not only ensure that data is not tampered with, but also prevent repudiation. However, if you are smart, you can easily imagine that digital signatures are also signed using public key algorithms. How to ensure the public key security of all signatories is another problem.

So imagine an organization that signs its own private key to the customer’s public key. Such an institution then exposes its public key, and since the middleman does not have the institution’s private key, the middleman cannot forge the signature. You can’t fake your client’s public key.

That raises the question of why we should trust the agency. At this point, we really have no choice but to trust the agency. This organization is the CA organization.

So, a certificate is essentially a digital signature for the customer’s public key and information. You can’t forge a public key.

However, the new problem is that when doing digital signature authentication, we need to use the public key of the CA organization for authentication. If we obtain the public key through the network, it is possible that the public key may be hijacked by the attacker. Therefore, it is better to build CA organization information into the operating system. It is not obtained from the network. But what if the number of CA organizations is too large? Therefore, CA organizations are divided into CA root organizations and CA intermediate organizations. In the operating system, all you need is a built-in root certificate. It does not matter if the root certificate is used to sign the lower-level organization and the intermediate organization is transmitted over the network. As long as the certificate is secure, the intermediate organization can be detected through the signature. This solves the transmission problem.

Now look at the handshake

From what we’ve learned about cryptography. Think more about the handshake process here. This time let’s clarify the purpose of the handshake. Look at the process in the figure above. The handshake process is ultimately for the data transfer service. That is, the application data process in subsequent data transmission.

Record layer protocol

Data transfer is defined in the record layer protocol. The record layer protocol mainly performs the following functions:

  1. Loads messages distributed by the upper-layer protocol
  2. Selective compression of data
  3. Encrypts, decrypts, and authenticates data by MAC

Therefore, after the handshake is complete, symmetric encryption algorithms are used to encrypt data and MAC authentication is performed. The handshake process is used to provide the session key for transferring data. TLS version 1.2 uses the SHA256 algorithm in PRF functions.

During the handshake, three keys are generated:

  1. Prepare the master key (pre_master_key)
  2. Master key (master_key)
  3. Session key (session_master_key)

In the previous handshake, we saw the first back-and-forth, where the main cipher suite was negotiated, and the client and server exchanged a pair of random numbers, client_RANDOM, and server_random. Forget about the structure of the Hello message.

The above master key is calculated using PRF function and random number:

PRF(pre_master_secret, "master secret",
    ClientHello.random + ServerHello.random)
    [0..47];
Copy the code

The calculation time, as mentioned earlier, is after the certificate has been verified.

In the case of RSA negotiation, the master key is prepared, which is randomly generated by the client and then sent through the public key, which is generated in the client_KEY_exchange message.

Here are some solutions based on the principles of cryptography:

  1. Why is the primary key generated by the client sent through the public key encryption.

A: In rsa key negotiation, a third party cannot decrypt the key information through the public key because the third party must use the asymmetric private key to decrypt the key, but the private key is not public. On the other hand, if the encryption is done using a private key, it is easy for a third party to decrypt the key because the public key is public.

  1. This section describes how to obtain the primary key in dh algorithm-related negotiations.

A: Join is not rsa as the key negotiation, then the two parties will exchange public information in the exchange information, and then calculate the pre-primary key, which will not be transmitted over the network. Essentially there is no change.

  1. In problem 1, isn’t there a man-in-the-middle attack through RSA key negotiation?

A: The cryptography chapter has discussed how certificate technology protects against man-in-the-middle attacks, meaning that the public key signed by the certificate is secure and cannot be forged by a man-in-the-middle. So there’s no man-in-the-middle attack.

The session key is the data generated through the master key in the recording layer. The data is split into several parts for encryption and decryption, and MAC authentication.

Session to restart

There are two ways to restart a session in TLS 1.2:

  1. Based on the Session Id
  2. Based on the Session Ticket

Process based on Session Id:

Session-based, the client sends the first Hello with the session_ID in the message, which is generated by the server during the full handshake and stored in the server’s memory. During the session restart, the original master key is retained and the session key is regenerated. This prevents forward security.

This restart mode has the common problem of session, that is, it consumes server resources, because the session is generated by the server and stored in the server.

Compared with the session restart mode, the ticket mode stores sessions on the client, but not on the server. However, each reboot updates the ticket. The process is as follows:

References:

  1. << Graphic cryptography >>
  2. << simple HTTPS >>
  3. https