Make writing a habit together! This is my first day to participate in the “Gold Digging Day New Plan · April More text challenge”, click to see the details of the activity.

Your support means a lot to me!

🔥 Hi, I’m Peng. This article has been included in GitHub · Android-Notebook. Here are Android advanced growth route notes & blog, like-minded friends, welcome to grow with me. (Contact & Group entry on GitHub)

preface

  • Network request packet capture is a common problem in the process of r&d. Whether it is interface debugging during development or data verification during testing, network request packet capture is required. With the promotion of HTTPS and the upgrade of mobile phone system security, the threshold of packet capture may gradually become higher.
  • In this article, I will give you a comprehensive understanding of HTTPS packet capture from principle to actual combat. You will not only understand the implementation principle behind HTTPS packet capture, but also master the existing packet capture schemes in the market. I also enumerate the pit points in some solutions one by one and give solutions. Please be sure to like and follow if you can help, it really means a lot to me.

1. How HTTPS works

To explain the principle of HTTPS packet capture, you need to first explain the working principle of SECURE data transmission through HTTPS, which is divided into three elements and three phases.

The three elements are:

  • Encryption: Implemented by symmetric encryption algorithm
  • Authentication: by digital signature (since the private key is only held by the “legitimate sender”, the forged digital signature of others cannot be verified)
  • Packet integrity: achieved through digital signature (because message digest is used in digital signature, messages tampered by others cannot pass verification)

The three stages are:

  • CA certificate verification: CA certificate verification occurs during the first two handshakes of TLS. The Client and Server obtain the CA certificate from the Server using packets such as Client Hello and Server Hello. The Client verifies the validity of the CA certificate to verify the validity of the public key in the CA certificate. That is, the server does not authenticate the client.
  • Key negotiation: The key negotiation takes place during the last two handshakes of TLS. The client and server communicate asymmetric encryption based on the public key and private key respectively, and obtain the Master Secret symmetric encryption private key through negotiation (the details of the negotiation process vary with different algorithms).
  • Data transfer: Data transfer occurs after the TLS handshake, and the client and server communicate symmetrically encrypted based on the negotiated symmetric key.

— Image quoted fromHTTPS Principle Template

2. HTTPS packet capture principle – Man-in-the-middle attack

Familiar packet capture tools such as Fiddler, Charles, and HttpCanary App employ a man-in-the-middle attack (MITM) scheme: Proxy the network traffic of the client to the MITM host, and then structurally present the network request through a series of panels or tools.

If you are intercepting an HTTP request, it is ok. If you are intercepting an HTTPS request, you will encounter the first problem — encryption:

  • Encryption: In HTTPS communication, the symmetric Master Secret is held only by the communication parties. MITM cannot decrypt the ciphertext. As a result, the packet capture tool can only see a bunch of meaningless garble characters.

To solve this problem, MITM can only find a way to obtain the symmetric key. At this point, MITM not only needs to intercept traffic, but also needs to pretend to be a real client and server and establish independent connections with real communication parties. Let’s look at the three phases of HTTPS under man-in-the-middle attack:

Connection 1: HTTPS connection between the client and the middleman:

  • CA certificate verification: The client shakes hands with MITM, and MITM returns a “swapped” CA certificate (to enable the client to verify the CA certificate, install the MITM certificate on the system in advance).
  • Key negotiation: The client and MITM communicate asymmetric encryption based on the public and private keys of the “switched” to obtain the symmetric key through negotiation.
  • Data transmission: The client and MITM carry out symmetric encryption communication based on the negotiated symmetric key, and MITM can decrypt the plaintext.

Connection 2: HTTPS connection between the middleman and the server:

  • CA certificate verification: MITM shakes hands with the server, and the server returns the CA certificate. Since the server certificate is valid, MITM can obtain the public key of the server.
  • Key negotiation: MITM and the server communicate asymmetric encryption based on public and private keys respectively, and obtain Master Secret symmetric encryption private key through negotiation.
  • Data transmission: MITM and the server communicate symmetrically and encrypted based on the negotiated symmetric key.

At this point, MITM successfully establishes an independent connection with the real client and server, and the ciphertext sent can be decrypted successfully on MITM.

Since HTTPS can be captured, does that mean HTTPS is not secure?

You need to distinguish the security parameters in different scenarios. In scenarios where users are unaware of HTTPS, data can be transmitted safely. In scenarios where users take the initiative to authorize users, users need to assume corresponding security risks for this initiative.

To summarize the basic steps of HTTPS packet capture:

  • Deploy the MITM proxy server.
  • Collect network traffic to MITM host by proxy;
  • The client trusts the MITM CA certificate.
  • MITM establishes independent connection with client and server respectively.
  • MITM decrypts the communication between client and server and presents it structurally.

3. Install the CA certificate on Android

Installing CA certificates on Android can be summarized into three types, where both system certificates and user certificates can be viewed in the trusted credentials in system Settings:

  • System Certificate:The system CA certificate is installed in the/system/etc/security/cacerts/Only Root permission is allowed to add and delete directories. Note that system CA certificates have special naming formats:The hash value. 0, the conversion method can be referred to:Android built-in certificate file.
  • User certificate:The user CA certificate is installed in the/data/misc/user/0/cacerts-added/Directory, installed by the user, can be set in the systemInstall the certificateTo install. Due to a behavior change in Android 7.0, the system no longer trusts user certificates by default for applications with targetSdkVersion ≥ 24.

Application level AndroidManifest. XML

<? xml version="1.0" encoding="utf-8"? > <manifest ... > <application android:networkSecurityConfig="@xml/network_security_config". >... </application> </manifest>Copy the code

network_security_config.xml

<? xml version="1.0" encoding="utf-8"? > <network-security-config> <base-config cleartextTrafficPermitted="true"> <trust-anchors> <! -- Trust preinstalled CAs --> <certificates src="system"/ > <! -- HERE: Additionaly trus user added CAs --> <certificates src="user" />
        </trust-anchors>
    </base-config>
</network-security-config>
Copy the code
  • Apply fixed Certificate Pinner: The client can directly build the trusted server Certificate to limit the set of certificates it accepts, and use a customized TrustManager to replace the default TrustManager. In the certificate verification phase of HTTPS request, only the certificate set accepted by HTTPS request can be verified, which is also a scheme to avoid man-in-the-middle attack. Such as:
protected static SSLSocketFactory getSSLSocketFactory(Context context, @ResId int[] certificates) {
    if (context == null) {
        throw new NullPointerException("context == null");
    }
    CertificateFactory certificateFactory;
    try {
        certificateFactory = CertificateFactory.getInstance("X.509");
        // Create a KeyStore containing our trusted CAs
        KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());
        keyStore.load(null.null);
        for (int i = 0; i < certificates.length; i++) {
            // Load the built-in certificate
            InputStream is = context.getResources().openRawResource(certificates[i]);
            keyStore.setCertificateEntry(String.valueOf(i), certificateFactory.generateCertificate(is));
            if (is! =null) {
                is.close(); }}// Create a TrustManager that trusts the CAs in our keyStore
        TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
        trustManagerFactory.init(keyStore);

        // Create an SSLContext that uses our TrustManager
        SSLContext sslContext = SSLContext.getInstance("TLS");
        sslContext.init(null, trustManagerFactory.getTrustManagers(), new SecureRandom());
        return sslContext.getSocketFactory();
    } catch (Exception e) {
        e.printStackTrace();
    }
    return null;
}
Copy the code
// Set the certificate through the OkHttp APIOkHttpClient.Builder builder = new OkHttpClient.Builder(); int[] certficates = new int[]{R.raw.media}; builder.socketFactory(getSSLSocketFactory(context, certficates)); .Copy the code

Breach the Android 7.0 USER CA certificate limitation

Due to a behavior change in Android 7.0, the system no longer trusts user certificates by default for applications with targetSdkVersion ≥ 24, Need in the application of AndroidManifest. Add android: XML networkSecurityConfig configuration. If you want to capture a third-party app and the app is not configured, you need to do something about it:

  • Method 1 – Use Android 7.0 or below: Erase user certificate restrictions from the source, this is the easiest and most straightforward;
  • Method 2 – Use virtual systems such as Parallel space: use virtual systems such as HttpCanary parallel space or VMOS App to create an Android 7.0 or less system environment on your mobile phone.
  • Method 3: Install the certificate to the system certificate directory: Install the CA certificate to the system certificate directory with the Root permission.

4. Summary of Fiddler’s using skills

Fiddler is currently mainly used as a network debugging tool on Windows system. The latest version of Fiddler Everywhere supports all platforms, but I feel that many functions are missing in my experience. I am looking forward to official updates. What follows is a new version of Fiddler Everywhere that works similarly to older versions.

4.1 Using Fiddler to Capture HTTPS Packets

Here is a summary of the main steps of using Fiddler to capture packets. In fact, the configuration is based on the basic steps of implementing HTTPS packet capture mentioned in Section 2:

  • 1. Deploy MITM proxy server:By default, Fiddler’s Web server will be deployed on port 8866 when you start Fiddler on your computer. You can change the port using the Settings page below. Because we’re on our phones, we still have to checkAllow remote computers to connect ：

  • 2. Collect network traffic to THE MITM host by proxy or other means: execute ipconfig on the command line of the computer to obtain the local IP address, then connect the mobile phone and the computer to the same LOCAL area network, modify the advanced Settings for the mobile phone to connect to Wifi, and set the IP address and port 8866 for the proxy to the computer. At this point, you can see the captured requests on Fiddler, but not the HTTPS data. There’s a “lock” icon that says:

  • 3. The client trusts the MITM CA certificate:To export the certificate, you need to enable the Https packet capture function on Fiddler. In the Settings page below, clickTrust root certificateCheck again,Capture HTTPS trafficOptions.

Then, you need to install the CA certificate on the mobile phone in two ways: Export root Certificate to Export the certificate file (by default, to the desktop) and then send the file to the mobile phone. You can also access ipv4. Fiddler :8866/ from your mobile browser and click on FiddlerRoot Certificate to download the certificate directly.

Now that you have successfully downloaded the CA certificate to your phone, you need to manually install the certificate. Search for the installation certificate in system Settings, find the CA certificate you just downloaded and install it (different mobile phone systems have different interfaces) :

At this point, you have no problem grabbing an HTTPS request on Fiddler. It is best to filter out interference items in the filter bar, such as non-important domain names, CONNECT handshake validation requests:

  • Filter domain name: Contains juejin
  • Method: is not equals to CONNECT and HEAD
  • Filter content-type: does not contain image/

Tip: Actually, Fiddler can do more than HTTP packet capture. It also supports a lot of advanced features. If you need to dig deeper into Fiddler, I suggest you buy Xiao Jia’s HTTP Packet Capture Game. Here are some of the tricks I’ve played.

4.2 Fiddler Packet Replay Test

Replay Attacks refer to Attacks in which an attacker captures packets sent by a client to a server and then resends the packets to the server. For example, an attacker captures a “like”, “vote”, or “award” request packet and sends it to the server again. Because the request itself is legitimate and the attacker has not tampered with the request, the server can mistake it for a valid request.

Replaying Requests on Fiddler is easy in two ways: right-click on a request and select Replay→Reissue Requests. Or select Edit in Composer to Edit and then replay.

I actually encountered an overload attack in the project. A smart user captured the request message of the App similar to receiving gold coins, and then collected a wave of wool with replay attack. The anti-replay method is to add identification parameters and digital signatures to the request (tamper-proof) :

  • Timestamp: The server compares the timestamp of the current request with the server time. If the request exceeds the threshold (for example, 60 seconds), the request is considered obsolete. The disadvantage is that playback requests within 60 seconds will still be considered valid.
  • Serial number: The server compares the serial number of the current request with the serial number recorded by the server. If a non-ascending (equal or less) request is received, it is considered obsolete. The disadvantage is that the order of packets needs to be ensured.
  • One-time password: The server uses the current one-time password to search the password table maintained by the server. If the password has already been used, the request is considered as outdated. The disadvantage is that the password table needs to be maintained. In practice, the timestamp + one-time password scheme can be used to avoid replay attacks within a short period of time, and the server only needs to maintain the password table within a short period of time.

4.3 Fiddler weak mesh simulation

The latest Version of Fiddler Everywhere does not support weak network simulation. Fiddler Classic of an earlier version needs to be used. The configuration paths are as follows: Rules→ Legends →Simulate Modem Speeds.

4.4 Fiddler Modifying HTTP Requests

Fiddler itself is a proxy server that intercepts HTTP requests/responses for modification before releasing them. To configure a Rule on Fiddler:

5. Summary of Charles’ use skills

5.1 Using Charles for HTTPS Packet Capture

Here is a summary of the main steps of using Charles to capture packets. In fact, the configuration is based on the basic steps of implementing HTTPS packet capture mentioned in Section 2:

  • 1. Deploy MITM proxy server: Start Charles on the computer, and by default, Charles Web server will be deployed on port 8888 of this computer. You can modify the port through the Settings page below, and it is better not to use the default port number here.

  • 2. Collect network traffic to MITM host by proxy or other means:Run ipconfig on the computer command line to get the local IP address (or Charles’s)Help to the Local Ip Address), and then connect the mobile phone and computer to the same LOCAL area network, and then modify the advanced Settings for the mobile phone to connect to Wifi, set the IP address and port 8888 proxy to the computer.
  • 3. The client trusts the MITM CA certificate:Charles packet capture also requires the trust of the Charles CA certificate on the phone. According to the instructions, it will be accessed in your mobile browserchls.pro/sslDownload the certificate and install it in the same way as Fiddler (pit log: access to CHLS. Pro/SSL will not download automatically when using the default port number 8888, just change the port number).

At this point, you have successfully fetched the HTTPS request on Charles. It is best to filter out the interference items in the filter:

5.2 Charles Packet Replay Test

Replaying requests on Charles is simple in two ways: right click on a request and select Repeat or Repeat Advanced to replay directly; Or select Compose to edit and then replay. Repeat Advanced is suitable for stress testing.

5.3 Charles weak network simulation

You can press Proxy→Throttle Settings to go to the weak-network configuration page. Throttle Preset provides multiple preset network environment simulation configurations and can be modified based on them. The meanings of each option are as follows:

  • Bandwidth: the amount of data transmitted over a communication link per second.
  • Utilisation: Bandwidth usage;
  • Round-trip latency: Transport layer concept, which refers to the round-trip latency of packets between the client and the server.
  • MTU: Data link layer concept. The maximum payload that an index frame can carry is 1500 bytes on an Ethernet. If the size of an IP packet exceeds the MTU, IP packets are fragmented.
  • Reliability/packet loss rate: Indicates the probability of data transmission failure.
  • Stability Stability/jitter rate: Indicates the Stability of the network environment. If the network is unstable, the success rate of data transmission is also unstable. This parameter is ideal for simulating mobile networks;
  • Unstable quality range: Specifies the Unstable range of the network environment.

5.4 Charles Modifies the HTTP Request

Charles itself is a proxy server that can intercept HTTP requests/responses for modification before releasing them. Configure Rewrite on Charles:

6. Mobile phone local packet capture scheme

The configuration of the packet capture schemes such as Fiddler, Charles, and Wireshark is cumbersome. For example, you need to configure the mobile phone proxy and install certificates. The biggest disadvantage is that both rely on a PC with a proxy server deployed, which cannot meet the requirements of packet capture anytime and anywhere. In practice, a comprehensive packet capture scheme can be adopted. If the local packet capture scheme cannot meet the requirements, a scheme such as Fiddler can be used to complete the packet capture.

6.1 VPNService API

VPNService is an API introduced in Android 4.0. It intercepts system traffic without root permission. Ap, which we are familiar with, actually runs a VPNService service. When the VPN is running, there is usually a message on the notification bar, such as “VPN activated”. To search for VPN in system Settings, you can view the applications that provide VPN services on the mobile phone. For example:

  • HttpCanary App

HttpCanary is a powerful network analysis tool for Android phones. It works by deploying a MITM proxy server based on VPNService to capture network packets. HttpCanary also supports bidirectional authentication packet capture. For details, see HTTPS Packet Capture Solution on the Android platform

However, HttpCanary has a problem fetching HTTPS requests on Android 11. Since the system only allows you to manually select the install certificate from the system Settings, the install certificate step is slow because the App has not yet implemented this issue. I offer two solutions:

  • 1. Use a lower version phone (extremely comfortable);
  • 2, use a mobile phone with Root permission to manually remove the certificate from the HttpCanary internal storage space, convert it into a system certificate, and install it. This scheme also solves the third-party applications, by the way not configured android: networkSecurityConfig problem. For details, see installing the Android 11 HttpCanary System certificate

  • Like mobile assistant App

The Technical team of Youzan is one of the teams I often follow. The local packet capture scheme of Youzan Mobile Assistant App is a mobile phone local packet capture scheme shared by them in 19 years. The principle of the sponsored hand App is also a proxy server built based on Android VPNService or IOS NetworkExtension. The helper App and the real server complete the HTTP request, which is equivalent to the self-implemented HttpCanary. At present, with the accumulation of small peng can not be fully digested to achieve this plan, the record here to provide a train of thought.

6.2 OkHttp interceptor

For applications that implement network requests based on OkHttp, network data within the application can be monitored through interceptors, and captured data can be viewed through the notification bar, desktop widgets, and other portals. There are already open source implementations that can be used directly in the industry:

  • Chunk: After Chunk is integrated, monitored network requests are displayed in the notification bar. Click to enter the analysis page. However, this project has not been maintained for many years, and does not support data filtering, some weak points.

  • Drops DoraemonKit

Didi Doraemon is familiar to everyone. It is an efficient platform for big front-end products. Currently, it has developed into A relatively complete ecosystem, supporting multiple platforms such as Android and iOS. DoraemonKit also provides network listening capabilities, again based on the OkHttp interceptor. The difference is that DoraemonKit adds interceptors via AOP injection, so we don’t need to manually add interceptors at initialization, which is a good way to solve the problem of having multiple OkHttpClients for different components. Related source code:

HttpUrlConnectionProxyUtil

private static void addInterceptor(OkHttpClient.Builder builder) {
    // Checks whether an interceptor is currently added and returns if so
    for (Interceptor interceptor : builder.interceptors()) {
        if (interceptor instanceof DokitMockInterceptor) {
            return;
        }
    }

    builder
        // Add a mock interceptor
        .addInterceptor(new DokitMockInterceptor())
        // Add the big graph detection interceptor
        .addInterceptor(new DokitLargePicInterceptor())
        // Add dokit interceptor
        .addInterceptor(new DokitCapInterceptor())
        // Add weaknet interceptor
        .addNetworkInterceptor(new DokitWeakNetworkInterceptor())
        // Add extension interceptor
        .addInterceptor(new DokitExtInterceptor());
}
Copy the code

  • Youku Ribut

Ribut is shared by the students in Xiaopeng’s communication group. I have read that this project is still in its early stage, and I look forward to its follow-up development. It is a visual debugging architecture developed by Alibaba Youku technical team, aiming to solve the daily pain points of r&d through instrumental means, such as network packet capture. The general idea of network packet capture is divided into two steps: 1. Establish a connection with PC by scanning codes; 2. 2. Capture network request data through OkHttp interceptor and forward to PC. Although strictly PC, the overall experience is OK.

7. To summarize

Having said so many schemes to capture packets, let’s change a perspective. Do you know what schemes App has to reverse capture packets? Follow me, tell you more, and I’ll see you next time.

The resources

  • HTTP Packet Capture Combat by Xiao Jia
  • Network Security Configuration – Official Android documentation
  • Note: An internal and domestic note for Android Traffic
  • Install System CA Certificate on Android Emulator — mitmProxy
  • HTTPS Packet Capture for Android by MegatronKing
  • Adding trusted Certificate Authentication using Https in Android by Joel Litterstar
  • Local Packet capture scheme of Youzan Mobile Assistant App — By Yang Bin (Youzan)

Your likes mean a lot to me! Wechat search public number [Peng Xurui], I hope we can discuss technology together, find like-minded friends, we will see you next time!