As a Web development engineer, I deal with the network every day. Charles is a network packet capture tool that has become almost standard for Web development.

This article is the result of my extensive use of Charles. Unlike other articles about Charles, this article does not go into the details of Charles’ various functions (such as remote and rewrite), but focuses on one question: Under what circumstances does Charles fail to capture a packet?

In order to solve this problem, I will analyze the failure problems in Charles captured packages one by one based on the principle of Charles and the official policies of Android/iOS. After reading if you feel useful, be sure to remember to give me a thumbs up 🌟, thank you, this is really important to me!

1. Charles packet capture principle

The principle behind the vast majority of packet capture software on the market is man-in-the-middle attack (MITM).

Wikipedia defines MITM as follows:

In the field of cryptography and computer security, a man-in-the-middle attack means that the attacker establishes an independent contact with the two ends of the communication and exchanges the data they receive, so that the two ends of the communication think they are talking to each other directly through a private connection, but in fact the whole conversation is completely controlled by the attacker.

The above definition is very clear. I drew a diagram with Charles, and the direction of HTTP Packets can be understood with the direction of arrows:

It’s a bit of a theoretical exercise, but let’s take a look at the inner workings of Charles as an example. I used Wireshark to capture an HTTP packet from cdn.staticfile.org:

Based on the Wireshark captured packets and Charles network analysis, we can see the HTTP request packet direction:

  1. The browser (Client) makes a request from port number 56075, and the request is sent to port 8888 (MITM Server) on which local Charles listens. The connection is made directly on the local machine

  2. After receiving the browser request, Charles makes a new request from port number 56076 (MITM Client). Since the network packet needs to be connected to the public network, the IP address is 192.168.31.44 (my computer’S IP address, described in the next section). The IP address of cdn.Staticfile.org (Server) is 111.63.183.223, and the port number is 80 because it is an HTTP request

  3. 111.63.183.223:80 returns an HTTP response to Charles’s proxy client at 192.168.31.44:56076

  4. Charles does some internal processing (Capture & Analysis) and sends the response to 127.0.0.1:56075 through port 8888, where the browser receives the response

In the second and third steps, the message passes through Charles, and Charles can then do some processing on the message, such as Remote redirection, Rewrite, caching and visualization, etc.

Charles packet capture configuration

For the sake of continuity, I will briefly explain Charles’ packet capture agent configuration. If you’re familiar with it, you can skip it, but if you’re not, you can bookmark it for subsequent retrieval.


Before starting the configuration, let’s review the basic network knowledge. If network data is to be transmitted from A to B, five key information must be determined in order to determine A connection: source IP address, source port, transport layer protocol, destination IP address, and destination port. These five key pieces of information are also called quintuples, and a quintuple determines a connection.

If Charles is used to capture packets, we need to compare and analyze the quintuple of proxy link between Client and MITM Server:

  • The source IP address: Indicates the IP address of the captured packet application. It is usually the IP address of the device and is not changed
  • Source port: Indicates the port number of the captured packet application. It is usually assigned to the operating system and cannot be changed
  • Transport layer protocolCharles’s main proxy is HTTP, generally TCP
  • Destination IP address: IP address of Charles. It is the IP address of the local PC and is not changed
  • Destination port: Proxy port for Charles, default 8888,You can modify

⚠️ Note: This analysis is only general, and you can change it if you really want to, but this hack behavior is outside the scope of this article

From the above analysis, we can see that in order to get through the quintuple, we mainly need to pay attention to two points: destination IP address and destination port.


Let’s analyze the destination IP address first. Since Charles is installed on the computer, Charles’ IP is the IP of the computer. If you’re a Mac, hold down the Option key and click on the Wi-Fi icon in the menu bar to get your COMPUTER’s IP address. You can also go to System Preferences -> Network to get the IP address. Here my IP address is 192.168.31.44, which I’ll use later.

The Proxy port number for Charles can be viewed and changed from Charles -> Proxy -> Proxy Setttings. The default port number is 8888. You are not advised to change it.

HTTP Proxy Configuration

With Charles’s IP and port number determined, we can configure the HTTP proxy on a per-device basis.

1. Configure the PC HTTP proxy

Whether you’re using Windows or MAC, Charles, as an APP installed on your PC, forwards network data locally when requested by the agent, so configuration on the PC side is relatively simple. We only need to follow the path Charles -> Proxy -> macOS Proxy click on the start Proxy.

Since the native Proxy is off by default, we can also click on macOS in Proxy Setttings and check Enable macOS Proxy on Lounch to Enable the Proxy every time Charles is opened.

2. Configure the iOS proxy

To capture iOS network packets, you only need to forward the iOS network packets to the proxy IP address and proxy port.

Before configuration, we should ensure that the mobile phone and computer are in the same LAN (generally mobile phone and computer together with a Wi-Fi will do), and then open iOS Settings -> WIRELESS LAN, enter the connected Wi-Fi Settings page, slide to the bottom to select configuration agent, and then manually configure agent.

The server will enter the Charles IP address obtained earlier, in this case 192.168.31.44; The port is the Charles proxy port in front, typically 8888.

3. Configure the Android proxy

In fact, the proxy configuration of Android is similar to that of iOS, but there are many Android manufacturers, the operation steps are not as uniform as iOS, and it is not very meaningful to cover one by one, so I will just do a simple step demonstration:

HTTPS Proxy Configuration

HTTPS is essentially HTTP + TLS. From the point of view of connection establishment, four TLS handshakes are added after the TCP three-way handshake, as shown in the following figure. During the TLS handshake, the public key certificate used for encryption is verified. Therefore, we need to manually install and trust Charles’s certificate to capture HTTPS Packets.

The addition of TLS enhances network security and increases the complexity of packet capture. I will explain in detail in the next section. Here I will first demonstrate the steps of certificate installation.

⚠️ Note: Before installing the certificate, ensure that the HTTP proxy has been configured

1. Install the certificate on the PC

Installing a certificate on a computer is the easiest. Please click on Charles -> Help -> SSL Proxying -> Install Charles Root Certificate on your PC. Then click on the newly installed Certificate, manually trust all permissions, and enter the password to save and modify it.

2. IOS installation certificate

IOS installation certificates are relatively complicated. First click on Charles -> Help -> SSL Proxying -> Install Charles Root Certificate on a Mobile Derive and a pop-up window will pop up. Then we follow the prompts to access CHLS. Pro/SSL on the mobile end, download and install the Charles CA certificate.

After installation, manually open permissions. Go to General > Description files & Device Management > Trust to install the certificate you just downloaded. Then go to General > About Native > Certificate Trust Settings > Enable Full Trust for Root certificate to manually install the certificate.

3.Android installation certificate

Installing a certificate for Android is cumbersome and not very useful.

First click on Charles -> Help -> SSL Proxying -> Save Charles Root Certificate and Save the Certificate file in *.cer format.

⚠️ Note: Charles saves the certificate file in two formats:. Pem and. Cer. The former is a certificate container format, which is base64 encoding of certificates. The latter is usually a binary certificate, and Android is more compatible with binary certificates, so we choose the.cer file.

After saving the file, we can import the CA certificate into Android by USB or other means. Finally, click Certificate Install.

Iii. Case analysis of Charles packet capture failure

This section is actually the focus of this paper. It analyzes the reasons for Charles’ packet capture failure from six aspects, from proxy server to TLS certificate, covering all knowledge points of computer network, which is worth collecting and learning.

1. Close the agent! Close the agent! Close the agent!

As programmers, we use “accessibility tools” all the time in order to access sites like GitHub smoothly. These tools typically automatically enable HTTP/HTTPS proxies to preempt ports, causing Charles proxies to fail.

The solution to this problem is also simple. Charles turns off the auxiliary tools on his computer and mobile phone before he captures the bag, so there is no agent conflict. We can check the Wi-Fi agent interface of the computer, just make sure that the following options are not checked before starting Charles packet capture.

It must have occurred to some people that we would have two proxies hanging locally, where the packets would first pass through the tool, then be captured by Charles, and finally transmitted to the client. First of all, this scheme is feasible, but the actual use will be very slow, and the delay is very high, so it is not recommended to use it.

2. Determine the HTTP protocol for data transfer?

To begin this section, let’s take a look at how Charles is officially defined:

Charles is an HTTP proxy / HTTP monitor / Reverse Proxy that enables a developer to view all of the HTTP and SSL / HTTPS traffic between their machine and the Internet.

Charles is a network tool focused on analyzing HTTP packets, so support for other protocols is very limited. For example, the current IM or audio and video applications are basically encapsulated by themselves based on a certain transport layer protocol for the sake of performance and security. Charles certainly cannot capture these data.

By reading Charles’ official documentation and its own tests, Charles supports the following protocols:

  • HTTP / 1.1
  • HTTPS
  • HTTP/2
  • Ws (WebSocket)
  • WSS (WebSocket Secure, TLS encrypted WebSocket)
  • SOCKS

⚠️ Note: Charles does not support HTTP/3, but most sites that enable HTTP/3 are degraded. For example, when using Chrome to access Google normally, the protocol is HTTP/3. When connecting to Charles proxy, the protocol is degraded to HTTP/2

The protocols listed above cover 90% of application scenarios in daily service development. To capture packets of other protocols, use the Wireshark.

3. I opened the black and white list before, but forgot to close the bag again

I’m sure you or your colleagues have come across this scenario in your daily work:

A BUG was reported in the test. I connected to Charles and tried to analyze HTTP packets to determine whether there was a front-end problem or a back-end problem. As a result, I found that the requests could not be connected all the time

The black and white list of the above cases is just a general term. Specifically, in Charles, the following configurations may cause misunderstandings:

  • Proxy SettingsSome urls are filtered in Options
  • SSL Proxying SettingsNot all urls matched
  • Block List/Allow ListI made a black and white list
  • DNS SpoofingI did a mapping to HOST
  • Map RemoteThe request was redirected
  • RewriteRewrote the request
  • .

I have written several Charles Tools with high frequency. These functions are likely to be forgotten to be turned off after you turn them on. Should we check them one by one if something goes wrong?

In fact, Charles has a very humble function, that is, the lower right corner of its UI interface will display the functions that Charles is enabling. If you suspect that your Charles has any interface restrictions, you can just glance at the functions that Charles is enabling in the lower right corner and check them one by one.

4. The higher the Android version, the harder it is to capture HTTPS packets

In the section “Installing Certificates for Android,” I said that this step doesn’t make much sense because the user’s self-installed CA certificate doesn’t have ROOT privileges.

Let’s start with a picture of the Android certificate trust page:

As you can see from the figure above, Android divides certificate trust into two chunks:

  • System CA certificate: Basically has all permissions
  • User CA certificate: This certificate is installed by users and has low permission

Our own Charles certificates are all user CA certificates. In addition to certificate permissions, different versions of Android have different rules for handling permissions:

✅ : Android 7.0 below: trusted user CA certificate, can be simply understood as we install the certificate directly obtain ROOT permission

✅ : Android 7.0 or above, targetSdkVersion < 24: trusted user CA certificate

❌ : Android 7.0 or higher, targetSdkVersion >= 24: do not trust the user CA certificate


Through the above analysis, we can get several solutions to make Android trust Charles certificate:

1.ROOT

ROOT Android phone directly, put the Charles certificate in the system certificate, realize the certificate whitewashing

2. Get a phone that is lower than Android 7.0

Android 7.0 was released in 2016, and with Android phones coming out every two years and every year, it’s hard to find one

3. Prepare an APP installation package with targetSdkVersion < 24

In 2019, all major domestic application markets require that the APP API version must be larger than 28. Such installation packages are hard to find, and Internet products are iterating so fast that the availability of installation packages may not be guaranteed

4. SAO operation

There are plenty of Android trails out there. There are various wheels on the community to get around the restrictions, but it doesn’t have anything to do with Charles, so I won’t go into that. For those of you who like to do this, you can study it.


The above schemes are for other apps, if you want to capture the APP is your own company, it is very simple.

Android has a res/ XML /network_security_config. XML file that, as its name suggests, controls network security.

For example, the release package only trusts system level certificates, and the debug package trusts both System and user level certificates, so we can happily capture packages with Charles in the debug environment. Of course, there is more to the security configuration. If you are interested, you can go to the Android developer website to learn about it.


      
<network-security-config>
    <base-config cleartextTrafficPermitted="true">
        <trust-anchors>
            <certificates overridePins="true" src="system" />
        </trust-anchors>
    </base-config>
    <debug-overrides>
        <trust-anchors>
            <certificates overridePins="true" src="system" />
            <certificates overridePins="true" src="user" />
        </trust-anchors>
    </debug-overrides>
</network-security-config>
Copy the code

5. Certificate Pinning

Certificate Pinning is when the client has the true public key certificates of the server built into it.

In an HTTPS request, the public key certificate sent by the server to the client must be the same as the built-in public key certificate on the client. Generally, companies that pay more attention to safety will adopt this operation.

In this case, when Charles is used to capture packets, the public key certificate of Charles is different from the public key certificate of the client. The forged request will be rejected, and the packet capture will fail. So what’s the solution to this situation?

As previously introduced, there are actually two roads:

  • One is Hack, swipe ROOT, remove the fixed public key certificate in APP with the help of tools;

  • The other is the right way, you have the development permission of this APP, then you generally have the public key certificate and the accompanying private key, we can import the certificate and private key into Charles, to solve the problem caused by the fixed certificate.

Charles To import the public key Certificate and private key, click Charles -> Proxy -> SSL Proxying Setting -> Root Certificate, and then import the. Pem or p12 file.

⚠️ Note:. P12 is a file format that contains a certificate and a key

6. Two-way certificate authentication

In most cases, the client authenticates the server, but in some security-sensitive scenarios (such as anonymous social networking), some apps enable two-way authentication of TLS, which means that the server also authenticates the client.

In this case, the client must have a built-in public key certificate and private key. Compared with the server side, APP has a great risk of shell cracking, so the public key certificate and private key are generally extremely hidden, such as written in. So, hidden in a confused random number algorithm function, which is difficult to decipher.

I’m not a security expert and I don’t know much about this, and I don’t have such difficult questions in my work. It can be seen from the function panel that Charles should also support this kind of packet capture in extreme scenarios, but I don’t have specific practice, so you can try it.

Four,

Charles packet capture is a very common professional skill. If you dig deeper, you’ll find it involves quintuples of network connections, proxy services for packet forwarding, MITM and public key certificates in cryptography. In general, only by mastering the basic knowledge at the bottom can we cope with all kinds of strange and strange problems at work.

Fifth, thank you

If you think this article is good, please don’t forget to like 🌟 to support the author, thank you, it really means a lot to me!

Welcome to pay attention to the public account “brine egg laboratory”, write some original dry goods without water injection.

Welcome to visit my personal website: SupercodePower.com, because every major platform to modify the article will be reviewed, personal blog modification and update more convenient.

6. Recommended reading

I have written a number of computer network related blog posts before, across a number of pits, welcome you to gather your opinion:

  • Full explanation of 15 beautiful GIFs CORS 🔥🔥🔥
  • HTTP | X – Forwarded – cold knowledge is For get the real IP?
  • HTTP cold knowledge | HTTP request, the blank space should be encoded as % 20 +?
  • Have you been through all of these HTTP holes?