Charles is a Network packet capture tool that can be used to analyze the packet capture of an App and get all the Network requests and responses that occur during the App’s operation. This is consistent with the results seen in the Developer tools Network section of the Web browser.
Charles is more powerful than Fiddler and has better cross-platform support. Therefore, Charles is selected as the main mobile terminal packet capture tool to analyze mobile App data packets and assist in App data capture.
I. Objectives of this section
In this section, we take JD App as an example to grasp the network data packets in the running process of the App through Charles, and then check the specific Request and Response content, so as to understand the usage of Charles.
Two, preparation work
Ensure that Charles has been correctly installed and the agent service has been enabled. The mobile phone and Charles are on the same LAN. The Charles agent and CharlesCA certificate have been set.
Third, the principle of
First Charles runs on his own PC. When Charles runs, he will enable a proxy service on port 8888 of the PC. This service is actually an HTTP/HTTPS proxy.
To ensure that the mobile phone and PC are on the same LAN, we can use the mobile phone simulator to connect to the virtual network, or we can use the real phone and PC to connect to the wireless network.
Set the phone agent as Charles’ agent address, so that the data packets from the phone accessing the Internet will flow through Charles, Charles will forward these data packets to the real server, and the data packets returned by the server will be forwarded back to the phone by Charles, Charles will act as the middleman. All traffic packets can be captured, and therefore all HTTP requests and responses can be captured. Charles also has the power to modify the request and response.
Four, caught
The running interface of Charles in the initial state is as shown in the figure below.
Charles is always listening to the network packets that are happening on the PC and the phone, and the captured packets are displayed on the left. Over time, more and more packets are captured, and the list on the left becomes larger and larger.
As you can see, the Request site captured by Charles is shown on the left of the figure. We can click on any item to view the detailed information of the corresponding Request, including Request, Response and other content.
To clear Charles’s fetching results, click the broom button on the left to clear all currently captured requests. Then click the second listen button and make sure the listen button is on. This indicates that Charles is listening to the App’s network data flow, as shown in the figure below.
At this time, open jd on your mobile phone, and pay attention to the fact that Charles’ agent must be set up in advance and the CA certificate must be configured, otherwise it will have no effect.
Open any product, such as an iPhone, and go to its product review page, as shown below.
As shown in the figure below, Charles captured all network requests that occurred in JD App during the process of continuously pulling up the comments.
A link to api.M.jd.com appears in the list on the left, and it’s flashing, most likely because Charles caught a request for comment data from the current App. We click to expand it and continue to pull up to refresh the comments. As the pull-up progresses, network request records appear again, and the new packet request is determined to be a request for a comment.
To verify this, we click to see the details of one of the entries. Go to the Contents TAB, and then we find some JSON data, check the results, and the results have the commentData field, which is consistent with the comments that we see in the App, as shown in the figure below.
You can then determine that the interface for this request is the one that gets the product review. We have successfully captured the request and response content that occurred during the pull-up refresh.
Five, analysis,
Now examine the details of the request and response. First, you can go back to the Overview TAB, where the requested interface URL is displayed, followed by the response Status Code, request Method, and so on, as shown in the figure below.
The results are similar in form to those originally captured on the Web side using the browser developer tools.
Next click the Contents TAB to see the details of the request and response.
The upper part displays the Request information, and the lower part displays the Response information. For example, for Reqeust, we can switch to the Headers TAB to see the Headers information of the Request; for Response, we can switch to the JSON TEXT TAB to see the Body information of the Response, and the content has been formatted, as shown in the figure below.
Since this request is a POST request, we also need to care about the Form information of the POST, which can be viewed by switching to the Form TAB, as shown in the figure below.
In this way, we can successfully grab the request and Response of the comment interface in the App, and we can view the JSON data returned by the Response.
As for other apps, we can also use this method to analyze. If we can directly analyze the requested URL and parameter rules, directly use the program simulation can be batch crawl.
Six, resend
Charles also has the power to modify the captured request and send the modified request. Click the modify button on the top, and there will be a link starting with the edit icon in the list on the left, which means that the request corresponding to this link is being modified by us, as shown in the figure below.
We can Remove a field from the Form, such as removing the Partner field here, and then click Remove. At this point, we have modified the Form Data carried in the original request, and then click the Execute button below to Execute the modified request, as shown in the figure below.
You can see that the request result of the interface appears again in the list on the left, and the content is still the same, as shown in the figure below.
Deleting the Partner field from the Form has no effect, so it is irrelevant.
With this function, we can easily use Charles to do debugging, can by changing the parameters, such as the interface to test the different request of the state, you can know which parameters are necessary which is unnecessary, and the parameters of what law respectively, the result was one of the most simple interface and call use parameter form for process simulation.
Seven, conclusion
This is the process of analyzing App requests through Charles packet capture. Through Charles, we successfully captured the network packets flowing through the App, captured the original data, and modified the original request and re-initiated the modified request for the interface test.
Knowing the specific information of the request and response, if we can analyze the law of the URL and parameters of the request, we can directly use the program to simulate the bulk capture, which is of course the best.
But with the development of technology, the App interface often has a key, we can not directly find these rules, so what to do? Next, we’ll see how Charles and Mitmdump can be used to directly interconnect Python scripts to process the crawled Response in real time.
This resource starting in Cui Qingcai personal blog still find: Python3 tutorial | static find web crawler development practical experience
For more information about crawlers, please follow my wechat official account: Coder
Weixin.qq.com/r/5zsjOyvEZ… (Qr code automatic identification)