This is the 10th day of my participation in the More text Challenge. For more details, see more text Challenge

1, the preface

Recently, the data are all web end, today to teach you how to climb the mobile app data (this article to ios Apple mobile phone as an example, in fact, Android and ios almost)!

This article will take “some” as a practical case, hand in hand teach you from the configuration to the code step by step to climb App data!

2. Configure the packet capture tool

1. Install software

The packet capture tool of choice for this article: Fiddler

The specific download installation here is not detailed details! (online search Fiddler installation, a lot of tutorials), this article in actual combat as an example, no longer here to waste time!

2. Configure the Fiddler

Once installed, it’s time to configure the Fiddler tool (this is the key, read it carefully!).

Configure Connections

To open Fiddler, go to Tools->Options

Click the Connections

Select the corresponding options

Configure HTTPS

Because most of the current APP is HTTPS encryption, including this actual “some” case is HTTPS encryption, so configure HTTPS, to grab HTTPS data packets!

Select the corresponding options

Finally, the packet capture tool Fiddler is configured

Remember to restart Fiddler! Restart the Fiddler! Restart the Fiddler! Otherwise it might not work

3. Configure the mobile agent

1. Set the proxy

The preparatory work

First look at installing the Fiddler host IP (the computer and phone must be on the same LAN)

Viewing the IP Command

Windows: ipconfigCopy the code

Start the configuration

IP address of the target agent host: 192.168.31.195

Port: 8888

Go in at wifi and click Configure Proxy

Fill in the relevant agent information

2. Install the certificate

In the browser type:

http://192.168.31.195:8888
Copy the code

Click to download the certificate, and then start the installation (see the picture).

Ok, so the mobile end configuration is complete, the following start to grab data!!

4. Grab data

1. Open an app

2. View the data packet list

Once you open the app, Fiddler is already grabbing the data

Here you can see what packets the app sent and received

In order to more accurately locate to a certain point (only look at the target packets), add a filter condition

In this way, the list of packets we get is all the target url within the filter condition

3. Search for packets

Like clicking on the hot list

The corresponding HTTPS encrypted data packets are as follows:

The data in the packet is as follows:

Extract the URL link

https://api.zhihu.com/topstory/hot-list?limit=10&reverse_order=0
Copy the code

After you get the URL, you start programming to crawl the saved data.

4. Write a crawler

# -*- coding: utf-8 -*-
Copy the code

Ok so you can get the data down!

5, summary

1. Configure the packet capture tool Fiddler (emphasis).

2. Ios iPhone configuration certificate and setup agent (Android phones are similar).

3. Simply use Fiddler (filter packets, view packets, etc.).

4. This article takes a certain family as actual combat, realized the Python crawl mobile phone app data (detailed tutorial, recommended collection).