Recently the data to crawl are all web, today to teach you how to crawl mobile app data (this article to ios Apple mobile phone as an example, in fact, Android and ios similar)!
This article will take “mou hu” as the actual combat case, hand in hand to teach you from configuration to code step by step to crawl App data!
2. Configure the packet capture tool
1. Install the software
[Data collection] The packet capture tool selected in this paper: Fiddler
The specific download and installation is not detailed here! (Fiddler installation online, a lot of tutorials), this article as an example, will not waste time here!
2. Configure the Fiddler
Once installed, it’s time to configure the Fiddler tool (here’s the key, read it carefully!).
Configure Connections
After opening Fiddler, click Tools->Options
Click the Connections
Check the corresponding option on
Configure HTTPS
Because most of the current APP is HTTPS encryption, including the actual combat “hu” case is HTTPS encryption, so configure HTTPS, to grab HTTPS packets!
Check the corresponding option on
Finally, the packet capture tool Fiddler is configured
Remember to restart Fiddler! Restart the Fiddler! Restart the Fiddler! Otherwise it might not work
3. Configure the mobile phone proxy
1. Configure the proxy
The preparatory work
Install Fiddler host IP (PC and phone must be on the same LAN)
Viewing the IP command
Window: ipconfig copies the codeCopy the code
Start the configuration
IP address of the target proxy host: 192.168.31.195
Port: 8888
Go to the wifi and click Configure Proxy
Fill in the relevant agent information
2. Install the certificate
Type in your browser:
Copy the code http://192.168.31.195:8888
Click to download the certificate and start the installation.
Ok, so the configuration of the mobile end is complete, the following start to grab data!!
4. Capture data
1. Open an app
2. View the packet list
Once the app is open, Fiddler grabs the data
Here you can see which packets the app sends and receives
Add a filter condition to make it more accurate to target the target packet
In this way, the packet list we get is the target URL in the filter condition
3. Search for data packets
Like clicking on the hot list
The corresponding HTTPS encrypted packets are as follows:
The data in the packet is as follows:
Extract the URL link
- Api.zhihu.com/topstory/ho…
Once you get the URL, you start programming to crawl and save the data.
4. Write crawlers
# -*- coding: utf-8 -*-
Copy the code
Ok so you can get the data down! 5. Summary 1. Configure the packet capture tool Fiddler (key). 2. Ios iPhone configuration certificate and setup agent (similar to Android). 3. Simply use Fiddler (filter packets, view packets, etc.). 4. This article is based on the actual situation of a household, to achieve python crawling mobile app data.
How to obtain the source code:
① More than 3000 Python ebooks ②Python development environment installation tutorial ③Python400 set self-learning video ④ software development common vocabulary ⑤Python learning roadmap ⑤ project source code case sharing if you use it can be directly taken away in my QQ technical exchange group group number: 754370353 (pure technical exchange and resource sharing, no advertising) to take away by yourself click here to collect