Rare weekend, I went to my friend Amu’s house to pull him out together, but found him still in front of the computer seems to be working hard. I was so surprised that I almost thought he had been abandoned by the object of his own imagination and obsessed with his work
After chatting for a while, I learned that I was preparing for the National Day outing. I was collecting hotel information for data comparison and avoiding thunder as much as possible. I noticed that what he was basically doing was going to the website, selecting information, copying information, pasting it into an Excel spreadsheet, repeating, and then filtering at the end, which reminded me of a crawler that should have saved him a lot of work: the Web scraper
This is a Chrome extension, used for automatic crawling site data tools, you can crawl Taobao, Zhihu, Ali baba, Douban, Jane and so on website information, and basically do not need to use the code, even do not understand the procedure of xiaobai also a little practice can also use freely
Of course, I said more magical operation is more simple, A mu also thought I was bragging, and did not have the patience to learn, so I decided to show him how to climb ctrip website hotel name, rating, housing prices and hotel contact information
First, the Web scraper plug-in is installed on Chrome.
If you can go to the Chrome App Store, search for “Web scraper” and click “Add to Chrome” to put it in your browser. If you can’t, you can either install a Google Assistant plug-in to go to the Google App Store, or you can get an offline CRX installation package, Zip it, unzip it, and load it into your browser. Once installed, you can see a spider web icon in the upper right corner
Step 2, open the site and developer tools that you want to grab data from.
On the Ctrip site you want to climb, press F12 to open developer tools. The first opening may appear on the right side of the browser, so you can click the three small dots in the upper right corner to make it appear on the bottom
Third, set up the Sitemap on the Web Scraper on the developer tools
1, create new sitemap create sitemap
2. Please fill in a Sitemap name, including the Id name, for remarks only. Copy and paste the URL of the page into the Start URL and submit it
3, I’m going to Add new Selector and I’m going to put in a random Id, and I’m going to call it home, just to recognize it
Select Element in Type.
Click Select, move your mouse over to one of the hotel areas, find a spot where the entire box turns red, and then move your mouse over to the next hotel and do the same.
After this time, you will find that all the hotels on the page have turned red, click Done selecting.
Check Multiple and commit to save.
5. The above step is equivalent to completing the form, and the next step is to put the information data into the form.
Click inside the home that you just created, and then Add New Selector to create a subset
Select Text for Type, select Multiple, select the Text you want to extract, and then save. Remember to set one by one, for example, after setting the parameters of the collection hotel name, click Add New Selector on the same page again to set the parameters of the collection price and save, collect several information to save several times.
7. In this page, you can crawl to the hotel name (Id: hotel), grade (Id: Grade) and price (Id: price). But the contact information needs to click into the hotel to the secondary page to see, so you need to create a way to let it automatically open the page and then climb. The method is the same as above, but the Text in Type is changed to Link, and then saved for automatic opening of the secondary interface
Eight points in a jump to the hotel details page, find the contact information, then we found that the contact details need to check it will display number, so we have to let it can also help us to click automatically, then can create two subsets in the link, click one for show contact, one for the climb take number
9. On the contact page, click F12 “Web Scraper” xiecheng “home, click on the link, click Add New Selector, fill in the Id, Type, select Element click, Then, Select selectors and Click selectors together Select page contacts and Click Done Selectors. (You can also Click Data Preview to test if the numbers are displayed automatically.) Submit and save this for selecting contacts.
Click Add New Selector again to create one more subset that will be used to crawl the contact text, just as we did for the title above
Step 4, take what he says and start crawling
If you click Scrape in Sitemap Xiecheng, the default value of the popup page is 2000 and you can ignore it.
Request interval (ms) : the default value is 2000ms, which is the interval between two requests.
Page load delay (ms) : specifies the Page load delay time. The default value is 2000ms.
Broaching will pop up a window — don’t close the browser or the browser — and just wait for it to crawl and close
When it’s done, it’ll pop out of the little window
Step 5: Download it as a computer file and open it with excel
In Sitemap Xiecheng, click Export data as CSV and Download Now! At this point, it downloads it by itself. But on the computer, you can click on it and see that the data has been tabulated and can be filtered and compared
This is only part of its functionality, there are many more ways to crawl information, very powerful. Wood is all kinds of dumbstruck, hurriedly tried, found that busy will be a morning of things, now a few minutes can be done by himself, he is angry ah, I hope he can screen to a good hotel, National Day crowded crowd can have a hotel to have a good rest
I need an instructional video of the Web scraper or any questions