Master Python crawlers from Scrapy to mobile applications

I can hear people screaming, “Appery. IO, a platform for mobile apps, and what does Scrapy have to do with it?” Well, seeing is believing. You may remember showing someone (a friend, a manager, a client) data on an Excel spreadsheet years ago. But unless your audience is sophisticated these days, their expectations are likely to be different. In the next few pages, you will see a simple mobile application, it is a just a few clicks can created the minimum visual products, its purpose is to convey to the stakeholders to extract data from power, and return to ecological system, in the form of network traffic source website to show the value of it can bring.

I’ll try to keep the heuristic examples short, and here they’ll show you how to make the most of your data.

1.1 Selecting a Mobile application framework

Providing data to mobile applications with the right tools is easy. There are many excellent cross-platform Mobile application development frameworks, such as PhoneGap, Appcelerator using Appcelerator cloud services, jQuery Mobile and Sencha Touch.

This article will use Appery. IO as it allows us to quickly create Mobile apps for iOS, Android, Windows Phone and HTML5 using PhoneGap and jQuery Mobile. Neither I nor Scrapy have any interest in Appery. IO. I would encourage you to do your own research to see if it meets your needs in addition to the features outlined in this article. Please note that this is a paid service and you can have a 14-day trial period, but in my opinion it is a no-brainer to develop a prototype quickly and is worth paying for, especially for those who are not web experts. The main reason I chose this service is that it provides both mobile applications and back-end services, meaning we don’t need to configure databases, write REST apis, or use some other language for the server and mobile applications. As you’ll see, we didn’t have to write a single line of code! We’ll use their online tools; At any time, you can download the app and use all of PhoneGap’s features as a PhoneGap project.

For this article, you will need access to an Internet connection in order to use Appery. Also note that the layout of the site may change in the future. Please use our screenshots for reference and don’t be surprised to find that the site looks different.

1.2 Creating databases and Collections

The first step is to Sign Up for the free Appery. IO scheme by clicking the sign-up button on the Appery. IO website and selecting the free scheme. You need to provide a username, email address and password, and the new account will be created. Wait a few seconds for the account to be activated. You can then log into the Appery. IO dashboard. Now you are ready to create the new database and collection, as shown in Figure 1.1.

Figure 1.1 Creating a new database and collection using Appery. IO

To complete the operation, perform the following steps.

1. Click the Databases TAB (1).

2. Then click the green Create New Database (2) button. Name your new database scrapy (3).

3. Now click the Create button (4). This automatically opens the Scrapy database dashboard, where you can create new collections.

In Appery. IO terms, a database is made up of a set of collections. Basically, an application uses a single database (at least initially), each containing multiple collections, such as users, properties, messages, and so on. Appery. IO already provides a Users collection by default, which includes a user name and password (they have many built-in features). Figure 1.2 shows the process of creating a collection.

Figure 1.2 Creating a new database and collection using Appery. IO

Now let’s add a user named root with a password of pass. Of course, you can also choose a more secure username and password. To do this, click the Users collection in the sidebar (1), and then click +Row to add a user/Row (2). Fill in the username and password (3) and (4) in the two fields that appear.

We also need to create a new collection to store the property data that Scrapy grabs and name it Properties. You can Create a new collection by clicking the green Create New Collection button (5), naming it Properties (6), and then clicking the Add button (7). Now we also have to do some customization to the collection. Click +Col to add database columns (8). Each database column has a type that validates the value. Most of the fields are simple string types, except that the price is a numeric type. We will add a few columns (8) by clicking +Col, fill in the column name (9), and select the type (10) if it is not a string, and then click the Create Column button (11). Repeat this process five times to create the columns shown in Table 1.1.

Table 1.1

At the end of collection creation, you should have created all the columns you need, as shown in Table 1.1. Now you’re ready to import some data from Scrapy.

1.3 Use Scrapy to populate the database

First, we need an API key. We can find it in the Settings TAB (1). Copy the value (2), then click the Collections TAB (3) to return to the property collection, as shown in Figure 1.3.

Figure 1.3 Creating a new database and collection using Appery. IO

Very good! You now need to import the data into Appery. IO. Let’s start by copying the project and a crawler named Easy (easy.py) and renaming the crawler tomobile (tomobile.py). In the meantime, edit the file and set its name to Tomobile.

You may have noticed one problem is that there is no use of Wen Jie before using the Web server (http://web:9312), but use, is the site of a publicly available a copy of a copy of this is my store in http://scrapybook.s3.amazonaws.com. I use this approach in this article because it makes both images and urls publicly available, making it very easy to share your application.

We will use the Appery. IO pipe to insert data. A Scrapy pipe is typically a small Python class with post-processing, cleaning, and storage of Scrapy items. For now, you can install it using easy_install or PIP, but if you’re using our Vagrant Dev machine, you don’t need to do anything because we’ve already installed it.

or

At this point, you’ll need to make some minor changes to your Scrapy main setup file, adding the API keys you copied earlier. Now, all we need to do is add the following line to the properties/settings.py file.

Don’t forget to replace APPERYIO_DB_ID with your API key. Also, make sure that the username and password in your Settings are the same as when you created the database user in Appery. IO. To populate the Appery. IO database, start scrapy Crawl as usual.

The output will be a little different this time. You can see that in the first few lines, one line is used to enable the ApperyIoPipeline Item pipeline; Most obviously, though, you’ll see 200 requests/responses despite fetching 100 items. This is because the Appery. IO pipeline performs an additional request to the Appery. IO server for each Item to be written. These requests with the URL api.appery. IO also appear in the log.

When you go back to Appery. IO, you can see that data (2) has been populated in the Properties collection (1), as shown in Figure 1.4.

Figure 1.4 populates the properties collection with data

1.4 Creating a Mobile Application

Creating a new mobile application is simple. We just clickAppsTAB (1), and then click the green oneCreate new appButton (2). Set the application name toproperties(3), and clickCreateButton to create, as shown in Figure 1.5.

Figure 1.5 Creating a new mobile application and database collection

1.4.1 Creating database Access Services

The number of options when creating a new application can be a bit excessive. It is possible to write complex applications using the Appery. IO application editor, but we will keep things as simple as possible. The first thing we needed was to create a service that would allow us to access our Scrapy database from within our application. To do this, click the rectangular green button CREATE NEW (5), then select Database Services (6). A new dialog will pop up and let us select the database we want to connect to. Select scrapy database (7). Most of the options in this menu are not used, so now just click to expand the Properties area (8) and select List (9). In the background, it writes code for us that makes the data we use Scrapy to crawl usable on the network. Finally, click the Import Selected Services button to complete (10).

1.4.2 Creating a User Interface

Now it’s time to start creating all the visual elements for the application, using the DESIGN TAB in the editor, as shown in Figure 1.6.

Figure 1.6 Creating the user interface

From the tree on the left side of the page, expand the Pages folder (1), then click startScreen (2). The UI editor will open the page and we can add some controls to it. Let’s use the editor to edit the title to make it more familiar. Click on the header title (3), and the properties area on the right side of the screen will change to show the title property, including a Text property. Change that property to Scrapy App, and the title in the middle of the screen will be updated accordingly.

Then, you need to add a Grid component by dragging the Grid control from the left panel (5). The control has two lines, and for our purposes, only one. Select the grid you just added. When the thumbnail area (6) at the top of the phone view turns gray, you know that the grid has been selected. If it is not selected, click on the grid to select it. The property bar on the right is then updated to the properties of the grid. Just set the Rows property to 1 and then click Apply (7) and (8). The grid will now be updated to have only one row.

Finally, drag and drop additional controls into the grid. First add a picture control to the left of the grid (9), then a link to the right of the grid (10), and finally a label below the link (11).

In terms of layout, this is enough. The next step is to enter data from the database into the user interface.

1.4.3 Mapping Data to the USER Interface

So far, we’ve spent a lot of time in the DESIGN TAB to create visualizations of the application. To link available DATA into these controls, switch to the DATA TAB (1), as shown in Figure 1.7.

Figure 1.7 Mapping data to the user interface

Select Service (2) as the data source type. Because the service created earlier is the only service available, it is automatically selected. You can then continue to click the Add button (3), and the service properties will be listed below it. Once you press the Add button, you will see events like Before Send and Success. We can customize what to do after a successful invocation of the service by clicking the Mapping button after Success.

The Mapping Action Editor opens and we can finish wiring from there. The editor has two sides. On the left are the fields available in the service response, and on the right you can see the properties of the UI control you added in the previous step. There is an Expand All link on both sides that you can click to see all the available data and controls. Next, you need to drag from the left to the right according to the five mappings given in Table 1.2.

1.4.4 Mapping between database fields and USER interface controls

The number of items in Table 1.2 May differ slightly from your case, but since there is only one of each control, the likelihood of an error is very small. By setting these maps, we tell Appery. IO to write all the code behind the scenes to load the controls using values from the database when the database query succeeds. Next, click the Save and Return button (6) to continue.

You’re back to the DATA TAB, as shown in Figure 1.7. Since you also need to return to the UI editor, you need to click the DESIGN TAB (7). At the bottom of the screen, you’ll find an EVENTS field (8), which has only just been expanded, although it has always been there. In the EVENTS section, we let Appery. IO do something in response to UI EVENTS. This is the last step we need to take. It makes the application call the service to retrieve the data as soon as the UI is loaded. To do this, we need to select startScreen as the component and leave the event as the default Load option. Then select the Invoke Service as the Action, leaving the Datasource as the default restService1 option (9). Finally, click Save (10), and that’s all we’ve done to create the mobile application.

1.4.5 Test, share and export your mobile app

Now you are ready to test the application. All we need to do is click the TEST button (1) at the top of the UI generator, as shown in Figure 1.8.

Figure 1.8 mobile application running in your browser

Mobile apps will run in the browser. These links are valid (2) and can be browsed. You can preview different Phone screen schemes and device orientation, or you can click the View on Phone button, which brings up a QR code that you can scan with your mobile device and preview the app. All you have to do is share the generated links, and others can try the app in their browsers.

With just a few clicks, we can organize the data that Scrapy grabs and display it in the mobile app. If you need to further customize the application, you can reference Appery. IO provide tutorial, the url is http://devcenter.appery.io/tutorials/. When all is ready, the application can be exported via the EXPORT button, and Appery. IO offers a wealth of EXPORT options, as shown in Figure 1.9.

Figure 1.9 You can export your application to most major mobile platforms

You can export project files and develop further in your favorite IDE; You can also get binaries and publish them to the mobile market on all platforms.

1.5 Summary of this paper

Using Scrapy and Appery. IO, we have a system that can crawl websites and insert data into a database. In addition, we got RESTful apis and a simple mobile application for Android and iOS. For advanced features and further development, you can dig deeper into these platforms, outsource some of the development to domain experts, or research alternatives. Now, you can have a minimal product that demonstrates the idea of your app with minimal coding.

You’ll notice that in such a short development time, our app looks pretty good. This is because it uses real data, not placeholders, and all links are available and meaningful. We managed to create a minimal usable product that respected its ecology (the source site) and returned value to the source site in the form of traffic.

Now we can learn how to use Scrapy crawlers to extract data in more complex scenarios.

This article is excerpted from Master Python crawler Framework Scrapy.

Python3 scrapy tutorial, a comprehensive analysis of web crawler technology implementation principle, through the crawling examples demonstrate the application of scrapy, covering from the desktop to the mobile side of the crawling, real-time crawling, all contents.

This book covers the basics of Scrapy, discussing how to extract data from arbitrary sources, how to clean it up, and how to use Python and third-party apis to tailor it to your needs. The book also explains how to efficiently feed crawled data into databases, search engines, and streaming data processing systems (such as Apache Spark). By the end of this book, you’ll know how to crawl data and use it in your own applications.

Read the original article and buy Master Python Crawler Framework Scrapy