Due to the business characteristics of our company, we need the APP to support the normal use of APP functions even when there is no network

Therefore, in order to make an APP implemented with the Web front-end to be able to run the program normally without network, the offline solution needs to achieve several key points:

  • Code offline, update
  • Data download, upload, update

This article would like to talk about, we in the offline application of the implementation scheme

Offline and updated code

Web applications, whether web pages or H5, are usually online services, and the code needs to be deployed to an online server

But offline applications can not only rely on the network, in the absence of network scenarios, also need to find ways to make the user’s client can obtain the program code, which needs to rely on the ability of native APP development

We adopt the cross-platform APP solution of Web + Cordova:

Cordova parses and runs web applications by running a WebView on a native APP. Web applications invoke native capabilities through Cordova plug-ins

When loading Web code, WebView supports loading local Web files, by which I mean, of course, the user’s mobile device

With the ability of APP native development, when the APP is packaged, the Web code is packaged into the APP installation package (APK/IPA), so that users can get the Web code locally after installing the APP

Take Android as an example, we can let Android developers package our Web code into assets, which is a directory that can store resource files, and the files inside will not be compressed and encoded when packaging

After packaging, the Android APP is an APK installation package, you can change the suffix to zip, and then unzip to see that our web files are already in the installation package (assets/ WWW) :

During APP installation, the mobile phone will decompress these Web code files to the private directory of the APP. When the APP is running, the WebView can load the local index. HTML of the user’s mobile phone, and the resource files are also loaded through the relative path. In this way, web code files can be taken offline and applications can be loaded and run directly independent of the network

For example, on Android, WebView uses the file protocol to load the local index.html file stored in the APP’s private directory:

This allows the code to be implemented offline, but it also poses a problem: How can the code be updated? That is, how to send version?

There are two options:

  1. Release a new version of the APP and package the new Web code files into the APP
  2. Hot update mechanism: Download new Web code packages, replace local Web resources with Cordova plug-ins, and reload web applications

Both schemes can achieve the effect of updating APP Web code files, but they have their own advantages and disadvantages and different application scenarios:

  • The first solution is that it relies on the launching and updating mechanism of the native APP, which leads to poor flexibility and timeliness. This solution is usually used only when the Cordova plug-in changes and the native development needs to be released

  • The second solution is that the front end implements hot update detection and download mechanism in the program, and then updates the new Web code file locally with the help of the Cordova plug-in. This solution has high flexibility and timeliness, and can be used for daily business iteration release

The two scenarios can simply be called: big release updates and hot updates

In fact, to be more precise, the hot update mechanism to be implemented by the front end includes two aspects: Web hot update and DB hot update, the former is to update the Web code file, the latter is to update the DB database table

Db hot updates can be covered in the next section on data, but web hot updates will be covered first. There are two key points to implement this mechanism: detecting and downloading updates

Therefore, a version management mechanism is needed to compare the local version of the user with the latest version of the server

A versioning mechanism can include many aspects of detection, one of the most important being the comparison of code versions

Consider a packet capture request:

The common detection logic is to compare the version. The version is composed of a large version (for example, 6.5.0) of the APP combined with a hot update version (for example, 00) of the Web. The former can be obtained through the Cordova plug-in, while the latter needs to be stored and maintained by itself and can be stored in localStorage

When a hot web update is detected, the interface will send the download address of the new version of the Web code file, which can be downloaded and updated locally with the help of the Cordova plug-in

Using the Cordova plug-in, you can download web code files using the native download capability. After the download is complete, you can use the Cordova plug-in to decompress the code update package to the directory storing the Web code files of the APP

Since the download is the download capability of the native APP called, this packet request cannot be caught in inspect debugging. If you want to capture the downloaded update packet, you can use the proxy to capture it (whistle is recommended).

In addition, the unzipping operation defaults to the same name replacement principle, which means that no historical files are deleted, so with this feature we can even do incremental updates by publishing only a few files that have changed in this iteration

If your phone has root permission, then you can go to the APP private directory (data/data/{package name}) to check:

This is the general idea of web hot updates

The AppCloud platform in the figure above is a project developed by our company specially for front-end developers to complete packaging behavior directly through the platform operation

Of course, there are many details about web hot updates, such as when to initiate update detection, how to package and compress the upload to a dedicated file server after local development, and so on. These aspects are usually developed according to different project and team specifications and are not developed

In this way, the offline of the code ensures that when the APP runs, the code file is already on the user’s local device, which can naturally support offline running

The Web hot update mechanism ensures that the user device can update the latest version of the Web code file in time

The combination of the two will solve the problem of the deployment and distribution of Web code, which is just the basis for the Web program to run offline on the user device. After the program runs, the interaction and data processing that business functions depend on is also the key to whether the application can support offline operation, so let’s continue to look at the processing of data

Data download, upload and update

The last section talked about the offline and hot update mechanism of the code, which is the basis to ensure that the APP can realize offline application. On this basis, each business group will develop and realize their own business offline function

The normal operation of business functions is supported by page data, so the key of offline is the processing of various data:

  • Offline of page data
  • Upload local offline data
  • Download updates of new data on the server

In a nutshell, that is: offline data download, upload and update mechanism

Let’s start with the first one

Offline of page data

By offline, we mean storage, and when the data for page rendering is taken locally, the interaction of the page is not dependent on the network

So, here’s the question:

  • Where should I deposit it?
  • What data do you store?

Storage is naturally to persistent storage, but for the front end, the ability of persistent storage is not very strong, localstorage has size limits; IndexedDB is a non-relational database and has compatibility problems, so we use Cordova to invoke native capabilities to store it in SQLite db files

Sqlite is a database storage solution supported by native APP. The syntax is basically the same as that of mysql. After storing, it is a DB file. Or take the DB file to the PC with the help of database software such as Navicat, or even if the phone has root permission, you can directly use sqlite3 command to debug:

Where to save the solution, the next is to think about what data to save?

Normally, for online Web applications, the front-end interface data is mostly real-time JSON data returned from THE API interface. Therefore, if we think from the perspective of interface data cache, we can also solve the problem of interface presentation, but only support the interface scene purely for display. Such as to-do pages, workbenches, and so on, only show interface scenarios that do not rely on business data

For these interface data caching scenarios, the storage can also be simply stored in localStorage, or you can directly experience the new features of ServiceWorker

However, some interfaces will change with the data generated by user operations. The interfaces are not simply displayed, but also have business interaction. In these scenarios, the interface data caching scheme is not feasible

For this type of scenario, in another Angle to think, now that chose to use sqlite in front of the relational database to store data, then the APP itself is actually have a similar backend role responsibilities, that the data you want to deposit, just like the back-end database table to store all kinds of business, and the list of nature, that is, some of the original data

It is important to note that some oF the apis used in such offline applications are somewhat different from those used in normal online Web applications. These apis can be regarded purely as the role of synchronizing back-end database tables with the local table data of the APP. The data returned by the interface is mostly SQL statements. Real data is inserted into db after SQL execution

Therefore, for this kind of offline application APP, its architecture will be relatively complicated. After all, in addition to the view layer, it also needs to take out the original data from DB, process the data through a series of business logic processing, and convert the data into the data structure required by the interface. Since the API is largely absent, the APP itself is responsible for some of the back-end roles it previously handled

To summarize briefly, the offline of page data needs different processing according to different scenarios:

  • For some displayed pages, such as to-do pages and workstations, interface data can be directly stored in localStorage. Interface data can be accessed when there is a network, and cache data can be accessed when there is no network
  • For pages that support user interaction, you need to download the data of each service table to SQLite DB in advance. No matter whether there is a network or not, you only fetch and save data from DB, and then process the service logic by yourself and throw it to the interface

Upload local offline data

As mentioned above, the interface that can interactively generate data, whether there is a network or not, only takes the local DB to fetch and store data processing, which means that, with the user’s continuous operation, the data in the local DB will change

Therefore, the offline data generated by user operations must be uploaded to the server. There are several aspects to consider in this regard:

  • When will it be uploaded?
  • What data is uploaded?

There are usually two types of upload timing: automatic upload and manual upload. Automatic upload needs to consider many scenarios, such as whether there is a network or not, whether data is valid or not, etc. Therefore, manual upload is more preferred in implementation

Manual upload requires a user interaction time to trigger the upload, which depends on the design of different service features

So, when you upload, what data should you upload?

The data generated by the user’s actions will certainly be inserted or updated into the local data table, which will be mixed with the native data, and the reported data will certainly not be the full data, so there needs to be a local mechanism to record the incremental data generated by the user’s actions

So in the database, in addition to some business tables, there will be a delta table:

The table structure consists of the type, operation, releation_id fields, and others, but these fields are required and cover most scenarios

The type and operation fields are used to indicate which type of data changes are generated by the operation. For example, type=add, operation= problem, indicating that a new problem is registered, and the related business table such as xxx_Problem, Releation_id is associated with the new data IDS in the business tables, so that the increments can be retrieved from each business table based on the increments

After the delta data is uploaded, the local system removes the delta data from the delta table to prevent repeated upload

This is the basic offline data upload processing, of course, there are a lot of details, such as performance: how do batch upload, how do background queue upload and so on

Another example is data processing: image file data must first be uploaded to the file server, and then backfill URL, and finally SQL to assemble into the format required by the interface to upload

For example, in the upload process, the scene processing and prompt when each step fails

These all depend on the product business characteristics to design and implement

Download updates of new data on the server

Data can be uploaded and downloaded in two scenarios. One is the initial download scenario when there is no local data, and the other is the synchronization and update of server data

Here, too, we need to consider:

  • Update what data?

  • When will the data be updated?

First of all, we need to know what data needs to be downloaded and updated. From the perspective of offline storage database DB, it can be divided into two types: table structure data and table data

The db hot update

Because the page data basically exists in the local database table, since it is a database, then there will be a library, library version management scenarios exist

Therefore, you need to have a database version management mechanism, also known as DB hot update

When building the database, it is best to build the database according to the user. Different users have different databases, so as to ensure that the use of switching users in the APP does not affect each other

When the user’s database is not found locally, it is natural to build the database first. SQL file, which is the latest structure of each table to build the table SQL, build the database, will execute this init. SQL to build the database. When a local database exists, you need to check whether the DB hot update is required

There are two methods for hot DB update: one is to upgrade THE SQL database in back-end maintenance, and the other is to upgrade the SQL database in front-end maintenance

If the DATABASE is used for back-end maintenance to upgrade SQL, a detection mechanism is required. Similar to web hot update, version comparison is required. The database will create a _version table, which will record the web version of the last database upgrade, and then at the end of each web hot update will be compared. If the version is inconsistent, the database will get the backend of the upgrade between the two versions SQL:

If it is to go pure front-end maintenance database upgrade SQL, then the front-end in addition to maintaining a latest database upgrade SQL, but also need to determine whether the need for database upgrade detection, can be simple to determine whether there is the field or table of ideas, can also make a version detection mechanism

Data download and update

Table structure data download and update solved, then can service table data download and update, for table data, we also need to clear some of its classification

From the perspective of our company’s business, table data can be roughly divided into two categories: basic data and business data

Basic data can be understood as: all users are the same, downloaded to the local basically unchanged data

Service data refers to data that is continuously added and changed based on the original data as different users perform operations

Different types of data download and update at different times and in different ways

For users to install and use the APP for the first time, they need to download the data before using the APP function normally, because the page data is dependent on the local DB database

Therefore, the basic data irrelevant to business naturally needs to be downloaded first. Usually, the download will be detected when the APP is opened and the home page is entered, and then the user can manually select the specified business data to download

This is the processing of data download. Whether it is basic data or business data, the criterion is that it needs to be downloaded when there is no such data locally for the first time. The difference is that basic data is automatically detected and downloaded by the program, while business data is usually manually triggered by the user

The reason for this is that the amount of service data is usually very large, and the current services only need a certain part of the service. Therefore, users can decide which part of the service data to download by themselves, which can avoid bad user experience caused by downloading a large amount of irrelevant data

Data download scenario is relatively simple, need to pay special attention to deal with the update of data, since the update, need to have a mechanism to detect the update

Detection mechanism is achieved by timestamp and version number, the timestamp mechanism can let the server know whether user local current data need to be updated, the version number is compatible to server processing, because there may be different users using different versions of the APP, table structure may be different, so the need according to the version control

When data needs to be updated, the server can deliver new data. In this case, there are two scenarios: full update or incremental update. For service data, incremental update is usually performed because the volume may be large. Basic data can choose to use full update according to the scene, on the one hand, the implementation is simple, and on the other hand, the amount of data is not large

But one thing to note, for the business data, if the user local offline data, so only after users upload to trigger the update of data, that is because the business data are all at the same time to modify the same data, it will cause the data conflict, and the data conflict once didn’t handle, might lead to a user local offline data is missing, Exceptions can occur because of overwriting or because of key-value conflicts

In this scenario, the best way is to let users upload data first, and the server resolves conflicts and backs up data based on the service scenario. In this way, the problem of user data loss can be minimized

conclusion

The offline solution of APP is to take all data offline during APP running, but the data involves page code and page data. The offline and updating mechanism of page code can be implemented by using the Cordova plug-in. The principle is to download the Web code file and store it on the user’s local device. The WebView then loads the local resources, combined with the Web hot update mechanism

Page available offline data requires each business group according to their respective business scenarios to determine what are the available offline data, what is the time to download data, what is the location of the storage, storage is what data, offline data how to upload, upload what time, what data upload, data update time, how to update, update the data, and so on

Different business needs, data download, upload and update in the specific implementation of some differences, but the knowledge involved is nothing more than: SQLite, delta table, timing stamps and so on