Create a front-end offline log (1): IndexedDB

preface

This article from the theoretical and practical two aspects of the front-end offline log system is how to build, because the content is more, I will be divided into three parts to describe the entire log system design.

• Front-end data storage design – IndexedDB
• Server design – Node + Express/KOA and data compression – Deflate /gzip
• (Explore) WebRTC implementation log acquisition

Why do you need offline logging

As front-end projects become more complex, the importance of front-end logging becomes more and more important. We often use network request reporting for logging, such as badJS, CNZZ for Allies, etc. Network request reporting has the following pain points:

1. Poor support for weak or disconnected network environments.
2. It has high requirements for servers.
3. Continuous log reporting may waste network resources.

In the development of BadJS, to solve the above problems, we reported some non-error logs by whitelist (easy to query problems), that is, only users who meet certain conditions will report data. The whitelist method also brings some problems. For example, a user feedback page is blank, but we can’t find the current user’s log in the badJS background. It may be that the user’s network is not good, and the JS fails to load, but you can’t give convincing evidence.

Therefore, offline logs came into being. Offline logs almost solved all the above pain points and have been widely used in clients. Why hasn’t there been a proper offline application platform on the front end? One is because the previous technology was flawed. Before IndexedDB, there was almost no good way to store offline logs on the front end. Although localstorage can meet the requirements to a certain extent, but its problems are obvious.

1. Synchronizing reading and writing data causes some blocking.
2. Data size limits.
3. It’s essentially strings, which leads to a lot of manipulation of strings.
4. Key-value storage leads to more complex CURD operations.

It wasn’t until the advent of IndexedDB and the support of browsers that front-end offline logging was ripe for the taking.

With IndexedDB profile

Simply put, IndexedDB is a browser-based implementation of an event-based key-value pair database that supports indexing. Although IndexedDB cannot use SQL statements, its storage requires structured data (it can store text, files, and blobs), and queries are performed through Pointers generated by the index.

IndexedDB has the following advantages:

• Key value pair storage based on JavaScript objects, easy to use.
• Asynchronous API. This is important for the front end, meaning that accessing the database does not block the calling thread.
• Very large storage space. Theoretically, there is no maximum limit, and if it exceeds 50 MB, the user will need to confirm the request permission.
• Support for transactions, in which any operation in IndexedDB occurs.
• Support Web Workers. The synchronization API must be used with Web Workers.
• Same-origin policy to ensure security.
• Not bad compatibility

The basic concept

Because IndexedDB is a low-level API, you need to understand some basic concepts before you can use IndexedDB.

• IDBFactory: window.indexedDB, which provides database access operations.
• IDBOpenDBRequest: The result of indexeddb.open (), which represents an open database request.
• IDBDatabase: represents an IndexedDB database connection that can only be used to obtain a database transaction.
• IDBObjectStore: Object repository. An IDBDatabase can have multiple IdBobjectStores, similar to a table or a document in MongoDB.
• IDBTransaction: represents a transaction that is created by specifying the scope and type of access (read or write).
• IDBCursor: Database index used to traverse the object storage space.

Basic operation

First, open the database

const request = window.indexedDB.open('test'.1)
Copy the code

Test indicates the name of the database. If the database does not exist, create the database. The second argument represents the version of the database and is an integer. The default is 1.

Indexeddb.open () returns an IDBOpenDBRequest object which needs to process database opening operations through three events onError, onSuccess, onUpgradenneeded.

let db
const request = indexedDB.open('test')
request.onerror = function(event) {
  console.error('open indexedDB error')
}
request.onsuccess = function(event) {
  db = event.target.result
  console.log('open indexedDB success')
}
request.onupgradeneeded = function(event) {
  db = event.target.result
  console.log('upgrade indexedDB success')}Copy the code

Onupgradenneeded events are triggered when creating a new database or increasing the version number of an existing database (specifying a larger version number when the database is opened).

Onsuccess and onUpgradenneeded need to retrieve database instances via event.target.result.

Second, create a new database and table

After using the indexeddb.open () method, the database has been created, but there is nothing in it yet. We create the table with db.createObjectStore().

request.onupgradeneeded = function(event) {
  db = event.target.result
  console.log('upgrade indexedDB success')
  if(! db.objectStoreNames.contains('logs')) {
    const objectStore = db.createObjectStore('logs', { keyPath: 'id'}}})Copy the code

This code creates a table called logs with a primary key id. If you want to generate the primary key automatically, you can also write:

const objectStore = db.createObjectStore('logs', { autoIncrement: true })
Copy the code

keyPath & autoIncrement

keyPath	autoIncrement	describe
No	No	You can store any type of value in the objectStore, but when you want to add a value, you must provide a separate key argument.
Yes	No	Only JavaScript objects can be stored, and the objects must have a property with the same name as the key Path.
No	Yes	You can store values of any type. The keys are generated automatically.
Yes	Yes	Only JavaScript objects can be stored. Usually when a key is generated, the value of the generated key is stored in a property of the object with the same name as key Path. However, if such a property already exists, the value of the property is used as a key and no new key is generated.

Third, create an index

Create index with objectStore:

// Create an index to search by time. Times may be repeated, so unique cannot be used.
objectStore.createIndex('time_idx'.'time', { unique: false })
// Create an index using mailboxes. To ensure that mailboxes are not repeated, use unique
objectStore.createIndex("email"."email", { unique: true })
Copy the code

The three parameters of idbobject.createIndex () are index name, attribute corresponding to the index, and index attribute (whether the index is unique or not).

Step four, insert data

Inserting data into IndexedDB must be done through a transaction.

// Use the onComplete event of the transaction to ensure that the repository is created before the data is inserted
objectStore.transaction.oncomplete = function(event) {
  // Save the data to the newly created object repository
  const transaction = db.transaction('logs'.'readwrite')
  const store = transaction.objectStore('logs')

  store.add({
    id: 18.level: 20.time: new Date().getTime(),
    uin: 380034641.msg: 'xxxx'.version: 1})}Copy the code

When initializing IndexedDB, onUpgradenneeded events are triggered, and subsequent calls to the DB only need to fire onSuccess events. So we will encapsulate the CURD operation of the database as follows.

CURD

The new data

Suppose you have created a database named logs with keyPath set to ‘id’.

function addLog (db, data) {
  const transaction = db.transaction('logs'.'readwrite')
  const store = transaction.objectStore('logs')

  const request = store.add(data)

  request.onsuccess = function (e) {
    console.log('write log success')	
  }

  request.onerror = function (e) {
    console.error('write log fail')	
  }
}

addLog(db, {
  id: 1.level: 20.time: new Date().getTime(),
  uin: 380034641.msg: 'add new log'.version: 1
})
Copy the code

When you write the data, you specify the table name, create a transaction, get the IDBObjectStore object from objectStore, and insert it with the add method.

Update the data

The data can be updated through the PUT method of the IDBObjectStore object.

function updateLog (db, data) {
  const transaction = db.transaction('logs'.'readwrite')
  const store = transaction.objectStore('logs')

  const request = store.put(data)

  request.onsuccess = function (e) {
    console.log('update log success')	
  }

  request.onerror = function (e) {
    console.error('update log fail')
  }	
}

updateLog(db, {
  id: 1.level: 20.time: new Date().getTime(),
  uin: 380034641.msg: 'this is new log'.version: 1
})
Copy the code

IndexeDB uses the PUT method to update data, but only if there is a unique index. IndexeDB uses the UNIQUE index as the key to update data. The put method is similar to upsert in that if the value corresponding to unique does not exist, new data is inserted directly.

Read the data

The data can be read through the GET method of the IDBObjectStore object. As with updating data, reading data through the GET method requires a unique index. The data read is viewed in the onSuccess event.

function getLog (db, key) {
  const transaction = db.transaction('logs'.'readwrite')
  const store = transaction.objectStore('logs')

  const request = store.get(key)
  request.onsuccess = function (e) {
    console.log('get log success')
    console.log(e.target.result)
  }

  request.onerror = function (e) {
    console.error('get log fail')	
  }	
}

getLog(db, 1)
Copy the code

Delete the data

function deleteLog (db, key) {
  const transaction = db.transaction('logs'.'readwrite')
  const store = transaction.objectStore('logs')

  const request = store.delete(key)
  request.onsuccess = function (e) {
    console.log('delete log success')
  }

  request.onerror = function (e) {
    console.error('delete log fail')}}Copy the code

Deleting data will enter the onSuccess event even if the data does not exist.

Using the cursor

Because there is no SQL capability in IndexedDB, many times we need to find some data by traversal.

function getAllLogs (db) {
  const transaction = db.transaction('logs'.'readwrite')
  const store = transaction.objectStore('logs')

  const request = store.openCursor()

  request.onsuccess = function (e) {
    console.log('open cursor success')
    const cursor = event.target.result
    if (cursor && cursor.value) {
      console.log(cursor.value)	
      cursor.continue()
    }
  }

  request.onerror = function (e) {
    console.error('oepn cursor fail')}}Copy the code

Cursor traverses the table in a recursively like manner, entering the next loop with the cursor.continue() method.

Using the index

In the previous example, we used the primary key to retrieve the data, and the index allows us to find the data using other attributes.

Assume that the UIN index is created when the table is created.

objectStore.createIndex('uin_index'.'uin', { unique: false })
Copy the code

When querying data, you can use the uIN index mode:

function getLogByIndex (db) {
  const transaction = db.transaction('logs'.'readonly')
  const store = transaction.objectStore('logs')

  const index = store.index('uin_index')
  const request = index.get(380034641) // Make sure the data types are consistent

  request.onsuccess = function (e) {
    const result = e.target.result
    console.log(result)
  }
}
Copy the code

Only the first data that meets the query criteria can be queried using the above index query method. If you want to query more data, you need to combine cursor to operate the query.

function getAllLogsByIndex (db) {
  const transaction = db.transaction('logs'.'readonly')
  const store = transaction.objectStore('logs')

  const index = store.index('uin_index')
  const request = index.openCursor(IDBKeyRange.only(380034641)) // You can write values directly here

  request.onsuccess = function (e) {
    const cursor = event.target.result
    if (cursor && cursor.value) {
      console.log(cursor.value)	
      cursor.continue()
    }	
  }
}
Copy the code

• Idbkeyrange. only(val) Obtains only the specified data
• IDBKeyRange. LowerBuund (val, isOpened) or small data before val, isOpened is a closed interval, false is a val (closed interval), true is not contain val (open interval)
• IDBKeyRange. UpperBuund (val, isOpened) after val or large data, isOpened is open closed interval value, false is a val (closed interval), true is not contain val (open interval)
• Idbkeyrange. buund(val1, val2, isOpened1, isOpened2) Obtains the data between value1 and value2. IsOpened1 and isOpened2 are the left and right open and closed intervals, respectively

The above methods on the IDBKeyRange object are generally used to perform multi-mode query operations.

How to design a front-end offline database

In the previous example, you can see the prototype of the database. The table structure is as follows:

• From – Indicates the log source
• Id – Indicates the id of the report
• Level – Indicates the log level
• MSG – Indicates log messages
• Time – Indicates the time when a log is generated
• Uin – The unique id of the user
• Version – Log version

So these are the reports, but what about the interface?

A front-end offline logging system must provide at least the following five interfaces:

1. Example Clear the log interface. User logs are generated constantly and data cannot be accumulated indefinitely. Therefore, you can set a fixed number of days to clear expired logs at each startup of the system.
2. Write log interface. Asynchronous log writing allows the system to continuously write new logs.
3. Search for relevant interfaces. You can search for logs of the current user, logs of a fixed period, and logs of a fixed level. This facilitates the reporting and collecting end to obtain proper log information.
4. Data collation and compression interface. The amount of user logs may be very large. Therefore, you can effectively reduce the size of reported data by organizing and compressing the data.
5. Data reporting interface.

For details, see the offline module in wardJS-Report project.

Because of the wX. getStorage(Object Object) interface in the small program, it can also simulate the storage function of offline logs. This branch feat_MINIProgram is our solution for reporting small programs offline.

IndexedDB performance test

IndexedDB performs very well and is mostly asynchronous, so regular read and write operations have little additional impact on the product, even though they work in the browser.

The test environment

MAC 4GHz I7 /16GB DDR3 macOS Majave 10.14.2 Chrome 72.0

Data preparation

1w logs of 500 length are inserted.

The test time

DB connection time (10 times for average) : 3.5ms Insert a data (10 times for average) : less than 1 ms Connect DB -> insert data -> release the connection (10 times for average) : 4.3ms Insert 10 data at the same time (10 times for average) : less than 1 ms

Mobile testing

IPhone 6SP iOS 12.1.4 Safari

The test time

DB connection time (10 times for average) : 2.3ms Insert a data (10 times for average) : less than 1 ms Connect DB insert data release connection (10 times for average) : 2.3ms Insert 10 data at the same time (10 times for average) : less than 1 ms

The test results were quite strange, and the results on mobile devices were better than those on PC devices. It may have something to do with the browser, and at the time of testing, there were many apps on the computer that were not related to the Chrome TAB, which may also have an impact.

However, I didn’t do the test again, because the data was amazing enough, you can understand that we simply insert the data is hardly time consuming. Yes, that’s how powerful IndexedDB is.

conclusion

Because the content is too much, the article is divided into several pieces. This section briefly introduces the use of IndexedDB and its performance testing. In the construction of the front-end offline logging system, this is the most critical link, data storage, not only to ensure the reliability of the data, but also to maintain a certain performance, at the same time can not have side effects on the user’s normal operation, through our simple test, I think IndexedDB is fully capable of doing this job.

Follow the public number: [IVWEB community], weekly push boutique technology weekly.

• Weekly articles collection: weekly
• Team open source project: Feflow