/ DataStore introduction /

Jetpack DataStore is a new and improved data storage solution that allows the use of protocol buffers to store key-value pairs or typed objects.

DataStore stores data in an asynchronous, consistent transactional manner, overcoming some of the drawbacks of SharedPreferences (collectively referred to as SP).

DataStore is implemented based on Kotlin coroutines and Flow, and can migrate SP data, aiming to replace SP.

DataStore provides two different implementations: Preferences DataStore and Proto DataStore. The Preferences DataStore is used to store key-value pairs. Proto DataStore is used to store typed objects. The following examples are provided.

/ SharedPreferences Disadvantages /

Before DataStore came along, the most common storage method we used was undoubtedly SP, which was widely praised for its simplicity, ease of use. However, SP is defined by Google as lightweight storage. If less data is stored, there is no problem in using SP. When a large amount of data needs to be stored, SP may cause the following problems:

1. When SP loads data for the first time, it needs to load data in full. When there is a large amount of data, THE UI thread may be blocked, resulting in delay

2. SP read and write files are not type safe, and there is no mechanism for sending error signals, and it lacks transactional API

3. Commit ()/apply() operations may cause ANR problems:

Commit () is a synchronous commit, which directly performs IO operations on the UI main thread. If the write operation takes a long time, the UI thread will be blocked, resulting in ANR. Although apply() commits asynchronously, if the onStop() method of Activity/Service is executed, it will also wait for SP to finish writing asynchronously. ANR problems will also occur if the wait time is too long. For apply(), let’s expand:

SharedPreferencesImpl#EditorImpl. Java finally executes apply() :

public final CountDownLatch writtenToDiskLatch = new CountDownLatch(1); public void apply() { final MemoryCommitResult mcr = commitToMemory(); final Runnable awaitCommit = new Runnable() { public void run() { try { mcr.writtenToDiskLatch.await(); } catch (InterruptedException ignored) { } } }; / / 8.0 before QueuedWork. Add (awaitCommit); / / 8.0 after QueuedWork. AddFinisher (awaitCommit); / / asynchronous execution of disk writes SharedPreferencesImpl. Enclosing enqueueDiskWrite (MCR, postWriteRunnable); / /... Other...... }Copy the code

Construct a Runnable task called awaitCommit and add it to QueuedWork, which internally calls the countdownlatch.await () method to perform wait operations directly on the UI thread. It depends on when the task is executed in QueuedWork.

The QueuedWork class is implemented differently on Android8.0 and above:

8.0 queuedwork.java:

public class QueuedWork { private static final ConcurrentLinkedQueue<Runnable> sPendingWorkFinishers = new ConcurrentLinkedQueue<Runnable>(); public static void add(Runnable finisher) { sPendingWorkFinishers.add(finisher); } public static void waitToFinish() { Runnable toFinish; // Fetch the task from the queue: if the task is empty, it breaks out of the loop and the UI thread can continue executing; / / vice task is not empty, take out the tasks and execution, the actual execution CountDownLatch. Await (), namely the UI thread will be blocked waiting for a while ((toFinish = sPendingWorkFinishers. Poll ())! = null) { toFinish.run(); }} / /... Other...... }Copy the code

8.0 queuedwork.java:

public class QueuedWork { private static final LinkedList<Runnable> sFinishers = new LinkedList<>(); public static void waitToFinish() { Handler handler = getHandler(); StrictMode.ThreadPolicy oldPolicy = StrictMode.allowThreadDiskWrites(); ProcessPendingWork (); } finally { StrictMode.setThreadPolicy(oldPolicy); } try { while (true) { Runnable finisher; Synchronized (sLock) {// Remove tasks from the queue finisher = sFinishers. Poll (); } if (finisher == null) {break; if (finisher == null) {break; } // The task is not empty, execute countdownlatch.await (), that is, the UI thread will block waiting for finisher. Run (); } } finally { sCanDelay = true; }}}Copy the code

As you can see, whether before or after 8.0, waitToFinish() attempts to fetch tasks from the Runnable task queue, if any, and executes them directly, looking directly at where waitToFinish() was called:

ActivityThread.java

private void handleStopActivity(IBinder token, boolean show, int configChanges, int seq) { //...... Other...... QueuedWork.waitToFinish(); } private void handleStopService(IBinder token) { //...... Other...... QueuedWork.waitToFinish(); }Copy the code

You can see that the waitToFinish() method is called in both the handleStopActivity and handleStopService methods in the ActivityThread. Both Activity onStop() and Service onStop() wait for the write task to complete before continuing.

So even though apply() writes to disk asynchronously, if onStop() of the Activity/Service is executed at this time, it may still block the UI thread and cause ANR.

Voice-over: SP ANR problem caused in use process, can be optimized by some Hook means, such as a byte * * headlines today released ANR series of best practice – * * SharedPreference wait (mp.weixin.qq.com/s/kfF83UmsG…

/ DataStore uses /

DataStore advantage:

  • DataStore processes data updates on a transactional basis.
  • DataStore accesses data based on Kotlin Flow and operates asynchronously in dispatchers. IO by default to avoid blocking UI threads and handle exceptions when reading data.
  • Methods to persist data are not provided for apply() and commit().
  • Supports automatic migration of SP data to datastores ata time.

Preferences DataStore

Adding dependencies

Implementation 'androidx. Datastore: datastore - preferences: 1.0.0'Copy the code

Build the Preferences DataStore

Val context.bookdatastorepf: DataStore<Preferences> by preferencesDataStoreCopy the code

With the above code, we have successfully created the Preferences DataStore, where preferencesDataStore() is a top-level function with the following parameters:

  • Name: Name of the file to create the Preferences DataStore.
  • CorruptionHandler: if the DataStore trying to read the data, the data cannot be deserialized, throws androidx. The DataStore. Core. CorruptionException, will perform corruptionHandler at this time.
  • ProduceMigrations: SP generation is migrated to the Preferences DataStore. The ApplicationContext is passed as a parameter to these callbacks, and the migration runs before any access to the data.
  • Scope: coroutine scope. Default IO operations are performed in dispatchers. IO threads.

/data/data/ project package name /files/ creates a file named pf_datastore as follows:

You can see that the suffix is not XML, but.preferences_pb. One thing to note here: Do not write the above initialization code into the Activity, otherwise when you repeat the Activity and use Preferences DataStore, you will try to create a.preferences_pb file with the same name, because it has already been created once. Will directly thrown exception: * Java. Lang. An IllegalStateException: There are multiple DataStores active for the same file: xxx.You should either maintain your DataStore as a singleton or confirm that there is no two DataStore’s active on the same file (by confirming that the scope is Cancelled). * error classes in androidx. Datastore: datastore – core: 1.0.0 androidx/datastore/core/SingleProcessDataStore under:

internal val activeFiles = mutableSetOf<String>() file.absolutePath.let { synchronized(activeFilesLock) { check(! activeFiles.contains(it)) { "There are multiple DataStores active for the same file: $file. You should " + "either maintain your DataStore as a singleton or confirm that there is " + "no two DataStore's active on the same file (by confirming that the scope" + " is cancelled)." } activeFiles.add(it) } }Copy the code

Which is the file via file (applicationContext filesDir, “datastore / $fileName”) generated file, namely the Preferences datastore final address to operation in the disk file, ActiveFiles stores the path of the generated file in the memory. If the activeFiles already exist, an exception is thrown directly, that is, repeated creation is not allowed.

  • Save the data

First declare an entity class, BookModel:

data class BookModel(
    var name: String = "",
    var price: Float = 0f,
    var type: Type = Type.ENGLISH
)

enum class Type {
    MATH,
    CHINESE,
    ENGLISH
} 
Copy the code

Store operations performed in bookrepo.kt:

const val KEY_BOOK_NAME = "key_book_name" const val KEY_BOOK_PRICE = "key_book_price" const val KEY_BOOK_TYPE = Key<T> type object PreferenceKeys {val P_KEY_BOOK_NAME = stringenceskey (KEY_BOOK_NAME)  val P_KEY_BOOK_PRICE = floatPreferencesKey(KEY_BOOK_PRICE) val P_KEY_BOOK_TYPE = stringPreferencesKey(KEY_BOOK_TYPE) } /** * Preferences DataStore to save data */ suspend fun saveBookPf BookModel) { context.bookDataStorePf.edit { preferences -> preferences[PreferenceKeys.P_KEY_BOOK_NAME] = book.name preferences[PreferenceKeys.P_KEY_BOOK_PRICE] = book.price preferences[PreferenceKeys.P_KEY_BOOK_TYPE] = book.type.name } }Copy the code

In the Activity:

lifecycleScope.launch { val book = BookModel( name = "Hello Preferences DataStore", price = (1.. 10).random().tofloat (), // Where the price changes with each click, to show that the UI layer can listen for data changes at any time type = type.math) mbookrebo.savepfData (book)}Copy the code

Through bookDataStorePf. Edit (transform: Suspend (MutablePreferences) -> Unit) suspends and stores a function that accepts a transform block that updates state in the DataStore transactionally.

  • Take the data

/** * Preferences */ val bookPfFlow: Flow<BookModel> = context.bookDataStorePf.data.catch { exception -> // dataStore.data throws an IOException when an error is encountered when reading data if (exception is IOException) { emit(emptyPreferences()) } else { throw exception <T> val bookName = preferences[preferencekeys.p_key_book_name]? : "" val bookPrice = preferences[PreferenceKeys.P_KEY_BOOK_PRICE] ? : 0f val bookType = Type.valueOf(preferences[PreferenceKeys.P_KEY_BOOK_TYPE] ? : Type.MATH.name) return@map BookModel(bookName, bookPrice, bookType) }Copy the code

In the Activity:

lifecycleScope.launch {
    mBookViewModel.bookPfFlow.collect {
        mTvContentPf.text = it.toString()
    }
} 
Copy the code

Bookdatastorepf. data returns a Flow, which is then used to do a series of processing on the data. IOExceptions are thrown if an error occurs when reading data from a file. You can use the catch() operator before map() and issue emptyPreferences() when the exception thrown is IOException. If another type of exception occurs, rethrow it.

Note: Key< T> and T can only store Int, Long, Float, Double, Boolean, String, Set< String>. This limitation in androidx/datastore/preferences/core/PreferencesSerializer class participation serialized getValueProto () method:

private fun getValueProto(value: Any): Value { return when (value) { is Boolean -> Value.newBuilder().setBoolean(value).build() is Float -> Value.newBuilder().setFloat(value).build() is Double -> Value.newBuilder().setDouble(value).build() is Int -> Value.newBuilder().setInteger(value).build() is Long -> Value.newBuilder().setLong(value).build() is String -> Value.newBuilder().setString(value).build() is Set<*> -> @Suppress("UNCHECKED_CAST") Value.newBuilder().setStringSet( NewBuilder ().addallstrings (value as Set<String>)).build() Else -> throw IllegalStateException("PreferencesSerializer does not support type: ${value.javaclass. Name}")}}Copy the code

As you can see in the last else logic, if it’s not of the above type, it throws an exception directly. Because the Key is the Preferences. The Key type < T >, the default system help us package a layer, is located in androidx. Datastore. Preferences. Core. PreferencesKeys. Kt:

public fun intPreferencesKey(name: String): Preferences.Key<Int> = Preferences.Key(name)

public fun doublePreferencesKey(name: String): Preferences.Key<Double> = Preferences.Key(name)

public fun stringPreferencesKey(name: String): Preferences.Key<String> = Preferences.Key(name)

public fun booleanPreferencesKey(name: String): Preferences.Key<Boolean> = Preferences.Key(name)

public fun floatPreferencesKey(name: String): Preferences.Key<Float> = Preferences.Key(name)

public fun longPreferencesKey(name: String): Preferences.Key<Long> = Preferences.Key(name)

public fun stringSetPreferencesKey(name: String): Preferences.Key<Set<String>> =
    Preferences.Key(name) 
Copy the code

Because the above declarations are in the top-level function, they can be used directly. For example, if we want to declare a preference. Key< T>, we can declare it as follows:

val P_KEY_NAME: Preferences.Key<String> = stringPreferencesKey("key")
Copy the code

  • Migrate the SP to the Preferences DataStore

If you want to migrate SP, add produceMigrations parameter in the Preferences DataStore build link as follows:

//SharedPreference file const val BOOK_PREFERENCES_NAME = "book_preferences" val context.bookdatastorepf: DataStore<Preferences> by preferencesDataStore( name = "pf_datastore", // Migrate SP to Preference DataStore, produceMigrations = {context -> listOf(SharedPreferencesMigration(context, BOOK_PREFERENCES_NAME)) } )Copy the code

The SP file will be deleted from the Preferences DataStore as follows:

Proto DataStore

One disadvantage of the SP and Preferences DataStore is the inability to define the schema to ensure that the correct data types are used when accessing keys. Proto DataStore available Protocol Buffers Protocol buffer (developers.google.com/protocol-bu… DataStore can know the type of storage and provide the type without using a key.

  • Adding dependencies

To use Proto DataStore and make the protocol buffer generate code for our architecture, we need to introduce the protobuf plugin in build.gradle:

plugins { ... "Version" id "com. Google. Protobuf 0.8.17"} android {/ /... Other Configurations.................. SourceSets {main {java.srcdirs = [' SRC /main/ Java '] proto {// Specify proto source file address srcDir 'SRC /main/protobuf' include '**/*.protobuf'}}} // Proto buffer protocol buffer-related configurations for DataStore protobuf {protoc {// Protoc version See also: https://repo1.maven.org/maven2/com/google/protobuf/protoc/ an artifact = "com. Google. Protobuf: protoc: 3.18.0"} / / Generates  the java Protobuf-lite code for the Protobufs in this project. See // https://github.com/google/protobuf-gradle-plugin#customizing-protobuf-compilation // for more information. GenerateProtoTasks {all().each {task -> task.builtins {Java {option 'lite'}}}} // Modify the location where Java classes are generated by default $buildDir/generated/source/proto generatedFilesBaseDir = "$projectDir/src/main/generated" } } dependencies { api 'androidx. Datastore: datastore: 1.0.0' API "com. Google. Protobuf: protobuf - javalite: 3.18.0"... }Copy the code

The number of libraries that need to be configured or imported seems quite large, so consider putting these configurations into a separate Module.

Define and use protobuf objects

Once you define how the data is structured, the compiler generates source code that makes it easy to write and read structured data. Protobuf: SRC /main/protobuf: proto: SRC /main/protobuf: proto: SRC /main/protobuf

Book.proto file contents:

Syntax = "proto3" must be specified on the first line. // javA_package: specifies the package name of the Java class generated by proto file option java_Package = "org.ninetriPods.mq.study "; // javA_outer_className: Specifies the name of the Java class generated by the proto file. Option javA_outer_className = "BookProto"; enum Type { MATH = 0; CHINESE = 1; ENGLISH = 2; } message Book { string name = 1; Float price = 2; Type Type = 3; / / type}Copy the code

After the above code is written, execute Build -> ReBuild Project, and the corresponding Java code will be generated in the path of the generatedFilesBaseDir configuration, as follows:

The serializer defines how to access the data types we define in the PROto file. If there is no data on disk, the serializer also defines a default return value. Here we create a serializer called BookSerializer:

object BookSerializer : Serializer<BookProto.Book> {
    override val defaultValue: BookProto.Book = BookProto.Book.getDefaultInstance()

    override suspend fun readFrom(input: InputStream): BookProto.Book {
        try {
            return BookProto.Book.parseFrom(input)
        } catch (exception: InvalidProtocolBufferException) {
            throw CorruptionException("Cannot read proto.", exception)
        }
    }

    override suspend fun writeTo(t: BookProto.Book, output: OutputStream) {
        t.writeTo(output)
    }
} 
Copy the code

Where, bookproto.book is code generated through the protocol buffer. If the Bookproto.book object or related method is not found, the project can be cleaned and rebuilt to ensure that the protocol buffer generates the object.

  • Build Proto DataStore

// Construct Proto DataStore val context.bookdatastorept: DataStore<BookProto.Book> by dataStore( fileName = "BookProto.pb", serializer = BookSerializer)Copy the code

DataStore is a top-level function. The following parameters can be passed:

  • FileName: indicates the name of the file to create the Proto DataStore.
  • Serializer: The Serializer serializer defines how to access formatted data.
  • CorruptionHandler: if the DataStore trying to read the data, the data cannot be deserialized, throw androidx. DataStore. Core. CorruptionException, call the corruptionHandler.
  • ProduceMigrations: This parameter is executed when SP is migrated to the Proto DataStore. The ApplicationContext is passed as a parameter to these callbacks, and the migration runs before any access to the data
  • Scope: coroutine scope. Default IO operations are performed in dispatchers. IO threads.

/data/data/ project package name /files/ create a file named bookproto.pb as follows:

Save the data

LifecycleScope. Launch {/ / build BookProto. Book val bookInfo = BookProto. Book. The getDefaultInstance () toBuilder () .setName("Hello Proto DataStore") .setPrice(20f) .setType(BookProto.Type.ENGLISH) .build() bookDataStorePt.updateData { bookInfo } }Copy the code

Proto DataStore provides a suspended function, datastore.updateData (), to store data, and when storage is complete, the coroutine completes.

Take the data

/** * Proto DataStore */ val bookProtoFlow: Flow<BookProto.Book> = context.bookDataStorePt.data .catch { exception -> if (exception is IOException) { Emit (BookProto. Book. GetDefaultInstance ())} else {throw exception}} / / Activity lifecycleScope launch { mBookViewModel.bookProtoFlow.collect { mTvContentPt.text = it.toString() } }Copy the code

The Proto DataStore retrives data in the same way as the Preferences DataStore.

  • The SP is migrated to the Proto DataStore

// Construct Proto DataStore val context.bookdatastorept: DataStore<BookProto.Book> by dataStore( fileName = "BookProto.pb", serializer = BookSerializer, // Migrate SP to Proto DataStore, produceMigrations = {context -> listOf( androidx.datastore.migrations.SharedPreferencesMigration( context, BOOK_PREFERENCES_NAME ) { sharedPrefs: SharedPreferencesView, currentData: bookProto.book -> String = sharedPrefs.getString(KEY_BOOK_NAME, "") ? : "" val bookPrice: Float = sharedPrefs.getFloat(KEY_BOOK_PRICE, 0f) val typeStr = sharedPrefs.getString(KEY_BOOK_TYPE, BookProto.Type.MATH.name) val bookType: BookProto.Type = BookProto.Type.valueOf(typeStr ?: Currentdata.tobuilder ().setName(bookName).setprice (bookPrice) .setType(bookType) .build() } ) } )Copy the code

Proto DataStore defines SharedPreferencesMigration class. Migrate specifies the following two parameters:

  • SharedPreferencesView: Can be used to retrieve data from SharedPreferences
  • Bookproto. Book: current data

Similarly, if produceMigrations are passed during creation, SP files will be migrated to the Proto DataStore and deleted after migration.

Here also need to note that the Preferences DataStore, Proto DataStore used to perform the migration during SharedPreferencesMigration classes, but these two places using the corresponding package name is different, Such as Proto DataStore package name path is androidx DataStore. Migrations. SharedPreferencesMigration, when writing them in a file and pay attention to one of them to use full path.

Summary /

SP and DataStore:

Follow me and share my knowledge every day