preface

Google has added a new Jetpack member, DataStore, to replace SharedPreferences, and Jetpack DataStore can be implemented in two ways:

  • Proto DataStore: Stores the typed objects of the class. The buffers are used to serialize the objects and store them locally
  • Preferences DataStore: Stored locally as key-value pairs, similar to SharedPreferences

Jetpack DataStore describes how SharedPreferences have those pits, And what the Preferences DataStore solves for us.

Proto DataStore uses protocol buffers to serialize objects locally, so you need to install Protobuf to compile Proto files. Protobuf compilation is roughly divided into Gradle plugin compilation and command line compilation. Both methods have been published on the blog, please click the link below to check them out.

  • Protobuf | Gradle plug-in installation compile proto file
  • Protobuf | how to compile on ubuntu installation Protobuf proto file
  • Protobuf | how to install Protobuf compile proto files on MAC

Since it is mainly developed on MAC and Ubuntu, only these two command line compilation methods are provided. If you are developing on Win, you can use Gradle plug-in compilation method.

An example of this article has been uploaded to GitHub. Go to the repository AndroidX-Jetpack-practice /DataStoreSimple and switch to the datastore_proto branch.

GitHub address: https://github.com/hi-dhl/AndroidX-Jetpack-Practice

In this article you will learn the following:

  • Why Proto DataStore?
  • Serialization of what? What is object serialization? What is the serialization of data?
  • What is a Protocol Buffer? Why is it needed? What problem has it solved for us?
  • How do I use Proto DataStore in a project?
  • How do I migrate SharedPreferences to Proto DataStore?
  • How to choose between Proto2 and Proto3 syntax?
  • Common Proto3 syntax parsing?
  • What is MAD Skills?

Why Proto DataStore

What is the difference between Preference DataStore and Proto DataStore?

  • Preference DataStore is primarily designed to address performance issues associated with SharedPreferences
  • Proto DataStore is more flexible and supports more types than Preference DataStore
    • Preference DataStore supportsInt 、 Long 、 Boolean 、 Float 、 String
    • Protocol Buffers are of the types supported by Proto DataStore
  • Preference DataStore stores key-value data as XML, which is very readable
  • Proto DataStore uses binary encoding compression, which is smaller and faster than XML

From the source point of view

If you are not familiar with the source code, you can ignore it, continue to read, and then go back and look at it.

  • Preference DataStore source code defines a proto file, using PreferencesSerializer to map each pair of key-value data to the proto file defined by the message type. The proto file contains the following contents:

    syntax = "proto2"; . message PreferenceMap { map<string, Value> preferences = 1; } message Value { oneof valueName { bool boolean = 1; float float = 2; int32 integer = 3; int64 long = 4; string string = 5; double double = 7; }}Copy the code

    The proto2 syntax is used in DataStore to Map key-value data from XML to Map, and only Int, Long, Boolean, Float, and String are defined in proto files.

  • Proto DataStore we can customize Proto files and implement the Serializer

    interface, so it is more flexible and supports more types

Proto DataStore uses binary encoding compression for protocol buffers to store object serialization locally. Let’s start with some basic concepts that will give us a better understanding of what’s to come.

serialization

Serialization: To convert an object into a state that can be stored or transferred, either locally or over Bluetooth or over a network. Serialization can be divided into object serialization and data serialization.

Serialization of objects

Java object serialization converts an object stored in memory into a transportable sequence of bytes that can be easily transferred over Bluetooth, over a network, or stored locally. The process of restoring byte sequences to Java objects stored in memory is called deserialization.

Object serialization can be implemented in Android through Serializable and Parcelable.

Serializable

Serializable is a Java native serialization method, which mainly implements object serialization and deserialization through ObjectInputStream and ObjectOutputStream, but uses a lot of reflection and temporary variables in the whole process, which frequently triggers GC. Serialization performance is very poor, but the implementation is very simple, look at ObjectInputStream and ObjectOutputStream source code has a lot of reflection.

ObjectOutputStream.java private void writeObject0(Object obj, boolean unshared) throws IOException{ ...... Class<? > cl = obj.getClass(); . } ObjectInputStream.java void readFields() throws IOException { ...... ObjectStreamField[] fields = desc.getFields(false); for (int i = 0; i < objVals.length; i++) { objVals[i] = readObject0(fields[numPrimFields + i].isUnshared()); objHandles[i] = passHandle; }... }Copy the code

There is a lot of cross-process communication in Android, and due to Serializable poor performance, Android needs a more lightweight and efficient object serialization and deserialization mechanism, hence Parcelable.

Parcelable

Parcelable solves the problem of poor cross-process communication in Android, and is much faster than Serializable since both writes and reads are custom serialized storage. WriteToParcel () method and describeContents() method to implement, do not need to use reflection to infer it, so the performance is improved, but more complex than Serializable.

To address complexity, AndroidStudio also has plug-ins to simplify the process. For Java, you can use the Android Parcelable Code Generator plug-in. You can quickly serialize Parcelable using @parcelize annotations in Kotlin.

Summarize the differences between Serializable and Parcelable in a table

Data serialization

Object serialization records a lot of information, including Class information, inheritance relationship information, variable information, and so on. However, data serialization does not have as much information as object serialization. The common methods of data serialization include JSON, Protocol Buffers, and FlatBuffers.

  • JSON: Interactive data format is a lightweight, cross-platform and cross-language, widely used in network transmission, JSON very readable, but the serialization and deserialization performance is the worst, the parsing process, to produce large Numbers of temporary variables, will often trigger GC, in order to guarantee the readability, and no binary compression, Performance is poor when there is a large amount of data.

  • Protocol Buffers. It is Google’s open source cross-language coding protocol, which can be used in C++, C#, Dart, Go, Java, Python and other languages. Almost all RPCS at Google use this protocol, which uses binary coding compression, smaller size, faster than JSON. But the downside is readability is sacrificed

    RPC refers to cross-process remote invocation, where one process calls another process’s methods.

  • FlatBuffers: Buffers, like Protocol Buffers, are an open-source cross-platform data serialization library for C++, C#, Go, Java, JavaScript, PHP, Python, and more. The space and time complexity is better than the other methods, no extra memory is required during use, and almost the size of the original data is in memory, but the disadvantage is that readability is sacrificed

Finally, we use a graph to analyze the serialization and de-sequence performance of JSON, Protocol Buffers and FlatBuffers. The data comes from JSON vs Protocol Buffers vs FlatBuffers

FlatBuffers and Protocol Buffers outperform JSON in both serialization and de-sequencing. FlatBuffers were originally developed by Google for games and other performance-demanding applications. Let’s take a look at the Protocol Buffer.

Protocol Buffer

Protocol Buffer (Protobuf for short) is an open-source cross-language coding Protocol that can be used in C++, C#, Dart, Go, Java, Python, and other languages. Almost all RPCS inside Google use this protocol, which uses binary encoding compression and is smaller and faster than JSON.

Proto3.0.0 Release Note: When Protocol Buffers were first open sourced, it implemented protocol Buffers language version 2 (called Proto2), which is why the number of versions starts from v2.0.0, and from v3.0.0, a new language version (Proto3) is introduced, Older versions (Proto2) continue to be supported. So so far there are two versions of Protobuf: Proto2 and Proto3.

Which version should Proto2 and Proto3 learn?

Proto3 simplifies the syntax of Proto2, making development more efficient, and therefore introducing version incompatibinities. Since stable versions of Proto3 were only released in 2019, companies that used Protocol Buffer prior to that, The proto2 syntax is used in DataStore, so both the proto2 syntax and the Proto3 syntax are used at the same time.

Proto3 syntax can be learned directly for beginners, in order to adapt to the changes of technology iteration, after mastering proto3 syntax, you can learn the Proto2 syntax and the difference between Proto3 and Proto2 syntax, so that you can better understand other open source projects.

To avoid confusion between proto3 and Proto2 grammars, this article will only examine proto3 grammars. Once we understand these basic concepts, we will examine how to use Proto DataStore in a project.

How do I use Proto DataStore in a project

Proto DataStore, like the Preferences DataStore, is primarily used in the Repository layer within the MVVM, making it very simple to use Proto DataStore in projects.

1. Add the Proto DataStore dependency

In the app module build.gradle file, add the following dependencies

/ / Proto DataStore implementation "androidx DataStore: DataStore - core: 1.0.0 - alpha01" / / protobuf implementation "Com. Google. Protobuf: protobuf - javalite: 3.10.0"Copy the code

Google recommends using Protobuf-Javalite for Android development because it’s much smaller and has a lot of optimizations.

After the dependency is added, a new proto file needs to be created. In the example project of this article, a new common-Protobuf module is created, and the new Person. proto file is placed in the SRC /main/proto directory of the common-Protobuf module.

The default path for storing proto files is SRC /main/proto. You can also change the default path for storing proto files by modifying gradle configuration

In the common-Protobuf module, build. Gradle file, add the following dependencies

Implementation "com. Google. Protobuf: protobuf - javalite: 3.10.0"Copy the code

2. NewPerson.protoFile, add the following

syntax = "proto3"; option java_package = "com.hi.dhl.datastore.protobuf"; option java_outer_classname = "PersonProtos"; Message Person {// format: field type + field name + field number string name = 1; }Copy the code
  • syntax: Specifies the version of a protobuf. If proto2 is not specified by default,It must be the first line of the.proto file except for empty lines and comment content
  • option: Indicates an optional field
    • java_package: Specifies the package name from which the Java classes will be generated
    • java_outer_classname: Specifies the name of the generated Java class
  • messageContains a string field (name).Pay attention to ：=The number is followed by a field number
  • Each field consists of three parts: field type + field name + field number. In Java, each field is compiled into a Java object

All you need to know here is the Proto syntax, which will be covered in more detail later in this article.

3. Execute protoc to compile the proto file

For example, run the following command to output the Java file. If the Gradle plug-in is configured, you can ignore this step and directly click Build -> Rebuild Project to generate the Java file.

protoc --java_out=./src/main/java -I=./src/main/proto  ./src/main/proto/*.proto
Copy the code
  • --java_out: Specifies the directory where the output Java files are located
  • -I: Specifies the directory where the proto file resides
  • *.proto: said in-IThe specified directory to find.protoClosing document

4. Build a DataStore

object PersonSerializer : Serializer<PersonProtos.Person> { override fun readFrom(input: InputStream): PersonProtos. Person {try {return PersonProtos. Person. ParseFrom (input) / / is the compiler automatically generated, To read and parse the input message} catch (exception: Exception) { throw CorruptionException("Cannot read proto.", exception) } } override fun writeTo(t: Personprotos. Person, output: OutputStream) = t.ritto (output) // t.Ritto (output) is automatically generated by the compiler to write serialized messages}Copy the code
  • To achieve theSerializer<T> Interface to tell the DataStore how to read and write data from a proto file
  • PersonProtos.PersonIs a Java class generated by compiling a proto file
  • Person.parseFrom(input)Is automatically generated by the compiler to read and parse input messages
  • t.writeTo(output)Is automatically generated by the compiler to write serialized messages

5. Read data from the Proto DataStore

fun readData(): Flow<PersonProtos.Person> {
    return protoDataStore.data
        .catch {
            if (it is IOException) {
                it.printStackTrace()
                emit(PersonProtos.Person.getDefaultInstance())
            } else {
                throw it
            }
        }
}
Copy the code
  • DataStore is implemented based on Flow, so it passesdataStore.dataWill return aFlow<T>Is reissued whenever the data changes
  • catchUsed to catch exceptions and throw an exception when reading data, if yesIOExceptionException, one will be sentPersonProtos.Person.getDefaultInstance()If it’s another exception, it’s better to throw it out

4. Write data to the Proto DataStore

Data is written to Proto DataStore using the datastore.updateData () method. Datastore.updatedata () is a suspend function, so it can only be used in coroutines. Whenever the suspend function is encountered and runs in suspend mode, the main thread is not blocked.

Run in suspended mode without blocking the main thread: that is, the coroutine scope is suspended and code outside the coroutine scope in the current thread does not block.

First, we need to create a suspend function and then call the datastore.updateData () method to write data.

suspend fun saveData(personModel: PersonModel) {
    protoDataStore.updateData { person ->
        person.toBuilder().setAge(personModel.age).setName(personModel.name).build()
    }
}
Copy the code

Person.tobuilder () is the Builder class that the compiler generates for each class to create message instances

Now that you’ve covered reading and writing data to the Proto DataStore, let’s look at how to migrate SharedPreferences to the Proto DataStore.

Migrate SharedPreferences to Proto DataStore

Migrating SharedPreferences to the Proto DataStore takes only 3 steps

1. Create a mapping

Migrating SharedPreferences data to the Proto DataStore requires a mapping that maps each pair of key-value data in SharedPreferences to the message type defined by the Proto file.

private val shardPrefsMigration = SharedPreferencesMigration<PersonProtos.Person>( context, SharedPreferencesRepository.PREFERENCE_NAME ) { sharedPreferencesView, The person - > / / SharedPreferences for data val follow. = sharedPreferencesView getBoolean (PreferencesKeys KEY_ACCOUNT, // add SharedPreferences to SharedPreferences. // add SharedPreferences to SharedPreferences. True data maps to Person's member variable followAccount in person.toBuilder().setFollowAccount(follow).build()}Copy the code
  • Gets the SharedPreferences storekey = ByteCodeThe value of the
  • willkey = ByteCodeThe data is mapped to the Person member variable followAccount

2. Construct the DataStore and pass in shardPrefsMigration

protoDataStore = context.createDataStore(
    fileName = FILE_NAME,
    serializer = PersonSerializer,
    migrations = listOf(shardPrefsMigration)
)
Copy the code
  • After the DataStore object is built, you need to perform a read or write operation to complete the migration of SharedPreferences to the DataStore. After the migration is successful, the files used by SharedPreferences are automatically deleted. The Proto DataStore and Preferences DataStore files are stored in the same path, as shown in the following figure

Proto DataStore is one of the implementation methods of Jetpack DataStore.

Common Proto3 syntax

I have summarized the common proto3 syntax, which should be able to meet most situations. For more syntax, please refer to the official Google tutorial. After mastering the Proto3 syntax, you can also learn the Proto2 syntax, although proto3 simplifies the use of Proto2 and improves the efficiency of development. However, most of the early Protocol Buffer teams used Proto2 syntax because of version-compatibility issues.

A basic message type

syntax = "proto3"; option java_package = "com.hi.dhl.datastore.protobuf"; option java_outer_classname = "PersonProtos"; Message Person {// format: field type + field name + field number string name = 1; int32 age = 2; bool followAccount = 3; repeated string phone = 4; Address address = 5; } message Address{ ...... }Copy the code
  • syntax: Specifies the version of a protobuf. If proto2 is not specified by default,It must be the first line of the.proto file except for empty lines and comment content
  • option: Indicates an optional field
    • java_package: Specifies the package name from which the Java classes will be generated
    • java_outer_classname: Specifies the name of the generated Java class
  • Multiple messages can be defined in a proto file
  • messageContains three fields: a string (name), an integer (age), and a bool (followAccount).Pay attention to ：=The number is followed by a field number
  • Each field consists of three parts: field type + field name + field number. In Java, each field is compiled into A Java object, and other languages are compiled into other language types

The field type

Each message type contains a number of message fields, and each message field has a type. Next, a table shows the types in the proto file, and the corresponding Java types, if other languages can view the official documentation.

.proto Type Notes Java Type
double double
float float
int32 Variable length encoding is used. If the field is negative, it is inefficient to use sint32 instead int
int64 Use variable-length coding. If the field is negative, which is inefficient, use sint64 instead long
sint32 Variable length encoding is more efficient if negative than plain INT32 int
sint64 Variable length encoding is more efficient if negative than plain INT64 long
bool boolean
string The string must always contain UTF-8 encoding or 7-bit ASCII text and cannot be longer than 23 characters String

The above types are commonly used, along with others: uint32, uint64, fixed32, fixed64, sfixed32, sfixed64, bytes, and more. For more, click here to view Encoding

Field defaults

Use the following rules in Proto3 to compile to Java language defaults:

  • For string, the default is an empty string ("")
  • For byte types, the default is an empty byte array of size 0
  • For bool, the default is false
  • For numeric types, the default value is 0
  • For enumeration types, the default value is the first enumerated value defined, and this value must be 0 (for compatibility with Proto2 syntax)
  • Use other message types as field types, the default is NULL (more on that below)
  • A field to be repeated. The default is an empty List of size 0

Field Numbers

Each message field = is followed by a field number, as follows:

string name = 1;
Copy the code

The field number is used to identify each field in the binary format of the message. The field number is very important and cannot be changed once it is used. The field number ranges from [1, 2^ 29-1], where [19000-19999] is reserved for Protobuf and cannot be used.

Note: Field numbers between ranges [1, 15] take up one byte during encoding, including field numbers and field types, and field numbers between ranges [16, 2047] take up two bytes. Therefore, field numbers between [1, 15] should be reserved for frequent message fields. Be sure to leave some room for elements that come up frequently in the future.

repeated

In the previous example, I added a repeated modifier to a field as follows:

repeated string phone = 4;
Copy the code

A List of Java types is repeated to a field. See the compiled code.

private com.google.protobuf.Internal.ProtobufList<java.lang.String> phone_;
Copy the code

ProtobufList is a subclass of List as follows:

public static interface ProtobufList<E> extends List<E>
Copy the code

Contains other message types

In addition to using int32, bool, string, etc., message fields can also use other message types as field types, as shown below:

Message Person {// format: Address Address = 5; } message Address{ ...... }Copy the code

Message nested

In a proto file, you can define multiple messages as follows:

Message Person {// format: field type + field name + field number string name = 1; int32 age = 2; bool followAccount = 3; repeated string phone = 4; Address address = 5; } message Address{ string city = 1 }Copy the code

Of course, messages can also be nested hierarchically. Here’s an example:

Message Person {// format: field type + field name + field number string name = 1; int32 age = 2; bool followAccount = 3; repeated string phone = 4; message Address{ string city = 1; } Address address = 5; }Copy the code

These messages are compiled into static inner classes, as shown below:

public  static final class Address extends
    com.google.protobuf.GeneratedMessageLite<
        Address, Address.Builder> implements AddressOrBuilder {
   ......
}
Copy the code

Enumerated type

We can also add an enumeration type to message or use an enumeration type as a field type, as follows:

message Person {    
    string name = 1;
    message Address{
        string city = 1;
    }
    Address address = 5;
    
    enum Weekday{
        SUN = 0;
        MON = 1;
        TUE = 2;
        WED = 3;
        THU = 4;
        FRI = 5;
        SAT = 6;
    }
    Weekday weekday = 6;
}
Copy the code

As you can see, message fields can use enumerated types as field types in addition to int32, bool, String, and other message types.

Note: The first enumeration value of each enumeration type must be 0, because:

  • There must be a value of 0 because 0 is required as the default
  • An element with a value of 0 must be the first enumeration value, which is compatible with the Proto2 syntax, where the default value is always the first enumeration value

oneof

Oneof has two meanings according to Google docs:

  • Oneof declares multiple fields, and only one field is assigned at a time, sharing a memory block, mainly to save memory
  • If one field of oneof is assigned and then the other fields are assigned, the values of the other assigned fields will be cleared, and only oneof the fields of oneof will have a value

Let’s look at a simple example

message PreferenceMap { map<string, Value> preferences = 1; } message Value { oneof valueName { bool boolean = 1; float float = 2; int32 integer = 3; int64 long = 4; string string = 5; double double = 7; }}Copy the code
  • A single oneof named valueName declares a number of fields, which share a single memory space, and only one field is assigned
  • A map is declared in a message named PreferenceMap. The Key is a string and the Value is actually a field declared in oneof. Each Key corresponds to only one Value at a time

At compile time, a Java enumeration type is actually generated for each oneof, as shown below

public enum ValueNameCase { BOOLEAN(1), FLOAT(2), INTEGER(3), LONG(4), STRING(5), DOUBLE(7), VALUENAME_NOT_SET(0); // If no values are assigned, 'VALUENAME_NOT_SET' private final int value; private ValueNameCase(int value) { this.value = value; }Copy the code

The compiler automatically generates the getValueNameCase() method to check which field is assigned, and returns VALUENAME_NOT_SET if none is assigned

Proto3 commonly used syntax is introduced here, the article only lists the commonly used syntax, if you need to complete the analysis of the Proto3 syntax, at least 2 articles are likely to be introduced, because of the length of the reason, the source analysis part will be analyzed in the subsequent article.

What is MAD Skills

Google has released MAD Skills (Modern Android Development), a new series of tutorials designed to help developers build better applications using the latest technology. This includes Kotlin, Android Studio, Jetpack, App Bundles, and more. Google only provides videos and articles. I’ve built on this and made some extensions:

  • Chinese and English subtitles have been added to the video to facilitate learning
  • The actual combat part of the video will provide corresponding actual combat cases
  • In addition to the actual case, will also provide the corresponding source analysis

Google releases a series of tutorials every few weeks, and has already started a series of videos on Navigation Components. The bilingual video has been synchronized to the GitHub warehouse and mad-Skills can watch the video first. Articles and cases are coming fast.

Refer to the article

  • Google-DataStore – Jetpack Alternative For SharedPreferences
  • Google-Language Guide (proto3)
  • GitHub-protobuf
  • FlatBuffers experience
  • Java object serialization
  • JSON vs Protocol Buffers vs FlatBuffers

conclusion

This concludes the article, and the relevant examples have been uploaded to GitHub. Welcome to the repository AndroidX-Jetpack-Practice/DataStoreSimple and switch to the datastore_proto branch.

GitHub address: https://github.com/hi-dhl/AndroidX-Jetpack-Practice

By the time I wrote this article, I had written four articles. Before I wrote this article, I had written three articles about the two command line compilation methods for MAC and Ubuntu and the way to compile proto files with Gradle plug-ins. Because the methods on the web are too old and not very clear. The Gradle plugin is mostly configured in 3.0.x ~ 3.7.x mode on the web. There are some differences when protoc >= 3.8, so we rewrote these three compilation modes and recorded the problems encountered in this process.

  • Protobuf | Gradle plug-in installation compile proto file
  • Protobuf | how to compile on ubuntu installation Protobuf proto file
  • Protobuf | how to install Protobuf compile proto files on MAC

Since it is mainly developed on MAC and Ubuntu at present, only these two command line compilation methods are provided. If you are developing on Win, you can use Gradle plug-in compilation method. If it is helpful, please give me a thumbs-up.

Finally, I recommend the projects and websites I have been updating and maintaining:

  • Androidx-jetpack-practice androidX-Jetpack-practice androidX-Jetpack-Practice androidX-Jetpack-Practice AndroidX-Jetpack-Practice

  • LeetCode/multiple thread solution, language Java and Kotlin, including a variety of solutions, problem solving ideas, time complexity, spatial complexity analysis

    • Job interview with major companies at home and abroad
    • LeetCode: Read online

  • Android10 Source code Analysis series of articles, understand the system Source code, not only help to analyze the problem, in the interview process, is also very helpful to us, the warehouse continues to update, welcome to check android10-source-Analysis

  • Collate and translate a series of selected foreign Technical articles, each Article will have a translator’s thinking part, a more in-depth interpretation of the original text, the warehouse continues to update, welcome to visit the Technical-Article-Translation

  • “Designed for Internet people, navigation of domestic and foreign famous stations” includes news, sports, life, entertainment, design, product, operation, front-end development, Android development and so on. Welcome to check the navigation website designed for Internet people

Article history

  • Kotlin’s Technique and Principle Analysis that few people know (1)
  • Kotlin’s Technique and Principle Analysis that few people know (II)
  • AndroidX App Startup practice and principle analysis of Jetpack’s latest member
  • Jetpack member Paging3 Practice and Source Code Analysis (PART 1)
  • Jetpack Member Paging3 Network Practice and Principle Analysis (II)
  • Jetpack member Paging3 retrieves network page data and updates it to the database
  • Jetpack member Hilt practice (1) Start a pit
  • Jetpack member Hilt with App Startup (ii) Advanced article
  • New member of Jetpack Hilt and Dagger are very different
  • All aspects of Hilt and Koin performance were analyzed
  • PokemonGo Jetpack + MVVM minimalism
  • What is Kotlin Sealed? Why does Google use them all
  • Kotlin StateFlow search features practice DB + NetWork
  • Bye-bye buildSrc, embrace Composing builds for faster Android builds
  • [Google] Bye SharedPreferences embrace Jetpack DataStore