Data transmission efficiency optimization
Data serialization and deserialization
Server Object—— Data flow —-> Client Object Object
Traditional serialization: Serializable/Parcelable is inefficient. For example, news users can download a large number of images and text when browsing. Traditional data transfer can waste memory and consume CPU computing time
Data serialization is an essential part of program code, and when we talk about the performance of data serialization, we need to know what the candidates are and what their pros and cons are. First of all, what is serialization? To illustrate:
The behavior of data serialization can occur at any stage of data transfer, such as network transfer, data transfer between different processes, parameter transfer between different classes, data storage to disk, and so on. Normally, we implement the Serializable interface for those classes that need to be serialized (as shown in the figure below), but this traditional approach is inefficient and consumes more memory.
Traditional methods consume too much memory
But if we use the GSON library to handle this serialization problem, not only the execution is faster, but the memory usage is also more efficient. Android’S XML layout files are converted at compile time into a more complex format with more efficient execution performance and memory usage.
Google Gson serialization
Here are three candidates for data serialization:
- Protocal Buffers: powerful and flexible, but consume a large amount of memory. Protocal Buffers are not the best choice for mobile terminals.
- Nano-proto-buffers: Based on Protocal, this method is specially optimized for mobile terminals to improve code execution efficiency and memory usage efficiency.
- FlatBuffers: This open source library was originally developed by Google with a focus on providing better performance. The performance data pairs of these schemes are shown below:
FlatBuffers almost beat other techniques in terms of spatial and temporal complexity. FlatBuffers is an open source cross-platform data serialization library that can be used in almost any language (C++, C#, Go, Java, JavaScript, PHP, Python). It was originally developed by Google for games and other performance-demanding applications. The project is on GitHub. The official document address.
The advantages of FlatBuffer
What are the advantages of FlatBuffer over other serialization technologies, such as XML, JSON, Protocol Buffers, etc.? The official document says as follows:
Parsing of serialized data without Parsing or Unpacking: The FlatBuffer stores the data hierarchy in a flat binary cache (a one-dimensional array), while maintaining direct access to the structured data without parsing, and ensuring forward and backward compatibility with changes in the data structure.
2. Efficient memory usage and speed: During the use of FlatBuffer, no extra memory is required, almost approaching the size of original data in memory.
3. Flexibility: Data is forward-backward compatible and has the flexibility to control your data structure.
4. Minimal code intrusion: This can be achieved with a small amount of automatically generated code.
5. Strong data class, easy to use, cross-platform, almost language independent.
An official performance comparison table is provided as follows:
JSON was the most commonly used data serialization technique in Android development. As we know, JSON is very readable, but the serialization and deserialization performance is the worst. When parsing, the JSON parser first needs to initialize a corresponding data structure in memory. This event usually takes 100ms to 200ms2. During parsing, a large number of temporary variables are generated, causing GC and memory jitter of the Java VIRTUAL machine. Parsing 20KB of data consumes about 100KB of temporary memory 2. FlatBuffers solve these problems.
FlatBuffer use
$git clone github.com/google/flat… 2. Download the corresponding cmake tool from cmake.org/download/ : You can use cmake to compile the flatC tool. But don’t know how to compile yet? Finally, this is where you can download the Windows EXE and use the file. Github.com/google/flat… Compare pit of the latest 1.7.1 without EXE without exe download. I didn’t look at it, thinking I needed to compile it myself. The last thing I saw was 1.7.0 being downloaded. Well…
Written description using FlatBuffers IDL Schema defined data structure, write detailed documentation of Schema Google here. Dead simple. IO/FlatBuffers… . Consider an example
namespace com.haocai.app.flatbuffer;
table Items {
ItemId : long;
timestemp : int;
basic:[Basic];
}
table Basic{
id:int;
name:string;
email:int;
code:long;
isVip:bool;
count:int;
carList:[Car];
}
table Car{
id:long;
number:long;
describle:string;
}
root_type Items;
Copy the codeUse tools to produce the corresponding data classes
Bin For example, repos_json.json (data repos_schema.fbs), the corresponding data structure is repos_schema.fbs Bin (in binary flatbuffer format) $./ flatc-J-b repos_schema.fbs repos_json.json
Project use to write corresponding description file, translated into the corresponding language class. Put the corresponding class in your project.
Call MainActivity. Java
public class MainActivity extends AppCompatActivity {
private static final String TAG ="main" ;
@Override
protected void onCreate(Bundle savedInstanceState) {
super.onCreate(savedInstanceState);
setContentView(R.layout.activity_main);
}
public void serialize(View v) {
//---------------- serialize -------------
FlatBufferBuilder builder = new FlatBufferBuilder();
int id1 = builder.createString(lamborghini);
// Prepare the Car object
int car1 = Car.createCar(builder,10001L.88888L,id1);
int id2 = builder.createString("The audi A8");
// Prepare the Car object
int car2 = Car.createCar(builder,10001L.88888L,id2);
int id3 = builder.createString("Audi A9");
// Prepare the Car object
int car3 = Car.createCar(builder,10001L.88888L,id3);
int[] cars = new int[3];
cars[0] = car1;
cars[1] = car2;
cars[2] = car3;
// Create Car collection in Basic object
int carList = Basic.createCarListVector(builder,cars);
int name = builder.createString("kpioneer");
int email = builder.createString("[email protected]");
int basic = Basic.createBasic(builder,10,name,email,100L.true.100,carList);
int basicOffset = Items.createBasicVector(builder,new int[]{basic});
/** * table Items { ItemId : long; timestemp : int; basic:[Basic]; } * /
Items.startItems(builder);
Items.addItemId(builder,1000L);
Items.addTimestemp(builder,2016);
Items.addBasic(builder,basicOffset);
int rootItems = Items.endItems(builder);
Items.finishItemsBuffer(builder,rootItems);
//============ Save the data to a file =================
File sdcard = Environment.getExternalStorageDirectory();
// Save path
File file = new File(sdcard,"Items.txt");
if(file.exists()){
file.delete();
}
ByteBuffer data = builder.dataBuffer();
FileOutputStream out = null;
FileChannel channel = null;
try {
out = new FileOutputStream(file);
channel = out.getChannel();
while(data.hasRemaining()){ channel.write(data); }}catch (Exception e) {
e.printStackTrace();
}finally {
try {
if(out! =null){
out.close();
}
if(channel! =null){ channel.close(); }}catch(IOException e) { e.printStackTrace(); }}/ / = = = = = = = = = = = = = = = = = = = deserialization = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
FileInputStream fis = null;
FileChannel readChannel = null;
try {
fis = new FileInputStream(file);
ByteBuffer byteBuffer = ByteBuffer.allocate(1024);
readChannel = fis.getChannel();
int readBytes = 0;
while((readBytes=readChannel.read(byteBuffer))! = -1){
System.out.println("Number of data read:"+readBytes);
}
// Return the pointer to its original state, ready to read data from byteBuffer
byteBuffer.flip();
// Parse the binary into Items objects.
Items items = Items.getRootAsItems(byteBuffer);
// Read the data test to see if it matches the saved data
Log.i(TAG,"items.id:"+items.ItemId());
Log.i(TAG,"items.timestemp:"+items.timestemp());
Basic basic2 = items.basic(0);
Log.i(TAG,"basic2.name:"+basic2.name());
Log.i(TAG,"basic2.email:"+basic2.email());
//carList
int length = basic2.carListLength();
for (int i=0; i<length; i++){ Car car = basic2.carList(i); Log.i(TAG,"car.number:"+car.number());
Log.i(TAG,"car.describle:"+car.describle()); }}catch (Exception e) {
e.printStackTrace();
}finally {
try {
if(readChannel! =null){
readChannel.close();
}
if(fis! =null){ fis.close(); }}catch(IOException e) { e.printStackTrace(); }}}}Copy the codeResult output:
09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/System. Out: Read the data number: 304 09-27 16:25:38. 567, 15008-15008 / com. Haocai. App. Flatbuffer I/main: Items. Id: 1000 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: Items. Timestemp: 2016 09-27 16:25:38. 567, 15008-15008 / com. Haocai. App. Flatbuffer I/main: Basic2. Name: kpioneer 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: Basic2. Email: 196 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: Car. Number: 88888 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: Car. Describle: lamborghini 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: Car. Number: 88888 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: Car. Describle: audi A8 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: Car. Number: 88888 09-27 16:25:38. 567, 15008-15008 / com haocai. App. Flatbuffer I/main: car. Describle: audi A9Copy the codeThe basic principle of
In the layout above, you need to note:
Each object is divided into two parts: the metadata part (or Vtable) on the left of the pivot point, and the real data part on the right. Each field corresponds to a slot in the VTable that stores the offset of the real data for that field. For example, the first slot of John’s table has a value of 1, indicating that John’s name is stored one byte to the right of Jonh’s pivot point.
For object fields, the offset in the VTABLE points to the pivot point of the child object. For example, the third slot in John’s Vtable points to Mary’s pivot point.
To indicate that a field has no value, we can use a 0 offset in a Vtable slot.
