Compress 20M files from 30 seconds to 1 second optimization process

There is a requirement to send 10 photos from the front end, and then compress them into a compressed package after processing by the back end. Before did not contact with Java compressed files, so directly on the Internet to find an example changed to use, after the change can also be used, but with the front end of the size of the image is bigger and bigger, the cost of time is also sharply increased, finally tested the compression of 20M files actually need 30 seconds. The code for the compressed file is as follows.

Here find a 2M size picture, and loop ten times to test. The printed result is as follows, and the time is about 30 seconds.

First optimization process – from 30 seconds to 2 seconds

The first thing that comes to mind when optimizing is using buffersBufferInputStream. inFileInputStreamThe read() method reads one byte at a time. The source code also has instructions.

This is a call to a local method that interacts with the native operating system to read data from disk. Interacting with the operating system by calling local methods every time a byte of data is read can be time-consuming. For example, we now have 30,000 bytes of data. If we use the FileInputStream, we need to call the local method 30,000 times to get the data, whereas if we use the buffer (assuming that the initial buffer size is large enough to hold 30,000 bytes of data) we only need to call it once. The buffer reads data directly from disk to memory the first time it calls the read() method. Then it slowly returns byte by byte.

BufferedInputStream internally encapsulates an array of bytes to hold the data. The default size is 8192

The optimized code is as follows

The output

You can see that the second optimization process is much more efficient than the first time using the FileInputStream – from 2 seconds to 1 second

Using buffers was sufficient for my needs, but with the idea of using what I learned, I wanted to optimize it with NIO knowledge.

The use of the Channel

Why do we use Channel? Because there are new channels and bytebuffers in NIO. Because their architecture is more in line with the way the operating system performs I/O, they offer a significant increase in speed compared to traditional IO. A Channel is like a mine that contains coal, and a ByteBuffer is the truck that delivers it to the mine. This means that all of our interactions with data are with the ByteBuffer.

There are three classes in NIO that can generate FileChannel. They are FileInputStream, FileOutputStream, and RandomAccessFile, which can both read and write.

The source code is as follows

As you can see, instead of using a ByteBuffer for data transfer, the transferTo method is used. The method is to connect the two channels directly.

This is the description text on the source code, which basically means that using transferTo is more efficient than looping over one Channel to read it and then looping over another Channel to write it. The operating system can transfer bytes directly from the file system cache to the target Channel without the need for an actual copy phase.

The copy phase is the process of moving from kernel space to user space and you can see that the speed has improved somewhat compared to using buffers.

Kernel space and user space

So why is the transition from kernel space to user space slow? The first thing we need to understand is what kernel space and user space are. In common operating systems, in order to protect the core resources in the system, the system is designed as four regions, the more the more access, so Ring0 is called the kernel space, used to access some key resources. Ring3 is called user space.

User mode, kernel mode: A thread in kernel space is called kernel mode, while a thread in user space belongs to user mode

So what if our applications (which are all in user mode) need access to core resources? You need to call the exposed interface in the kernel, called a system call. Let’s say our application needs to access a file on disk. At this point, the application calls the open method of the system call interface, and the kernel accesses the file on disk and returns the file contents to the application. The general process is as follows

Direct and indirect buffers

Since we’re going to read a file on a disk, it takes so much work. Is there an easy way for our application to work directly with the disk files without kernel transfer? Yes, that is to create a direct buffer.

Indirect buffers: The indirect buffers are the ones we talked about above as middlemen, requiring the kernel to be in the middle every time.

Direct buffer: The direct buffer does not require kernel space as the transfer copy data, but directly applies for a piece of physical memory space, this space is mapped to the kernel address space and user address space, application and disk data access through this directly applied physical memory interaction.

If direct buffers are so fast, why don’t we all use direct buffers? In fact, direct buffers have the following disadvantages. Disadvantages of direct buffers:

  • unsafe
  • It consumes more because it doesn’t carve out space directly in the JVM. This part of memory can only be reclaimed by the garbage collection mechanism, and when the garbage is reclaimed is not under our control.
  • When data is written to the physical memory buffer, the program loses control of the data, that is, when the data is finally written to disk is determined by the operating system, and the application can no longer interfere.

In summary, we use the transferTo method to create a direct buffer directly. So the performance is much better

Use memory-mapped files

Another new feature in NIO is memory-mapped files. Why are memory-mapped files so fast? In fact, the reason is the same as the above mentioned, is to open up a direct buffer in memory. Interact directly with data. The source code is as follows

Print the following

You can see that the speed is about the same as using a Channel.

Use Pipe

A Java NIO pipe is a one-way data connection between two threads. Pipe has a source channel and a sink channel. The source channel is used to read data, and the sink channel is used to write data. As you can see in the source code, the writing thread blocks until a reader thread reads from the channel. If there is no data to read, the reader thread also blocks until the writer thread writes the data. Until the channel closes.

Whether or not a thread writing bytes to a pipe will block until another thread reads those bytes

I want it to look like this. The source code is as follows:

Source address github.com/modouxiansh…

conclusion

Life is all about learning, and sometimes it’s just a simple optimization that allows you to deeply learn different kinds of knowledge. So you have to learn not so much, you have to know this but you have to understand why you’re doing it.