Learning Java IO system, the focus is to learn IO model, understand a variety of IO models after you can better understand Java IO
Java IO is a set of Java apis for reading and writing data (input and output). Most programs process some input and produce some output from that input. Java provides the java.io package for this purpose
IO system in Java can be divided into BIO, NIO, AIO three IO models
About BIO, we need to know what the synchronous blocking IO model is, what objects the BIO operates on: streams, and how to use BIO for network programming, the question of using BIO for network programming
About NIO, we need to know what is the synchronous non-blocking IO model, what is the multiplexing IO model, the concepts of Buffer,Channel,Selector in NIO, and how to use NIO for network programming
About AIO, we need to know what the asynchronous non-blocking IO model is, how many ways can AIO implement asynchronous operations, and how can AIO be used for network programming
A, BIO
If you want to process multiple streams at the same time, you need to use multiple threads
A stream consists of character streams and byte streams. A stream is conceptually a continuous stream of data. When the program needs to read data, it needs to use the input stream to read data, and when it needs to write data out, it needs to use the output stream
1. Block IO model
In Linux, when an application calls recvFROM, the kernel does not return the data immediately if the data is not ready. Instead, the kernel waits for the data to be ready, and the data is copied from the kernel to user space and then returns. During this time, the application process blocks until it returns, which is called the blocking IO model
2, flow
There are two main types of streams that operate in BIO, byte streams and character streams, both of which can be divided into input streams and output streams based on the direction of the stream
According to type and input and output direction, it can be divided into:
(1) Input byte stream: InputStream
(2) Output byte stream: OutputStream
(3) Input character stream: Reader
(4) Output character stream: Writer
Byte streams are primarily used to process byte or binary objects, and character streams are used to process character text or strings
InputStreamReader is used to convert an input byte stream into an input character stream
Reader reader = new InputStreamReader(inputStream);Copy the codeUsing OutputStreamWriter, you can turn output byte streams into output character streams
Writer writer = new OutputStreamWriter(outputStream)Copy the codeWe can read data from a data source in an application through InputStream and Reader, and then we can output data to a target medium in an application through OutputStream and Writer
When using byte streams, InputStream and OutputStream are abstract classes, and we instantiate their subclasses, each with its own scope
Reader and Writer are abstract classes, and we instantiate their subclasses, each of which has its own scope
Take reading and writing files as an example
(1) Read data from the data source
Input byte stream: InputStreamCopy the codepublic static void main(String[] args) throws Exception{
File file = new File("D:/a.txt");
InputStream inputStream = new FileInputStream(file);
byte[] bytes = new byte[(int) file.length()];
inputStream.read(bytes);
System.out.println(newString(bytes)); inputStream.close(); } Input character stream:Reader
public static void main(String[] args) throws Exception{
File file = new File("D:/a.txt");
Reader reader = new FileReader(file);
char[] bytes = new char[(int) file.length()];
reader.read(bytes);
System.out.println(new String(bytes));
reader.close();
}Copy the code(2) Output to target media
OutputStream public static void main(String[] args) throws Exception{String var ="hai this is a test";
File file = new File("D:/b.txt"); OutputStream outputStream = new FileOutputStream(file); outputStream.write(var.getBytes()); outputStream.close(); } Output character stream: Writer public static void main(String[] args) throws Exception{String var ="hai this is a test";
File file = new File("D:/b.txt");
Writer writer = new FileWriter(file);
writer.write(var);
writer.close();
}Copy the code(3) the BufferedInputStream
When you use InputStream, you read or write byte by byte. BufferedInputStream provides a buffer for the InputStream, which reads one piece of data at a time and puts it into the buffer. When the data in the buffer is read, the InputStream fills the buffer. Until the input stream is read, having a buffer can improve I/O speed a lot
Use to wrap the input stream into BufferedInputStream
/** * inputStream inputStream * 1024 internal buffer size of 1024 bytes */ BufferedInputStream BufferedInputStream = new BufferedInputStream(inputStream,1024);Copy the code(4) BufferedOutputStream
BufferedOutputStream can provide a buffer for the output byte stream, similar to BufferedInputStream
Wrap the output stream in a BufferedOutputStream
/** * outputStream outputStream * 1024 internal buffer size of 1024 bytes */ BufferedOutputStream BufferedOutputStream = new BufferedOutputStream(outputStream,1024);Copy the codeIf the byte stream provides buffered characters, the character stream must also provide BufferedReader and BufferedWriter
(5) BufferedReader
Provides a buffer for the input character stream as follows
BufferedReader bufferedReader = new BufferedReader(reader,1024);
(6) BufferedWriter
Provides a buffer for output character streams as follows
BufferedWriter bufferedWriter = new BufferedWriter(writer,1024);Copy the code3. BIO model network programming
When using the BIO model for Socket programming, the server usually calls the Accept method in a while loop. When there is no client request, the Accept method blocks until the request is received and the corresponding processing is returned. This process is linear. Subsequent requests are processed only after the current request has been processed, usually resulting in the communication thread being blocked for a long time
The BIO model handles multiple connections:
In this mode we usually use a single thread to receive requests, and then use a single thread pool to process requests, and manage multiple Socket client connections concurrently, like this:
Using BIO model for network programming ability of problem is lack of flexibility, the client number of concurrent access and server is 1:1, the relationship between number of threads and due to obstruction at ordinary times there will be a large number of threads in a wait state, waiting for input or output data in place, cause the waste of resources, in the face of a large number of concurrent conditions, If you do not use the thread pool to directly new threads, it will be roughly ballooning, system performance will degrade, may lead to stack memory overflow, and frequent creation and destruction of threads, more waste of resources
Using a thread pool may be a bit better, but you can’t solve the problem of blocking IO obstruction, but also need to consider if the number of thread pool set will be refused to a lot of smaller Socket client connection, if the number of thread pool set is larger, can lead to a large number of context switches, and the program to allocate memory for each thread of the call stack, The default value ranges from 64 KB to 1 MB, which wastes VM memory
The BIO model is suitable for architectures with a fixed number of links and a small number of links, but the code written using this model is more intuitive, simpler and easier to understand
Second, the NIO
Since JDK 1.4, the JDK has released a new I/O class library, or NIO, which is a synchronous non-blocking IO model
1. Non-blocking IO model
Synchronous non-blocking IO model implementation:
Non-blocking IO model
The application calls the recvFROM system call, which returns an EWOULDBLOCK error if the kernel data is not ready. The application does not block, but needs to poll recvFROM continuously until the kernel data is ready. It then waits for the data to be copied from the kernel to user space (which blocks but takes very little time) and returns when the copy is complete
2. IO reuse model
IO reuse model, using the Select provided by Linux system, poll system call, pass one or more file handles (client links in network programming) to select or poll system call, and the application process blocks on the SELECT, thus forming a process corresponding to multiple Socket links. Select /poll then scans the set of Socket links linearly, resulting in a decrease in efficiency when only a few sockets have data, and is limited by the number of file handles held by select/poll. The default value is 1024
3. Signal driven IO model
The system call SIGAction executes a signal-handler function that does not block the application process. When the data is ready, a SIGIO signal is generated for the process, and the signal callback tells the application to call recvFROM to read the data
4. Core concepts of NIO
Buffer
Buffer is an object, it contains some to write or read data, all data is used in the NIO Buffer processing, reading data from the Buffer when you need to read, writing the data will be written to the Buffer, the Buffer is essentially a can write data, which can then be read from an array of data, Provides structured access to data and maintains information such as read and write locations internally
ByteBuffer =ByteBuffer. Allocate (1024); allocate(1024); allocate(1024);Copy the codeHow to use Buffer
(1) Write data to Buffer
(2) Call flip() method to switch Buffer from write mode to read mode
(3) Read data from Buffer
(4) Call clear() or Compact () to clear the buffer so that it can be written again
Channel
Data is always read from a Channel to a Buffer, or written from a Buffer to a Channel. A Channel is only responsible for transporting data, while the operational data is a Buffer
A channel is similar to a stream.
(1) Because a channel is bidirectional, read and write operations can be carried out at the same time, while a stream is unidirectional and can only be written or read
(2) The read and write of the stream is blocked, and the channel can read and write asynchronously
Data is read from Channel to Buffer
inChannel.read(buffer);Copy the codeData is written from Buffer to Channel
outChannel.write(buffer);Copy the codeTake copying files as an example
FileInputStream fileInputStream=new FileInputStream(new File(src));
FileOutputStream fileOutputStream=new FileOutputStream(new File(dst));
// Get the input/output channel
FileChannel inChannel=fileInputStream.getChannel();
FileChannel outChannel=fileOutputStream.getChannel();
// Create a buffer of 1024 bytes
ByteBuffer buffer=ByteBuffer.allocate(1024);
while(true) {// Read data from inChannel. If no bytes are read, return -1
int eof =inChannel.read(buffer);
if(eof==-1) {break; }// Switch Buffer from write mode to read mode
buffer.flip();
// Start writing data to outChannel
outChannel.write(buffer);
/ / empty buffer
buffer.clear();
}
inChannel.close();
outChannel.close();
fileInputStream.close();
fileOutputStream.close();Copy the codeA Selector for multiplexing.
Selector is the basis of NIO programming. The main function is to register multiple channels with a Selector. If a read or write event occurs on a Channel, the Channel will be in a ready state and will be polled by the Selector. The SelectionKey is then used to retrieve the set of channels in place for IO operations
The relationship between Selector and Channel and Buffer
5. NIO model network programming
NIO uses the MULTIPLEXing IO model in JDK. By multiplexing multiple IO blocks to a single select block, NIO can simultaneously process multiple client requests in a single thread, saving system overhead. JDK Selector is implemented based on the Select /poll model. In JDK versions 1.5 and later, epoll is used instead of select/poll
Epoll has the following advantages over SELECT /poll:
(1) Epoll supports unlimited number of open file descriptors. Select /poll can open a limited number of file descriptors
(2) SELECT /poll uses polling to traverse the entire set of file descriptors. Epoll is based on the callback function of each file descriptor
Select, poll, and epoll are all MECHANISMS for I/O multiplexing. I/O multiplexing is a mechanism by which a process can monitor multiple descriptors and, once a descriptor is ready (usually read or write), inform the program to do the corresponding read/write operation. But SELECT, poll, and epoll are all synchronous I/ OS in nature, because they are responsible for reading and writing after the read and write event is ready, that is, the read and write process is blocked, while asynchronous I/ OS are not responsible for reading and writing
NIO provides two different sets of socket channels for network programming, ServerSocketChannel and client SocketChannel, both of which support blocking and non-blocking modes
Server code
The server accepts the message output sent by the client and sends a message to the client
SelectorSelector = selector. Open (); ServerSocketChannel Channel = serverSocketChannel.open ().bind(new InetSocketAddress(9001)); / / set the channel for nonblocking channel. ConfigureBlocking (false); // ** * 1. Selectionkey. OP_CONNECT: connection event * 2. Selectionkey. OP_ACCEPT: receive event * 3. * 4. Selectionkey. OP_WRITE: Write event * * bind channel to selector and register OP_ACCEPT event */ channel.register(selector, selectionkey.op_accept);while (trueSelect () returns (a key) only if the OP_ACCEPT event arrives, and blocks selector. Select () until the event arrives; // When an event arrives, select() is not blocking, and then selectionKeys () takes the Set of selectionkeys that reached the event. Set keys = selectedKeys(); Iterator iterator = keys.iterator();while(iterator.hasNext()){ SelectionKey key = (SelectionKey) iterator.next(); Iterator.remove (); iterator.remove(); iterator.remove(); iterator.remove(); // Determine by SelectionKey stateif(key.isConnecTable ()){// Connection successful}else if(key.isacceptable ()){/** * accepts client requests ** Since we only register the OP_ACCEPT event, the client link will only go to this point * All we need to do is read the client data, So we need to get the serverChannel based on SelectionKey * get the client Channel based on the serverChannel, and register it with an OP_READ event */ // 1, Get ServerSocketChannelServer SocketChannel serverChannel = (ServerSocketChannel) key. The channel (); SocketChannel clientChannel = serverChannel.accept(); SocketChannel clientChannel = serverChannel.accept(); / / 3, set the channel to a non-blocking clientChannel. ConfigureBlocking (false); // 4, register the OP_READ event clientchannel.register (key.selector(), selectionkey.op_read); }else if(key.isreadable ()){// The channel can read data /** * because after the client is connected to the server, An OP_READ event is registered to send some data * so we still need to fetch clientChannel * and then read clientChannel data through Buffer */ SocketChannel clientChannel = (SocketChannel) key.channel(); ByteBuffer byteBuffer = ByteBuffer.allocate(BUF_SIZE); long bytesRead = clientChannel.read(byteBuffer);while (bytesRead>0){
byteBuffer.flip();
System.out.println("The client data."+new String(byteBuffer.array())); byteBuffer.clear(); bytesRead = clientChannel.read(byteBuffer); } /** * After the server receives the message, we send a data to the client */ bytebuffer.clear (); byteBuffer.put("Hello client, I'm the server, look how hard NIO is.".getBytes("UTF-8"));
byteBuffer.flip();
clientChannel.write(byteBuffer);
}
else if(key. IsWritable () && key. The isValid ()) {/ / channel data can be written}}}Copy the codeClient code
After connecting to the server, the client sends a message to the server and receives the message from the server
Selector selector = Selector.open();
SocketChannel clientChannel = SocketChannel.open();
// Set channel to non-blocking
clientChannel.configureBlocking(false);
// Connect to the server
clientChannel.connect(new InetSocketAddress(9001));
// Register the OP_CONNECT event
clientChannel.register(selector, SelectionKey.OP_CONNECT);
while (true) {// If the event does not arrive, it will remain blocked
selector.select();
Iterator iterator = selector.selectedKeys().iterator();
while (iterator.hasNext()){
SelectionKey key = iterator.next();
iterator.remove();
if (key.isConnectable()){
/**
* 连接服务器端成功
*
* 首先获取到clientChannel，然后通过Buffer写入数据，然后为clientChannel注册OP_READ时间
*/
clientChannel = (SocketChannel) key.channel();
if (clientChannel.isConnectionPending()){
clientChannel.finishConnect();
}
clientChannel.configureBlocking(false);
ByteBuffer byteBuffer = ByteBuffer.allocate(1024);
byteBuffer.clear();
byteBuffer.put("Hello server, I am the client, do you see this NIO difficult?".getBytes("UTF-8"));
byteBuffer.flip();
clientChannel.write(byteBuffer);
clientChannel.register(key.selector(),SelectionKey.OP_READ);
}
else if (key.isReadable()){
// Channels can read data
clientChannel = (SocketChannel) key.channel();
ByteBuffer byteBuffer = ByteBuffer.allocate(BUF_SIZE);
long bytesRead = clientChannel.read(byteBuffer);
while (bytesRead>0){
byteBuffer.flip();
System.out.println("Server data."+newString(byteBuffer.array())); byteBuffer.clear(); bytesRead = clientChannel.read(byteBuffer); }}else if (key.isWritable() && key.isValid()){
// Channels can write data}}}Copy the codeIt is very complicated to use the native NIO class library. NIO class library and Api are complicated and troublesome to use. In order to write high quality NIO programs, it is not recommended to directly use the native NIO for network programming, but to use some mature frameworks, such as Netty
Third, AIO
JDK1.7 upgraded the Nio library to Nio2.0, most importantly providing asynchronous file IO operations and event-driven IO. AIO’s asynchronous socket channel is truly asynchronous non-blocking IO
1. Asynchronous IO model
On Linux, an application initiates a read operation and is immediately ready to do something else. The kernel tells the application that the data has been copied to the used space and that the read operation is complete
2. Aio model network programming
Asynchronous operations
Aio simplifies NIO’s programming model by enabling asynchronous reads and writes without polling registered channels through a multiplexer
Aio implements asynchronous operations through asynchronous channels, which provide two ways to obtain operation results:
(1) Use the Future class to retrieve the results of asynchronous operations, but note that future.get() is a blocking method that blocks the thread
(2) Asynchronous callback is implemented by passing in an implementation class of CompletionHandler, which defines two methods. Completed and failed correspond to success and failure respectively
Channels in Aio support both methods
AIO provides corresponding asynchronous socket channel to realize the network programming, the service side: AsynchronousServerSocketChannel AsynchronousSocketChannel and client
The service side
The server sends messages to the client and receives messages sent by the client
AsynchronousServerSocketChannel server =AsynchronousServerSocketChannel.open().bind(new InetSocketAddress("127.0.0.1", 9001)); Server. accept(null, newCompletionHandler() {/ / success @ Override public void completed (AsynchronousSocketChannel result, Void attachment) { try { ByteBuffer buffer = ByteBuffer.allocate(1024); buffer.put("I'm the server. Hello client.".getBytes());
buffer.flip();
result.write(buffer, null, new CompletionHandler(){
@Override
public void completed(Integer result, Void attachment) {
System.out.println("The server sent the message successfully");
}
@Override
public void failed(Throwable exc, Void attachment) {
System.out.println("Send failed"); }}); ByteBufferreadBuffer = ByteBuffer.allocate(1024);
result.read(readBuffer, null, new CompletionHandlerPublic void completed(Integer result, void attachment) {system.out.println (new String();readBuffer.array())); } @override public void failed(Throwable exc, void attachment) {system.out.println ("Read failed"); }}); } catch (Exception e) { e.printStackTrace(); }} @override public void void failed(Throwable exc, void attachment) {exc.printStackTrace(); }}); Timeunit.seconds.sleep (1000L);Copy the codeThe client
The client sends messages to and receives messages from the server
AsynchronousSocketChannel client = AsynchronousSocketChannel.open();
Future future = client.connect(new InetSocketAddress("127.0.0.1".9001));
// block, get the connectionfuture.get(); ByteBuffer buffer = ByteBuffer.allocate(1024);
/ / read the data
client.read(buffer, null.new CompletionHandler() {
// Called on success
@Override
public void completed(Integer result, Void attachment) {
System.out.println(new String(buffer.array()));
}
// called on failure
@Override
public void failed(Throwable exc, Void attachment) {
System.out.println("Client failed to receive message"); }}); ByteBuffer writeBuffer = ByteBuffer.allocate(1024);
writeBuffer.put("Hello, client. Hello, server.".getBytes());
writeBuffer.flip();
// block method
Future write = client.write(writeBuffer);
Integer r = write.get();
if(r>0){
System.out.println("Client message sent successfully");
}
// Hibernate the thread
TimeUnit.SECONDS.sleep(1000L);Copy the codeFour,
Comparison of IO models:
Pseudo asynchronous IO refers to the Bio model that uses thread pools to process requests