1. Background

In the process of video download, especially M3U8 video download, the following exceptions often occur:

W/System.err: java.net.ProtocolException: unexpected end of stream
W/System.err:     at com.android.okhttp.internal.http.Http1xStream$FixedLengthSource.read(Http1xStream.java:398)
W/System.err:     at com.android.okhttp.okio.RealBufferedSource$1.read(RealBufferedSource.java:372)
W/System.err:     at java.io.InputStream.read(InputStream.java:101)
......
W/System.err:     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
W/System.err:     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
W/System.err:     at java.lang.Thread.run(Thread.java:919)
Copy the code

Take an example video url: www.fztxylgy.com:65/20210218/7E… Website URL: www.smqzj.com/play-49384-…

Download www.fztxylgy.com:65/20210218/7E… Video, you get this exception, why is this problem?

One of the shard: www.fztxylgy.com:65/20210218/7E…

contentLength=255304, totalLength=203951, fileLength=203951, url=https://www.fztxylgy.com:65/20210218/7EeCKw5j/1500kb/hls/iOzaXozO.ts
Copy the code

ContentLength is larger than fileLength. If inputStream finishes reading the content, it checks the contentLength. If inputStream finishes reading the content, it checks the contentLength.

2. Source tracking

According to the Android source code analysis: / external/okhttp/okhttp/SRC/main/Java/com/squareup/okhttp/internal/HTTP/Http1xStream Java

377  /** An HTTP body with a fixed length specified in advance. */
378  private class FixedLengthSource extends AbstractSource {
379    private long bytesRemaining;
380
381    public FixedLengthSource(long length) throws IOException {
382      bytesRemaining = length;
383      if (bytesRemaining == 0) {
384        endOfInput();
385      }
386    }
387
388    @Override public long read(Buffer sink, long byteCount) throws IOException {
389      if (byteCount < 0) throw new IllegalArgumentException("byteCount < 0: " + byteCount);
390      if (closed) throw new IllegalStateException("closed");
391      if (bytesRemaining == 0) return -1;
392
393      long read = source.read(sink, Math.min(bytesRemaining, byteCount));
394      if (read == -1) {
395        unexpectedEndOfInput(); // The server didn't supply the promised content length.
396        throw new ProtocolException("unexpected end of stream");
397      }
398
399      bytesRemaining -= read;
400      if (bytesRemaining == 0) {
401        endOfInput();
402      }
403      return read;
404    }
405
406    @Override public void close() throws IOException {
407      if (closed) return;
408
409      if (bytesRemaining != 0
410          && !Util.discard(this, DISCARD_STREAM_TIMEOUT_MILLIS, MILLISECONDS)) {
411        unexpectedEndOfInput();
412      }
413
414      closed = true;
415    }
416  }
Copy the code

Paste core code:

393 long read = source.read(sink, Math.min(bytesRemaining, byteCount)); 394 if (read == -1) { 395 unexpectedEndOfInput(); // The server didn't supply the promised content length. 396 throw new ProtocolException("unexpected end of stream"); 397}Copy the code

The initial value of bytesRemaining is contentLength, which is when we send the request and receive the content-Length in the response.

Here’s the flow:

  • Read data from the byte stream in Response. Before each read, determine whether bytesRemaining is 0. If 0, it indicates that the response has no data (regardless of whether there is data in fact, contentLength is used as the standard in this case). Anyway, I read a contentLength bitstream until I finish.
  • If bytesRemaining is not 0 each time, but the Byte stream has finished reading, and there is no valid data to read, then there is a problem, because the server tells me that I should read contentLength, But I have not read the contentLength length, there is no data, this is not a problem?

When is an exception thrown? ===== reads the requested byte stream, and raises this exception if it finds that the end of the byte stream is read and the length of the byte stream is inconsistent with the contentLength.

If the “transfer-encoding” header in the response header is “chunked”, create a newChunkedSource instance. If contentLength is in the response header, create a FixedLengthSource instance. NewChunkedSource is a request of undetermined length.

135 private Source getTransferStream(Response response) throws IOException { 136 if (! HttpEngine.hasBody(response)) { 137 return newFixedLengthSource(0); 138 } 139 140 if ("chunked".equalsIgnoreCase(response.header("Transfer-Encoding"))) { 141 return newChunkedSource(httpEngine); 142 } 143 144 long contentLength = OkHeaders.contentLength(response); 145 if (contentLength ! = -1) { 146 return newFixedLengthSource(contentLength); 147 } 148 149 // Wrap the input stream from the connection (rather than just returning 150 // "socketIn" directly here),  so that we can control its use after the 151 // reference escapes. 152 return newUnknownLengthSource(); 153}Copy the code

Since FixedLengthSource is a fixed length request processing, it indicates that contentLength is a very important verification value in the process of the request. If the length read is inconsistent with contentLength, the system will consider that the current requested data cannot be verified. So the exception that we started with in this article occurs.

The above code provides us very meaningful thinking, since we are using FixedLengthSource java.net.ProtocolException: happens when instance request Unexpected end of stream problem, can we use newChunkedSource to try to solve this problem?

3. Think about

If in the process of download video files java.net.ProtocolException: happened unexpected end of stream abnormal problem

  • (1) in sell java.net.ProtocolException: Unexpected end of stream: verify that fileLength is consistent with contentLength. If not, Transfer-Encoding: chunked can be set to avoid client validation
  • (2) For frequent requests from a resource or the same server, it is best to set Connection: Close, which can prevent the server from the problem of anti-attack. The server finds that the request for a short time is cut too frequently and keeps long links for a long time, which will cause the server to break down. Therefore, the long link request for the server will be dismissed, resulting in the current request being abnormal.