Image credit: unsplash.com

By Xie Fugui

background

WebView can be seen everywhere in mobile applications, and in cloud music as an entry point for many core businesses. To meet the increasingly complex business scenarios of cloud music, we are constantly optimizing WebView performance. One of the techniques that can improve WebView loading speed in a short time is offline package technology. This technology can save the network loading time, especially for large web pages. The most critical link in offline package technology is to intercept requests from WebView and map resources to local offline package. For WKWebView request interception, iOS system native does not provide direct ability, so this article will focus on WKWebView request interception to discuss.

research

We have studied the existing WKWebView request interception schemes in the industry, which are mainly divided into the following two types:

NSURLProtocol

By default, NSURLProtocol intercepts all requests that pass through the URL Loading System. Therefore, requests issued by WKWebView that pass through the URL Loading System can be intercepted. After our attempts, we found that WKWebView runs independently of the application process, and the sent request does not pass through the URL Loading System by default, which requires us to hook additional to support. For the specific method, please refer to the processing of WKWebView by NSURLProtocol.

WKURLSchemeHandler

WKURLSchemeHandler is a new feature introduced in iOS 11 that manages data for custom requests, To support data management for HTTP or HTTPS requests, hook the handlesURLScheme: method of WKWebView and return NO.

After some attempts and analysis, we compare the two schemes from the following aspects:

  • Isolation:NSURLProtocolOnce registered, it is globally enabled. Normally we would only block our own business pages, but useNSURLProtocolIn this way, the pages of the three parties cooperating within the app will also be blocked and thus contaminated.WKURLSchemeHandlerYou can isolate the page as a dimension because it is followingWKWebViewConfigurationPerform configuration.
  • Stability:NSURLProtocolThe Body is lost during interception,WKURLSchemeHandlerBefore iOS 11.3 (not included) the Body will also be lost, after iOS 11.3 WebKit has been optimized to only lose Blob type data.
  • Consistency:WKWebViewThe request issued isNSURLProtocolBehavior may change after interception. For example, a video that wants to unload the video tag would normally set the resource address (SRC) to empty, but in this casestopLoadingMethod is not called, by contrastWKURLSchemeHandlerBehave normally.

The survey concluded that:WKURLSchemeHandlerThe performance is superior in isolation, stability and consistencyNSURLProtocol, but the problem of missing bodies must be addressed in order to be used in production.

Our solution

It can be seen from the above that request interception using WKURLSchemeHandler alone cannot cover all request scenarios, because there is a case of Body loss. So we focused on making sure that the Body data could not be lost or that we could get the Body data in advance and then assemble it into a complete request. Obviously, the former required a costly change to the WebKit source code, so we chose the latter. By modifying the JavaScript native Fetch/XMLHttpRequest interface implementation to get the Body data in advance, the scheme design is as shown below:

The specific process is mainly as follows:

  • Inject custom when HTML documents are loadedFetch / XMLHttpRequestObject script
  • Before sending a request, collect parameters such as BodyWKScriptMessageHandlerPass to the native application for storage
  • The native application is notified when the store is complete by calling a convention JavaScript functionWKWebViewSave your
  • Call nativeFetch / XMLHttpRequestWait for the interface to send the request
  • The request isWKURLSchemeHandlerManage, take out the corresponding parameters such as the Body to assemble and then send

Script injection

Replace Fetch implementation

Script injection needs to modify the processing logic of the Fetch interface. Before the request is sent, parameters such as Body can be collected and passed to the native application. The main problems to solve are as follows:

  • The Body is lost before iOS 11.3
  • IOS 11.3 after the BodyBlobType data loss problem

1. For the first point, it is necessary to determine whether the request sent by the device before iOS 11.3 contains the request body. If yes, the request body data needs to be collected and passed to the native application before calling the native Fetch interface.

2. For the second point, it is also necessary to determine whether the request sent by the device after iOS 11.3 contains the request body and whether the request body contains Blob type data. If the request body is satisfied, the processing is the same as above.

For the rest of the case, simply call the native Fetch interface directly, keeping the native logic.

var nativeFetch = window.fetch
var interceptMethodList = ['POST'.'PUT'.'PATCH'.'DELETE'];
window.fetch = function(url, opts) {
  // Determine whether to include the request body
  varhasBodyMethod = opts ! =null&& opts.method ! =null&& (interceptMethodList.indexOf(opts.method.toUpperCase()) ! = = -1);
  if (hasBodyMethod) {
    // Check whether the state is before iOS 11.3 (via navigate. UserAgent)
    var shouldSaveParamsToNative = isLessThan11_3;
    if(! shouldSaveParamsToNative) {// Whether the request body contains Blob type data after iOS 11.3shouldSaveParamsToNative = opts ! =null ? isBlobBody(opts) : false;
    }
    if (shouldSaveParamsToNative) {
      // In this case, the request body data needs to be collected and saved to the native application
      return saveParamsToNative(url, opts).then(function (newUrl) {
        // Call the native FETCH interface after the application saves
        returnnativeFetch(newUrl, opts) }); }}// Call the native FETCH interface
  return nativeFetch(url, opts);
}
Copy the code

Save the request body data to the native application

The WKScriptMessageHandler interface is used to save the request body data to the native application and needs to generate a unique identifier corresponding to the specific request body data for subsequent retrieval. The idea is to generate the standard UUID as the identifier and pass it along with the request body data to the native application for storage, and then concatenate the UUID identifier into the request link. After the request is managed by the WKURLSchemeHandler, it retrieves the specific request body data through this identifier and then assembles it into a request.

function saveParamsToNative(url, opts) {
  return new Promise(function (resolve, reject) {
    // Construct the identifier
    var identifier = generateUUID();
    var appendIdentifyUrl = urlByAppendIdentifier(url, "identifier", identifier)
    // Parse the body data and save it to the native application
    if (opts && opts.body) {
      getBodyString(opts.body, function(body) {
        // Set the callback when the save is complete. The native application will call this JS function to issue the request after the save is complete
        finishSaveCallbacks[identifier] = function() {
          resolve(appendIdentifyUrl)
        }
        // Notify the native application to save the request body data
        window.webkit.messageHandlers.saveBodyMessageHandler.postMessage({'body': body, 'identifier': identifier}})
      });
    }else{ resolve(url); }}); }Copy the code

Request body resolution

In the Fetch interface, the request body parameter, namely opTS.body, can be obtained through the second OPTS parameter, and there are seven types of request bodies according to MDN Fetch Body. After analysis, these seven data types can be divided into three types for parsing and encoding processing. ArrayBuffer, ArrayBufferView, Blob, File are classified as binary type, string, URLSearchParams are classified as string type. FormData is classified as a compound type and returned to the native application as a string type.

function getBodyString(body, callback) {
  if (typeof body == 'string') {
    callback(body)
  }else if(typeof body == 'object') {
    if (body instanceof ArrayBuffer) body = new Blob([body])
    if (body instanceof Blob) {
      // Convert Blob type to base64
      var reader = new FileReader()
      reader.addEventListener("loadend".function() {
        callback(reader.result.split(",") [1])
      })
      reader.readAsDataURL(body)
    } else if(body instanceof FormData) {
      generateMultipartFormData(body)
      .then(function(result) {
        callback(result)
      });
    } else if(body instanceof URLSearchParams) {
      // Iterate over URLSearchParams for key-value pair concatenation
      var resultArr = []
      for (pair of body.entries()) {
        resultArr.push(pair[0] + '=' + pair[1])
      }
      callback(resultArr.join('&'))}else{ callback(body); }}else{ callback(body); }}Copy the code

Binary types are uniformly converted to Base64 encoding for ease of transmission. The string type URLSearchParams iterates through to get the key-value pair. Compound types are stored in a structure similar to a dictionary, and the value may be of type string or Blob, so you need to iterate and concatenate in Multipart/form-data format.

other

The main content of the injected script is as shown above. In this example, the Fetch implementation is replaced, and XMLHttpRequest is replaced along the same lines. Cloud music due to the lowest version supports to the iOS 11.0, while the FormData. Prototype. Entries are only after the iOS 11.2 support, and for the previous version can modify the FormData. The prototype. The realization of the set method to save key-value pairs, I won’t go into much detail here. In addition, the request may be issued by an embedded iframe. Calling finishSaveCallbacks[Identifier]() directly is invalid because finishSaveCallbacks are mounted on the Main Window. Consider using the window.postMessage method to communicate with child Windows.

WKURLSchemeHandler intercepts the request

The registration and use of WKURLSchemeHandler is no longer described here, the specific can refer to the survey section and the Apple documentation above, here we mainly talk about the intercept process to pay attention to the point

redirect

Some readers may have noticed that when we introduced WKURLSchemeHandler in the survey section above, its role was defined as data management for custom requests. So why not custom request data interception? In theory interception does not require the developer to care about the request logic, the developer only needs to process the data in the process. For data management, the developer needs to pay attention to all the logic in the process and then return the final data. With these two definitions in mind, let’s compare WKURLSchemeTask and NSURLProtocol. It can be seen that the latter has more redirection, authentication and other related request processing logic than the former.

API_AVAILABLE(macos(10.13), ios(11.0))
@protocol WKURLSchemeTask <NSObject>

@property (nonatomic.readonly.copy) NSURLRequest *request;

- (void)didReceiveResponse:(NSURLResponse *)response;

- (void)didReceiveData:(NSData *)data;

- (void)didFinish;

- (void)didFailWithError:(NSError *)error;

@end
Copy the code
API_AVAILABLE(macos(10.2), ios(2.0), watchos(2.0), tvos(9.0))
@protocol NSURLProtocolClient <NSObject>

- (void)URLProtocol:(NSURLProtocol *)protocol didReceiveResponse:(NSURLResponse *)response cacheStoragePolicy:(NSURLCacheStoragePolicy)policy;

- (void)URLProtocol:(NSURLProtocol *)protocol didLoadData:(NSData *)data;

- (void)URLProtocolDidFinishLoading:(NSURLProtocol *)protocol;

- (void)URLProtocol:(NSURLProtocol *)protocol didFailWithError:(NSError *)error;

- (void)URLProtocol:(NSURLProtocol *)protocol didReceiveAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge;

- (void)URLProtocol:(NSURLProtocol *)protocol didCancelAuthenticationChallenge:(NSURLAuthenticationChallenge *)challenge;

@end
Copy the code

So how do you handle the redirection response during interception? We try to call didReceiveResponse every time we receive a response: Method to find that the intermediate redirection response will be overwritten by the last received response, so that WKWebView will not be aware of the redirection, so that the address and other related information will not change, for some pages with routing judgment may have some unexpected effects. Once again, we are stuck with the fact that the WKURLSchemeHandler does not support redirection when fetching data, because Apple designed it as purely data management. In fact, we can get the response every time, but can not pass the complete WKWebView. After some evaluation, we finally chose the reload method to solve the HTML document request redirection problem for three reasons.

  • The only thing that can be changed right now isFetch 和 XMLHttpRequestThe implementation of the interface, for both document requests and HTML tag initiating requests, is an internal behavior of the browser, and it is too costly to modify the source code.
  • Fetch 和 XMLHttpRequestBy default, only the final response is returned, so the final data is guaranteed to be correct at the server interface level, and the loss of the redirection response has little impact.
  • The same goes for images/videos/forms/style sheets/scripts as long as the final data of the relationship is correct.

A redirection response received from an HTML document is returned directly to WKWebView and further loading is cancelled. For redirection of other resources, discard is selected.

- (void)URLSession:(NSURLSession *)session task:(NSURLSessionTask *)task willPerformHTTPRedirection:(NSHTTPURLResponse *)response newRequest:(NSURLRequest *)request completionHandler:(void(^) (NSURLRequest * _Nullable))completionHandler {                  
  NSString *originUrl = task.originalRequest.URL.absoluteString;
  if ([originUrl isEqualToString:currentWebViewUrl]) {
    [urlSchemeTask didReceiveResponse:response];
    [urlSchemeTask didFinish];
    completionHandler(nil);
  }else{ completionHandler(request); }}Copy the code

WKWebView after receiving the response data is called a webView: decidePolicyForNavigationResponse: decisionHandler method to determine the final jump, In this method, you can get the destination Location of the redirect to reload.

- (void)webView:(WKWebView *)webView decidePolicyForNavigationResponse:(WKNavigationResponse *)navigationResponse decisionHandler:(void(^) (WKNavigationResponsePolicy))decisionHandler
{
  // Intercept is enabled
  if (enableNetworkIntercept) {
    if ([navigationResponse.response isKindOfClass:[NSHTTPURLResponse class]]) {
        NSHTTPURLResponse *httpResp = (NSHTTPURLResponse *)navigationResponse.response;
        NSInteger statusCode = httpResp.statusCode;
        NSString *redirectUrl = [httpResp.allHeaderFields stringForKey:@"Location"];
        if (statusCode >= 300 && statusCode < 400 && redirectUrl) {
            decisionHandler(WKNavigationActionPolicyCancel);
            // the 307 and 308post jump scenarios are not supported
            [webView loadHTMLWithUrl:redirectUrl]; 
            return;
        }
    }
  }
  decisionHandler(WKNavigationResponsePolicyAllow);
}
Copy the code

At this point, the redirection of HTML documents is basically over. We haven’t seen any boundary issues until this article is published, but if you have any other ideas, please feel free to discuss them.

Cookies are synchronous

Because WKWebView is not in the same process as our application, WKWebView and NSHTTPCookieStorage are not in sync. Instead of going through the entire process of WKWebView Cookie synchronization, we will focus on Cookie synchronization during interception. Since the request is ultimately made by the native application, the Cookie is read and stored at NSHTTPCookieStorage. Note that the response returned by WKURLSchemeHandler to the WKWebView contains set-cookie information, but the WKWebView is not Set to the Document. Cookie. The WKURLSchemeHandler is only responsible for data management. The logic involved in the request needs to be handled by the developer.

Cookie synchronization of WKWebView can be achieved through the WKHTTPCookieStore object

- (void)URLSession:(NSURLSession *)session dataTask:(NSURLSessionDataTask *)dataTask didReceiveResponse:(NSURLResponse *)response completionHandler:(void(^) (NSURLSessionResponseDisposition))completionHandler
{
  if ([response isKindOfClass:[NSHTTPURLResponse class]]) {
    NSHTTPURLResponse *httpResp = (NSHTTPURLResponse *)response;
    NSArray <NSHTTPCookie *>*responseCookies = [NSHTTPCookie cookiesWithResponseHeaderFields:[httpResp allHeaderFields] forURL:response.URL];
    if ([responseCookies isKindOfClass:[NSArray class]] && responseCookies.count > 0) {
        dispatch_async(dispatch_get_main_queue(), ^{
            [responseCookies enumerateObjectsUsingBlock:^(NSHTTPCookie * _Nonnull cookie, NSUInteger idx, BOOL * _Nonnull stop) {
                // Synchronize to WKWebView
                [[WKWebsiteDataStore defaultDataStore].httpCookieStore setCookie:cookie completionHandler:nil];
            }];
        });
    }
  }
  completionHandler(NSURLSessionResponseAllow);
}
Copy the code

In addition to synchronizing the native application’s Cookie to WKWebView during the interception process, the modification of document. Cookie should also be synchronized to the native application. After an attempt, it was found that the document.cookie on the real device would actively delay synchronization to NSHTTPCookieStorage after modification, but the emulator did not do any synchronization. For some requests sent immediately after modifying the document.cookie, the request may not carry the changed cookie information immediately, because the cookie is NSHTTPCookieStorage after interception.

Our solution is to modify the document.cookie setter method implementation to synchronize to the native application before the cookie is set. Caution In native applications, cross-domain verification is required to prevent malicious pages from arbitrarily modifying cookies.

(function() {
  var cookieDescriptor = Object.getOwnPropertyDescriptor(Document.prototype, 'cookie') || Object.getOwnPropertyDescriptor(HTMLDocument.prototype, 'cookie');
  if (cookieDescriptor && cookieDescriptor.configurable) {
    Object.defineProperty(document, 'cookie', {
      configurable: true,
      enumerable: true,
      set: function (val) {
        // Pass the Settings to the native application before they take effect
        window.webkit.messageHandlers.save.postMessage(val);
        cookieDescriptor.set.call(document, val);
      },
      get: function () {
        returncookieDescriptor.get.call(document); }}); }}) ()Copy the code

Memory leak caused by NSURLSession

Through NSURLSession sessionWithConfiguration: delegate: delegateQueue constructor to create the object when the delegate is NSURLSession strong references, this is easy to ignore you. We will create an NSURLSession object for each WKURLSchemeHandler object and set the former as the delegate of the latter, resulting in a circular reference. It is recommended to call the invalidateAndCancel method of NSURLSession on WKWebView destruction to remove a strong reference to the WKURLSchemeHandler object.

Stability improvement

As we can see from the above, if we “work against” the system (WKWebView itself does not support HTTP/HTTPS interception), many unexpected things will happen and there may be many boundaries to cover, so we must have a set of comprehensive measures to improve the stability of the interception process.

Dynamic distributed

We can turn off the blocking of some pages by dynamically sending the blacklist. By default, cloud Music preloads two empty WkWebViews. One is a WKWebView registered with WKURLSchemeHandler to load the master page, and supports blacklist closing. The other is a normal WKWebView to load some three-party pages (because the logic of three-party pages is varied and complex, and there is no need to block three-party page requests). In addition, for some teams who just try to solve the loss of the request body through script injection, they may not be able to cover all scenarios. They can try to update the script by dynamic delivery, and also sign the script content to prevent malicious tampering.

monitoring

Log collection can help us better identify potential problems. All the request logic during interception is consolidated in a WKURLSchemeHandler, and we can log on some key links. For example, you can collect whether the injected script performed abnormally, whether the Body received was lost, whether the response status code returned was normal, and so on.

Full proxy request

In addition to the above measures, we can also fully proxy network requests such as the server API interface to the client. The front-end only passes the corresponding parameters to the native application through JSBridge and then obtains the data through the network request channel of the native application. In addition to reducing potential problems during interception, this method can also reuse some network related capabilities of native applications such as HTTP DNS, anti-cheating, etc. And it is worth noting that iOS 14 Apple in the WKWebView on the default enabled ITP (Intelligent Tracking Prevention) Intelligent Tracking function, affected by the main place is the use of cross-domain cookies and Storage. For example, there are some three-party pages in our application that need to be authorized through an iframe embedded with our page. At this time, because the cross-domain default is unable to obtain the Cookie under the domain name of our master site, the proxy request of the native application can solve similar problems. Finally, if you use this method, remember to do a good authentication check, to prevent some malicious pages to invoke the ability, after all, native application requests are not cross-domain restrictions.

summary

In this paper, the iOS native WKURLSchemeHandler is combined with JavaScript script injection to realize the request interception capability required by WKWebView in offline package loading, stream-free and other services. The problem of redirection, request body loss and Cookie asynchronization can be solved in the interception process, and the interception and isolation can be carried out in the dimension of page. As we explored, we increasingly realized that technology knows no boundaries, and sometimes it is impossible to achieve a complete set of capabilities on one side, perhaps due to the limitations of the platform. Only by combining the technical capabilities of relevant platforms can a reasonable technical solution be developed. Finally, this article is some of our exploration practices in WKWebView request interception, if there are errors welcome corrections and communication.

This article is published by netease Cloud Music front end team. Any unauthorized reprint of the article is prohibited. We hire front end, iOS, Android all the year around. If you’re ready for a career change and you love cloud music, join us GRP. Music-fe (at) Corp.netease.com!