Recently, I met a demand in my work, that is, to open an interface to receive the data pushed by suppliers. The project uses The Python Django framework. Without thinking about it, I wrote the following code directly:
class XXDataPushView(APIView) :
"" receive xx data push ""
#...
@white_list_required
def post(self, request, **kwargs) :
req_data = request.data or {}
#...
Copy the codeBut then, we found that there was no change in the daily data, and asked whether the supplier did not push them and cheated us. The reply was that they were pushing a Gzzip-compressed data stream, and the receiver needed to proactively decompress it. I have never dealt with such compressed data before, and the specific push of the other side is also a black box for me.
Therefore, I asked for a simple example to be pushed, but I was surprised that the other party did not talk about martial virtue, but still came back with a piece of Java code that could not be run alone:
private byte[] compress(JSONObject body) {
try {
ByteArrayOutputStream out = new ByteArrayOutputStream();
GZIPOutputStream gzip = new GZIPOutputStream(out);
gzip.write(body.toString().getBytes());
gzip.close();
return out.toByteArray();
} catch (Exception e) {
logger.error("Compress data failed with error: " + e.getMessage()).commit();
}
return JSON.toJSONString(body).getBytes();
}
public void post(JSONObject body, String url, FutureCallback<HttpResponse> callback) {
RequestBuilder requestBuilder = RequestBuilder.post(url);
requestBuilder.addHeader("Content-Type"."application/json; charset=UTF-8");
requestBuilder.addHeader("Content-Encoding"."gzip");
byte[] compressData = compress(body);
int timeout = (int) Math.max(((float)compressData.length) / 5000000.5000);
RequestConfig.Builder requestConfigBuilder = RequestConfig.custom();
requestConfigBuilder.setSocketTimeout(timeout).setConnectTimeout(timeout);
requestBuilder.setEntity(new ByteArrayEntity(compressData));
requestBuilder.setConfig(requestConfigBuilder.build());
excuteRequest(requestBuilder, callback);
}
private void excuteRequest(RequestBuilder requestBuilder, FutureCallback<HttpResponse> callback) {
HttpUriRequest request = requestBuilder.build();
httpClient.execute(request, new FutureCallback<HttpResponse>() {
@Override
public void completed(HttpResponse httpResponse) {
try {
int responseCode = httpResponse.getStatusLine().getStatusCode();
if(callback ! =null) {
if (responseCode == 200) {
callback.completed(httpResponse);
} else {
callback.failed(new Exception("Status code is not 200")); }}}catch (Exception e) {
logger.error("Get error on " + requestBuilder.getMethod() + "" + requestBuilder.getUri() + ":" + e.getMessage()).commit();
if(callback ! =null) {
callback.failed(e);
}
}
EntityUtils.consumeQuietly(httpResponse.getEntity());
}
@Override
public void failed(Exception e) {
logger.error("Get error on " + requestBuilder.getMethod() + "" + requestBuilder.getUri() + ":" + e.getMessage()).commit();
if(callback ! =null) { callback.failed(e); }}@Override
public void cancelled(a) {
logger.error("Request cancelled on " + requestBuilder.getMethod() + "" + requestBuilder.getUri()).commit();
if(callback ! =null) { callback.cancelled(); }}}); }Copy the codeAs you can see from the above code, the other party compresses the JSON data into a GZIP data stream. I searched django’s documentation and found only this paragraph describing the decorator for gzip processing:
Django. Views. Decorators. Gzip decorator control based on the content of each view in compression.
¶ gzip_page ()
If the browser allows gzip compression, this decorator compresses the content. It sets the Vary header accordingly so that the cache will be stored based on the Accept-Encoding header.
However, this decorator simply compresses the content of the request response to the browser, and our current requirement is to decompress the received data. That’s not what we want.
Fortunately, there is an extension in Flask called flask-inflate that automatically uncompresses the requested data. See the code handling for this extension:
# flask_inflate.py
import gzip
from flask import request
GZIP_CONTENT_ENCODING = 'gzip'
class Inflate(object) :
def __init__(self, app=None) :
if app is not None:
self.init_app(app)
@staticmethod
def init_app(app) :
app.before_request(_inflate_gzipped_content)
def inflate(func) :
""" A decorator to inflate content of a single view function """
def wrapper(*args, **kwargs) :
_inflate_gzipped_content()
return func(*args, **kwargs)
return wrapper
def _inflate_gzipped_content() :
content_encoding = getattr(request, 'content_encoding'.None)
ifcontent_encoding ! = GZIP_CONTENT_ENCODING:return
# We don't want to read the whole stream at this point.
# Setting request.environ['wsgi.input'] to the gzipped stream is also not an option because
# when the request is not chunked, flask's get_data will return a limited stream containing the gzip stream
# and will limit the gzip stream to the compressed length. This is not good, as we want to read the
# uncompressed stream, which is obviously longer.
request.stream = gzip.GzipFile(fileobj=request.stream)
Copy the codeThe core of the above code is:
request.stream = gzip.GzipFile(fileobj=request.stream)
Copy the codeSo, in Django, you can do something like this:
class XXDataPushView(APIView) :
"" receive xx data push ""
#...
@white_list_required
def post(self, request, **kwargs) :
content_encoding = request.META.get("HTTP_CONTENT_ENCODING"."")
ifcontent_encoding ! ="gzip":
req_data = request.data or {}
else:
gzip_f = gzip.GzipFile(fileobj=request.stream)
data = gzip_f.read().decode(encoding="utf-8")
req_data = json.loads(data)
#... handle req_data
Copy the codeOk, problem solved perfectly. You can also test the request as follows:
import gzip
import requests
import json
data = {}
data = json.dumps(data).encode("utf-8")
data = gzip.compress(data)
resp = requests.post("http://localhost:8760/push_data/",data=data,headers={"Content-Encoding": "gzip"."Content-Type":"application/json; charset=utf-8"})
print(resp.json())
Copy the code