Remember the MongoDB data garble processing problem once

The front end group of the company is separating the front end from the back end, which needs to be provided with background interface. The company has not built a usable Mock server, and it happened to have a small Demo made by Flask. Now Docker can still use it, so I will provide the background interface without further ado. With the help of their own Ali Cloud set up my micro service architecture.

Flask start

The API provided by Flask is quite simple and fully serves the design concept of micro-service architecture. What we plan to do is a book list management, involving book list, details, addition and deletion.

The API design is as follows

@app.route('/books')
def books():
    return book_model.get_books()
@app.route('/books/<string:book_id>')
def book(book_id):
    ......
@app.route('/books/add', methods=['GET'])
def add():
    ......
@app.route('/books/<string:book_id>/delete')
    ......Copy the code

error

When the data from the database is transferred to the front end, it becomes Unicode encoding and all Chinese is converted.

The corresponding data processing code is as follows

## book_model.py from bson import json_util as jsonb #...... def get_books(): db = MongoDB().get_db('liberiry') result = {} db_result = db.books.find() if db_result: Result ['status']= True result['data'] = jsonb.dumps(list(db_result)) Return result ## controller.py import json #...... @app.route('/books') def books(): return Response(json.dumps(book_model.get_books()))Copy the code

The general idea is to convert the Bson data obtained from MongoDB to strings, which are then converted back to JSON by the DUMPS method of JSON. The result of the conversion is wrong.

~ docker container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c17296ed0c05 flask:latest "gunicorn -b :8080 A..." 4 days a line Up 4 days 8080 / TCP liberiry_flask. 1. 451 d71a2dc8a vu70ur9cuaimefxl01979haqw mongo: 3.6 "docker - entrypoint. S..." 4 days ago Up 4 days 27017/tcp liberiry_mongo.1.prlfqd3nuuo3y985dznw8nmcs ~ docker exec -it 451 /bin/bash root@451d71a2dc8a:/# mongo MongoDB shell version v3.6.6 Connecting to: Mongo: / / 127.0.0.1:27017 mongo server version: 3.6.6 Welcome to the MongoDB shell. For interactive help, type "help". For more comprehensive documentation, see http://docs.mongodb.org/ Questions? Try the support group http://groups.google.com/group/mongodb-user Server has startup warnings: 2018-07-27T03:21:16.488+0000 I STORAGE [initandListen] 2018-07-27T03:21:16.488+0000 I STORAGE [initandListen] ** WARNING: Strongly Recommended with the WiredTiger Storage Engine 2018-07-27T03:21:16.488+0000 I STORAGE [initandlisten] * * See http://dochub.mongodb.org/core/prodnotes-filesystem 2018-07-27 T03: mark. 877 + 0000 I CONTROL [initandListen] 2018-07-27T03:21:20.878+0000 I CONTROL [initandListen] ** WARNING: Access Control is not enabled for the database. 2018-07-27T03:21:20.878+0000 I control [initandListen] ** Read and write Access to data and Configuration is unrestricted. 2018-07-27T03:21:20.878+0000 I CONTROL [initandListen] 2018-07-27T03:21:20.878+0000 I CONTROL [initandListen] 2018-07-27T03:21:20.878+0000 I CONTROL [initandListen] ** WARNING: / sys/kernel/mm/transparent_hugepage/enabled is' always'. The 2018-07-27 T03: mark. 878 + 0000 I CONTROL [initandlisten] We * * Suggest setting it to 'never' 2018-07-27T03:21:20.878+0000 I CONTROL [initandListen] 2018-07-27T03:21:20.878+0000 I CONTROL [initandlisten] ** WARNING: / sys/kernel/mm/transparent_hugepage/defrag is' always'. The 2018-07-27 T03: mark. 878 + 0000 I CONTROL [initandlisten] We * * 2018-07-27T03:21:20.878+0000 I CONTROL [initandlisten] > use liberiry switched to DB Liberiry > db.books.find(). Pretty () {"_id" : ObjectId("5b59934719fe22fc8ecfad19"), "name" : "Content" : "Test China "," Labels ": "book,bussiness", "author" :" Ben Horowitz "} {"_id" : ObjectId(" 5b5993b843314814F50D263a "), "name" : "Zero to one "," Content ":" None of these answers are good. Diyi and the second statement may be true, but many have already agreed. The third simply paraphrases one side of a common debate. A good answer goes something like this: "Most people believe X, but the truth is the opposite of X." I will give my own answer later in this chapter. So what does this countermainstream problem have to do with the future? In small ways, the future is just a collection of moments yet to come. But what makes the future so unique and important is not that it didn't happen, but that it will be a different world. In this way, if our society hasn't changed in the next 100 years, the future is more than 100 years away. If the world changes in the next 10 years, the future will be within reach. No one can predict the future accurately, but we do know two things: the world is bound to be different, but the change has to be based on the world today. Most of the answers to this counterintuitive question are different views of the present, and the best answers are as far into the future as possible." , "Labels" : "book,bussiness", "author" : "Peter Thile, Blake Masters"} {"_id" : "labels" : "Book,bussiness"," Author ":" Peter Thile, Blake Masters "} {"_id" : "labels" : "Book,bussiness"," Author ":" Peter Thile, Blake Masters "} {"_id" : ObjectId(" 5B5993e443314814F50D2642 "), "Name" : "Alliance "," Content ":" It is time to rebuild the relationship between employer and employee. The business world needs a new framework of employment relations that is conducive to mutual trust, mutual investment, and mutual benefit. "The ideal employment relationship framework should encourage people to develop personal networks and get things done, rather than become job-hopping professionals seeking to make money." , "labels" : "book,bussiness", "author" : "Reid · Hoffman"} {"_id" : ObjectId(" 5b59947343314814F50d2650 "), "name" : Labels ": "book,bussiness", "author" :" Reid · Hoffman "} {"_id" : ObjectId(" 5b59947343314814F50d2650 "), "name" : Along the way, there were three important revolutions: About 70,000 years ago, history began in earnest with the Cognitive Revolution. About 12,000 years ago, the "Agricultural Revolution" accelerated history. Only about 500 years ago, the "Scientific Revolution" ended history and made innovation happen. This book is about how these three revolutions have changed human beings and other creatures." , "labels" : "book,bussiness", "author" : "Yoval Helari"} {"_id" : ObjectId(" 5b59954a43314814F50d2670 "), "name" : Labels ": "book,bussiness", "author" :" Yoval Helari "} {"_id" : ObjectId(" 5b59954a43314814F50d2670 "), "name" : "Pay Wars," "Content" : "My unsettling experience on the job, and the disorganization of my first day on the job, showed that Confiniti was anything but an organized company. As a young company, IT is understandable that IT does not have enough resources to invest in hr and IT departments to provide planned orientation to new employees. Still, the apparent chaos is a little disconcerting. I hated the thought, but at noon that first day I found myself wondering whether my blind faith in Peter, and my eagerness to make a quick buck, had led me to make a terrible mistake." , "Labels" : "book,bussiness", "author" : "Eric Jackson"} {"_id" : ObjectId(" 5b59AC30CDc864001148ba19 "), "name" : Labels ": "book,bussiness", "author" :" Eric Jackson "} {"_id" : ObjectId(" 5b59AC30CDc864001148ba19 "), "name" : "asdfs", "content" : "asdfasdf", "lables" : "dfasdf,sadf", "author" : "test" } { "_id" : ObjectId("5b59ac38cdc864001148ba1b"), "name" : "asdsfs", "content" : "asdfasdf", "lables" : "dfasdf,sadf", "author" : "test" }Copy the code

There is no problem in MongoDB query, so the problem should be in the Bson to JSON part; The result from the database is the following structure

"_id" : ObjectId("5b59ac38cdc864001148ba1b"),Copy the code

Json cannot convert Object, baidu Google once, everyone’s method is to query the result of traversal processing, as follows:

Results = collection.find({'age': 20}) print(results) for result in results: print(result) #Copy the code

Obviously, it will take time and effort to do the traversal again for the results, and the performance will surely decline greatly with a large amount of data, so this kind of processing method is not desirable.

solution

Loads the data into bson objects, then json.loads() converts it to a string that json can handle, and then converts it to JSON objects. From this, we can write a general approach,

## utils.py
from bson import json_util as jsonb
import json
def bson_to_json(documentList):
    return json.dumps(json.loads(jsonb.dumps(documentList)))Copy the code

validation

## book_modal.py
from .mongo_connection import MongoDB
from .utils import bson_to_json
def get_books():
    db = MongoDB().get_db('liberiry')
    result = {}
    db_result = db.books.find() #'_id': False
    if db_result:
        result['status']= True
        result['data'] = db_result
        del db_result
        return bson_to_json(result)Copy the code

The front-end calls

conclusion

Transformations between data are always related; Json_util dumps in bson are converted to parsed data with Object. Loads (Obj)), and the DUMPS method of JSON parses the strings into standard JSON objects.

Existing immediately and reasonably.

mo4tech.com (Moment For Technology) is a global community with thousands techies from across the global hang out!Passionate technologists, be it gadget freaks, tech enthusiasts, coders, technopreneurs, or CIOs, you would find them all here.

Remember the MongoDB data garble processing problem once

Flask start

The API design is as follows

error

solution

validation

The front-end calls

conclusion

Personal public account

Remember the MongoDB data garble processing problem once

Flask start

The API design is as follows

error

solution

validation

The front-end calls

conclusion

Personal public account

Related Posts

The Spring source reading notes (6) the main line registerBeanPostProcessors interpretation

Read the pros and cons of blue-green publishing, A/B testing, and Canary publishing

Interceptor and proxy classes 🐣