Layout: Post Date: 2020-06-09 Tag: Note Author: BY Zhi-kai Yang

The first version of Frodo has already been implemented, and I’ve organized the current development ideas into three articles for the next version: Data, Communications, and Asynchrony.

The project address

Blog address

This article looks at the logical flow that implements specific functions. In the Web application summary, I personally prefer to refer to business processes as “communications.” Because it’s all about organizing and processing data in the background and sending it to the front end, This process can vary in protocol (HTTP (S), WebSocket), method (RCP, Ajax, MQ), content returned in different formats (JSON, XML, HTML (templates), early Flash, etc.); What WE just talked about is the communication between the front and the back. In fact, the communication between logical modules, between processes and even between subsequent containers is involved. This article first introduces the core of Web communication, front and background communication.

Template technology is separated from the front and back ends

  • Template technology: Widely adopted Web technology in the first decade of this century, it is better known as the MVC pattern. The idea is to write data in an HTML template using back-end code, and the template engine will return the rendered HTML. Django has this technique built into it, and other Python frameworks rely on separate templates such as Jinjia,Mako, etc. JSPS in other languages, such as Java, also use this pattern. His characteristics are direct operation, directly in the need to write the corresponding data. Also can directly use back-end language to write logic in the page, the development speed is fast. However, the disadvantages are also obvious, the front and back ends are seriously coupled, difficult to maintain, and not suitable for large projects.

    • Protocol: HTTP
    • Methods: Both can be used
    • Content: the HTML (templates)

  • Front and back end separation: As projects get bigger and bigger, the need for front-end engineering gives rise to WebPack tools. Later, the Vue,React, and Angular frameworks focused on the MVVC model, which only took data, rendering, and business logic from the back end into the front end framework. This allows maximum separation between the front and back end developers.

    • Agreement: both are acceptable
    • Methods: Both can be used
    • Content: the json/XML

Mako template and his friend Fastapi-Mako

Frodo uses a template as the front desk for the blog, considering that this section has fewer pages, simple logic, and is easy to maintain on the back end. There is no outdated technology, only inappropriate technology.

Mako is one of the main python templates. Its native interface can be used directly, but some repetitive logic needs to be wrapped up:

  • Several context variables are fixed in the template
    • Request object (the request object used by the backend framework, inFlask.Django.fastapiThe template needs to use some of its methods and properties, such as reverse addressingrequest.url_for().request.hostAnd evenrequest.SessionThe contents of the
    • Request context (body) : Formdata, QueryParam, PathParam, these may be used in templates.
    • Return context (without encapsulating ye Tao supply)
  • Template files are addressed automatically
  • Static file addressing
  • Template Exception Handling

Like Flask, FastAPI routing is functional, and the direct way to encapsulate the above template functionality into routing functions is through Python’s decorators. Finally, the following results are achieved:

from fastapi import APIRouter, Request, HTTPException
from fastapi.responses import HTMLResponse
from models import cache, Post
from ext import mako

@router.get('/archives', name='archives', response_class=HTMLResponse)
Template ('archives.html') # specify the template file name
@cache(MC_KEY_ARCHIVES)
async def archives(request: Request): Request needs to be passed explicitly
    post_data = await Post.async_filter(status=Post.STATUS_ONLINE)
    post_obj = [Post(**p) for p in post_data]
    post_obj = sorted(post_obj, key=lambda p: p.created_at, reverse=True)
    rv = dict()
    for year, items in groupby(post_obj, lambda x: x.created_at.year):
        if year in rv: rv[year].extend(list(items))
        else: rv[year] = list(items)
    archives = sorted(rv.items(), key=lambda x: x[0], reverse=True)
    Only context is returned
    return {'archives': archives}
Copy the code

It’s easy to understand, but the only thing that needs to be explained is why the request is passed explicitly. Fastapi mostly avoids passing requests, which is exactly the same as Flask’s idea, using a Local thread stack to distinguish the context of different requests. But often reverse addressing is required in templates, like this:

% for year, posts in archives:
  <h3 class="archive-year-wrap">
      <a href="${ request.url_for('archives', year=year) }" class="archive-year">
      ${ year }
    </a>
  </h3>
% endfor
Copy the code

@Mako Simple decorator complete at LouisYZK/ FastAPi-Mako, interested friends can have a look.

In addition, there is the @cached decorator, which caches the return template of the function. If the current page data has not changed, the next access will fetch the data directly from Redis. The detailed logic will be explained in the CRUD logic below.

Communication logic for CRUD

This section is about all data models. Some of them, such as Posts and Activity, have multiple data storage modes, and they need more trick. All data operations follow the following process:

The DataModel in the control use case is a data class designed in a data piece that has several methods to address CRUD requirements, of which two are the most important:

  • Generate the SQL for the operation
  • Generate the key used by KV database cache

Both of these uses a little SQLalchemy and a little trickery from python decorators. You can focus on the source code models/base.py and models/ Mc.py.

It is worth mentioning the implementation of the update delete cache:

Clear_mc method in ## Mc.py
async def clear_mc(*keys):
    redis = await get_redis()
    print(f'Clear cached: {keys}')
    assert redis is not None
    await asyncio.gather(*[redis.delete(k) for k in keys],
                        return_exceptions=True)

The __flush__ method in the ## base class
from models.mc import clear_mc
@classmethod
async def __flush__(cls, target):
    await asyncio.gather(
        clear_mc(MC_KEY_ITEM_BY_ID % (target.__class__.__name__, target.id)),
        target.clear_mc(), return_exceptions=True
    )

## target is a concrete data instance that overwrites the clear_mc() method to remove a different key, such as the following override of the Post class:
async def clear_mc(self):
    keys = [
        MC_KEY_FEED, MC_KEY_SITEMAP, MC_KEY_SEARCH, MC_KEY_ARCHIVES,
        MC_KEY_TAGS, MC_KEY_RELATED % (self.id, 4),
        MC_KEY_POST_BY_SLUG % self.slug,
        MC_KEY_ARCHIVE % self.created_at.year
    ]
    for i in [True.False]:
        keys.append(MC_KEY_ALL_POSTS % i)
    for tag in await self.tags:
        keys.append(MC_KEY_TAG % tag.id)
    await clear_mc(*keys)
Copy the code

This ensures that each creation, update, and deletion of data removes the associated cache and maintains data consistency. As you may have noticed, operations to remove the cache are awaitable, which means asynchrony can take advantage of concurrency here. So we see the use of asyncio. Gather (*coros), which can remove multiple keys concurrently, because Redis creates a connection pool that does not use multithreading, which is how Asyncio achieves I/O concurrency. (This should have been covered in asynchrony, but it’s important).

certification

The requirements for certification come from two sources:

  • The background of content management system can only be operated by the blog owner, such as blog publishing, password modification, etc.

  • Visitor comments require authentication.

Administrator authentication – using JWT

JWT is one of the widely used authentication methods at present, and its advantages over cookies can be referred to relevant articles. Fastapi has built-in support for JWT, which makes it easy to use for validation.

Before we talk about the implementation, we should first think about its communication logic:

The above process represents the logic of LOGIN and the general logic of accessing the required authentication API. Do you see the problem? Where are tokens stored?

Where do tokens exist? The server generates a Token for the client to receive, and the next request takes it with it. This frequently used and small volume of data is best stored directly in memory. Global variables must be shared in programming languages, such as multiprocess.Value. Contextvar is specifically designed to address asynchronous variable sharing issues, requiring Python greater than 3.7

Fastapi maintains this Token for us by simply defining it as follows:

from fastapi.security import OAuth2PasswordBearer
oauth2_scheme = OAuth2PasswordBearer(tokenUrl='/auth')
Copy the code

The Token generation path is /auth, and the oauth2_scheme is used as the globally dependent Token source. Whenever the interface needs to use the Token, only:

@router.get('/users')
async def list_users(token: str = Depends(oauth2_scheme)) -> schemas.CommonResponse:
    users: list = await User.async_all()
    users = [schemas.User(**u) for u in users]
    return {'items': users, 'total': len(users)}
Copy the code

Depends is a feature of FastAPI. It is written directly in the parameters of the interface function and performs some logic before the request, similar to middleware. The logic here is to check if the request header has an Auth: Bear+Token, otherwise the request cannot be issued.

The Token generation logic is done in the login interface, which is almost the most complex logic in Frodo:

@app.post('/auth')
async def login(req: Request, username: str=Form(...)., password: str=Form(...).):
    user_auth: schemas.User = \
            await user.authenticate_user(username, password)
    if not user_auth:
        raise HTTPException(status_code=400, 
                            detail='Incorrect User Auth.')
    access_token_expires = timedelta(
        minutes=int(config.ACCESS_TOKEN_EXPIRE_MINUTES)
    )
    access_token = await user.create_access_token(
                        data={'sub': user_auth.name},
                        expires_delta=access_token_expires)
    return {
        'access_token': access_token,
        'refresh_token': access_token,
        'token_type': 'bearer'
    }
Copy the code

Basically follow the sequence diagram in this section.

Access authentication – Use Session

Github authentication is used for Frodo, so the logics are as follows:

The whole logic is very simple, follow the Github authentication logic, change the way such as wechat scan code will change a set. Notice the jump url. JWT is not used to store visitor information at the same time, because there is no restriction on obsolescence and so on. Session cookies are the most direct.

@router.get('/oauth')
async def oauth(request: Request):
    if 'error' in str(request.url):
        raise HTTPException(status_code=400)
    client = GithubClient()
    rv = await client.get_access_token(code=request.query_params.get('code'))
    token = rv.get('access_token'.' ')
    try:
        user_info = await client.user_info(token)
    except:
        return RedirectResponse(config.OAUTH_REDIRECT_PATH)
    rv = await create_github_user(user_info)
    ## Use session storage
    request.session['github_user'] = rv
    return RedirectResponse(request.session.get('post_url'))
Copy the code

Note that fastAPI requires middleware to enable sessions

from starlette.middleware.sessions import SessionMiddleware

app.add_middleware(SessionMiddleware, secret_key='YOUR KEY')
Copy the code

What is a starlette? Isn’t it FastAPI? Simply speaking, the relationship between Starlette and FastAPI is the same as the relationship between Werkzurg and Flask. The difference between WSGI and ASGI is that ASGI wants to go beyond WSGI. Of course, it also needs to develop a set of basic standards and tool libraries.

Fine, the communication logic is basically there, Frodo uses very few communication models. The next “asynchronous article” has to do with both communication and data, which is where asynchronous blogging differs from blogs implemented in python in general.