Djangos source-1 startup process analysis

Django is a classic Python Web development framework and one of the most popular Python open source projects. Unlike the Flask framework, Django is highly integrated and helps developers build Web projects quickly. Starting from this week, we will get into the source code analysis of Djaong project, deepen our understanding of Django, and master Python Web development. Djangos source code is 4.0.0. We’ll start with a brief overview of how a Django project is created and started.

• A quick review of Django
• Overview of the Django Project
• Django command scaffolding
• The startproject and startApp commands are implemented
• The runserver command is implemented
• The setup process for your app
• summary
• tip

A quick review of Django

Set up your virtual environment and install Django version 4.0.0. This version is consistent with the latest official documentation and has a complete Chinese translation. You can follow the Quick Setup guide to create a Django project and get a taste of its magic. First, use the startProject command to create a project called Hello. Django sets up a basic project structure locally. Once in the Hello project, you can start the project directly using the runServer command.

python3 -m django startproject hello
cd hello  && python3 manage.py runserver
Copy the code

Django provides good modularity support for large Web projects. Modules under project are called apps, and a project can contain multiple APP modules, similar to flask’s blue-print. We continue to use the startApp command to create an app called API.

python3 -m django startapp api
Copy the code

For the views.py of apI-app, we need to improve it by adding the following:

from django.shortcuts import render

# Create your views here.

from django.http import HttpResponse


def index(request):
    return HttpResponse("Hello, Game404. You're at the index.")
Copy the code

Create a module file called urls.py, define the mapping between the URL and view, and fill it in:

from django.urls import path

from . import views

urlpatterns = [
    path('', views.index, name='index'),
]
Copy the code

After completing the implementation of apI-app, we need to add a custom API module to the project. This requires two steps. First, configure the api-app in setting.py of hello-project:

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'api',
]
Copy the code

Import the URL and view defined in api-app in the hello-project urls.py:

from django.contrib import admin
from django.urls import path, include

urlpatterns = [
    path('admin/', admin.site.urls),
    path('api/', include('api.urls')),
]
Copy the code

The rest can be implemented using the template’s standard implementation. Then we start the project again and access the following path:

➜ ~ Hello, curl http://127.0.0.1:8000/api/ Game404. You 'r e at the index. %Copy the code

Now that we’ve basically created and started a simple Django project, let’s take a look at how the process works.

Overview of the Django Project

The django project source code probably includes the following packages:

package	Functional description
apps	Django’s app manager
conf	Configuration information, including project template and APP template, etc
contrib	Django provides a standard app by default
core	Django core Features
db	Database model implementation
dispatch	Signal for module decoupling
forms	Form implementation
http	HTTP protocol and service-related implementation
middleware	Standard middleware provided by Django
template && templatetags	The template function
test	Unit testing support
urls	Some URL handling classes
utils	Utility class
views	View-specific implementation

I’ve also done a quick comparison of Django and Flask code:

-------------------------------------------------------------------------------
Project                      files          blank        comment           code
-------------------------------------------------------------------------------
django                         716          19001          25416          87825
flask                           20           1611           3158           3587
Copy the code

As you can see from the number of files and lines of code, Django is a massive framework, unlike Flask, which is a simple framework with over 700 module files and nearly 90,000 lines of code. Before reading flask’s source code, we need to learn about sqlAlchemy, Werkzeug, and pyProject.toml dependencies. Django doesn’t have any other dependencies by default. Flask is a bit like ios and Android, but requires a variety of plugins and is much more open; Django is fully integrated, and uses the default framework to handle most open Web requirements.

Django command scaffolding

Django provides a series of scaffolding commands to assist developers in creating and managing Django projects. The list of commands can be viewed using the help parameter:

python3 -m django --help

Type 'python -m django help <subcommand>' for help on a specific subcommand.

Available subcommands:

[django]
    check
    compilemessages
    createcachetable
    dbshell
    diffsettings
    dumpdata
    flush
    inspectdb
    loaddata
    makemessages
    makemigrations
    migrate
    runserver
    sendtestemail
    shell
    showmigrations
    sqlflush
    sqlmigrate
    sqlsequencereset
    squashmigrations
    startapp
    startproject
    test
    testserver
Copy the code

The startproject, startApp and runserver commands are the focus of this article, and the other 20 commands will be introduced when they are used. This is the essence of general reading, focusing only on the backbone, first establishing a global vision, and then gradually deepening.

The main function of a Django module is provided in __main__.py:

"""
Invokes django-admin when the django module is run as a script.

Example: python -m django check
"""
from django.core import management

if __name__ == "__main__":
    management.execute_from_command_line()
Copy the code

As you can see, the Django.core. management module provides scaffolding functionality. Also in project manager.py, the project is started by calling the management module:

#! /usr/bin/env python """Django's command-line utility for administrative tasks.""" import os import sys def main(): """Run administrative tasks.""" os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hello.settings') try: from django.core.management import execute_from_command_line except ImportError as exc: ... execute_from_command_line(sys.argv) if __name__ == '__main__': main()Copy the code

ManagementUtility’s main structure is as follows:

class ManagementUtility: """ Encapsulate the logic of the django-admin and manage.py utilities. """ def __init__(self, argv=None): Argv = argv or sys.argv[:] self.prog_name = os.path.basename(self.argv[0]) if self.prog_name == '__main__.py': self.prog_name = 'python -m django' self.settings_exception = None def main_help_text(self, commands_only=False): """Return the script's main help text, as a string.""" pass def fetch_command(self, subcommand): """ Try to fetch the given subcommand, printing a message with the appropriate command called from the command line (usually "django-admin" or "manage.py") if it can't be found. """ pass ... def execute(self): """ Given the command-line arguments, figure out which subcommand is being run, create a parser appropriate to that command, and run it. """ passCopy the code

• The init function parses the command-line arguments to argv
• Main_help_text Command help information
• Fetch_command finds a django module subcommand
• Execute Executes the subcommand

A good command line tool requires clear help output. The default help message is to call the main_help_text function:

def main_help_text(self, commands_only=False):
    """Return the script's main help text, as a string."""

    usage = [
        "",
        "Type '%s help <subcommand>' for help on a specific subcommand." % self.prog_name,
        "",
        "Available subcommands:",
    ]
    commands_dict = defaultdict(lambda: [])
    for name, app in get_commands().items():
        commands_dict[app].append(name)
    style = color_style()
    for app in sorted(commands_dict):
        usage.append("")
        usage.append(style.NOTICE("[%s]" % app))
        for name in sorted(commands_dict[app]):
            usage.append("    %s" % name)
    # Output an extra note if settings are not properly configured
    if self.settings_exception is not None:
        usage.append(style.NOTICE(
            "Note that only Django core commands are listed "
            "as settings are not properly configured (error: %s)."
            % self.settings_exception))

    return '\n'.join(usage)
Copy the code

Main_help_text makes use of the following four functions to find command lists:

def find_commands(management_dir):
    """
    Given a path to a management directory, return a list of all the command
    names that are available.
    """
    command_dir = os.path.join(management_dir, 'commands')
    return [name for _, name, is_pkg in pkgutil.iter_modules([command_dir])
            if not is_pkg and not name.startswith('_')]

def load_command_class(app_name, name):
    """
    Given a command name and an application name, return the Command
    class instance. Allow all errors raised by the import process
    (ImportError, AttributeError) to propagate.
    """
    module = import_module('%s.management.commands.%s' % (app_name, name))
    return module.Command()

def get_commands():
    commands = {name: 'django.core' for name in find_commands(__path__[0])}

    if not settings.configured:
        return commands

    for app_config in reversed(list(apps.get_app_configs())):
        path = os.path.join(app_config.path, 'management')
        commands.update({name: app_config.name for name in find_commands(path)})

    return commands

def call_command(command_name, *args, **options):
    pass
Copy the code

Find_commands finds commands under management/commands. Load_command_class imports commands using import_module, a method that dynamically loads modules.

Subcommands are executed by finding the subcommand through fetch_command and then executing the run_from_argv method of the subcommand.

def execute(self):
    ...
    self.fetch_command(subcommand).run_from_argv(self.argv)

def fetch_command(self, subcommand):
    """
    Try to fetch the given subcommand, printing a message with the
    appropriate command called from the command line (usually
    "django-admin" or "manage.py") if it can't be found.
    """
    # Get commands outside of try block to prevent swallowing exceptions
    commands = get_commands()
    try:
        app_name = commands[subcommand]
    except KeyError:
        ...
    if isinstance(app_name, BaseCommand):
        # If the command is already loaded, use it directly.
        klass = app_name
    else:
        klass = load_command_class(app_name, subcommand)
    return klass
Copy the code

Django’s core includes the following commands, most of which are directly derived from BaseCommand:

BaseCommand’s main code structure:

The most important run_from_argv and execute methods are the entry points to the command:

def run_from_argv(self, argv):
    ...
    parser = self.create_parser(argv[0], argv[1])

    options = parser.parse_args(argv[2:])
    cmd_options = vars(options)
    # Move positional args out of options to mimic legacy optparse
    args = cmd_options.pop('args', ())
    handle_default_options(options)
    try:
        self.execute(*args, **cmd_options)
    except CommandError as e:
        ...

def execute(self, *args, **options):
    output = self.handle(*args, **options)
    return output
Copy the code

The handler method is left to subclasses to override the implementation:

def handle(self, *args, **options): """ The actual logic of the command. Subclasses must implement this method. """ raise NotImplementedError('subclasses of  BaseCommand must provide a handle() method')Copy the code

The startProject && startapp command is implemented

The startProject and startApp commands, which create the project and app respectively, are derived from TemplateCommand. The main functionality is implemented in TemplateCommand.

# startproject
def handle(self, **options):
    project_name = options.pop('name')
    target = options.pop('directory')

    # Create a random SECRET_KEY to put it in the main settings.
    options['secret_key'] = SECRET_KEY_INSECURE_PREFIX + get_random_secret_key()

    super().handle('project', project_name, target, **options)

# startapp
def handle(self, **options):
    app_name = options.pop('name')
    target = options.pop('directory')
    super().handle('app', app_name, target, **options)
Copy the code

The structure of the startProject command looks something like this:

├ ─ ─ the hello │ ├ ─ ─ just set py │ ├ ─ ─ asgi. Py │ ├ ─ ─ Settings. Py │ ├ ─ ─ urls. Py │ └ ─ ─ wsgi. Py └ ─ ─ the manage. PyCopy the code

This is the same as the template file in the project_template directory under conf:

. ├ ─ ─ the manage. Py - TPL └ ─ ─ project_name ├ ─ ─ just set py - TPL ├ ─ ─ asgi. Py - TPL ├ ─ ─ Settings. The py - TPL ├ ─ ─ urls. Py - TPL └ ─ ─ wsgi.py-tplCopy the code

Manage.py – TPL template file contents:

#! /usr/bin/env python """Django's command-line utility for administrative tasks.""" import os import sys def main(): """Run administrative tasks.""" os.environ.setdefault('DJANGO_SETTINGS_MODULE', '{{ project_name }}.settings') try: from django.core.management import execute_from_command_line except ImportError as exc: .... if __name__ == '__main__': main()Copy the code

As you can see, the function of the startProject command is to take the project_name entered by the developer, render it into a template file, and regenerate it into a project file.

The main functions handled by TemplateCommand are as follows:

from django.template import Context, Engine

def handle(self, app_or_project, name, target=None, **options):
    ...
    base_name = '%s_name' % app_or_project
    base_subdir = '%s_template' % app_or_project
    base_directory = '%s_directory' % app_or_project
    camel_case_name = 'camel_case_%s_name' % app_or_project
    camel_case_value = ''.join(x for x in name.title() if x != '_')
    ...
    context = Context({
        **options,
        base_name: name,
        base_directory: top_dir,
        camel_case_name: camel_case_value,
        'docs_version': get_docs_version(),
        'django_version': django.__version__,
    }, autoescape=False)   
    ...
    template_dir = self.handle_template(options['template'],
                                            base_subdir)
    ...   
    for root, dirs, files in os.walk(template_dir):
        for filename in files:
            if new_path.endswith(extensions) or filename in extra_files:
                with open(old_path, encoding='utf-8') as template_file:
                    content = template_file.read()
                    template = Engine().from_string(content)
                    content = template.render(context)
                    with open(new_path, 'w', encoding='utf-8') as new_file:
                        new_file.write(content)
Copy the code

• Build the Context Context used by the template parameters
• Traverse the template file directory
• Use Djangos template Engine to render the content
• Generate project scaffolding files with rendered results

How django.template.Engine works and how it differs from Mako will be explained later in this chapter.

The runserver command is implemented

Runserver provides an HTTP service for developing tests, helps launch Django projects, and is Django’s most frequently used command. Django projects follow the WSGi specification and need to look for WSGi-Application before execution starts. (If you are not familiar with the WSGI specification, please check out the previous article.)

def get_internal_wsgi_application():
    """
    Load and return the WSGI application as configured by the user in
    ``settings.WSGI_APPLICATION``. With the default ``startproject`` layout,
    this will be the ``application`` object in ``projectname/wsgi.py``.

    This function, and the ``WSGI_APPLICATION`` setting itself, are only useful
    for Django's internal server (runserver); external WSGI servers should just
    be configured to point to the correct application object directly.

    If settings.WSGI_APPLICATION is not set (is ``None``), return
    whatever ``django.core.wsgi.get_wsgi_application`` returns.
    """
    from django.conf import settings
    app_path = getattr(settings, 'WSGI_APPLICATION')
    if app_path is None:
        return get_wsgi_application()

    try:
        return import_string(app_path)
    except ImportError as err:
        ...
Copy the code

This will load the application defined by the developer in the project Settings:

WSGI_APPLICATION = 'hello.wsgi.application'
Copy the code

By default, the custom WSGi-application looks like this:

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'hello.settings')

application = get_wsgi_application()
Copy the code

Continue with the execution of runserver, mainly with the inner_run function:

from django.core.servers.basehttp import run
    
def inner_run(self, *args, **options):
    try:
        handler = self.get_handler(*args, **options)
        run(self.addr, int(self.port), handler,
            ipv6=self.use_ipv6, threading=threading, server_cls=self.server_cls)
    except OSError as e:
        ...
        # Need to use an OS exit because sys.exit doesn't work in a thread
        os._exit(1)
    except KeyboardInterrupt:
        ..
        sys.exit(0)
Copy the code

The realization of the function of the HTTP service by django. Core. The servers. Basehttp, completing the HTTP service and wsgi the cohesion between the architecture diagram is as follows:

The code for the run function:

def run(addr, port, wsgi_handler, ipv6=False, threading=False, server_cls=WSGIServer): server_address = (addr, port) if threading: httpd_cls = type('WSGIServer', (socketserver.ThreadingMixIn, server_cls), {}) else: httpd_cls = server_cls httpd = httpd_cls(server_address, WSGIRequestHandler, ipv6=ipv6) if threading: # ThreadingMixIn.daemon_threads indicates how threads will behave on an # abrupt shutdown; like quitting the server by the user or restarting # by the auto-reloader. True means the server will not wait for thread # termination before it quits. This will make auto-reloader faster # and will prevent the need to kill the server  manually if a thread # isn't terminating correctly. httpd.daemon_threads = True httpd.set_app(wsgi_handler) httpd.serve_forever()Copy the code

• If multithreading is supported, create a new class for WSGIServer and ThreadingMixIn
• Create the HTTP service and start it

Djangos wsGi-Application implementation and HTTP protocol implementation will be covered in the next chapter, which we’ll skip for now. There is another very important feature in RunServer: automatic restart service. If we change the project code, the service will restart automatically, which can improve development efficiency.

For example, if we modify the VIEW function of the API and add a few random characters, we can see something like the following output on the console:

# python manage.py runserver Watching for file changes with StatReloader Performing system checks... System Check identified no issues (0 silenced). March 05, 2022-09:06:07 Django Version 4.0, Using Settings 'hello. Settings' Starting development server at http://127.0.0.1:8000/ Quit the server with control-c. /Users/yoo/tmp/django/hello/api/views.py changed, reloading. Watching for file changes with StatReloader Performing system checks... System Check identified no issues (0 silenced). March 05, 2022-09:12:59 Django Version 4.0, Using Settings 'hello. Settings' Starting development server at http://127.0.0.1:8000/ Quit the server with control-c.Copy the code

The runserver command detects that /hello/ API /views.py has been modified and automatically restarts the service using StatReloader.

Inner_run has one of the following startup modes, which by default uses autoreload:

def run(self, **options):
    """Run the server, using the autoreloader if needed."""
    ...
    use_reloader = options['use_reloader']

    if use_reloader:
        autoreload.run_with_reloader(self.inner_run, **options)
    else:
        self.inner_run(None, **options)
Copy the code

The autoreload inheritance is as follows:

class BaseReloader:
    pass
    
class StatReloader(BaseReloader):
    pass

class WatchmanReloader(BaseReloader):
    pass

def get_reloader():
    """Return the most suitable reloader for this environment."""
    try:
        WatchmanReloader.check_availability()
    except WatchmanUnavailable:
        return StatReloader()
    return WatchmanReloader()
Copy the code

The current version preferentially uses the WatchmanReloader implementation, which relies on the pywatchman library and requires additional installation. Otherwise use the StatReloader implementation, which was introduced earlier in werkzeug, which essentially continuously listens for file state changes.

class StatReloader(BaseReloader):
    SLEEP_TIME = 1  # Check for changes once per second.

    def tick(self):
        mtimes = {}
        while True:
            for filepath, mtime in self.snapshot_files():
                old_time = mtimes.get(filepath)
                mtimes[filepath] = mtime
                if old_time is None:
                    logger.debug('File %s first seen with mtime %s', filepath, mtime)
                    continue
                elif mtime > old_time:
                    logger.debug('File %s previous mtime: %s, current mtime: %s', filepath, old_time, mtime)
                    self.notify_file_changed(filepath)

            time.sleep(self.SLEEP_TIME)
            yield
Copy the code

• StatReloader checks file timestamp changes at 1s intervals.
• Tick is a generator mode that can be invoked continuously using Next

When the file changes, exit the current process and start a new process using subprocess:

def trigger_reload(filename): logger.info('%s changed, reloading.', filename) sys.exit(3) def restart_with_reloader(): new_environ = {**os.environ, DJANGO_AUTORELOAD_ENV: 'true'} args = get_child_arguments() while True: p = subprocess.run(args, env=new_environ, close_fds=False) if p.returncode ! = 3: return p.returncodeCopy the code

The setup process for your app

Another important step before executing the runServer command is setup: loading and initializing the developer’s custom app content. This is started in ManagementUtility’s execute function:

def execute(self):
    ...
    try:
        settings.INSTALLED_APPS
    except ImproperlyConfigured as exc:
        self.settings_exception = exc
    except ImportError as exc:
        self.settings_exception = exc
    ...
Copy the code

Settings contains the following modules, mainly INSTALLED_APPS:

class Settings:
    def __init__(self, settings_module):
        ...
        # store the settings module in case someone later cares
        self.SETTINGS_MODULE = settings_module

        mod = importlib.import_module(self.SETTINGS_MODULE)

        tuple_settings = (
            'ALLOWED_HOSTS',
            "INSTALLED_APPS",
            "TEMPLATE_DIRS",
            "LOCALE_PATHS",
        )
        self._explicit_settings = set()
        ...
Copy the code

INSTALLED_APPS is defined in the project setting:

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'api',
]
Copy the code

This allows the Django framework to load developer-defined content dynamically.

summary

Django is a highly integrated Python Web development framework that supports modular development. Django also provides a number of scaffolding commands, such as startProject and StartApp, to help create project and module templates; Use RunServer to assist with testing and development projects. The Django project also complies with the WSGI specification, creating WSGIServer in the startup of its HTTP service and supporting multithreaded mode. Django is a framework that dynamically loads a developer’s business implementation using the convention’s setting configuration file.

tip

Django commands support intelligent prompts. For example, in the runserver command, we accidentally typed the letter u into an I. The command will automatically remind us if we want to use the runserver command:

python -m django rinserver
No Django settings specified.
Unknown command: 'rinserver'. Did you mean runserver?
Type 'python -m django help' for usage.
Copy the code

The smart prompt feature is useful for command-line tools. The common implementation is to compare user input to known commands to find the closest command. This is a practical application of the string edit distance algorithm. In my opinion, it is more useful to understand the scene than the deadbrush algorithm, and I occasionally use this example in interviews to gauge the level of the algorithm of the interviewer. Djangos use difflib, the Python standard library, directly:

from difflib import get_close_matches possible_matches = get_close_matches(subcommand, commands) sys.stderr.write('Unknown command: %r' % subcommand) if possible_matches: sys.stderr.write('. Did you mean %s? ' % possible_matches[0])Copy the code

An example of using get_close_matches:

>>> get_close_matches("appel", ["ape", "apple", "peach", "puppy"])
['apple', 'ape']
Copy the code

Students who are interested in the algorithm can learn more about its implementation details by themselves.

Another trick is the naming of an alienated detail. Class is usually a keyword in many development languages. If we define a variable name of class type, avoid keyword conflicts. One way is to use Klass instead:

# 
if isinstance(app_name, BaseCommand):
    # If the command is already loaded, use it directly.
    klass = app_name
else:
    klass = load_command_class(app_name, subcommand)
Copy the code

Another option is to use Clazz instead:

if ( classes.length ) { while ( ( elem = this[ i++ ] ) ) { curValue = getClass( elem ); . if ( cur ) { j = 0; while ( ( clazz = classes[ j++ ] ) ) { // Remove *all* instances while ( cur.indexOf( " " + clazz + " " ) > -1 ) { cur =  cur.replace( " " + clazz + " ", " " ); }}... }}}Copy the code

Which one do people usually use?

Refer to the link

• Docs.djangoproject.com/zh-hans/4.0…
• Docs.python.org/zh-cn/3/lib…
• Leetcode-cn.com/problems/ed…