This article has participated in the activity of “New person creation Ceremony”, and started the road of digging gold creation together.

Structure determines nature and creates beauty. — Commemorating a cooperative development

The scattered logic is combined into beautiful code in a delicate architecture, just as water molecules form a stable and beautiful ice crystal by hydrogen bonding, small and grand.

preface

Here’s a famous quote from chemistry class that I secretly think is wonderful:

Die Entropie der Welt strebt einem Maximum zu. The world strives for maximum entropy. (1865, Rudolph J. E. Clausius)

This is the law of the world, just as the room always gets messy and the code always gets annoying.

In terms of student affairs, the idea has been around for years. When there was a bug (31/30) in the school’s extended course, I really wanted to make another one. Later, due to the huge traffic explosion of sports course selection, I was more determined to make this idea, but I always felt that I had no ability and energy to realize it, because the application was huge and the demand was complicated.

Later, with the help of team members, I felt relaxed and had the strength of a war, so I fought bravely.

It was a team effort and achievement to finish such a somewhat complicated project in an organized way. I would like to express my special thanks to the school for its support, especially for providing students with such a valuable opportunity. I would also like to express my thanks to all the leaders who participated in the development of the project and contributed to the project:

  • CmdBlock
  • Phantomlsh
  • Tina
  • Queenie

By the way, this article is written by Phantomlsh.

Initial goal

The story begins in a tiny, tiny office filled with smoke, fans, flashing lights and crackling keyboards as two people whisper conspirates…….

Hold on, hold on! It’s just two students talking about a little app that lets students upload their own photos to make a campus card.

In the past, the school always organized students to queue up to take photos, which was not only very troublesome, but also the same profile picture taken by everyone. Therefore, the information group hoped to have a web page for students to upload photos by themselves. No problem! Little things! The next day I went face to face with CmdBlock and quickly built a simple application: Golang + Mongodb on the back end, Vuejs on the front end plus a bunch of weird plug-ins (in fact, I also took some code from previous projects like login). Once logged in, users can select photos, crop them, and upload them. The server saves the photo file according to the user, by controlling the size of the front clipping box to make the last saved photo size uniform.

This may be the prototype of student affairs system, everything is very simple, also did not consider what security and structure, after all, just to achieve a small function! Did not expect to have not been put into use, found that also need to do management end. In other words, teachers can log in to view and approve photos of all the students in a class. In fact, it is not a big problem, after a disorderly knocking added a management page for the teacher to view the photo.

CmdBlock and I had a lot of fun with the app at the time (and it turned out to be a good photo collection for the new class). But no one would have guessed that a gadget created to collect photos would eventually lead to a huge project.

Technical details:

  • Back end: Gin framework builds HTTP server to provide Restful API service.
  • Login: learn from the experience of the previous authentication system, take two authentication to handle the user login request (the first request to send the user name, the server returns a random string; Second request to send user password hash summary of password and random string) to avoid plaintext transmission of user password. In two-step authentication, the back end requires a three-second interval to prevent brute force cracking.
  • Front-end page: because there is no CDN (cry), considering the server traffic problem when the user loads the page, front-end page does not use single page application, while trying to use js library on the public CDN.
  • Form verification: check the legitimacy of the ID number in the front end to prevent users from sending login requests in vain because of incorrect ID number, and further reduce server load.

A light on the horizon

The story of the last section is still about summer vacation. At that time, CmdBlock and I played and studied in the small office every day. So in fact, the photo-uploading app is getting more complicated before it even has a name.

The idea was simple: if you have a photo uploading service, you might as well do other services that students use. CmdBlock and I put our heads together and decided that updating the school’s course selection system was a top priority. Specifically, there are three major problems in school course selection:

  1. The limit is easy to fail. 31 people signed up for a class of 30 places.
  2. The server will either explode and restart several times, or it will be very, very slow.
  3. The page is ugly and not very mobile.

Aside from the third point, it’s mostly Tina and Queenie. CMB Block and I are not very good at front-end web design and all that stuff. In terms of the first two points, it is actually a contradiction that is difficult to solve: the essence of the lock is to limit concurrency, so that the database read and write single thread, so that students’ requests “queue” processing, so as to solve the problem of two requests crowded in at the same time; However, if you make database reads and writes single-threaded, concurrency is worse. (This problem was actually caused by the fact that we were too unfamiliar with Redis at that time.)

CmdBlock and I gave it a lot of thought and finally made a bold decision. We read the remaining number of students in each course into memory as a global variable before the start of class selection, so that we can process the number of students in the process queue of the program. For persistence, synchronize with Mongodb periodically. The problem is so serious that once the server crashes, the course selection process has to be started all over again because the remaining number of people in memory is lost.

Tube he! Just use it anyway! We’re just glad the service didn’t blow up on class selection day. I secretly thought it would be impossible to blow up an advanced coroutine concurrency like Golang, but I always knew.

Again, I would like to thank the dean’s office and the information team for actually approving the new course selection system for the new class of students. CmdBlock and I were in the office nervously staring at the server monitoring on the big screen and the running status of the server on the laptop. We were probably the only ones who knew how fragile and chaotic the system was.

Unfortunately, there’s a bug before we even get started. The bug is in time: the front-end page uses JS to obtain the local time of the computer to do the front-end time limit. There was a time limit on the back end, but the clock on the server running the server was several minutes fast!! As a result, some students whose local time is also fast entered the course selection interface and finished the selection in advance before the time is up, so I would like to apologize to other students!

And then what? And then, of course, everything was fine! CmdBlock and I were very excited to see the logs being flushed and running smoothly on the server. Occasionally, one or two students came to the office for special processing (they forgot their ID number), and one of the Chrome browser versions in the computer room was too old to support async/await in JS, so they could not log in. However, in general, everything was perfect and achieved the ideal high concurrency effect! I just remember that there was no moment in the server at that time. Many students arrived late and found that the courses were all finished in an instant.

After we joined the class selection service, this little app was officially renamed “Student Affairs” and started to flourish.

Technical details:

  • Course selection: The selected course data is read into memory for quick processing and full useGolangHigh concurrency features.
  • Modularity: The different service controllers are written separately, which begins to reflect the modularity design idea.

Sail at the end

The course selection and photo upload tasks were successfully completed, but we also saw several problems:

  1. Course selection data cached in memory is unstable and load balancing cannot be used
  2. The code is starting to get a little messy, which is not conducive to adding more task types
  3. There is not enough flexibility for different course selection needs
  4. No management side

I’m not quite sure why, perhaps because after looking at continuous integration tools like Jenkins, CmdBlock and I came up with the idea that maybe we should make the student affairs system task-based. That is, administrators can post a task, set its template (for example, selecting courses), fill in the corresponding data (which courses are available, how many places are available), select students to participate (a certain grade), and then students complete their corresponding tasks.

As a student affairs system, such a design can meet the needs of complex affairs, such as sports and school-based course selection at the same time. However, such a design has great difficulty in implementation. How can we make different tasks and their different logic run under the same program framework? In particular, the back end is a strongly typed compiled language like Golang.

It was almost time for me to start college, so I just pitched the idea and left the rest to CmdBlock. He did come up with an extremely subtle design, which I asked him about while writing this article. Here’s an excerpt:

CmdBlock: (reference below)

The first thing that comes to mind is a mechanism to refresh the user’s tokens on every request (for added security). Tokens need to be stored somewhere, not directly using the back-end memory for possible load balancing. However, storing databases was slow and stupid, so a high-performance in-memory database, Redis, was found. During the trial it was found to be not only fast, but also satisfy the single thread operation.

The goal of a transactional system is to be universal, preferably in the form of a schedule, in the form of a timeline. The initial process for the task was designed as follows:

  1. Create and specify the required information (for example, course catalogs, etc.)
  2. Specify start and end times to accept user submissions within the valid time
  3. The administrator exports data and destroys the task

Later, considering the reuse of tasks (selecting the same course for the same group of people for many times, and not selecting the courses previously selected), I added the concept of opening and closing (a sudden idea of evening self-study) :

  1. Create a task
  2. Input information
  3. Specify the start time and start the task
  4. The start time is reached and user submissions are accepted
  5. Stop accepting user submissions when the end time is reached
  6. Export the results and close the task
  7. Go back to 2, or destroy

In this way, the data of the same task can be used (and modified) multiple times, and the task logic can read the user’s previous submission records.

Different tasks correspond to different program logic, the goal is modularization. Adding a file is a module, written as an anonymous function that runs different modules by routing incoming task types.

For course selection, the discovery of course selection data can be put in Redis, using Lua script to operate. Therefore, when the task is started, the data in Mongodb can be copied into Redis to achieve single-thread operation and high concurrency and support load balancing.

The above is the design of CmdBlock, have to sigh a: evening self-study is really too idle! CmdBlock is too strong!

At that time, I was obsessed with my studies and CmdBlock was back in school, so the whole project progressed very slowly. Fortunately, there is no urgent need, CmdBlock leisurely swim zai to write, leisurely swim zai to open a new voyage.

Technical details:

  • The cache: Cache data usageRedisSupports load balancing while ensuring high concurrency and single threading of operations.
  • User credentials: the user credentials will be refreshed every time the request is made, and passed through the requested header. The front-end page automatically saves the header to the sessionStorage of the web page by writing axios intercept, and reads and sets the header before sending.
  • Back-end verification: Middleware is used for verification to prevent the controller from repeatedly reading data from the database.
  • Task reuse: Multiple uses of tasks by managing their on and off status. When the task is closed, the status stored in the cache for the user to complete the task is cleared. When the task is enabled again, the user can complete the task again.
  • Server time: Automatically calibrates the server time using NTP.
  • The front page: Start to adopt a dual front-end design, the high concurrency front-end page used by students is loaded separately (as before), and the administrative side used by administrators is usedVuecliWrite a single page application.

Decisive metamorphosis

The semester under quarter system passed quickly, I returned to the familiar high school campus between Christmas and New Year’s Day. There were some new insights in the course of a semester, such as understanding and using Nodejs for the back end, but the most surprising was CmdBlock’s discovery of Redis. After some play, I looked at the code that CmdBlock had slowly accumulated, and a man (CmdBlock had gone to class) was looking grave in his familiar little office.

It’s not that the writing is bad, it’s that the writing is so good that only the creator can understand it. Many architectural designs make me feel redundant, such as task types, which can be obtained by database query, but need to be routed; The design of CmdBlock is to write a corresponding structure for each task in the code, and open a separate collection in Mongodb to store the task information, which I think is completely unnecessary. There were so many of these issues that I found CmdBlock’s class to discuss with him between classes, and we finally made the important decision: refactoring.

Love dearly? Of course! So much code thrown away! Fortunately, CmdBlock’s code snippets and functions are very easy to use and refreshing, and many things can be used in refactored applications, so we’ll be quick (maybe a day or two?) Most of the refactoring is done.

At the time, I felt that when developing the framework, the course selection task was too complex, and I should have chosen a primary task as a template. So I chose the notification function, the administrator issued the notification, students view the notification, and have read the receipt.

The most important part of the project is of course the modular design. The first is the task information, which can be read and processed directly using Golang’s Map[string]interface{} type. No special structure is required. Second, I think the type of interface Golang provides is designed to accomplish modularity. Modularity can be achieved by calling the interface methods implemented by the structure by simply mapping the task type to the corresponding module structure with a map. Specifically, the following functions:

  1. Open Start a task (copy task information to cache)
  2. RealTime Task real-time data (reading cached data)
  3. Response Response task (read/write cache and persistent storage)

A module is a task type, corresponding to a code file and the implementation code of three interface functions.

As for the distinction of task permissions, I also had an idea that I would use a tree-like structure to store different permission nodes. For example, the nodes of each class under the node of grade 2019 serve as child nodes. Each user must belong to a certain permission node, and each node stores a list of tasks authorized by that node. This idea comes from lazy tags in segment trees, although they are completely different. When querying the task list authorized by a user, the program starts from the permission node to which the user belongs, traces the permission tree to the root node, records the task list authorized by each node, and generates all the tasks that a user has permission to access. This makes it possible to quickly give a list of tasks without retrieving all tasks or all users. At the same time, the permission tree structure brings unprecedented flexibility to task permission allocation: the administrator can assign a task to all classes of class 2019 and class 18 of class 2018 without special processing.

User nodes (leaf nodes) are not stored in the permission tree to prevent the query of the permission tree from becoming too large. In order to achieve special authorization for a single user, temporary authorization is added to Redis token store that can exceed permission limits.

Management model is also really nervous, added a new role role field, super administrator to manage the permission tree and users, teacher role to manage users, students to complete tasks. Tasks are added to the super node by default, but the teacher role also needs to obtain the task list and management authorization according to the structure of the permission tree. In this way, the permission tree is managed hierarchically.

Due to the complexity of the program structure, I added a half layer between the traditional model layer and the controller layer, called Services, which encapsulates some complex logic pieces that can be used in multiple controllers.

The process for transferring task data to the backend is as follows:

  1. Request access, inGinFramework to middleware
  2. The middleware invokes the service layer for layer upon layer verification and stores the relevant database query results at the same timeGinFramework of thecontext.keys
  3. The request data arrives at the controller and the controller parses the request data
  4. The controller invokes the service layer and the data goes to the sorting function of the task module
  5. The sorting function readscontext.keysTo call the handler function in the correct task module
  6. The handler writes to the model and returns the result
  7. In the controller the processing result is packaged asjsonFormat, back to the front end

Is just like the water flow in complex pipelines, and, like the electron beam under the action of magnetic force of constraint on and off at this point, the student affairs system back-end structured scheme were designed, while maintaining the performance of the maximum retained the flexibility, has realized the multiple modules, a variety of authorization scheme, multiple tasks simultaneously, multiple back-end load balancing.

Data flows through a system to create value in the aesthetic of the structure.

Technical details:

  • Task informationUse:GolangIn theMap[string]interface{}Type read, write, and process directlyMongodbIn thejsonFormat task information.
  • Module interfaceUse:GolangInterface to achieve different module structure, elegant code.
  • Permission tree: The tree structure stores the user permission model. Each node records the task authorization information corresponding to the node. To keep the program running correctly, the backend is automatically checked to see if key permission nodes (such as the super administrator) are lost. If they are lost, they are created again.
  • User roles: Considering the instability of the permission tree based on individual collections, add a user role field, which the application uses to distinguish between super administrators, teachers, and students.
  • Request context: Use the request context to store related database query results and prevent multiple redundant queries.

Full concurrency!

By full concurrency, I don’t mean the high concurrency supported by the server side, but rather the fact that at this stage, the entire project has multiple lines of code running in parallel, facilitated by many team members. CmdBlock and I basically finished the back-end code over the New Year holiday, but there are many details and tweaks that CmdBlock will continue to handle for some time to come. On the front end, CmdBlock had already written an old version of the management side, so he still took the lead on the management side, and I helped with the student front end for a limited time.

The project was broken down into three projects:

  • The back-end
  • Students in the front
  • Management of the front

After collecting students’ opinions on the front end of the previous version, the new front end version returned to the light color system, while many details were further optimized. As for the login interface, I learned the two-step input user name and password scheme of Gooogle and Microsoft, which fit the two-step verification algorithm of back-end login. Here’s an interesting twist: because the back-end two-step authentication has to be three seconds apart (to prevent blowups), sometimes the front-end password is entered too fast in less than three seconds and will be denied login. When CmdBlock asked me how to solve it, I said: “No need to solve, input a relatively complicated password and animation time, it must be more than three seconds. If there really is a password that fast, either the password is simple, or a copy and paste password is not recommended.” So we ended up with a magical feature where if you entered your password too fast and were denied login, it would pop up a “security risk” message.

The reason for splitting the front end into two projects is still to reduce peak load. But because the management end does not have the login interface, the development debugging time is very troublesome. For this reason, I specially made a debug-Web project to proxy API requests to the real server, which is used to debug the student front end. Vuecli builds the management front end, so you can reverse proxy. You can first proxy to the real login interface, and then continue development and debugging after login.

The back-end project remains structurally unchanged, maintaining the design from the previous section.

There’s not much to be said for the student front end either, except that callback can be set up on the task page to support link access. You can access tasks through links, and then switch to login. After login, the task page is automatically redirected, skipping the loading task list and time limit for front-end tasks.

Managing the front end can be quite complex. In order to effectively manage complex data structures such as permission trees and various information of the task, we adopted the primitive method of more basic tabs and JSON editor to adapt to the flexible needs of the task. This part is mainly CmdBlock and Tina’s design, I am not very clear, but the function is very comprehensive, it is very comfortable to use, the administrator can see the task real-time report.

During this phase, Tina and Queenie also hopped back and forth between the two front-end projects, completing a lot of code and more important design work.

At this point, the framework of the student affairs system is complete. As of the end of this paper, it supports two task modules: notice and course selection. But I believe that as long as there are more requirements, a new module will emerge quickly and accomplish its task in an efficient and stable manner.

Technical details:

  • Login front: The user name and password are divided into two interfaces, which fit the two-step login authentication of the back end.
  • Student front-end callback: The user can access the task directly from the link, jumping to the login will automatically callback to the original task link.
  • Real-time reports on the management end: Obtained directlyRedisTo provide fast and effective real-time task reports.

Afterword.

Thank you for spending your precious time reading this nonsense!

Writing this article is also because this project dragged on for too long, the amount of code, I feel a sense of achievement is worth remembering. We all learned a lot from this project, not only in terms of technology, but also in terms of teamwork and design. We have seen the beauty of structuring from the gradual growth of student affairs system, experienced the architecture design and details processing, and believe that we have also brought an outstanding transaction system!

The ice crystals, formed by structuring water molecules, act like prisms, directing light beams in the right direction. Under the sun’s rays, a beautiful rainbow can also be reflected.