Summary: Industry-level application architecture is constantly evolving and iterating, but I have always felt that enterprise application architecture is being shaped in a way that looks scientific, but is not completely scientific.
Author: Jiao Fangfei
Day, watching the Chinese team’s 1:3 Vietnam team competition, in thinking about the deep cause of the Chinese football go from bad to worse, could not help but recall the years did some large enterprise digital transformation project, the trade-off, eventually return to the source of “how to design and implement a complex software engineering”, during the lunar New Year holiday, I wrote down some of my thoughts and learning essays on architecture design in a hurry, hoping to inspire and discuss with you. Of course, the software development mentioned in this article is mainly the development of business application software, while middleware, database and other technical component development concerns are in some other aspects, not here.
How to solve complex business design
Software architecture design is inherently complex, but there is a consensus in the industry that “componentization reduces local complexity by achieving separation of concerns.” In fact, whether we use containers, middleware, messages, databases, etc., are in a sense the product of componentization. This has the advantage of being reusable across different systems. In the rise of cloud native today, it is easier for us to use the form of universal and componentized services, so now if you do not enjoy the bonus of cloud native technology, then you will be abandoned by The Times.
Cloud native technology addresses many of the non-functional qualities and technical requirements to the greatest extent possible (see figure), and as an enterprise application architecture, the focus naturally shifts to the functional design of the business application itself. Nowadays, to design a complex business architecture quickly and well, there are no more than two situations: one is that the architect himself has a deep understanding of the business, strong ability and consummate skills; Second, the original business system itself has a clear model with enough “high cohesion and low coupling”, which can quickly analyze business changes and form a new business architecture design on its basis. What we should pursue is the second situation, which means that model design and business process should be carefully treated from the beginning of enterprise-level model construction. Only with solid foundation can we have “rapid iteration”.
Let’s get back to the essence of architectural design, why we design before code is implemented. Design first addresses the complexity of the problem. So someone built an architecture, handed it over to a team to implement, and quickly realized that the architecture and design of the implementation were completely different things. Of course the reason is clear — lack of communication and communication; The second is to establish a consensus of teamwork and communication. Even if we do a team consensus on architectural design, all dedicated to design into reality, a long-term problems appeared in the software industry, the demand is always changing, whatever upfront design “accurate”, always find the next hole is not far away, the results tend to be more and more worse, that is what we call “corruption” of architecture, Eventually, they had to accept a rewrite. These experiences have gradually made it clear that the essence of software architecture design is to reduce complexity through the separation of core problems and enable the system to respond more quickly to changes in external business and enable the system to continue to evolve. There is no need to start from scratch when changes are made, ensuring implementation costs are effectively controlled.
Therefore, from the perspective of architectural design, I think the following three points are the most critical:
-
Keeping our models, components, and business divisions as close to the nature of change as possible, such as users, goods, transactions, payments, etc. in the case of a general e-commerce system, allows us to “isolate” change within a certain scope (business module), which helps us effectively reduce change points.
-
Design, internal business model is high cohesion, low coupling between model, that is, their complete business is relatively independent, not because of one party drops and implications for the other party, such as commodity recommendation feature to hang out, but trading and payment services should continue to provide normal service, may prompt the user temporarily unable to provide recommendation service, or downgraded to out strategy.
-
Models and components should be reused as much as possible in business. It is such reuse that makes today’s Internet-level architecture possible. We will not start from scratch every time we build an e-commerce system. The business modules that are “reused” the most will obviously focus on design and operation and become core business modules. Of course, the architecture of such an e-commerce system will be more robust.
The above three points undoubtedly point to the business. Starting from the business and facing business changes is the key to the success of our modern architecture design. Therefore, the core essence of complex business architecture design is to ensure that we can have fast enough response capability when facing business changes.
Areas of design
I mentioned a common problem with business software development: evolving from a small project to a large business system, only to turn into a nightmare for the development team as new requirements are added. Most of these nightmares stem from the fact that the conceptual integrity of software (” conceptual integrity “comes from the software engineering classic, The Myth of the Man-month) has been broken. This business code can be stacked on top of each other by generations of developers (aka “shit mountains”) with no conscious effort to maintain the conceptual integrity of the software. DDD domain design, especially the strategic modeling level concepts provided by DDD, is the best medicine for maintaining software conceptual integrity.
“Technology serves the business and business drives technology” is now the consensus of most people, especially for business companies. The DDD domain design proposition that the business domain itself should be the focus of software design (in other words, software developers should understand the business) fits this idea very well; Moreover, DDD provides practical solutions to complex business software design, which I strongly advocate as an architect to learn and discuss DDD design knowledge in depth.
Strategic modeling
At the strategic level, DDD places a strong emphasis on the analysis and decomposition of business problems, reducing their complexity by identifying core problems. DDD maintains the conceptual integrity of models at the strategic level. The two most important concepts are Bounded Context and anticorrosion Layer.
- Define bounded contexts
The definition of bounded context will be explained in detail in any DDD book, but I just want to share some of my own understanding. At this point, one might ask: How large is the bounded context appropriate, and are there rules to follow for partitioning contexts?
It may be too much of a stretch to say that the rules for dividing context are nothing more than a universal “high cohesion, low coupling”. What really makes people struggle is the connection between things they don’t know how to slice, and sometimes even put into a context. In fact, RATHER than focusing on the “size” of the context, I think it’s better to focus on the “quality” of the model and whether the integrity of the concept is easily compromised. I think the right size depends on the ability of the application development team to control the conceptual integrity of the software. As long as the development team doesn’t have a problem, this range can be as large as possible.
If the development team is upstream in the industry, the range of maintenance contexts is often large; Some companies have uneven development teams, so projects may need to be broken down into relatively small contexts to minimize the “shit mountain” that keeps piling up.
- Make anticorrosive coating
The boundary context needs to always protect the boundary it maintains and the integrity of the concepts within the boundary. At this time, the place where the concepts of one context need to be transformed into the concepts of another context is called “anti-corrosion layer”. There are many kinds of anticorrosive coating, typically implemented as an Adaptor Adaptor, for example, in addition a broad sense, the Gateway is a typical coating components, of course, the anticorrosive layer of code and other internal business model to obvious physical boundaries between (of course not to say to want to the coating as a separate deployment process), At the very least, we can consider building and maintaining the embalming layer as a separate class library, which is the idea behind ali’s internal products such as Star Ring.
Tactical modeling
The core tactical concepts of DDD are entities and aggregations. To better understand what aggregations, aggregations root, and aggregations internal entities are, here are some examples. Imagine an order-dependent model for an e-commerce system. We might have three interrelated concepts: Order, Order header OrderHeader, and Order row item OrderItem:
- An aggregation called Order.
- The aggregation root of this order aggregation is an entity called OrderHeader, whose ID is called OrderId (Order number).
- Through the OrderHeader entity, we can access an aggregation of OrderItem entities. The local ID of the entity OrderItem is called ProductId. Because the business is fickle and does not allow the same product to appear in different order items of the same order, we can select the product ID as the local ID of the order item.
“Unit” data aggregation is modified, based on this principle, we can do it “aggregation strong consistent outside, aggregation, eventually consistent”, for example, we can not accept all order items within an order, the total amount of the order amount is not equal to the sum of the first we have to put the order and order line item within the two entities into the same polymerization.
- Principles of design aggregation
Let’s take a look at the description of aggregation design principles in the book Implementing Domain-Driven Design, which is a little confusing. Let me explain a little bit:
-
Model true immutable conditions within consistent boundaries. Aggregation is used to encapsulate true immutability rather than simply grouping objects together. There is a set of unchanging business rules within an aggregation, and entities and value objects run according to the same business rules to achieve consistency of object data. Anything outside the boundary is irrelevant to the aggregation, which is why the aggregation can achieve high business cohesion.
-
Design aggregates as small as possible. If the aggregation is designed too large, it will contain too many entities, resulting in complex management between entities, concurrent conflicts or database locks for frequent operations, and ultimately poor system availability. The small aggregation design can reduce the possibility of aggregation refactoring due to large business and make the domain model more adaptable to business changes.
-
Reference other aggregations by unique identity. Aggregations are referred to each other by associating external aggregate root ids rather than by direct object references. The external aggregation object is managed within the aggregation boundary, which will easily lead to unclear aggregation boundary and increase the degree of coupling between aggregates.
-
Use final consistency outside of boundaries. Data is highly consistent within aggregations and ultimately consistent between aggregations. You can change the state of at most one aggregation at a time. If a business operation involves multiple aggregation state changes, use domain events to asynchronously modify related aggregations to decouple them (I’ll explain this in the domain events section).
-
Service invocation across aggregation is implemented through the application layer. To achieve decoupling between aggregations within microservices, as well as future aggregation-based microservice composition and unassembly, domain service calls across aggregations and database table associations across aggregations should be avoided.
These principles are some of the general design principles of DDD, and the old saying, “What works for you is best.” In the system design process, you must consider the specific situation of the project, if faced with the convenience of use, high performance requirements, lack of technical ability and global transaction management and other influencing factors, these principles are not impossible to breakthrough, in short, all to solve practical problems as the starting point.
- Design the steps for aggregation
DDD domain modeling usually adopt similar storm events, usually through case analysis, scenario analysis and user journey analysis, through the brainstorming list all possible business behavior and events, and then find out the behavior of domain objects, and combing the relationship between the domain objects, find the aggregate root, with close connection of the aggregate root business entity and value object, The aggregate is then constructed by combining the aggregate root, entity, and value object.
We are all familiar with the e-commerce system, and there are many mature models for e-commerce business can be directly used for reference; Now let’s take another scenario — insurance business as an example to see what steps the aggregation construction process mainly includes. Of course, this example I saw from other learning materials is relatively typical and can be used as an example to illustrate:
-
Step 1: Use case analysis or event storm and other methods are adopted to sort out all entities and value objects of these behaviors in the process according to business behaviors, such as application policy, subject matter, customer, insured and so on.
-
Step 2: From the list of entities, select the root entity that is suitable for the object manager, the aggregation root. To determine whether an entity is an aggregate root, as discussed in the previous chapter, you can combine the following scenarios: Does it have a separate life cycle? Is there a globally unique ID? Can I create or modify other objects? Is there a dedicated module to manage this entity? The aggregation roots in the figure are the application form and the customer entity respectively.
-
Step 3: Identify all closely dependent entities and value objects associated with the aggregation root, as described in the previous chapter for designing aggregation. Build a collection of objects containing an aggregate root, multiple entities, and value objects, which is called an aggregate. In the figure we build the customer and insurance aggregations.
-
Step 4: Draw the object reference and dependency model within the aggregate based on the dependencies of the aggregate root, entity, and value object. Here need to explain: the applicant and the insured person’s data, is through the polymerization of associated customer ID from customers, they are in the insured polymerization insurance application value of the object, the data value objects are the redundant data of customers, even if the future customers aggregated data changes, also won’t affect the value of the insurance application object data. From the figure, we can also see the reference relationship between entities. For example, in the insurance aggregation, the application aggregation root refers to the quotation entity, while the quotation entity refers to the quotation rule sub-entity.
-
Step 5: Multiple aggregations are grouped together into the same bounded context based on business semantics and context.
So that’s the whole process of polymerization.
Domain modeling strategies for different scenarios
Due to the various situations within the enterprise, the development process is also different, including the micro-service transformation of the legacy single system, the business modeling and system design in the new unknown field, and the local optimization of the legacy system. The strategies for domain modeling vary from scenario to scenario. Let’s break down a few scenarios to see how to model the domain.
The new system
For complex business domains, domains may require multiple levels of fragmentation before domain modeling can begin. Domains are split into subdomains, and even subdomains need to be split further. For example: insurance needs to be split into underwriting, claims, payment and reinsurance subdomain, underwriting subdomain and then split into insurance, policy management subdomain. If the complex domain is not further subdivided, the engineering amount of domain modeling will be very large because the problem domain is too large. It’s not easy to model a large domain through event storms, and even if you do, it’s not always good.
For complex domains, we can accomplish domain modeling and microservice design in three steps.
-
According to the characteristics of the business domain, the domain model is built by disassembling the molecular domain and referring to boundary factors such as process node boundary or function aggregation module. Combined with the discussion of domain experts and project team, the domain is decomposed into appropriate subdomains step by step. For the subdomains, event storms are adopted to divide aggregation and limiting contexts, and domain models within the subdomains are preliminarily determined.
-
Fine-tuning of domain model The domain models of all subdomains in the domain are sorted out to fine-tune the domain models of each subdomain. The fine-tuning process focuses on the reorganization of aggregations in models of different domains. The final domain model is determined by simultaneous consideration of domain model and aggregation boundaries, and dependencies between services and events.
-
Design and disassembly of microservices According to the domain model and the principles of microservice disassembly, complete the disassembly and design of microservices.
Unitary legacy system
If we are dealing with a monolithic legacy system, we only need to separate some functions into microservices, while the rest remain as monolithic and the whole remains unchanged, such as splitting the modules facing performance bottlenecks into microservices. We just need to understand this particular function as a simple subdomain, and follow the simple domain modeling approach. In microservice design, we should also consider service and business compatibility between old and new systems, introducing preservatives if necessary.
Challenges in the cloud native age
With the rise of cloud native technologies, enterprise architectures are now more cloud-based, and cloud-based architectural styles have a new focus: elastic boundaries. Elastic boundary is the core concept of a cloud native enterprise application architecture. It refers to the system boundary defined by taking elasticity as the top consideration factor, which determines whether we can give full play to the full capabilities of the cloud native platform. So we need new ways to complement the old business model to meet the needs of the new cloud biogenesis. Now, microservices are basically based on a cloud native architecture, and using microservices on a fixed, flexible platform has a high implementation cost. Arguably, cloud native should actually be a prerequisite for microservices.
In the cloud native era, we need to make elasticity a primary consideration in our modeling. So the elastic boundary is an important basis for dividing systems. Furthermore, we need to consider the dependency between elastic boundaries and avoid elastic coupling as much as possible. For business modeling, in order to match the architecture of the cloud native era, I think there are several things to do:
-
To establish a model structure that can reflect elastic boundary, it is necessary to consider the principle of different elastic boundary to divide the boundary context. If two contexts clearly have different elastic appeals, they should be split. And if there is a consistent elastic appeal, you can consider not demolition. How “micro” can it be at this point? In short, it is “micro” to be able to better use flexibility to control the size of the cost.
-
From the perspective of asynchronous model, to optimize business logic; The typical example is an MQ message queue system where, thanks to brokers, producers and consumers do not have to be available at the same time with the same throughput, and producers do not have to wait for a reply immediately.
-
Loose coupling of locations: A typical example is a service registry, where the consumer does not need to know the location of the provider directly, but looks for services through the registry to access them.
-
When an elastic boundary is segmented into business context, strong service consistency is maintained within the same elastic boundary.
-
When an asynchronous invocation generates an intermediate state exception, the business consistency needs to be maintained.
Don’t ignore the impact of organizational structure
“Conway’s Law” tells us that organizational structure determines the structure of team communication, as well as the structure of products. The sorting of organizational structure is often done in demand research. In the case of information collection, business architecture design is nothing special here. The difference is that business architecture is aimed at enterprise-level capability planning, with the hope of breaking down barriers and forming synergies. Just for this reason, organizational structure has a great counterforce to business architecture design. Enterprise-level digital transformation plan should match with organizational structure, otherwise it will be difficult when it is implemented. Arguably, departmental interests are one of the biggest obstacles to doing enterprise architecture, and overcoming this barrier is also one of the requirements for the architect’s ability. Of course, there are situations where standing still is an option when no better solution is available.
In my experience, there is no good way to solve this problem, but there are two ways: one is to have the leadership of super-competent people, with the support of the top level, to push forward this kind of decision. However, the bigger the enterprise, especially in the business-dominated enterprise, the more difficult it is to form this kind of structure. The second is to strengthen the business enterprise internal architecture personnel’s ability and the number of departments (preferably has a similar role), make the enterprise personnel fully involved in the project in the form of partnership, in the process of the implementation of the project set up collaboration network, improve the decision-making efficiency, to make the organization structure is no longer the bottleneck of enterprise digital transformation.
Soa-microservices-mid-stage: The Art of compromise
Many years ago, these traditional large ERP business software, in fact, maintained the integrity of the business concept on a large scale. After an ERP is installed, the database has 800 or 800 tables (that is, 800 or 800 entities) in the same bounded context. But these ERPs maintain the integrity of the business concept within the context of such a large boundary, which is admirable.
However, it is very difficult to implement, but very easy to destroy. After the implementation of a set of ERP customization projects, the database may have hundreds more tables, not to mention the non-standard naming looks strange. ERP implementation consultants and developers from these vendors work day and night to maintain this huge “shit mountain”. We can’t let these huge “monomer apps” grow unchecked, so we’re once again embracing the “divide and conquer” mantra. Software componentization technologies such as SOA give us tools for fragmentation. We divide a large bounded context into several relatively small bounded contexts according to its advantages; In physics, we split a large monolithic application into services.
In general, we will let the physical boundary of the service and the domain boundary of the boundary context be basically stacked, a boundary context for one or more independently deployable service applications, service applications contain the implementation of the core business logic of the boundary context. The physical boundaries of SOA’s service components make it difficult to invoke services between them, which allows developers to simplify relationships between objects and write more “cohesive, low-coupling” code. When there aren’t many service components, it’s not a lot of work to build the anticorrosion layer; we just have to deal with the code between the components.
However, our architects and developers like to “divide and conquer” too much, the widespread use and even abuse of microservices, let’s see that many microservices are really “micro”, almost an aggregation of DDDS for a microservice that can be deployed independently. Such microservices cannot do much business by themselves, which requires more and more microservices to be “aggregated” together to provide business services.
Of course, the development of microservices technology infrastructure also provides more convenience for the invocation between services, crossing the boundaries of microservices has become the norm; At this point, the business developer needs to keep his mind on the distinction between “service invocation in the same context” and “embalming between contexts”, and the physical boundaries of the bounded context and microservices are often difficult to align, which inevitably increases the difficulty of maintaining the conceptual integrity of each bounded context.
Since it is increasingly difficult to maintain the conceptual integrity of “tiny” independent bounded contexts, why don’t we aggregate them again? Put them together in a context of modest boundaries, and this is what is called enterprise business architecture, or what we now call business mid-stage, with the ultimate goal of achieving “enterprise” harmony.
So to a certain extent, software engineering is the art of compromise, “the golden mean.” We want China to China, no matter the size of the enterprise, should be combined with its own business goals and resources, in “the wider concept of integrity maintenance” and “more anticorrosive layer code maintenance”, to make a balance between that this is also an enterprise architects have to do one of the most core.
Our team has indeed accumulated and practiced some “business Middle-stage methodology” over the years, and practiced it in some projects. Of course, one of the most soul parts is the domain design mentioned above. In the past, many people said that the most difficult thing about DDD design and even business platform methodology is that there is no appropriate tool or platform to practice it. In fact, COLA, Star Ring and BizWorks used internally by Alibaba are very good tools and platforms.
conclusion
Enterprise application architecture is constantly evolving and iterating, but I always feel that the formation of enterprise application architecture is achieved in a process that looks scientific but is not completely scientific. Carefully think about it, do what software architecture is very envy of building architecture, because building architecture have precise mechanics foundation as base, there are a lot of things can be exactly calculated, and the software architecture is not much can be precisely calculated, so that “compromise” continuously losing is a kind of feasible design thinking and design art; In fact, it is true that there is no silver bullet.
Due to the short time, some content is briefly mentioned, and some places that should be illustrated with examples are also briefly mentioned in this paper. If I have time later, I will combine more practical cases to supplement this point of view. I also hope that this paper can inspire everyone to think and discuss the design of enterprise application architecture in the current cloud native era, learn from each other and make progress together.
The original link
This article is the original content of Aliyun and shall not be reproduced without permission.