During the conversation, Lu Zhi asked “why don’t you open source?” That quote stuck with me. – Lao wang
In the early spring of February, in the drizzle by the west Lake, I visited the middleware team of Ant Financial and had an in-depth conversation with Lu Zhi, the technical director of SOFA. What’s more, Lu Zhi is also responsible for the promotion of SOFA open source affairs. I am very interested in such an open source project that earnestly practices the open core mode.
The conversation between the two technologists is simple and straightforward, and the topic mainly revolves around SOFA and open source. I hope it can also arouse the resonance of you who are also technologists.
intro
Respondent: Lu Zhi, principal of ANT Financial SOFA Open source.
Interviewer: Lao Wang, an open source evangelist, is a 20-year tech veteran of the Internet industry.
Although Lu Zhi and I have been in touch on wechat for a long time, this is the first time we have met. During our conversation, I learned that Lu Zhi joined Alibaba in 2009, which has been ten years. At the beginning, I worked as a business system in 1688.COM. I was very interested in middleware technology and often studied the realization and functions of various middleware. Later, in 2013, in order to further study the middleware framework, I transferred to ant Financial Middleware team and have been working on SOFA since then.
At present, Lu zhi’s team in SOFA is mainly responsible for several parts. One of the main parts is the work related to open source SOFA. SOFA product system is very broad, including the open source parts, the entire internal micro-service system, as well as SOFA framework and so on – and these open source related work is mainly promoted by Lu Zhi.
Of course, as the person in charge of technology, Lu Zhi should not only take the technical team but also do technical work. Referring to this, Lu Zhi said:
“I think technical management is different from ordinary management, because the most important point of technical management is to maintain a certain degree of technical judgment and acuity in addition to management. You should have some sense of direction for new technologies, including major technical problems in your team. Although it may not be your specific solution at the end of the day, we need to set the direction together on the technology and technology selection of the whole team.”
I used to do more than ten years of technical management, I can feel this situation, the major problems of the technical person to grasp the nettle.
SOFA 5 includes Service Mesh
As far as I know, SOFA has been developed into SOFA5. In THE SOFA4 phase, the main task is to clarify the open source architecture and then start to open source step by step. And now it’s SOFA5. I would like to know from the development of SOFA4 to SOFA5, what makes the middleware team of Ant Financial judge that the stage goal of SOFA4 has been achieved and it is ready to move to the new SOFA5 stage?
“In terms of the overall industry trend, THE ARCHITECTURE of SOFA4 is still relatively traditional, more of a rearrangement of our previous technical framework. At this stage, SOFA code after a lot of optimization and reconstruction, to meet the requirements of open source, thus SOFA stepped onto the open source core model, gradually open source each part of the stage.” Lu zhi said, “However, from our overall judgment of the industry, the future is undoubtedly the era of cloud, so we should consider how to make all business systems can provide cloud capabilities, such as Serverless.”
Following this topic, Lu Zhi talked about his understanding of cloud computing: “On the one hand, cloud computing must provide more convenient basic resources for the development of the whole business, without caring about the underlying infrastructure. Serverless literally means’ no server ‘- I don’t care where the server came from, I don’t care about the infrastructure, I just care about the business code. In turn, for cloud service providers, after this layer of abstraction, their resource utilization will be higher, they can have more profit space, which is a win-win situation. “For users, the benefits are real, you can focus less on the infrastructure and more on the code.”
“We hope that in the direction of SOFA5, in this new iteration, the business — including in the future when we open source all kinds of functionality, all kinds of service models — will care more about their business code and less about the infrastructure.” LuZhi said.
In SOFA5, one important direction is the direction of the Service Mesh, which will be a very important feature in SOFA5. Lu emphasizes his enthusiasm for Service Mesh: “I think Service Mesh is a very critical step forward into the future, where businesses don’t care about infrastructure anymore. With Service Mesh, we can put a lot of technical capabilities directly into the infrastructure without the business being aware of this layer. “Infrastructure problems that might have taken hours or more to solve can now be solved through Service Mesh.”
“So far we have applied Service Mesh in production. We are very committed to this, and we hope to be able to implement Service Mesh on a larger scale this year. The current phase is more focused on the internal implementation of this technology, hoping to make more contributions to the community once it is well used.”
The term Service Mesh was first proposed by Buoyant, a company that developed Linkerd, in 2016. With the introduction of Linkerd, Service Mesh has entered the domestic technology community. Service Mesh is also translated as “Service Mesh”. Linkerd was the industry’s first Service Mesh.
The Service Mesh is an infrastructure layer that handles communication between services and is responsible for the reliable delivery of requests. In practice, service grids are typically implemented as lightweight network proxies that are typically deployed with, but transparent to, applications.
There are two deployment models for Service Mesh:
◈ For a simple request, the client application instance as the initiator of the request first sends the request in a simple way to the local Service Mesh instance. These are two separate processes with remote calls between them. The Service Mesh completes the entire inter-service invocation process, such as Service discovery load balancing, and finally sending requests to the target Service. This is Sidecar, which adds a proxy between the existing client and server. ◈ In the case of multiple Service invocations, the Service Mesh appears underneath all the services. This layer is called the dedicated infrastructure layer for inter-service communication. The Service Mesh takes over the network and forwards all requests between services. In this case, the service above is no longer responsible for passing the specific logic of the request, but for completing the business processing. The communication between services is stripped from the application, presenting a layer of abstraction.
If there are a large number of services, the connections between sidecars form a network, hence the name of the service grid.
“We’re going to use Service Mesh as a springboard to go further.” “Serverless should be focused more on its literal meaning, which means’ Serverless ‘,” lu said. “The technology behind it is all about making Serverless host specific business.”
Although the concept of Serverless has been put forward for several years, AWS is at the forefront of Serverless and FaaS, but in China, the development of Serverless and FaaS technologies is still relatively lagging behind.
“I think Serverless needs to be able to cover the entire breadth of its business in order to be successful,” lu said. “Otherwise it might stay on FaaS, which would be a more limited scenario.”
Service Mesh will be the next era of microservices, and it is still under theoretical and practical exploration.
“To be honest, I think istio’s concept is very good, but the overall engineering design would need some work to run in a larger environment like Ant Financial,” lu said. We hope that after Service Mesh is launched on a larger scale in Ant Financial this year, we can apply some of our practical experience in Service Mesh to the engineering of product environment and contribute to it. “There’s a lot of work going on to take the whole system and apply it to more businesses, and then feed that experience back into the design of the entire Service Mesh to make it go further.”
In other words, Ant Financial follows istio’s technical path in Service Mesh, but will promote its development more from an engineering perspective.
We want to be able to push it out to the open source community carefully after we have done production verification. This is the philosophy behind ant’s open source contribution.
Zhi LU: “We want to be able to carefully push it out to the open source community after we have done production verification. This is the philosophy behind ant’s open source contributions — we want things to mature internally for a while before we open source. After a large scale of internal verification, its stability was guaranteed, so it was contributed to the external community for use, and then to expand more use scenarios, including improving and solving some problems that had not been encountered before.”
Synergy Seata distributed transaction framework
Since 2007, Ant Financial has independently developed the distributed transaction middleware XTS, which is widely used internally to solve the problem of cross-database and cross-service data consistency in financial core scenarios. Finally, it is presented and opened to the outside world as DTX cloud product. At the same time, Alibaba middleware team released TXC to provide distributed transaction services for applications within the group. After years of technological precipitation, it was transformed into GTS in 2016 and implemented in many external customers through Ali Cloud solution.
In January 2019, based on technology accumulation, Alibaba middleware team launched the open source project Fescar. Ant Financial also opened its own distributed transaction framework and merged with Fescar to build distributed transaction solutions. This development is both expected and unexpected, and I do wonder what happened during this period, how it relates to the development of the SOFA middleware team, and what’s next for them?
“Distributed transactions are an innovation made by Ant Financial in 2007, based on the TCC principle, and we have implemented this model internally,” lu said. TCC theory is relatively simple, but it needs to be polished in a long project to be implemented. The technology of distributed transaction has been in Ant Financial for 12 years. Ant Financial uses this distributed transaction framework to solve and incubate some of its most core businesses, including payment, transaction, accounting and other systems.”
In the field of distributed transaction, the industry is relatively blank at present, there is no very good distributed transaction framework. Speaking of the original intention of the merger, Lu said, “Since alibaba and Ant Financial have both done some open source work in this direction, we combined the efforts of the two parts to learn from each other and apply to more distributed transaction business scenarios. Ant Financial joined the Seata community to build the business. TCC mode was added in Seata 0.4.0 to provide a broader solution for distributed transactions.”
To be specific, “Alibaba’s Seata provides AT mode, which does not have too much perception for business, but it covers limited scenes. If such a situation is acceptable, AT mode is better. Ant Financial has stronger financial requirements, so it needs to adopt TCC mode, which has higher cost of business access, but it can achieve very good distributed execution. In the future, there will be models like XA to adapt to a wider range of business scenarios, where Ant Financial and Alibaba will combine to provide a framework for convergence.”
Seata provides a unique solution to the problem of distributed transactions in microservices architecture. Seata’s vision is to make the use of distributed transactions as simple and efficient as the use of local transactions, hopefully making Seata applicable to all distributed transaction scenarios.
How to Open Source
As a project of open source core mode, I hope to know what work the open source middleware of Ant Financial will generally do, and whether there are relatively perfect processes and rules?
“First of all, is the most basic code, and provide corresponding sample, then we can provide contributors to guide such guidance document, because essentially we want to make an open source community, community participation is very important for us, some to mention issue at the meeting, also somebody to answer, some functional requirements, Someone mentioned PR and so on, “Lu zhi said.
Linux China had developed a is used to calculate the open source project a model of activity, we think that from the perceptual to see an open source project is active, has begun to gradually progress to through rational data evaluation, but the need to rule out some data to cheat, as before a lot of people will with the star number to assess the activity of the project, Of course it’s crude. We will continue to provide data support for the 2019 open Source annual report this year, and we will definitely make an investigation of SOFA related projects then. Hopefully, it can reflect SOFA’s work in open source.
From the previous context, SOFA5 will continue to follow the open core model, that is, the core is open source, strongly associated with the local business, but the core is not strongly associated with the open source.
From the perspective of ant Financial’s own practice, they have practically implemented the open source core mode.
There are pros and cons to the open source core model. However, from the perspective of ant Financial’s own practice, they have practically implemented the open source core mode, which is the first real implementation of the open source core mode project that I have deeply understood in China.
Speaking of the open source model, Lu zhi said: “To do open source, I think first of all, you must make a psychological preparation, that is, you have to have a core part, and then expand on this basis, there will be a certain increase in maintenance costs, but you have to accept such costs — I think such costs are acceptable. … The project itself should be well designed, with a certain possibility of disassembly. If there’s no possibility of a spin-off, it can’t be done. A design approach like the microkernel — open source a core module and then extend it, and the modules are pluggable.”
For the open source work is how to do, LuZhi said: “we don’t have special person for open source, also does not have special person for internal code, we are put the two parts together, both open source and do internal code, because such a benefit is that both familiar external code, and familiar with the internal code, this boundary can grasp better myself. We’re more of a rule-maker. For example, if you open source the parts that are strongly related to the business layer, no one will use them. If it’s not relevant to the business, why don’t you open source? Because you open source the product to do better, these capabilities are not too big a problem, so generally our standard is to see whether it is related to the internal system, whether it is strongly related to the business, if not related to open source.”
During the conversation, Lu Zhi asked “why don’t you open source?” That quote stuck with me.
During the conversation, Lu Zhi asked “why don’t you open source?” I was impressed by this sentence, which actually represents their original intention of open source. However, from the perspective of businesses and companies, does open source bring real benefits to companies? This is not just a matter of feelings. I believe that everyone who loves open source actually has feelings or even idealized ideas about open source. But on the other hand, open source should have real benefits and promote the development of the company’s business, no matter in terms of the company’s mechanism or company’s performance. As the head of an open source project, how does he feel the benefits of open source?
It was clear that he had given mature thought to the question:
“The immediate benefit is more lasting. In the short term, is your reputation out there making it easier to hire someone better? This is the shortest term gain. In the long run, there are a lot of ideas shared in the open source community, and in practice, too. For example, if you work in a company, the business scene of the company is limited. Although Ant Financial covers a variety of businesses, almost all of the financial aspects are covered. But the rest of the industry does not necessarily have, they encounter this problem, we may not meet, but these problems may be able to meet in the future, if you work in a way that a project to open source, means, said more use case scenarios are more likely to find bugs, the more people use, the more likely to trigger this bug, That is the possibility of further improvement.
In addition, with this kind of community development, the project can move forward more quickly when more people are involved, rather than just you. In a company, the number of people on the team is certainly limited, and having so many people on the team is a huge benefit to the evolution of the project, which in turn brings more benefits to the company — both potentially and directly.
Finally, if you have commercial support for your product, such as support from other systems, you are better able to provide commercial support.”
As far as I know, there are about 30 enterprises using this open source framework since SOFA was opened. Of course, this is mainly domestic users, so where is the reason why the influence in foreign countries is not as big as that in China? Is it because our project is not international enough, or not enough people know the framework?
“I think there are two reasons,” Lu said. On the one hand, it is true that we have not done much in internationalization, and we will try to do more in this year. On the other hand, there are more cultural differences, and people may have different ways of thinking. Of course we’re going to try to go international, because open source doesn’t respect borders.”
Further, in terms of SOFA’s community governance, “We hope to adopt and refer to the Apache Foundation’s approach, which is a very perfect governance model, and we will try to adopt this approach to community governance. There are a lot of benefits to internationalizing products, and it’s more about governance, whether it’s done in a community way, whether it’s done with respect to the community, etc.” Lu said, “We will consider making direct contact with the Apache Foundation and CNCF, and if appropriate, we will donate projects to the foundation. If it is just a commercial company without the support of the foundation, people will have more concerns. Donating the project to the foundation will give people more confidence. Through the foundation’s trusteeship, more participants will participate, rather than ant Financial, and people will have more confidence to participate.”
Finally, Lu Zhi hopes to send a message to the open source community, “In fact, Ant Financial will open source more things, including some AI technologies, than just SOFA middleware framework. We also hope that the whole community can pay more attention to ant Financial’s future open source initiatives.”
Financial Class Distributed Architecture (Antfin_SOFA)