Basic data provide data support for various business systems of auto finance. In order to realize this ambitious goal, our R&D team has witnessed the experience of starting from scratch and pioneering all the way. Let’s begin to understand that journey, to remember those extraordinary years. As a witness of this process, I write an article to recall the spring and autumn events.
Car finance past life album serial series, follow wechat public account [Autumn night without frost]
Financial | car GPS incarnations of the audit system
Car financial | basic data platform of incarnations
Car incarnations of financial center | contract system
Car financial | financial product rules engine incarnations (last)
Car financial | financial product rules engine incarnations (medium)
Car financial | financial product rules engine incarnations (next)
Car financial | I M in the two years
A preface,
The whole article has more than 4,400 words, all of which are not easy to use during the subway journey. Please forgive me for my poor writing. Thank you for reading carefully, I believe you gain a lot from it and enlightenment. Thank you for your attention, and I will continue to publish articles in this series.
(This article was written from November 11, 2020 to November 14, 2020, the final version was published on November 13, 2020, and the illustrations were added from November 14, 2020 to Saturday, covering a period of 4 days, on the subway between Shunyi and Dawang Road)
Second, the introduction
It took nearly a year for the basic data platform to emerge from zero. In order to promote the establishment of this project, it is natural that the strong support of the boss and the hard work and unremitting efforts of our financial product team partners are indispensable.
Basic data platform, as an upstream application service of auto finance business line, provides basic data management and services for downstream related business systems. From a business perspective, it provides basic data support for the entire business process from order making to lending (e.g. data dictionary, administrative code, dealer units and stores).
In addition, in order to meet the development scale of the company’s business and support the future business growth, from the beginning of the project, from the technical framework and application service performance, made a lot of RESEARCH and development design work, achieved good results.
The age of chaos
At the beginning of the entry, I the letter D team had an old system, to provide the function of the operating personnel management related functions including basic data platform, as part of the early these closely linked with the business system data dictionary, administrative code, dealer units and stores, because does not have a unified system used to provide interface service, a database system and the service, So they are to query the data for business logic support.
Although these business systems, such as the order receiving system, contract system and audit system, which have been split in the early stage, belong to different team leaders, we often find that the business logic codes of query data dictionary or dealer unit stores can be seen everywhere and are common.
After all, they do not belong to the same application, but share the same database, since no application service provides an interface. In fact, they caused the system late business split, leaving a fault. Without this problem, there would be no basic data platform. The need of history coincides with the birth of heroes. The accident of history makes a glorious age. And I certainly volunteered, for the sake of the team, willing to take on this difficult task.
In the process of reconstruction, since financial products are closely connected with data dictionary and dealer units and stores, in view of this, in order to consider the clear positioning of future system architecture and business system, I led the team to build a unified basic data platform for the product operation team to use. At the same time, a basic data service was established to provide interface services for not only the LETTER D team, but also to provide support for the whole team of auto finance business line.
It can be found that the operation platform and application service modules share the same database, and all business systems on the operation platform are directly connected to the database.
4. Reconstruction process
In the second half of 2018, it has been half a year since I joined the company. During this time, car finance lines mushroomed and a number of new systems were reconstituted, For example, the audit system (Poseidon), order receiving system (CAR-Yulv), financial product system (Car-Heil and CAR-product), GPS audit system (GPS-Web and GPS-provider), contract system (Doraemon) and D-after system (CAR-After) Loan), and so on. However, the reality of these business systems is that they find their business scenarios related to the underlying data. At this time the relevant business system, some of their own split out of their own database, and some are still common to the original database. And the old systems because they need to, so open up some new interfaces to provide these business systems with their own independent database to provide basic data support.
Time goes by in the iteration of requirements. With the development of business, technology and products need to constantly adapt to the scale of business development and policy environment, and constantly innovate themselves. Twinkling of an eye to the APP side for the first time refactoring, when refactoring a loud code within the team “do ChanChaoRen (later renamed the ‘do ChanXia)”, because of the need to provide data dictionary APP end, so the financial product center (this time still rely on the original large database) these interfaces service function, thus provided to simba system services.
Later, it was considered that the Financial Products Center needed to disassemble the database, but after disassembling the database, such as the data dictionary, the dealer unit store, should the database have redundant related tables, or should it be separated into an independent service to invoke its interface service?
After careful consideration, I propose two technical solutions:
- 1. Data source table synchronization is realized based on the open source project DataLink of Shenzhou Special Vehicle.
- 2. Build a basic data service and provide basic data interface service based on the original library.
The first technical solution, in my old employer shenzhou special car is actually very familiar. The scenario of this technical solution is applied to the premise that the relevant systems within each business line depend on the basic table of other business systems, such as the driver system depends on the model table. The technical solution at that time is that the architecture team realizes the synchronization between data sources through the data synchronization platform.
You might ask, what if the dependent database table field definition changes? This is a natural question for the architecture group middleware team at the beginning of the design process. The platform supports two options, full scale field synchronization and on-demand field synchronization. Each business system can choose one of the options to meet its own scenario requirements.
Again, we realize that the data is synchronized, and we must consider data consistency issues, such as how long it takes to delay synchronization. Therefore, if you think about this problem, it is naturally due to the strict requirement of its own business to synchronize data all the time. Frankly speaking, the platform can expand the number of nodes (workers) to support the needs of business scale. At least the delay is small and negligible.
This data synchronization feature, provided by the platform, can solve the data dependency problem for many teams. Through this platform, configuration separation can liberate more productivity without relying on business systems to provide interfaces.
Our team members also agree that we are willing to try this scheme. After all, the car finance business line was investigated and not applied in practice at that time, so we can try it. However, unexpectedly, the boss was not optimistic about this plan, he preferred the other side to provide interface services to the business side.
Unfortunately, the boss doesn’t approve, and we can’t do it under the table. In fact, we ignored an important problem later. The data synchronization platform configuration provided data source and required database account and password, but at that time, the DBA provided encrypted password to each line of business, and naturally did not provide plaintext to the line of business. Later, we had to choose the second option, which paved the way for the birth of basic data platforms.
In order to provide basic data services for each business line, we set up a new application service (CAR-Transfer), which I originally translated as “Evangelist” in Chinese. The main purpose of basic data service is to provide basic data support for business parties. The so-called “evangelist” is also called “Evangelist”. This application service provides the most basic data query, including data dictionary, dealer units and stores, administrative code.
After the application service is launched, the business system within our financial product team, including financial product system and GPS audit system, will be the first to be connected. Then we promoted it to the credit team and other RESEARCH and development teams, and connected with the audit system, bill of lading system, contract system, post-loan system, etc.
5. Architecture design
The boss said that since the basic service provides interface services for numerous business systems in the car finance business line, the stability of the service is crucial. In view of this, we have achieved more innovative design results through research in the system architecture and design for this great goal.
throughput
Service throughput. Under the premise of distributed cluster deployment, we can increase the number of server nodes to improve the overall application service throughput. Under the premise of server cost, under the strict server cost budget, we are thinking about how to improve the throughput of single node. Frankly speaking, it is to use one node to give play to the throughput capacity of multiple nodes.
Therefore, based on the premise of single node, we have designed some interface services to increase the QPS value of the interface by adding the first and second level cache (the first level cache is local cache, and the second level cache is based on Redis distributed cache) in the scenario of more read and less write.
The core design of primary and secondary caches lies in different business modules, and cache management is isolated. For each independent cache module, the interface is first loaded from the level-1 cache, which cannot be read, and then loaded from the level-2 cache, which cannot be loaded, and finally loaded from the database, and then callback to the level-2 cache. When the interface is invoked for the second time, it can be loaded from level 1 cache on the premise that the cache period is not invalid, so as to reduce frequent query operations to the database and make full use of the performance of the cache to improve the QPS and throughput of the service.
For details, see article: Design and implementation of primary and secondary cache based on Google Guava and Redis
The first load of the cache is designed to be initialized the first time it is called externally. The first time an interface is called, the reader might ask, it must take a long time. You’re right, but at least that’s the size of the business. We can also choose timing to check the level 1 cache in the design of initial loading, but this design implementation is not necessary at present, will only increase the complexity of system design, we choose Simple design to achieve business goals, the so-called “Simple is perfect” is also.
Some readers will wonder how you solve the cache consistency problem. If you are asking, I will tell you that in this case, if the database field values change, we choose manual intervention to reset the cache. Since our data rarely changes, we provide cache management capabilities in the base data platform and provide both full reset and specified data reset of the cache module to meet the needs of our business scenarios. Resetting the cache simply means clearing the primary and secondary caches and reloading them from the database.
Since the introduction of cache, we will also consider such serious problems as cache breakdown, cache penetration and cache avalanche. However, I can tell you that our company belongs to the auto finance industry, and the level of finance is definitely lower than other industries of e-commerce. In addition, our application services only open Intranet domain names, and our data volume is not high, such as data dictionary, the current business is only tens of thousands, so it is unnecessary to worry about. Ha ha!
availability
Since the online construction and release is based on docker containerization, the platform supports the rapid replication of N nodes based on the completed application service image, so it is beneficial for us to quickly remove the faulty node and create a new node to replace the faulty node.
The platform provides a fast CI/CD build, and our service is exactly deployed on multiple nodes according to the business volume, which is load balanced through NGINx. With two K8S clusters deployed online at the same time, the application can support multiple clusters for failover and disaster recovery, so the chance of service unavailability is almost zero unless the ENTIRE K8S cluster is down, which is possible.
6. Promotion Ambassador
As more and more business systems are connected, the target positioning of basic services is becoming clearer and clearer. Its purpose is to provide support for all business lines of auto finance and lay the core position of basic data services.
Therefore, the focus of our team’s next docking work is the APP server application – Simba system, after all, all the functions of the APP are connected to this application. We and simba team chose to connect our application service (CAR-Transfer) in the requirements iteration. Just like that, we made another milestone, from the credit team to the Simba team and then to the payroll team.
After the migration and docking of all business systems, our next goal is to build the brand image of the basic data platform. If the functions involved in relevant requirements can serve multiple business systems, we are willing to do the functions on the basic data platform, and it is necessary. With its reputation spreading across teams, we achieved this goal.
Seven, set the universe
As more relevant business systems of auto Finance r&d team are connected to the basic data platform, we gradually migrate other related interface services from the old system to enrich its functions and expand its strength.
We have built a basic data platform from scratch, and the core strategy is to keep the characteristics of our financial product team and ensure that our team is always young and shining through continuous self-innovation. Although the boss wanted our team to hand over this basic data service later, as A witness of its birth and gradual growth, I was naturally reluctant to give it up, let alone others. It is just like the child I raised. Blood dissolves in water. I hope that under the leadership of the team, we can keep the glorious image of basic data service forever.
Later, after the dewarehousing of financial products and GPS audit system was successively implemented, we also split the relevant business tables of the old system on which basic data services depended into a new database (CAR_transfer), and finally achieved a perfect transformation.
In order to better promote the brand image, it was necessary to use a high profile name. We brought the financial product platform and presented it to the boss. We gave two names, taichu (basic data platform) and Taiji (financial product platform). Then we asked our designer to design the LOGO, and finally became the Buddha. 
The above LOGO is designed by master Baipeng (류패븡), special thanks.
Eight, the tail
The basic data platform is born for the car finance business and constantly innovates for the development of the car finance business. The greatest value of a system application is that it can serve more business systems and provide a “superhuman” capability.
I believe that the essence of technical architecture lies in the research and design of practical business development for the company, and the essence of reconstruction is to further improve the scalability and scalability of the system to make continuous efforts to try.
The success of the story and journey told behind this article could not have been possible without the support of the team behind it and the rest of the boss’s team, for which I would like to thank you.
In order not to reveal their names, we chose to mark them in Korean to remember their strong support.
Thanks Everybody
| Team | List |
|---|---|
| Finance Product Team | 반강강 상영욱 조 commonly used to wish for future happiness 해토 |
| Receive Order Team | Breaking down 정짐 월 |
| Audit Team | 롱석서 강 평 taitae 효 효 효 명 효 롱 |
| Contract Team | 장 서 교 |
| Post Loan Team | Taitae 해회 강팅팅 |
| Simba Team | 덩효롱 양흐욱 양 덩효롱 양흐욱 양 |
| Order Center Team | 엄방새 량 조 량 조 량 조 |
| Salary Team | 레레 란용 transliteration |
| UE Team | 잔 민 류패븡 |
| Operation Team | 류효환 강임민 |
| FE Team | 상 y 립 우 립 우 명 |
| Product Team | 서군임 우효빈 |
| Other | Kevin 조 엔 |