Golang greatly reduces the threshold of service development and makes it easier for enterprises to directly structure businesses due to its significant advantages of high efficiency, speed and natural support for concurrency. These characteristics make more and more traditional technology departments choose to embrace Go and explore rich practical experience based on their own business characteristics. In Gopher China2017, which was co-hosted by Go China and Seven Niuyun, Zhang Zewu, technical manager from Master Jin, shared a detailed discussion about developing a complete security, futures, quotation processing and distribution system using Go. The background of the project is that the team members are not fully in place, after using the easy-to-use Go language, in three months to complete the development. And in the subsequent development of the project, this system gradually evolved into a set of its own rapid development framework.

The project story

When the project was started, the leader had several requirements for the market system: how to deliver in the shortest time, how to meet a large number of concurrent requests, how to ensure low delay, and how to send the market data of the exchange to the client as soon as possible. After these basic requirements are met, data services such as time-sharing line, K line, pen by pen and index will be developed.

With these steps sorted out, we begin to consider the pressing issues we are facing today, including the following:

1. How to access the data source? Directly access the data of the exchange or from the secondary platform?

2, how to improve the market rate, hoping to reduce the flow of market data in the server as much as possible.

3. Computing services of time-sharing, K line and index data should be considered.

4. Set the number of access service nodes to 10,000.

5. Nodes are delivered before eleven.

Next is building the team. There are several problems with building a team. First of all, I would like to choose C++ as the development language for the team. Since I have a long experience in C++, choosing C++ at the initial stage of the project can make it quicker to start. Second, it is best to recruit the development of industry experience, securities, futures industry experience, know how business is going to return a responsibility. Three is to have the experience of the service end, because market system is a service. What if I can’t recruit anyone who meets the above conditions all the time? In the end, I thought, even those who have traded stocks can do it, and at least know some business knowledge. In the recruitment process, every time I found a person with Go development experience, I was very happy. These are some of the early experiences of the team.

Our first engineer arrived on June 9th, and our fourth partner joined us on August 10th. In such a succession to the post, but also before 11 to complete the delivery, pressure can think and know. So when I joined two C++ partners, I tried to use Go to develop this market system. We did some basic service development in July, such as access service, computing service, data source collection and receiving the market data of gold exchange. By August, these basic services will be integrated and debugged. We did a test launch in September, and although we still had some minor issues, we went live and initially achieved our goal.

To sum up, it is because we choose Go that we can deliver. At that time, the boss was worried that it would be difficult to recruit people after using Go. But with C++ a few people can’t do it in 3 months. There was another important reason I chose to Go, because I had just attended my first Gopher conference and I was so inspired and confident that I stuck with it. A special thanks to Asta.

The market system

What is the market? It is an instant quote, and the instant quote becomes a historical quote after this point in time has passed. Historical quotation will have some statistics, processing needs. For example, let’s say I’m buying potatoes. This is a user requirement. I asked the boss how much a kilo of potatoes, this is my inquiry behavior, the boss answered 3 yuan a kilo, this is the price reported by the boss. When the price changes, the boss informs me, this is the quotation service. There are a lot of people ask the boss at the same time, how much is the price of potatoes and vegetables, at this time it is a concurrent inquiry behavior, the boss directly reply has been unbearable, put your way of communication to stay, when the price changes to inform you, this is the market subscription service.

The stock market mainly comes from the data generated after trading on the exchange. The data we usually get is the real-time data of the real-time transaction. After the transaction data is processed into other data, this is what the market system to do.

What are the requirements of the market system? We must be quick, accurate and steady. Fast is the market to respond fast, low delay, if slow, user value will decline, and will cause problems. Accuracy refers to the data can not have deviation, sent to the data down when the processing should be accurate. There is also the possibility that the data sent over is incorrect, the data of the data service provider has problems, the data quality is not qualified, and the fault tolerance processing should be done. Foreign holidays will also have an impact on the data, which is to be taken into account by the market system to filter out garbage data. Stable means that the service should be stable and available at least 99.99%.

Talk about the characteristics of market system design requirements. The concurrency requirement is that the system should be able to scale flexibly when the user size changes. When the data volume of the project was not very large in the initial stage, the concurrency pressure was not very large. With the promotion of the project, the concurrency pressure will come. Fault tolerance requirement refers to once a fault occurs, the market system can control the scope of influence of the fault in the local, does not affect the overall service. The failure of one service cannot cause the failure of the entire service. Fault response requirements refer to the ability to recover quickly once a fault occurs and cannot be controlled locally. To some extent, this requirement can be solved by means of operation and maintenance. If support is provided in the early architecture design, the later operation and maintenance pressure will be much less.

The system service design is introduced below. The first part is access service, which is mainly client-oriented to meet the requirements of high concurrency and high online. The second part of computing services, mainly processing market data, get market data for calculation, calculation after storage to persist, according to the characteristics of the data classification management. The third part is the collection service. We are connected with a lot of exchanges, which provide different interfaces and organize data in different ways, and the protocols are also very different. These differences are handled in the collection service. We set up an internal data protocol, and in the collection process, the external protocol is converted into internal protocol.

    

The above general service made some classification, a lot of clarity. Next is some development work. In order for developers to get started quickly, they need to focus on their own work and lower the barriers to development. So I put the development to use the library, made some abstraction, classification, the same content and business to do some of the library. There are eight categories. The first one, main.go, takes the general initialization actions, including monitoring, statistics, external commands, some parameters, exit mechanism, and makes some conventions. When you want to develop a new feature, you can take main.go and do the business logic directly under its framework without caring about the surrounding content.

When a service is running, it needs some control, and the command framework does this. When we are dealing with market calculation, we receive market data is real-time market, and we need to calculate the time-sharing line, minute K line, day K, week K, month K and other data in parallel. The scheduling function in the middle has also been library. The configuration library means that many service attributes are the same. After the loading process and configuration description are determined, a configuration library is formed. Other developers can refer to this and quickly add the configuration they need. There is also a tool library for basic tools.

Having such a framework makes it very easy for new colleagues to get started.

Next comes the business-specific base business library. After assembling from the previous common libraries, you can do some basic business development, as well as library. When you do this, you decouple to a certain extent, abstracting objects and components, and then servitizing those objects and components. Later, if there are more services, you can solve the problem of service management. Such a design is expected to make it easier for the team to focus on business delivery in the development process, simplifying the development process and further lowering the development threshold.

We made some basic business libraries, such as protocol library, request protocol, time-sharing line, K line and other data protocols, and put the whole set of protocols in a database. For example, the trading day processing problems in futures and spot markets are prominent. The gold Exchange will open at 20:00 on Friday night and close at 15:30 next Monday. A trading day will span three days and there are holidays.

Sometimes the market source will be a problem, multi-way competition, one is a road market source failure, will not lead to market interruption, two is under competition, can ensure the fastest estrus, this is some of the best processing. Time – sharing, K – line, pen – by – pen, price – dividing algorithm and index algorithm are library – based. At the beginning, we compiled the Go algorithm library and put it into the C++ client because it was too late to use C++ to develop the index algorithm on the PC client. It only took one week.

Compression algorithm library is a compression library according to the characteristics of market data. For in-memory data caching done in access services, there are some cache policies, encapsulated as cache policy libraries. Other business libraries can be added quickly in later business cycles.

    

Access to the service

The access service supports the status removal of back-end development, fault recovery, and load balancing. Finally, I’ll show you some test data for the access service. Access services mainly provide stable, timely and high-quality services for clients. The access service has four objectives. First, it should be stable for a long time and basically no change will occur after it goes online. The second is to ensure timely service response, the back-end service sent, it can quickly forward. Third, when the server is abnormal, so that the client no sense. Fourth, it can be flexible up and down the line, the back end can add services online, also can choose to remove a service directly.

Under the above objectives, several key points are collated. One is to decouple the business, independent of the specific business. Provide service registration function, back-end business service program can register with access service, any business can register with access service. Multiple service programs of the same service can be registered at the same time, and multicast policy and multi-policy load balancing service can be provided. To provide a business routing service, the business in the request protocol is routed to the service provider. Provides a business state hosting service that can store requests directly to the state service in context while the back-end service is being developed. The access service does not care what the context content is, but it provides a hosting service. Supports online extension of routing policies, multicast policies, and load balancing policies.

Access service in the process of implementation, divided into six modules, one is the network transceiver module, which is also available, all the services can use the network module. There’s a service scheduling, scheduling all of these things, like routing, service management, state hosting, command processing, do some scheduling of them. The forwarding service in the access service is the most basic service. When a client initiates a request to the access service, the request is directly forwarded to the back-end service and then sent back to the client.

The service status of the access service refers to the back-end service, which can be stateless. When the client initiates a request, the state context is an empty state. This empty state and the request are forwarded to the backend service. When the backend service returns, the state of result 1 is saved in the context state and then forwarded to the client. As the back-end service continues to push subsequent results, the result status is constantly updated into the context. In this way, the backend quote service can remove the state and not keep the state in the service.

Fault recovery in access services. If our current services are normal and one of the nodes suddenly fails, when access services forward requests to it, the faulty server is unable to deal with it. In this case, the scheduling module of the access service will re-route the request to the status N, and bring the status N to the normal service node, and the normal service will return the correct result. When you push new results later, the status is n+1. The faulty node can be removed at any time.

Load balancing and dynamic online in access services. If you have N services on the back end, and it’s in my service management list, the service management list it’s going to put this information in the routing table. When a new service is added to the backend, it registers with the service list. The service management list will put this information into the routing table, and the existing service will not be affected. The client can send requests for new services directly.

The control of routing policy is supported by command module. The client sends the configured policy to the command module to control routing table and service list.

The test part

Here’s the data from our test. The test scenarios are divided into connection number test, module ability test, receiving and sending test, a small number of clients and multiple clients, as well as complete business test, receiving, sending and replying, delayed data and internal queue data. The packet we use is 200 bytes in size and runs over a 1-gigabyte network.

The following is an example, testing the client, which uses a dynamic quote to put the business ID into the protocol, and then packets it to the server. The intermediate process will access the server and route the delivery service. Server-side development is very simple, use to access services SDK, service object, service object access registered service address to us, provide their own service name, server and he told us the service which is to solve business, for which a business providing services, business development can be processed directly on the Client, Take out the data for further development.

The receiving capacity is 500,000 packets per second, reaching the upper limit of the network card. It started with Go1.4, but after the 1.6 upgrade, the CPU dropped quite a bit.

The forwarding test has back-end services. The first data is 250,000 packets/second due to the limited server resources of the testing scene. After the subsequent server is upgraded, it can reach 500,000 packets/second.

Multi-client test, respectively in the case of 5000, 10000, 20000,25000 clients, each client sends 10 packets, 20 packets, 25 packets per second, up to 250,000 packets/second, which is an 8-core CPU speed.

        

The request response test, which is a complete business process, can receive and receive 450 megabits per second, plus 900 megabits per second, up to the line.

    

Data delay test, also made a statistics, every 100,000 packets do statistics, no obvious delay.

    

Queues, ranging from 1, 48, 400 to 800, were tested. As the number of queues increased, CPU consumption decreased and memory consumption increased. When queues are small, it’s a bottleneck.

This access service has a better support for the rapid development of later business. This technical scheme can be further discussed if you are interested in it later.

—————————————-

Beijing on August 19

Qiuniuyun and Lianjia jointly launched the architect Practice Day with the theme of “Latest Scenario-based Application practice of Big Data”

Produced jointly by Chen Chao, technical director of Qiuniuyun, and Lv Yi, head of the big data Department of Lianjia, selected dry goods:

  • Deng Fangyuan, senior Big data R&D engineer of Lianjia, shared how to use “open source + own business customization” to solve the pain points of multi-dimensional big data analysis

  • Dang Hexuan, senior engineer of Big data of Seven Niuyun, fully revealed the practice of seven Niuyun’s independent research and development of 100 billion big data platform

  • Wang Muling, head of Mafengwo’s big data platform, brought mafengwo, as one of the earliest companies to use Druid in China, the “pits” and optimization experience in the actual combat process

  • Wei Xiangjun, former Sina Weibo technical expert and CTO of Sanhao.com, explained how to use big data technology solutions to add value in practical business applications

Go Chinese Fans benefits:

Click “Read the original” to enjoy free registration

Provide a wonderful message to the most praised three readers can get seven cattle yun hand do a set!