This article has been authorized by the author Liu Chao netease cloud community.
Welcome to visit netease Cloud Community to learn more about Netease’s technical product operation experience.
10. Understand big data platforms based on Hadoop and Spark
As for data architecture, there are actually three processes: Hadoop Map-Reduce 1.0, Map-Reduce 2.0 based on Yarn, and Spark.
The following figure shows the process of Map-Reduce 1.0.
In the map-reduce process, a large Task, split, is called multiple Map tasks, which are distributed to multiple machines for parallel processing and the processing results are saved locally. In the second stage, the Reduce Task copies the intermediate results and processes the results in a centralized manner to obtain the final result.
In Map-Reduce 1.0, there was only one way to run tasks. In order to cope with complex scenarios, task scheduling and resource scheduling were divided into two layers. Resource invocation is performed by Yarn. Yarn finds free resources and allocates them to Yarn as long as it requests them, whether Map or Reduce.
When each task is started, an Application Master is specially started to manage task scheduling and is aware of Map and Reduce. This is Map-Reduce 2.0 as shown below.
Here Yarn is equivalent to the boss of an outsourcing company. All employees are workers and their resources. The boss of the outsourcing company is not clear about every project he or she receives.
The Application Master is equivalent to the project manager of each project, and he knows the specific situation of the project. When executing the project, he needs to apply to the boss of the outsourcing company if he needs employees to work.
Yarn is a universal scheduling platform. If map-Reduce 2 runs, Spark can run.
Spark also creates its own Application Master for scheduling tasks.
Spark is faster, because early planning to do good, not like a Map – Reduce, every task assigned tasks and polymerization to write a hard disk, but the task is divided into multiple stages, all in a Map made the synthesis of a phase, so don’t fall among dish, but to the place where need to merge, and still need to trading.
I wrote the following article on the fundamentals of Hadoop and Spark.
Map-reduce process based on Yarn is commonly described
Popular said that Spark
There are many methodologies for actually writing map-Reduce programs. Here are a few for your reference.
Big data methodology for optimizing map-Reduce processes
Map-reduce algorithm for Web page weight elimination in big data methodology
Map-reduce computation for PageRank in Big data methodology
Nutch’s Map-reduce crawler approach for big data methodology
Search engines based on Lucene and ElasticSearch
After big data is processed, collected data is stored in two places. One is a forward index, which can be stored in Hbase and Cassandra files, and the other is a reverse index, which is easy to search. The data is stored in ElasticSearch based on Lucene.
For Lucene, early in my career, I wrote a Complete Lucene Principles and Code Analysis edition that ran to over 500 pages.
For search engine general principle, wrote the following article.
It doesn’t take technology to understand search engines
Search engine design (1) : dictionary design
Search engine design (2) : inversion list design on
Search engine design (3) : inversion list design under
Understand microservices based on SpringCloud
Finally, application architecture, or microservices.
Here are ten things you should know about microservices architecture design.
Design point 1: Load balancing + API gateway
In the process of implementing microservices, it is inevitable to face the aggregation and disaggregation of services.
When the split of back-end services is relatively frequent, as a mobile App, it often needs a unified entrance to route different requests to different services. No matter how to split and aggregate them later, they are transparent to the mobile end.
With the API gateway, simple data aggregation can be completed at the gateway layer, which does not need to be completed at the App end of the mobile phone, so that the mobile App consumes less power and has better user experience.
With a unified API gateway, authentication and authentication can also be unified, although the calls between services are more complex and there are more interfaces.
The API gateway exposes only required external interfaces and implements unified authentication and authentication on the interfaces. In this way, internal services do not need to be authenticated and authenticated when accessing each other.
With A unified API gateway, you can set certain policies at this layer, do A/B testing, blue-green release, pre-release environment diversion, and so on.
API gateways tend to be stateless and scaleable so that they do not become performance bottlenecks.
Design point two: stateless and independent stateful clusters
Application state is an important factor affecting application migration and horizontal scaling. Stateless services are designed to move this state outward, storing Session data, file data, and structured data in a unified storage at the back end, so that applications contain only business logic.
States are unavoidable, such as ZooKeeper, DB, Cache, etc., and all these stateful things converge into a very centralized cluster.
The whole business is divided into two parts, one is stateless part, one is stateful part.
Stateless parts can achieve two things:
· Arbitrary deployment across machine rooms, also known as mobility.
· Flexible expansion, easy to expand.
Stateful parts, such as ZooKeeper, DB, and Cache, have their own high availability mechanism, and use their own high availability mechanism to implement this state of clustering.
Although stateless, but the current processing of data, or in memory, the current process hangs the data, must also have a part of the loss.
To achieve this, the service needs to have a retry mechanism, the interface needs to have idempotent mechanism, and through the service discovery mechanism, another instance of the back-end service can be called again.
Design point three: horizontal expansion of the database
The database is the state to save, is the most important and most prone to bottlenecks. With a distributed database, the performance of the database increases linearly with the number of nodes.
At the bottom of distributed database is RDS, which is in active/standby mode. Through MySQL kernel development capability, we can achieve zero data loss in active/standby switchover.
So the data falls in this RDS, it is very safe, even if a node is lost, after the switch, your data will not be lost.
On top of that, there is a load balancing NLB with LVS, HAProxy, Keepalived, and a layer of Query Server.
Query Server can scale horizontally based on monitoring data, and if a failure occurs, it can be replaced and fixed at any time, with no awareness of the business layer.
Another is the dual room deployment, DDB developed a data canal NDC component, which enables synchronization between different DDBS in different rooms.
In this case, it is not only distributed in one DATA center, but also has a backup similar to hypermetro in multiple data centers. High availability is very good.
Design point 4: Cache
Caching is important in high concurrency scenarios. Have hierarchical caching so that the data is as close to the user as possible. The closer the data is to the user, the greater the concurrency and the shorter the response time.
There should be a layer of cache on the mobile client App. Not all data should be taken from the back end all the time, but only important, critical and constantly changing data.
Especially for static data, it can be fetched once after a period of time, and there is no need to fetch it from the data center. Data can be cached on the nearest node to the client through CDN for nearby download.
Sometimes there is no CDN, or to go back to the data center to download, called back source, in the outermost layer of the data center, we call access layer, you can set a layer of cache, will be most of the request interception, so as not to cause pressure on the background database.
If the data is dynamic and needs to access the application, it can be generated by the business logic in the application or read from the database. To reduce the pressure on the database, the application can use the local cache or the distributed cache.
Such as Memcached or Redis allows most requests to read from the cache without accessing the database.
Of course, dynamic data can also be statically, or demoted to static data, to reduce the stress on the back end.
Design point 5: Service split and service discovery
When the system is overwhelmed and applications change rapidly, it is often necessary to consider splitting the larger services into a series of smaller services.
The first benefit is that development is more independent. When many people are maintaining the same repository, changes to the code often affect each other.
I often failed the test without changing anything, and when the code is submitted, there will often be conflicts and the code needs to be merged, which greatly reduces the efficiency of development.
Another advantage is the independence of online delivery. The logistics module connects with a new express company and needs to be online together with the order, which is very unreasonable.
I didn’t change it and asked me to restart it, I didn’t change it and asked me to release it, I didn’t change it and asked me to have a meeting, all of which are times that should be split.
In addition, the capacity expansion during the time of high concurrency, only the most critical order and payment process is usually the core. As long as the key transaction link is expanded, if many other services are attached at this time, the capacity expansion is not only uneconomical, but also very risky.
In addition to disaster recovery and degradation, it may be necessary to sacrifice some of the functions of the corners when promoting, but if all the code is coupled together, it is difficult to degrade some of the functions of the corners.
Of course, after the separation, the relationship between applications is more complex, so the mechanism of service discovery is needed to manage the relationship between applications, to achieve automatic repair, automatic correlation, automatic load balancing, automatic fault-tolerant switchover.
Design point 6: Service orchestration and elastic scaling
When services are unbundled, there are so many processes that service choreography is required to manage the dependencies between services and to code the deployment of services, which is often referred to as infrastructure as code.
In this way, services can be published, updated, rolled back, expanded, or scaled down by modifying the orchestration file, increasing traceability, manageability, and automation capabilities.
Since the choreographer file can also be managed with a code repository, it is possible to update five of the 100 services by changing the configuration of five of the services in the choreographer file.
When the choreographer file is submitted, the repository automatically triggers the automatic deployment of upgrade scripts to update the environment online.
When problems are found in the new environment, you want to roll back these five services atomically, and if there are no choreography files, you need to manually record which five services were upgraded this time.
With marshalling files, you can simply Revert to the previous version in the repository. All operations are visible in the code repository.
Design point 7: unified configuration center
After services are split, the number of services is so large that it is difficult to manage if all the configurations are stored in the local application as configuration files.
It can be imagined that when there is a configuration problem in hundreds or thousands of processes, it is difficult to find it out. Therefore, there needs to be a unified configuration center to manage all configurations and deliver unified configurations.
In microservices, configurations tend to fall into the following categories:
· One type is almost invariable configuration, which can be directly typed into the container image.
· The second type is the configuration that is determined at startup, usually through environment variables, when the container is started.
· The third type is unified configuration, which needs to be delivered through the configuration center. For example, if some functions need to be degraded, you can configure the functions that can and cannot be degraded in the configuration file.
Design Point 8: Unified log center
When there are a large number of processes, it is difficult to log in to hundreds of containers one by one to view logs. Therefore, a unified log center is required to collect logs.
In order to make the collected logs easy to analyze, certain requirements are required for the log specification. When all services comply with the uniform log specification, a transaction process can be traced uniformly in the log center.
For example, in the final log search engine, search for the transaction number and you can see where the error or exception occurred in the process.
Design point nine: fusing, limiting current, downgrade
The service should have the ability of fusing, limiting flow, and degrading. When a service calls another service and there is a timeout, it should be returned in time, rather than blocked in that place, thus affecting the transactions of other users. It can return the default base data.
When a service finds that the called service is too busy, the thread pool is full, the connection pool is full, or there are always errors, it should be disconnected in time to prevent the fault or busy of the next service from causing the abnormal of the service and gradually spreading forward, resulting in the avalanche of the whole application.
When it is found that the whole system is really overburdened, you can choose to downgrade some functions or some calls to ensure that the most important transaction flows pass, and the most important resources are devoted to the most core processes.
Another means is current limiting. When both the fusing and degradation strategies are set, the supporting capacity of the whole system should be known through the pressure test of the whole link.
Therefore, a traffic limiting policy needs to be formulated to ensure that the system can provide services within the tested supporting capacity. Services beyond the supporting capacity can be denied.
When you place an order and the system pops up a dialog box saying “system busy, please try again”, it does not mean that the system is down, but it means that the system is working properly, but the traffic limiting policy is working.
Design point 10: Comprehensive monitoring
When the system is very complex, there are two main aspects of unified monitoring, one is health, and one is where the performance bottlenecks are.
When the system is abnormal, the monitoring system can cooperate with the alarm system to discover, inform and intervene in time, so as to ensure the smooth operation of the system
When the pressure test, often encountered bottlenecks, also need to have a comprehensive monitoring to find bottlenecks, while preserving the site, so that traceability and analysis, all-round optimization.
I will write more detailed articles related to micro services.
Service decomposition and service discovery in microservitization
Design of microservitization cache
Statelessness and containerization of microservitization
Microservitization of database design and read and write separation
The access layer of microservices is designed to be isolated from dynamic and static resources
Continuous integration is the cornerstone of microservitization
The following articles have been written about the combination of microservices and containers.
Why is Kubernetes a natural fit for microservices
Different ways to play Kubernetes in different stages of microservitization
Financial innovation business is based on microservitization practice of container cloud
Dive into the technical details behind Service Mesh
Read the data side Envoy of the Service Mesh in depth
At last.
Liu Chao director of Netease Cloud Technology Architecture Department
Long-term commitment to cloud computing open source technology sharing, preaching and implementation, netease internal best practices to serve customers and the industry.
Technology sharing: Published “Lucene Application Development and Decryption”, geek Time column “Interesting Talk about Network Protocol”, personal public account “Liu Chao’s Popular Cloud Computing” article Kubernetes and micro services series 18, Mesos series 30, KVM series 25, Openvswitch series 31, There are 24 OpenStack series and 10 Hadoop series. The article “finally someone put cloud computing, big data, artificial intelligence to understand” accumulated 100,000 +
Conference sermon: Star lecturer of InfoQ Architect Summit, as invited lecturer to share netease’s best practices at more than 10 large technology summits including QCon, LC3, SACC, GIAC, CEUC, SoftCon and NJSD
Industry landing: Netease’s container and micro-service products will be launched in banking, securities, logistics, video surveillance, intelligent manufacturing and other industries.
Related Reading: Advanced Cloud Architect Walkthrough (2)
Advanced Cloud Architect Guide (1)
The basic cloud computing services of netease deeply integrate IaaS, PaaS and container technologies, provide elastic computing, DevOps tool chain and micro-service infrastructure services, help enterprises solve IT, architecture, operation and maintenance problems, and make enterprises more focused on business. As a new-generation cloud computing platform, you can click to try IT for free.
【 recommended 】 use of the responsibility chain pattern -Netty ChannelPipeline and Mina IoFilterChain analysis