The top 50 list of domestic big data companies in 2018 has been published, and Big will soon break into the top 50 as a dark horse and win a number of titles. Hanlp natural language processing technology also won the “2018 China Data Star Technology” award. Those who are interested in this list should check it out. Following the previous article “DKM Platform Monitoring Parameters Description”, this paper continues to introduce and explain the platform monitoring parameters in the big data integration processing architecture. DKhadoop big data processing platform architecture installation related articles have been shared, detailed content can be found. In the previous section, monitoring parameters such as cluster load average, cluster disk usage, HDFS monitoring page, and Hbase monitoring page have been described. Today, the rest of the monitoring parameters are introduced together, and the introduction of the monitoring parameters of the big and fast big data processing platform is complete.
1. Yarn Monitoring interface (1) Failed application
(3) Running applications
Monitoring the number of applications running in YARN resource management The vertical axis indicates the number of applications, and the horizontal axis indicates the time, in minutes. (4) Applications waiting to be executed
Monitoring the number of applications waiting to be executed in YARN resource management The vertical axis indicates the number of applications, and the horizontal axis indicates the time, in minutes
(5) Completed applications
Monitoring the number of completed applications in YARN resource management The vertical axis indicates the number of applications, and the horizontal axis indicates the time (unit: minute) (6) Killed applications
Monitor the number of applications that are killed in YARN resource management. The vertical axis indicates the number of applications, and the horizontal axis indicates the time, in minutes. 2
Monitor the maximum available memory capacity in the Spark cluster. The vertical axis indicates the memory capacity, in MB. The horizontal axis indicates the time, in minute. (2) Used memory
Monitor the remaining memory capacity in the Spark cluster. The vertical axis indicates the memory capacity and the unit is MB. The horizontal axis indicates the time and the unit is minute
Monitor the number of stages that are running in the Spark cluster. The vertical axis indicates the number of stages, and the horizontal axis indicates the time (unit: minute)
(7) The number of failed stages
Monitor the number of stages that are ready in the Spark cluster. The vertical axis indicates the number of stages, and the horizontal axis indicates time, and the unit is minute. Note: (The time on the horizontal axis indicates the collection time, which is usually once a minute.)