Cloud + Introduction: On April 22, The Tencent Cloud + Community technology salon Kafka Meetup shenzhen station hosted by Tencent Cloud and Kafka community and co-organized by Open Source China was held in Tencent Building. The whole event was full of goods, not only Kafka’s past life and outlook, but also in-depth analysis of Kafka’s technical architecture. At the same time, kafka-based application platform related development and operation and maintenance practice. The activity was extremely popular, with nearly 300 participants participating in the exchange. Many participants said that the quality of the activity was very high and the trip was worthwhile.

The scene audience

Apache Kafka’s past, present, and future

LinkedIn started working on Kafka in 2010, and by 2011 Kafka was in the Apache incubator, becoming one of the world’s top open source projects and quickly being adopted by various enterprises. Kafka is currently used by more than 70% of fortune 500 companies. Rao Jun, principal author, project committee chair and Confluent co-founder of Kafka, first shared Apache Kafka’s past, present and future, introducing the key points and prospects of Kafka along the way. “Kafka is going to be more than just a real-time data collection and transmission system,” he says. “It’s going to be more of a real-time data stream processing, exchange and sharing platform.”

Rao Jun: Apache Kafka’s past, present and future

Kafka cluster automation operation practices

Yang Yuan, senior engineer of Tencent Infrastructure Department, shared “Kafka Cluster Automation Operation Practice”, he introduced Tencent Cloud CKafka system in automatic operation, We need to face the problems of large Kafka cluster failure node removal, new node joining, data skew between nodes, dynamic lifting of Kafka instances required by users and unbalanced utilization of Borker resources, etc., and also give corresponding solutions for Tencent.

Yang Yuan: Kafka cluster automation operation practice

Design and implementation of Zhihu Kafka platform based on Kubernetes

The next speaker, Bai Yuqing, is the head of Zhihu Technology Center. He will bring “Design and Implementation of Kafka Platform Based on Kubernetes in Zhihu”, sharing the application of Kafka in Zhihu, why Kafka platform based on K8S, and how to make such a platform. Zhihu has applied Kafka in logging, data transmission and message queue services. Kafka has become an important component of Zhihu’s technology platform and has been running smoothly for two years, according to Bai Yuqing.

Design and implementation of Zhihu Kafka platform based on Kubernetes

During the tea break, the participants asked questions around the sharing guests. The atmosphere was lively and the technical atmosphere was strong.

Tea to ask questions

Application of CKafka in face recognition PaaS

Back to the second half of sharing, the first is the application of CKafka in Face recognition PaaS brought by Chen Xinyu, the head of research and development of Deep Pupil Cloud data stream of Gling. Spirit deep pupil’s massive picture face recognition service using the optimized tencent cloud CKafka, used in business data import, cache, output, and monitor and log collection and so on each link, connecting the various data processing module, for real time human face feature extraction, comparison and analysis provides the security, is the core of image flow real-time face recognition platform. Chen Xinyu introduced the position and importance of Kafka in deep learning face recognition platform architecture from the application level, and introduced some applications and problems of Kafka ecological components such as Connect.

Chen Xinyu: Application of CKafka in face recognition PaaS

Kafka high performance decryption and optimization

Finally, Yan Yanfei, senior engineer of Tencent Cloud Infrastructure Department, shared “Kafka High Performance Decryption and Optimization”, in-depth analysis of Kafka’s technical architecture, and share the reasons for Kafka’s high performance from storage and message formats, and also put forward some insights on optimizing its performance.

Yan Yanfei: Kafka high performance decryption and optimization

Tencent Cloud + Community Technology Salon Kafka Meetup is an offline technology sharing activity hosted by Tencent Cloud + community and Kafka community. This is the first Kafka Meetup in Shenzhen, co-organized by Open Source China. The event not only expanded the reach of the Kafka open source community, but also prompted developers to think further about Kafka’s application scenarios and future directions.

The scene photos

Part of the audience
The guests had a group photo taken
Some of the staff and volunteers

Field data


Apache Kafka past, present, and future. PDF

Kafka – High performance disclosure and optimization. PDF

Application of Kafka in face recognition PAAS. PDF

Tencent Cloud Kafka automation operation practice.pdf

Design and implementation of Kafka platform based on Kubernetes in Zhihu. PDF