The author | yuan yi Ali cloud intelligent enterprise group of senior development engineer

Kafka event source support is now available in Knative, so how to implement message push based on Kafka? The author of this article will use Ali Cloud Kafka product as an example to unlock this new pose.

background

Message queue for Apache Kafka is a distributed, high-throughput, scalable message queue service provided by Ali Cloud. Message queue for Apache Kafka is widely used in big data fields such as log collection, monitoring data aggregation, streaming data processing, online and offline analysis, and has become an integral part of the big data ecosystem.

With the support of KafkaSource event source in Knative, you can easily connect to Kafka message service.

You can also install community Kafka 2.0.0 or later.

Create Kafka instance on Aliyun

Create a Kafka instance

Log in to the Message queue Kafka console and select Buy Instance. The current Kafka event source in Knative supports version 2.0.0 or higher. To create Kafka instances on Ali Cloud, you need to purchase the professional version of the Kafka event source. After purchase, you can upgrade to version 2.0.0.

Deploy the instance and bind it to a VPC

After purchase, deploy the Knative cluster. During deployment, set the VPC where the Knative cluster resides:

Create Topic and Consumer groups

Next we create topics and consumer groups. Enter Topic Management and click Create Topic. Here we create a Topic named Demo:

Enter “Consumer Group Management” and click “Create Consumer Group”. Here we create a Consumer Group named demo-consumer:

Deploy the Kafka data source

Deploy the Kafka Addon component

Log in to the Container Service Console, enter Knative Component Management, and deploy the Kafka Addon component.

Create a KafkaSource instance

Start by creating the event-display service to receive events:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: event-display
spec:
  template:
    spec:
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/knative-sample/eventing-sources-cmd-event_display:bf45b3eb1e7fc4cb63d6a5a6416cf696295484a7662e0cf9ccdf5c080542c21d
Copy the code

Next create KafkaSource:

apiVersion: sources.eventing.knative.dev/v1alpha1
kind: KafkaSource
metadata:
  name: alikafka-source
spec:
  consumerGroup: demo-consumer
  # Broker URL. Replace this with the URLs for your kafka cluster,
  # which is in the format of my-cluster-kafka-bootstrap.my-kafka-namespace:9092.BootstrapServers: 192.168.0.6 x: 9092192168:0.7 x 9092192168 x 0.8:9092 switchable viewer: demo sink: apiVersion: serving.knative.dev/v1alpha1 kind: Service name: event-displayCopy the code

Description:

  • BootstrapServers: Kafka VPC access address
  • ConsumerGroup: sets the consumerGroup
  • Topics: Set Topic

Once created, we can check that the corresponding instance is running:

[root@iZ2zeae8wzyq0ypgjowzq2Z ~]# kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
alikafka-source-k22vz-db44cc7f8-879pj   1/1     Running   0          8h
Copy the code

validation

In the Kafka console, select Topic to send the message, and note that the message format must be JSON:

We can see that we have received the sent Kafka message:

[root@iZ2zeae8wzyq0ypgjowzq2Z ~]# kubectl logs event-display-zl6m5-deployment-6bf9596b4f-8psx4 user-container☁️ CloudEvent: Valid ✅ Context Attributes, SpecVersion: 0.2 Type: dev.knative. Kafka. Event Source: /apis/v1/namespaces/default/kafkasources/alikafka-source#demo
  ID: partition:7/offset:1
  Time: 2019-10-18T08:50:32.492Z
  ContentType: application/json
  Extensions: 
    key: demo
Transport Context,
  URI: /
  Host: event-display.default.svc.cluster.local
  Method: POST
Data,
  {
    "key": "test"
  }
Copy the code

summary

Combine ali Cloud Kafka product, through the event driven trigger service (function) execution, is not simple and efficient. In this way, we can fully unleash the power of cloud native with Knative, bringing us more space for imagination. Welcome to communicate with those who are interested in Knative.

Hands-on video demonstration link: v.qq.com/x/page/d301…

About the author: Yuan Yi, senior development engineer of Ali Cloud Container platform, responsible for the work related to Ali Cloud container platform Knative.

Welcome to Knative Communication Group

“Alibaba cloudnative wechat public account (ID: Alicloudnative) focuses on micro Service, Serverless, container, Service Mesh and other technical fields, focuses on cloudnative popular technology trends, large-scale implementation of cloudnative practice, and becomes the technical public account that most understands cloudnative developers.”