Because the VPC Flow Log is not real-time, after the VPC Flow Log captures the data during the aggregation interval, additional time is needed to process the data and publish it to CloudWatch Logs or Amazon S3. This additional time is about 5 minutes for publishing to CloudWatch Logs and 10 minutes for publishing to Amazon S3. The stream logging service is provided to the maximum extent possible during this additional time. In some cases, your log may be delayed more than the additional 5 to 10 minutes mentioned above, which cannot meet the real-time requirements of some customers for traffic analysis. Therefore, customers will ask us if we support some third-party traffic analysis system to monitor and analyze traffic in real time.

On February 23, 2021, Amazon Cloud China announced the landing of VPC Traffic Mirroring function [1]. With this function, customers’ real-time demand for Traffic analysis can be solved by combining some third-party Traffic analysis systems. In document [2], an example of traffic analysis combined with Suricata is listed, which is tested and extended in this document.

According to the following test steps in the document [3], only the mirror message of VXLAN can be seen in the log, and the contents of the message (such as HTTP request) cannot be recorded and analyzed: Step 1: Install the Suricata software on the EC2 instance target Step 2: Create a traffic mirror target Step 3: Create a traffic mirror filter Step 4: Create a traffic mirror session

According to the following configuration and tests, the client’s HTTP request was successfully logged.

  1. Use the following command to update Suricata Rule sudo Suricata-Update
  2. HTTP $HOME_NET any -> $EXTERNAL_NET any = “HTTP $HOME_NET any -> $EXTERNAL_NET any” = “HTTP $HOME_NET any -> $EXTERNAL_NET any”; $HOME_NET and $EXTERNAL_NET is in Suricata configuration file/etc/Suricata/Suricata yaml do definition, so change the configuration file, set the $HOME_NET and $EXTERNAL_NET are any, after the configuration is as follows: HOME_NET: “any” EXTERNAL_NET: “any”
  3. By Traffic Mirror Mirror of message is Vxlan encapsulation, thus requiring Suricata message for Vxlan decapsulation, change the configuration file/etc/Suricata/Suricata yaml make its message for Vxlan decapsulation, configuration is as follows: decoder: vxlan: enabled: true ports: $VXLAN_PORTS # syntax: ‘8472, 4789’
  4. Change the configuration file/etc/suricata/suricata yaml open HTTP Log, after the configuration is as follows:- http-log: enabled: yes filename: http.log append: yes
  5. Restart the Suricata process to take effect (kill the current Suricata process first, delete rm rf /var/run/ file, delete rm rf /var/run/ file, delete rm rf /var/run/ file, delete rm rf /var/run/ file) Then run the command again “suricata – c/etc/suricata/suricata yaml -k none -i eth0 – D” launch process), restart the process does not delete before logging have been generated.
  6. Test, in the mirror the flow of traffic source device access curl on the image of the target device (installed the Suricata equipment) on the view/var/log/Suricata/HTTP log, you can see the request has been successfully recorded: 03/15/2021-07:04:00. 347144 [] / [] curl / 7.61.1 HTTP / 1.1 * * GET [] [] [] 200 2381 bytes [] – >

Additional Notes:

1.Suricata message processing flow chart

2. Suricate feature detection was used to identify abnormal flow, the feature database are stored in the file/var/lib/suricata/rules/suricata rules

3. For more information about Suricata, please refer to [4].

Reference documents: