This article is a summary of the microservices and cloud Native https://github.com/wx-chevalier/Backend-Series series, with related resources declared in the Awesome Serverless List.

Serverless💖 node. js Puppeteer penetration test crawler practice

As defined by CNCF, Serverless is the concept of building and running applications that do not require server administration; Serverless is an Internet-based system in which applications are developed without the use of regular service processes. Instead, they rely only on a combination of third-party services (such as AWS Lambda services), client-side logic, and service-hosted remote procedure calls.

The main forms of Serverless are BaaS and FaaS. BaaS refers to back-end cloud services, such as cloud databases, object storage, and message queues. With BaaS, we can greatly simplify our application development. FaaS functions are services, which are custom code running in a staging container, and functions are the smallest unit of abstract language runtimes in a serverless architecture. In FaaS, the user pays primarily for the running time of the function, without having to worry about CPU or RAM or any other resources and their operational and maintenance burdens.

Referring to the definition of BaaS and FaaS, we can know that Serverless has the following main features:

  • Event-driven: Functions in FaaS require a series of events to drive function execution.

  • Stateless: Because each function execution may use a different container, there is no memory or data sharing. If you want to share data, you can only do so through third-party services such as Redis.

  • No O&M: With Serverless, we do not need to care about servers and O&M. This is also at the heart of Serverless’s thinking.

  • Billing by Actual Use: Using Serverless is cheap because we only pay for each function run. If the function does not run, it costs nothing and does not waste server resources.

The essence of these features is the most direct way for users to speak about the development of cloud applications, and even the user-friendliness and low mental burden of so-called cloud native applications. This expectation of simplicity, economy and reliability is also the original intention of cloud computing. Of course, Serverless is not limited to Function, but should coexist with multiple deployment modes. For example, application deployment follows the principle of single responsibility, but can trigger multiple events. It can also be deployed at the container level and can include any language, any runtime, such as solutions like Knative. We also discussed more Serverless landing patterns in the section microservices vs. Cloud Native /Serverless.

Front-end perspective Serverless

In the foreword to the https://github.com/wx-chevalier/Web-Series series on fundamentals of Modern Web Development and Engineering Practices, we also discuss Web technology and ecology through template rendering, front-end and back-end separation and single-page applications, engineering and micro-front-end, Big front-end and Serverless these three different stages. Naturally, from the perspective of the front end, Serverless also gives more freedom and possibility to the front end. There is a lot of room for server-side rendering, simple server-side support for small program development, including BFF interface aggregation and so on.

BFF not only solves the problem of interface coordination and aggregation, but also bears the concurrent pressure of the original backend, which also brings a lot of development and operation pressure to the front-end engineer. Serverless mitigated this problem by using functions to implement aggregate clipping of interfaces; The front-end request for BFF is translated into the triggering of HTTP triggers for FaaS, and the business logic for the request is implemented in this function, such as calling multiple microservices to get data and then returning the processing results to the front-end. In this way, the pressure of operation and maintenance is shifted from the former BFF Server to FaaS service, and the front end no longer needs to care about the Server. Serverless can also be applied to server-side rendering, breaking up each route into functions and deploying the corresponding functions on FaaS. Thus, the path requested by the user corresponds to each individual function. In this way, the operation and maintenance operations are transferred to the FaaS platform, and the front-end server rendering is done, so there is no need to care about the operation and maintenance deployment of the server program. In addition, small programs such as wechat and Alipay also provide a cloud development platform in line with the Serverless concept, enabling rapid iteration of the front end of the business.

Fun💖 Cendertron of Aliyun

The Serverless service of Ali Cloud provides us with functional computing service. Functional computing is an event-driven service, through which users do not need to manage the running conditions of servers, but only need to write codes and upload. Function computing prepares computing resources and runs user code on an elastic scale, with users paying only for the resources consumed by the actual code running.

Fun is a tool designed to support Serverless application deployment, helping you easily manage functions, API gateways, logging services, and more. It helps you develop, build, and deploy through a resource configuration file (template.yml).

Cendertron https://github.com/wx-chevalier/xe-crawler is a dynamic crawler based on Puppeteer that focuses on pre-interface extraction for penetration testing, with built-in features such as action emulation, URL de-repetition, interface forms and request extraction.

In the standard Docker deployment mode, we need to package it into the Docker container:

# build image
$ docker build -t cendertron .

# run as contaner
$ docker run -it --rm -p 3000:3000 --name cendertron-instance cendertron

# run as container, fix with Jessie Frazelle seccomp profile for Chrome.
$ wget https://raw.githubusercontent.com/jfrazelle/dotfiles/master/etc/docker/seccomp/chrome.json -O ~/chrome.json
$ docker run -it -p 3000:3000 --security-opt seccomp=$HOME/chrome.json --name cendertron-instance cendertron

# or
$ docker run -it -p 3000:3000 --cap-add=SYS_ADMIN --name cendertron-instance cendertron

# use network and mapping logs
$ docker run -d -p 5000:3000 --cap-add=SYS_ADMIN --name cendertron-instance --network wsat-network cendertron
Copy the code

Now, using the functional computing of Ali Cloud, we can convert it to FC mode deployment. For example, see Cendertron-FC:

# Install fun cli
$ brew tap vangie/formula
$ brew install fun
$ brew install fcli

# Config fun cli
$ fun config

# Install dependences
$ fun instll

# If you want to dynamically pull Puppeteer remotely at runtime, In the https://fc-demo-public.oss-cn-hangzhou.aliyuncs.com/fun/examples/headless_shell.tar.gz download headless_shell. Tar. Gz, Or build with scripts/ buildchrome.sh; Then upload it to oss://${BUCKET}/headless_shell.tar.gz

# Deployment

Deploy with fun
$ fun deploy

# Or manually package local upload, sync package local Chrome to remote
$ npm run package
# Do not pack Chrome, dynamically pull from remote
$ npm run package-nochrome
Publish Function in the corresponding Service directory
$ fcli shell
>>> mkf/upf cendertron-fc/crawl -h index.handler -f package.zip -t nodejs8

: << !
using region: cn-beijing
using accountId: ***********6367
using accessKeyId: ***********5Kd5
using timeout: 10

Waiting for service cendertron-fc to be deployed...
        Waiting for function crawl to be deployed...
                Waiting for packaging function crawl code...
                package function crawl code done
        function crawl deploy success
service cendertron-fc deploy success
!

# Test crawl
# Invoke remote
$ fcli function invoke -s cendertron-fc -f crawl
$ fcli function invoke -s cendertron-fc -f crawl --event-str 'http://www.baidu.com'

# Local debug
$ fun local invoke crawl <<<'http://www.baidu.com'
Copy the code