Author: KubeVela Community
At present, KubeVela has supported AWS, Azure, GCP, Ali Cloud, Tencent Cloud, Baidu Cloud, UCloud and other cloud manufacturers. It also provides simple and fast command line tools [1] to introduce cloud resources of cloud service providers. However, supporting cloud resources from cloud service providers one by one in KubeVela is not a good way to quickly meet users’ demand for cloud resources. This article provides a solution to quickly introduce the top 50 most popular AWS cloud resources in less than 100 lines of code.
We also expect users to be inspired by this article and contribute cloud resources to other cloud services.
Where are AWS’s most popular cloud resources?
Terraform’s official website provides Terraform Modules of various cloud service providers, such as AWS cloud resource Terraform Modules [2]. Cloud resources are ranked by popular usage (downloads), such as 18.7 million AWS VPC downloads.
Through the simple analysis, we found that 50 years before the AWS Terraform modules can request the registry data. Terraform. IO/v2 / modules? … To obtain.
Prior to the start
The code accepts two user-passed parameters: • the name of the provider • the URL of the Terraform Modules to which the provider corresponds
For AWS, the Provider name is “AWS”, The corresponding Terraform Modules is the Terraform Modules JSON format interface [3] (that is, the 50 most popular cloud resources when the provider is AWS is searched in Terraform Registry[4]). You need to verify that the providerName(AWS) and Modules links are correct before executing the code.
Execute the code
You can quickly import the top 50 AWS cloud resources in bulk with the following 100 or so lines of code (named Gen.go).
import (
"encoding/json"
"fmt"
"io"
"log"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
"github.com/pkg/errors"
)
type TFDownload struct {
Data []DataItem `json:"data"`
Included []IncludedItem `json:"included"`
}
type IncludedItem struct {
Id string `json:"id"`
Attributes Attributes `json:"attributes"`
}
type DataItem struct {
Attributes Attributes `json:"attributes"`
Relationships Relationships `json:"relationships"`
}
type Relationships struct {
LatestVersion RelationshipLatestVersion `json:"latest-version"`
}
type RelationshipLatestVersion struct {
Data RelationshipData `json:"data"`
}
type RelationshipData struct {
Id string `json:"id"`}var errNoVariables = errors.New("failed to find main.tf or variables.tf in Terraform configurations")
type Attributes struct {
Name string `json:"name"`
Downloads int `json:"downloads"`
Source string `json:"source"`
Description string `json:"description"`
Verified bool `json:"verified"`}func main() {
if len(os.Args) < 2 {
fmt.Println("Please provide the cloud provider name and an official Terraform modules URL")
os.Exit(1)
}
providerName := os.Args[1]
terraformModulesUrl := os.Args[2]
resp, err := http.Get(terraformModulesUrl)
iferr ! = nil { log.Fatal(err) } defer resp.Body.Close() body, err := io.ReadAll(resp.Body)iferr ! = nil { log.Fatal(err) }var modules TFDownload
iferr := json.Unmarshal(body, &modules); err ! = nil { fmt.Println(err.Error()) os.Exit(1)}if _, err = os.Stat(providerName); err == nil {
iferr := os.RemoveAll(providerName); err ! = nil { log.Fatal(err) } fmt.Printf("Successfully deleted existed directory %s\n", providerName)
}
if _, err = os.Stat(providerName); os.IsNotExist(err) {
if err := os.Mkdir(providerName, 0755); err ! = nil {if! os.IsExist(err) { log.Fatal(err) } fmt.Printf("Successfully created directory %s\n", providerName)
}
}
for _, module := range modules.Data {
var description string
for _, attr := range modules.Included {
if module.Relationships.LatestVersion.Data.Id == attr.Id {
description = attr.Attributes.Description
}
}
if description == "" {
description = strings.ToUpper(providerName) + "" + strings.Title(module.Attributes.Name)
}
outputFile := fmt.Sprintf("%s/terraform-%s-%s.yaml", providerName, providerName, module.Attributes.Name)
if_, err := os.Stat(outputFile); ! os.IsNotExist(err) {continue
}
if providerName == "aws" && (module.Attributes.Name == "rds" || module.Attributes.Name == "s3-bucket" ||
module.Attributes.Name == "subnet" || module.Attributes.Name == "vpc") {
continue
}
if err := generateDefinition(providerName, module.Attributes.Name, module.Attributes.Source, "", description); err ! = nil { fmt.Println(err.Error()) os.Exit(1)}}}func generateDefinition(provider, name, gitURL, path, description string) error {
defYaml := filepath.Join(provider, fmt.Sprintf("terraform-%s-%s.yaml", provider, name))
cmd := fmt.Sprintf("vela def init %s --type component --provider %s --git %s.git --desc \"%s\" -o %s",
name, provider, gitURL, description, defYaml)
ifpath ! ="" {
cmd = fmt.Sprintf("%s --path %s", cmd, path)
}
fmt.Println(cmd)
stdout, err := exec.Command("bash"."-c", cmd).CombinedOutput()
iferr ! = nil {return errors.Wrap(err, string(stdout))
}
fmt.Println(string(stdout))
return nil
Copy the codeExecute command:
go run gen.go aws "https://registry.terraform.io/v2/modules? filter%5Bprovider%5D=aws&include=latest-version&page%5Bsize%5D=50&page%5Bnumber%5D=1"
Copy the codeCode brief
Parse cloud resource data
Access the URL passed in by the user and parse the returned JSON data into a structure in Go.
The json format of the resource is as follows:
{
"data": [{"type": "modules"."id": "23"."attributes": {
"downloads": 18440513."full-name": "terraform-aws-modules/vpc/aws"."name": "vpc"."namespace": "terraform-aws-modules"."owner-name": ""."provider-logo-url": "/images/providers/aws.png"."provider-name": "aws"."source": "https://github.com/terraform-aws-modules/terraform-aws-vpc"."verified": true
},
"relationships": {
"latest-version": {
"data": {
"id": "142143"."type": "module-versions"}}},"links": {
"self": "/v2/modules/23"}},... ] ."included": [{"type": "module-versions"."id": "36806"."attributes": {
"created-at": "2020-01-03T11:35:36Z"."description": "Terraform module Terraform module for creating AWS IAM Roles with heredocs"."downloads": 260030."published-at": "2020-02-06T06:26:08Z"."source": ""."tag": "v2.0.0"."updated-at": "2022-02-22T00:45:44Z"."version": "2.0.0"
},
"links": {
"self": "/v2/module-versions/36806"}},... ] . }Copy the codeIn json data for Modules, we only care about two key-value pairs, namely:
• Data: contains the list of Modules names and attributes • Included: specifies the details of the selected version of Modules
Where, for each Module element in data, parse its attribute, Id and the Id corresponding to the latest-version in relationship; Parse the attributes and IDS for each Module version element Included.
Attributes are also resolved as follows:
• Name • Downloads • Source • Description • Verified
The structure is defined in the structure TFDownload. Json data is obtained from the HTTP library, and the structure of Terraform modules is resolved by json.Unmarshal.
Batch produce cloud resources
1. Create a directory to generate files required for resources
After the resolution is complete, create a new folder in the current directory and name the folder provider name. For each Module element in the parsed data, do the following to generate configuration files, definitions, and documentation for it.
2. Generate a definition file
The following Vela command reads the corresponding information from the github repository of the module to generate the definition file.
vela def init {ModuleName} --type component --provider {providerName} --git {gitURL} --desc {description} -o {yamlFileName}
Copy the codeThe directives that need to be filled in are passed in by the parsed Module structure.
• gitURL: {Module. The Attributes. The Source}. The git
Description: If the Included element ID is the same as the latest-version ID in module relationship, description is the description of the element attribute Included. Otherwise, description is a concatenation of providerName and module name
• yamlFileName: terraform – {providerName} – {Module. The Attributes. The Name}. The yaml
Why don’t you try it?
There are plenty of cloud services that offer Terraform Modules, for example
GCP: registry. Terraform. IO/namespaces /…
Ali cloud: registry. Terraform. IO/namespaces /…
Would you also like to introduce cloud resources for KubeVela from your current or favorite cloud service provider?
A link to the
[1] Simple and fast command line tools
Kubevela. IO/docs/next/p…
[2] AWS Cloud resource Terraform Modules
Registry. Terraform. IO/namespaces /…
[3] Terraform Modules Json format interface
registry.terraform.io/v2/modules? …
[4] Terraform Registry
aregistry.terraform.io/