In past work, we built technology platforms using microservices, containerization, and service choreography. To increase development team productivity, we also provide a CICD platform for rapid deployment of code to Openshift (enterprise-class Kubernetes) clusters.

The first step in deployment is the containerization of the application, with continuous integration deliverables moving from jar packages, Webpacks, and so on to container images. Containerization packages software code together with all required components (libraries, frameworks, runtime environments) so that it can run consistently in any environment, on any infrastructure, and is “isolated” from other applications.

Our code needs to go from source code to compilation to a final runnable image, and even deployment, all in CICD’s pipeline. Initially, we added three files to each repository, which were also injected into the new project through the project generator (similar to Spring Initializer) :

  • Jenkinsfile.groovy: The Pipeline used to define Jenkins. There are several versions for different languages
  • Manifest YAML: Used to define the Kubernetes resource, which is a description of the workload and its execution
  • Dockerfile: Used to build objects

These three files also need to be constantly evolving in the work. At the beginning, when there were fewer projects (more than a dozen), our base team could also go to each code repository to maintain and upgrade. With the explosive growth of projects, the cost of maintenance is getting higher and higher. We iterated through the CICD platform and removed “Jenkinsfile.groovy” and “manifest YAML” from the project, leaving the Dockerfile with fewer changes.

As the platform evolves, we need to consider decoupling the only “nail” Dockerfile from the code and upgrading the Dockerfile if necessary. So I researched BuildPacks, which led to this article.

What is a Dockerfile

Docker automatically builds the image by reading the instructions in the Dockerfile. A Dockerfile is a text file that contains instructions that a Docker can execute to build an image. Let’s take the Dockerfile we used to test Tekton’s Java project:

FROM openjdk:8-jdk-alpine

RUN mkdir /app
WORKDIR /app
COPY target/*.jar /app/app.jar
ENTRYPOINT ["sh"."-c"."java -Xmx128m -Xms64m -jar app.jar"]
Copy the code

Mirror layered

You may have heard that Docker images contain multiple layers. Each layer corresponds to each command in the Dockerfile, such as RUN, COPY, ADD. Certain instructions create a new layer, which is fetched from the cache during mirror building if certain layers have not changed.

Buildpack below also uses image layering and caching to speed up the build of images.

What is a Buildpack

BuildPack is a program that converts source code into container images and can run in any cloud environment. Buildpack typically encapsulates a single language ecosystem tool chain. Suitable for Java, Ruby, Go, NodeJs, Python, etc.

What is a Builder?

Some buildPacks are sequentially assembled as Builder. In addition to BuildPacks, Lifecycle and Stack container images are added to Builder.

The stack container image consists of two images: build Image, which is used to run buildPack, and Run Image, which is the base image for building application images. This is the runtime environment in Builder.

How Buildpack works

Each BuildPack runtime consists of two phases:

1. Detection phase

Determine if the current BuildPack works by examining certain files/data in the source code. If applicable, it will enter the construction phase; Otherwise they quit. Such as:

  • Java Maven’s Buildpack checks to see if it is present in the source codepom.xml
  • Python’s buildpack checks to see if it is present in the source coderequirements.txtorsetup.pyfile
  • Node buildpack will findpackage-lock.jsonFile.

2. Construction phase

During the construction phase, the following operations occur:

  1. Set up the build environment and runtime environment
  2. Download the dependencies and compile the source code (if necessary)
  3. Set up the correct EntryPoint and startup scripts.

Such as:

  • Java Maven BuildPack has been checked to havepom.xmlAfter the file is executedmvn clean install -DskipTests
  • Python buildpack checks that there arerequrements.txtAfter that, it will executepip install -r requrements.txt
  • The Node build pack has been checkedpackage-lock.jsonAfter the implementationnpm install

BuildPack to fit

So how do you build an image using BuilderPack without Dockerfile? By looking at the above, you can basically see that the core is written and used in BuildPack.

There are many open source buildPacks available today that you don’t have to write yourself without specific customization. For example, the following Buildpacks are open source and maintained by major manufacturers:

  • Heroku Buildpacks
  • Google Buildpacks
  • Paketo

But before we get into the details of open source BuildPacks, let’s take a closer look at how BuildPacks work by creating our own. As for the test project, we still use the Tekton Test Java project.

All of the content below is submitted to Github and can be accessed at github.com/addozhang/b… Get the relevant code.

The final directory buildpacks-sample looks like this:

├ ─ ─ builders │ └ ─ ─ builder. Toml ├ ─ ─ buildpacks │ └ ─ ─ buildpack - maven │ ├ ─ ─ bin │ │ ├ ─ ─ build │ │ └ ─ ─ detect │ └ ─ ─ │ ├─ build │ ├─ build │ ├─ buildCopy the code

Create buildpack

Pack buildpack new examples/maven \ -- API 0.5 \ --path buildpack-maven \ --version 0.0.1 \ --stacks io.buildpacks.samples.stacks.bionicCopy the code

Take a look at the generated buildpack-maven directory:

Buildpack - ├─ bin │ ├─ build │ ├─ detect ├─ buildpackCopy the code

The default preliminary data in each file is not useful. Something to add:

bin/detect:

#! /usr/bin/env bash

if [[ ! -f pom.xml ]]; then
    exit 100
fi

plan_path=$2

cat >> "${plan_path}" <<EOL
[[provides]]
name = "jdk"
[[requires]]
name = "jdk"
EOL
Copy the code

bin/build:

#! /usr/bin/env bashset -euo pipefail layers_dir="$1" env_dir="$2/env" plan_path="$3" m2_layer_dir="${layers_dir}/maven_m2" if [[ !  -d ${m2_layer_dir} ]]; then mkdir -p ${m2_layer_dir} echo "cache = true" > ${m2_layer_dir}.toml fi ln -s ${m2_layer_dir} $HOME/.m2 echo "---> Running Maven" mvn clean install -B -DskipTests target_dir="target" for jar_file in $(find "$target_dir" -maxdepth 1 -name "*.jar" -type f); do cat >> "${layers_dir}/launch.toml" <<EOL [[processes]] type = "web" command = "java -jar ${jar_file}" EOL break; doneCopy the code

buildpack.toml:

API = "0.5" [buildpack] id = "examples/maven" version = "0.0.1" [[stacks]] id = "com.atbug.buildpacks.example.stacks.maven"Copy the code

Create a stack

We use Maven: 3.5.4-JDK-8-SLIM as the base image of the build image. Therefore, openJDK: 8-JDK-Slim is used as the base image of run Image.

Create build and run directories in the stacks directory:

build/Dockerfile

FROM maven:3.5.4-jdk-8-slim

ARG cnb_uid=1000
ARG cnb_gid=1000
ARG stack_id

ENV CNB_STACK_ID=${stack_id}
LABEL io.buildpacks.stack.id=${stack_id}

ENV CNB_USER_ID=${cnb_uid}
ENV CNB_GROUP_ID=${cnb_gid}

# Install packages that we want to make available at both build and run time
RUNapt-get update && \ apt-get install -y xz-utils ca-certificates && \ rm -rf /var/lib/apt/lists/*

# Create user and group
RUN groupadd cnb --gid ${cnb_gid} && \
  useradd --uid ${cnb_uid} --gid ${cnb_gid} -m -s /bin/bash cnb

USER ${CNB_USER_ID}:${CNB_GROUP_ID}
Copy the code

run/Dockerfile

FROM openjdk:8-jdk-slim

ARG stack_id
ARG cnb_uid=1000
ARG cnb_gid=1000
LABEL io.buildpacks.stack.id="${stack_id}"

USER ${cnb_uid}:${cnb_gid}
Copy the code

Then use the following command to build two images:

export STACK_ID=com.atbug.buildpacks.example.stacks.maven

docker build --build-arg stack_id=${STACK_ID} -t addozhang/samples-buildpacks-stack-build:latest ./build
docker build --build-arg stack_id=${STACK_ID} -t addozhang/samples-buildpacks-stack-run:latest ./run
Copy the code

Create a Builder

Now that we have buildPack and Stack, it’s time to create the Builder. First create the Builder.toml file and add the following:

[[buildpacks]]
id = "examples/maven"
version = "0.0.1"
uri = ".. /buildpacks/buildpack-maven"

[[order]]
[[order.group]]
id = "examples/maven"
version = "0.0.1"

[stack]
id = "com.atbug.buildpacks.example.stacks.maven"
run-image = "addozhang/samples-buildpacks-stack-run:latest"
build-image = "addozhang/samples-buildpacks-stack-build:latest"
Copy the code

Then run the –pull-policy if-not-present command to remove the need to push the stack’s two mirrors to the repository:

pack builder create example-builder:latest --config ./builder.toml --pull-policy if-not-present
Copy the code

test

Now that we have Builder, we can use the builder we created to build the image.

The –pull-policy if-not-present parameter is also added to use the local Builder image:

#Buildpacks -sample is the same directory as Tekton-test, and execute the following command in buildpacks-samplepack build addozhang/tekton-test --builder example-builder:latest --pull-policy if-not-present --path .. /tekton-testCopy the code

If you see something similar to the following, the image build is successful. The first build may take a long time due to the need to download maven dependencies.

. ===> EXPORTING [exporter] Adding 1/1 app layer(s) [exporter] Reusing layer 'launcher' [exporter] Reusing layer 'config' [exporter] Reusing layer 'process-types' [exporter] Adding label 'io.buildpacks.lifecycle.metadata' [exporter] Adding label 'io.buildpacks.build.metadata' [exporter] Adding label 'io.buildpacks.project.metadata' [exporter] Setting default  process type 'web' [exporter] Saving addozhang/tekton-test... [exporter] *** Images (0d5ac1158bc0): [exporter] addozhang/tekton-test [exporter] Adding cache layer 'examples/maven:maven_m2' Successfully built image addozhang/tekton-testCopy the code

Start the container and see that the Spring Boot application starts normally:

docker run --rm addozhang/tekton-test:latest . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \ / _ ` | \ \ \ \ \ \ / ___) | | _) | | | | | | | (_ | |))))' there comes | |. __ _ - | | | _ _ - | | | _ \ __, | / / / / = = = = = = = = = | _ | = = = = = = = = = = = = = = | ___ / = / _ / _ / _ / : : Spring the Boot: : (v2.2.3. RELEASE)...Copy the code

conclusion

There are many open source buildPacks available today that you don’t have to write yourself without specific customization. For example, the following Buildpacks are open source and maintained by major manufacturers:

  • Heroku Buildpacks
  • Google Buildpacks
  • Paketo

The buildPacks libraries above are fairly comprehensive and will be implemented slightly differently. For example, Heroku uses Shell scripts for its execution phase, while Paketo uses Golang. The latter is scalable, supported by the Cloud Foundry Foundation and has a full-time core development team sponsored by VMware. These small modular buildpacks can use different scenarios by combining extensions.

Again, it’s much easier to understand how Buildpack works if you write one yourself.