1. Introduction

Build systems are automated tools used to generate target artifacts from source code, including libraries, executables, generated scripts, etc. Build systems typically provide platform-specific executables that externally trigger builds by executing commands such as GUN Make, Ant, CMake, Gradle, etc. Gradle is a flexible and powerful open source build system that provides a cross-platform executable for outsiders to execute Gradle builds from a command line window with commands such as./gradlew Assemble that trigger Gradle build tasks.

Modern and mature ides will integrate the required build system, combine various command line tools, package into a set of automated build tools, and provide build view tools, improve developer productivity. In IntelliJ IDEA, a Gradle task can be triggered by the Gradle view tool, but this is not implemented by packaging the command line tool. Gradle Tooling API is required to implement Gradle Tooling. This API allows you to embed Gradle build capabilities into your IDE or other tools:

Why did Gradle specifically provide a Tooling API for external integration calls, rather than just a command modality based on executable programs, as other build systems do? The Tooling API is a significant extension to Gradle that provides more control and depth of build control than the command approach, enabling ides and other tools to more easily and closely integrate Gradle capabilities. Tooling API the Tooling API can directly return build results without manually parsing the log output of a command line program as in command mode. It can be run independent of version, meaning that the Tooling API of the same version can handle builds of different Gradle versions with forward and backward compatibility.

2. Interface function and Invocation example

2.1 Interface Function

Tooling API Provides the Tooling API to perform and monitor builds and query build information:

  • Query build information, including project structure, project dependencies, external dependencies, and project tasks;
  • Execute build tasks and monitor build progress;
  • Cancel a build task in progress;
  • Automatically download the Gradle version that matches your project.

Key apis are as follows:

2.2 Invocation Example

Query project structure and tasks

try (ProjectConnection connection = GradleConnector.newConnector()
         .forProjectDirectory(new File("someFolder"))
         .connect()) {
   GradleProject rootProject = connection.getModel(GradleProject.class);
   Set<? extends GradleProject> subProject = rootProject.getChildren();
   Set<? extends GradleTask> tasks = rootProject.getTasks();
}
Copy the code

As described above, a ProjectConnection to the participating project was created using the Tooling API entry class GradleConnector. Then, getModel(Class

modelType) is used to obtain the structural information model of this project, GradleProject, which contains the project structure, project tasks and other information we want to query.

Executing a Build Task

String[] gradleTasks = new String[]{"clean", "app:assembleDebug"}; try (ProjectConnection connection = GradleConnector.newConnector() .forProjectDirectory(new File("someFolder")) .connect()) { BuildLauncher build = connection.newBuild(); build.forTasks(gradleTasks) .addProgressListener(progressListener) .setColorOutput(true) .setJvmArguments(jvmArguments);  build.run(); }Copy the code

In this example, we create a BuildLauncher to execute the build task using the newBuild() method of ProjectConnection, and then use forTasks(String… Tasks) configures Gradle tasks to be executed, monitors the progress of the execution, etc. Finally, run() triggers the execution of the task.

3. Principle analysis

3.1 How do I communicate with the Gradle build process?

Gradle Tooling API Gradle Tooling API does not implement Gradle build capabilities. Instead, it provides an entry to native Gradle programs to communicate with Gradle in coded form. After invoking Gradle build capabilities in our own Tooling program, You also need to communicate across processes with the real Gradle builder. Gradle Tooling API, Gradle Tooling API, Gradle Tooling API, Gradle Tooling API, Gradle Tooling API, Gradle Tooling API, Gradle Tooling API Gradle build process is the Gradle build process that actually performs the tasks.

Gradle Daemon process is a long-running Gradle build process that allows you to build faster by bypassing the JVM environment and memory caching. Daemon Process is always enabled for Gradle Client that integrates the Gradle Tooling API. In other words, tools that integrate the Gradle Tooling API always communicate with daemon processes across processes to invoke Gradle build capabilities. Gradle daemon processes are dynamically created. If you want to connect to a dynamically created daemon process, you need to register it and open it up for query through service registration and service discovery. DaemonRegistry provides such a mechanism.

Client – Gradle Client

The Gradle Tooling API implements cross-process communication from a source point of view.

try (ProjectConnection connection = GradleConnector.newConnector()
         .forProjectDirectory(new File("someFolder"))
         .connect()) {
   GradleProject rootProject = connection.getModel(GradleProject.class);
}
Copy the code

From the code, although ProjectConnection looks like it establishes a link to the Daemon process, it doesn’t, Instead, the link to daemon process is established in the getModel(Class

modelType) method. Within the method, the Tooling API makes a call to the Gradle source code. Finally in DefaultDaemonConnector. Java daemon process finds the available:

public DaemonClientConnection connect(ExplainingSpec<DaemonContext> constraint) { final Pair<Collection<DaemonInfo>, Collection<DaemonInfo>> idleBusy = partitionByState(daemonRegistry.getAll(), Idle); final Collection<DaemonInfo> idleDaemons = idleBusy.getLeft(); final Collection<DaemonInfo> busyDaemons = idleBusy.getRight(); // Check to see if there are any compatible idle daemons DaemonClientConnection connection = connectToIdleDaemon(idleDaemons, constraint); if (connection ! = null) { return connection; } // Check to see if there are any compatible canceled daemons and wait to see if one becomes idle connection = connectToCanceledDaemon(busyDaemons, constraint); if (connection ! = null) { return connection; } // No compatible daemons available - start a new daemon handleStopEvents(idleDaemons, busyDaemons); return startDaemon(constraint); }Copy the code

Through the above daemon process to find the logic and relevant code, it can be concluded that:

  1. Daemon processes include Idle, Busy, Canceled, StopRequested, Stopped, and Broken.
  2. When running a Gradle build in Daemon Process mode, it tries to find Idle and Canceled daemon processes that are compatible with the environment. Create a daemon process that is compatible with Gradle Client.
  3. All Daemon processes are logged inDaemonRegistry.javaIn the Gradle Client registry.
  4. Daemon process environment compatibility checks include Gradle version, file code, JVM heap size, etc.
  5. When a compatible Daemon process is obtained, the Socket connects to the port on which the daemon process listens, and then communicates with the process through the Socket.

Server – Daemon process

When a Gradle client calls Gradle build capability, it triggers the creation of a daemon process. The process entry function is in gradledaemon. Java, and then it goes to daemonmain. Java to initialize the process. Finally in TcpIncomingConnector. Open the Socket Server in Java and bind to monitor a specified port:

public ConnectionAcceptor accept(Action<ConnectCompletion> action, boolean allowRemote) { final ServerSocketChannel serverSocket; int localPort; try { serverSocket = ServerSocketChannel.open(); serverSocket.socket().bind(new InetSocketAddress(addressFactory.getLocalBindingAddress(), 0)); localPort = serverSocket.socket().getLocalPort(); } catch (Exception e) { throw UncheckedException.throwAsUncheckedException(e); }... }Copy the code

Then in DaemonRegistryUpdater. Java daemon process will be record in the registry:

public void onStart(Address connectorAddress) {
    LOGGER.info("{}{}", DaemonMessages.ADVERTISING_DAEMON, connectorAddress);
    LOGGER.debug("Advertised daemon context: {}", daemonContext);
    this.connectorAddress = connectorAddress;
    daemonRegistry.store(new DaemonInfo(connectorAddress, daemonContext, token, Busy));
}
Copy the code

In this way, Gradle Client can obtain compatible Daemon processes and their ports in the registry, and establish a connection to the daemon process to communicate with it.

Tooling API: Gradle Daemon Process: Tooling API: Gradle Daemon Process

  1. Tooling API: The tolling API itself was not too much code, and was invoked to obtain project information through the interfaceModelProducerAfter encapsulation, it enters Gradle source code, but still belongs to Gradle client process.
  2. inDefaultDaemonConnectorWill try to learn fromDaemonRegistryGet a valid, compatible daemon process, or create one if none is available.
  3. After the Daemon process starts, it uses Socket binding to listen to fixed ports and records information about the listening portsDaemonRegistryFor Gradle Client to query, obtain, and establish a connection.

3.2 How to Achieve Forward and backward Compatibility?

Tooling API Supports Gradle 2.6 and later versions, indicating that one version of Gradle is Tooling API compliant forward and backward with other Gradle versions. Tooling API supports making calls to older or newer Gradle builds. However, the Tooling API included interface functions that did not apply to all Gradle versions; Gradle 5.0 and later also required Tooling API version. Tooling API 3.0 and later were required. How was compatibility implemented between Gradle and Tooling API versions?

Consider this question: If we have two pieces of software: main software A and tool software B, which is dedicated to calling A, how can we achieve maximum and elegant version compatibility between A and B? Following is an in-depth analysis of the Tooling API and Gradle source code to see what technical solutions Gradle adopted in terms of version compatibility are of interest.

Gradle version adaptation

Gradle Tooling API source repository: Gradle Tooling API

In the figure, we will focus only on DefaultConnection- the key class that was invoked from the Tooling API to the Gradle Launcher module:

DefaultConnection has entry points to accept calls from different ToolingAPI versions

Tooling API side eventually DefaultToolingImplementationLoader. In Java by custom URLClassLoader load DefaultConnection, A custom URLClassLoader class load path specifies the jar package under Gradle version lib, which can load DefaultConnection for different Gradle versions.

private ClassLoader createImplementationClassLoader(Distribution distribution, ProgressLoggerFactory progressLoggerFactory, InternalBuildProgressListener progressListener, ConnectionParameters connectionParameters, BuildCancellationToken cancellationToken) {
    ClassPath implementationClasspath = distribution.getToolingImplementationClasspath(progressLoggerFactory, progressListener, connectionParameters, cancellationToken);
    LOGGER.debug("Using tooling provider classpath: {}", implementationClasspath);
    FilteringClassLoader.Spec filterSpec = new FilteringClassLoader.Spec();
    filterSpec.allowPackage("org.gradle.tooling.internal.protocol");
    filterSpec.allowClass(JavaVersion.class);
    FilteringClassLoader filteringClassLoader = new FilteringClassLoader(classLoader, filterSpec);
    return new VisitableURLClassLoader("tooling-implementation-loader", filteringClassLoader, implementationClasspath);
}
Copy the code

Tooling API makes a tolling call to the source code of the native version of Gradle via the custom Java class loader. Note that DefaultConnection is part of the Gradle client process. That is, in tools such as IDE software programs.

Model class adaptation

The getModel(Class

modelType) method is used to obtain the engineering structure information model GradleProject from the Gradle Daemon process. GradleProject may be defined in different Gradle versions. How to implement the following information model structure in the Tooling API?

Tooling API Determines whether the model is supported in versiondetails. Java according to Gradle version before requesting the information model. If required, the Tooling API issues a request to daemon Process. Daemon process will return to the corresponding version of the information model, ProtocolToModelAdapter in Tooling API in Java to encapsulate a layer of dynamic Proxy, eventually returns to the Proxy:

private static <T> T createView(Class<T> targetType, Object sourceObject, ViewDecoration decoration, ViewGraphDetails graphDetails) { ...... // Create a proxy InvocationHandlerImpl handler = new InvocationHandlerImpl(targetType, sourceObject, decorationsForThisType, graphDetails); Object proxy = Proxy.newProxyInstance(viewType.getClassLoader(), new Class<? >[]{viewType}, handler); handler.attachProxy(proxy); return viewType.cast(proxy); }Copy the code

Final Tooling API returned GradleProject was only a dynamic proxy interface, as follows:

public interface GradleProject extends HierarchicalElement, BuildableElement, ProjectModel {
    ......
    File getBuildDirectory() throws UnsupportedMethodException;
}
Copy the code

As you can see, even the support of information model, some contents may also be due to Gradle version mismatch does not support available, call throws UnsupportedMethodException anomalies.

The Tooling API received only the model information interface, which was not a real model entity class. After receiving the model information, one layer of transformation was required when the whole model information class was serialized or transmitted. Construct a real contains the content of the entity class, Android SDKTools library for the AndroidProject model, construct a real contains the content of the entity class IdeAndroidProjectImpl.

4. To summarize

Gradle Tooling API (Tooling API) Gradle Tooling API (Tooling API) Gradle Tooling API (Tooling API) Finally, in terms of principle analysis, the Tooling API implements cross-process communication and version compatibility, two important mechanisms.

After learning Gradle Tooling API, you learned the architecture principle of Gradle Tooling API in detail, enabling you to develop Gradle Tooling software based on the Tooling API. In addition, you learned some methodology in similar technical architecture scenarios. Service registration and service discovery mechanisms can be introduced to query and connect dynamic services when they need to communicate with dynamically created services. As a tool program for external access, when similar programs only provide command line mode, we should dare to break the convention and provide a new way, so that we can empower other software to a greater extent and achieve a win-win situation for both sides.

5. Refer to the article

  • Org. Gradle. Tooling gradle API (7.2)

    Docs.gradle.org/current/jav…

  • Gradle & Third-party Tools

    Docs.gradle.org/current/use…

  • Gradle | Gradle Features

    Gradle.org/features/#e…

  • The Gradle Daemon

    Docs.gradle.org/current/use…

6. Join us

We are the Developer Tools team under The Client Infrastructure team of Bytedance, responsible for building company-wide r&d Tools for different business scenarios to improve the efficiency of mobile application r&d. It is urgent to find Android mobile R&D engineer/iOS mobile R&D engineer/server R&D engineer.

For more information please contact: [email protected], email subject Resume – Name – Job Intention – Desired city – Tel.