Image courtesy of Unsplash by Marc Reichelt

The Jetpack Room library provides an abstraction layer on TOP of SQLite, providing the ability to validate SQL queries at compile time without any boilerplate code. It implements this behavior by handling code annotations and generating Java source code.

Annotation handlers are very powerful, but they increase build time. This is generally acceptable for code written in Java, but for Kotlin, the compile time consumption is significant because Kotlin does not have a built-in annotation processing pipeline. Instead, it generates stub Java code through the Kotlin code to support the annotation processor, which it then pipes into the Java compiler for processing.

Because not everything in Kotlin’s source code can be represented in Java, some information is lost in this transformation. Again, Kotlin is a multi-platform language, but KAPT only works for Java bytecode.

knowKotlin symbol processing

With the widespread use of annotation processors on Android, KAPT has become a compile-time performance bottleneck. To solve this problem, the Google Kotlin compiler team began working on an alternative that would provide first-class annotation processing support for Kotlin. When this project was born, we were very excited because it would help Room better support Kotlin. Starting with Room 2.4, which has experimental support for KSP, we’ve seen a 2x improvement in compilation speed, especially for full compilation.

This article is not about annotation processing, Room, or KSP. Rather, it highlights the challenges and trade-offs we faced in adding KSP support to Room. You don’t need to know Room or KSP to understand this article, but you do need to be familiar with annotation processing.

Note: we started using KSP before the stable release. So it is not clear that some of the decisions made before are applicable now.

The purpose of this article is to give annotation processor authors a good idea of what to look out for before adding KSP support to their projects.

Introduction to Room working principle

Annotation processing for Room is divided into two steps. There are several “Processor” classes that walk through the user’s code, validating and extracting the necessary information into the “value object”. These value objects are sent to the “Writer” classes, which convert them into code. Like many other annotation processors, Room relies heavily on frequently referenced classes in auto-common and the Javax.Lang. model package (the Java annotation processing API package).

To support KSP, we have three options:

  1. Copy each “Processor” class of JavaAP and KSP, they will have the same value object as output, which we can input into Writer;
  2. Create an abstraction layer on top of KSP/Java AP so that the processor has an implementation based on that abstraction layer;
  3. Replace JavaAP with KSP and require developers to use KSP for Java code as well.

Option C is actually not feasible because it would cause significant disruption to Java users. As the number of Room uses increases, this disruptive change is impossible. Given the choice between “A” and “B”, we decided to choose “B” because the processor has A considerable amount of business logic that is not easy to decompose.

knowX-Processing

Creating a common abstraction on JavaAP and KSP is not easy. Kotlin and Java can interoperate, but not in the same pattern, for example, the types of special classes in Kotlin such as Kotlin’s value classes or static methods in Java. In addition, Java classes have fields and methods, while Kotlin has properties and functions.

Instead of trying to achieve perfect abstractions, we decided to implement “What Room needs”. Literally, find every file in Room that imports javax.lang.Model and move it into an X-Processing abstraction. As a result, TypeElement becomes XTypeElement, ExecutableElemen becomes XExecutableElemen, and so on.

Unfortunately, the Javax.lang. model API is very widely used in Room. Creating all of these X classes at once creates a very serious psychological burden on the reviewer. Therefore, we need to find a way to iterate over this implementation.

On the other hand, we need to prove that it can work. So we prototyped it first, and once it was a reasonable choice, we re-implemented all the X classes one by one with their own tests.

A good example of what I say is needed to implement “Room “can be seen in the field changes on the class. When Room processes the fields of a class, it is always interested in all of its fields, including the fields in the parent class. So when we created the corresponding X-Processing API, we just added the ability to get all the fields.

interface XTypeElement {
  fun getAllFieldsIncludingPrivateSupers(a): List<XVariableElement>
}
Copy the code

If we were designing a common library, it would probably never pass API review. But because our target is just Room, and it already has a helper method that has the same functionality as TypeElement, copying it reduces the risk of the project.

Once we have the basic X-Processing apis and their test methods, the next step is to let Room call this abstraction. This is where “what’s needed for Room” pays off well. Room already has extension functions/properties on the Javax.lang.model API for basic functions (such as methods to get TypeElement). We first updated these extensions to look like the X-Processing API, and then migrated Room to X-Processing in 1 CL.

Improve API usability

Keeping javaAP-like apis doesn’t mean we can’t improve anything. After migrating Room to X-Processing, we implemented a number of API improvements.

For example, Room calls MoreElement/MoreTypes multiple times to convert between different Javax.lang. model types (such as MoreElements. AsType). The related calls are usually as follows:

val element: Element ...
if (MoreElements.isType(element)) {
  val typeElement:TypeElement = MoreElements.asType(element)
}
Copy the code

We put all the calls into Kotlin contracts so that we can write:

val element: XElement ... If (element.istypeElement ()) {// the compiler recognizes that the element is an XTypeElement}Copy the code

Another good example is finding methods in a TypeElement. Typically in JavaAP, you call the ElementFilter class to get the methods in TypeElement. Instead, we set it directly as an attribute in XTypeElement.

/ / before
val methods = ElementFilter.methodsIn(typeElement.enclosedElements)
/ / after
val methods = typeElement.declaredMethods
Copy the code

The last example, and this is probably one of my favorites, is distributability. In JavaAP, if you want to check whether a given TypeMirror can be assigned by another TypeMirror, you call types.isAssignable.

val type1: TypeMirror ...
val type2: TypeMirror ...
if (typeUtils.isAssignable(type1, type2)) {
  ...
}
Copy the code

This code is really hard to read, because you can’t even guess whether it verifies that type 1 can be specified by type 2, or just the opposite. We already have an extension function as follows:

fun TypeMirror.isAssignableFrom(
  types: Types,
  otherType: TypeMirror
): Boolean
Copy the code

In X-Processing, we can convert it to a regular function on XType, as follows:

interface XType {
  fun isAssignableFrom(other: XType): Boolean
}
Copy the code

Implement the KSP back end for X-processing

Each of these X-Processing interfaces has its own test suite. We didn’t write them to test AutoCommon or JavaAP. Rather, we wrote them so that when we had their KSP implementation, we could run test cases to verify that it met Room’s expectations.

Since the original X-Processing apis were modeled after avax.lang.model, they were not always suitable for KSP, so we also improved them to provide better support for Kotlin when needed.

This creates a new problem. The existing Room code base was written to handle Java source code. When an application is written by Kotlin, Room can only recognize what that Kotlin looks like in the Java stub. We decided to maintain similar behavior in the KSP implementation of X-Processing.

For example, the suspend function in Kotlin generates the following signature at compile time:

// kotlin
suspend fun foo(bar:Bar):Baz
// java
Object foo(bar:Bar, Continuation<? extends Baz>)
Copy the code

To maintain the same behavior, the XMethodElement implementation in KSP syntheses a new parameter for the suspend method, along with a new return type. (KspMethodElement.kt)

Note: This works well because Room generates Java code, even in KSP. When we add support for Kotlin code generation, it may cause some changes.

Another example relates to attributes. A Kotlin property may also have a synthetic getter/setter (accessor) based on its signature. Because Room expects to find these accessors as methods (see: kspTypeElement.kt), XTypeElement implements these composition methods.

Note: We have plans to change the XTypeElement API to provide properties instead of fields, because that’s what Room really wants. As you can guess by now, we decided not to do this “for the time being” to reduce Room modifications. Hopefully one day we’ll be able to do that, and when we do, XTypeElement’s JavaAP implementation will bundle methods and fields together as properties.

The last interesting issue when adding a KSP implementation to X-Processing is API coupling. The apis of these processors often talk to each other, so you can’t implement XTypeElement in KSP without implementing XField/XMethod, which itself references XType, and so on. As we added these KSP implementations, we wrote separate test cases for their implementation parts. As the KSP implementation became more complete, we gradually started all x-Processing tests through the KSP back end.

It is important to note that at this stage we are only running tests in x-Processing projects, so even if we know what is being tested, there is no guarantee that all Room tests will pass (also known as unit tests vs integration tests). We needed a way to run all Room tests using the KSP back end, and “X-Processing-testing” was born.

knowX-Processing-Testing

The annotation handler is written with 20% of the processor code and 80% of the test code. You need to consider all possible developer errors and make sure that error messages are reported truthfully. To write these tests, Room has provided a helper method as follows:

Copy the code

RunTest uses the Google Compile Testing library underneath and allows us to simply unit test the processor. It synthesizes a Java annotation handler and calls the processor-supplied process method in it.

val entitySource : JavaFileObject // example @entity annotation class
val result = runTest(entitySource) { invocation ->
  val element = invocation.processingEnv.findElement("Subject")
  valentityValueObject = EntityProcessor(...) .process(element)/ / assertion entityValueObject
}
// Assert whether the result is wrong, warning, etc
Copy the code

Unfortunately, Google Compile Testing only supports Java source code. To test Kotlin we needed another library, fortunately Kotlin Compile Testing, which allowed us to write tests for Kotlin, and we contributed KSP support to the library.

Note: We later replaced Kotlin Compile Testing with an internal implementation to simplify Kotlin/KSP updates in the AndroidX Repo. We also added better assertion APIS, which required us to perform API-incompatible modification operations on KCT.

As a final step to enable KSP to run all tests, we created the following test API:

fun runProcessorTest(
  sources: List<Source>,
  handler: (XTestInvocation) - >Unit
): Unit
Copy the code

The main difference between this and the original is that it runs tests through both KSP and JavaAP (or KAPT, depending on the source). Because it runs the test multiple times and KSP and JavaAP have different results, it cannot return a single result.

So we came up with an idea:

fun XTestInvocation.assertCompilationResult(
  assertion: (XCompilationResultSubject) - >Unit
}
Copy the code

After each compilation, it invokes the result assertion (if there is no failure message, the compilation is checked for success). We refactor each Room test as follows:

val entitySource : Source // example @entity annotation class
runProcessorTest(listOf(entitySource)) { invocation ->
  // This code block is run twice, once using JavaAP/KAPT and once using KSP
  val element = invocation.processingEnv.findElement("Subject")
  valentityValueObject = EntityProcessor(...) .process(element)/ / assertion entityValueObject
  invocation.assertCompilationResult {
    // The result is asserted as error, warning, etc
    hasWarningContaining("...")}}Copy the code

The rest of the story is simple. Migrate compile tests for each Room to the new API, report new KSP/X-Processing errors as soon as they are found, and then implement interim solutions; This is repeated. As KSP is under heavy development, we do have a lot of bugs. Each time we reported the bug, linked to it from the Room source, and moved on (or fixed it). After each KSP release, we searched the code base to find fixed issues, removed interim solutions, and started testing.

Once the compile test coverage is good, we will run Room’s integration tests using KSP in the next step. These are actual Android test apps that also test their behavior at run time. Fortunately, Android supports Gradle variants, so running our Kotlin integration tests using KSP and KAPT is fairly easy.

The next step

Adding KSP support to Room is just the first step. Now, we need to update Room to use it. For example, nullability is ignored by all type checks in Room because the TypeMirror of javax.lang.model does not understand Nullability. Therefore, when your Kotlin code is called, Room sometimes raises a NullPointerException at runtime. With KSP, these checks now create new KSP bugs (such as B /193437407) in Room. We’ve added some temporary solutions, but ideally we still want to improve Room to handle these situations correctly.

Again, even though we support KSP, Room still generates only Java code. This limitation prevents us from adding support for certain Kotlin features, such as Value Classes. Hopefully, in the future, we’ll also have some support for generating Kotlin code to provide first-class support for Kotlin in Room. Next, maybe more :).

Can I use X-Processing on my project?

The answer is not yet; At least not the way you would use any other Jetpack library. As mentioned earlier, we only implemented what Room needed. Writing a real Jetpack library involves a lot of work — documentation, API stability, Codelabs, etc. — that we can’t afford. Having said that, Dagger and Airbnb (Paris, DeeplinkDispatch) both started supporting KSP with X-Processing (and contributed what they needed 🙏). Maybe one day we’ll break it out of Room. Technically, you can still use it just as you would the Google Maven library, but there is no API that guarantees this, so you should definitely use the Shade technology.

conclusion

We added KSP support to Room, which wasn’t easy but was definitely worth it. If you are maintaining annotation processors, add support for KSP to provide a better Kotlin developer experience.

Special thanks to Zac Sweers and Eli Hart, excellent KSP contributors, for reviewing an earlier version of this article.

More resources

  • Issue Tracker about Room’s support for KSP

  • X-ray source Processing

  • X – Processing – Testing the source code

  • KSP source

Please click here to submit your feedback to us, or share your favorite content or questions. Your feedback is very important to us, thank you for your support!