Testing data shapes using Go-Lookslike by Andrew Cholakian

It is my pleasure to introduce you to a new open source Go test/architecture validation library that we are developing at Elastic. It’s called Lookslike. LooksLike allows you to match Golang data structure shapes in a manner similar to JSON architectures, but it is more powerful than JSON and more similar to Go. It has many features that are not available in any of the existing Go test libraries.

Let’s look directly at what it does with an example:

// This library lets us check that a data-structure either similar to, or exactly like a certain schema.
// For example we could test whether a pet is a dog or a cat, using the code below.

// A dog named rover
rover := map[string]interface{}{"name": "rover", "barks": "often", "fur_length": "long"}
// A cat named pounce
pounce := map[string]interface{}{"name": "pounce", "meows": "often", "fur_length": "short

Running the above code will show the following results.

Checked rover, validation status true, errors: []
Checked pounce, validation status false, errors: [@Path 'barks': expected this key to be present @Path 'meows': unexpected field encountered during strict validation]

Here, we can see that the dog “Rover” matches as expected, but the cat “Pounce” does not match as expected, resulting in two errors. One error is not defining the “barks” key; Another error was the accidental occurrence of an extra key “MEOWS”.

Since looksLike is typically used in a Test context, we developed the TestsLike.test helper, which produces well-formed Test output. Simply change the last few lines in the above example to the ones shown below.

testslike.Test(t, dogValidator, rover)
testslike.Test(t, dogValidator, pounce)

composite

One of the key concepts in LooksLike is the ability to merge validators. Suppose we need separate cat and dog validators, but don’t want to redefine common fields like “name” and “fur_length” in each validator. Let’s take a look at how this is done in the following example.

pets := []map[string]interface{}{
    {"name": "rover", "barks": "often", "fur_length": "long"},
    {"name": "lucky", "barks": "rarely", "fur_length": "short"},
    {"name": "pounce", "meows": "often", "fur_length": "short"},
    {"name": "peanut", "meows": "rarely", "fur_length": "long"},
}

// We can see that all pets have the "fur_length" property, but that only cats meow, and dogs bark.
// We can concisely encode this in

Why build LooksLike

Lookslike is derived from the Heartbeat project of Elastic. Heartbeat is the proxy behind the run-time solution that pings the endpoint and then reports whether or not it is up and running. The final output of Heartbeat is the Elasticsearch document, which is represented in the Golang codebase as a “Map String Interface {}” type. Testing these output documents was part of the requirement to create the library, although it is now used elsewhere in the Beats code base.

The challenge is:

  • For some fields, the data does not need to match exactly, such as “monitor.duration,” which is used to calculate the time taken for execution. It may produce different results on each run. We need a way to loosely match data
  • In any given test, many fields are shared with other tests, and only a few are different. We want to be able to reduce code duplication by writing different field definitions.
  • We want good test output to show a single error for each field failure, so we need the “TestsLike” test helper.

Given these challenges, we made the following design decisions:

  • We wanted the architecture to be flexible, and we wanted developers to be able to create new matchers easily.
  • We want all schemas to be composable and nested, so that if you nest one document within another, you can merge schemas without having to copy a bunch of code.
  • We need a good test helper to make test failures easy to read.

Main types of

Lookslike’s architecture resolves two main types: “Validator” and “ISDEF.” “Validator” is the result of compiling a given schema. It is a function that takes any data structure and returns the result. “IsDef” is the type used to match a single field. You might wonder why there is a difference. In fact, we may merge the two types in the future. However, the main reason is that “IsDef” takes extra parameters about its location in the document structure, allowing it to perform advanced validation based on that context. “Validator” functions do not accept additional context, but are easier for the user to execute (they just accept “interface{}” and validate it).

For an example of writing a custom “IsDefs”, just see the source file. You can extend this by adding the new “IsDef” to your own source.

The sample

We use Lookslike heavily in Beats. A large number of usage examples can be found through this GitHub search.

We need your help!

If you are interested in LooksLike, submit a pull request on the repo! In particular, we can use a more comprehensive set of “ISDEF”.

To learn more about Elastic technology, please follow and sign up for the webinar. The upcoming schedule is as follows:

Wednesday, February 19, 2020 15:00-16:00 Building Omni-Observable Instances Using Elastic Stack

Wednesday, February 26, 2016 15:00-16:00 Kibana Lens Webinar

Wednesday, March 4, 2020 15:00-16:00 Elastic Endpoint Security Overview Network

Monitor website resources with Elastic Stack Wednesday, March 11, 2020 15:00-16:00