In the previous article “Elasticsearch: I show how to use Runtime Field to shadow an existing field in the mapping, such as Duration. In today’s exercise, I’ll show you how to create a brand new field and count the data. Note here that the newly added Runtime field is not added to the source, but is only generated at query time, i.e., schema on read.
In the following exercise, it contains a demonstration of creating a Runtime field that calculates the day of the week from a timestamp field containing the date. Then use the index field and the newly created Runtime Field to create the visualization file in Kibana Lens. Runtime field is the name provided for the implementation of the schema when read in Elasticsearch.
show
Let’s create an index mapping:
#Create the index mapping
PUT date_to_day
{
"mappings": {
"properties": {
"timestamp": {
"type": "date",
"format": "yyyy-MM-dd"
},
"response_code": {
"type": "integer"
}
}
}
}
Copy the codeAbove, we show two fields: TIMESTAMP and response_code. Since we have a timestamp, we can export the day of week from this timestamp, which is the day of the week. And this is very useful for what we want to do for every day of the week.
We import the data using the following BULK API:
#Load a few documents to work with
POST date_to_day/_bulk
{"index":{}}
{"response_code": 200, "timestamp": "2021-01-01"}
{"index":{}}
{"response_code": 300, "timestamp": "2021-01-03"}
{"index":{}}
{"response_code": 200, "timestamp": "2021-01-04"}
{"index":{}}
{"response_code": 400, "timestamp": "2021-01-01"}
{"index":{}}
{"response_code": 300, "timestamp": "2021-01-05"}
{"index":{}}
{"response_code": 200, "timestamp": "2020-12-21"}
{"index":{}}
{"response_code": 200, "timestamp": "2021-01-02"}
{"index":{}}
{"response_code": 200, "timestamp": "2021-01-08"}
{"index":{}}
{"response_code": 300, "timestamp": "2021-01-09"}
{"index":{}}
{"response_code": 400, "timestamp": "2021-01-09"}
Copy the codeBecause we want to count every day of the week. One way is to create a new mapping from scratch. Include the day of week definition in this new mapping. And we process the data before we import it. In practical use, such processing can be laborious in the face of large amounts of existing data. We can use the Runtime Field to accomplish the desired function.
Used when searching for requests
Because the Runtime field is dynamically generated, it requires a computer to process it. Most of the time, we don’t want to modify mapping to do this. We only want to generate the Runtime field for some searches, or just as an exercise to verify that the Runtime field is correct before modifying the mapping. We use the following command to produce this type of Runtime field:
#Create an ephemeral runtime field for day of week and aggregate on it
GET date_to_day/_search
{
"runtime_mappings": {
"day_of_week": {
"type": "keyword",
"script": {
"source": """emit(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.SHORT, Locale.ROOT))"""
}
}
},
"size": 0,
"aggs": {
"terms": {
"terms": {
"field": "day_of_week"
}
}
}
}
Copy the codeIn the command above, we can read this section carefully:
"runtime_mappings": {
"day_of_week": {
"type": "keyword",
"script": {
"source": """emit(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.SHORT, Locale.ROOT))"""
}
}
},
Copy the codeInstead of using script to generate a runtime field called day_of_week. And that field only exists in this search. After performing this search, the field will automatically disappear. The day_of_week field is based on timestamp guidance and is completely absent in the previous mapping.
The command output is as follows:
{
"took" : 16,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"terms" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Fri",
"doc_count" : 3
},
{
"key" : "Sat",
"doc_count" : 3
},
{
"key" : "Mon",
"doc_count" : 2
},
{
"key" : "Sun",
"doc_count" : 1
},
{
"key" : "Tue",
"doc_count" : 1
}
]
}
}
}
Copy the codeAs shown above, we made statistics for every day of the week.
This parameter is used in index Mapping
Of course, in many cases, we want this field to always exist in the index mapping. The advantage of this is that we can use the defined Runtime Fields directly in the visualization in Kibana. Runtime fields can be mapped by adding a Runtime section under the Mapping definition and defining Painless script. The script has access to the entire context of the document, including the original _source and any mapped fields and their values. At query time, the script runs and generates values for each script field required for the query.
When defining the Painless script to use with the runtime fields, you must include emit to emit computed values. For example, the script in the following request extracts the day of the week from the @TIMESTAMP field, which is defined as a date type. The script calculates the day of the week based on the value of TIMESTAMP and returns the calculated value using emit.
We can define it as follows:
#Add the runtime field to the index mapping
PUT date_to_day/_mapping
{
"runtime": {
"day_of_week": {
"type": "keyword",
"script": {
"source": """emit(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.SHORT, Locale.ROOT))"""
}
}
}
}
Copy the codeThe Runtime section can be any of the following data types:
booleandatedoublegeo_pointipkeywordlong
To view the date_to_day index mapping, run the following command:
{
"date_to_day" : {
"mappings" : {
"runtime" : {
"day_of_week" : {
"type" : "keyword",
"script" : {
"source" : "emit(doc['timestamp'].value.dayOfWeekEnum.getDisplayName(TextStyle.SHORT, Locale.ROOT))",
"lang" : "painless"
}
}
},
"properties" : {
"response_code" : {
"type" : "integer"
},
"timestamp" : {
"type" : "date",
"format" : "yyyy-MM-dd"
}
}
}
}
}
Copy the codeKibana index Pattern creates an index pattern and looks at its field definition:
From the above we can see a newly added day_of_week field.
We can use this field directly in Kibana and visualize it:
Of course, if you are wondering if the source of the index already contains the newly generated day_of_week field, we can check by using the following command:
GET date_to_day/_search
Copy the codeThe command above shows:
{ "took" : 1, "timed_out" : false, "_shards" : { "total" : 1, "successful" : 1, "skipped" : 0, "failed" : 0 }, "hits" : {" total ": {" value" : 10, "base" : "eq"}, "max_score" : 1.0, "hits" : [{" _index ":" date_to_day ", "_type" : "_score" : 1.0, "_source" : {"response_code" : 200, "timestamp" : "2021-01-01" } }, { "_index" : "date_to_day", "_type" : "_doc", "_id" : "_yyDlXcBjSpwk8PH7vNz", "_score" : 1.0, the "_source" : {" response_code ": 300," timestamp ":" 2021-01-03 "}}, {" _index ":" date_to_day ", "_type" : "_doc", "_id" : "ACyDlXcBjSpwk8PH7vRz", "_score" : 1.0, "_source" : {"response_code" : 200, "timestamp" : "2021-01-04"}},...Copy the codeObviously, our source hasn’t changed at all. Day_of_week is just schema on read.