EsClientRHL

EsClientRHL is an elasticSearch wrapper that can be invoked on the ElasticSearch Java client based on SpringBoot. Built-in ES index structure tools, ES index data add, delete, change tools, ES query tools, ES data analysis tools or ES usage scaffolding, can be easily integrated and very convenient to use.

The reason why EsClientRHL is selected

Currently, the es TransportClient is used for Spring-Data-ElasticSearch. However, ES plans to abandon TransportClient. The tool is encapsulated with RestHighLevelClient recommended by ES
Spring-data-elasticsearch supports limited apis, while EsClientRHL supports richer API calls
The Java Client API has been greatly simplified and updated to make the more advanced features of ES easier to use
Support for two types of automation to ease developers’ workload and allow them to focus more on business development

Enable automatic scanning of ElasticSearch index entities and automatically create index structures for entities that do not have index structures
Support developers define only one interface and have the dark magic of frequently interacting with ES

EsClientRHL provides the following API tools for elasticSearch: es index data add/delete, ES query, ES data analysis, etc
Some oF the apis in EsClientRHL combine best practices in real-world scenarios
In short, ESClientRHL can help you, then it has the value of existence, if there is some help for you, please don’t hesitate to Star

Gitee.com/zxporz/ESCl…

Release notes

Elasticsearch7 + is the only version of elasticSearch7 that is compatible with elasticSearch7. You are advised to add the elasticSearch version number to the poM file in the imported Springboot project; otherwise, it may be overwritten by the Springboot parent project.

< the properties >... < elasticsearch version > 7.3.1 < / elasticsearch version >... </properties>Copy the code

Update log

Updated date	Update the content
2019-03-19	Fixed a search suggestion mapping bug, added an API to delete by criteria, added an API to retrieve multiple indexes when searching, and annotated configuration (see API update)
2019-04-16	@ EnableESTools was optimized for the logical address of the entity, and add the support of esclientrhl – start, increase ImportBeanDefinitionRegistrar (eggs)
2019-06-07	As required by the message, I submit a compressed package (the root directory is testdemo.zip). The compressed package contains the demo called by the test code and the PDF version of the document used
2019-09-01	Added support for ES7 +, addedThe branch of es7With support for version 7+, the default version is stilles6May I have your attention
2019-09-01	Since support for ES7 + has been added to the MAVEN POM version, the corresponding version number of 7 is 7.0.0 in order to keep consistent with es. The version number of 6 will not be adjusted for the time being (it will be adjusted later).
2019-09-05	Optimized some logic of internal automation functions
2019-09-05	– Added a new feature for minimal development that allows you to directly define interfaces to call simple common methods to interact with ElasticSearch (similar to JPA)
2019-09-05	The default version is 7.0.0, and the component version of ES6 is 6.0.0
2019-09-06	IndexType is no longer required in ESMetaData. ElasticsearchTemplate adds a batch update method, as described in the API section below
2019-09-17	The query method searchMore was added to directly specify the maximum return result and add this method to the interface proxy
2019-10-10	Added batch batch new A method to update index data The paging, highlighting, sorting, and query methods add the ability to return the results of specified fields The 7+ version changes the default master shard number to 1 Added null_value support for mapping annotation Added methods that support uri QueryString Added methods to support SQL queries A slew of utility updates will follow
2019-10-11	To avoid ambiguity, the autoComplete customization of mapping was renamed to Ngram and the functionality remains the same The Phrase Suggester search method was added Improved the template search method
2019-10-11	The best practices and usage instructions of disMaxQuery, multiMatchQuery (three kinds), functionScoreQuery and boostingQuery are updated
2019-10-14	Advanced query methods are added: support the original paging, sorting, highlighting query, move the specified field policy to the advanced query method, and increase the function of route setting Added the route saving method Added the route deletion method
2019-10-17	Added the step scroll method to avoid memory overflow caused by large quantity, and set the original scroll method as not recommended Support for search after has been added to advanced queries Fix sort field error bug
2019-10-23	Added the configuration of user authentication
2019-11-21	Fixed SQL query error with user authentication
2019-11-21	Optimize query results for assigning primary key fields for binding ESID annotations (to prevent null primary key field query results)
The 2020-1-1	Thanks for adding support for nested types. Please refer to the “Instructions for nested type support” and “Instructions for nested queries”jangojingIdeas provided
2020-09-09	Fixed several issues Added the ability for aliases to manipulate indexes Added scrolling index support
2020-09-10	Added support for geo latitude and longitude coordinates
2020-10-14	Upload to maven central repository, notice groupId changed to cn.zxporz
2020-11-05	Cardinality adds an option to the precision_threshold parameter
2021-01-30	Index configuration increases the maximum page depth of maxResultWindow Add the configuration of HTTP connection pools
2021-02-03	Optimization should not be added`@ESMapping`When annotating, datA_type is automatically determined by type
2021-02-04	Added the rollover function to automatically perform scheduled tasks
2021-02-09	Added the configuration of automatically suffixing indexName Added a configuration that does not automatically create indexes on startup
2021-03-09	Added method to clear scroll
2021-03-10	Add sample projects to use (constantly updated)demo
2021-04-07	Added highlighting of complex parameter Settingsissue
2021-04-09	Added non-Springboot integration demoESClientRHLSpring

What skills should you have before using it

springboot
maven
Basic concepts of ElasticSearch
Elasticsearch Rest API description and usage

This section describes the functions of the tool

Automatic proxy interfaces simplify development

Automatic proxy interface to simplify development

Index structure Management

Check whether the index exists
Index structure creation
Automatically customize index structure mapping
Drop index structure
Support for scrolling indexes

CRUD

LowLevelClient query
Adding index data
Add index data in batches
Add index data in batches
Overwrite updates index data
Partially update index data
Batch update index
Update indexes in batches
Delete index data
Check whether index data exists
Native queries
Supports uri query String queries
SQL query support
Customize query criteria
Supports customized query based on search criteria and maximum number of returned items
Supports custom queries for paging, highlighting, sorting, and query conditions
Advanced queries (support paging, highlighting, sorting, routing, specifying result fields, query conditions, search after)
Count query
Scroll query (for large data volume query)
Template query – Save template
Template query – Register template
Template query – Inline template
Search for completion suggest
Search suggestions phrase suggest
Query by ID
Mget query
Query by multiple indexes (2019-03-19 added)
Supports Alias query and Alias writable index switching

Aggregation query

Native aggregate query
Plain aggregate query
Group common aggregate query
Drill down (layer 2) to aggregate queries
Statistical aggregate query
Group statistics aggregate query
Base query
Percentage aggregate query
Percentile aggregated query
Filter aggregate query
Histogram aggregation query
Date histogram aggregation query

Tool source code structure introduction

annotation

Store annotations to simplify component usage

config

Springboot-based automation features, including automatic configuration of ES client components

auto

Automatic index structure management function Automatically generates interface proxies

enums

Enumeration of underlying data

index

Index structure management feature pack

repository

CURD+ converged feature pack

util

Internal tool kit

Begin to use

Use the sample

Gitee.com/zxporz/escl…

Instructions before use

The component is compiled based on JDK1.8, please note the choice of JDK version
Currently, only springboot integration is supported. To integrate with common Spring systems, simple modifications are required
Note If you do not need to use the built-in integration method of components, you only need to refer to the integration method with ESElasticsearchTemplateImplClass code

Maven rely on

The introduction of pom

<dependency> <groupId>cn. Zxporz </groupId> <artifactId>esclientrhl</artifactId> <version>7.0.0</version> </dependency>Copy the code

Note: It is recommended to add the elasticSearch version number to the poM file in the imported Springboot project, otherwise it may be overwritten by the Springboot parent project.

< the properties >... < elasticsearch version > 7.3.1 < / elasticsearch version >... </properties>Copy the code

The introduction of the component

Add the EnableESTools annotation

Add on the SpringBoot boot class

The @Enableestools annotations help developers automatically inject tool services, simplify configuration, and introduce the ability to automatically discover es index structure entity classes, recognize the ESCRepository interface, and automatically generate proxies

EnableESTools Note Usage: Usage 1: Most fully configured

@enableestools (basePackages = {"ESCRepository interface package path array "}, entityPath = {" Entity package path array "}, printregmSG = true)// See o&MCopy the code

Usage 2: Minimizes default scanning for all classes in the boot class package

@EnableESTools
Copy the code

Usage 3: Scan entity classpath only

@enableestools ({" entity class package path array "})Copy the code

Configure @enableestools on the SpringBoot boot class

@SpringBootApplication @EnableESTools public class EsdemoApplication { public static void main(String[] args) { SpringApplication.run(EsdemoApplication.class, args); }}Copy the code

Application. properties Adds the URI of the ElasticSearch service

Properties Specifies the URIs for elasticSearch. If multiple URIs exist, separate them by commas (,)

Elasticsearch. Host = 127.0.0.1:9200Copy the code

Application. Properties Adds elasticSearch HttpClient connection pool parameter Settings

Elasticsearch. Max_connect_per_route =10 # HTTP number of requests that a server can receive in parallel Connection_request_timeout_millis =2000 # select * from connetcion pool Connect_timeout_millis =2000 elasticSearch.socket_timeout_millis =30000Copy the code

Add elasticSearch username and password (based on es7+ only version of XPack)

None can be configured without

elasticsearch.username=elastic
elasticsearch.password=changeme
Copy the code

Using the component

ElasticsearchTemplate and elasticsearch dex are automatically injected into the spring-managed bean and the related API is called

@Service
public class CompletionSuggestServiceImpl implements CompletionSuggestService {
    @Autowired
    ElasticsearchTemplate<Book,String> elasticsearchTemplate;
Copy the code

If you just want to introduce some simple basic functions, you are recommended to use interface proxy, which makes it easier to define the interface. You only need to inherit the ESCRepository interface

public interface Main2Repository<Main2,String> extends ESCRepository<Main2,String> {
}
Copy the code

To use it, you only need to import the defined interface

@Autowired
Main2Repository main2Repository;
Copy the code

The method used is similar to ElasticsearchTemplate, except for some basic methods that remove the Clazz class type input

Main2 main2 = new Main2(); main2.setProposal_no("qq123549440"); Main2. SetBusiness_nature_name (" channels "); main2.setAppli_name("esclientrhl"); main2Repository.save(main2); System.out.println(main2Repository.getById("22222")); Map map2 = main2Repository.aggs("proposal_no", AggsType.count,null,"appli_name"); map2.forEach((o, o2) -> System.out.println(o + "=====" + o2));Copy the code

Note: If the automatic proxy interface is used, note the following:

The interface must inherit from ESCRepository, and the interface must be defined with the true type of the generic
The corresponding entity class must be annotated with ESMetaData for the component to automatically recognize it
The entity class name must be unique throughout the project; otherwise, proxy class generation will fail

Index management

Metadata Configuration

Metadata for customizing the entity class corresponding to the ES index structure

@ESMetaData(indexName = "index", number_of_shards = 5,number_of_replicas = 0)
Copy the code

The main configuration information and default values are as follows

*/ String[] searchIndexNames() default {}; */ String[] searchIndexNames(); /** * indexName, must be set */ String indexName(); /** * indexType() default ""; /** * indexType() default ""; /** * number of fragments */ int number_of_shards() default 5; /** * Number of backup fragments */ int number_of_replicas() default 1; /** * Whether to print logs * @return */ Boolean printLog() default false; * indexName = aliasName * @return */ Boolean alias() default false; /** * The current configuration only takes effect when alias is configured but rollover is not configured * Note: all configured indexes must exist * @return */ String[] aliasIndex() default {}; /** * This configuration only takes effect when alias is configured but rollover is not configured * Note: The configured index must exist in aliasIndex * @return */ String writeIndex() default ""; /** * when rollover is set to true, rollover is enabled (and other aliases are ignored) * aliasName is indexName * indexName specifications are: IndexName - YYYY.mm. Dd-00000n * The index scrolling generation policy is as follows * @return */ Boolean Rollover () default false; /** * Automatic rollover configuration * automatic Rollover switch * @return */ Boolean autoRollover() default false; * @return */ long autoRolloverInitialDelay() default 0L; /* long autoRolloverInitialDelay() default 0L;  * @return */ long autoRolloverPeriod() default 4L; long autoRolloverPeriod() default 4L; /** * Automatically perform rollover related configuration * unit time configuration, * @return */ TimeUnit autoRolloverTimeUnit() default timeunit.hours; / * * * after the current index over the configuration of time to generate new index * @ return * / long rolloverMaxIndexAgeCondition () default 0 l; / * * * with rolloverMaxIndexAgeCondition combined, Corresponding unit rolloverMaxIndexAgeCondition * @ return * / TimeUnit rolloverMaxIndexAgeTimeUnit () the default TimeUnit. DAYS; / * * * the current index documents more than Numbers of the configuration generated new index after * @ return * / long rolloverMaxIndexDocsCondition () default 0 l; / * * * after the current index size over the configuration of digital generated new index * @ return * / long rolloverMaxIndexSizeCondition () default 0 l; / * * * with rolloverMaxIndexSizeCondition combined, Corresponding unit rolloverMaxIndexSizeCondition * @ return * / ByteSizeUnit rolloverMaxIndexSizeByteSizeUnit () the default ByteSizeUnit. GB;  /** * Max page depth * @return */ long maxResultWindow() default 10000L; /** * Index names automatically include suffix * @return */ Boolean suffix() default false; /** * Whether to automatically create index * @return */ Boolean autoCreateIndex() default true;Copy the code

Index structure configuration

Used to customize the index structure of the entity class corresponding to the ES index structure to simplify index creation. Configure annotations on the entity class Field to identify information about the elasticSearch index structure fields that field corresponds to

@ESID
private String proposal_no;
@ESMapping(datatype = DataType.keyword_type)
private String risk_code;
@ESMapping(datatype = DataType.text_type)
private String risk_name;
Copy the code

@esID identifies the ES primary key (which automatically corresponds to the ES index data _id field). Note that the primary key type must be the same as the second generic of ElasticsearchTemplate

@esMapping Indicates information about the ES index structure field corresponding to the field

/** * DataType (including keyword type) */ DataType DataType () default DataType. Text_type; /** * Boolean keyword() default true; /** * keyword ignore word count */ int ignore_above() default 256; /** * Whether to support ngram, efficient full text search prompt (custom gram segmentation, Please refer to the official https://www.elastic.co/guide/en/elasticsearch/reference/7.x/analysis-ngram-tokenizer.html) * / Boolean ngram () default false; */ Boolean suggest() default false; /** * Index tokenizer Settings (research type) */ Analyzer () default Analyzer.standard; /** * Search content Analyzer set */ search_Analyzer () default Analyzer. Standard; /** * whether to allow search */ Boolean allow_search() default true; /** * replace _all */ String copy_to() default ""; /** * null_value specifies that the default null_value string will not be added to the mapping. Null_value * @return */ * Null_value * @return */ String null_value() default ""; /** * Nested corresponding type, default is Object.class. * Generate nested type indexes with annotations that need to be added only if DataType is nested_type * for example:  * @ESMapping(datatype = DataType.nested_type, nested_class = EsFundDto.class) * * @return */ Class nested_class() default Object.class;Copy the code

If you do not have high requirements on field types, the component supports automatic adaptation to mapping

Automatically create index structure mapping based on the configuration information

When the project starts, the component automatically recognizes the @esMetadata annotation configured on the ES entity class. If the corresponding index structure is not created, it automatically creates the corresponding index structure based on the Mapping annotation configuration.

If the entity class is not in the package path of the startup class, you need to configure the entity classpath on the startup annotation to enable this feature.

@EnableESTools(entityPath = "com.*.esdemo.domain")
Copy the code

GEO type

www.elastic.co/guide/en/el…

If you want to support latitude and longitude search, you need to add a latitude and longitude field

    @ESMapping(datatype = DataType.geo_point_type)
    GeoEntity geo;
Copy the code

The data_type is DataType. Geo_point_type, and the entity class is fixed to GeoEntity

The reference code is as follows:

GeoEntity gp1 = new GeoEntity(1,1);
GeoPojo g1 = new GeoPojo();
g1.setGeo(gp1);
g1.setPlace("1");
g1.setUserId(1L);
g1.setUserName("1");
elasticsearchTemplate.save(Arrays.asList(g1));
GeoPoint topLeft = new GeoPoint(32.030249,118.789703);
        GeoPoint bottomRight = new GeoPoint(32.024341,118.802171);
        GeoBoundingBoxQueryBuilder geoBoundingBoxQueryBuilder =
                QueryBuilders.geoBoundingBoxQuery("geo")
                .setCorners(topLeft,bottomRight);
        List<GeoPojo> search = elasticsearchTemplate.search(geoBoundingBoxQueryBuilder, GeoPojo.class);
Copy the code

A description of nested type support

Indexes containing nested member variables do not support partial field updates (override updates are supported) variables in a nested object do not support aggregate queries Support automatic creation of mapping by nested annotations. However, at present, only one layer of mapping is supported and complex mapping configurations (only type configurations are supported) are not supported.

In the configuration example (list mode below, but non-list mode is also supported), datatype needs to specify the nesteD_TYPE enumeration type as well as the nested class type

@ESMapping(datatype = DataType.nested_type,nested_class = Actors.class)
private List<Actors> actors;
Copy the code

You can configure @esMapping in Actors

public class Actors {
    @ESMapping
    private String first_name;
    @ESMapping
    private String last_name;
Copy the code

The difference between ngram completion suggest and phrase suggest

ngram

Ngram requires custom setting and Mapping
Still use inverted index, the letter or word carry in the word form inverted index (edge ngram), can be efficient prefix retrieval, can be retrieved from the middle word
Use match_PHRASE for retrieval

completion suggest

Completion Suggest requires customization of mapping
Did not use inverted index, but used inverted index dictionary FST data structure and cache and memory, has very good performance, only strict prefix search, do not support the middle word search
Retrieval requires a separate API

phrase suggest

Phrase Suggest does not require customization
It is not as good as completion suggest, which focuses on matching similar words (incorrectly typed words), and provides search suggestions based on relatively vague input terms to provide a better search experience and a basis for preparing the next retrieval
Retrieval requires a separate API

Manually create or drop index structures

elasticsearchIndex.dropIndex(Main2.class);
elasticsearchIndex.createIndex(Main2.class);
Copy the code

Check whether the index exists

@param clazz passes the entity Class corresponding to the ES index structure * @throws Exception */ public Boolean exists(Class<T> clazz) throws Exception;Copy the code

elasticsearchIndex.exists(Main2.class)
Copy the code

Create index Mapping using JSON

@param Settings Settings map information @param settingsList Settings Map information * @param mappingJson mapping JSON * @param indexName indexName * @throws Exception */ public void createIndex(Map<String,String> Settings,Map<String,String[]> settingsList,String mappingJson,String indexName) throws Exception;Copy the code

Configuring not to create indexes automatically (at startup)

Configure autoCreateIndex in ESMetaData annotations this configuration is false by default, that is, indexes are not created automatically

@ESMetaData(indexName = "main10", number_of_shards = 5,number_of_replicas = 0,autoCreateIndex = false
)
public class Main10
Copy the code

Configure automatic basis for index names`suffix`Configure suffixes for index names

This configuration is not valid for searchIndexNames

Set suffix to true in the ESMetaData annotation, which defaults to false when set to true, Applciation. Yml need to increase the configure elasticsearch. Index. The suffix = dev will automatically set to indexName_dev indexName, not configure this does not automatically add suffix for trueindexName suffix

@ESMetaData(indexName = "main10", number_of_shards = 5,number_of_replicas = 0,suffix = true
)
public class Main10
Copy the code

elasticsearch.index.suffix=dev
Copy the code

Alias operation

When you need to operate on an Alias, you can set Alias to true in the @esmetadata annotation. The index must be created in advance, and the Alias is automatically created. AliasIndex is the list of indexes bound to the Alias. Otherwise, the alias cannot be created and writeIndex must be set. WriteIndex is the name of the index currently written in the index list bound to the alias. Note that this index name must not only exist but also exist in the aliasIndex configuration

The indexName configured in @esmetadata is actually the alias name

For example, in the following code, the alias is testAlias, the bound index names are testalia-01, testalia-02, testalia-03, and the write operation corresponds to testalia-02

@ESMetaData(indexName = "testalias",
        number_of_shards = 1,
        number_of_replicas = 0,
        printLog = true,
        alias = true,
        aliasIndex = {"testalias-01","testalias-02","testalias-03"},
        writeIndex = "testalias-02")
public class TestAlias {


}
Copy the code

If frequent modification or deletion operations are required, using Alias is cumbersome because only one index can be configured. If other indexes need to be written, the switchAliasWriteIndex method needs to be called to switch the index that can be written

/** * Switch Alias to write index * @param clazz * @throws Exception */ public void switchAliasWriteIndex(Class<T> clazz,String writeIndex) throws Exception;Copy the code

Rollover Indicates the scrolling index operation

Configure your POJOs as follows when you need to automatically generate new indexes according to some index rolling build strategy and manipulate them uniformly through alias

in@ESMetaDataConfiguration in annotationsrolloverfortrue
rolloverMaxIndexAgeConditionA new index is generated when the current index exceeds the specified time
rolloverMaxIndexAgeTimeUnitCombining with rolloverMaxIndexAgeCondition, the corresponding rolloverMaxIndexAgeCondition units
rolloverMaxIndexDocsConditionWhen the number of current index documents exceeds this number, a new index is generated
rolloverMaxIndexSizeConditionIf the current index size exceeds the specified number, a new index is generated
rolloverMaxIndexSizeByteSizeUnitCombining with rolloverMaxIndexSizeCondition, the corresponding rolloverMaxIndexSizeCondition units
The name of the automatically generated index isindexName-yyyy-mm-dd-00000n
@ESMetaDataIndexName is the alias name

For example, the following code, alias testRollover2, automatically generates a new index when the number of indexes exceeds two. The name of the index is testRollover2 – current date-000001

@ESMetaData(indexName = "testrollover2",
        number_of_shards = 1,
        number_of_replicas = 0,
        printLog = true,
        rollover = true,
        rolloverMaxIndexDocsCondition = 2)
public class TestRollover {
Copy the code

You are advised not to modify (save the same ID) or delete rollover operations, but only add and query operations. Because the current writable index can only be the latest index, the modified or deleted data may not exist and various unexpected situations may occur

If you need to customize the rollover automatic execution policy, perform the following configuration. However, this method is not a good choice for distributed cluster scenarios, which may cause multiple Rollover tasks to run simultaneously

in@ESMetaDataConfiguration in annotationsautoRolloverfortrueThis option does not automatically run Rollover after saving index data
autoRolloverInitialDelayAutoRolloverInitialDelay after the project starts
autoRolloverPeriodAutoRolloverPeriod is executed every interval after the project starts
autoRolloverPeriodAutoRolloverPeriod is executed every interval after the project starts
By default, it runs every 4 hours

CRUD function description

LowLevelClient query

In this case, you can directly send JSON packets to query or operate the ES server. This method does not do too much encapsulation, and basically retains the native entry and exit parameters

// Autowired ElasticsearchTemplate ElasticsearchTemplate; Request Request = new Request("GET","/esdemo/_search"); Request. SetEntity (new NStringEntity (" {\ "query \" : {\ "match_all \" : {\ "boost \" : 1.0}}} ", ContentType. APPLICATION_JSON)); Response response = elasticsearchTemplate.request(request); RequestLine requestLine = response.getRequestLine(); HttpHost host = response.getHost(); int statusCode = response.getStatusLine().getStatusCode(); Header[] headers = response.getHeaders(); String responseBody = EntityUtils.toString(response.getEntity()); System.out.println(responseBody);Copy the code

Adding index data

Unless otherwise specified, the following queries inject utility classes by default. The first generic of utility classes is the es index structure class to be operated on, and the second generic is the primary key class for the index

@Autowired
ElasticsearchTemplate<Main2,String> elasticsearchTemplate;
Copy the code

The primary key of type Main2 is String

@ESMetaData(indexName = "index", number_of_shards = 5,number_of_replicas = 0)
public class Main2 implements Serializable {
    private static final long serialVersionUID = 1L;
    @ESID
    private String proposal_no;
Copy the code

Main2 main = new Main2();
main.setProposal_no("main2");
main.setAppli_code("123");
main.setAppli_name("456");
elasticsearchTemplate.save(main);
Copy the code

You can customize the Routing field to specify which shard data is indexed to

Main2 main2 = new Main2();
main2.setProposal_no("qq360");
main2.setAppli_name("zzxxpp");
elasticsearchTemplate.save(main2, "R01");
Copy the code

Add index data in batches

List<Main2> list = new ArrayList<>();
Main2 main1 = new Main2();
main1.setProposal_no("main1");
main1.setAppli_code("123");
main1.setAppli_name("456");
Main2 main2 = new Main2();
main2.setProposal_no("main2");
main2.setAppli_code("123");
main2.setAppli_name("456");
Main2 main3 = new Main2();
main3.setProposal_no("main3");
main3.setAppli_code("123");
main3.setAppli_name("456");
list.add(main1);
list.add(main2);
list.add(main3);
elasticsearchTemplate.save(list);
Copy the code

Add index data in batches

Compared with batch adding index data, batch adding index data takes into account the memory bottleneck of es server batch adding index data. The incoming data list is split and indexed sequentially according to some simple strategies

You can modify the org. ZXP. Esclientrhl. Util. Constant configuration in the class the following variables, combined with the best practices, the default in 5000 as a batch

Public static int BULK_COUNT = 5000; public static int BULK_COUNT = 5000;Copy the code

Batch adding index data provides a new approach:

Main2 main1 = new Main2();
main1.setProposal_no("aaa");
main1.setBusiness_nature_name("aaaaaa2");
Main2 main2 = new Main2();
main2.setProposal_no("bbb");
main2.setBusiness_nature_name("aaaaaa2");
Main2 main3 = new Main2();
main3.setProposal_no("ccc");
main3.setBusiness_nature_name("aaaaaa2");
Main2 main4 = new Main2();
main4.setProposal_no("ddd");
main4.setBusiness_nature_name("aaaaaa2");
elasticsearchTemplate.saveBatch(Arrays.asList(main1,main2,main3,main4));
Copy the code

Partially update index data

Main2 main1 = new Main2();
main1.setProposal_no("main1");
main1.setInsured_code("123");
elasticsearchTemplate.update(main1);
Copy the code

Overwrite updates index data

Main2 main1 = new Main2();
main1.setProposal_no("main1");
main1.setInsured_code("123");
elasticsearchTemplate.updateCover(main1);
Copy the code

Partial updates differ from override updates in that partial updates only update set field values

Batch update index

/** * * * @param queryBuilder * @Param limitCount * @Param Clazz * @Param limitCount * @Param Clazz * @Param LimitCount The value cannot exceed LimitCount * @Param ASyn true Asynchronous processing; otherwise, synchronous processing * @Return * @throws Exception */ public BulkResponse batchUpdate(QueryBuilder queryBuilder, T t, Class clazz, int limitcount, boolean asyn) throws Exception;Copy the code

Main2 main1 = new Main2(); main1.setSum_amount(1000); elasticsearchTemplate.batchUpdate(QueryBuilders.matchQuery("appli_name","123"),main1,Main2.class,30, true);Copy the code

Batch update indexes do not support overwrite updates

Update indexes in batches

For details about how to adjust batch policies, see section “Adding Index Data by Batch”

Main2 main1 = new Main2();
main1.setProposal_no("aaa");
main1.setBusiness_nature_name("aaaaaa2");
Main2 main2 = new Main2();
main2.setProposal_no("bbb");
main2.setBusiness_nature_name("aaaaaa2");
Main2 main3 = new Main2();
main3.setProposal_no("ccc");
main3.setBusiness_nature_name("aaaaaa2");
Main2 main4 = new Main2();
main4.setProposal_no("ddd");
main4.setBusiness_nature_name("aaaaaa2");
elasticsearchTemplate.bulkUpdateBatch(Arrays.asList(main1,main2,main3,main4));
Copy the code

Delete index data

Main2 main1 = new Main2(); main1.setProposal_no("main1"); main1.setInsured_code("123"); elasticsearchTemplate.delete(main1); / / delete by ID elasticsearchTemplate deleteById (" main1, "Main2. Class);Copy the code

Custom routing Delete the routing information

/ / can only be deleted successfully elasticsearchTemplate routing information correctly. The delete (main2, "R02"); / / whether or not the index data is specified the routing can delete elasticsearchTemplate. Delete (main2);Copy the code

Delete index data based on query conditions

elasticsearchTemplate.deleteByCondition(QueryBuilders.matchQuery("appli_name","2"),Main5.class);
Copy the code

Check whether index data exists

Main2 main1 = new Main2();
main1.setProposal_no("main1");
main1.setInsured_code("123");
boolean exists = elasticsearchTemplate.exists("main1",Main2.class);
System.out.println(exists);
Copy the code

Native queries

SearchRequest is the official native query input, which is used when the tool cannot meet requirements

SearchRequest searchRequest = new SearchRequest(new String[]{"index"});
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(new MatchAllQueryBuilder());
searchSourceBuilder.from(0);
searchSourceBuilder.size(10);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = elasticsearchTemplate.search(searchRequest);

SearchHits hits = searchResponse.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
    Main2 t = JsonUtils.string2Obj(hit.getSourceAsString(), Main2.class);
    System.out.println(t);
}
Copy the code

Supports uri query String queries

(in the form of a uri + parameters query string) query and returns the API usage reference official documentation: www.elastic.co/guide/en/el…

/ / q = "aaa" contained in the query field aaa matching results List < Main2 > List = elasticsearchTemplate. SearchUri (q = "aaa" Main2. Class); List<Main2> List =sum_premium:100 elasticsearchTemplate.searchUri("q=sum_premium:100",Main2.class);Copy the code

Note: Some advanced URI queries, such as range queries, may not be available

SQL query support

The query method that passes SQL statements into the method and returns them in various forms (the method returns text) is worth noting that this method does not support automatic index recognition, requiring that the index name after from be defined correctly (case sensitive) and does not support automatic generic conversion

. / / all the query String result = elasticsearchTemplate queryBySQL (" SELECT * FROM the index ORDER BY sum_premium DESC LIMIT 5 ", SqlFormat.TXT); / / query count String result = elasticsearchTemplate. QueryBySQL (" SELECT count (*) FROM the index ", SqlFormat. TXT); . / / grouping query String result = elasticsearchTemplate queryBySQL (" SELECT risk_code, sum (sum_premium) FROM the index group by risk_code", SqlFormat.TXT);Copy the code

The second argument to this method is the enumeration type that returns the result, as listed below:

CSV("csv","text/csv"),
JSON("json","application/json"),
TSV("tsv","text/tab-separated-values"),
TXT("txt","text/plain"),
YAML("yaml","application/yaml"),
CBOR("cbor","application/cbor"),
SMILE("smile","application/smile");
Copy the code

Customize query criteria

@param queryBuilder * @Param Clazz generic class type * @return generic es index structure entity class query result set * @throws Exception */ public List<T> search(QueryBuilder queryBuilder, Class<T> clazz) throws Exception;Copy the code

/ / here is simple by matchall inquiry (all) in a way that shows some examples of / / criteriabuilder will introduce a List in a separate chapter < Main2 > main2List = elasticsearchTemplate. Search (new MatchAllQueryBuilder(),Main2.class); main2List.forEach(main2 -> System.out.println(main2));Copy the code

Supports customized query based on search criteria and maximum number of returned items

/** * unpaginated query, Specifies the maximum number of returned items * @param queryBuilder * @Param limitSize Specifies the maximum number of returned items * @Param clazz * @return * @throws Exception */ public List<T> searchMore(QueryBuilder queryBuilder,int limitSize, Class<T> clazz) throws Exception;Copy the code

/ / return 7 most data List < Main2 > main2List = elasticsearchTemplate. SearchMore (new MatchAllQueryBuilder (), 7, Main2. Class); System.out.println(main2List.size()); main2List.forEach(main2 -> System.out.println(main2));Copy the code

Proxy interface usage:

/ / returns at most six data List < Main2 > ll = main2Repository. SearchMore (new MatchAllQueryBuilder (), 6); System.out.println(ll.size()); ll.forEach(s -> System.out.println(s));Copy the code

Supports custom queries for paging, highlighting, sorting, and query conditions

@param pageSortHighLight supports paging, sorting, and highlighting queries @Param pageSortHighLight supports paging, sorting, and highlighting queries @Param Clazz generic class types * @return Generic definition es index structure entity class query result set * @throws Exception */ public PageList<T> Search (QueryBuilder QueryBuilder, PageSortHighLight pageSortHighLight, Class<T> clazz) throws Exception;Copy the code

Int currentPage = 1; int pageSize = 10; PSH = new PageSortHighLight(currentPage,pageSize); Keyword String sorter = "proposal_no. Keyword "; Sort.Order order = new Sort.Order(SortOrder.ASC,sorter); psh.setSort(new Sort(order)); Psh.sethighlight (new HighLight().field("risk_code"))); // New HighLight().setpretag ("<em>"); //new HighLight().setPostTag("</em>"); PageList<Main2> pageList = new PageList<>(); pageList = elasticsearchTemplate.search(new MatchAllQueryBuilder(), psh, Main2.class); pageList.getList().forEach(main2 -> System.out.println(main2));Copy the code

Advanced query

Advanced queries include custom queries for paging, sorting, and highlighting, as well as custom queries for specifying return result fields and routing queries

Set paging, sorting, and highlighting information in advanced queries

PageSortHighLight psh = new PageSortHighLight(1, 50);
Attach attach = new Attach();
attach.setPageSortHighLight(psh);
elasticsearchTemplate.search(QueryBuilders.termQuery("proposal_no", "qq360"), attach, Main2.class)
.getList().forEach(s -> System.out.println(s));
Copy the code

Specify return fields

Attach attach = new Attach(); // Returns a result containing only the proposal_no field String[] includes = {"proposal_no"}; attach.setIncludes(includes); elasticsearchTemplate.search(QueryBuilders.termQuery("proposal_no", "qq360"), attach, Main2.class);Copy the code

You can also customize the following return policies

// Except business_nature_NAME Field, other Mandatory String[] Excludes = {"business_nature_name"}; // Return a field starting with risk. String[] includes = {"risk*"}; Attach attach = new Attach(); attach.setIncludes(includes); attach.setExcludes(includes); elasticsearchTemplate.search(QueryBuilders.termQuery("proposal_no", "qq360"), attach, Main2.class);Copy the code

Query routes by route name

Attach Attach = new Attach(); attach.setRouting("R01"); elasticsearchTemplate.search(QueryBuilders.termQuery("proposal_no", "qq360"), attach, Main2.class) .getList().forEach(s -> System.out.println(s));Copy the code

TrackTotalHits support does not count hits if the number of matches is greater than 10,000. This is to avoid the unnecessary overhead of calculating exactly matching documents for a given query. We can by track_total_hits = true as a request parameter to force accurately matching calculation of www.elastic.co/guide/en/el…

Attach attach = new Attach();
attach.setTrackTotalHits(true);
elasticsearchTemplate.search(QueryBuilders.termQuery("proposal_no", "qq360"), attach, Main2.class)
.getList().forEach(s -> System.out.println(s));
Copy the code

The search after the query

Search after mainly solved the elasticsearch deep paging (deep paging), but only from the first page turn back from page to page www.elastic.co/guide/en/el…

Attach attach = new Attach(); // [must] setSearchAfter mode attach. SetSearchAfter (true); // [optional] Set PageSortHighLight PageSortHighLight = new PageSortHighLight(1, 10); String sorter = "sum_amount"; Sort.Order order = new Sort.Order(SortOrder.ASC,sorter); pageSortHighLight.setSort(new Sort(order)); attach.setPageSortHighLight(pageSortHighLight); PageList page = elasticsearchTemplate.search(new MatchAllQueryBuilder(),attach,Main2.class); Page.getlist ().foreach (s -> system.out.println (s)); Object[] sortValues = page.getSortValues(); While (true) {attach. SetSortValues (sortValues); page = elasticsearchTemplate.search(new MatchAllQueryBuilder(),attach,Main2.class); // If (page.getList()! = null && page.getList().size() ! = 0) { page.getList().forEach(s -> System.out.println(s)); sortValues = page.getSortValues(); } else { break; }}Copy the code

Count query

The amount of data combined with the query criteria

long count = elasticsearchTemplate.count(new MatchAllQueryBuilder(),Main2.class);
System.out.println(count);
Copy the code

Scroll to query

In order to prevent memory overflow caused by large amount of data, the Scroll method is divided into multi-step execution

/** * scroll Create scroll * @param queryBuilder * @param clazz * @param time * @param size * @return * @throws Exception */ public ScrollResponse<T> createScroll(QueryBuilder queryBuilder, Class<T> clazz, Long time, Integer size) throws Exception; /** * scroll Create scroll * @param queryBuilder * @param clazz * @param time * @param size * @param indexs * @return * @throws Exception */ public ScrollResponse<T> createScroll(QueryBuilder queryBuilder, Class<T> clazz, Long time, Integer size , String... indexs) throws Exception;Copy the code

/ / create the scroll and the first batch of data obtained ScrollResponse < Main2 > ScrollResponse = elasticsearchTemplate. CreateScroll (new MatchAllQueryBuilder (),  Main2.class, 1L, 100); scrollResponse.getList().forEach(s -> System.out.println(s)); String scrollId = scrollResponse.getScrollId(); / / get other batches of data through scrollId while (true) {scrollResponse = elasticsearchTemplate. QueryScroll (Main2. Class, 1 l, scrollId); if(scrollResponse.getList() ! = null && scrollResponse.getList().size() ! = 0){ scrollResponse.getList().forEach(s -> System.out.println(s)); scrollId = scrollResponse.getScrollId(); }else{ break; }Copy the code

/** * clearScroll snapshot * @param scrollId * @return * @throws Exception */ public ClearScrollResponse clearScroll(String... scrollId) throws Exception; ClearScrollResponse clearScrollResponse = elasticsearchTemplate.clearScroll(scroll.getScrollId()); System.out.println(clearScrollResponse.isSucceeded());Copy the code

The following two methods are not recommended to avoid memory overflow

/** * scroll * @param queryBuilder query criteria * @Param Clazz class type * @Param time Retention hours * @return * @throws Exception */ @Deprecated public List<T> scroll(QueryBuilder queryBuilder, Class<T> clazz, Long time) throws Exception; /** * scroll for large volume of data query, The default retention time is 2 hours (constant.default_scroll_time) * @param queryBuilder * @Param clazz * @return * @throws Exception */ @Deprecated public List<T> scroll(QueryBuilder queryBuilder, Class<T> clazz) throws Exception;Copy the code

/ / the default scroll mirror for two hours the List < Main2 > main2List = elasticsearchTemplate. Scroll (new MatchAllQueryBuilder (), Main2. Class); main2List.forEach(main2 -> System.out.println(main2)); Keep 5 hours / / / / specified scroll image List < Main2 > main2List = elasticsearchTemplate. Scroll (new MatchAllQueryBuilder (), Main2. Class, 5 l);Copy the code

Template query – Save template

/** * Save Template * @param templateSource * @param templateSource * @return */ public Response saveTemplate(String templateName,String templateSource) throws Exception;Copy the code

Save a template named tempdemo1, whose contents are shown in the variable templatesource value

String templatesource = "{\n" +
"  \"script\": {\n" +
"    \"lang\": \"mustache\",\n" +
"    \"source\": {\n" +
"      \"_source\": [\n" +
"        \"proposal_no\",\"appli_name\"\n" +
"      ],\n" +
"      \"size\": 20,\n" +
"      \"query\": {\n" +
"        \"term\": {\n" +
"          \"appli_name\": \"{{name}}\"\n" +
"        }\n" +
"      }\n" +
"    }\n" +
"  }\n" +
"}";
elasticsearchTemplate.saveTemplate("tempdemo1",templatesource);
Copy the code

Template query – Register template

/** * Template; Template has been saved in the script directory * @param template_params Template parameter * @param templateName Template name * @param clazz * @return */ public List<T> searchTemplate(Map<String, Object> template_params,String templateName,Class<T> clazz) throws Exception;Copy the code

Apply the query to the template tempdemo1 saved above

Map param = new HashMap();
        param.put("name","123");
        elasticsearchTemplate.searchTemplate(param,"tempdemo1",Main2.class).forEach(s -> System.out.println(s));
Copy the code

Template query – Inline template

/** * Template; * @param template_params template_params * @param templateSource * @param clazz * @return */ public List<T> searchTemplateBySource(Map<String, Object> template_params,String templateSource,Class<T> clazz) throws Exception;Copy the code

Directly upload template content for query

Map param = new HashMap();
param.put("name","123");
String templatesource = "{\n" +
"      \"query\": {\n" +
"        \"term\": {\n" +
"          \"appli_name\": \"{{name}}\"\n" +
"        }\n" +
"      }\n" +
"}";
elasticsearchTemplate.searchTemplateBySource(param,templatesource,Main2.class).forEach(s -> System.out.println(s));
Copy the code

Search for Completion Suggester suggestions

The search suggestion function can prompt the content to be searched quickly (see Baidu search function). Set the suggest property to true in the search suggestion field

/** * search suggestion method * @param fieldName The field to search for (set the field mapping configuration to true) * @param fieldValue The content to search for * @param clazz * @return Returns the search result List. The default number of search suggestions is 10. * @throws Exception */ public List<String> completionSuggest(String fieldName,String fieldValue,Class<T> clazz) throws Exception;Copy the code

List<String> list = elasticsearchTemplate.completionSuggest("appli_name", "1", Main2.class);
list.forEach(main2 -> System.out.println(main2));
Copy the code

@ESMapping(suggest = true)
private String appli_name;
Copy the code

Search suggestions phrase suggest

Phrace Suggester * @param fieldName * @param fieldValue * @param param Suggester Have customized the Phrace Suggester parameter * @param clazz * @return * @throws Exception */ public List<String> phraseSuggest(String fieldName, String fieldValue, ElasticsearchTemplateImpl.PhraseSuggestParam param, Class<T> clazz) throws Exception;Copy the code

ElasticsearchTemplateImpl.PhraseSuggestParam param = new ElasticsearchTemplateImpl. PhraseSuggestParam (5, 1, null, "always"); elasticsearchTemplate.phraseSuggest("body", "who is good boy zhangxinpen may be a goop",param, Sugg.class).forEach(s -> System.out.println(s)); // You can use the default parameters, Param into the null elasticsearchTemplate. PhraseSuggest (" body ", "who is the good boy zhangxinpen may be a goop", null, Sugg.class).forEach(s -> System.out.println(s));Copy the code

How to define the parameters as shown in the official document: www.elastic.co/guide/en/el…

Query by ID

/** * select * from @param ID; Corresponding to the second generic type * @param clazz * @return * @throws Exception */ public T getById(M ID, Class<T> clazz) throws Exception;Copy the code

Main2 main2 = elasticsearchTemplate.getById("main2", Main2.class);
System.out.println(main2);
Copy the code

Mget query

@param ids primary key array * @param clazz * @return * @throws Exception */ public List<T> mgetById(M[] ids, Class<T> clazz) throws Exception;Copy the code

String[] list = {"main2","main3"};
List<Main2> listResult = elasticsearchTemplate.mgetById(list, Main2.class);
listResult.forEach(main -> System.out.println(main));
Copy the code

QueryBuilder common usage demo

Precise query

// Exact queries require the keyword attribute (true by default), Keyword QueryBuilder QueryBuilder = QueryBuilders. TermQuery ("appli_name. Keyword ","456"); List<Main2> list = elasticsearchTemplate.search(queryBuilder,Main2.class); list.forEach(main2 -> System.out.println(main2)); Keyword is not required if the field type is keywordCopy the code

The phrase query

My hero / / / / China must query conditions of adjacent criteriabuilder criteriabuilder. = QueryBuilders matchPhraseQuery (" appli_name ", "the good");Copy the code

Relevance query

// SloP set to 2, And most men can move two times and complete matching criteriabuilder criteriabuilder. = QueryBuilders matchPhraseQuery (" appli_name ", "male"). The slop (2);Copy the code

Range queries

QueryBuilder queryBuilder = QueryBuilders.rangeQuery("sum_premium").from(1).to(3);
Copy the code

The full text matching

QueryBuilder QueryBuilder = QueryBuilders. MatchQuery ("appli_name"," middle man ");Copy the code

MinimumShouldMatch Minimum match parameters

// The minimum number of matching words is 75%, QueryBuilder QueryBuilder = QueryBuilders. MatchQuery ("appli_name"," appli_name"). MinimumShouldMatch ("75%");Copy the code

Match query integrates fuzzy error correction query

QueryBuilder queryBuilder = QueryBuilders.matchQuery("appli_name","spting");
((MatchQueryBuilder) queryBuilder).fuzziness(Fuzziness.AUTO);
Copy the code

Match Query Sets the logical relationship of query conditions

OR QueryBuilder QueryBuilder = QueryBuilders. MatchQuery ("appli_name","spring SPS "). Operator (operator.and);Copy the code

Fuzzy error correction query

QueryBuilder QueryBuilder = QueryBuilders. FuzzyQuery ("appli_name","spting");Copy the code

Boost weight Settings

// Query result appli_name = spring QueryBuilder queryBuilder1 = QueryBuilders. TermQuery ("appli_name.keyword","spring"). Boost (5); QueryBuilder queryBuilder2 = QueryBuilders.termQuery("appli_name.keyword","456").boost(3); QueryBuilder queryBuilder3 = QueryBuilders.termQuery("appli_name.keyword","123"); BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery(); queryBuilder.should(queryBuilder1).should(queryBuilder2).should(queryBuilder3);Copy the code

Prefix Prefix query

The shorter the prefix, the more doc to process, the worse the performance. As far as possible with the prefix search / / query appli_name prefix field value of 1 content criteriabuilder criteriabuilder. = QueryBuilders prefixQuery (" appli_name ", "1");Copy the code

Wildcard Query the wildcard

// Not recommended for poor performance //? : any character / / * : zero or any number of characters criteriabuilder criteriabuilder. = QueryBuilders wildcardQuery (" appli_name ", "1? 3 ");Copy the code

Regexp re – queries

/ / poor performance is not recommended criteriabuilder criteriabuilder. = QueryBuilders regexpQuery (" appli_name ", "[0-9] +"); //[0-9] : numbers in the specified range //[A-z] : letters in the specified range //. : one character //+ : The preceding regular expression can appear once or more timesCopy the code

Combined logic query

A combined logical query is a logical combination of multiple query methods, including must, should, and not mustNot

//select * from Main2 where (appli_name = 'spring' or appli_name = '456') and risk_code = '0101' and proposal_no ! = '1234567' QueryBuilder queryBuilder1 = QueryBuilders.termQuery("appli_name.keyword","spring"); QueryBuilder queryBuilder2 = QueryBuilders.termQuery("appli_name.keyword","456"); QueryBuilder queryBuilder3 = QueryBuilders.termQuery("risk_code","0101"); QueryBuilder queryBuilder4 = QueryBuilders.termQuery("proposal_no.keyword","1234567"); BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery(); queryBuilder.should(queryBuilder1).should(queryBuilder2); queryBuilder.must(queryBuilder3); queryBuilder.mustNot(queryBuilder4);Copy the code

Dis Max Query

Only the score of the condition with the highest matching degree is taken as the final score

// Assume that the first data is 0.8 for title and Bryant Fox, and 0.1 for body and Bryant Fox. The final score of this data is 0.8 // the final score of the second data is 0.6 for the title matching Bryant Fox and 0.7 for the body matching Bryant Fox. The final score of this data is 0.7. // If dis_max is not used, the score of the second data is 1.4, higher than the score of the first data is 0.9 // If other matching results are added, TieBreaker needs to be specified // get the best match _score // Multiply the score of other matching statements by tie_breaker // Sum and normalize the score above //Tier Breaker: a floating point number between 0 and 1 (0 means use the best match; 1 is for all the statement equally important) QueryBuilders. DisMaxQuery (). The add (QueryBuilders. MatchQuery (" title ", "Bryant Fox ").add(QueryBuilders. MatchQuery ("body"," Bryant Fox ")).tiebreaker (0.2f);Copy the code

Multi Match Query

Best match BEST_fields: same as Dis Max Query

QueryBuilders.multiMatchQuery("Quick pets", "Title", "body"). MinimumShouldMatch (" 20% "). The type (MultiMatchQueryBuilder. Type. BEST_FIELDS). The tieBreaker (0.2 f);Copy the code

Most_fields: Records that match more fields are preferred (no matter how much one of them matches)

PUT /mult/_doc/1 { "s1": "shanxi shanxi shanxi shanxi shanxi", "s2": "shanxi", "s3": "ttttt", "s4": "Brown rabbits are commonly seen." } PUT /mult/_doc/2 { "s1": "datong", "s2": "datong", "s3": "datong", "s4": "Brown Rabbits are commonly seen."} // Though Shanxi appears many times in S1, only two fields s1, S2, datong appears in three fields S123, So more priority fields of the record / / doc2 scores than doc1 QueryBuilders score. MultiMatchQuery (" shanxi datong ", "s1","s2","s3","s4") .type(MultiMatchQueryBuilder.Type.MOST_FIELDS);Copy the code

Cross_fields: the best overall match, which is similar to combining all fields into one field and searching the field to see who scored higher. Best practice: Replacing copy_to saves inversion space for a field

PUT /mult/_doc/3 { "s1": "sichuan sichuan", "s2": "sichuan", "s3": "eee", "s4": "sichuan" } PUT /mult/_doc/4 { "s1": "chengdu t chengdu chengdu", "s2": "rr", "s3": "ss", "s4": "Brown Rabbits are commonly seen."} // Although Chengdu appears 3 times in S1, together it is obvious that DOC3 matches better (TF is higher, document length is smaller, IDF). / / so doc3 scores higher QueryBuilders multiMatchQuery (Szechwan "chengdu sichuan", "s1","s2","s3","s4") .type(MultiMatchQueryBuilder.Type.CROSS_FIELDS);Copy the code

Function Score Query

www.elastic.co/guide/en/el… You can perform a series of recalculations for each matching document at the end of the query, sorting the score based on the newly generated score

// New score = old score * log(1 + factor*votes) // This is equivalent to weighting the contents of the data, not specifying the weighting value directly, but specifying the weighting strategy, and the fields in the data can directly affect the ScoreFunctionBuilder<? > scoreFunctionBuilder = ScoreFunctionBuilders .fieldValueFactorFunction("votes") Modifier (FieldValueFactorFunction. Modifier. LOG1P) factor (0.1 f); QueryBuilders.functionScoreQuery(QueryBuilders.matchQuery("title", "Bryant fox"),scoreFunctionBuilder).boostMode(combinefunction.multiply)// Default is.maxboost (3f);Copy the code

boosting Query

www.elastic.co/guide/en/el… Boosting query is a set of queries that customize a set of negative query criteria and set the criteria to reduce the degree of matching. Negative_boost (the second parameter to boostingQuery) determines the relevancy of the search results. ● Score relativity increases when boost is greater than 1 ● Score relativity decreases when boost is less than 1

QueryBuilders.boostingQuery(QueryBuilders.matchQuery("title", "bryant fox"), QueryBuilders.matchQuery("flag", "123")). NegativeBoost (0.2 f);Copy the code

Boosting Query, unlike Boost, has a fixed strategy

The filter

Filter queries need to be used in conjunction with Boolean queries, and the effect is no different from normal queries

For example, the relationship between must and filter is and

But filter queries don’t compute the correlation score first, whereas normal queries compute the correlation score and sort it by correlation pairs, so if you don’t need the correlation score, filter queries first

In addition, filter query has a built-in bitset caching mechanism, which can greatly improve query performance and reduce the probability of reverse index scanning. Therefore, in daily development, we need to carefully evaluate and actively use filter query

//select * from Main2 where appli_name = '456' and risk_code = '0101' QueryBuilder queryBuilder = Querybuilders.boolquery (). Must (querybuilders.termQuery ("appli_name. Keyword ","456")) Is more suitable for cache by filter. The filter (QueryBuilders. MatchPhraseQuery (" risk_code ", "0101")); List<Main2> list = elasticsearchTemplate.search(queryBuilder,Main2.class); list.forEach(main2 -> System.out.println(main2));Copy the code

More criteriabuilder see www.elastic.co/guide/en/el…

Nested query

NestedQueryBuilder queryBuilder = QueryBuilders.nestedQuery("actors",QueryBuilders.matchQuery("actors.first_name","DiCaprio1"), ScoreMode.Total); elasticsearchTemplate.search(queryBuilder, MyMovies.class).forEach(s -> {System.out.println(s); });Copy the code

NestedQueryBuilder needs to pass in a nestedPath

NestedQueryBuilder queryBuilder = QueryBuilders.nestedQuery("nestedPath",QueryBuilders.termsQuery("nestedPath.id",1),ScoreMode.Total);
Copy the code

Follow the multi-index query instructions

There are two ways to do multi-index queries

Configuring annotationssearchIndexNamesIf configured, the corresponding query method will query multiple indexes and return the result of the current POJI field. However, the configuration through annotations is not flexible, so it is not recommended for uncertain scenarios.

@ESMetaData(indexName = "main5",searchIndexNames = {"main5","index"}, number_of_shards = 5,number_of_replicas = 0,printLog = false) public class Main5 implements Serializable { // The query API calls do not change at allCopy the code

This method is more flexible and reliable than the annotation method. This method is recommended for cross-index query services. Currently, the multi-index entry API corresponding to common query has been added, and cross-index query of aggregate query will be added later

List< main6 > List = elasticsearchTemplate.search(QueryBuilders.matchAllQuery(),Main6.class,"main5","main6"); System.out.println(list.size()); List.foreach (main6 -> system.out.println (main6));Copy the code

This is a new example of an interface method that adds a variable long entry parameter at the end

@param queryBuilder @param clazz * @return * @throws Exception */ public List<T> search(QueryBuilder queryBuilder, Class<T> clazz,String... indexs) throws Exception;Copy the code

== It is recommended that the field consistency between multiple indexes be as high as possible ==

ScriptSortBuilder

Sort by custom script (random in this case)

Script script = new Script("Math.random()");
ScriptSortBuilder scriptSortBuilder = SortBuilders.scriptSort(script,ScriptSortBuilder.ScriptSortType.NUMBER).order(SortOrder.DESC);
SearchRequest searchRequest = new SearchRequest(new String[]{"index"});
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
searchSourceBuilder.query(new MatchAllQueryBuilder());
searchSourceBuilder.from(0);
searchSourceBuilder.size(10);
searchSourceBuilder.sort(scriptSortBuilder);
searchRequest.source(searchSourceBuilder);
SearchResponse searchResponse = elasticsearchTemplate2.search(searchRequest);

SearchHits hits = searchResponse.getHits();
SearchHit[] searchHits = hits.getHits();
for (SearchHit hit : searchHits) {
Main2 t = JsonUtils.string2Obj(hit.getSourceAsString(), Main2.class);
System.out.println(t);
}
Copy the code

Geo*QueryBuilder

GeoBoundingBoxQueryBuilder

The query returns the coordinates in the rectangle between the left vertex of topLeft and the bottomRight point of bottomRight

GeoPoint topLeft = new GeoPoint(32.030249,118.789703);
GeoPoint bottomRight = new GeoPoint(32.024341,118.802171);
GeoBoundingBoxQueryBuilder geoBoundingBoxQueryBuilder =
QueryBuilders.geoBoundingBoxQuery("geo")
.setCorners(topLeft,bottomRight);
List<GeoPojo> search = elasticsearchTemplate.search(geoBoundingBoxQueryBuilder, GeoPojo.class);
Copy the code

GeoDistanceQueryBuilder

The query returns the coordinates of the given latitude and longitude

QueryBuilders.geoDistanceQuery("pin.location")                             
    .point(40, -70)                                          
    .distance(200, DistanceUnit.KILOMETERS);  
Copy the code

GeoPolygonQueryBuilder

The query returns the coordinates of the range of the given latitude and longitude list

List<GeoPoint> points = new ArrayList<>();           
points.add(new GeoPoint(40, -70));
points.add(new GeoPoint(30, -80));
points.add(new GeoPoint(20, -90));
QueryBuilders.geoPolygonQuery("pin.location", points);
Copy the code

Aggregation query

Native aggregate query

/** * Aggregate base method, this method is less encapsulation, The purpose is to meet the aggregation of queries that tools cannot implement * @param aggregationBuilder aggregates content * @Param queryBuilder search criteria * @param clazz * @return aggregation results (official native aggregation results) *  @throws Exception */ public Aggregations aggs(AggregationBuilder aggregationBuilder, QueryBuilder queryBuilder, Class<T> clazz) throws Exception;Copy the code

SumAggregationBuilder aggregation = AggregationBuilders.sum("agg").field("sum_amount");
Aggregations aggregations = elasticsearchTemplate.aggs(aggregation,null,Main2.class);
Sum agg = aggregations.get("agg");
double value = agg.getValue();
System.out.println(value);
Copy the code

The more detail please refer to the official documentation www.elastic.co/guide/en/el…

The official all aggregation Builder please refer to www.elastic.co/guide/en/el…

Plain aggregate query

/** * The metric is aggstypes, which does not group, Sum, count, AVg, min, and Max of the metricName field * @param metricName Field to be analyzed * @param aggsType Sum, count, AVg, min, and Max types * @param queryBuilder query criteria, * @param clazz * @return Returns the result of sum, count, AVg, min, and Max. * @throws Exception */ public double aggs(String) metricName, AggsType aggsType, QueryBuilder queryBuilder, Class<T> clazz) throws Exception;Copy the code

double sum = elasticsearchTemplate.aggs("sum_premium", AggsType.sum,null,Main2.class); double count = elasticsearchTemplate.aggs("sum_premium", AggsType.count,null,Main2.class); double avg = elasticsearchTemplate.aggs("sum_premium", AggsType.avg,null,Main2.class); double min = elasticsearchTemplate.aggs("sum_premium", AggsType.min,null,Main2.class); // if translated into SQL: select max(sum_premium) from Main2 double max = elasticsearchTemplate.aggs("sum_premium", AggsType.max,null,Main2.class); System.out.println("sum===="+sum); System.out.println("count===="+count); System.out.println("avg===="+avg); System.out.println("min===="+min); System.out.println("max===="+max);Copy the code

Group common aggregate query

* @param bucketName * @param bucketName * @param bucketName * @param bucketName * @param bucketName * @param metricName Specifies the field to be analyzed * @param aggsType * @param clazz * @return */ public Map aggs(String) metricName, AggsType aggsType,QueryBuilder queryBuilder, Class<T> clazz, String bucketName) throws Exception;Copy the code

// if translated into SQL: select appli_name,max(sum_premium) from Main2 group by appli_name Map map = elasticsearchTemplate.aggs("sum_premium", AggsType.sum,null,Main2.class,"appli_name"); map.forEach((k,v) -> System.out.println(k+" "+v));Copy the code

== By default, the aggregation results are sorted in descending order ==

Drill down (layer 2) to aggregate queries

/** * Drill down aggregate query (no sort default policy), group twice, Note that this method currently supports only two levels of grouping * Bucket grouping with aggstypes metric measurement * @param metricName Field to be analyzed statistically * @param aggsType * @Param queryBuilder * @param clazz * @param bucketNames Drilldown group field * @return * @throws Exception */ public List<Down> aggsWith2Level (String) metricName, AggsType aggsType,QueryBuilder queryBuilder, Class<T> clazz ,String... bucketNames) throws Exception;Copy the code

//select appli_name,risk_code,sum(sumpremium) from Main2 group by appli_name,risk_code List<Down> list = elasticsearchTemplate.aggswith2level("sum_premium", AggsType.sum,null,Main2.class,"appli_name","risk_code"); list.forEach(down -> { System.out.println("1:"+down.getLevel_1_key()); System.out.println("2:"+down.getLevel_2_key() + " "+ down.getValue()); });Copy the code

Statistical aggregate query

/** * Statistical aggregation metric metric * @param metricName Field to be analyzed * @Param queryBuilder * @Param clazz * @return * @throws Exception */ public Stats statsAggs(String metricName, QueryBuilder queryBuilder, Class<T> clazz) throws Exception;Copy the code

// The sum, count, avg, min, Max indexes of metricName are returned in a single query elasticsearchTemplate.statsAggs("sum_premium",null,Main2.class); System.out.println("max:"+stats.getMax()); System.out.println("min:"+stats.getMin()); System.out.println("sum:"+stats.getSum()); System.out.println("count:"+stats.getCount()); System.out.println("avg:"+stats.getAvg());Copy the code

Group statistics aggregate query

/** * Group by bucket, Statistical aggregation metric metric * @Param bucketName Groups by bucketName field * @Param metricName Field to be analyzed statistically * @Param queryBuilder * @Param clazz * @return * @throws Exception */ public Map<String,Stats> statsAggs(String metricName, QueryBuilder queryBuilder, Class<T> clazz, String bucketName) throws Exception;Copy the code

Compared to the previous method, this method increases the grouping capability

Map<String,Stats> stats = elasticsearchTemplate.statsAggs("sum_premium",null,Main2.class,"risk_code");
stats.forEach((k,v) ->
    {
        System.out.println(k+"    count:"+v.getCount()+" sum:"+v.getSum()+"...");
    }
);
Copy the code

Base query

www.elastic.co/guide/en/el…

/** * Cardinality query, that is, count(distinct) returns an approximation, Not always accurate * @param metricName Field to be analyzed * @param queryBuilder * @Param clazz * @return * @throws Exception */ public long cardinality(String metricName, QueryBuilder queryBuilder, Class<T> clazz) throws Exception; /** * base query, Combined with https://gitee.com/zxporz/ESClientRHL/wikis/Elasticsearch-ESClientRHL, please use the * @ param metricName measure field name * @ param @param precisionThreshold Set precisionThreshold, Default 3000 Max. 40000 * @param clazz Index POJO class type * @return * @throws Exception */ Public Long Cardinality (String metricName, QueryBuilder queryBuilder, long precisionThreshold, Class<T> clazz) throws Exception;Copy the code

//select count(distinct proposal_no) from Main2 long value = elasticsearchTemplate.cardinality("proposal_no",null,Main2.class); System.out.println(value); As long value = elasticsearchTemplate. Cardinality (" proposal_no ", null, 3000 l, Main2. Class);Copy the code

Percentage aggregate query

/** * Percentage aggregation defaults to 50%,95%,99% (TP50 TP95 TP99) * @param metricName Specifies the field for statistical analysis, which must be a numeric type * @param queryBuilder * @param clazz * @return * @throws Exception */ public Map percentilesAggs(String metricName, QueryBuilder queryBuilder, Class<T> clazz) throws Exception; /** * Custom percentage segment aggregated by percentage * @param metricName Specifies the field to be analyzed statistically, which must be a numeric type * @param queryBuilder * @param clazz * @param customSegment Custom percentage * @return * @throws Exception */ Public Map percentilesAggs(String metricName, QueryBuilder QueryBuilder, Class<T> clazz,double... customSegment) throws Exception;Copy the code

Percentage aggregation is the query statistics field 50% of the data within what value, 95% of the data within what value

/ / the following example is taken sum_premium this field TP50 TP95 TP99 Map Map. = elasticsearchTemplate percentilesAggs (" sum_premium ", null, Main2. Class);  map.forEach((k,v) -> { System.out.println(k+" "+v); }); // The output is: 50.0 3.5 95.0 6.0 99.0 6.0 // that is 50% SUM_premium below 3.5 // that is 95% SUM_premium below 6.0 // that is 99% SUM_premium below 6.0 // You can also customize percentage segment Map Map = elasticsearchTemplate. PercentilesAggs (" sum_premium ", null, Main2. Class, 10,20,30,50,60,90,99);Copy the code

Percentile aggregated query

* @param metricName Specifies the field to be analyzed. It must be a numeric type * @param queryBuilder * @param clazz * @param customSegment * @return * @throws Exception */ public Map percentileRanksAggs(String metricName, QueryBuilder queryBuilder, Class<T> clazz,double... customSegment) throws Exception;Copy the code

Percentile aggregation is given a range of statistical results and queried to see what percentage of the range occurs

Sum_premium = 1,4,5,9 That is, below 1, below 4, below 5, below 9 // Finally obtain the ratio of percentile Map of data appearing in the above range Map = ElasticsearchTemplate. PercentileRanksAggs (" sum_premium ", null, Main2. Class, 1,4,5,9); map.forEach((k,v) -> { System.out.println(k+" "+v); }); // The output is: 8.3333333333332 1.0 58.333333333333336 4.0 75.0 5.0 100.0 9.0 // 58.3% of the data is sum_premium below 4 // 75% of the data is sum_premium below 5 // 100% of the data is sum_premium below 9Copy the code

Filter aggregate query

/** * filter aggregation, Grouping that can be "variable" * @param metricName Field that needs statistical analysis * @Param aggsType Statistical type * @param clazz * @Param queryBuilder * @param filters FiltersAggregator filters encapsulate objects, * @return * @throws Exception */ Public Map filterAggs(String metricName, AggsType AggsType, QueryBuilder queryBuilder,Class<T> clazz, FiltersAggregator.KeyedFilter... filters) throws Exception;Copy the code

Filter aggregation makes es aggregation very flexible, it can be flexibly customized group specifications

// The following example takes risk_code as group 0101 and risk_code as group 0103 or 0104. O sum_premium within the group and Map Map = elasticsearchTemplate. FilterAggs (" sum_premium, "AggsType sum, null, Main2. Class, new FiltersAggregator.KeyedFilter("0101", QueryBuilders.matchPhraseQuery("risk_code", "0101")), New FiltersAggregator. KeyedFilter (" 0103 or 104, "QueryBuilders. MatchQuery (" risk_code", "0103 0104"))); map.forEach((k, v) -> System.out.println(k + " " + v) );Copy the code

Histogram aggregation query

/** * histogram aggregation * @param metricName Field for statistical analysis * @Param aggsType Statistical type * @param queryBuilder * @param clazz * @param * @param Interval Specifies the interval for grouping field values. * @return * @throws Exception */ public Map histogramAggs(String)  metricName, AggsType aggsType,QueryBuilder queryBuilder,Class<T> clazz,String bucketName,double interval) throws Exception;Copy the code

Histogram aggregation is statistics on grouped fields that measure values every X interval

// Count sum_premium for every multiple of 3, Statistics of the number of a proposal_no Map Map. = elasticsearchTemplate histogramAggs (" proposal_no, "AggsType. Count, null,Main2.class,"sum_premium",3); map.forEach((k, v) -> System.out.println(k + " " + v) ); 0.0 2 3.0 3 6.0 1Copy the code

Date histogram aggregation query

/** * Date histogram aggregation * @param metricName Field for statistical analysis * @Param aggsType Statistical type * @param queryBuilder * @param clazz * @param BucketName * @param interval Date group interval * @return * @throws Exception */ public Map dateHistogramAggs(String metricName, AggsType aggsType, QueryBuilder queryBuilder, Class<T> clazz, String bucketName, DateHistogramInterval interval) throws Exception;Copy the code

Date histograms are similar to histograms except that grouped fields are replaced with date types

/ / statistical input_date sum_premium amount every two hours combined Map Map = elasticsearchTemplate. DateHistogramAggs (" sum_premium, "AggsType sum, null,Main2.class,"input_date", DateHistogramInterval.hours(2)); map.forEach((k, v) -> System.out.println(k + " " + v) );Copy the code

More ways to aggregate queries

Es supports much more than that. The tool simply encapsulates some of the most common query methods to reduce the amount of code

For example, we need to implement a range aggregation, which can be done with the following example:

// Group by sum_premium, AggregationBuilder Aggregation = AggregationBuilders. Range (" range "). The field (" sum_premium "). AddUnboundedTo (1). The addRange (1, 4). AddRange (4100). AddUnboundedFrom (100); aggregation.subAggregation(AggregationBuilders.count("agg").field("proposal_no.keyword")); Aggregations aggregations = elasticsearchTemplate.aggs(aggregation,null,Main2.class); Range range = aggregations.get("range"); for (Range.Bucket entry : range.getBuckets()) { ValueCount count = entry.getAggregations().get("agg"); long value = count.getValue(); System.out.println(entry.getKey() + " " + value); }Copy the code

More aggregate API usage see www.elastic.co/guide/en/el…

Operations function

Example Print es service logs

If you need to obtain JSON packets requesting ES for debugging, you can configure printLog = true to print packets for es index structure entity class annotations

@ESMetaData(indexName = "index", number_of_shards = 5,number_of_replicas = 0,printLog = true)
public class Main2 implements Serializable {
Copy the code

This function is disabled by default. After this function is enabled, only some common functions can be printed. If you want to print more functions, add the following codes in corresponding positions

if(metaData.isPrintLog()){
    logger.info(searchSourceBuilder.toString());
}
Copy the code

Prints automatic registration

The EnableESTools annotation printregmSG property can be set to true if you need to validate automatic registration (including information about automatic index structure creation and automatic generation of the ESCRepository proxy class)

@EnableESTools(entityPath = {printregmsg = true)

Copy the code

Testdemo package (attached testdemo.zip) description

Build a SpringBoot program and import esclientrhl. After configuring the ES service, you can invoke relevant test demo

TestAggs is a method for testing aggregation
TestCRUD is a method for testing index data
TestIndex is a test method for creating a drop index
TestLowLevelClient is the method that tests LowLevelClient

I didn’t write a comment on the method, so please refer to readme

Special thanks to

Special thanks to JetBrains for their IDEA products

Special thanks to the enthusiastic participants of the project: Jane gojing, tongxuefang welcome to participate in Elasticsearch-ESClientRHL

Esclientrhl-elasticsearch Java client open source tool

EsClientRHL

The reason why EsClientRHL is selected

Release notes

Update log

What skills should you have before using it

This section describes the functions of the tool

Automatic proxy interfaces simplify development

Index structure Management

CRUD

Aggregation query

Tool source code structure introduction

annotation

config

auto

enums

index

repository

util

Begin to use

Use the sample

Instructions before use

Maven rely on

The introduction of the component

Add the EnableESTools annotation

Application. properties Adds the URI of the ElasticSearch service

Application. Properties Adds elasticSearch HttpClient connection pool parameter Settings

Add elasticSearch username and password (based on es7+ only version of XPack)

Using the component

Index management

Metadata Configuration

Index structure configuration

Automatically create index structure mapping based on the configuration information

GEO type

A description of nested type support

The difference between ngram completion suggest and phrase suggest

Manually create or drop index structures

Check whether the index exists

Create index Mapping using JSON

Configuring not to create indexes automatically (at startup)

Configure automatic basis for index namessuffixConfigure suffixes for index names

Alias operation

Rollover Indicates the scrolling index operation

CRUD function description

LowLevelClient query

Adding index data

Add index data in batches

Add index data in batches

Partially update index data

Overwrite updates index data

Batch update index

Update indexes in batches

Delete index data

Delete index data based on query conditions

Check whether index data exists

Native queries

Supports uri query String queries

SQL query support

Customize query criteria

Supports customized query based on search criteria and maximum number of returned items

Supports custom queries for paging, highlighting, sorting, and query conditions

Advanced query

Count query

Scroll to query

Template query – Save template

Template query – Register template

Template query – Inline template

Search for Completion Suggester suggestions

Search suggestions phrase suggest

Query by ID

Mget query

QueryBuilder common usage demo

Precise query

The phrase query

Relevance query

Range queries

The full text matching

Fuzzy error correction query

Boost weight Settings

Prefix Prefix query

Configure automatic basis for index names`suffix`Configure suffixes for index names