I used ElasticSearch for 10 million IP library data cache, but I didn’t use it because it took too long to query it. I’ve seen some videos before, just to show how to use it.
ElasticSearch is now available in version 7.1.1. The API is ElasticSearch – Rest-high-level -client7.1.1. Since I just joined the new company, my priority is to get familiar with the business, and I will further learn ElasticSearch if I have time.
How to use ElasticSearch-rest-high-level client in a Java project
Direct HTTP interface call
Actually using Kibana or ElasticSearch-rest-high-level client ends up sending HTTP requests. Different from services such as Redis, it is necessary to understand its communication protocol, and then realize communication through Socket programming, so they are directly using the API encapsulated by others. ES provides RESTFUL interfaces that do not require us to understand the protocol. Therefore, the simplest way is to directly construct a request body and send an HTTP request to ES.
String esUrl = String.format("%s/%s/_search",elasticsearchConfig.getClusterNodes(),INDEX);
// Send an HTTP request
String responseStr = HttpUtil.requestPostWithJson(esUrl,searchBuilder.toString());
// Parse the responseStr result
Copy the codeAgain, the ES API is used to construct the request body using the SearchBuilder. Of course, you can do it yourself without using it.
SearchSourceBuilder searchBuilder = newSearchSourceBuilder(); BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery() .must(....) ; searchBuilder.query(boolQueryBuilder);Copy the codeBut constructing a request body can be tedious, so a wrapped API is often chosen.
Api-based encapsulation is used
Add the ElasticSearch-rest-high-level client dependency. Select * from elasticSearch; select * from elasticSearch; select * from elasticSearch; Elasticsearch/rest-Client dependencies for elasticSearch/rest-Client (The project uses Maven; I prefer gradle.)
<! -- es -->
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-high-level-client</artifactId>
<version>${elasticsearch.version}</version>
<exclusions>
<! -- default introduced low version so exclude re-dependency -->
<exclusion>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
</exclusion>
<exclusion>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
</exclusion>
</exclusions>
</dependency>
<! -- redependence -->
<dependency>
<groupId>org.elasticsearch.client</groupId>
<artifactId>elasticsearch-rest-client</artifactId>
<version>${elasticsearch.version}</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>${elasticsearch.version}</version>
</dependency>
Copy the codeThen add the es configuration to the application.yml configuration file with a name of your own.
# Request ES 7.x version
es:
host: 127.0. 01.
port: 9400
scheme: http
Copy the codeReading Configuration Information
@Component
@ConfigurationProperties(prefix = "es")
public class ElasticSearchPropertys {
private String host;
private int port;
private String scheme;
}
Copy the codeThen create a RestHighLevelClient to inject into the Spring container based on the configuration.
@Configuration
public class ElasticSearchConfig {
@Resource
private ElasticSearchPropertys elasticSearchPropertys;
@Bean
public RestHighLevelClient restHighLevelClient(a) {
RestClientBuilder restClientBuilder = RestClient.builder(
new HttpHost(elasticSearchPropertys.getHost(),
elasticSearchPropertys.getPort(), elasticSearchPropertys.getScheme()));
return newRestHighLevelClient(restClientBuilder); }}Copy the codeRestClient builder supports multiple HttpHost.
You can then happily implement the CURD operation using the API provided by RestHighLevelClient. For ease of use, you can wrap another layer based on RestHighLevelClient.
/** * Encapsulates ES generic API **@author wujiuye
* @date2020/03/04 * /
@Component
@ConditionalOnBean(RestHighLevelClient.class)
public class ElasticSearchService {
@Resource
RestHighLevelClient restHighLevelClient;
/** * Determines whether an index contains **@paramThe index index name * /
public boolean existIndex(String index) throws Exception {}/** * Create index (test only) **@paramIndex Indicates the index name *@paramMappings Index Description *@paramNumber of Shards fragments *@paramReplicas */
public void createIndex(String index, EsIndexMappings mappings, int shards, int replicas) throws Exception {}/** * Inserts or updates a single record **@param index index
* @paramThe entity object * /
public void insertOrUpdate(String index, EsEntity entity) throws Exception {}/** * Insert data in batches **@param index index
* @paramList with insert list */
public void insertBatch(String index, List<EsEntity> list) throws Exception {}/** * search for **@param index index
* @paramBuilder Query parameter *@paramC Result object type */
public <T> List<T> search(String index, SearchSourceBuilder builder, Class<T> c) throws Exception {}... }Copy the codeDuring development, we need local tests, or ES connected to the test environment for testing. For convenience, I will write the action of creating the index in the code, and delete the index when judging that the environment is dev. Therefore, I have also encapsulated the logic for creating indexes.
The first step is to define an annotation that annotates the fields of the entity class to construct the mapping when creating the index. If you need more information, add the EsField annotations and refine the parsing logic.
/** * ES index field mapping for code to create indexes (for testing only) **@author wujiuye
* @date2020/03/04 * /
@Target({ElementType.FIELD})
@Retention(RetentionPolicy.RUNTIME)
@Documented
public @interface EsField {
/** * Field type **@return* /
String type(a) default "text";
}
Copy the codeSuch as:
public class Product {
/** * product id */
@EsField(type = "integer")
private Integer leId;
/** * brand id */
@EsField(type = "integer")
private Integer brandId;
/** * brand name */
private String brandName;
/** ** date */
private String date;
}
Copy the codeImplement annotation EsField parsing to generate an EsIndexMappings object.
/** * for code to create indexes (for testing only) **@author wujiuye
* @date2020/03/04 * /
public class EsIndexMappings {
private boolean dynamic = false;
private Map<String, Map<String, Object>> properties;
/** * Generate index field mapping information **@param dynamic
* @param type
* @return* /
public static EsIndexMappings byType(booleandynamic, Class<? > type) {
EsIndexMappings esIndexMappings = new EsIndexMappings();
esIndexMappings.setDynamic(dynamic);
esIndexMappings.setProperties(new HashMap<>());
Field[] fields = type.getDeclaredFields();
for (Field field : fields) {
Map<String, Object> value = new HashMap<>();
EsField esField = field.getAnnotation(EsField.class);
if (esField == null) {
value.put("type"."text");
value.put("index".true);
} else {
value.put("type", esField.type());
value.put("index", esField.index());
}
esIndexMappings.getProperties().put(field.getName(), value);
}
returnesIndexMappings; }}Copy the codeCreate an implementation of the index method
/** * Create index (test only) **@paramIndex Indicates the index name *@paramMappings Index Description *@paramNumber of Shards fragments *@paramReplicas */
public void createIndex(String index, EsIndexMappings mappings, int shards, int replicas) throws Exception {
if (this.existIndex(index)) {
return;
}
CreateIndexRequest request = new CreateIndexRequest(index);
request.settings(Settings.builder()
/ / shard
.put("index.number_of_shards", shards)
/ / replications
.put("index.number_of_replicas", replicas));
/ / specify the mappings
request.mapping(JSON.toJSONString(mappings), XContentType.JSON);
CreateIndexResponse res = restHighLevelClient.indices().create(request, RequestOptions.DEFAULT);
if(! res.isAcknowledged()) {throw new RuntimeException("So creation failed!"); }}Copy the codeUse examples:
elasticService.createIndex(INDEX, EsIndexMappings.byType(false, Product.class), 1.1);
Copy the codeWe may have a need to specify a document ID when inserting objects, so to encapsulate the more general insert and bulk insert methods, we need to abstract an intermediate object, EsEntity.
public class EsEntity {
/** * the id of the index. If this parameter is not specified, */ is automatically generated by es
private String id;
/** * convert the intermediate object into a JSON string to avoid the waste of memory resources */
private String jsonData;
}
Copy the codeProvides a static method to convert an arbitrary object into an EsEntity, with or without an ID. When an ID is not specified, ES is automatically generated.
/** * Convert any type of object to EsEntity * without specifying _id **@paramObj A document *@param <T>
* @return* /
public static <T> EsEntity objToElasticEntity(T obj) {
return objToElasticEntity(null, obj);
}
/** * Convert an object of any type to EsEntity **@paramId NULL: _id is not specified. Non-null: _id * is specified@paramObj A document *@param <T>
* @return* /
public static <T> EsEntity objToElasticEntity(Integer id, T obj) {
EsEntity elasticEntity = new EsEntity();
String data = JSON.toJSONString(obj);
elasticEntity.setId(id == null ? null : String.valueOf(id));
elasticEntity.setData(data);
return elasticEntity;
}
Copy the codeInsert and batch insert implementation:
/** * Inserts or updates a single record **@param index index
* @paramThe entity object * /
public void insertOrUpdate(String index, EsEntity entity) throws Exception {
IndexRequest request = new IndexRequest(index);
request.id(entity.getId());
request.source(entity.getData(), XContentType.JSON);
IndexResponse response = restHighLevelClient.index(request, RequestOptions.DEFAULT);
if(response.status() ! = RestStatus.OK) {throw newRuntimeException(response.toString()); }}/** * Insert data in batches **@param index index
* @paramList with insert list */
public void insertBatch(String index, List<EsEntity> list) throws Exception {
BulkRequest request = new BulkRequest();
list.forEach(item -> request.add(new IndexRequest(index)
.id(item.getId())
.source(item.getData(), XContentType.JSON)));
BulkResponse response = restHighLevelClient.bulk(request, RequestOptions.DEFAULT);
if (response.hasFailures()) {
throw newRuntimeException(response.buildFailureMessage()); }}Copy the codeExamples of batch inserts:
List<Product> products = new ArrayList<>();
elasticService.insertBatch(INDEX,
products.stream().map(EsEntity::objToElasticEntity)
.collect(Collectors.toList()));
Copy the codeconclusion
Personally, I prefer apI-based encapsulation, which is simple and universal. However, when inserting data in batches, do not create too many intermediate objects, resulting in memory space waste. For example, the results from the database are converted into an intermediate object, which is then converted into a Map object and inserted into ES.
Of course, if the team members don’t like it and feel bad about it, I’ll stick with the way it’s used in existing projects.
Welcome to leave a message and tell me how to use ES in your project. I also want to learn about it, because everything I see online is similar.