This is the 14th day of my participation in the November Gwen Challenge. Check out the event details: The last Gwen Challenge 2021

The background of this problem arises from the burying point of a feature that needs to be launched in the near future. The main manifestation is that the memory usage has been increasing for a period of time after application startup.

The following figure shows the internal usage trend after multiple lines are displayed on jConsole.

The actual environment is limited by configuration and memory does not expand

Background & Issues

Application A uses the REST template to invoke application B through HTTP. The actuator is enabled in the application project, and the API uses Micrometer. When the client calls, the actuator generates metrics with the name HTTP.client.requests, and the metric tag contains the DESTINATION URI.

The interfaces provided by application B are as follows:

@RequestMapping("test_query_params")
public String test_query_params(@RequestParam String value) {
    return value;
}

@RequestMapping("test_path_params/{value}")
public String test_path_params(@PathVariable String value) {
    return value;
}


Copy the code

http://localhost:8080/api/test/test_query_params?value=

http://localhost:8080/api/test/test_path_params/{value}_

The metric collection should include two metrics. The main difference is that the URI in the tag is API /test/test_query_params. /test/test_path_params/{value}; In fact, there is a big difference in metrics data. Here, take the metric of Pathvariable as an example, and the data are as follows:

tag: "uri",
values: [
"/api/test/test_path_params/glmapper58"."/api/test/test_path_params/glmapper59"."/api/test/test_path_params/glmapper54"."/api/test/test_path_params/glmapper55"."/api/test/test_path_params/glmapper56"."/api/test/test_path_params/glmapper57"."/api/test/test_path_params/glmapper50"."/api/test/test_path_params/glmapper51"."/api/test/test_path_params/glmapper52"."/api/test/test_path_params/glmapper53"."/api/test/test_path_params/glmapper47"."/api/test/test_path_params/glmapper48"."/api/test/test_path_params/glmapper49"."/api/test/test_path_params/glmapper43"."/api/test/test_path_params/glmapper44"."/api/test/test_path_params/glmapper45"."/api/test/test_path_params/glmapper46"."/api/test/test_path_params/glmapper40"."/api/test/test_path_params/glmapper41"."/api/test/test_path_params/glmapper42"."/api/test/test_path_params/glmapper36"."/api/test/test_path_params/glmapper37"."/api/test/test_path_params/glmapper38"."/api/test/test_path_params/glmapper39"."/api/test/test_path_params/glmapper32"."/api/test/test_path_params/glmapper33"."/api/test/test_path_params/glmapper34"."/api/test/test_path_params/glmapper35"."/api/test/test_path_params/glmapper30"."/api/test/test_path_params/glmapper31"."/api/test/test_path_params/glmapper25"."/api/test/test_path_params/glmapper26". ]Copy the code

You can clearly see that the {value} parameter is used as part of the URI component and is embodied in the tag, not the expected API /test/test_path_params/{value}.

Problem causes and solutions

Two questions, 1, this buried point is how effective, first figure out this problem, to follow the trail. 2, how to solve.

How does the default burying point work

Since the call access is made through the RestTemplate, the burial point must also be based on the proxy to the RestTemplate; According to this train of thought, the author find the org. Springframework. Boot. Actuate. Metrics. Web. Client. MetricsRestTemplateCustomizer this class. RestTemplateCustomizer is customized for resttemplate MetricsRestTemplateCustomizer phase by name can also be learned that role is to increase the metric for resttemplate ability.

Moving on to RestTemplateCustomizer, you can use RestTemplateCustomizer for more advanced customization when building a RestTemplate using RestTemplateBuilder, All RestTemplateCustomizer Beans are automatically added to the automatically configured RestTemplateBuilder. That is to say, if you want to MetricsRestTemplateCustomizer effect, then build resttemplate must build by RestTemplateBuilder, rather than new.

HTTP. Client. Requests in the uri

Plug the tag code in org. Springframework. Boot. Actuate. Metrics. Web. Client. RestTemplateExchangeTags class, Timing is in the MetricsClientHttpRequestInterceptor interceptors. When the call completes, the request metric is recorded, and RestTemplateExchangeTags is used to populate the tags. Only part of the URI code is given below


	/**
	 * Creates a {@code uri} {@code Tag} for the URI of the given {@code request}.
	 * @param request the request
	 * @return the uri tag
	 */
	public static Tag uri(HttpRequest request) {
		return Tag.of("uri", ensureLeadingSlash(stripUri(request.getURI().toString())));
	}

	/**
	 * Creates a {@code uri} {@code Tag} from the given {@code uriTemplate}.
	 * @param uriTemplate the template
	 * @return the uri tag
	 */
	public static Tag uri(String uriTemplate) {
		String uri = (StringUtils.hasText(uriTemplate) ? uriTemplate : "none");
		return Tag.of("uri", ensureLeadingSlash(stripUri(uri)));
Copy the code

Other tag names include status and clientName.

From the breakpoint, you can see that request.geturi () gets the full request link with the parameters.

These tag final assembly in DefaultRestTemplateExchangeTagsProvider completion, and returns a list.

private Timer.Builder getTimeBuilder(HttpRequest request, ClientHttpResponse response) {
    return this.autoTimer.builder(this.metricName)
                / / tagProvider DefaultRestTemplateExchangeTagsProvider
				.tags(this.tagProvider.getTags(urlTemplate.get().poll(), request, response))
				.description("Timer of RestTemplate operation");
}
Copy the code

To solve

Here’s the official explanation for Request. getURI


	/**
	 * Return the URI of the request (including a query string if any,
	 * but only if it is well-formed for a URI representation).
	 * @return the URI of the request (never {@code null})
	 */
	URI getURI(a);
Copy the code

Returns the URI of the request, including any query parameters. Is it ok to get path without arguments?

Here we try to get the desired path via request.geturi ().getPath() (@pathvariable gets the template).

DefaultRestTemplateExchangeTagsProvider again, all the tag are assembled here, this class is clearly a default implementation (Spring system under the basic as long as it is Defaultxxx, usually extension). Check the interface class RestTemplateExchangeTagsProvider is as follows:


/**
 * Provides {@link Tag Tags} for an exchange performed by a {@link RestTemplate}.
 *
 * @author Jon Schneider
 * @author Andy Wilkinson
 * @since2.0.0 * /
@FunctionalInterface
public interface RestTemplateExchangeTagsProvider {

	/**
	 * Provides the tags to be associated with metrics that are recorded for the given
	 * {@code request} and {@code response} exchange.
	 * @param urlTemplate the source URl template, if available
	 * @param request the request
	 * @param response the response (may be {@code null} if the exchange failed)
	 * @return the tags
	 */
	Iterable<Tag> getTags(String urlTemplate, HttpRequest request, ClientHttpResponse response);

}
Copy the code

RestTemplateExchangeTagsProvider role is to provide resttemplate tag, so through a custom RestTemplateExchangeTagsProvider here, To replace DefaultRestTemplateExchangeTagsProvider, in order to achieve our goal, roughly as follows:


@Override
 public Iterable<Tag> getTags(String urlTemplate, HttpRequest request, ClientHttpResponse response) {
    Tag uriTag;
    // Take request.geturi ().getPath() as the value of the URI
    if (StringUtils.hasText(request.getURI().getPath())) {
      uriTag = Tag.of("uri", ensureLeadingSlash(stripUri(request.getURI().getPath())));
    } else {
      uriTag = (StringUtils.hasText(urlTemplate) ? RestTemplateExchangeTags.uri(urlTemplate)
                    : RestTemplateExchangeTags.uri(request));
    }
    return Arrays.asList(RestTemplateExchangeTags.method(request), uriTag,
                RestTemplateExchangeTags.status(response), RestTemplateExchangeTags.clientName(request));
    }
Copy the code

Will the OOM

In theory, should have different parameters, in the case of using default DefaultRestTemplateExchangeTagsProvider, meter with different rapid expansion of the tags, in micrometer, the data are exist in the map

// Even though writes are guarded by meterMapLock, iterators across value space are supported
// Hence, we use CHM to support that iteration without ConcurrentModificationException risk
private final Map<Id, Meter> meterMap = new ConcurrentHashMap<>();
Copy the code

Generally, this is because the Spring Boot Actuator provides its own protection mechanism. By default, tags in the same metric can only be 100 at most

 

/**
* Maximum number of unique URI tag values allowed. After the max number of
* tag values is reached, metrics with additional tag values are denied by
* filter.
*/
private int maxUriTags = 100;
Copy the code

If you want to make this number larger, you can configure it as follows

management.metrics.web.client.max-uri-tags=10000
Copy the code

If the configuration value is too large, there may be oom risks.