background

Hello, everyone, long time no see.

XXX promotion activity happened, the database connection of short link application suddenly soared, the monitoring found that SQL was updating frantically, one of which was to update the click number of short link. Check the interface function is actually very simple: determine whether the IP is valid, and then the click number of the short link +1, update to the database table.

Problem analysis

Although the interface function is simple, but if it is in the statistics of several Taobao super seller member click number, if we do not pay attention to it is easy to bring down the system. The following questions can be drawn from the above:

  • 1. The short link is directly updated to the database. If the concurrency is too high, the pressure on the database will be increased and other services will be affected.

  • 2. The interface is only IP verification, without any high concurrency and anti-brush limit, which makes it vulnerable to external attacks.

The solution

Cache hits are asynchronously stored

Since the requirement is to update click data in real time, you can’t cache it for too long.

1. Mq can be used to de-peak traffic and achieve asynchronous processing, but mq is mainly rabbitMQ in the project, which does not work well in the case of large piles. (If you have RocketMQ, of course it’s the first choice.)

2. Similar results can be achieved with Redis.

  • 2.1. Just rPush the clicked link ID + IP into a Redis list.
  • 2.2. Start the thread and execute it once for 1min to obtain the total llEN length of the current Redis list.
  • 2.3 Take out the maximum 1W click data each time for statistics, and update the click number in batches.
  • 2.4. After the statistics are completed, use redis pipeline circulation to pop lPOP 1W pieces of data that have just been processed.
  • 2.5. Loop steps 3 and 4 to get llen clicks.

Make sure that the second and third steps are in the same transaction, otherwise it is easy to duplicate calculations.

A click data = short link ID + IP, about 25 bytes, in fact, 1 G of Redis memory can store 40 million people click the amount of interface, the specific amount of data to estimate plus memory or make a choice. (The boss gave five GS, the drop strategy.)

One might say, well, it would be nice to have multithreading later. In fact, each time processing 1W, if there are 100 million hits in 1min, in fact, only need to perform 1W update operation, the whole process only takes up the majority of the time, 1W times in 1min cycle can be achieved, there is no need to open multithreading to bring more concurrent problems (such as concurrent update of the same row easy to lock table).

Core code:

1. External receiving click request:

 @Override
    public String visitLink(String shortUrl) {
        if (StringUtils.isEmpty(shortUrl)) {
            return null;
        }
        // Add the short links generated in the last day to the cache to improve the response speed.
        // Cache click counts and use asynchronous threads to batch update.
        String resultStr = redisUtil.get(RedisKey.LINK_LIST_LAST + shortUrl);
        if(! StringUtils.isEmpty(resultStr)) { redisUtil.lRightPush(RedisKey.LINK_CLICK_COUNT, shortUrl);return resultStr;
        }
        switch (shortUrl.length()) {
            case 4:
                // Extremely short link
                MinShortUrl originUrl = minShortUrlMapper.getOriginUrl(shortUrl);
                if(originUrl ! =null) {
                    minShortUrlMapper.updateShortUrl(originUrl);
                }
                resultStr = originUrl.getUrl();
                break;
            case 6:
                // Plain short links
                ShortUrl oUrl = shortUrlMapper.getOriginUrl(shortUrl);
                if(oUrl ! =null) {
                    shortUrlMapper.updateShortUrl(oUrl);
                }
                resultStr = oUrl.getUrl();
                break;
            default:
                break;
        }
        if(! StringUtils.isEmpty(resultStr)) { redisUtil.setEx(RedisKey.LINK_LIST_LAST + shortUrl, resultStr,1, TimeUnit.DAYS);
        }
        return resultStr;
    }
Copy the code

2. Timed task processing click number storage:

/** * Count short link scheduled task */
@Component
public class ShortUrlSchedule {

    @Autowired
    private RedisUtil redisUtil;

    @Autowired
    ShortUrlService shortUrlService;

    // This command is executed every 10 minutes
    @Scheduled(cron = "0 0/10 * * * ? ")
    @Transactional(rollbackFor = Exception.class)
    public void calculateClickCount(a) {
        Long size = redisUtil.size(RedisKey.LINK_CLICK_COUNT);
        if(size ! =null && size > 0) {
            // Count the number of clicks on short links
            Map<String, Integer> urlMap = new HashMap<>();
            Long batchSize = 10000L;
            do {
                Long pageSize = size > batchSize ? batchSize : size;
                List<String> tmpList = redisUtil.lRange(RedisKey.LINK_CLICK_COUNT, 0, pageSize);
                if (CollectionUtils.isEmpty(tmpList)) {
                    return;
                }
                for (String shortUrl : tmpList) {
                    // Handle the number of clicks on short links
                    Integer count = urlMap.get(shortUrl);
                    if (count == null || count == 0) {
                        count = 0;
                    }
                    urlMap.put(shortUrl, ++count);
                }
                // Batch update
                int i = shortUrlService.batchUpdateClickCount(urlMap);
                / / the pop-up
                redisUtil.getRedisTemplate().executePipelined(new RedisCallback<String>() {
                    @Override
                    public String doInRedis(RedisConnection redisConnection) throws DataAccessException {
                        RedisConnection pl = redisConnection;
                        for (int i = 0; i <= tmpList.size(); i++) {
                            pl.lPop(RedisKey.LINK_CLICK_COUNT.getBytes());
                        }
                        return null; }}); size = size - tmpList.size(); }while (size > 0); }}}Copy the code

The IP address of the interface is brush resistant

Problem: You want someone on an interface to only request N times in a certain period of time.

Principle: When you request, the server through Redis records the number of times you request, if the number exceeds the limit will not give access. Keys saved in Redis are time-limited and will be deleted when they expire.

The detailed core code is as follows:

/** * Request to intercept */
@Slf4j
@Component
public class RequestLimitIntercept extends HandlerInterceptorAdapter {

    @Autowired
    private RedisTemplate<String,Object> redisTemplate;

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
        /** * isAssignableFrom() determines whether the Class or interface represented by this Class object is the same as that represented by the Class argument. IsAssignableFrom () is the parent of a class. The instanceof keyword is a subclass of a class
        if(handler.getClass().isAssignableFrom(HandlerMethod.class)){
            //HandlerMethod encapsulates information related to the method definition, such as classes, methods, parameters, etc
            HandlerMethod handlerMethod = (HandlerMethod) handler;
            Method method = handlerMethod.getMethod();
            // Whether the method contains annotations
            RequestLimit methodAnnotation = method.getAnnotation(RequestLimit.class);
            // Gets whether the class contains annotations, that is, whether the controller has annotations
            RequestLimit classAnnotation = method.getDeclaringClass().getAnnotation(RequestLimit.class);
            // select method parameters first if there are annotations on the method, otherwise the class parametersRequestLimit requestLimit = methodAnnotation ! =null? methodAnnotation:classAnnotation;if(requestLimit ! =null) {if(isLimit(request,requestLimit)){
                    resonseOut(response,Result.error(ApiResultEnum.REQUST_LIMIT));
                    return false; }}}return super.preHandle(request, response, handler);
    }
    // Determine whether the request is restricted
    public boolean isLimit(HttpServletRequest request,RequestLimit requestLimit){
        // Restricted redis cache key, because I'm using the browser test here, I'll use the sessionID as the unique key, if it's app, you can use a unique ID like the user ID.
        String limitKey = request.getServletPath()+request.getSession().getId();
        // From the cache, the current request is accessed several times
        Integer redisCount = (Integer) redisTemplate.opsForValue().get(limitKey);
        if(redisCount == null) {// The initial count
            redisTemplate.opsForValue().set(limitKey,1,requestLimit.second(), TimeUnit.SECONDS);
        }else{
            if(redisCount.intValue() >= requestLimit.maxCount()){
                return true;
            }
            // The number of times increases
            redisTemplate.opsForValue().increment(limitKey);
        }
        return false;
    }

    /** * write back to the client *@param response
     * @param result
     * @throws IOException
     */
    private void resonseOut(HttpServletResponse response, Result result) throws IOException {
        response.setCharacterEncoding("UTF-8");
        response.setContentType("application/json; charset=utf-8");
        PrintWriter out = null; String json = JSONObject.toJSON(result).toString(); out = response.getWriter(); out.append(json); }}Copy the code

For more information, see xbmchina.cn/AAAAAD

Next time share how to design a small short link small module design. Like the reference link above.

If you like this article, please give it a thumbs up.