background

Recently, I received a demand that, in a word, shows the dynamics of follower posts, which involves the design of feed streaming system. This article focuses on a Feed streaming solution available to the general enterprise.

Relevant concepts

Let’s start with a simple concept of Feed flows.

What is a feed stream

  • Feed: Every status or message in a Feed stream is a Feed. For example, a tweet in a microblog is a Feed.
  • The Feed flow: An information stream that continually updates and presents content to users. Everyone’s moments, Micro blog page and so on is a Feed stream.

The feed flow classification

There are two common classifications of Feed flows:

  • TimelineTimeline indicates that there are not many feeds in the Feed stream, but each Feed is important and needs to be seen by users. Similar to wechat moments, Weibo and so on.
  • Rank: Sort by a non-time factor, generally according to the user’s preference, generally used for news recommendation, product recommendation, etc.

design

Design a Feed stream system, two key steps, one is Feed stream initialization, one is push. The storage of the Feed stream is also a core point, but the main persistence of the pen is still using MySQL, which can be optimized later.

The Feed stream is initialized

The initialization of a Feed stream refers to creating a Feed stream for a user that does not already exist. It’s really easy. Just go through the follow list, pull out all the feeds from the people you follow, and put your feedId into Redis’ sortSet. There are a few key points:

  • Initialize data: The initialized data needs to be loaded from the database.
  • Key value: The key value of sortSet must be identified by the current user ID.
  • Score value: For Timeline, select the timestamp created by the feed. If it is a rank type, set the weight of your business to it.

push

After the initialization above, the feed has been exiled into the Redis cache. Next up is the need to update the feed stream in the following four cases:

  1. Followers post new feeds:
  2. The following users delete the feed.
  3. New user concerns.
  4. The user unfollowed the user.

Publish/delete Feed process

How to operate the above four steps will be described in detail in the following implementation steps. Here we will focus on the first and second cases. Because when we deal with big V [tens of millions of fans], we need to deal with the feed stream of all big V fans, and the amount involved at this time will be very huge, which needs to be considered. There are generally two types of push/pull.

  • push: When user A posts A new feed, push it to all of user A’s fan feed streams.
  • la: When user A releases A new dynamic, he does not push it at first. When fans come in, he takes the initiative to pull the latest feed from user A’s personal page TimeLine, and then performs A merge. If you care about multiple large V’s, you can simultaneously pull multiple large V’s from the individual page TimeLine.
Push-pull combination mode

When a user posts a new Feed, the process is as follows:

  1. Read your fan list from the following list and determine whether you are a big V.
  2. Write your own Feed messages to the personal page Timeline. If it’s big V, the write process ends there.
  3. If you’re a regular user, you need to write your Feed to your followers, and if you have 100 followers, you need to write your Feed to 100 users.

When refreshing your Feed stream, the process is as follows:

  1. Start by reading the list of big Vs you care about
  2. To read your own Feed stream.
  3. If there are big Vs that are concerned, the individual page Timeline of each big V will be read concurrently again. If there are 10 big Vs that are concerned, 10 times of access will be required.
  4. The results of steps 2 and 3 are merged, sorted by time, and returned to the user.

At this point, with push-pull publishing, the process of reading the Feed stream is complete.

Push mode

If you just use push mode, it will be easier:

  • Publish the Feed:
    • Regardless of whether it’s big V or not, the process is the same for all users, with three steps.
  • Read the Feed stream:
    • No need for the first step, no need for the third step, just need the second step, reduce the previous 2 + N(N is the number of large V of concern) network cost to 1 network cost. Read latency is significantly reduced.
Summary of two modes:

One drawback of push and pull combination is that when refreshing its own Feed stream, big V’s personal page Timeline will have great reading pressure.

How to solve:

  1. The optimization method of large V/ ordinary users is not used. The active fans are pushed and inactive fans are pulled.
  2. Using the push mode completely solves this problem, but with the increase of storage, the total time to send the big V Feed will increase, from the first fan to the last fan can take several minutes.

implementation

The pen master mainly uses pure push mode to achieve a common enterprise basic available Feed stream system, the following introduces the specific implementation code, mainly including three parts:

  1. Initialize the Feed stream.
  2. Followers post/delete feeds, and followers of that user update their feed stream.
  3. Users add/unfollow and update their Feed streams.

Initialize the Feed stream

When the user first comes in to refresh the Feed stream, and the Feed stream doesn’t exist yet, we need to initialize it. The initialization code is as follows: The core idea is to load the Feed from the database, stuff it into zSet, and then page back.

    /** * The stream of people who get attention */
    public List<FeedDto> listFocusFeed(Long userId, Integer page, Integer size) {
        String focusFeedKey = "focusFeedKey" + userId;

        // If zset is empty, initialize first
        if(! zSetRedisTemplate.exists(focusFeedKey)) { initFocusIdeaSet(userId); }// If zset exists, but 0 exists
        Double zscore = zSetRedisTemplate.zscore(focusFeedKey, "0");
        if(zscore ! =null && zscore > 0) {
            return null;
        }

        / / paging
        int offset = (page - 1) * size;

        long score = System.currentTimeMillis();
        // Select FeedId from zSet in order of score
        List<String> list = zSetRedisTemplate.zrevrangeByScore(focusFeedKey, score, 0, offset, size);

        List<FeedDto> result = new ArrayList<>();
        if (QlchatUtil.isNotEmpty(list)) {
            for (String s : list) {
                // Load the feed from the cache based on feedId
                FeedDto feedDto = this.loadCache(Long.valueOf(s));
                if(feedDto ! =null) { result.add(feedDto); }}}return result;
    }

    /** * Initializes the following person's information flow zSet */
    private void initFocusFeedSet( Long userId) {
        String focusFeedKey = "focusFeedKey" + userId;
        zSetRedisTemplate.del(focusIdeaKey);

        // Load feeds from the database of people the current user follows
        List<Feed> list = this.feedMapper.listFocusFeed(userId);

        if (QlchatUtil.isEmpty(list)) {
            // Save 0 to avoid frequent library searches for empty data
            zSetRedisTemplate.zadd(focusFeedKey, 1."0");
            zSetRedisTemplate.expire(focusFeedKey, RedisKeyConstants.ONE_MINUTE * 5);
            return;
        }

        // Iterate through the FeedList and save the FeedId to zSet
        for (Feed feed : list) {
            zSetRedisTemplate.zadd(focusFeedKey, feed.getCreateTime().getTime(), feed.getId().toString());
        }

        zSetRedisTemplate.expire(focusFeedKey, 60 * 60 * 60);
    }
Copy the code

Followers post/delete new feeds

Whenever a user posts/deletes a new feed, we need to update the feed stream of all the user’s fans. This step is usually time-consuming, so we suggest asynchronous processing. In order to avoid loading too much fan data at one time, we use circular paging query here. In order to avoid large fan Feed flows, we will limit the Feed stream length to 1000 and remove the oldest Feed when the stream length exceeds 1000.

    /** * Handle fan feed streams ** when adding/deleting feeds@paramUserId userId * for adding/deleting feeds@paramFeedId Add or delete feedId *@paramType feed_add = Add feed feed_sub = delete feed */
    public void handleFeed(Long userId, Long feedId, String type) {

        Integer currentPage = 1;
        Integer size = 1000;
        List<FansDto> fansDtos;

        while (true) {
            Page page = new Page();
            page.setSize(size);
            page.setPage(currentPage);
            fansDtos = this.fansService.listFans(userId, page);

            for (FansDto fansDto : fansDtos) {
                String focusFeedKey = "focusFeedKey" + userId;

                // If the fan zSet does not exist, exit
                if (!this.zSetRedisTemplate.exists(focusFeedKey)) {
                    continue;
                }

                / / the new Feed
                if ("feed_add".equals(type)) {
                    this.removeOldestZset(focusFeedKey);
                    zSetRedisTemplate.zadd(focusFeedKey, System.currentTimeMillis(), feedId);
                }
                / / remove the Feed
                else if ("feed_sub".equals(type)) { zSetRedisTemplate.zrem(focusFeedKey, feedId); }}if (fansDtos.size() < size) {
                break; } currentPage++; }}/** * Delete the oldest data in zSet */
    private void removeOldestZset(String focusFeedKey){
        If the current zSet is greater than 1000, delete the oldest data
        if (this.zSetRedisTemplate.zcard(focusFeedKey) >= 1000) {
            // Get the smallest score in the current zSet
            List<String> zrevrange = this.zSetRedisTemplate.zrevrange(focusFeedKey, -1, -1, String.class);
            if (QlchatUtil.isNotEmpty(zrevrange)) {
                this.zSetRedisTemplate.zrem(focusFeedKey, zrevrange.get(0)); }}}Copy the code

A user added or unfollowed a user

Here, it is easier to add/unfollow new feeds to/remove new feeds from your Feed stream, but also asynchronous processing is required.

    /** * Follow/close zSet ** with followerId@paramFollowId is followed by people *@paramFollowerId Current user *@paramType focus = focus unfocus = close */
    public void handleFocus( Long followId, Long followerId, String type) {

        String focusFeedKey = "focusFeedKey" + userId;

        // If the fan zSet does not exist, exit
        if (!this.zSetRedisTemplate.exists(focusFeedKey)) {
            return;
        }
        List<FeedDto> FeedDtos = this.listFeedByFollowId(source, followId);
        for (FeedDto feedDto : FeedDtos) {

            / / concern
            if ("focus".equals(type)) {
                this.removeOldestZset(focusFeedKey);
                this.zSetRedisTemplate.zadd(focusFeedKey, feedDto.getCreateTime().getTime(), feedDto.getId());
            }
            / / take off
            else if ("unfocus".equals(type)) {
                this.zSetRedisTemplate.zrem(focusFeedKey, feedDto.getId()); }}}Copy the code

The above is the core code, just to provide you with an implementation idea, not directly executable code, after all, the real implementation will involve a lot of other irrelevant classes.

The last

Here we have a simple and usable Feed streaming system. Welcome to point out any errors and suggestions!

Reference article:

  • How to build a ten-million-level Feed streaming system