Redis persistence mechanism

Redis is a persistent in-memory database. Through the persistence mechanism, the data in the memory is synchronized to the hard disk file to ensure the data persistence. When Redis restarts, the data can be recovered by reloading the disk file into the memory.

Implementation: create a separate child of fock(), copy the current parent’s database data to the child’s memory, then write the child to a temporary file, the persistence process ends, and then replace the last snapshot file with this temporary file. Then the child exits and the memory is freed.

Redis has two types of persistence:

  1. RDB is the default redis persistence mode. According to a certain period policy, memory data is saved in the form of snapshots to disk binary files, that is, SnapShot SnapShot storage. The generated data file is Dmp. RDB.

  2. AOF: Redis will append each received write command to the end of the file through the write function, similar to mysql binlog. When Redis restarts, it will re-execute the write command saved in the file to reconstruct the entire database contents in memory.

    In general, Redis adopts RDB+AOP for persistence. When the two methods are used, REDis will preferentially use AOP for data recovery.

Cache avalanche, cache penetration, cache warm-up, cache update, cache degradation

1. What is cache avalanche?

With high concurrency, multiple requests visit to cache the data set with the same expiration time, a large cache in the past, cause had access to a large number of requests of cache poured on to the database, the database CPU and memory pressure instantly increases, serious will cause the database downtime, thus a chain reaction, cause the collapse of the system.

2. What can be done to prevent cache avalanche?

You can use locks or queues to keep data from being requested by a large number of threads, to avoid a large number of concurrent requests flooding the database in the event of a failure, or to segment the cache expiration time.

3. What is cache penetration?

Cache penetration refers to that when the user queries data, there is no payer in the database and the cache at the same time, resulting in repeated database queries but returned empty data. In this way, the request bypasses the cache and accesses the database directly, which is also a problem of cache hit ratio.

4. Is there any way to prevent cache penetration?

The most common way is to use a Bloom filter to hash all possible data into a bitmap large enough that a certain non-existent data will be intercepted by the bitmap, thus avoiding the query pressure on the underlying storage system. Another way is if the query result returns null, Still cache it and set the minimum expiration time so that the second query will fetch the data in the cache instead of directly accessing the database.

4. What is cache preheating?

Cache preheating is to load the relevant cache data to the cache system when the system is started, so that users can avoid the problem of querying the database first and then caching the data.

Solution:

1. Manually operate the development cache refresh page when it goes online

2. If the amount of data is not large, it can be automatically loaded into the cache when the project is started

3. Refresh the cache periodically

5. What are cache updates?

In addition to the cache invalidation side column provided by the cache server, we can also customize cache elimination policies based on specific business requirements. There are two common policies:

1) Regularly clean past data (a large number of keys need to be maintained)

2) When the user requests, first judge whether the cache used in this request is invalid. If it is invalid, query the database for new data and update it to the cache system (each request requires cache judgment, which is relatively complex logic).

6. What is cache degradation?

In the case of high concurrency, when the server is faulty or non-core services affect the performance of core processes, the service still needs to be available. The system can degrade some key data automatically or manually by configuring switches.

Is there any way to degrade the cache?

Log Level Setting plan:

1), general: for example, some servers occasionally send network jitter or service is online and times out, can be automatically degraded

2) Warning: For example, if the success rate of some services fluctuates within a certain period of time, they can be automatically degraded to manual degradation and send an alarm

3). Error: For example, when the system availability rate is lower than 90%, or the database connection pool is overwhelmed, or the page view suddenly increases, the system’s capacity reaches the threshold. At this time, the system can be automatically degraded or manually degraded according to the situation

4) Critical error: Due to some special reasons, an emergency manual downgrade is required

In the case of high concurrency, if the Redis service is unavailable, the default value can be directly returned to the user, so as to avoid a large number of requests to the database, database downtime, chain reaction, resulting in system crash

What are hot data and cold data?

Hot data is data that is frequently accessed in large quantities

Cold data is data that is frequently modified or accessed infrequently

Frequently accessed data can be directly for caching, such doing can avoid a large number of requests high concurrent access database, database pressure instantly increases, only the data is read at least twice before update to the cache, to be changed frequently and frequently accessed data can also be cached, reduce the pressure of the database (thumb up number, the number of share, the collection number)

What are the differences between Memcache and Redis?

Memcache stores all data in memory. After a breakpoint, it will hang. The deployment cannot exceed the memory size.

All values of memcache are simple string types, while Redis has rich data types, providing storage of data structures such as list, set, Zset and Hash.

3, the use of the underlying model is not the same, memcahce and Redis the underlying implementation of the client communication between the application protocol is not the same, Redis directly build their own VM mechanism, because the general system will waste a certain amount of time to request system functions

Redis can be up to 1GB, while memcache is only 1MB

Redis is much faster than Memcache

6. Redis supports data backup, that is, data backup in master/slave mode

Why is single-threaded Redis so fast?

1. Pure memory operation

2. Single-threaded operations avoid a lot of context switching

3. Use a non-blocking I /O multiplexing mechanism

6. Data types of Redis and usage scenarios of each data type

String: Regular JSON string storage

Hash type: Value is a structured object that can be easily manipulated into a field

List type: can do simple message queue, can also do redis based paging function

Set type: because set is an unrepeatable set, so it can do the function of global reduplication, and can use the intersection, union, difference and other operations to calculate common preferences, all preferences, preferences and other functions

Sorted set type: Sorted set has a weight parameter score. The elements in the set can be sorted according to the score element, which can be used as ranking version

Redis expiration strategy and cache elimination mechanism

Redis uses a periodic delete + lazy delete strategy

Why not use a timed deletion policy?

A timer is used to monitor the key for periodic deletion. If the key expires, it is automatically deleted. Although the memory is released in time, it consumes CPU resources.

How does periodic deletion + lazy deletion work?

By default, Redis randomly selects every 100ms to check whether there are expired keys, and deletes expired keys. However, if periodic deletion is used only, there are still many expired keys that have not been deleted, so lazy deletion takes effect, that is, when obtaining a key, Redis checks that the key is deleted if it has been set to a past time and has expired

Is there no problem with periodic deletion + lazy deletion?

No, if you periodically delete a key without deleting it, and then you don’t request the key, redis’s memory will get higher and higher, then cache elimination should be used.

There is a line of configuration in the redis.conf configuration file

maxmenory-policy volatile-lru
Copy the code

This configuration is configured with a memory elimination policy

Volatile – LRU: Selects the least recently used data from a set that has been set to expire for elimination

Volatile – TTL: Selects data to be obsolete from a set with an expiration time

Volatile -random: Randomly selects data from a set whose expiration time has been set

Allkeys-lru: Cull the least recently used data from the dataset

Allkeys-random: Randomly select any data from the data set

No-enviction: forbids eviction of data. New write operations will report an error

The policy behavior of Volaltile-LRU, Volaltile-Random and Volaltile-TTL is designed to match noeviction if the expire key is not set and the preconditions are not met

Why is Redis single threaded?

Since Redis is memory-based, CPU is not the bottleneck of Redis. The bottleneck of Redis is most likely the size of the machine’s memory or network bandwidth. Since single-threading is easy to implement and the CPU is not a bottleneck, the single-threaded solution can be adopted smoothly.

The vast majority of requests are pure memory operations

2. Using a single thread can avoid frequent context switches and race conditions

3. Non-blocking IO is a bit

  • Fast, because the data is stored in memory, similar to a Hashmap, which has the advantage of having o(1) time for both lookup and operation.

  • Supports rich data types, such as list, set, zset, Hash, and sorted set

  • Transactions are supported, and operations are atomic, meaning that changes to data either all succeed or all fail

  • Rich features, can be used for cache, message, by key set expiration time, will be automatically deleted after the expiration

How to solve the problem of Redis competing for keys in high concurrency?

  • If no order is required for this key, a distributed lock can be used

  • If the key operation is sequential, a distributed lock with a timestamp can be used

  • You can take advantage of queues

Common performance problems and solutions of Redis?

1. It is best for the master not to do any persistent work, such as RDB memory snapshots and AOF log files

2. If data is important, a slave enables AOF backup and synchronizes policy Settings every second

3. For the speed and connection stability of master and slave, it is best to deploy master and slave within the swimsuit LAN

4. Try to avoid adding slave libraries to stressed master libraries

5, master and slave do not use graph structure, use a single necklace table structure is more stable, that is: master < -slave1 < -slave2 < -slave3

X. Why redis operation is atomic, how to ensure atomic?

The atomicity of Redis means that an operation is not separable, and that the operation either succeeds or fails.

Redis operations are atomic because Redis is single-threaded

All apis provided by Redis are atomic, and transactions in Redis are meant to be atomic for batch operations

Are multiple commands atomic in concurrency?

Change get and set to a single command, incr. Use redis transaction, or use redis+lua==

Redis transactions

Redis transaction function is realized by four primitives: Multi, exec, discard and Watch

Redis serializes all commands in a transaction and executes them sequentially

1. Redis does not support rollback operations. Redis does not roll back a transaction if it fails, but continues to execute the remaining commands, so it can be kept simple and fast internally

2. In the same transaction, if an error occurs on one command, no other command is executed

3. If a command execution error occurs within the same transaction, the correct command will be executed

1) The multi command is used to start a transaction and always returns OK. After the multi command is executed, the client can send any number of commands to the server. These commands are not executed immediately, but are placed in a queue. When the exex command is invoked, all commands in the queue are executed.

2) exec executes all the commands in the food block and returns the return values of all the commands in the food block in order of execution. When the operation is interrupted, null is returned

3) By calling discard, the client can condition the transaction queue and abandon the execution of the transaction, and the client exits from the transaction state

4) The watch command can provide a check-and-set (CAS) behavior for redis objects, which can monitor one or more keys. Once one of the keys is modified, the subsequent objects will not be executed, and the monitoring continues until the ECEX command.