Redis study notes

Edit the blog version, I was on CSDN when I wrote my blog before, but I don’t like its interface style (there are many advertisements), so I plan to migrate and tidy up, and transfer all the blogs there before to Nuggets.

Introduction to NoSQL database

“NOT ONLY SQL” means more than just a database. It refers to a non-relational database, stored in a simple key-value schema. Does not support transactions, far exceeding SQL performance.

Ii. Introduction to Redis database

Memcashed introduction:

  • Early onNoSqlThe database
  • The data is in memory and generally does not support persistence
  • Support only simplekey-valuemodel
  • Typically a database that acts as a cache database to aid persistence

Redis database introduction:

  • Almost coveredMemcashedMost of the functions of
  • Data is stored in memory for persistence and is mainly used for backup and restoration
  • In addition to supporting simple key-value mode, it also supports multiple data structure storage, such aslist,set,hash,zsetAnd so on.
  • It is typically used as a cache database to assist persistent databases

Redis is an open source key-value storage system. Like Memcached, it supports storing a relatively large number of value types, including String (string), list(linked list), set(collection), zset(sorted set — ordered set), and hash (hash). All of these data types support push/ POP, add/remove, intersection union and difference sets, and richer operations, all of which are atomic. On this basis, Redis supports sorting in various ways. Like memcached, the data is cached in memory for efficiency. The difference is that Redis periodically writes the updated data to disk or the modified operation to the appended record file, and on this basis realizes master-slave synchronization.

Three, application scenarios

1. cache

First of all, 'Redis' is single-threaded. In addition to its own memory based, simple data structure, no multi-threaded lock events, more importantly,' Redis' uses I/O multiplexing technology. Because of these features, redis is often used to cache hot data and reduce database IO. In addition, it is also used for **session sharing in distributed environments.Copy the code

2. Store data

Due to the persistence capability of Redis, a variety of business scenarios can be applied by utilizing its diverse data structure features:

  1. Natural chronological ordering of data – list structure
  2. list–zsetAn ordered set
  3. Timeliness data–redisKey to set the validity period
  4. Counter, seckill – atomicity of increment method

Four, installation tutorial

  1. Download the redis-3.2.5.tar.gz package from the official website

  2. Copy the compressed package to the opt directory of the Linux system

  3. Use the decompression command tar -zxvf radis-3.2.5.tar.gz to decompress the folder

  4. Go to the decompressed folder and runmakeCommand, an error is found as follows:

  5. Yum install GCC: yum install gcc-C ++

  6. Use the make distclean command to clear the cache for the above execution

  7. Run the make command in the decompressed redis directory to install redis. The default installation directory is usr/local/bin

5. Start Redis

1. Use the Redis -server foreground to start the Redis server

2. Change redis to background boot

  • Copy the redis.conf file from the extracted directory to the opt/myRedis directory (your own installation directory).

  • usevi redis.confCommand to enter the configuration file and searchdaemonize, change its value to yes, save and exit.

  • Run the redis-server /opt/myRedis/redis.conf command to start the redis configuration file. Run redis-cli -h host name -p port number to access the Redis client

3. View and close Redis

  • Use the ps – ef | grep redis check to see if the server and the client is running in the background

  • Start Redis in the foreground and use the shutdown command once inside

  • Use kill P in the process

  • Run redis-cli shutdown

The use of Redis

Redis has 16 databases by default, like arrays starting from 0, using database 0 by default. Select a database using select Index. Redis’s five data types — String, List, set, zset, and Hash — all exist in key-value pairs

4. Basic grammar

1) String:
  • keys *: View all keys
  • exists key: Checks whether a key/value pair exists, returns 1 if it exists, and 0 if it does not
  • type key: Displays the key type
  • expire key secends: Sets the lifetime of an existing key
  • setex key secends value: Sets the expiration time of a key-value pair.
  • ttl key: Displays the time to Live of the key. -1 indicates that it will never expire, and -2 indicates that it has expired.
  • dbsize: Displays the number of keys in the current database.
  • flushdb: Clears the current library.
  • flushall: Clears all libraries.
  • append key value: Appends content to the value corresponding to key.
  • strlen key: Gets the length of the string.
  • setnx key value: Sets the key value only when the key cannot exist.
  • incr/decr key: Adds/subtracts a value, only for numeric values.
  • incrby/decrby keyStep size: Add or subtract the value by step size
  • mset key1 value2 key2 value2...: Sets multiple key-value pairs.
  • mget key1 key2...: Gets multiple key-value pairs.
  • msetnx key1 key2 ...: Sets both key and value pairs at the same time. This setting takes effect only if all given keys do not exist.
  • getrange key start end: gets the values of the specified range.
  • setrange key start value: duplicates the value from a position.
  • getset key value: Overrides the key and returns the value of the key.
2) List

Simple string list, the underlying implementation is a bidirectional linked list, the operation performance of two sections is very high, the middle operation performance is poor, ordered repeatable

  • lpush key value1 value2 value3… : Inserts the linked list value from the left

  • rpush key value1 value2… : Inserts the value of the list from the right

  • Lpop Key/RPOP Key: Pops a value from a linked list left/right.

  • Rpoplpush key1 key2: pop a value from the right side of key1 to the left side of key2.

  • Lrange key start stop: Finds elements in a specified range in the linked list. 0 indicates the table header and -1 indicates the table tail.

  • Lindex key index: Get the list elements by index (left to right)

  • Llen key: Gets the length of the list.

  • Linsert key before value newvalue: inserts a newvalue before the value

  • Lrem key N value: Deletes n values from left to right. If n is negative, deletes n values from right to left. 0 indicates that all files are deleted.

3） set

The underlying hash table has a value of null, so the complexity of operations is 1. It is unordered and unrepeatable

  • sadd key value1 value2 value3...: Creates a key of type set and adds data to it.
  • smembers key: Retrieves an element from the set.
  • sismember key value: Checks whether there are elements in the set whose value is value.
  • spop key: Randomly popping up an element in the collection will remove that element.
  • srandmembers key n: Randomly pops n elements from the collection without deleting them.
  • sinter key1 key2 : Returns the intersection of two collections.
  • sunion key1 key2: Returns the union of two collections.
  • sdiff key1 key2: Returns the difference set of two elements.
4) the hash

Especially good for storing objects. In Java, for example, you can convert a class to a JSON string and store it in a hash in Redis

  • hset key filed valueFiled is the name of the hash key. It is usually named in the format of user:1010: UID
  • hmset key filed1 value1 filed2 value2: Creates hash keys in batches.
  • hexist key filedFiled: Checks whether keys are filed in the hash.
  • hkeys key: Lists all key names in the hash set
  • hvals key: Lists all the values in the hash set.
  • hgetall key: Lists all key-value pairs in the hash.
  • hincrby key filed increment: Adds INCREMENT to the field key and returns the result.
  • hsetnx key filed value: Sets the field value of the hash table to value.
5) zSet

Sort set, an ordered set without repetition, uses score to sort from smallest to largest

  • zadd key score1 value1 score2 value2: Sets the value of multiple members and score. It can also be used to modify the score value of elements and add new elements.
  • zrange key start stop: Finds the elements of the zSet in the specified range by the subscript, 0 being the first and -1 being the last.
  • zrangebyscore key min max: finds all elements of score in the specified range. When min== Max, it means that elements are searched by score.
  • zrevrange key start stop: Sorts the elements of a specified range from largest to smallest.
  • zincrby key increment value: increments the element’s score.
  • zrem key value: Deletes elements.
  • zcount key min max: Counts the number of elements in the specified range score.
  • zrank key value: Returns the ranking of the value in the collection.

5. Related configuration of Redis

Run CD /opt/myRedis/redis.conf to open the redis configuration file. You can view the redis configuration information.

  • Case insensitive

  • Keyword Specifies the included configuration file

  • Bind: to bind a specific IP host to access the service (used with protected-mode)

  • Tcp-backlog: Queue for interface processes after the request arrives

  • Timeout: indicates the duration during which an idle client is enabled. 0 indicates that the client is not disabled.

  • Daemonize: Indicates whether to set it as a background process

  • Requirepass: Sets the redis access password

  • TCP Keepalive: checks the heartbeat of the client once every n seconds. The recommended value is 60 seconds

  • Pidfile: The location of the PID file. Each instance will generate a different PID file.

  • Loglevel: sets the loglevel debug verbose notice warning

  • Logfile: indicates the name of a logfile

  • Syslog: indicates whether to input redis logs to the Linux system log service

  • Database: Sets the number of libraries

  • Maxclient: indicates the maximum number of client connections

  • Maxmemory: The maximum amount of memory redis can use

Java connection

1. Preparation

  1. Download the necessary JAR packages:
  • Commons - pool2-2.4.2. Jar
  • Jedis - 2.7.2. Jar
  1. Modify the redis configuration file
  • Change protected-mode to no
  • Comment out bind 127.0.0.1
  1. Disable the firewall on centos
  • sudo systemctl stop firewalldtemporarily closed
  • sudo systemctl disable firewalld Then reboot shuts it down permanently
  • sudo systemctl status firewalldCheck the firewall status.

2. Code

import redis.clients.jedis.Jedis;
public class Client {
    public static void main(String[] args) {
        // Create a redis connection
        Jedis jedis = new Jedis("192.168.17.135".6379);
        // Create a String of data
        jedis.set("achao"."12");
        System.out.println(jedis.get("achao"));
        // Close the connectionjedis.close(); }}Copy the code

3, use,

Methods for redis operations are encapsulated in the Jedis class. See the basic syntax section above for details.

Redis transaction

A Redis transaction is a separate isolated operation: all commands in the transaction are serialized and executed sequentially. A transaction is not interrupted by command requests sent by other clients.

The main purpose of redis transactions is to concatenate multiple commands to prevent other commands from jumping the queue.

1, multi

Use this command to signal the start of a transaction, and all subsequent commands are written as a queue waiting to be executed.

2, the discard

Exit the multi state and do not execute all commands in the queue

3, the exec

After a transaction is started using MULTI, the commands in the queue to be executed can be executed sequentially using exec.

4. About error handling:

When we start a transaction and use exec to batch execute commands.

If the command is compiled with an error, the entire queue command is not executed.

If a command is run with an error, it will not be executed, and other commands will be executed sequentially — unlike our relational databases.

9. Redis Monitoring (Watch)

The Watch command in Redis can monitor one or more keys. If one of the keys is changed during a transaction, the transaction will not be executed and will be unmonitored. Let’s simulate this process with an example of multithreading changes to a key.

1. When the Watch is not used

127.0.0.1:6379 >set money 1000	# we set an initial amount of moneyOK 127.0.0.1:6379 > multi# start transaction execution
OK
127.0.0.1:6379> decrby money 100
QUEUED
127.0.0.1:6379> incrby money 100
QUEUED
The first client performed a transaction that was subtracted by 100 and then incremented by 100

# # # # # # # # # # # # # # # # # # # # # another client to perform 100 operations # # # # # # # # # # # # # # # # # #127.0.0.1:6379 > decrby money 100# # # # # # # # # # # # # # # # # # # # # the first client begins to execute transaction # # # # # # # # # # # # # # # # # # # # # # #127.0.0.1:6379 >exec(1)integer), 800 (2)integer) 900	The result is 900, the second client affecting our transaction
Copy the code

Obviously, we do not want this situation to happen, so we use the watch command to prevent this situation from happening

2. When using watch

127.0.0.1:6379 >setMoney 1000 OK 127.0.0.1:6379> Watch money# Start monitoring moneyOK 127.0.0.1:6379 > multi# execute transaction
OK
127.0.0.1:6379> decrby money 100
QUEUED
127.0.0.1:6379> incrby money 100
QUEUED

# # # # # # # # # # # # # # # # # # # # # another client to perform 100 operations # # # # # # # # # # # # # # # # # #
127.0.0.1:6379> decrby money 100
(integer) 900

# # # # # # # # # # # # # # # # # # # # # the first client begins to execute transaction # # # # # # # # # # # # # # # # # # # # # # #127.0.0.1:6379 >exec
(nil)		The transaction failed to execute with the return value null127.0.0.1:6379 > get the money"900" 	Get the value of money, which is 900

Copy the code

3, summarize

The watch command in a transaction is equivalent to an “optimistic lock” on the monitored key. The mechanism of this optimistic lock is equivalent to cas operation in Java, which records the monitored data and version of the monitored key. When the monitored key performs a transaction, the data or version is found to be inconsistent with the previously recorded data. The transaction is terminated to ensure data consistency.

When the second client in the above example subtracts from the monitored key, the version of the key changes. (In the above example, both the data and the version of the monitored key have changed. The reader can try to have the second client add or subtract 100 from the key.) See if the first client transaction executes successfully (the answer is no).

SpringBoot integration with Redis

1. Create the SpringBoot project

  • Open IDEA, select a new SpringBoot project, and check it when using dependencies in the boot pagespringwebandspringredisRely on
  • Problem solving (maven cannot import dependence) : real solution Could not transfer an artifact org. Springframework. The boot: spring – the boot – starter – parent: pom

2, description,

After SpringBoot2.x, the jedis used is replaced with the lettuce, the difference between the two is as follows:

Jedis: Direct connection is used. Multiple threads are not safe. If you want to avoid unsafe connection, you can only use the Jedis pool connection pool

Lettuce: With Netty, instances can be shared between multiple threads. There is no thread insecurity.

3. Connect and use Redis

  • Start Redis in Linux services, as described in Chapter 7.
  • Use the Properties file to configure the runtime environment for Redis
# configuration redis
spring.redis.url=127.0.0.1
spring.redis.port=6379
Copy the code
  • Basic data types to operate on Redis:
    • Injection RedisTemplate;
    • redisTemplate.opsForValue(): Operation string
    • redisTemplate.opsForList(): Operate linked lists
    • redisTemplate.opsForSet(): the operation is Set
    • Wait…
  • In addition, we use some transactions, delete and other common operations directly usedredisTemplateMethod operations in
  • Clearing the database passed. ProcedureredisTempldate.getConnectionFactory.getConnection()To get the connection to operate
  • More operations to explore

Redis persistence

Redis is an in-memory database, and if you do not save the in-memory database state to disk, the database state in the server will disappear once the server process exits. So Redis provides persistence! Redis persistence has the following two mechanisms:

1, RDB (Redis database)

The trigger event will write the data in memory to disk at a specified interval, which is commonly called snapshot. The snapshot file will be read directly into memory during recovery.

Redis will create a separate subprocess for persistence. It will first write data to a temporary file, and then replace the last persistent file with this temporary file after the persistence process is complete. ** The main process does not perform any IO operations during the entire process. ** This ensures extremely high performance. If large-scale data recovery is required and the integrity of the data recovery is not very sensitive, the RDB approach is higher than the AOF approach. The disadvantage is that the data may be lost after the last persistence. (Because persistence has trigger conditions and time intervals)

RDB saves the dump. RDB file. When we want to use a dump. RDB file (usually a redis data migration), we simply place the file in the bin directory of the redis program. Run the config get dir command

Trigger mechanism:

  • The save rule meets the case
  • performflushallWhen commanding
  • Exit Redis normally

advantages

  • Suitable for large-scale data recovery (due to efficient backup and small backup files)
  • Low data integrity requirements (because the last persistence may be incomplete)

disadvantages

  • It takes a certain amount of time for the process to operate. If Redis unexpectedly goes down, the last modified data will be lost
  • When forking, it takes up some content space

2, AOF (append only file)

Log each write operation, record all commands executed by Redis (read operations are not logged), only file is appended but file is not overwritten, redis reads the file and overwrites the build data (equivalent to re-executing the recorded command), the file is saved as appendone.aof by default

Appendonly is disabled by default and needs to be changed to yes in the redis startup configuration file. If there are some errors in the aOF reply file, you can use redis-check-aof –fix

advantages

  • Each change can be synchronized, resulting in better file integrity
  • If you synchronize once per second, you may lose one second of data
  • Never synchronized for maximum efficiency

disadvantages

  • Compared to data files, AOF is much larger than RDB, and repairs are slower than RDB
  • Aof is also slower than RDB, so our redis default persistence is RDB

Xii. Redis publishing and subscription

Redis publish subscribe is a message communication model: the sender sends the message and the subscriber receives the message. Such as wechat, Weibo.

The Redis client can subscribe to any number of channels.

Use the following command:

  • SUBSCRIBE channel1 channel2.. : Subscribe to multiple channels

  • PSUBSCRIBE pattern1 pattern2… : Subscribe to channels that fit a certain pattern

  • UNSUBSCRIBE channel1 channel2: UNSUBSCRIBE multiple channels

  • PUBLISH CHANEL Message: Publishes the message to the specified channel

Usage Scenarios:

  • Real-time messaging system, real-time chat
  • A slightly more complex implementation uses MQ middleware

Master/slave replication

1, concept,

Primary/secondary replication refers to the replication of data from one Redis server to other Redis servers. The former is called the master node and the leader node, and the latter is called the slave node

(slava/followers); Data replication is one-way and can only go from the master node to the slave node. The Master mode is mainly written, while the Slave mode is mainly read.

By default, each Redis server is a master node, and a master node can have multiple slave nodes (or none), but a slave node can only have one master node.

The primary and secondary replication functions include:

1. Data redundancy: Master/slave replication implements hot backup of data and is a data redundancy mode other than persistence.

2. Fault recovery: When the primary node has problems, multiple secondary nodes can provide services to achieve rapid fault recovery; It’s actually redundancy of services.

3. Load balancing: On the basis of master-slave replication and read/write separation, the master node can provide write services and the slave node can provide read services. (That is, when redis data is written, the application connects to the master node and when redis data is read, the application connects to the slave node to share the load of the server. Especially in the scenario of less write and more read, multiple nodes can share the read load. Can greatly increase the concurrency of redis server.

4. High availability cornerstone: In addition to the above, master-slave replication is the foundation upon which sentry and clustering can be implemented, hence master-slave replication is the foundation of High availability in Redis.

In general, using only one Redis server is not a good idea for a project (because there is always a chance that redis will go down) for the following reasons:

  1. Structurally, a single Redis server can have a single point of failure, and it is stressful for a single server to handle all the request load.
  2. In terms of capacity, the memory capacity of a single Redis server is limited. Even if the memory capacity of a Redis server is 256G, all the memory cannot be used as REDis memory storage. Generally speaking, the maximum memory used by a single Redis server should not exceed 20G.

The goods on the e-commerce website are generally uploaded once and viewed for countless times, which means “read more and write less”.

2. Environment configuration

For local testing, use the following methods to set up the cluster environment:

  • In Linux, use a different redis. Config file to start the redis service and the corresponding client:
    • Example Modify port information in the configuration file
    • Example Modify the PID name in the configuration file
    • Modify the log file in the configuration file
    • Modify the dump. RDB file in the configuration file
  • To bind a client that needs to be changed to a slave host, use the following command:
    • SLAVEOF host port
    • Use Info Replication to view primary and secondary information
    • Note: Using the client command line only temporarily establishes the master-slave relationship

Features:

  • The host can write and read, while the slave machine can only read the redis data in the host
  • When the host unexpectedly downtime, from the machine can still read the host downtime inside the redis data; After the host restarts, the slave server automatically connects to the host.
  • Restarting from an outage cannot connect to the host or continue reading values from the host. This is because the master-slave relationship established on the command line is temporary.

Replication principle:

After the Slave starts successfully, it automatically connects to the master and sends a sync command

The master receives the command to start the background saving process and collect all received commands to modify the data set. After the background process is complete, the master sends the entire data file to the slave and completes a complete synchronization. The synchronization modes are as follows:

Full synchronization: Synchronize all redis data from the master to the slave

Incremental synchronization: To synchronize new REDis data from master to Redis. The incremental basis is full

3. Sentinel

Problem: when we host goes down, our data can only achieve read operation, but we still want to continue to operate in the redis server, this time we hope to connect directly to the host from the machine into a new host, can be used in this machine from slaveof no one command manually.

Sentinel mode:

Sentinel mode is a special mode that can monitor whether the host fails in the background. If the host fails, it will automatically convert the slave library to the master library based on the number of votes.

Sentinel mode is a special mode, first of all redis provides sentinel command, sentinel is an independent process, as a process, it will run independently. The idea is that sentry monitors multiple Running Redis hosts by sending commands and waiting for redis servers to respond.

Sentinel mode requires additional processes to be started. Sentinel has two main functions:

  • The Redis server is sent back to monitor its running status, both primary and secondary, by sending commands.
  • When the sentry detects that the master is down, it automatically switches the slave to master and notifies other slave servers by publishing subscriptions. Modify the configuration file and have them switch hosts.

Problems: However, a sentinel process monitoring the Redis server can have problems (out-of-process termination), for which we can use multiple sentinels to monitor each other. This creates the multi-sentry mode to restart or create a new process for an unexpectedly terminated sentry.

Note that: A sentry when we detect a redis server goes down, the system will not be repaired immediately, because a sentry test may have deviation, this phenomenon is what we call subjective offline, when more than one sentry were detected by the server goes down and the number reaches a certain value, the sentinels will perform certain master-slave switch work, This is called objective downsizing.

Core commands:

Sentinel Monitor Name Host Port 1# create one in the bin folder
Copy the code

Advantages:

  • Sentinel cluster, based on the master-slave replication mode, has all the advantages of a master-slave configuration
  • Master and slave can be switched, failures can be transferred, and the system is more usable
  • Sentinel mode is an upgrade from master-slave mode, manual to automatic, and more robust

Disadvantages:

  • Redis is not easy to expand online, once the cluster capacity reaches online, it is very troublesome to expand online!
  • Implementing sentinel mode configuration is cumbersome and there are many options (link complexity). It is recommended to find related resources on the Internet for sentinel mode configuration

Cache penetration and avalanche

The cache to penetrate

1. Concept:

The concept of cache penetration is very simple. When the user wants to query a data and finds that the redis memory data is not available, the user will skip the Redis cache and directly look up the data in the database (the cache is not hit). If a malicious user (attacker) launches a large number of such queries against the database using data that is not in the Redis cache, the database will be under a lot of pressure. This phenomenon is called cache penetration.

2. Solutions

Bloom filter: IT is a data structure that stores all possible query parameters in the hash form and verifies them at the control layer. Any query parameters that do not match are discarded or returned with a fixed value, thus avoiding the query pressure on the underlying database

Existing problems:

  1. If null values can be cached, this means that the cache needs more space to store more keys, because there may be a lot of null keys.
  2. Even if the expiration time is set for null values, data at the cache layer and storage layer will be inconsistent for a period of time, which will affect services that need to maintain consistency.

Cache breakdown

1, concept,

This is a very hot key in a redis. Continuously accessed by the outside world (large concurrent requests), at the moment of the key failure, continuous large concurrent access will penetrate the Redis cache to directly access the database, and the underlying database will instantly bear a huge amount of traffic, which is the concept of breakdown.

2. Solutions

  • Set the hotspot to never expire
  • Add a mutex

Cache avalanche

1, concept,

The expiration of several caches during a period of time. Like redis down! At this point all data access falls on the underlying database instantly!

For example, when buying!

2. Solutions

  • Redis high availability: Set up several Redis (cluster)
  • Traffic limiting degradation: Controlling the number of threads accessing the underlying database after a cache failure
  • Data preheating: Cache the data that may be accessed to Redis and set different expiration times to avoid excessive concentration of expiration times