The problem background

  • The first time a Django project was released to production, it started up and was unable to connect to the Redis cluster.
  • The redis cluster of the test environment does not have a password, resulting in a different configuration format for the production of the test environment
  • The redis configuration used in the production environment has not been verified before, whether it is single machine or cluster
  • The password in the production environment redis specifically matches @. There might be a URI cut symbol conflict in Django Cache configuration. He encountered this pit in the celery broker_URL configuration before.

Thinking of the February

  • The configuration of the test environment is different from that of the production environment. Therefore, be careful that the configuration of the production environment cannot be copied from that of the test environment
  • First use the official Redis – CLI to determine whether Redis is a cluster or a client, and then modify the code related configuration
  • The production environment cannot be connected locally because you need to log in to the production environment via a jumper and debug directly with the Django shell instead of resubmitting and building after each code change.
  • When something goes wrong, make it a priority to suspect your own code rather than the framework itself, especially a framework as large as Django.

The screening process

Confirm the client type

  • Do the simplest set command test and report an error directly.
rhea-flaskapi-live-sg(sg2|10.xxx:40681)@/workspace$ redis-cli -h redis.xxx.shopee.io -p 10010 -a xxxx@123
redis.xxx.shopee.io:10010> set a 123
(error) MOVED 15495 10.xxx.104:10011
Copy the code
  • After the search engine query, confirm that this redis is a cluster, the cluster connection command isRedis -cli -c -h Connection address -p port -a PasswordThere are fewer orders from above-cparameter

Start the Django Shell connection to Redis

from django_redis import get_redis_connection
conn = get_redis_connection()
Copy the code

Just one connection failed

After confirming that the account password and port were ok, the question colleague asked whether the configuration had been verified. He said that the CV was directly from the Internet, which had not been verified at all. Pit!!

Modify the configuration and test again

Several changes were made to the production configuration and test configuration

  • LOCATIONfromstrtoList[str], plus defaultdb, that is,[ f'redis://{DEFAULT_REDIS_URL}:{DEFAULT_REDIS_PORT}/0']
  • REDIS_CLIENT_CLASS': 'rediscluster.RedisClusterThe cluster connects to the required clients
  • The connection pool configuration is changed to'CONNECTION_POOL_CLASS': 'rediscluster.connection.ClusterConnectionPool'

Confident enough to restart the Django shell test, the connection still works! At this time the mood began a little bad ~

Calm down, Django shell won’t work. How about using Python shell direct connection?

Python Shell directly connects to Redis

There’s nothing wrong with it! It’s connected!

A face meng force, this exactly is what problem!

Never give up, repeatedly modify the configuration, and test Django Shell with Redis

The result is still not connected.

Thinking on the way home from work

Imperceptibly already arrived at 9 o ‘clock in the evening, very tired, do not want to roll. Come home from work

I thought about it all the way home. Does Django really not connect to a redis cluster whose password contains the @ sign? Ask yourself repeatedly.

I asked my colleagues if there were other Redis clusters in the production environment that could be used for debugging. Unfortunately, it doesn’t.

How about I create my own Redis cluster and set the password to contain the @ sign?

However, creating your own redis cluster is a hassle. To install a virtual machine locally, think of a bunch of configurations and simply quit.

Back home after the struggle

After taking a bath, I chatted with my wife for about 1 hour. It’s past 11:00. Ready for bed?

It’s impossible, it’s hard to sleep with the problem! Ah, this is an old problem.

I hit on a little trial-and-error. Setting up your own local cluster is a hassle, and the company doesn’t have any extra clusters.

How about buying a cloud redis cluster? Just get out of bed and turn on the computer.

On the principle of minimal trial and error, buy cloud Redis clusters rather than build them yourself

Then the question came again, Ali cloud or Tencent cloud?

Considering that Double 11 bought Tencent cloud 2C 4G 8m server, only 199 can be 3 years.

Before buying ali cloud that 1C 2G 1M server, 3 years also want more than 100.

Instant Tencent cloud good impression multiplication, decided to buy Tencent cloud first.

Buy Tencent Cloud Redis cluster

A meal of operation, found Tencent cloud is really difficult to use:

  • Purchase page selected configuration, after submission, because I have no balance, prompted me to recharge. When I finished my recharge, the configuration I had selected was gone, so I had to choose again.
  • Redis cluster does not have restart function.
  • When setting a security group, you can bind instances only on the page of the security group. Security group cannot be bound on instance page.

Most importantly, after the instance is bound to a security group, the external network is still inaccessible. (Anyway, I’m mad anyway)

Buy Aliyun Redis cluster

Tencent Cloud was too disappointed, had to put the last straw on Ali Cloud.

Fortunately, Aliyun did not disappoint me!

Kaka kaka, a meal of operation:

  • Create instance, this ali cloud is much slower than Tencent cloud, about 7 minutes (pure feeling)

  • Configure external network access, very convenient. This operation logic is simply second Tencent cloud.

  • Configure whitelist, redis- CLI connection test, successfully passed!

  • Modify the configuration to start a Django shell test.

Password contains @ sign, but the connection is fine!!

At this point, the problem is finally solved!!

I can’t wait to go to the company to verify tomorrow, but when I look back, it is already 1:30 in the night.

He said to himself, “Go to sleep, Curly.”

Check in the next day

Rhea - flaskapi - live - sg (sg2 | XXXX: 26863) @ $python3.8 / workspace. The manage py shell -- Settings = rhea. Settings. Prod Rhea. Settings. prod Python 3.8.7 (default, Oct 1 2021, 14:58:33) [GCC 5.4.0 20160609] on Linux Type"help"."copyright"."credits" or "license" for more information.
(InteractiveConsole)
>>> from django_redis import get_redis_connection
>>> conn = get_redis_connection()
>>>>conn.get("rhea:access_token:abc")
'[email protected]'

Copy the code

After comparison, it is found that the configuration only needs the production configuration and only needs to add one more configuration to the test configuration:

Fixing the most annoying bugs usually requires only a small change

Why didn’t the test environment report any errors??

Because the Redis cluster in the test environment does not require a password

conclusion

  • Any code that has not been verified by itself must be used with caution

  • Minimum trial and error principle. When you want to verify something, do it at the lowest cost possible

  • Try to keep the production and test environments in the same configuration