“This is the second day of my participation in the Gwen Challenge in November. See details: The Last Gwen Challenge in 2021”

Focus on the public account [High-performance architecture exploration], reply [PDF], free access to classic computer books

Hello everyone, I’m Yule!

A few days ago, I suddenly received an alarm, the online service crashed, and then restarted automatically.

As it is the double Eleven period, the business is mainly stable, and the online service crashes, which is not a small matter, log in the online server immediately, analyze the reason, and solve the problem quickly.

Use this article to document the entire breakdown analysis and resolution process.

Receive the alarm

After going to work in the morning, I was paddling and suddenly received an email alarm as follows:

Problem analysis

Log in to the online server immediately and debug the stack information by GDB.

Stack information is as follows:

#0  0x0000003ab9a324f5 in raise () from /lib64/libc.so.6
#1  0x0000003ab9a33cd5 in abort () from /lib64/libc.so.6
#2  0x0000003abcebea8d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x0000003abcebcbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x0000003abcebcc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x0000003abcebcd32 in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x00000000006966bf in Json::throwRuntimeError(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#7  0x0000000000681019 in Json::Reader::readValue() ()
#8  0x000000000068277c in Json::Reader::readArray(Json::Reader::Token&) ()
#9  0x0000000000681152 in Json::Reader::readValue() ()
#10 0x00000000006823a6 in Json::Reader::readObject(Json::Reader::Token&) ()
#11 0x00000000006810f5 in Json::Reader::readValue() ()
#12 0x0000000000680e6e in Json::Reader::parse(char const*, char const*, Json::Value&, bool) ()
#13 0x0000000000680c52 in Json::Reader::parse(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Json::Value&, bool) ()
......
Copy the code

Json::Reader:: throwRuntimeError is thrown when Json::Reader::readValue is called after Json::Reader::readValue is called The exception.

See where the Json::throwRuntimeError function is called:

src/lib_json/json_writer.cpp:    throwRuntimeError("commentStyle must be 'All' or 'None'");
src/lib_json/json_reader.cpp:  if (stackDepth_g >= stackLimit_g) throwRuntimeError("Exceeded stackLimit in readValue().");
src/lib_json/json_reader.cpp:  if (stackDepth_ >= features_.stackLimit_) throwRuntimeError("Exceeded stackLimit in readValue().");
src/lib_json/json_reader.cpp:    if (name.length() >= (1U<<30)) throwRuntimeError("keylength >= 2^30");
src/lib_json/json_reader.cpp:    throwRuntimeError(errs);
src/lib_json/json_value.cpp:    throwRuntimeError(
src/lib_json/json_value.cpp:    throwRuntimeError(
src/lib_json/json_value.cpp:JSONCPP_NORETURN void throwRuntimeError(JSONCPP_STRING const& msg)
src/lib_json/json_valueiterator.inl:  throwRuntimeError("ConstIterator to Iterator should never be allowed.");
Copy the code

Go to the corresponding function

bool Reader::readValue() { if (stackDepth_g >= stackLimit_g) throwRuntimeError("Exceeded stackLimit in readValue()."); ++stackDepth_g; . . --stackDepth_g; return successful; }Copy the code

Found in satisfying the condition

stackDepth_g >= stackLimit_g
Copy the code

StackDepth_g and stackLimit_g declare a throwRuntimeError.

static int const stackLimit_g = 1000;
static int       stackDepth_g = 0; 
Copy the code

The question is pretty straightforward:

❝

StackDepth_g is a static global variable, thread unsafe, and the service in question is multithreaded

❞

In this preparation for ridicule, the author uses the jSONCPP object, is a local variable in the thread, so there will be no multi-thread access to the same local JSONCPP object, so it is determined that the global variable multi-thread access caused by. An open source project with global variables is not allowed in the specification.

Then I googled everyone who has had similar problems.

Problem solving

To solve the crash problem, you need to first look at how it is used, find a thread-safe interface, or replace it with another library.

Modify jSONCPP source code

There are two solutions to the thread-safety problem: In order to improve business performance, internal locks are optimized in other ways. For example, mutex uses double buffer to replace them, although the process of mutex unlocking a lock is only 100ns.

2. Put the above global variables into the Json object, so that the local variables will not crash. However, there is a problem with this scheme, that is, the change point is very large, and a lot of strict testing is required, so give up.

Therefore, considering the above two points, we decided to adopt other safer and more reliable ways to solve the online crash problem.

Using rapidjson

Rapidjson is used because there are dozens of services online, most of which use RapidJSON. Only a few services, such as the one that crashed online, use JSONCPP for historical reasons.

Rapidjson: RapidJSON

  • RapidJSON is a C++ JSON parser and generator. It was inspired by RapidXml.
  • RapidJSON is small and complete. It supports both SAX and DOM style apis. The SAX parser has only about 500 lines of code.
  • RapidJSON fast. Its performance is comparable to strlen(). Supports SSE2/SSE4.2 acceleration.
  • RapidJSON independence. It does not rely on external libraries such as BOOST. It doesn’t even depend on STL.
  • RapidJSON is memory friendly. On most 32/64-bit machines, each JSON value takes up only 16 bytes (excluding strings). It uses a fast memory allocator by default, allowing the parser to allocate memory compactly.
  • RapidJSON is Unicode-friendly. It supports UTF-8, UTF-16, and UTF-32 (big-endian/little-endian), and internally supports detection, verification, and transcoding of these encodings. For example, RapidJSON can parse a UTF-8 file into the DOM and transcode the JSON string in it to UTF-16. It also supports surrogate pairs and “\u0000” (null characters).

However, RapidJSON needs to be used with extreme care for performance.

❝

Rapidjson doesn’t use the local string right away. Instead, rapidJSON accesses the contents of the local string at the end, when the local string is already out of scope, resulting in garbled content.

❞

conclusion

When using open source projects, do your research and, if necessary, go through the source code implementation (this is a bit difficult 😁), otherwise you can easily fall into the trap.

When using libcurl as an httpclient, there was a bug in libcurl that crashed online, and it took two nights to fix it.

A C++ deep as the sea, from XX is passers-by.

To wait jie in <<STL source code analysis >> on a sentence as the conclusion of this article:

❝

Before the source code, no secrets.

❞

‘.