Troubleshooting Process

A test colleague prompts the application popup occasionally disappear Bug.

By checking the internal native crash stack information of the device, it can be seen that the crash occurs once every 20 minutes, with a relatively consistent frequency. Attached is a portion of the Tombstone log

Model: 'rk3288' Build fingerprint: 'Android/rk3288 / rk3288: the 6.0.1 / MXC89K/ci07121450: userdebug/test - keys' ABI:' arm 'pid: 3829, tid: 4346, name: Thread-749 >>> com.xxxxxx <<< signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x9c Abort message: 'art/runtime/thread.cc:1237] Native thread exited without calling DetachCurrentThread: Thread[12,tid=4346,Native,Thread*=0xb8933778,peer=0x331c5100,"Thread-749"]'Copy the code

Based on the crash information, it is very clear that the DetachCurrentThread was not called in pairs after AttachCurrentThread.

View the code, pseudo-code implementation is as follows:

// Destructor to DetachCurrentThread
static void detachDestructor(void* arg)
{
    pthread_t thd = pthread_self(a); JavaVM* jvm = (JavaVM*)arg;LOGI("detach thread");
    jvm->DetachCurrentThread(a); }static JNIEnv* tryAttach(a){
    JNIEnv *env = NULL;
    if(javaVM == NULL) {return env;
    }
    if (javaVM->GetEnv((void**) &env, JNI_VERSION_1_4) ! = JNI_OK) {pthread_key_t detachKey;
        pthread_key_create(&detachKey, detachDestructor);
        javaVM->AttachCurrentThread(&env, NULL);
        pthread_setspecific(detachKey, javaVM);
    }
    return env;
}
Copy the code

The code looks fine at first glance, using pthread_key_create() to define the destructor that will be called before the thread exits and then DetachCurrentThread()

If the Tombstone file prints “detach Thread “from the detachDestructor destructor, we will check the log and see that LOGI(“detach Thread “) is not called. This line means that JVM ->DetachCurrentThread(); It’s not called at all. What’s the reason?

  • Normally, we call the same pthread the first timejavaVM->GetEnv((void **) &env, JNI_VERSION_1_4)Returns theJNI_EDETACHEDAnd a subsequent call to the function on this thread will return directly since it was attachedJNI_OK.

Check the call flow of tryAttach and find the pseudocode as follows:

void *runSomthing(void *arg) {
	JNIEnv* env = tryAttach(a); jstring result = (jstring) env->CallObjectMethod(object, xxMethodId);
}

void jniMethod(a) {
	while(1) {
		if(xxxxxxxx) {
			pthread_create(&newthread, &attr, &runSomthing,
	                           (void(*)intptr_t) clientxxx)
		}
	}
}
Copy the code

When the JNI method is called, a ServerSocket will be opened and waiting to receive the client’s request. Once there is a request for change, a PThread will be created and processed in it. Therefore, for this logic, every time tryAttach is executed, JavaVM ->GetEnv is not JNI_OK. The pthread_KEY_CREATE process is required each time.

The canvass has stalled again. No ideas.

Looking at the normal log again, “Detach Thread” is printed as normal, indicating the process that caused the destructor detachDestructor not to execute. Doubt pthread_key_create (& detachKey detachDestructor); There is a problem with the process. If you look at the C++ interface documentation,

This function has a return value indicating whether it was successfully created.

int pthread_key_create(pthread_key_t *key, void (*destructor)(void*));

//If successful, the pthread_key_create() function shall store 
//the newly created key value at *key and shall return zero.
// Otherwise, an error number shall be returned to indicate the error.

//The pthread_key_create() function shall fail if:
//[EAGAIN]
//The system lacked the necessary resources to create another thread-specific data key, 
//or the system-imposed limit on the total number of keys 
//per process {PTHREAD_KEYS_MAX} has been exceeded.
//[ENOMEM]
//Insufficient memory exists to create the key.
Copy the code

EAGAIN (11) indicates that the current process has allocated more PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX PTHREAD_KEYS_MAX

Try adding two lines to print, and increase the number after each successful creation

  static int pthread_count = 0;

	static JNIEnv* tryAttach(a){
					int pthreadKeyCreateResult = pthread_key_create(&detachKey, detachKeyDestructor);
	        if (pthreadKeyCreateResult == JNI_OK) {
	            if(lastKey ! = detachKey) { pthread_count++; } lastKey = detachKey;LOGI("create thread key Success is: %d, Max key size:%d, total Count:%d", detachKey, PTHREAD_KEYS_MAX, pthread_count);
	        } else {
	            LOGI("create thread key failed, errorCode: %d,result is: %d, Max key size:%d, total Count:%d", pthreadKeyCreateResult, detachKey, PTHREAD_KEYS_MAX, pthread_count); }}Copy the code

The results are as follows:

create thread key failed, errorCode: 11,result is: 0, Max key size:128, total Count:110
Copy the code

Check the errno-base.h file. EAGAIN is 11, indicating that the pthread_key allocated by the current process exceeds the limit. The 110 we actually created, plus other system usage, could not be created when PTHREAD_KEYS_MAX (128) was exceeded.

pthread_key_create(&detachKey, detachDestructor); The JVM ->DetachCurrentThread() will not be called when the thread exits.

The solution

Knowing that the crash was caused by a pthread_KEY_CREATE failure, it was easy to think of solving the problem by using global pthread_key reuse.

Create pthread_KEY_CREATE in system. onload and use it later.

The code is as follows:

static pthread_key_t g_key;
static JNIEnv* tryAttach(a) {
	JNIEnv *env = NULL;
    if(javaVM == NULL) {return env;
    }
    if (javaVM->GetEnv((void**) &env, JNI_VERSION_1_4) ! = JNI_OK) { javaVM->AttachCurrentThread(&env, NULL);
        pthread_setspecific(detachKey, javaVM);
    }
    return env;
}

JNIEXPORT jint JNICALL JNI_OnLoad(JavaVM *vm, void *reserved) {
	int ret = pthread_key_create(&g_key, detachDestructor);
    if(ret ! = JNI_OK) {LOGI("create thread key error ret:%d.", ret); }}Copy the code