1. Introduction

Because of the difference of background applications running environment including container operation parameters, application configuration, etc., in the online application problems are could miraculously screening problem, such as in an interface delay problems or framework container loading to different expected configuration and common problem scenarios, such as is usually the code update log on again to play everywhere, repackaged deployment, It’s a waste of time and efficiency, especially during the critical launch phase. With the development of technology, APM link tracking has solved some problems to a certain extent, but some strange problems need to be located, analyzed and solved by online debugging. Arthas has become the preferred tool for troubleshooting problems in Java background application development. Arthas has been in use for almost two years now, but it is extremely useful. This article focuses on the implementation principles of Arthas and some common scenarios.

2. Aarthas is introduced

Arthas is an open source Java diagnostics tool from Alibaba that is a favorite among developers.

  • Official documentation: arthas.aliyun.com/zh-cn/index…
  • Source code repository: github.com/alibaba/art…

3. Implementation principle

3.1 Java Agent Technology

Arthas is implemented mainly through Java Agent technology, which is a jar package that functions as a plug-in. This JAR package is loaded through JVMTI (JVM Tool Interface). Finally with the help of JPLISAgent (Java Programming Language Instrumentation Services Agent) to complete the modification of the target code.

The Java Agent technology provides the following functions:

  • Interception modifies bytecode before loading Java files

  • Changes to the bytecode of the loaded class at run time

  • Gets all classes that have been loaded

  • Gets all classes that have been initialized

  • Gets the size of an object

  • Add a JAR to the bootstrapClasspath as a high priority to be loaded by the bootstrapClassloader

  • Add a JAR to the classpath for the AppClassloard to load

  • Set the prefix of some native methods, mainly for rule matching when searching for native methods

A simple Java Agent implementation is as follows:

public class AgentApplication { public static void premain(String arg, Instrumentation instrumentation) { instrumentation.addTransformer(new DumpClassesService()); } } public class DumpClassesService implements ClassFileTransformer { @Override public byte[] transform(ClassLoader loader, String className, Class<? > classBeingRedefined, ProtectionDomain ProtectionDomain, byte[] classfileBuffer){CtClass cl = null; try { ClassPool classPool = ClassPool.getDefault(); cl = classPool.makeClass(new ByteArrayInputStream(classfileBuffer)); For (CtMethod method: cl.getDeclaredMethods()) {method.addLocalVariable("start", ctclass.longType); method.insertBefore("start = System.currentTimeMillis();" ); String methodName = method.getLongName(); InsertAfter (" system.out.println (\"" + methodName + "cost: \" + (System" +" .currentTimeMillis() - start));" ); return cl.toBytecode(); } catch (Exception e) { e.printStackTrace(); } } } <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-jar-plugin</artifactId> <configuration> <archive> <manifestEntries> <Premain-Class>com.demo.AgentApplication</Premain-Class> <Agent-Class>com.demo.AgentApplication</Agent-Class> <Can-Redefine-Classes>true</Can-Redefine-Classes> <Can-Retransform-Classes>true</Can-Retransform-Classes> </manifestEntries> </archive> </configuration> </plugin>Copy the code

  • Loading on Startup: The startup parameters are added. -javaAgent :[path], where path is the jar package path of the agent

    java -javaagent:/java-agent.jar hello

  • In the operation of loading: use the com. Sun. View the attach the VirtualMachine load,

    String jvmPid = [pid]; VirtualMachine jvm = VirtualMachine.attach(jvmPid); Jvm.loadagent ([jar path for agent]); jvm.detach();

Through Java Agent technology for class bytecode modification is the main use of Java Instrumentation API. It depends on the Attach API mechanism of JVMTI. Currently, the Instrument classes support changes to class definitions at runtime. As shown above, to use Instrument’s class modification capabilities, we need to implement its provided ClassFileTransformer interface and define a ClassFileTransformer. The transform() method in the interface is called when the class file is loaded, and in the Transform method, you can use tools like ASM or Javassist to rewrite or replace the bytecode passed in, generate new bytecode and return it.

/** * retriggers class loading for classes already loaded by the JVM. The Transformer registered above is used. * retransformation can modify the method body, but cannot change the method signature, add and delete method/Class member attributes */ void retransformClasses(Class<? >... classes) throws UnmodifiableClassException; Long getObjectSize(Object objectToSize); / * * * to add a jar to the bootstrap classpath in this * / void appendToBootstrapClassLoaderSearch (JarFile JarFile); Class[] getAllLoadedClasses();Copy the code

3.2 Arthas main structure

Arthas doesn’t have a lot of code and the structure is fairly straightforward. Below are the main components of Arthas and how they relate to each other.

The main components are as follows:

  • Arthas-core. jar is a server-side startup entry class that calls VirtualMachine#attach to the target process and loads arthas-agent.jar as the agent package.
  • Arthas-agent. jar can be used either in premain (statically specified with the -agent parameter before the target process starts) or in AgentMain (attached after the process starts). Arthas-agent loads the Configure class and ArthasBootstrap in arthas-core-jar using a custom ArthasClassLoader. Arthas-spy.jar is also used when the program runs.
  • Arthas-spy.jar contains only the Spy class. The purpose is to load the Spy class using BootstrapClassLoader so that the target process’s Java applications can access the Spy class. Using ASM to modify bytecode, you can weave the Spy class’s methods ON_BEFORE_METHOD, ON_RETURN_METHOD, and so on into the target class.
  • Arthas-client. jar is a client application that connects to the arthas-core.jar startup server code using Telnet. Arthas-boot.jar and as.sh are generally responsible for the startup.

3.3 Initializing the Main Process

Arthas can be launched either as an Agent plug-in or directly through the Arthas class. The first is to specify the Arthas agent startup class in the maven package, which will be initialized by calling the AgentBootStrap#main method once the java-agent is started.

<archive>
    <manifestEntries>
        <Premain-Class>com.taobao.arthas.agent.AgentBootstrap</Premain-Class>
        <Agent-Class>com.taobao.arthas.agent.AgentBootstrap</Agent-Class>
        <Can-Redefine-Classes>true</Can-Redefine-Classes>
        <Can-Retransform-Classes>true</Can-Retransform-Classes>
    </manifestEntries>
</archive>
Copy the code

In AgentBootStrap#main, you initialize the hook object used in the Advice enhancement and load the Spy class into the BootstrapClassLoader.

private static synchronized void main(final String args, final Instrumentation inst) { try { ... Final ClassLoader agentLoader = getClassLoader(inst, spyJarFile, agentJarFile); // Initialize the hook object initSpy(agentLoader); Thread bindingThread = new Thread() {@override public void run() {try {// bind(inst, agentLoader, agentArgs); } catch (Throwable throwable) { throwable.printStackTrace(ps); }}}; bindingThread.setName("arthas-binding-thread"); bindingThread.start(); bindingThread.join(); } catch (Throwable t) { ... } } private static ClassLoader getClassLoader(Instrumentation inst, File spyJarFile, The File agentJarFile) throws Throwable {/ / add Spy to BootstrapClassLoader inst. AppendToBootstrapClassLoaderSearch (new JarFile(spyJarFile)); Return loadOrDefineClassLoader(agentJarFile); return loadOrDefineClassLoader(agentJarFile); }Copy the code

There are various Advice hooks in the Spy class that reference static Method objects that are used when byte enhancement occurs.

private static void initSpy(ClassLoader classLoader) throws ClassNotFoundException, NoSuchMethodException { Class<? > adviceWeaverClass = classLoader.loadClass(ADVICEWEAVER); Method onBefore = adviceWeaverClass.getMethod(ON_BEFORE, int.class, ClassLoader.class, String.class, String.class, String.class, Object.class, Object[].class); Method onReturn = adviceWeaverClass.getMethod(ON_RETURN, Object.class); Method onThrows = adviceWeaverClass.getMethod(ON_THROWS, Throwable.class); Method beforeInvoke = adviceWeaverClass.getMethod(BEFORE_INVOKE, int.class, String.class, String.class, String.class); Method afterInvoke = adviceWeaverClass.getMethod(AFTER_INVOKE, int.class, String.class, String.class, String.class); Method throwInvoke = adviceWeaverClass.getMethod(THROW_INVOKE, int.class, String.class, String.class, String.class); Method reset = AgentBootstrap.class.getMethod(RESET); Spy.initForAgentLauncher(classLoader, onBefore, onReturn, onThrows, beforeInvoke, afterInvoke, throwInvoke, reset); }Copy the code

In AgentBootStrap#bind, the ArthasBootStrap class will be loaded and the bind method will be called, which will start the server side.

private static void bind(Instrumentation inst, ClassLoader agentLoader, String args) throws Throwable { Class<? > classOfConfigure = agentLoader.loadClass(ARTHAS_CONFIGURE); Object configure = classOfConfigure.getMethod(TO_CONFIGURE, String.class).invoke(null, args); int javaPid = (Integer) classOfConfigure.getMethod(GET_JAVA_PID).invoke(configure); Class<? > bootstrapClass = agentLoader.loadClass(ARTHAS_BOOTSTRAP); Object bootstrap = bootstrapClass.getMethod(GET_INSTANCE, int.class, Instrumentation.class).invoke(null, javaPid, inst); boolean isBind = (Boolean) bootstrapClass.getMethod(IS_BIND).invoke(bootstrap); }Copy the code

The other option is to start directly with the Arthas class, first iterate through all the processes, attach the Agent class to the target process using the VirtualMachine class, and initialize the Agent.

private void attachAgent(Configure configure) throws Exception { VirtualMachineDescriptor virtualMachineDescriptor = null; For (VirtualMachineDescriptor: VirtualMachine.list()) {String pid = Description.id (); if (pid.equals(Integer.toString(configure.getJavaPid()))) { virtualMachineDescriptor = descriptor; } } VirtualMachine virtualMachine = null; Try {// Use attach(String PID) this way if (null == virtualMachine descriptor) {virtualMachine = virtualMachine. Attach ("" + configure.getJavaPid()); } else { virtualMachine = VirtualMachine.attach(virtualMachineDescriptor); } Properties targetSystemProperties = virtualMachine.getSystemProperties(); String targetJavaVersion = targetSystemProperties.getProperty("java.specification.version"); String currentJavaVersion = System.getProperty("java.specification.version"); . virtualMachine.loadAgent(configure.getArthasAgent(), configure.getArthasCore() + ";" + configure.toString()); } finally { if (null ! = virtualMachine) { virtualMachine.detach(); }}}Copy the code

3.4 Main communication process

Arthas applications are designed based on a C/S communication architecture that supports Telnet and Http client protocol communication. Upon receiving a connection from a client, the Arthas application generates a session window for each connection, and then parses the request content and generates a command for task control to complete the response. Each client communication corresponds to a unique ShellImpl implementation, which contains a unique Session instance and holds JobControllerImpl and InternalCommandManager objects used to assemble asynchronous tasks to execute the command.

During startup on the server side ArthasBootstrap#bind is called, which launches a server instance of the Telnet and Http communication protocols and receives requests. The InternalCommandManager class records all commands, and the corresponding Command implementation class can be searched by name, where the Command class is wrapped in AnnotatedCommand class and placed in the list.

public class ArthasBootstrap { public void bind(Configure configure) throws Throwable { try { shellServer = new ShellServerImpl(options, this); BuiltinCommandPack builtinCommands = new BuiltinCommandPack(); List<CommandResolver> resolvers = new ArrayList<CommandResolver>(); resolvers.add(builtinCommands); / / initializes the Telnet protocol server shellServer registerTermServer (new TelnetTermServer (configure. GetIp (), the configure. GetTelnetPort (), options.getConnectionTimeout())); / / initialize the HTTP protocol server shellServer registerTermServer (new HttpTermServer (configure. GetIp (), the configure. GetHttpPort (), options.getConnectionTimeout())); / / server registered command parser instance for (CommandResolver resolver: resolvers) {shellServer. RegisterCommandResolver (resolver); }... } catch (Throwable e) { ... } } } public class BuiltinCommandPack implements CommandResolver { ... Private static void initCommands() {// wrapper AnnotatedCommandImpl class, Add commands. Add (Command. Create (HelpCommand. Class)); commands.add(Command.create(KeymapCommand.class)); } } public class InternalCommandManager { private final List<CommandResolver> resolvers; Private static Command getCommand(CommandResolver CommandResolver, String name) { List<Command> commands = commandResolver.commands(); if (commands == null || commands.isEmpty()) return null; } for (Command command : commands) { if (name.equals(command.name())) { return command; } } return null; }Copy the code

Each request of the client responds in the form of an asynchronous task. Taking the request-response process of TelnetTermServer as an example, the Server holds an implementation instance of JobController to task-process the response. JobControllerImpl generates a command handling object that calls the specific methods of the command.

Public interface JobController {// Get all jobs Set<Job> jobs(); Job getJob(int id); CreateJob createJob(InternalCommandManager commandManager, List<CliToken> tokens, ShellImpl Shell); // The task postprocessor void close(Handler< void > completionHandler); // Task close void close(); } public class JobControllerImpl implements JobController { @Override public Job createJob(InternalCommandManager CommandManager, List < CliToken > tokens, ShellImpl shell) {/ / generated in the form of incremental task id int jobId = idGenerator. IncrementAndGet (); . Process Process = createProcess(tokens, commandManager, jobId, shell.term()); process.setJobId(jobId); JobImpl job = new JobImpl(jobId, this, process, line.toString(), runInBackground, shell); Jobs. put(jobId, job); return job; } private Process createProcess(List<CliToken> line, InternalCommandManager commandManager, int jobId, Term term) { try { ListIterator<CliToken> tokens = line.listIterator(); while (tokens.hasNext()) { CliToken token = tokens.next(); If (token. IsText ()) {/ / obtain corresponding Command from the Command list Command Command = commandManager. GetCommand (token) value ()); . }}... } catch (Exception e) { ... } } private Process createCommandProcess(Command command, ListIterator<CliToken> tokens, int jobId, Term term) { .... ProcessOutput ProcessOutput = new ProcessOutput(stdoutHandlerChain, cacheLocation, term); return new ProcessImpl(command, remaining, command.processHandler(), ProcessOutput); }Copy the code

3.5 Mechanism of Bytecode Enhancement

It can be seen from the source code that various commands input from the control end will be encoded into communication protocols by the Telenet client and then sent to the server, where they will be parsed into various commands, such as common commands such as trace/watch, etc. These commands will be monitored with specific class # methods. The monitoring results are returned to the client for display, similar to an AOP implementation where the target class or object is enhanced in the Enhance class to do the statistics and present the results.

The main logic in Enhance is shown below. This class inherits from the ClassFileTransformr class and can be enhanced in two main ways: by calling transform when the class is loaded, or by calling the Enhance method after receiving the command to Enhance the target class. Enhancements use the ClassReader/ClassWriter class of the ASM framework for class enhancements. The specific enhancement logic is in the AdviceWeave class.

public class Enhancer implements ClassFileTransformer { ... @Override public byte[] transform(...) { final ClassReader cr; Final byte[] byteOfClassInCache = classBytesCache.get(classBeingRedefined); final byte[] byteOfClassInCache = classBytesCache.get(classBeingRedefined); if (null ! = byteOfClassInCache) { cr = new ClassReader(byteOfClassInCache); } else {// If not in the cache from the original bytecode enhancement cr = new ClassReader(classfileBuffer); } / / bytecode enhancement final ClassWriter the cw = new ClassWriter (cr, COMPUTE_FRAMES | COMPUTE_MAXS) {... }; New AdviceWeaver(adviceId, isTracing, skipJDKTrace, cr.getClassName(), methodNameMatcher, affect, cw), EXPAND_FRAMES); final byte[] enhanceClassByteArray = cw.toByteArray(); ClassBytesCache. Put (classBeingRedefined, enhanceClassByteArray); . try { spy(inClassLoader); } catch (Throwable t) { ... } return enhanceClassByteArray; } catch (Throwable t) { ... } return null; } public static synchronized EnhancerAffect enhance(...) {... Final Enhancer = new Enhancer(adviceId, isTracing, skipJDKTrace, enhanceClassSet, methodNameMatcher, affect); try { ... // Add class transformer inst.addTransformer(enhancer, true); / / batch enhance the if (GlobalOptions isBatchReTransform) {final int size = enhanceClassSet. The size (); final Class<? >[] classArray = new Class<? >[size]; arraycopy(enhanceClassSet.toArray(), 0, classArray, 0, size); If (classarray. length > 0) {if (classarray. length > 0) {if (classarray. length > 0) {if (classarray. length > 0) {if (classarray. length > 0) {if (classarray. length > 0) { } } else { for (Class<? > clazz: enhanceClassSet) {try {// Convert the class definition, then call the Enhance#tranformClass method inst.retransformclasses (clazz); . } catch (Throwable t) { .... } } } catch(Exception ex) { ... } finally {// Remove class transformer inst.removeTransformer(enhancer) from Instrument; } return affect; }}Copy the code

The enhancement here is very similar to AOP in that it uses a ClassVisitor and then inserts the before-and-after methods into the target method body, triggering the listener’s methods when requested.

public class AdviceWeaver extends ClassVisitor implements Opcodes { ... Private final static Map<Integer/*ADVICE_ID*/, AdviceListener> advices = new ConcurrentHashMap<Integer, AdviceListener>(); . public static void methodOnBegin( int adviceId, ClassLoader loader, String className, String methodName, String methodDesc, Object target, Object[] args) { try { ... final AdviceListener listener = getListener(adviceId); . Before (Listener, loader, className, methodName, methodDesc, target, args); . } finally { ... } } @Override public MethodVisitor visitMethod(...) { final MethodVisitor mv = super.visitMethod(access, name, desc, signature, exceptions); . return new AdviceAdapter(ASM5, new JSRInlinerAdapter(mv, access, name, desc, signature, exceptions), access, name, desc) { private final Type ASM_TYPE_SPY = Type.getType("Ljava/arthas/Spy;" ); . @Override protected void onMethodEnter() { codeLockForTracing.lock(new CodeLock.Block() { @Override public void code() {  final StringBuilder append = new StringBuilder(); _debug(append, "debug:onMethodEnter()"); // Load the before method loadAdviceMethod(KEY_ARTHAS_ADVICE_BEFORE_METHOD); _debug(append, "debug:onMethodEnter() > loadAdviceMethod()"); // Push the first argument to method.invoke () pushNull(); // Method argument loadArrayForBefore(); _debug(append, "debug:onMethodEnter() > loadAdviceMethod > loadArrayForBefore()"); // Call method invokeVirtual(ASM_TYPE_METHOD, ASM_METHOD_METHOD_INVOKE); pop(); _debug(append, "debug:onMethodEnter() > loadAdviceMethod() > loadArrayForBefore() > invokeVirtual()"); }}); mark(beginLabel); }}Copy the code

4. Common service scenarios

4.1 Viewing the Implementation

sc *Wrapper* ... jad org.apache.dubbo.qos.protocol.QosProtocolWrapper jad org.apache.dubbo.registry.ListenerRegistryWrapper jad org.apache.dubbo.registry.RegistryFactoryWrapper sc *Adaptive* ... Sc - d org. Apache. Dubbo.. RPC protocol. The InvokerWrapper view class complete informationCopy the code

4.2 Call recording and replay

tt -t com.demo.provider.UserServiceImpl saveUser ... INDEX TIMESTAMP COST(ms) IS-RET IS-EXP OBJECT CLASS METHOD ------------------------------------------------------------------------------------------------------------------------ -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- - 1000 2021-03-28 16:33:58 0.183831 false true 0 x1b936e8 UserServiceImpl saveUser Tt -- play-i 1000... RE-INDEX 1000 GMT-REPLAY 2021-03-28 16:34:57 OBJECT 0x1b936e8 CLASS com.seewo.demo.provider.UserServiceImpl METHOD saveUser PARAMETERS[0] @String[test] PARAMETERS[1] @Integer[18] IS-RETURN false IS-EXCEPTION true THROW-EXCEPTION java.lang.RuntimeException: testCopy the code

4.3 Dynamic Log Level Adjustment

ognl '@org.apache.dubbo.rpc.protocol.dubbo.filter.TraceFilter@logger.logger' @Log4jLogger[ FQCN=@String[org.apache.dubbo.common.logger.support.FailsafeLogger], logger=@Logger[org.apache.log4j.Logger@7b89497f], ] sc -d org.apache.log4j.Logger ... class-info org.apache.log4j.Logger code-source / Users/Martin/m2 / repository/org/slf4j/log4j - over - slf4j / 1.7.25 / log4j - over - slf4j - 1.7.25. Jar name org. Apache.. Log4j Logger isInterface false isAnnotation false isEnum false ognl '@org.slf4j.LoggerFactory@getLogger("root").getLevel().toString()' logger --name [ROOT] --level [error] -c [this hashCode]Copy the code

4.4 Obtaining Spring Context Information

4.4.1 based on for SpringMVC

tt -t org.springframework.web.servlet.mvc.method.annotation .RequestMappingHandlerAdapter invokeHandlerMethod -n 3 // You must request it or you will not be recorded... INDEX TIMESTAMP COST(ms) IS-RET IS-EXP OBJECT CLASS METHOD ------------------------------------------------------------------------------------------------------------------------ -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 1000 2021-04-10 08:15:20 911.495905 true or false 0 x5f50206c RequestMappingHandlerAdapter invokeHandlerMethod / / 1000 here is the result of the above indexes tt - 1000 - w I 'target. GetApplicationContext ()' method is called after / / access to specific objects, Then add into the participation in the tt - 1000 - w I refs' target. GetApplicationContext () getBeanFactory () .singletonObjects.get("agoalEpassDataPanelFacadeImpl") .getEmployeeInfoByWorkNos({"xxxxxxx"})' '{params, returnObj}' -x 3 ... @AnnotationConfigServletWebServerApplicationContext[ reader=@AnnotatedBeanDefinitionReader[org.springframework.context.annotation.AnnotatedBeanDefinitionReader@522b017d], scanner=@ClassPathBeanDefinitionScanner[org.springframework.context.annotation.ClassPathBeanDefinitionScanner@2f18e9bb],  annotatedClasses=@LinkedHashSet[isEmpty=true;size=0], basePackages=null, logger=@Slf4jLocationAwareLog[org.apache.commons.logging.LogAdapter$Slf4jLocationAwareLog@7dea2bff], DISPATCHER_SERVCopy the code

4.4.2 Dubbo based

sc -d 'org.apache.dubbo.config.spring.extension.SpringExtensionFactory'


ognl -c 5197848c '#context=@com.alibaba.dubbo.config.spring.extension.SpringExtensionFactory@contexts.iterator.next'ognl -c 5197848c '#context=@com.alibaba.dubbo.config.spring.extension.SpringExtensionFactory@contexts.iterator.next, #context.getBean("tianqiRedisFactory").jedisPools' -x 3
Copy the code

4.5 Viewing Thrown Exceptions

watch com.seewo.demo.provider.UserServiceImpl * -e -x 2 '{params,throwExp}' watch com.seewo.demo.provider.UserServiceImpl * "{params,returnObj}" -x 2 ... method=com.seewo.demo.provider.UserServiceImpl.saveUser location=AtExceptionExit ts=2021-03-28 16:17:50; [cost] = 0.599027 ms result = @ ArrayList [@ Object [] [@ String [test], @ Integer, [18]], Java. Lang. RuntimeException: test watch com.seewo.demo.provider.UserServiceImpl -x 2 '{params,returnObj}'Copy the code

4.6 Viewing the Call Link

trace com.seewo.demo.provider.UserServiceImpl run "#cost > 10"

...

trace -E com.seewo.demo.provider.UserServiceImpl method1|method2|method3

.....
Copy the code

4.7 Debugging code online

redefine /**/UserServiceImpl.class

...
redefine success, size: 1, classes:
com.demo.provider.UserServiceImpl
Copy the code

5. To summarize

This is a comprehensive description of Arthas, a frequently used troubleshooting tool, including implementation technology principles and common scenarios. Through understanding Java Agent, ASM and other technical principles, we have basically understood how Arthas applications work. The open API for the JVM is written with logic enhanced with ASM bytecode, which is great to learn!

reference

Blog.csdn.net/wangzhongsh… A simple Java Ageng implementation

Blog.csdn.net/can007/arti… Compilation and execution of Java code

www.jianshu.com/p/63c328ca2… Java agent implementation

www.jianshu.com/p/4e34d0ab4… Arthas implementation principles

Developer.aliyun.com/article/675… When Dubbo met Arthas- the practice of troubleshooting problems