Auxiliary mode is the ultimate test of imagination, let’s see how to use! | Accessibility

A, sequence

Hi, everyone, I am Chengxiang Ink shadow!

Android Accessibility is quite powerful. Basically, once authorized, it can listen for any event on the phone, such as: screen clicks, window changes, as well as simulated clicks, simulated system keystrokes and so on.

A more common practical example of this is the general app market, which recommends enabling accessibility mode to automatically click the “Next” and “Install” buttons for you when you install Apk. Another example is wechat grab red envelope plug-in, is based on it to achieve.

The org.eclipse.swt.accessibility permissions is very high, basically you authorization to open some others provide AccessibilityService, he can do a lot of things and not to let you know, this is not need Root access. Therefore, for small products, it may not be useful to support it, because the trust is so low that most users will not open it at all. More common is some tool class apps, to help users save some click time.

Although most of the time Accessibility is not used for commercial products, this doesn’t prevent us from doing interesting things with Accessibility.

Second, the use of auxiliary mode steps

Accessibility supports third party development, that is, we can support it according to documentation. As long as users authorize access to this service, we can use the standard APIS provided by Accessibility to implement many interesting functions.

If you want to use auxiliary mode, you need to do the following:

1. Implement a service class that inherits from AccessibilityService.
2. Set the configuration information so that the system knows some basic information about the helper mode, such as which events to listen for.
3. In the manifest file (Androidmanifest.xml), register the service.
4. In system Settings, find “Accessibility” and enable this service.

Let’s go through the steps and details step by step.

2.1 inheritance AccessibilityService

Helper mode is essentially a service, and if we want to support it, we first need to inherit AccessibilityService.

The AccessibilityService class provides a number of methods that need to be overridden, two of which are mandatory:

public abstract void onAccessibilityEvent(AccessibilityEvent event);
public abstract void onInterrupt(a);
Copy the code

When an AccessibilityService is enabled, the system calls its onAccessibilityEvent() method whenever an event that the AccessibilityService is listening for occurs, passing the event information as a parameter. If you listen for enough events, it will be called frequently.

The onInterrupt() method, on the other hand, calls back when a system event is interrupted and is called frequently, usually without extra processing.

The onAccessibilityEvent() method is usually used to write the core logic.

2.2 Configuring the Auxiliary Mode

Once we have created an AccessibilityService, we need to do some basic configuration, otherwise we will not be able to write the service in the “accessibility” of the system Settings.

There are two ways to configure AccessibilityService.

• Through XML configuration files
• Dynamically configured through Java code.

But there are some properties that can only be configured through XML configuration files, and the Java code just makes certain configuration items more flexible, as I’ll explain later.

1. XML configuration files

To use an XML configuration file, you first need to create an RES/XML directory and create an XML file in it with an optional file name and an internal swt.swt.service tag. Set the AccessibilityService configuration. For example, I’ll create an accessibility_config.xml file that I’ll use later.

XML configuration AccessibilityService is one of our common configuration methods, very clear and convenient.

<accessibility-service
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:accessibilityEventTypes="typeAllMask"
    android:accessibilityFeedbackType="feedbackAllMask"
    android:accessibilityFlags="flagReportViewIds"
    android:canRetrieveWindowContent="true"
    android:packageNames="com.forwarding.wechat"
    android:description="@string/accessbility_desc"
    android:notificationTimeout="100" />
Copy the code

For example, the above is a common configuration, if there is no special requirements, directly copy the past, modify some individual parameters can be used.

Meanings of attributes:

• AccessibilityEventTypes: Event types to listen for. For example, typeAllMask represents all events while typeViewClicked represents only click events.
• AccessibilityFeedbackType: listen event feedback mode.
• CanRetrieveWindowContent: Whether to allow access to the view level, if it is set to false,node.getSource()Method calls fail.
• AccessibilityFlags: Specifies the Flag, which is used to specify the permission to obtain the View ID based on Node.
• PackageNames: specifies the name of the application package for which listening is enabled. Multiple packageNames can be separated by commas (,). If this attribute is not set, global listening is identified.
• Description: Indicates the description of the auxiliary function. It will be displayed in the description information under Accessibility in system Settings.
• NotificationTimeout: The number of milliseconds for a response.

The system provides optional configuration parameters for these configurable parameters. If you do not need to customize additional parameters, use the default Settings. If you need to customize parameters, refer to the official documents for a complete description.

AccessibilityService:

https://developer.android.com/reference/android/accessibilityservice/AccessibilityService

2. Dynamic configuration in Java code

In addition to the XML file configuration, you can override the AccessibilityService’s onServiceConnected() method. First, you need to build an AccessibilityServiceInfo object. Configure it through its standard Api and use the setServiceInfo() method to set it to auxiliary mode.

OnServiceConnected () is called when your application successfully connects to the helper service, usually by doing some initialization.

override fun onServiceConnected(a) {
	super.onServiceConnected()
    var serviceInfo = AccessibilityServiceInfo()
    serviceInfo.eventTypes = AccessibilityEvent.TYPES_ALL_MASK
    serviceInfo.feedbackType = AccessibilityServiceInfo.FEEDBACK_ALL_MASK
    serviceInfo.notificationTimeout = 100
    serviceInfo.packageNames = arrayOf("com.forwarding.wechat")
    serviceInfo.flags =  AccessibilityServiceInfo.FLAG_REPORT_VIEW_IDS
    setServiceInfo(serviceInfo)
}
Copy the code

The examples provided here are exactly the same as the previous ones using XML configuration. The recommended XML configuration is clearer and more flexible, and attributes like description are not available in AccessibilityServiceInfo as setDescription() is by design. After all, if the service is not running, the description information does not exist and cannot be read on the “accessibility” page set by the system.

That said, even if you use the setServiceInfo() method to set it up dynamically, you can’t get away with using an XML configuration file. I strongly recommend using an XML configuration file for all ancillary services, mainly for ease of use.

2.3 Register the service in the manifest file

AccessibilityService is essentially a Service, and to use it we need to configure it in the manifest file.

<service android:label="Auxiliary tools for Chengxiang ink shadow"
    android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"
    android:name=".WeForwardServer">
    <intent-filter>
        <action android:name="android.accessibilityservice.AccessibilityService"/>
    </intent-filter>
    <meta-data
        android:name="android.accessibilityservice"
        android:resource="@xml/accessibility_config"/>
</service>
Copy the code

This is a standard Service in which the label is resolved to be “displayed in secondary mode” in the system setting, while the intent-filter and meta-data are just formatted, for no reason.

In meta-data, the path to the configuration file we edited in step 2 is specified by the Android :resource attribute.

2.4 Enabling auxiliary Mode

Once the above steps are complete, you should be able to see the auxiliary mode switch you wrote in the system’s “accessibility” Settings.

By default, it is closed. When you open it, you will receive a warning popup indicating that you are currently opening a barrier-free service and what permissions it has. This dialog box is beyond our control.

Note that the Title is the Android :label configured in the manifest file and the description is the Android: Description information configured in the XML configuration file.

When you see this switch in the system Settings, it means that your service in auxiliary mode is configured correctly, and then you need to think about how to use it.

Write logical code

In AccessibilityService, the onAccessibilityEvent() method will be called back when the event that we are listening to occurs and will pass information about that event.

Let’s take a look at how to write the actual logic inside the onAccessibilityEvent() callback.

Now, programmer’s mind is on line, and it takes a few steps to turn off the elephant. Let’s break down the steps of the helper mode.

1. Judge events,onAccessibilityEvent()It will be called back multiple times, and we only need to deal with the events we care about and ignore the rest and filter out.
2. Find the key nodes to control for later control.
3. For key nodes, send the appropriate action events to complete our step.
4. Reclaim resources to prevent resource leakage.

Pretty simple, right? So let’s go through the methods and properties associated with these steps.

3.1 Determining events

When onAccessibilityEvent() is called back by the system, an AccessibilityEvent object is also passed, which contains a lot of information related to the current event.

1. EventType determines the eventType

Determine the type of event by eventType, which we can retrieve using the getEventType() method.

These events are easy to recognize, for example: TYPE_NOTIFICATION_STATE_CHANGED is when a window View changed, TYPE_VIEW_CLICKED is when a View clicked, and so on.

2. PackageName App that determines the event

Use getPackageName() to determine which App the event occurred in.

3, className Determines which class the current event is

GetClassName () determines which class is currently involved in the event, such as the display of a page. ClassName may point to an Activity, a Button click, or a Button.

4. Text Determines the text on the trigger source of the current event

GetText () gets the text attribute of the current event source, either the text of the TextView or the Label attribute of the Activity, again depending on the actual situation.

Generally we can guess whether the event is the one we want to listen for in the above ways. The next step is to find the source we want to operate on.

3.2 Finding key Nodes (Nodes) to Be Controlled

Usually we use helper mode to manipulate an element on the page, so this step is to find it.

In the auxiliary mode, each element on the page is actually an AccessibilityNodeInfo node, which is a similar tree structure. Its interior is consistent with the layout level in our real App, but it cannot be simply understood as a ViewTree.

NodeInfo = NodeInfo = NodeInfo = NodeInfo = NodeInfo = NodeInfo = NodeInfo = NodeInfo = NodeInfo

• event.getSource()
• getRootInActiveWindow()

Both methods return an AccessibilityNodeInfo object. Is AccessibilityEvent getSource () method, it is the premise of the previous configuration android: canRetrieveWindowContent, is set to True. So I recommend using the getRootInActiveWindow() method. The two methods are slightly different and can be interrupted to see information if you are interested, but for the most part, they are consistent for us users.

Once we get the AccessibilityNodeInfo of the root node, we can use it to find the key node we want to operate on. In AccessibilityNodeInfo, we provide the following two methods to find the key node.

• findAccessibilityNodeInfosByViewId(String viewId)
• findAccessibilityNodeInfosByText(String text)

One relies on ViewId and the other relies on Text information.

Using ViewId to find key nodes is a safe bet, whereas using Text to find key nodes may not be found.

Either way, it is possible to find multiple NodeInfo nodes, so both methods simply return a List

, so we need to filter again through other criteria. Usually through Text message filtering.

var mNodeInfo = rootInActiveWindow
var listItem = mNodeInfo.findAccessibilityNodeInfosByViewId("com.tencent.mm:id/lp")
for (item in listItem) {
    if (item.text.toString().equals("Fragrant ink shadow")){
		nodeClick(item)
	}
}
Copy the code

Contain () : findXxxByText() : findXxxByText() : findXxxByText() : findXxxByText() : findXxxByText() : findXxxByText() : findXxxByText(); We also need to add judgment conditions to this.

If you have tried all these methods and still can’t find the key node, you can find it by traversing.

Since AccessibilityNodeInfo is a tree structure, it also provides a way to traverse the tree.

• GetParent () : Finds the parent node.
• GetChild () : Returns the child node.
• GetChildCount () : Number of children of the current node.

With getChild() and getChildCount(), we can traverse the entire ViewNodeTree to find the key nodes we care about. This is a last resort option and is not recommended.

3.3 Triggering Events

Helper modes typically help us respond to events that, broadly speaking, fall into two categories.

• Global system events.
• View events.

For global system events, we don’t really need the key nodes found in step 2. AccessibilityService provides a performGlobalAction() method that allows you to manipulate global system events, such as simulating a return key click, simulating a HOME key click, locking the screen, and so on.

/ / return key
performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK);
/ / HOME button
performGlobalAction(AccessibilityService.GLOBAL_ACTION_HOME);
Copy the code

These events are encapsulated in AccessibilityService, prefixed with GLOBAL_, as shown in the attribute description.

In addition to global system events, we usually want to manipulate the key nodes retrieved in step 2.

In AccessibilityNodeInfo, we provide a performAction() method that passes the desired event to the key node.

These events are defined in AccessibilityNodeInfo, prefixed with ACTION_. For example, ACTION_CLICK is a click event, and ACTION_SET_TEXT sets an input.

Only some common operations are described here, and more operations are used in similar ways.

1. View click

After finding the key nodes, you can send AccessibilityNodeInfo. ACTION_CLICK click operation simulation of this View.

But sometimes it doesn’t work, mainly because you found a key node whose isClickable() is false.

For example, if we want to find “send to friends” in the public account of wechat, the best way is to find the key node (NodeInfo) represented by the TextView control and click on it. The TextView does not actually have a clickable effect; its isClickable() is false.

Find the parent node of the key node (NodeInfo), and then check whether it is clickable. If it is clickable, click it, otherwise continue to look up.

private fun nodeClick(node : AccessibilityNodeInfo?).{
    var clickNode = node;
    while(clickNode! =null) {if(clickNode.isClickable){
            clickNode.performAction(AccessibilityNodeInfo.ACTION_CLICK)
            break; } clickNode = node? .parent } }Copy the code

Although AccessibilityNodeInfo actually has the setClickable() method open, I don’t recommend using it because it sometimes throws an exception and is unstable.

2. EditText Input text

To enter text into EditText, you need at least two parameters, the key node and the typed text. This requires another overloaded method of performAction(), which allows you to pass an additional Bundle to specify parameters.

private fun nodeSetText(node : AccessibilityNodeInfo? ,text:String){
    varargument = Bundle() argument.putString(AccessibilityNodeInfo.ACTION_ARGUMENT_SET_TEXT_CHARSEQUENCE,text) node? .performAction(AccessibilityNodeInfo.ACTION_SET_TEXT,argument) }Copy the code

Any additional arguments that support definition are defined in AccessibilityNodeInfo and are prefixed with ACTION_ARGUMENT_.

3. ListView scrolling

AccessibilityNodeInfo can only operate on nodes visible under the current screen, so use ListView or RecycleView to scroll through it.

There are two kinds of rolling events:

• ACTION_SCROLL_FORWARD
• ACTION_SCROLL_BACKWARD

private fun nodeScrollList(node : AccessibilityNodeInfo?).{ node? .performAction(AccessibilityNodeInfo.ACTION_SCROLL_FORWARD) }Copy the code

One forward and one backward. That’s enough.

3.4 Reclaiming Resources

After using AccessibilityNodeInfo, don’t forget to call the Recycle () method to release resources.

nodeInfo.recycle();
Copy the code

Four, summary

How to use the auxiliary mode, to now has explained very clearly, the back is basically to rely on their own imagination to do small functions.

You can also do a lot of interesting things with the help of your imagination.

---

Public number background reply growth “growth”, will get the learning materials I prepared, can also reply “add group”, learning progress together; You can also reply to “questions” and ask me questions.

Recommended reading:

• Small Program UI Layout Guide (1)
• Password management for programmers
• Manually refresh MediaStore and the saved images immediately appear in the album
• Pseudocode, humor, and the art of Google!
• App prevents Fiddler from grabbing your bag!