The introduction

Since Apple released ARKit, I’ve wanted to learn and do something fun, and I signed up for the 8th event of Code Salon in Shanghai to talk about ARGitHubCommits. Read a lot of ARKit articles by Zhang Jiafu, and also read some ARKit open source projects. At that time, there was a dynamic picture of ARKit constantly popping up Windows alarm dialog box on Weibo that was very popular. When I saw it, I thought, this pile of pop-up dialog box looks like a path, which is also the original inspiration of Find Me App.

explore

Later, I wanted to make use of this feature to explore some aspects of pathfinding. At first, I wanted to find things at home. My imagination is as follows:

  • Find a fixed object (like a TELEVISION) in your home as a reference point.
  • Record a path for each item to this reference point.
  • Find this reference point when you want to find an item, and then follow the recorded path to find the item.

But there were two particularly big pain points:

  • Things often change their positions, and their paths change with them.
  • Things that can’t be found are often misplaced, so of course there is no such thing as a path record.

So the vision was approved and the project was shelved for a while.

Then one day, I went to the mall with a friend. He got there first, but I couldn’t find the shop. After asking for a long time, I found it in a very hidden corner. After the appointment, I was inspired to go home. This scene can be done with the idea of looking for something before:

  • Those who arrive first find an easy spot, such as the north entrance.
  • Record a route to the store from this location and share it with subsequent friends.
  • Later friends find the starting point of this path, which is the north gate of the shopping mall mentioned before. Open the path shared by those who arrive first to find the store they have agreed to meet along the path.

Key issue: No pain points in terms of usage scenarios.

This is what I’m going to focus on today with Find Me.

Now this App has been released on the shelf: Find Me, and the code is open source, the address is: mmoaay/Findme (please click Star if you like), why open source? Because it is just an idea, there are not too many technical barriers, and currently there are two problems in technology:

  • Path recording and search processARKitThe Session of the virtual path cannot be interrupted, because the virtual path will be in an uncontrollable state due to the deviation of the virtual coordinate after recovery.
  • ARKitCurrently unstable, the virtual coordinate system will shake, so that the virtual path will be offset, and the farther the distance, the greater the deviation.

These two problems are mainly caused by virtual world coordinates. As we all know, when ARKit is initialized, it will set up a virtual world coordinate system based on your current position as the origin of the world:

So, the paths that Find Me records and shares have two basic requirements for ARKit’s virtual world coordinates:

  • When recording the path and searching according to the path, the origin of the virtual world must be mapped to the same point in the real world, that is, the starting point of sharing and searching must be the same.
  • The virtual world should have the same horizontal and vertical orientation as the real world.

To optimize the

I also made some optimizations for these two problems.

Two optimizations that failed so far

Location-based optimization

The idea is simple: segment the path according to location and fix the virtual path. This is equivalent to adding a real world coordinate correction to the virtual path.

After practice, I found that positioning is not accurate than ARKit, especially in the shopping mall, positioning based on WiFi can basically make your position jump around. Pass!

Range-based optimization

The idea is to track your path in segments based on how far you move in the virtual world.

Then there were the problems: ARKit initialization was too slow, taking 3 seconds on my iPhone 7… And ARKit Seesion can not be concurrent, because there is only one camera, only the last one ends, the next one can start, finally found that the path can not record…

Successful optimization

ARKitUsing the optimization

Set ARWorldTrackingConfiguration worldAlignment to gravityAndHeading.

First let’s look at the WorldAlignment type:

/**
Enum constants forWhere did I send it? */ DID I send it where DID I send it? */ DID I send it where DID I send it? Int { /** Aligns the world with gravity that is defined by vector (0, -1, 0). */case gravity

        
    /** Aligns the world with gravity that is defined by the vector (0, -1, 0)
     and heading (w.r.t. True North) that is given by the vector (0, 0, -1). */
    caseGravityAndHeading /** Aligns the world with the camera's orientation. */case camera
}
Copy the code
  • .gravity: Only gravity, the vertical direction of the coordinate system is the same as the real world, but the horizontal direction is variable.
  • .gravityAndHeadingGravity and pointing north, vertical and horizontal are the same as in the real world.
  • .camera: The camera direction, that is, the coordinate system should be consistent with the mobile phone.

So that’s why we chose.gravityandheading. In this way, the person looking for someone according to the path only needs to find the real location of the path starting point, the direction of the phone is not important, greatly reducing the threshold to use.

Provides a picture of the path start point

The optimized content is: when sharing the path, provide a photo of the starting point. In this way, the person who gets the path will make a general comparison between the picture and their own scene to determine the location of the person who shared the path at that time, and take a look at the effect:

We directly use ARKit sceneView. This photo session. CurrentFrame attributes, as follows:

if let currentFrame = sceneView.session.currentFrame, let image = UIImage(pixelBuffer: currentFrame.capturedImage, context:CIContext()) {
}
Copy the code

Use advice

With that in mind, if you do want to use Find Me in your daily life, here are some very important tips:

  • If there is a new message in the process of recording or finding the way, do not cut out to read, after all, the shopping mall navigation is only about 10 minutes, 10 minutes late reply to the message is ok.
  • The pathfinder must find the exact starting point of the path! Very important! This directly determines the accuracy of the path end position.

Some other interesting technical points

Arrow implementation in path

Take a look at the results:

This technical point consists of two parts:

  • How do I draw arrows?
  • What if I let the arrow point to the next point?

How do I draw arrows?

First, we generate a UIBezierPath that looks like the following:

The code is as follows:

private static func vertexCoordinates() -> [CGPoint] {
    return [CGPoint(x: 0, y: 0),
            CGPoint(x: 20, y: 0),
            CGPoint(x: 20, y: 10),
            CGPoint(x: 10, y: 10),
            CGPoint(x: 10, y: 20),
            CGPoint(x: 0, y: 20)
    ]
}
    
private static func arrowPath() -> UIBezierPath {
    let path = UIBezierPath()
    let points = NodeUtil.vertexCoordinates()
        
    var count = 0
    for point in points {
        if 0 == count {
            path.move(to: point)
        } else {
            path.addLine(to: point)
        }
        count += 1
    }
        
    path.close()
        
    return path
}
Copy the code

Then generate the SCNNode with the following code:

let path = NodeUtil.arrowPath()
let shape = SCNShape(path: path, extrusionDepth: 2)

let node = SCNNode(geometry: shape)
Copy the code

Thus an arrow node is generated.

What if I let the arrow point to the next point?

I’m just going to rotate this arrow by a certain Angle. Here’s the math: calculate the Angle between two lines and some coordinate axis. I’m not good at math, so I won’t talk about the principle, but I’ll talk about how to rotate a SCNNode in SceneKit:

First we need two points, the current point and the last point, so we record the last point with a last variable of type SCNVector3. SCNVector3 contains x, Y and Z attributes, which correspond to the values of x, Y and Z axes respectively.

RotateBy is used to generate a rotation action based on the current and previous positions, and the SCNNode runAction method is used to perform this action.

The final code is as follows:

node.runAction(SCNAction.rotateBy(x: CGFloat(float.pi /2.0*3.0), y:CGFloat(float.pi /4.0+atan2(current.x-last.x, current.z-last.z)), z: 0.0, duration: 0.0)Copy the code

File sharing a pit

Find me now share the path used way is to file sharing, sharing out USES is UIDocumentInteractionController, accept share the path through the main func application (_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool callback, where it returns the path of the shared file via the URL argument, there is a dot here, if you open the file directly, you will find that the file does not exist…

The solution is interesting: use this path to copy the file to another path, and then open the file.

conclusion

ARKit, as a newly released framework by Apple, will definitely be further optimized in the future, so the accuracy of Find Me path is worth looking forward to in the future. Of course, I will continue to make improvements to Find Me. Welcome to pay attention to it.

Another personal feeling ARKit framework learning threshold is really not high, the main threshold instead in SceneKit or SpriteKit, you can also have a look at some of zhang Jiafu’s tutorials, high quality.