Cardboard and gaze
Cardboard is one of the earliest examples of mobile VR. In a narrow sense, it refers to a Google box with a double convex lens, and in a broader sense, it refers to a smartphone + box VR experience platform.
Cardboard
Its interaction mode is relatively simple. It uses the gyroscope of mobile phones to trigger events in the scene through gaze behavior, such as pop-up price information of a product when the user looks at it in the virtual store.
Gaze interaction example diagram
Watching events is the most basic way of interaction, WebVR users to change the line of sight toward the head movement, when a user view facing objects, binding event trigger object, concrete can be divided into three basic events, gazeEnter, gazeTrigger, gazeLeave. We can set a collimate in the center of the camera to describe these three basic events (specifically, two in VR mode, one in the center of the left and right cameras).
- GazeEnter: Triggered once when centering enters an object, i.e. the user looks at the object
- GazeLeave: Fires once when centering leaves the object, that is, when the user stops looking at the object
- GazeTrigger: The gazeTrigger is triggered when the collimation is on an object. Unlike the gazeEnter, the gazeTrigger is triggered every frame until the collimation is away from the object
Principle of gaze events
The trigger condition of the gaze event is that the object is “hit” by the user’s line of sight. In every frame of animation rendering, rays are emitted from the collimation along the negative direction of z-axis. If the rays intersect with the object, that is, the object is hit by rays, it means that the object in front is watched by the user. Here, raycaster object provided by Three is used to pick up 3d objects in the scene.
Here is a simple example of picking up objects using THREE.Raycaster:
// Create an instance rayCaster
const raycaster = new THREE.Raycaster(a);
raycaster.setFromCamera(origin.camera); // Sets the ray source point
raycaster.intersectObjects(targetList); // Check whether the targetList object intersects the ray
if (intersects.length > 0) {
// Gets the first object that intersects the ray triggered from the source point
const target = intersects[0].object;
// TODO
}
Copy the codeIt is mainly divided into three steps:
- New three.raycaster () creates a Raycaster;
- SetFromCamera (Origin,camera) is called to set the location of the ray emission source. The first parameter Origin is passed into the NDC normalized device coordinates, that is, the normalized screen coordinates, and the second parameter is passed into the camera.
- Call. IntersectObjects (targetList) to detect whether objects in the targetList intersect. Raycaster uses the raycasting method to pick up objects
GazeEnter, gazeLeave, gazeTrigger implementation
According to the above description of the basic event gaze, now began to create a look at the listener Gazer, provide event binding on and tie off, update the update utility method, the object can be registered gazeEnter, gazeLeave, gazeTrigger event callback, the following is a complete code.
// Watch the event listener
class Gazer {
constructor(a) {
// Initializes the ray emission source
this.raycaster = new THREE.Raycaster(a);
this._center = new THREE.Vector2(a);
this.rayList = {},this.targetList = [];
this._lastTarget = null;
}
/** Public method of object binding gaze event
* @param {three. Object3D} target listens to the 3D grid
* @param {String} eventType specifies the eventType
* @param {Function} callback event callback
* * /
on(target. eventType. callback) {
const noop = (a) = > {};
// target binds the event for the first time, creates a listener, adds it to the RayList listener list, and initials the callbacks of the three base events as null methods
if (!this.rayList[target.id]) this.rayList[target.id] = { target. gazeEnter: noop. gazeTrigger: noop. gazeLeave: noop };
// Update the event callback based on the eventType and callback passed in
this.rayList[target.id] [eventType] = callback;
this.targetList = Object.keys(this.rayList).map(key = > this.rayList[key].target);
}
off(target) {
delete this.rayList[target.id];
this.targetList = Object.keys(this.rayList).map(key = > this.rayList[key].target);
}
update(camera) {
if (this.targetList.length < = 0) return;
// Update ray position
this.raycaster.setFromCamera(this._center.camera);
const intersects = this.raycaster.intersectObjects(this.targetList);
if (intersects.length > 0) { // The current frame ray hits the object
const currentTarget = intersects[0].object;
if (this._lastTarget) { // The last ray hit the object
if (this._lastTarget.id ! = = currentTarget.id) { // The last ray hit a different object than the current frame
this.rayList[this._lastTarget.id].gazeLeave(a);
this.rayList[currentTarget.id].gazeEnter(a);
}
} else { // The last ray failed to hit the object
this.rayList[currentTarget.id].gazeEnter(a); // Trigger the gazeEnter event for the current frame object
}
this.rayList[currentTarget.id].gazeTrigger(a); // The current frame ray hits the object, triggering the object's gazeTrigger event
this._lastTarget = currentTarget;
} else { // I hit the object in the current frame
if ( this._lastTarget ) this.rayList[this._lastTarget.id].gazeLeave(a); // Trigger the last frame gazeLeave
this._lastTarget = null;
}
}
}
Copy the codeLet’s take a look at the three steps Gazer implements, where “hit” means the ray intersects the object.
The first step is to initialize with the constructor function:
- Initialize the rayCaster instance;
- Create rayList to record object objects registered with Gaze events;
- Create lastTarget to record the object hit by the ray in the previous frame, starting with null.
Second, create an ON method to provide an event binding API
By calling the gazer. On (target, eventType, callback), to bind the event Obect3D object target, binding event type eventType and event callback callback three parameters.
- Check whether the target exists. If it does not, create a listener. If it does, update the event function in the object. This object contains the target itself passed in, along with the three basic event callback functions (methods with an initial value of null) :
this.rayList[target.id] = {
target.
gazeEnter.
gazeTrigger.
gazeLeave
}
Copy the codeAssign this object as a key-value pair to the RayList [target.id] listener sequence object;
- Raylist object processing into [target1,…, targetN] assigned to the form of enclosing targetList, as raycaster. IntersectObjects into the refs.
Third, create an Update method that listens for three basic events in the animation frame
- Call RayCaster. SetFromCamera to update the ray origin and direction.
- Call raycaster. Detection of intersectObjects listening sequence enclosing targetList whether there are objects and ray intersection;
- Based on the gazeEnter and gazeLeave and gazeTrigger implementations, the following logical diagrams are summarized for the triggering of these three events.
Gaze basic event logic diagram
The three conditions in the logical diagram are expressed in code as follows:
If (Intersects. Length > 0)
If (this._lastTarget)
If (this._lasttarget.id! == currentTarget.id)
if (intersects.length > 0) { // The current frame ray hits the object
const currentTarget = intersects[0].object;
if (this._lastTarget) { // The last ray hit the object
if (this._lastTarget.id ! = = currentTarget.id) {
// The gazeLeave event of the previous frame is triggered, and the gazeEnter event of the current frame is triggered
this.rayList[this._lastTarget.id].gazeLeave(a);
this.rayList[currentTarget.id].gazeEnter(a);
}
} else { // The last ray failed to hit the object
this.rayList[currentTarget.id].gazeEnter(a); // The gazeEnter event for the current frame is triggered when the last ray did not hit the object
}
this.rayList[currentTarget.id].gazeTrigger(a); // The current frame ray hits the object, triggering the object's gazeTrigger event
this._lastTarget = currentTarget;
} else { // I hit the object in the current frame
if ( this._lastTarget ) this.rayList[this._lastTarget.id].gazeLeave(a); // The previous frame gazeLeave is triggered when the ray hits the object in the previous frame
this._lastTarget = null;
}
Copy the codeFinally, we need to update this._lastTarget for the next frame, this._lastTarget = currentTarget if the current frame is hit, otherwise this._lastTarget = null.
Event binding example
Next, we call the Gazer class we defined earlier to develop the Gaze interaction to implement a simple example: randomly create 100 cubes that are translucent when the user looks at them. Start by creating a collimate. Set it to a dot as the cursor to be displayed to the user. Of course you can create other collimating shapes such as a cross or a circle.
// Create collimation
createCrosshair (a) {
const geometry = new THREE.CircleGeometry( 0.002. 16 );
const material = new THREE.MeshBasicMaterial({
color: 0xffffff.
opacity: 0.5.
transparent: true
});
const crosshair = new THREE.Mesh(geometry.material);
crosshair.position.z = -0.5;
return crosshair;
}
Copy the codeNext, create objects and bind events in the start() method and listen for events in the update.
// The scene object is initialized
start() {
const { scene. camera } = this;
. Create lights, floors, etc
// Add collimation to the camera
camera.add(this.createCrosshair());
this.gazer = new Gazer(a);
// Create a cube
for (let i = 0; i < 100; i++) {
const cube = this.createCube(2.2.2 );
cube.position.set( 100*Math.random(a) - 50. 50*Math.random(a) -10. 100*Math.random(a) - 50 );
scene.add(cube);
// Bind the gaze event
this.gazer.on(cube.'gazeEnter', () = > {
cube.material.opacity = 0.5;
});
this.gazer.on(cube.'gazeLeave', () = > {
cube.material.opacity = 1;
});
}
}
// Animation update
update() {
const { scene. camera. renderer. gazer } = this;
gazer.update(camera);
renderer.render(scene. camera);
}
Copy the codeIn this example, we follow the code structure of the previous WebVRApp, and add a center in the start method to bind the gazeEnter event and gazeLeave event to 100 cubes. When the gazeEnter event is triggered, the cube is translucent, and when the gazeLeave event is triggered, The cube restores opacity.
Gaze implements the demo
Demo Address:
Yonechen. Making. IO/WebVR – hello…
Source code address:
Github.com/YoneChen/We…
In addition to the above three basic events, gaze events are also derived from gaze delay events and gaze click events. These gaze events can be expanded in gazeTrigger.
Gaze click event
The Second generation of Cardboard provides a button on the box that is triggered when the user clicks on the screen by looking at an object and clicking on it. The gazeTrigger method determines whether to execute the callback based on the flag bit. The key code is as follows:
// Button event listener
window.addEventListener('click'. e = > this.state._clicked = true);
this.gazer.on(cube.'gazeTrigger', () = > {
// Triggered when the user clicks
if (this.state._clicked) {
this.state._clicked = false; // Reset the click flag bit
cube.scale.set(1.5.1.5.1.5); // TODO
}
});
Copy the codeFixation delay event
When centering on an object is triggered for more than a certain amount of time, a progress bar animation is usually set at the centering.
Fixation delay event
In gazeEnter, record the start time. In gazeTrigger, calculate whether the time difference exceeds the preset delay time. If so, execute the callback.
// Collimate into the object, turn on the event trigger timing
this.gazer.on(cube.'gazeEnter', () = > {
this.state._wait = true; // The timer has started
this.animate.loader.start(a); // Start the centering progress bar animation
this.state.gazeEnterTime = Date.now(a); // Record the start time of the timer
});
this.gazer.on(cube.'gazeTrigger', () = > {
// Trigger when the timer has started and the delay is more than 1.5 seconds
if (this.state._wait && Date.now(a) - this.state.gazeEnterTime > 1500) {
this.animate.loader.stop(a); // Stop the centering progress bar animation
this.state._wait = false; // The timer ends
cube.material.opacity = 0.5; // TODO
}
});
this.gazer.on(cube.'gazeLeave', () = > {
this.animate.loader.stop(a); // Stop the centering progress bar animation
this.state._wait = false; // The timer ends
.
});
Copy the codeHere, the progress bar loader animation using tween. js, here will not expand, more can be viewed in the source address.
Demo Address:
Yonechen. Making. IO/WebVR – hello…
Source code address:
Github.com/YoneChen/We…
In conclusion, this article introduces the concept and principles of Gaze events for Cardboard and the development process of three basic events. Examples are provided to illustrate how to implement gaze interactions. The above mentioned look and click is also the most common way for Gear VR to interact. However, Gear VR offers a richer Touchpad instead of buttons. The next issue will cover the event development of Gear VR and TouchPad in detail, so stay tuned.