Component Type: Sensor (Subcategory: Computer Vision)
OS: Windows (x86, x64), Linux (x86, x64, RPi3), Mac OSX
This component provides a cross-platform face tracking computer vision algorithm which puts out estimated movement of a user’s nose and chin in x and y coordinates. The underlying mechanism builds upon the JavaCV library for frame grabbing and computer vision processing. In particular a trained cascade of haar-like features is used in combination with a Lukas Kanade optical flow algorithm to track a face and its movement. The x- and y- coordinates can be used in camera-mouse configurations or to enable selection or control tasks. The values are only provided if a face can be tracked by the algorithm. Note that the x- and y- values represent relative movement in pixels and have to be accumulated (e.g. via the integrate component) to generate e.g. absolute mouse positions.
A camera has to be available (this can be any webcam or a camera which is available as image acquisition device via the operating system). The picture below shows the Logitech Webcam 9000 Pro. Also the camera device of a Kinect sensor or PS3Eye should work. Furthermore, a video stream of an IP camera can be used for tracking.
Logitech Webcam 9000 Pro
Note for Raspberry Pi
The plugin can be used on a Raspberry Pi 3 (tested with Raspbian Jessie) with USB cameras, the Kinect camera or the Raspberry Pi camera module.
In order for the raspi cam to work, the following steps are necessary:
Enable the camera with
Add the v4l driver for the camera, to make it accessible as /dev/videoXX device:
sudo modprobe bcm2835-v4l2 max_video_width=2592 max_video_height=1944
Output Port Description
- noseX [integer]: This value specifies the relative change in the x coordinate of the user’s nose with respect to the previous image frame.
- noseY [integer]: This value specifies the relative change in the y coordinate of the user’s nose with respect to the previous image frame.
- chinX [integer]: This value specifies the relative change in the x coordinate of the user’s chin with respect to the previous image frame.
- chinY [integer]: This value specifies the relative change in the y coordinate of the user’s chin with respect to the previous image frame.
Event Listener Description
- init: if this event is triggered, the face recognition procedure is initiated. This can be useful if the correct face position has been lost due to drifting of the LK algorithm.
- showCameraSettings an incoming event displays the settings window for the camera device (only on Windows), where parameters like image brightness or contrast can be adjusted.
- **frameGrabber[string, combobox selection]*Name of FrameGrabber to use (Default, VideoInput, OpenCV, FFmpeg, OpenKinect, PS3Eye, IPCamera).
- **frameGrabberFormat[string]*Format for grabber, e.g. FFmpeg: ‘dshow’ (default), ‘vfwcap’, ‘gdigrab’.
- **deviceList [string, combobox selection]*List of available devices, if supported by grabber
- cameraSelection [string]: The camera device to be used - use camdIdx e.g. ‘0’ (VideoInput, OpenCV, OpenKinect, PS3Eye), or camName e.g. ‘video=Integrated Camera’ or ‘desktop’ (FFmpeg), stream-url for IPCamera.
- cameraResolution [string, combobox selection]: This selection box provides several standard camera resolutions. Changing the resolution affects accuracy and performance (CPU load of the runtime system). Provided selections include �160x120�, �320x240�, �640x480�, �800x600�, �1024x768� and �1600x1200�.
- **frameRate [integer]*The frame rate to use for frame grabbing. In case of 0 or a negative value, the maximum frame rate will be used.
- **titleVideoFrameWindow [string]*The title of the window showing the video frame.
- **displayGUI [boolean]*if selected, the GUI window will be shown.
- **enableOverlaySettings [boolean]*if selected, the video frame shows the current frame rate and device name as overlayed text.
Selects the best frame grabber for the platform the ARE is running on. (Linux: FFmpeg, Mac OSX: OpenCV, Windows: VideoInput)
VideoInput (Windows only)
FFmpeg (Linux only)
- frameGrabberFormat (optional): e.g. vfwcap or dshow
- cameraSelection (device name or device number): e.g. /dev/video0 or 0 (will be mapped to /dev/video0)
MJPEG stream of SmartPhone camera
Install the following Android app on your smartphone and start streaming the front camera: Camera Stream - WiFi IP Webcam (Web Host LLC)
Enter the http url displayed at your smartphone and add /video to the url
Set the cameraSelection property of the XFacetrackerLK plugin to this url.