Technologies used for system control may differ based on the area where are used. For different system control (gesture navigation, voice navigation, eye tracking, etc.) also different hardware is used.
HW used in gesture recognition process:
- Gesture Control devices – wired gloves were used in the past to capture hand gestures and motions. The gloves use tactile switches, optical or resistance sensors to measure the bending of joints.
- Vision Based Gesture Recognition - uses a generic camera and/or range camera to capture and derive the hand gesture. There are multiple methods for camera-based gesture recognition.
- 3D cameras - can perceive depth. 3D cameras have become much more broadly available and cheaper in recent years [3].
Touchscreens
Generally, we can distinguish two type of touch screens: resistive and capacitive.
A resistive touchscreen consists of several layers, out of which the flexible plastic and glass layers are two important electrically resistive layers.
Both the layers face each other and between them is a thin gap. When a finger or stylus tip presses down on the outer surface, both the films meet. It is the measure of the resistance of both the layers on the place of contact and get an accurate measurement of the touch position.
Advantages of Resistive Touchscreen:
- High resistance to dust and water
- Best used with a finger, gloved hand or stylus
- Best suited for handwriting recognition
Disadvantages of Resistive Touchscreen:
- Not too sensitive, you have to press down harder
- Poor contrast because of having additional reflections from extra layer of material placed over the screen
- Does not support multi-touch
A capacitive touchscreen also consists of two spaced layers of glass, which are coated with conductor such as Indium Tin Oxide (ITO).
Human body is an electrical charge conductor. When a finger touches the glass of the capacitive surface, it changes the local electrostatic field. The system continuously monitors the movement of each tiny capacitor to find out the exact area where the finger had touched the screen.
Advantages of Capacitive Touchscreen:
- Because capacitive touchscreen has glass layer instead of plastic, it looks brighter and sharper
- Highly touch sensitive and doesn’t need a stylus
- Supports multi-touch
Disadvantages of Capacitive Touchscreen:
- Because the technology is dependent on the conductive nature of human body, it doesn’t work if the user is wearing gloves
- Because of having a complex structure, these are quite expensive
- Glass is more prone to breaking
Microphones
A microphone converts acoustic waves into electric signal. A diaphragm reacts to acoustic waves with vibration, which produce electric charges of corresponding intensity. There are several techniques to do so, for example condenser, dynamic, piezoelectric or even laser. Mobile phones usually use electret or MEMS (MicroElectrical-Mechanical System) microphones.
Condenser Microphone
Two plates are powered to create a condenser. One of the plates acts as a membrane and moves based on incoming acoustic waves. The movement changes output voltage, generating the signal.
Usually a studio microphone, it is more sensitive than dynamic microphone. It is used to record musical instruments.
The microphone requires power source.
An electret microphone is a technological update of condenser microphone, making it more resilient. Electret microphone is used in majority of mobile devices nowadays.
Dynamic Microphone
Microphone membrane connects to a coil located around a permanent magnet. Pressure applied to the membrane forces the coil to move along the magnet and generate electric current.
Dynamic microphone is less sensitive, making it more suitable for recording live stage singing.
Other technologies for sound acquisition include carbon, piezoelectric, ribbon, MEMS, liquid, or laser.
Based on the shape of basic components, a microphone can have different sensitivity to different angles of sound source. The most common is cardioid pattern, which makes microphone pick up sound waves in front of it but does not pick up sound from behind of it. Other sensitivity patterns include omnidirectional, bi-directional and directional.
A combination of several microphones (for example, ordered in a row) creates a microphone array. The array can better focus on selected area while ignoring the unnecessary area. An important feature of microphone arrays is the ability to deduce direction from which comes the speaker’s voice. This helps targeting other systems of the multimedia system.
Eye tracking and BCI
Eye tracking is the process of measuring the gaze point position (where the tracked person is looking) or the motion of eyes relative to the head.
The systems for eye tracking are mostly based on cameras that capture the eye or eyes images and based on this evaluate the gaze position. The cameras capture the eye image typically with framerate starting with 30Hz (entry level, gaming) up to 1200Hz (research grade). There are basically two constructions of eye trackers – mobile variant, where the cameras that capture the eyes position are mounted on glasses or built into HMD, or fixed variant, where the cameras are placed in box below screen/monitor. The mobile binocular versions (each eye is tracked by dedicated camera) of the eyetracker are generally more precise than monocular (only one eye is tracked) and allow bigger eye movements. Besides eye cameras, in the mobile variant is used also “world” camera, that captures the environment and allows mapping the gaze position on the surrounding image.
There are basically two constructions of eye trackers – mobile variant, where the cameras are mounted on glasses or built into HMD, or fixed variant, where the cameras are placed in box below screen/monitor. Besides above mentioned “eye” cameras, in the mobile variant is used also “world” camera, that captures the environment and allows mapping the gaze position on the surrounding image.
A Brain Computer Interface (BCI) is a technology that allows communication between a human brain and an external system (typically computer based). BCI can refer to a technology that reads signals from the brain to external system and/or a technology that sends signals to the brain.
For device control is primary interest to read the brain signals and interpret the user intention. The sending signals to brain can be used as feedback channel. BCI technology for sending the signals to the brain can use e.g. transcranial magnetic stimulation (TMS) [7]. TMS is a non-invasive approach in which a changing magnetic field is used to cause electric current in the target brain region via electromagnetic induction. BCI technology for reading the brain signals mostly uses electroencephalography (EEG) – electrical signals created by neurons and captured on the skin over the skull using electrodes, which are typically gold plated or wet. Typically, BCI systems use 2 electrodes (entry level, gaming) – 4 and above, up to 256 (research grade). The important part of captured signals (brain waves) lie in frequency in the 2Hz-30Hz band, are very weak (2-30mV) and need to be amplified. This frequency range is split among more sub-bands (called also brain waves), as beta, theta, etc ... Presence of energy in these sub-bands can indicate various situations. Depends also on measurement location. For example Delta waves are up 4 Hz, located in frontal part and for adults it is present in more sleep stages. Alpha waves are form 7 Hz to 13 Hz located on posterior regions of head, both sides, they emerge with eye closing and with relaxation, and attenuates with eye opening or mental exertion. Mu waves are from 8Hz to 13 Hz are located on sensorimotoric cortex (central upper part on scalp on both sides) and are present during motoric actions or even imagination of motoric actions. One problem is, that also myo-signals (signals generated because of muscle movements) are captured and these are in one magnitude order stronger (10-300mV). So, the careful postprocessing is necessary. There are also systems that are based on myo-signals, e.g. captured e.g. on wrist or forearm to capture the gestures. Mostly the myo-signal based systems actually focus to be used in the neuroprosthetic solutions. One special case is oculography (capturing the signals from eye movement muscles). These systems are now widely replaced by the above-mentioned camera-based eye tracking systems.