6MMI applications

Modern application research is focused on natural multimodal interface and its integration into multimedia system used on daily base. Usage of multimodal interface is not limited only to the TV system but has many different applications. The vision of the future applications and system is not only to control the TV and access multimedia content, but to build system that is aware of its user’s habits, preferences and interests. Such system can become an intelligent concierge of the household, and can provide advanced interconnections between various services (shopping while watching, voting or digital doorman).

In order to make the best use of multimodal interface, it’s important to know which modality should be used to perform every action. Some actions will always be better executed by using a different modality. I.e. entering text would be difficult, time consuming and by all means uncomfortable using gestures, but can be easily and faster performed with voice recognition. With this in mind, it becomes necessary to introduce an integration platform that will provide multimodal applications with requested inputs. The application does not need to know the source modality, if not required explicitly.

Simple example of possible scenario: The MMI application for TV control shows a list of users identified by speech or face recognition modules. Only users in this list are permitted to control the TV using predefined set of gestures, voice commands or other modality. When the user leaves the room, he is automatically removed from the list. The other part of application shows recommended channels. Depending on user viewing preferences, system shows recommendations that best suit all users in front of the TV. Using swipe gestures a user is able to browse in this list, play or stop the video. Some of the recommended channels are locked. It means, that users without permission are not allowed to watch such content until they enter the secret pattern. For entering the secret pattern we apply dynamic gestures.