|
|
@@ -82,7 +82,7 @@ Python.
|
|
|
should allow for extensions to be added to any implementation.
|
|
|
|
|
|
The reference implementation is a Proof of Concept that translates TUIO
|
|
|
- events to some simple touch gestures that are used by some test
|
|
|
+ messages to some simple touch gestures that are used by some test
|
|
|
applications.
|
|
|
%Being a Proof of Concept, the reference implementation itself does not
|
|
|
%necessarily need to meet all the requirements of the design.
|
|
|
@@ -118,7 +118,6 @@ Python.
|
|
|
|
|
|
\section{Gesture recognition software for Windows 7}
|
|
|
|
|
|
- % TODO
|
|
|
The online article \cite{win7touch} presents a Windows 7 application,
|
|
|
written in Microsofts .NET. The application shows detected gestures in a
|
|
|
canvas. Gesture trackers keep track of stylus locations to detect specific
|
|
|
@@ -160,7 +159,182 @@ Python.
|
|
|
of gesture detection code, thus keeping a code library manageable and
|
|
|
extendable, is to user different gesture trackers.
|
|
|
|
|
|
-\chapter{Requirements}
|
|
|
+% FIXME: change title below
|
|
|
+\chapter{Design - new}
|
|
|
+
|
|
|
+ % Diagrams are defined in a separate file
|
|
|
+ \input{data/diagrams}
|
|
|
+
|
|
|
+ \section{Introduction}
|
|
|
+
|
|
|
+ % TODO: rewrite intro, reference to experiment appendix
|
|
|
+ This chapter describes a design for a generic multi-touch gesture detection
|
|
|
+ architecture. The architecture constists of multiple components, each with
|
|
|
+ a specific set of tasks. Naturally, the design is based on a number of
|
|
|
+ requirements. The first three sections each describe a requirement, and a
|
|
|
+ solution that meets the requirement. The following sections show the
|
|
|
+ cohesion of the different components in the architecture.
|
|
|
+
|
|
|
+ To test multi-touch interaction properly, a multi-touch device is required.
|
|
|
+ The University of Amsterdam (UvA) has provided access to a multi-touch
|
|
|
+ table from PQlabs. The table uses the TUIO protocol \cite{TUIO} to
|
|
|
+ communicate touch events. See appendix \ref{app:tuio} for details regarding
|
|
|
+ the TUIO protocol.
|
|
|
+
|
|
|
+ \subsection*{Position of architecture in software}
|
|
|
+
|
|
|
+ The input of the architecture comes from some multi-touch device
|
|
|
+ driver. For example, the table used in the experiments uses the TUIO
|
|
|
+ protocol. The task of the architecture is to translate this input to
|
|
|
+ multi-touch gestures that are used by an application, as illustrated in
|
|
|
+ figure \ref{fig:basicdiagram}. At the end of this chapter, the diagram
|
|
|
+ is extended with the different components of the architecture.
|
|
|
+
|
|
|
+ \basicdiagram{A diagram showing the position of the architecture
|
|
|
+ relative to a multi-touch application.}
|
|
|
+
|
|
|
+ \section{Supporting multiple drivers}
|
|
|
+
|
|
|
+ The TUIO protocol is an example of a touch driver that can be used by
|
|
|
+ multi-touch devices. Other drivers do exist, which should also be supported
|
|
|
+ by the architecture. Therefore, there must be some translation of
|
|
|
+ driver-specific messages to a common format in the arcitecture. Messages in
|
|
|
+ this common format will be called \emph{events}. Events can be translated
|
|
|
+ to multi-touch \emph{gestures}. The most basic set of events is
|
|
|
+ ${point\_down, point\_move, point\_up}$.
|
|
|
+
|
|
|
+ A more extended set could also contain more complex events. However, a
|
|
|
+ object can also have a rotational property, like the ``fiducials'' type in
|
|
|
+ the TUIO protocol. This results in $\{point\_down, point\_move, point\_up,
|
|
|
+ object\_down, object\_move, object\_up,\\object\_rotate\}$.
|
|
|
+
|
|
|
+ The component that translates driver-specific messages to events, is called
|
|
|
+ the \emph{event driver}. The event driver runs in a loop, receiving and
|
|
|
+ analyzing driver messages. The event driver that is used in an application
|
|
|
+ is dependent of the support of the multi-touch device.
|
|
|
+
|
|
|
+ When a sequence of messages is analyzed as an event, the event driver
|
|
|
+ delegates the event to other components in the architecture for translation
|
|
|
+ to gestures.
|
|
|
+
|
|
|
+ \driverdiagram{Extension of the diagram from figure \ref{fig:basicdiagram},
|
|
|
+ showing the position of the event driver in the architecture.}
|
|
|
+
|
|
|
+ \section{Restricting gestures to a screen area}
|
|
|
+
|
|
|
+ An application programmer should be able to bind a gesture handler to some
|
|
|
+ element on the screen. For example, a button tap\footnote{A ``tap'' gesture
|
|
|
+ is triggered when a touch object releases the screen within a certain time
|
|
|
+ and distance from the point where it initially touched the screen.} should
|
|
|
+ only occur on the button itself, and not in any other area of the screen. A
|
|
|
+ solution to this program is the use of \emph{widgets}. The button from the
|
|
|
+ example can be represented as a rectangular widget with a position and
|
|
|
+ size. The position and size are compared with event coordinates to
|
|
|
+ determine whether an event should occur within the button.
|
|
|
+
|
|
|
+ \subsection*{Widget tree}
|
|
|
+
|
|
|
+ A problem occurs when widgets overlap. If a button in placed over a
|
|
|
+ container and an event occurs occurs inside the button, should the
|
|
|
+ button handle the event first? And, should the container receive the
|
|
|
+ event at all or should it be reserved for the button?.
|
|
|
+
|
|
|
+ The solution to this problem is to save widgets in a tree structure.
|
|
|
+ There is one root widget, whose size is limited by the size of the
|
|
|
+ touch screen. Being the leaf widget, and thus the widget that is
|
|
|
+ actually touched when an object touches the device, the button widget
|
|
|
+ should receive an event before its container does. However, events
|
|
|
+ occur on a screen-wide level and thus at the root level of the widget
|
|
|
+ tree. Therefore, an event is delegated in the tree before any analysis
|
|
|
+ is performed. Delegation stops at the ``lowest'' widget in the three
|
|
|
+ containing the event coordinates. That widget then performs some
|
|
|
+ analysis of the event, after which the event is released back to the
|
|
|
+ parent widget for analysis. This release of an event to a parent widget
|
|
|
+ is called \emph{propagation}. To be able to reserve an event to some
|
|
|
+ widget or analysis, the propagation of an event can be stopped during
|
|
|
+ analysis.
|
|
|
+ % TODO: insprired by JavaScript DOM
|
|
|
+
|
|
|
+ % TODO: add GTK to bibliography
|
|
|
+ Many GUI frameworks, like GTK \cite{GTK}, also use a tree structure to
|
|
|
+ manage their widgets. This makes it easy to connect the architecture to
|
|
|
+ such a framework. For example, the programmer can define a
|
|
|
+ \texttt{GtkTouchWidget} that synchronises the position of a touch
|
|
|
+ widget with that of a GTK widget, using GTK signals.
|
|
|
+
|
|
|
+ \subsection*{Callbacks}
|
|
|
+ \label{sec:callbacks}
|
|
|
+
|
|
|
+ When an event is propagated by a widget, it is first used for event
|
|
|
+ analysis on that widget. The event analysis can then trigger a gesture
|
|
|
+ in the widget, which has to be handled by the application. To handle a
|
|
|
+ gesture, the widget should provide a callback mechanism: the
|
|
|
+ application binds a handler for a specific type of gesture to a widget.
|
|
|
+ When a gesture of that type is triggered after event analysis, the
|
|
|
+ widget triggers the callback.
|
|
|
+
|
|
|
+ \subsection*{Position of widget tree in architecture}
|
|
|
+
|
|
|
+ \widgetdiagram{Extension of the diagram from figure
|
|
|
+ \ref{fig:driverdiagram}, showing the position of widgets in the
|
|
|
+ architecture.}
|
|
|
+
|
|
|
+ \section{Event analysis}
|
|
|
+
|
|
|
+ The events that are delegated to widgets must be analyzed in some way to
|
|
|
+ from gestures. This analysis is specific to the type of gesture being
|
|
|
+ detected. E.g. the detection of a ``tap'' gesture is very different from
|
|
|
+ detection of a ``rotate'' gesture. The \cite[.NET
|
|
|
+ implementation]{win7touch} separates the detection of different gestures
|
|
|
+ into different \emph{gesture trackers}. This keeps the different pieces of
|
|
|
+ detection code managable and extandable. Therefore, the architecture also
|
|
|
+ uses gesture trackers to separate the analysis of events. A single gesture
|
|
|
+ tracker detects a specific set of gesture types, given a sequence of
|
|
|
+ events. An example of a possible gesture tracker implementation is a
|
|
|
+ ``transformation tracker'' that detects rotation, scaling and translation
|
|
|
+ gestures.
|
|
|
+
|
|
|
+ \subsection*{Assignment of a gesture tracker to a widget}
|
|
|
+
|
|
|
+ As explained in section \ref{sec:callbacks}, events are delegated from
|
|
|
+ a widget to some event analysis. The analysis component of a widget
|
|
|
+ consists of a list of gesture trackers, each tracking a specific set of
|
|
|
+ gestures. No two trackers in the list should be tracking the same
|
|
|
+ gesture type.
|
|
|
+
|
|
|
+ When a handler for a gesture is ``bound'' to a widget, the widget
|
|
|
+ asserts that it has a tracker that is tracking this gesture. Thus, the
|
|
|
+ programmer does not create gesture trackers manually. Figure
|
|
|
+ \ref{fig:trackerdiagram} shows the position of gesture trackers in the
|
|
|
+ architecture.
|
|
|
+
|
|
|
+ \trackerdiagram{Extension of the diagram from figure
|
|
|
+ \ref{fig:widgetdiagram}, showing the position of gesture trackers in
|
|
|
+ the architecture.}
|
|
|
+
|
|
|
+ \section{Example usage}
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+
|
|
|
+% FIXME: Delete the 2 following chapters
|
|
|
+
|
|
|
+\chapter{Experiments}
|
|
|
\label{chapter:requirements}
|
|
|
|
|
|
% testimplementatie met taps, rotatie en pinch. Hieruit bleek:
|
|
|
@@ -174,72 +348,14 @@ Python.
|
|
|
% wellicht in een ander programma nodig om maar 1 hand te gebruiken, en
|
|
|
% dus punten dicht bij elkaar te kiezen (oplossing: windows).
|
|
|
|
|
|
- % TODO: Move content into the following sections:
|
|
|
- \section{Introduction}
|
|
|
- \section{Supporting multiple drivers}
|
|
|
- \section{Restricting gestures to a screen area}
|
|
|
- \section{Separating and extending code}
|
|
|
-
|
|
|
-
|
|
|
\section{Introduction}
|
|
|
|
|
|
- % TODO
|
|
|
-
|
|
|
To test multi-touch interaction properly, a multi-touch device is required.
|
|
|
The University of Amsterdam (UvA) has provided access to a multi-touch
|
|
|
table from PQlabs. The table uses the TUIO protocol \cite{TUIO} to
|
|
|
communicate touch events. See appendix \ref{app:tuio} for details regarding
|
|
|
the TUIO protocol.
|
|
|
|
|
|
- \section{Experimenting with TUIO and event bindings}
|
|
|
- \label{sec:experimental-draw}
|
|
|
-
|
|
|
- When designing a software library, its API should be understandable and
|
|
|
- easy to use for programmers. To find out the basic requirements of the API
|
|
|
- to be usable, an experimental program has been written based on the
|
|
|
- Processing code from \cite{processingMT}. The program receives TUIO events
|
|
|
- and translates them to point \emph{down}, \emph{move} and \emph{up} events.
|
|
|
- These events are then interpreted to be (double or single) \emph{tap},
|
|
|
- \emph{rotation} or \emph{pinch} gestures. A simple drawing program then
|
|
|
- draws the current state to the screen using the PyGame library. The output
|
|
|
- of the program can be seen in figure \ref{fig:draw}.
|
|
|
-
|
|
|
- \begin{figure}[H]
|
|
|
- \center
|
|
|
- \label{fig:draw}
|
|
|
- \includegraphics[scale=0.4]{data/experimental_draw.png}
|
|
|
- \caption{Output of the experimental drawing program. It draws the touch
|
|
|
- points and their centroid on the screen (the centroid is used
|
|
|
- as center point for rotation and pinch detection). It also
|
|
|
- draws a green rectangle which responds to rotation and pinch
|
|
|
- events.}
|
|
|
- \end{figure}
|
|
|
-
|
|
|
- One of the first observations is the fact that TUIO's \texttt{SET} messages
|
|
|
- use the TUIO coordinate system, as described in appendix \ref{app:tuio}.
|
|
|
- The test program multiplies these with its own dimensions, thus showing the
|
|
|
- entire screen in its window. Also, the implementation only works using the
|
|
|
- TUIO protocol. Other drivers are not supported.
|
|
|
-
|
|
|
- Though using relatively simple math, the rotation and pinch events work
|
|
|
- surprisingly well. Both rotation and pinch use the centroid of all touch
|
|
|
- points. A \emph{rotation} gesture uses the difference in angle relative to
|
|
|
- the centroid of all touch points, and \emph{pinch} uses the difference in
|
|
|
- distance. Both values are normalized using division by the number of touch
|
|
|
- points. A pinch event contains a scale factor, and therefore uses a
|
|
|
- division of the current by the previous average distance to the centroid.
|
|
|
-
|
|
|
- There is a flaw in this implementation. Since the centroid is calculated
|
|
|
- using all current touch points, there cannot be two or more rotation or
|
|
|
- pinch gestures simultaneously. On a large multi-touch table, it is
|
|
|
- desirable to support interaction with multiple hands, or multiple persons,
|
|
|
- at the same time. This kind of application-specific requirements should be
|
|
|
- defined in the application itself, whereas the experimental implementation
|
|
|
- defines detection algorithms based on its test program.
|
|
|
-
|
|
|
- Also, the different detection algorithms are all implemented in the same
|
|
|
- file, making it complex to read or debug, and difficult to extend.
|
|
|
-
|
|
|
\section{Summary of observations}
|
|
|
\label{sec:observations}
|
|
|
|
|
|
@@ -412,10 +528,6 @@ Python.
|
|
|
|
|
|
\section{Diagrams}
|
|
|
|
|
|
- \input{data/diagrams}
|
|
|
- \simplediagram
|
|
|
- \completediagrams
|
|
|
-
|
|
|
\section{Example usage}
|
|
|
|
|
|
This section describes an example that illustrates the communication
|
|
|
@@ -450,12 +562,6 @@ Python.
|
|
|
start server
|
|
|
\end{verbatim}
|
|
|
|
|
|
-\chapter{Reference implementation}
|
|
|
-
|
|
|
-% TODO
|
|
|
-% alleen window.contains op point down, niet move/up
|
|
|
-% een paar simpele windows en trackers
|
|
|
-
|
|
|
\chapter{Test applications}
|
|
|
|
|
|
% TODO
|
|
|
@@ -534,4 +640,60 @@ client application, as stated by the online specification
|
|
|
values back to the actual screen dimension.
|
|
|
\end{quote}
|
|
|
|
|
|
+\chapter{Experimental program}
|
|
|
+\label{app:experiment}
|
|
|
+
|
|
|
+% TODO: rewrite intro
|
|
|
+When designing a software library, its API should be understandable and easy to
|
|
|
+use for programmers. To find out the basic requirements of the API to be
|
|
|
+usable, an experimental program has been written based on the Processing code
|
|
|
+from \cite{processingMT}. The program receives TUIO events and translates them
|
|
|
+to point \emph{down}, \emph{move} and \emph{up} events. These events are then
|
|
|
+interpreted to be (double or single) \emph{tap}, \emph{rotation} or
|
|
|
+\emph{pinch} gestures. A simple drawing program then draws the current state to
|
|
|
+the screen using the PyGame library. The output of the program can be seen in
|
|
|
+figure \ref{fig:draw}.
|
|
|
+
|
|
|
+\begin{figure}[H]
|
|
|
+ \center
|
|
|
+ \label{fig:draw}
|
|
|
+ \includegraphics[scale=0.4]{data/experimental_draw.png}
|
|
|
+ \caption{Output of the experimental drawing program. It draws the touch
|
|
|
+ points and their centroid on the screen (the centroid is used as center
|
|
|
+ point for rotation and pinch detection). It also draws a green
|
|
|
+ rectangle which responds to rotation and pinch events.}
|
|
|
+\end{figure}
|
|
|
+
|
|
|
+One of the first observations is the fact that TUIO's \texttt{SET} messages use
|
|
|
+the TUIO coordinate system, as described in appendix \ref{app:tuio}. The test
|
|
|
+program multiplies these with its own dimensions, thus showing the entire
|
|
|
+screen in its window. Also, the implementation only works using the TUIO
|
|
|
+protocol. Other drivers are not supported.
|
|
|
+
|
|
|
+Though using relatively simple math, the rotation and pinch events work
|
|
|
+surprisingly well. Both rotation and pinch use the centroid of all touch
|
|
|
+points. A \emph{rotation} gesture uses the difference in angle relative to the
|
|
|
+centroid of all touch points, and \emph{pinch} uses the difference in distance.
|
|
|
+Both values are normalized using division by the number of touch points. A
|
|
|
+pinch event contains a scale factor, and therefore uses a division of the
|
|
|
+current by the previous average distance to the centroid.
|
|
|
+
|
|
|
+There is a flaw in this implementation. Since the centroid is calculated using
|
|
|
+all current touch points, there cannot be two or more rotation or pinch
|
|
|
+gestures simultaneously. On a large multi-touch table, it is desirable to
|
|
|
+support interaction with multiple hands, or multiple persons, at the same time.
|
|
|
+This kind of application-specific requirements should be defined in the
|
|
|
+application itself, whereas the experimental implementation defines detection
|
|
|
+algorithms based on its test program.
|
|
|
+
|
|
|
+Also, the different detection algorithms are all implemented in the same file,
|
|
|
+making it complex to read or debug, and difficult to extend.
|
|
|
+
|
|
|
+\chapter{Reference implementation in Python}
|
|
|
+\label{app:implementation}
|
|
|
+
|
|
|
+% TODO
|
|
|
+% alleen window.contains op point down, niet move/up
|
|
|
+% een paar simpele windows en trackers
|
|
|
+
|
|
|
\end{document}
|
Taddeus Kroes