|
|
@@ -426,25 +426,24 @@ detection for every new gesture-based application.
|
|
|
sufficient to detect many common gestures, like rotation and dragging. The
|
|
|
imperative programming style is also familiar and understandable for a wide
|
|
|
range of application developers. Therefore, the architecture should support
|
|
|
- an imperative style of gesture detection. A problem with an imperative
|
|
|
- programming style is that the explicit detection of different gestures
|
|
|
- requires different gesture detection components. If these components are
|
|
|
- not managed well, the detection logic is prone to become chaotic and
|
|
|
- over-complex.
|
|
|
+ this style of gesture detection. A problem with an imperative programming
|
|
|
+ style is that the explicit detection of different gestures requires
|
|
|
+ different gesture detection components. If these components are not managed
|
|
|
+ well, the detection logic is prone to become chaotic and over-complex.
|
|
|
|
|
|
A way to detect more complex gestures based on a sequence of input events,
|
|
|
is with the use of machine learning methods, such as the Hidden Markov
|
|
|
- Models \footnote{A Hidden Markov Model (HMM) is a statistical model without
|
|
|
- a memory, it can be used to detect gestures based on the current input
|
|
|
- state alone.} used for sign language detection by Gerhard Rigoll et al.
|
|
|
- \cite{conf/gw/RigollKE97}. A sequence of input states can be mapped to a
|
|
|
- feature vector that is recognized as a particular gesture with a certain
|
|
|
+ Models (HMM)\footnote{A Hidden Markov Model (HMM) is a statistical model
|
|
|
+ without a memory, it can be used to detect gestures based on the current
|
|
|
+ input state alone.} used for sign language detection by Gerhard Rigoll et
|
|
|
+ al. \cite{conf/gw/RigollKE97}. A sequence of input states can be mapped to
|
|
|
+ a feature vector that is recognized as a particular gesture with a certain
|
|
|
probability. An advantage of using machine learning with respect to an
|
|
|
- imperative programming style is that complex gestures can be described
|
|
|
+ imperative programming style, is that complex gestures are described
|
|
|
without the use of explicit detection logic, thus reducing code complexity.
|
|
|
For example, the detection of the character `A' being written on the screen
|
|
|
- is difficult to implement using an imperative programming style, while a
|
|
|
- trained machine learning system can produce a match with relative ease.
|
|
|
+ is difficult to implement using explicit detection code, whereas a trained
|
|
|
+ machine learning system can produce a match with relative ease.
|
|
|
|
|
|
To manage complexity and support multiple styles of gesture detection
|
|
|
logic, the architecture has adopted the tracker-based design as described
|
|
|
@@ -503,35 +502,35 @@ detection for every new gesture-based application.
|
|
|
machines, thus distributing computational load. The other machine may even
|
|
|
use a different operating system.
|
|
|
|
|
|
- \section{Example usage}
|
|
|
- \label{sec:example}
|
|
|
-
|
|
|
- This section describes an extended example to illustrate the data flow of
|
|
|
- the architecture. The example application listens to tap events on a button
|
|
|
- within an application window. The window also contains a draggable circle.
|
|
|
- The application window can be resized using \emph{pinch} gestures. Figure
|
|
|
- \ref{fig:examplediagram} shows the architecture created by the pseudo code
|
|
|
- below.
|
|
|
-
|
|
|
-\begin{verbatim}
|
|
|
-initialize GUI framework, creating a window and nessecary GUI widgets
|
|
|
-
|
|
|
-create a root event area that synchronizes position and size with the application window
|
|
|
-define 'rotation' gesture handler and bind it to the root event area
|
|
|
-
|
|
|
-create an event area with the position and radius of the circle
|
|
|
-define 'drag' gesture handler and bind it to the circle event area
|
|
|
-
|
|
|
-create an event area with the position and size of the button
|
|
|
-define 'tap' gesture handler and bind it to the button event area
|
|
|
-
|
|
|
-create a new event server and assign the created root event area to it
|
|
|
-
|
|
|
-start the event server in a new thread
|
|
|
-start the GUI main loop in the current thread
|
|
|
-\end{verbatim}
|
|
|
-
|
|
|
- \examplediagram
|
|
|
+%\section{Example usage}
|
|
|
+%\label{sec:example}
|
|
|
+%
|
|
|
+%This section describes an extended example to illustrate the data flow of
|
|
|
+%the architecture. The example application listens to tap events on a button
|
|
|
+%within an application window. The window also contains a draggable circle.
|
|
|
+%The application window can be resized using \emph{pinch} gestures. Figure
|
|
|
+%\ref{fig:examplediagram} shows the architecture created by the pseudo code
|
|
|
+%below.
|
|
|
+%
|
|
|
+%\begin{verbatim}
|
|
|
+%initialize GUI framework, creating a window and nessecary GUI widgets
|
|
|
+%
|
|
|
+%create a root event area that synchronizes position and size with the application window
|
|
|
+%define 'rotation' gesture handler and bind it to the root event area
|
|
|
+%
|
|
|
+%create an event area with the position and radius of the circle
|
|
|
+%define 'drag' gesture handler and bind it to the circle event area
|
|
|
+%
|
|
|
+%create an event area with the position and size of the button
|
|
|
+%define 'tap' gesture handler and bind it to the button event area
|
|
|
+%
|
|
|
+%create a new event server and assign the created root event area to it
|
|
|
+%
|
|
|
+%start the event server in a new thread
|
|
|
+%start the GUI main loop in the current thread
|
|
|
+%\end{verbatim}
|
|
|
+%
|
|
|
+%\examplediagram
|
|
|
|
|
|
\chapter{Implementation and test applications}
|
|
|
\label{chapter:testapps}
|
|
|
@@ -589,6 +588,11 @@ have been implemented using an imperative programming style. Technical details
|
|
|
about the implementation of gesture detection are described in appendix
|
|
|
\ref{app:implementation-details}.
|
|
|
|
|
|
+%\section{Basic usage}
|
|
|
+
|
|
|
+% TODO
|
|
|
+% example usage uit H3 hierheen halen
|
|
|
+
|
|
|
\section{Full screen Pygame application}
|
|
|
|
|
|
%The goal of this application was to experiment with the TUIO
|
|
|
@@ -608,10 +612,38 @@ application, the detection logic of all gestures is combined in a single class
|
|
|
file. As predicted by the GART article \cite{GART}, this leads to over-complex
|
|
|
code that is difficult to read and debug.
|
|
|
|
|
|
-The application has been rewritten using the reference implementation of the
|
|
|
-architecture. The detection code is separated into two different gesture
|
|
|
-trackers, which are the ``tap'' and ``transformation'' trackers mentioned in
|
|
|
-section \ref{sec:implementation}.
|
|
|
+The original application code consists of two main classes. The ``multi-touch
|
|
|
+server'' starts a ``TUIO server'' that translates TUIO events to ``point
|
|
|
+\{down,move,up\}'' events. Detection of ``tap'' and ``double tap'' gestures is
|
|
|
+performed immediately after an event is received. Other gesture detection runs
|
|
|
+in a separate thread, using the following loop:
|
|
|
+
|
|
|
+\begin{verbatim}
|
|
|
+60 times per second do:
|
|
|
+ detect `single tap' based on the time since the latest `tap' gesture
|
|
|
+
|
|
|
+ if points have been moved, added or removed since last iteration do:
|
|
|
+ calculate the centroid of all points
|
|
|
+ detect `drag' using centroid movement
|
|
|
+ detect `rotation' using average orientation of all points to centroid
|
|
|
+ detect `pinch' using average distance of all points to centroid
|
|
|
+\end{verbatim}
|
|
|
+
|
|
|
+There are two problems with the implementation described above. In the first
|
|
|
+place, low-level events are not grouped before gesture detection. The gesture
|
|
|
+detection uses all events for a single gesture. Therefore, only one element at
|
|
|
+a time can be rotated/resized etc. (see also section \ref{sec:areas}).
|
|
|
+Secondly, all detection code is located in the same class file. To extend the
|
|
|
+application with new gestures, a programmer must extend the code in this class
|
|
|
+file and therefore understand its structure. Since the main loop calls specific
|
|
|
+gesture detection components explicitly in a certain order, the programmer must
|
|
|
+alter the main loop to call custom gesture detection code. This is a problem
|
|
|
+because this way of extending code is not scalable over time. The class file
|
|
|
+would become more and more complex when extended with new gestures. The two
|
|
|
+problems have been solved using event areas and gesture trackers from the
|
|
|
+reference implementation. The gesture detection code has been separated into
|
|
|
+two different gesture trackers, which are the ``tap'' and ``transformation''
|
|
|
+trackers mentioned in section \ref{sec:implementation}.
|
|
|
|
|
|
The positions of all touch objects and their centroid are drawn using the
|
|
|
Pygame library. Since the Pygame library does not provide support to find the
|
|
|
@@ -644,13 +676,35 @@ The application creates a main window, whose size and position are synchronized
|
|
|
with the root event area of the architecture. The synchronization is handled
|
|
|
automatically by a \texttt{GtkEventWindow} object, which is a subclass of
|
|
|
\texttt{gtk.Window}. This object serves as a layer that connects the event area
|
|
|
-functionality of the architecture to GTK+ windows.
|
|
|
+functionality of the architecture to GTK+ windows. The following Python code
|
|
|
+captures the essence of the synchronization layer:
|
|
|
|
|
|
-The main window contains a number of polygons which can be dragged, resized and
|
|
|
-rotated. Each polygon is represented by a separate event area to allow
|
|
|
-simultaneous interaction with different polygons. The main window also responds
|
|
|
-to transformation, by transforming all polygons. Additionally, double tapping
|
|
|
-on a polygon changes its color.
|
|
|
+\begin{verbatim}
|
|
|
+class GtkEventWindow(Window):
|
|
|
+ def __init__(self, width, height):
|
|
|
+ Window.__init__(self)
|
|
|
+
|
|
|
+ # Create an event area to represent the GTK window in the gesture
|
|
|
+ # detection architecture
|
|
|
+ self.area = RectangularArea(0, 0, width, height)
|
|
|
+
|
|
|
+ # The "configure-event" signal is triggered by GTK when the position or
|
|
|
+ # size of the window are updated
|
|
|
+ self.connect("configure-event", self.sync_area)
|
|
|
+
|
|
|
+ def sync_area(self, win, event):
|
|
|
+ # Synchronize the position and size of the event area with that of the
|
|
|
+ # GTK window
|
|
|
+ self.area.width = event.width
|
|
|
+ self.area.height = event.height
|
|
|
+ self.area.set_position(*event.get_coords())
|
|
|
+\end{verbatim}
|
|
|
+
|
|
|
+The application window contains a number of polygons which can be dragged,
|
|
|
+resized and rotated. Each polygon is represented by a separate event area to
|
|
|
+allow simultaneous interaction with different polygons. The main window also
|
|
|
+responds to transformation, by transforming all polygons. Additionally, double
|
|
|
+tapping on a polygon changes its color.
|
|
|
|
|
|
An ``overlay'' event area is used to detect all fingers currently touching the
|
|
|
screen. The application defines a custom gesture tracker, called the ``hand
|
|
|
@@ -659,22 +713,35 @@ between detected fingers to detect which fingers belong to the same hand. The
|
|
|
application draws a line from each finger to the hand it belongs to, as visible
|
|
|
in figure \ref{fig:testapp}.
|
|
|
|
|
|
+Note that however it is a full screen event area, the overlay is not used as
|
|
|
+the root of the event area tree. Instead, the overlay is the right sibling of
|
|
|
+the application window area in the tree. This is needed, because the
|
|
|
+application window and its children stop the propagation of events to the root
|
|
|
+event area. The overlay area needs all events to keep its hand tracker
|
|
|
+up-to-date. Therefore, the tree is arranged in such a way that the overlay
|
|
|
+event area handles an event first, before the application window can stop its
|
|
|
+propagation. The event area implementation delegates events to its children in
|
|
|
+right-to left order, because area's that are added to the tree later are
|
|
|
+assumed to be positioned over their previously added siblings.
|
|
|
+
|
|
|
\begin{figure}[h!]
|
|
|
\center
|
|
|
\includegraphics[scale=0.35]{data/testapp.png}
|
|
|
- \caption{Screenshot of the second test application. Two polygons can be
|
|
|
- dragged, rotated and scaled. Separate groups of fingers are recognized as
|
|
|
- hands, each hand is drawn as a centroid with a line to each finger.}
|
|
|
+ \caption{
|
|
|
+ Screenshot of the second test application. Two polygons can be dragged,
|
|
|
+ rotated and scaled. Separate groups of fingers are recognized as hands,
|
|
|
+ each hand is drawn as a centroid with a line to each finger.
|
|
|
+ }
|
|
|
\label{fig:testapp}
|
|
|
\end{figure}
|
|
|
|
|
|
-To manage the propagation of events used for transformations, the applications
|
|
|
-arranges its event areas in a tree structure as described in section
|
|
|
-\ref{sec:tree}. Each transformable event area has its own ``transformation
|
|
|
-tracker'', which stops the propagation of events used for transformation
|
|
|
-gestures. Because the propagation of these events is stopped, overlapping
|
|
|
-polygons do not cause a problem. Figure \ref{fig:testappdiagram} shows the tree
|
|
|
-structure used by the application.
|
|
|
+To manage the propagation of events used for transformations and tapping, the
|
|
|
+applications arranges its event areas in a tree structure as described in
|
|
|
+section \ref{sec:tree}. Each transformable event area has its own
|
|
|
+``transformation tracker'', which stops the propagation of events used for
|
|
|
+transformation gestures. Because the propagation of these events is stopped,
|
|
|
+overlapping polygons do not cause a problem. Figure \ref{fig:testappdiagram}
|
|
|
+shows the tree structure used by the application.
|
|
|
|
|
|
Note that the overlay event area, though covering the whole screen surface, is
|
|
|
not the root event area. The overlay event area is placed on top of the
|
|
|
@@ -689,7 +756,7 @@ first.
|
|
|
|
|
|
\testappdiagram
|
|
|
|
|
|
-\section{Results}
|
|
|
+\section{Conclusion}
|
|
|
|
|
|
\emph{TODO: Tekortkomingen aangeven die naar voren komen uit de tests}
|
|
|
|
Taddeus Kroes