Skip to content

M
multitouch
Commit 4e493e3c authored by Taddeüs Kroes's avatar Taddeüs Kroes
Browse files
Options

Clarified some sections.

parent b3dd89c3
Hide whitespace changes
Inline Side-by-side
Showing with 154 additions and 82 deletions
docs/data/diagrams.tex
View file @ 4e493e3c
......@@ -309,39 +309,42 @@
\begin{figure}[h!]
\center
\architecture{
\tikzstyle{area} = [block, fill=gray!15];
\tikzstyle{tracker} = [block, draw=gray!50];
\node[block, below of=driver] (eventdriver) {Event driver}
edge[linefrom] node[right, near end] {device-specific messages} (driver);
\node[block, below of=eventdriver] (rootarea) {Screen area}
\node[area, below of=eventdriver] (rootarea) {Screen area}
edge[linefrom] (eventdriver);
\node[block, below of=rootarea, xshift=-5em] (appwindow) {Application window area}
\node[area, below of=rootarea, xshift=-5em] (appwindow) {Application window area}
edge[lineto, <->] (rootarea);
\node[block, left of=appwindow, xshift=-4em, text width=7em] {Transformation tracker}
\node[tracker, left of=appwindow, xshift=-4em, text width=7em] {Transformation tracker}
edge[lineto, dotted, bend right=10] (appwindow)
edge[linefrom, bend left=10] (appwindow);
\node[block, below of=rootarea, xshift=5em] (overlay) {Overlay area}
\node[area, below of=rootarea, xshift=5em] (overlay) {Overlay area}
edge[lineto, <->] (rootarea);
\node[block, right of=overlay, xshift=4em] (tracker) {Hand tracker}
\node[tracker, right of=overlay, xshift=4em] (tracker) {Hand tracker}
edge[lineto, dotted, bend left=10] (overlay)
edge[linefrom, bend right=10] (overlay);
\node[block, below of=appwindow, xshift=-5em] (rectangle) {Rectangle area}
\node[area, below of=appwindow, xshift=-5em] (rectangle) {Rectangle area}
edge[lineto, <->] (appwindow);
\node[block, left of=rectangle, xshift=-4em, yshift=2em, text width=7em] (recttracker) {Transformation tracker}
\node[tracker, left of=rectangle, xshift=-4em, yshift=2em, text width=7em] (recttracker) {Transformation tracker}
edge[lineto, dotted, bend left=10] (rectangle)
edge[linefrom, bend right=10] (rectangle);
\node[block, left of=rectangle, xshift=-4em, yshift=-2em, text width=7em] {Tap tracker}
\node[tracker, left of=rectangle, xshift=-4em, yshift=-2em, text width=7em] {Tap tracker}
edge[lineto, dotted, bend right=10] (rectangle)
edge[linefrom, bend left=10] (rectangle);
\node[block, below of=appwindow, xshift=5em] (triangle) {Triangle area}
\node[area, below of=appwindow, xshift=5em] (triangle) {Triangle area}
edge[lineto, <->] (appwindow);
\node[block, right of=triangle, xshift=4em, yshift=2em, text width=7em] {Transformation tracker}
\node[tracker, right of=triangle, xshift=4em, yshift=2em, text width=7em] {Transformation tracker}
edge[lineto, dotted, bend right=10] (triangle)
edge[linefrom, bend left=10] (triangle);
\node[block, right of=triangle, xshift=4em, yshift=-2em, text width=7em] (taptracker) {Tap tracker}
\node[tracker, right of=triangle, xshift=4em, yshift=-2em, text width=7em] (taptracker) {Tap tracker}
edge[lineto, dotted, bend left=10] (triangle)
edge[linefrom, bend right=10] (triangle);
......@@ -353,14 +356,16 @@
\group{recttracker}{eventdriver}{tracker}{taptracker}{Architecture}
}
\caption{Diagram representation of the second test application. A full
screen event area contains an application window and a full screen
overlay. The application window contains a rectangle and a triangle.
the application window and its children can be transformed, and thus
each have ``transformation tracker''. The rectangle and triangle also
have a ``tap tracker'' that detects double tap gestures. Dotted arrows
represent a flow of gestures, regular arrows represent events (unless
labeled otherwise).}
\caption{
Diagram representation of the second test application. A full
screen event area contains an application window and a full screen
overlay. The application window contains a rectangle and a
triangle. the application window and its children can be
transformed, and thus each have ``transformation tracker''. The
rectangle and triangle also have a ``tap tracker'' that detects
double tap gestures. Dotted arrows represent a flow of gestures,
regular arrows represent events (unless labeled otherwise).
}
\label{fig:testappdiagram}
\end{figure}
}
docs/report.tex
View file @ 4e493e3c
......@@ -426,25 +426,24 @@ detection for every new gesture-based application.
sufficient to detect many common gestures, like rotation and dragging. The
imperative programming style is also familiar and understandable for a wide
range of application developers. Therefore, the architecture should support
an imperative style of gesture detection. A problem with an imperative
programming style is that the explicit detection of different gestures
requires different gesture detection components. If these components are
not managed well, the detection logic is prone to become chaotic and
over-complex.
this style of gesture detection. A problem with an imperative programming
style is that the explicit detection of different gestures requires
different gesture detection components. If these components are not managed
well, the detection logic is prone to become chaotic and over-complex.
A way to detect more complex gestures based on a sequence of input events,
is with the use of machine learning methods, such as the Hidden Markov
Models \footnote{A Hidden Markov Model (HMM) is a statistical model without
a memory, it can be used to detect gestures based on the current input
state alone.} used for sign language detection by Gerhard Rigoll et al.
\cite{conf/gw/RigollKE97}. A sequence of input states can be mapped to a
feature vector that is recognized as a particular gesture with a certain
Models (HMM)\footnote{A Hidden Markov Model (HMM) is a statistical model
without a memory, it can be used to detect gestures based on the current
input state alone.} used for sign language detection by Gerhard Rigoll et
al. \cite{conf/gw/RigollKE97}. A sequence of input states can be mapped to
a feature vector that is recognized as a particular gesture with a certain
probability. An advantage of using machine learning with respect to an
imperative programming style is that complex gestures can be described
imperative programming style, is that complex gestures are described
without the use of explicit detection logic, thus reducing code complexity.
For example, the detection of the character `A' being written on the screen
is difficult to implement using an imperative programming style, while a
trained machine learning system can produce a match with relative ease.
is difficult to implement using explicit detection code, whereas a trained
machine learning system can produce a match with relative ease.
To manage complexity and support multiple styles of gesture detection
logic, the architecture has adopted the tracker-based design as described
......@@ -503,35 +502,35 @@ detection for every new gesture-based application.
machines, thus distributing computational load. The other machine may even
use a different operating system.
\section{Example usage}
\label{sec:example}
This section describes an extended example to illustrate the data flow of
the architecture. The example application listens to tap events on a button
within an application window. The window also contains a draggable circle.
The application window can be resized using \emph{pinch} gestures. Figure
\ref{fig:examplediagram} shows the architecture created by the pseudo code
below.
\begin{verbatim}
initialize GUI framework, creating a window and nessecary GUI widgets
create a root event area that synchronizes position and size with the application window
define 'rotation' gesture handler and bind it to the root event area
create an event area with the position and radius of the circle
define 'drag' gesture handler and bind it to the circle event area
create an event area with the position and size of the button
define 'tap' gesture handler and bind it to the button event area
create a new event server and assign the created root event area to it
start the event server in a new thread
start the GUI main loop in the current thread
\end{verbatim}
\examplediagram
%\section{Example usage}
%\label{sec:example}
%
%This section describes an extended example to illustrate the data flow of
%the architecture. The example application listens to tap events on a button
%within an application window. The window also contains a draggable circle.
%The application window can be resized using \emph{pinch} gestures. Figure
%\ref{fig:examplediagram} shows the architecture created by the pseudo code
%below.
%
%\begin{verbatim}
%initialize GUI framework, creating a window and nessecary GUI widgets
%
%create a root event area that synchronizes position and size with the application window
%define 'rotation' gesture handler and bind it to the root event area
%
%create an event area with the position and radius of the circle
%define 'drag' gesture handler and bind it to the circle event area
%
%create an event area with the position and size of the button
%define 'tap' gesture handler and bind it to the button event area
%
%create a new event server and assign the created root event area to it
%
%start the event server in a new thread
%start the GUI main loop in the current thread
%\end{verbatim}
%
%\examplediagram
\chapter{Implementation and test applications}
\label{chapter:testapps}
......@@ -589,6 +588,11 @@ have been implemented using an imperative programming style. Technical details
about the implementation of gesture detection are described in appendix
\ref{app:implementation-details}.
%\section{Basic usage}
% TODO
% example usage uit H3 hierheen halen
\section{Full screen Pygame application}
%The goal of this application was to experiment with the TUIO
......@@ -608,10 +612,38 @@ application, the detection logic of all gestures is combined in a single class
file. As predicted by the GART article \cite{GART}, this leads to over-complex
code that is difficult to read and debug.
The application has been rewritten using the reference implementation of the
architecture. The detection code is separated into two different gesture
trackers, which are the ``tap'' and ``transformation'' trackers mentioned in
section \ref{sec:implementation}.
The original application code consists of two main classes. The ``multi-touch
server'' starts a ``TUIO server'' that translates TUIO events to ``point
\{down,move,up\}'' events. Detection of ``tap'' and ``double tap'' gestures is
performed immediately after an event is received. Other gesture detection runs
in a separate thread, using the following loop:
\begin{verbatim}
60 times per second do:
detect `single tap' based on the time since the latest `tap' gesture
if points have been moved, added or removed since last iteration do:
calculate the centroid of all points
detect `drag' using centroid movement
detect `rotation' using average orientation of all points to centroid
detect `pinch' using average distance of all points to centroid
\end{verbatim}
There are two problems with the implementation described above. In the first
place, low-level events are not grouped before gesture detection. The gesture
detection uses all events for a single gesture. Therefore, only one element at
a time can be rotated/resized etc. (see also section \ref{sec:areas}).
Secondly, all detection code is located in the same class file. To extend the
application with new gestures, a programmer must extend the code in this class
file and therefore understand its structure. Since the main loop calls specific
gesture detection components explicitly in a certain order, the programmer must
alter the main loop to call custom gesture detection code. This is a problem
because this way of extending code is not scalable over time. The class file
would become more and more complex when extended with new gestures. The two
problems have been solved using event areas and gesture trackers from the
reference implementation. The gesture detection code has been separated into
two different gesture trackers, which are the ``tap'' and ``transformation''
trackers mentioned in section \ref{sec:implementation}.
The positions of all touch objects and their centroid are drawn using the
Pygame library. Since the Pygame library does not provide support to find the
......@@ -644,13 +676,35 @@ The application creates a main window, whose size and position are synchronized
with the root event area of the architecture. The synchronization is handled
automatically by a \texttt{GtkEventWindow} object, which is a subclass of
\texttt{gtk.Window}. This object serves as a layer that connects the event area
functionality of the architecture to GTK+ windows.
functionality of the architecture to GTK+ windows. The following Python code
captures the essence of the synchronization layer:
The main window contains a number of polygons which can be dragged, resized and
rotated. Each polygon is represented by a separate event area to allow
simultaneous interaction with different polygons. The main window also responds
to transformation, by transforming all polygons. Additionally, double tapping
on a polygon changes its color.
\begin{verbatim}
class GtkEventWindow(Window):
def __init__(self, width, height):
Window.__init__(self)
# Create an event area to represent the GTK window in the gesture
# detection architecture
self.area = RectangularArea(0, 0, width, height)
# The "configure-event" signal is triggered by GTK when the position or
# size of the window are updated
self.connect("configure-event", self.sync_area)
def sync_area(self, win, event):
# Synchronize the position and size of the event area with that of the
# GTK window
self.area.width = event.width
self.area.height = event.height
self.area.set_position(*event.get_coords())
\end{verbatim}
The application window contains a number of polygons which can be dragged,
resized and rotated. Each polygon is represented by a separate event area to
allow simultaneous interaction with different polygons. The main window also
responds to transformation, by transforming all polygons. Additionally, double
tapping on a polygon changes its color.
An ``overlay'' event area is used to detect all fingers currently touching the
screen. The application defines a custom gesture tracker, called the ``hand
......@@ -659,22 +713,35 @@ between detected fingers to detect which fingers belong to the same hand. The
application draws a line from each finger to the hand it belongs to, as visible
in figure \ref{fig:testapp}.
Note that however it is a full screen event area, the overlay is not used as
the root of the event area tree. Instead, the overlay is the right sibling of
the application window area in the tree. This is needed, because the
application window and its children stop the propagation of events to the root
event area. The overlay area needs all events to keep its hand tracker
up-to-date. Therefore, the tree is arranged in such a way that the overlay
event area handles an event first, before the application window can stop its
propagation. The event area implementation delegates events to its children in
right-to left order, because area's that are added to the tree later are
assumed to be positioned over their previously added siblings.
\begin{figure}[h!]
\center
\includegraphics[scale=0.35]{data/testapp.png}
\caption{Screenshot of the second test application. Two polygons can be
dragged, rotated and scaled. Separate groups of fingers are recognized as
hands, each hand is drawn as a centroid with a line to each finger.}
\caption{
Screenshot of the second test application. Two polygons can be dragged,
rotated and scaled. Separate groups of fingers are recognized as hands,
each hand is drawn as a centroid with a line to each finger.
}
\label{fig:testapp}
\end{figure}
To manage the propagation of events used for transformations, the applications
arranges its event areas in a tree structure as described in section
\ref{sec:tree}. Each transformable event area has its own ``transformation
tracker'', which stops the propagation of events used for transformation
gestures. Because the propagation of these events is stopped, overlapping
polygons do not cause a problem. Figure \ref{fig:testappdiagram} shows the tree
structure used by the application.
To manage the propagation of events used for transformations and tapping, the
applications arranges its event areas in a tree structure as described in
section \ref{sec:tree}. Each transformable event area has its own
``transformation tracker'', which stops the propagation of events used for
transformation gestures. Because the propagation of these events is stopped,
overlapping polygons do not cause a problem. Figure \ref{fig:testappdiagram}
shows the tree structure used by the application.
Note that the overlay event area, though covering the whole screen surface, is
not the root event area. The overlay event area is placed on top of the
......@@ -689,7 +756,7 @@ first.
\testappdiagram
\section{Results}
\section{Conclusion}
\emph{TODO: Tekortkomingen aangeven die naar voren komen uit de tests}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment