|
|
@@ -8,7 +8,7 @@
|
|
|
\hypersetup{colorlinks=true,linkcolor=black,urlcolor=blue,citecolor=DarkGreen}
|
|
|
|
|
|
% Title Page
|
|
|
-\title{Universal multi-touch event mechanism}
|
|
|
+\title{A universal detection mechanism for multi-touch gestures}
|
|
|
\author{Taddeüs Kroes}
|
|
|
\supervisors{Dr. Robert G. Belleman (UvA)}
|
|
|
\signedby{Dr. Robert G. Belleman (UvA)}
|
|
|
@@ -65,6 +65,7 @@ events.
|
|
|
% Deelvragen
|
|
|
To design such a mechanism properly, the following questions are relevant:
|
|
|
\begin{itemize}
|
|
|
+ \item What are the requirements of the mechanism to be universal?
|
|
|
\item What is the input of the mechanism? Different touch drivers have
|
|
|
different API's. To be able to support different drivers (which is
|
|
|
highly desirable), there should probably be a translation from the
|
|
|
@@ -94,20 +95,44 @@ events.
|
|
|
|
|
|
\chapter{Related work}
|
|
|
|
|
|
- \section{GART}
|
|
|
+ \section{Gesture and Activity Recognition Toolkit}
|
|
|
|
|
|
- % TODO
|
|
|
- \cite{GART}
|
|
|
+ The Gesture and Activity Recognition Toolkit (GART) \cite{GART} is a
|
|
|
+ toolkit for the development of gesture-based applications. The toolkit
|
|
|
+ states that the best way to classify gestures is to use machine learning.
|
|
|
+ The programmer trains a program to recognize using the machine learning
|
|
|
+ library from the toolkit. The toolkit contains a callback-mechanism that
|
|
|
+ the programmer uses to execute custom code when a gesture is recognized.
|
|
|
|
|
|
- \section{Processing implementation of simple gestures in Android}
|
|
|
+ Though multi-touch input is not directly supported by the toolkit, the
|
|
|
+ level of abstraction does allow for it to be implemented in the form of a
|
|
|
+ ``touch'' sensor.
|
|
|
|
|
|
- % TODO
|
|
|
- \cite{processingMT}
|
|
|
+ The reason to use machine learning is the statement that gesture detection
|
|
|
+ ``is likely to become increasingly complex and unmanageable'' when using a
|
|
|
+ set of predefined rules to detect whether some sensor input can be seen as
|
|
|
+ a specific gesture. This statement is not necessarily true. If the
|
|
|
+ programmer is given a way to separate the detection of different types of
|
|
|
+ gestures and flexibility in rule definitions, over-complexity can be
|
|
|
+ avoided.
|
|
|
+
|
|
|
+ % oplossing: trackers. bijv. TapTracker, TransformationTracker gescheiden
|
|
|
|
|
|
\section{Gesture recognition software for Windows 7}
|
|
|
|
|
|
% TODO
|
|
|
- \cite{win7touch}
|
|
|
+ The online article \cite{win7touch} presents a Windows 7 application,
|
|
|
+ written in Microsofts .NET. The application shows detected gestures in a
|
|
|
+ canvas. Gesture trackers keep track of stylus locations to detect specific
|
|
|
+ gestures. The event types required to track a touch stylus are ``stylus
|
|
|
+ down'', ``stylus move'' and ``stylus up'' events. A
|
|
|
+ \texttt{GestureTrackerManager} object dispatches these events to gesture
|
|
|
+ trackers. The application supports a limited number of pre-defined
|
|
|
+ gestures.
|
|
|
+
|
|
|
+ An important observation in this application is that different gestures are
|
|
|
+ detected by different gesture trackers, thus separating gesture detection
|
|
|
+ code into maintainable parts.
|
|
|
|
|
|
\section{The TUIO protocol}
|
|
|
|
|
|
@@ -147,6 +172,10 @@ events.
|
|
|
calculated. For more complex objects such as fiducials, arguments like
|
|
|
rotational position is also included.
|
|
|
|
|
|
+ ALIVE and SET messages can be combined to create ``point down'', ``point
|
|
|
+ move'' and ``point up'' events (as used by the \cite[.NET
|
|
|
+ application]{win7touch}).
|
|
|
+
|
|
|
TUIO coordinates range from $0.0$ to $1.0$, with $(0.0, 0.0)$ being the
|
|
|
left top corner of the screen and $(1.0, 1.0)$ the right bottom corner. To
|
|
|
focus events within a window, a translation to window coordinates is
|
|
|
@@ -159,6 +188,25 @@ events.
|
|
|
scale these values back to the actual screen dimension.
|
|
|
\end{quote}
|
|
|
|
|
|
+ In other words, the design of the gesture detection mechanism should
|
|
|
+ incorporate a translation from driver-specific coordinates to pixel
|
|
|
+ coordinates.
|
|
|
+
|
|
|
+ \section{Processing implementation of simple gestures in Android}
|
|
|
+
|
|
|
+ An implementation of a detection mechanism for some simple multi-touch
|
|
|
+ gestures (tap, double tap, rotation, pinch and drag) using
|
|
|
+ Processing\footnote{Processing is a Java-based development environment with
|
|
|
+ an export possibility for Android. See also \url{http://processing.org/.}}
|
|
|
+ can be found found in a forum on the Processing website
|
|
|
+ \cite{processingMT}. The implementation is fairly simple, but it yields
|
|
|
+ some very appealing results. The detection logic of all gestures is
|
|
|
+ combined in a single class. This does not allow for extendability, because
|
|
|
+ the complexity of this class would increase to an undesirable level (as
|
|
|
+ predicted by the GART article \cite{GART}). However, the detection logic
|
|
|
+ itself is partially re-used in the reference implementation of the
|
|
|
+ universal gesture detection mechanism.
|
|
|
+
|
|
|
% TODO
|
|
|
|
|
|
\chapter{Experiments}
|
|
|
@@ -185,7 +233,13 @@ events.
|
|
|
|
|
|
\chapter{Design}
|
|
|
|
|
|
-% TODO: link naar appendix met schema
|
|
|
+ \section{Requirements}
|
|
|
+
|
|
|
+ % TODO
|
|
|
+ % ondersteunen van meerdere drivers
|
|
|
+ % gesture detectie koppelen aan bepaald gedeelte van het scherm
|
|
|
+ % scheiden van detectiecode voor verschillende gesture types
|
|
|
+ % eventueel te gebruiken in meerdere talen
|
|
|
|
|
|
\section{Input server}
|
|
|
|
|
|
@@ -195,18 +249,24 @@ events.
|
|
|
|
|
|
\section{Gesture server}
|
|
|
|
|
|
- \subsection{Windows}
|
|
|
+ % TODO
|
|
|
+ % vertaling naar pixelcoordinaten
|
|
|
+ % toewijzing aan windows
|
|
|
|
|
|
- % TODO
|
|
|
- % toewijzen even aan deel v/h scherm:
|
|
|
- % TUIO coördinaten zijn over het hele scherm en van 0.0 tot 1.0, dus moeten
|
|
|
- % worden vertaald naar pixelcoördinaten binnen een ``window''
|
|
|
+ \section{Windows}
|
|
|
|
|
|
- \subsection{Trackers}
|
|
|
+ % TODO
|
|
|
+ % toewijzen even aan deel v/h scherm:
|
|
|
+ % TUIO coördinaten zijn over het hele scherm en van 0.0 tot 1.0, dus moeten
|
|
|
+ % worden vertaald naar pixelcoördinaten binnen een ``window''
|
|
|
|
|
|
- % TODO
|
|
|
- % event binding/triggering
|
|
|
- % extendability
|
|
|
+ \section{Trackers}
|
|
|
+
|
|
|
+ % TODO
|
|
|
+ % event binding/triggering
|
|
|
+ % extendability
|
|
|
+
|
|
|
+% TODO: link naar appendix met schema
|
|
|
|
|
|
\chapter{Reference implementation}
|
|
|
|
Taddeus Kroes