Skip to content

M
multitouch
Commit 7c17c783 authored by Taddeüs Kroes's avatar Taddeüs Kroes
Browse files
Options

Wrote 'Experiments' and started writing 'Design' chapters in report.

parent c0c33d52
Hide whitespace changes
Inline Side-by-side
Showing with 320 additions and 113 deletions
docs/data/server_scheme.tex → docs/data/diagram.tex
View file @ 7c17c783
\documentclass{minimal}
\usepackage{tikz}
\usetikzlibrary{shapes,arrows}
\begin{document}
\pagestyle{empty}
% Block+line styles
\tikzstyle{block} = [rectangle, draw, fill=blue!20, text width=5em,
text centered, rounded corners, minimum height=4em]
......@@ -17,18 +9,18 @@
\begin{tikzpicture}[node distance = 7em, auto]
% Servers
\node [block] (inputserver) {Input server};
\node [right of=inputserver] (inputserverdots) {$\cdots$};
\node [block] (eventserver) {Event server};
\node [right of=eventserver] (eventserverdots) {$\cdots$};
\node [impl, right of=inputserverdots] (tuioserver) {TUIO server};
\node [impl, right of=eventserverdots] (tuioserver) {TUIO server};
\node [block, below of=inputserver] (gestureserver) {Gesture server};
\path [line] (inputserver) -- node {triggers [down, move, up]}(gestureserver);
\node [block, below of=eventserver] (gestureserver) {Gesture erver};
\path [line] (eventserver) -- node {trigger events of all touch points}(gestureserver);
% Window
\node [block, below of=gestureserver] (window) {Window};
\node [right of=window] (windowdots) {$\cdots$};
\path [line] (gestureserver) -- node {triggers [down, move, up]}(window);
\path [line] (gestureserver) -- node {trigger events of containing touch points}(window);
\node [impl, right of=windowdots] (rectangularwindow) {Rectangular window};
\node [impl, right of=rectangularwindow] (circularwindow) {Circular window};
......@@ -36,7 +28,7 @@
% Tracker
\node [block, below of=window] (tracker) {Gesture tracker};
\node [right of=tracker] (trackerdots) {$\cdots$};
\path [line] (window) -- node {triggers [down, move, up]}(tracker);
\path [line] (window) -- node {delegate events}(tracker);
\node [impl, right of=trackerdots] (taptracker) {Tap tracker};
\node [impl, right of=taptracker, text width=7em] (transformtracker) {Transformation tracker};
......@@ -44,16 +36,12 @@
% Gesture
\node [block, below of=tracker] (gesture) {Gesture};
\node [right of=gesture] (gesturedots) {$\cdots$};
\path [line] (tracker) -- node {triggers gesture}(gesture);
\path [line] (tracker) -- node {trigger by calling gesture handler}(gesture);
% Client application
\node [app, left of=window, xshift=-14em] (app) {Client application};
\path [line, dashed] (app) -- node [near start] {binds gesture handler}(tracker);
\path [line, dashed] (tracker) -- node [near start] {triggers gesture handler}(app);
\path [line, dashed] (app) -- node [near start] {starts}(gestureserver);
\path [line, dashed] (app) -- node [near start] {adds to server}(window);
\node [app, left of=window, xshift=-11em] (app) {Client application};
\path [line, dashed] (app) -- node [left=20, near end] {add to window, bind gesture handler}(tracker);
\path [line, dashed] (app) -- node [near start] {start}(gestureserver);
\path [line, dashed] (app) -- node {add to gesture server}(window);
\end{tikzpicture}
\end{document}
docs/report.bib
View file @ 7c17c783
......@@ -51,3 +51,36 @@
note = {10.1007/978-3-540-73110-8\_78},
year = {2007}
}
@article{OSC,
author = {Wright, Matthew},
title = {Open Sound Control: an enabling technology for musical networking},
journal = {Org. Sound},
issue_date = {December 2005},
volume = {10},
number = {3},
month = {dec},
year = {2005},
issn = {1355-7718},
pages = {193--200},
numpages = {8},
url = {http://dx.doi.org/10.1017/S1355771805000932},
doi = {10.1017/S1355771805000932},
acmid = {1109064},
publisher = {Cambridge University Press},
address = {New York, NY, USA},
}
@misc{pyOSC,
author = {v2lab},
title = {pyOSC - A Simple OpenSoundControl implementation, in Pure Python.},
year = {2008},
howpublished = {\url{https://trac.v2.nl/wiki/pyOSC}}
}
@misc{pyTUIO,
author = {Jannis Leidel},
title = {pyTUIO - A Python library that understands the TUIO protocol.},
year = {2008},
howpublished = {\url{http://code.google.com/p/pytuio/}}
}
docs/report.tex
View file @ 7c17c783
......@@ -2,7 +2,8 @@
\usepackage[english]{babel}
\usepackage[utf8]{inputenc}
\usepackage{hyperref,graphicx,float}
\usepackage{hyperref,graphicx,float,tikz}
\usetikzlibrary{shapes,arrows}
% Link colors
\hypersetup{colorlinks=true,linkcolor=black,urlcolor=blue,citecolor=DarkGreen}
......@@ -41,8 +42,8 @@ these frameworks have no access to their multi-touch events.
% Aanleiding
This problem was observed during an attempt to create a multi-touch
``interactor'' class for the Visualization Toolkit (VTK \cite{VTK}). Because
VTK provides the application framework here, it is undesirable to use an entire
``interactor'' class for the Visualization Toolkit \cite[VTK]{VTK}. Because VTK
provides the application framework here, it is undesirable to use an entire
framework like Qt simultaneously only for its multi-touch support.
% Ruw doel
......@@ -65,7 +66,6 @@ events.
% Deelvragen
To design such a mechanism properly, the following questions are relevant:
\begin{itemize}
\item What are the requirements of the mechanism to be universal?
\item What is the input of the mechanism? Different touch drivers have
different API's. To be able to support different drivers (which is
highly desirable), there should probably be a translation from the
......@@ -73,25 +73,33 @@ events.
\item How can extendability be accomplished? The set of supported events
should not be limited to a single implementation, but an application
should be able to define its own custom events.
\item How can the mechanism be used by different programming languages?
A universal mechanism should not be limited to be used in only one
language.
\item Can events be shared with multiple processes at the same time? For
example, a network implementation could run as a service instead of
within a single application, triggering events in any application that
needs them.
\item Is performance an issue? For example, an event loop with rotation
detection could swallow up more processing resources than desired.
% FIXME: gaan we nog wat doen met onderstaand?
%\item Is performance an issue? For example, an event loop with rotation
% detection could swallow up more processing resources than desired.
\item How can the mechanism be integrated in a VTK interactor?
\end{itemize}
% Afbakening
The scope of this thesis includes the design of an multi-touch triggering
mechanism, a reference implementation of this design, and its integration
into a VTK interactor. To be successful, the design should allow for
extensions to be added to any implementation. The reference implementation
is a Proof of Concept that translates TUIO events to some simple touch
gestures that are used by a VTK interactor.
The scope of this thesis includes the design of a universal multi-touch
triggering mechanism, a reference implementation of this design, and its
integration into a VTK interactor. To be successful, the design should
allow for extensions to be added to any implementation.
The reference implementation is a Proof of Concept that translates TUIO
events to some simple touch gestures that are used by a VTK interactor.
Being a Proof of Concept, the reference implementation itself does not
necessarily need to meet all the requirements of the design.
\section{Structure of this document}
% TODO
% TODO: pas als het klaar is
\chapter{Related work}
......@@ -134,7 +142,25 @@ events.
detected by different gesture trackers, thus separating gesture detection
code into maintainable parts.
\section{Processing implementation of simple gestures in Android}
An implementation of a detection mechanism for some simple multi-touch
gestures (tap, double tap, rotation, pinch and drag) using
Processing\footnote{Processing is a Java-based development environment with
an export possibility for Android. See also \url{http://processing.org/.}}
can be found found in a forum on the Processing website
\cite{processingMT}. The implementation is fairly simple, but it yields
some very appealing results. The detection logic of all gestures is
combined in a single class. This does not allow for extendability, because
the complexity of this class would increase to an undesirable level (as
predicted by the GART article \cite{GART}). However, the detection logic
itself is partially re-used in the reference implementation of the
universal gesture detection mechanism.
\chapter{Preliminary}
\section{The TUIO protocol}
\label{sec:tuio}
The TUIO protocol \cite{TUIO} defines a way to geometrically describe
tangible objects, such as fingers or fiducials on a multi-touch table. The
......@@ -142,15 +168,13 @@ events.
information is sent to the TUIO UDP port (3333 by default).
For efficiency reasons, the TUIO protocol is encoded using the Open Sound
Control (OSC)\footnote{\url{http://opensoundcontrol.org/specification}}
format. An OSC server/client implementation is available for Python:
pyOSC\footnote{\url{https://trac.v2.nl/wiki/pyOSC}}.
Control \cite[OSC]{OSC} format. An OSC server/client implementation is
available for Python: pyOSC \cite{pyOSC}.
A Python implementation of the TUIO protocol also exists:
pyTUIO\footnote{\url{http://code.google.com/p/pytuio/}}. However, the
execution of an example script yields an error regarding Python's built-in
\texttt{socket} library. Therefore, the reference implementation uses the
pyOSC package to receive TUIO messages.
A Python implementation of the TUIO protocol also exists: pyTUIO
\cite{pyTUIO}. However, the execution of an example script yields an error
regarding Python's built-in \texttt{socket} library. Therefore, the
reference implementation uses the pyOSC package to receive TUIO messages.
The two most important message types of the protocol are ALIVE and SET
messages. An ALIVE message contains the list of session id's that are
......@@ -188,26 +212,10 @@ events.
scale these values back to the actual screen dimension.
\end{quote}
In other words, the design of the gesture detection mechanism should
incorporate a translation from driver-specific coordinates to pixel
coordinates.
\section{Processing implementation of simple gestures in Android}
\section{The Visualization Toolkit}
\label{sec:vtk}
An implementation of a detection mechanism for some simple multi-touch
gestures (tap, double tap, rotation, pinch and drag) using
Processing\footnote{Processing is a Java-based development environment with
an export possibility for Android. See also \url{http://processing.org/.}}
can be found found in a forum on the Processing website
\cite{processingMT}. The implementation is fairly simple, but it yields
some very appealing results. The detection logic of all gestures is
combined in a single class. This does not allow for extendability, because
the complexity of this class would increase to an undesirable level (as
predicted by the GART article \cite{GART}). However, the detection logic
itself is partially re-used in the reference implementation of the
universal gesture detection mechanism.
% TODO
% TODO
\chapter{Experiments}
......@@ -227,6 +235,71 @@ events.
% Proof of Concept: VTK interactor
\section{Experimenting with TUIO and event bindings}
\label{sec:experimental-draw}
When designing a software library, its API should be understandable and
easy to use for programmers. To find out the basic requirements of the API
to be usable, an experimental program has been written based on the
Processing code from \cite{processingMT}. The program receives TUIO events
and translates them to point \emph{down}, \emph{move} and \emph{up} events.
These events are then interpreted to be (double or single) \emph{tap},
\emph{rotation} or \emph{pinch} gestures. A simple drawing program then
draws the current state to the screen using the PyGame library. The output
of the program can be seen in figure \ref{fig:draw}.
\begin{figure}[H]
\center
\label{fig:draw}
\includegraphics[scale=0.4]{data/experimental_draw.png}
\caption{Output of the experimental drawing program. It draws the touch
points and their centroid on the screen (the centroid is used
as center point for rotation and pinch detection). It also
draws a green rectangle which responds to rotation and pinch
events.}
\end{figure}
One of the first observations is the fact that TUIO's \texttt{SET} messages
use the TUIO coordinate system, as described in section \ref{sec:tuio}.
The test program multiplies these with its own dimensions, thus showing the
entire screen in its window. Also, the implementation only works using the
TUIO protocol. Other drivers are not supported.
Though using relatively simple math, the rotation and pinch events work
surprisingly well. Both rotation and pinch use the centroid of all touch
points. A \emph{rotation} gesture uses the difference in angle relative to
the centroid of all touch points, and \emph{pinch} uses the difference in
distance. Both values are normalized using division by the number of touch
points. A pinch event contains a scale factor, and therefore uses a
division of the current by the previous average distance to the centroid.
There is a flaw in this implementation. Since the centroid is calculated
using all current touch points, there cannot be two or more rotation or
pinch gestures simultaneously. On a large multi-touch table, it is
desirable to support interaction with multiple hands, or multiple persons,
at the same time.
Also, the different detection algorithms are all implemented in the same
file, making it complex to read or debug, and difficult to extend.
\section{VTK interactor}
% TODO
% VTK heeft eigen pipeline, mechanisme moet daarnaast draaien
\section{Summary of observations}
\label{sec:observations}
\begin{itemize}
\item The TUIO protocol uses a distinctive coordinate system and set of
messages.
\item Touch events occur outside of the application window.
\item Gestures that use multiple touch points are using all touch
points (not a subset of them).
\item Code complexity increases when detection algorithms are added.
\item % TODO: VTK interactor observations
\end{itemize}
% -------
% Results
% -------
......@@ -234,49 +307,176 @@ events.
\chapter{Design}
\section{Requirements}
\label{sec:requirements}
% TODO
% ondersteunen van meerdere drivers
% gesture detectie koppelen aan bepaald gedeelte van het scherm
% scheiden van detectiecode voor verschillende gesture types
% eventueel te gebruiken in meerdere talen
\section{Input server}
From the observations in section \ref{sec:observations}, a number of
requirements can be specified for the design of the event mechanism:
% TODO
% vertaling driver naar point down, move, up
% TUIO in reference implementation
\section{Gesture server}
% TODO
% vertaling naar pixelcoordinaten
% toewijzing aan windows
\begin{itemize}
% vertalen driver-specifieke events naar algemeen formaat
\item To be able to support multiple input drivers, there must be a
translation from driver-specific messages to some common format
that can be used in gesture detection algorithms.
% events toewijzen aan GUI window (windows)
\item An application GUI window should be able to receive only events
occuring within that window, and not outside of it.
% scheiden groepen touchpoints voor verschillende gestures (windows)
\item To support multiple objects that are performing different
gestures at the same time, the mechanism must be able to perform
gesture detection on a subset of the active touch points.
% scheiden van detectiecode voor verschillende gesture types
\item To avoid an increase in code complexity when adding new detection
algorithms, detection code of different gesture types must be
separated.
\end{itemize}
\section{Windows}
\section{Components}
Based on the requirements from section \ref{sec:requirements}, a design
for the mechanism has been created. The design consists of a number of
components, each having a specific set of tasks.
\subsection{Event server}
% vertaling driver naar point down, move, up
% vertaling naar schermpixelcoordinaten
% TUIO in reference implementation
The \emph{event server} is an abstraction for driver-specific server
implementations, such as a TUIO server. It receives driver-specific
messages and tanslates these to a common set of events and a common
coordinate system.
A minimal example of a common set of events is $\{point\_down,
point\_move, point\_up\}$. This is the set used by the reference
implementation. Respectively, these events represent an object being
placed on the screen, moving along the surface of the screen, and being
released from the screen.
A more extended set could also contain the same three events for a
surface touching the screen. However, a surface can have a rotational
property, like the ``fiducials'' type in the TUIO protocol. This
results in as $\{point\_down, point\_move, point\_up, surface\_down,
surface\_move, surface\_up,\\surface\_rotate\}$.
An important note here, is that similar events triggered by different
event servers must have the same event type and parameters. In other
words, the output of the event servers should be determined by the
gesture servers (not the contrary).
The output of an event server implementation should also use a common
coordinate system, that is the coordinate system used by the gesture
server. For example, the reference implementation uses screen
coordinates in pixels, where (0, 0) is the upper left corner of the
screen.
The abstract class definition of the event server should provide some
functionality to detect which driver-specific event server
implementation should be used.
\subsection{Gesture trackers}
A \emph{gesture tracker} detects a single gesture type, given a set of
touch points. If one group of points on the screen is assigned to one
tracker and another group to another tracker, multiple gestures, an be
detected at the same time. For this assignment, the mechanism uses
windows. These will be described in the next section.
% event binding/triggering
A gesture tracker triggers a gesture event by executing a callback.
Callbacks are ``bound'' to a tracker by the application. Because
multiple gesture types can have very similar detection algorithm, a
tracker can detect multiple different types of gestures. For instance,
the rotation and pinch gestures from the experimental program in
section \ref{sec:experimental-draw} both use the centroid of all touch
points.
If no callback is bound for a particular gesture type, no detection of
that type is needed. A tracker implementation can use this knowledge
for code optimization.
% scheiding algoritmiek
A tracker implementation defines the gesture types it can trigger, and
the detection algorithms to trigger them. Consequently, detection
algorithms can be separated in different trackers. Different
trackers can be saved in different files, reducing the complexity of
the code in a single file. \\
% extendability
Because tacker defines its own set of gesture types, the application
developer can define application-specific trackers (by extending a base
\texttt{GestureTracker} class, for example). In fact, any built-in
gesture trackers of an implementation are also created this way. This
allows for a plugin-like way of programming, which is very desirable if
someone would want to build a library of gesture trackers. Such a
library can easy be extended by others.
\subsection{Windows}
A \emph{window} represents a subset of the entire screen surface. The
goal of a window is to restrict the detection of certain gestures to
certain areas. A window contains a list of touch points, and a list of
trackers. A window server (defined in the next section) assigns touch
points to a window, but the window itself defines functionality to
check whether a touch point is inside the window. This way, new windows
can be defined to fit over any 2D object used by the application.
The first and most obvious use of a window is to restrict touch events
to a single application window. However, the use of windows can be used
in a lot more powerful way.
For example, an application contains an image with a transparent
background that can be dragged around. The user can only drag the image
by touching its foreground. To accomplish this, the application
programmer can define a window type that uses a bitmap to determine
whether a touch point is on the visible image surface. The tracker
which detects drag gestures is then bound to this window, limiting the
occurence of drag events to the image surface.
% toewijzen even aan deel v/h scherm:
% TUIO coördinaten zijn over het hele scherm en van 0.0 tot 1.0, dus
% moeten worden vertaald naar pixelcoördinaten binnen een ``window''
% TODO
\subsection{Gesture server}
% luistert naar point down, move, up
The \emph{gesture server} delegates events from the event server to the
set of windows that contain the touch points related to the events.
% toewijzing point (down) aan window(s)
The gesture server contains a list of windows. When the event server
triggers an event, the gesture server ``asks'' each window whether it
contains the related touch point. If so, the window updates its gesture
trackers, which can then trigger gestures.
\section{Diagram of component relations}
\begin{figure}[H]
\input{data/diagram}
% TODO: caption
\end{figure}
\section{Example usage}
% TODO
% toewijzen even aan deel v/h scherm:
% TUIO coördinaten zijn over het hele scherm en van 0.0 tot 1.0, dus moeten
% worden vertaald naar pixelcoördinaten binnen een ``window''
% vertellen hoe je tracker aanmaakt, binnen een window
\section{Trackers}
%\section{Network protocol}
% TODO
% event binding/triggering
% extendability
% TODO: link naar appendix met schema
% ZeroMQ gebruiken voor communicatie tussen meerdere processen (in
% verschillende talen)
\chapter{Reference implementation}
% TODO
% alleen window.contains op point down, niet move/up
\chapter{Integration in VTK}
% VTK interactor
\chapter{Conclusions}
%\chapter{Conclusions}
% TODO
% Windows zijn een manier om globale events toe te wijzen aan vensters
......@@ -286,27 +486,17 @@ events.
\chapter{Suggestions for future work}
% TODO
% Network protocol (ZeroMQ)
% State machine
\bibliographystyle{plain}
\bibliography{report}{}
% Network protocol (ZeroMQ) voor meerdere talen en simultane processen
% Hierij ook: extra laag die gesture windows aanmaakt die corresponderen met window manager
\appendix
\chapter{Diagram of mechanism structure}
\label{app:schema}
% State machine
\begin{figure}[H]
\hspace{-14em}
\includegraphics{data/server_scheme.pdf}
\caption{}
%TODO: caption
\end{figure}
% Window in boomstructuur voor efficientie
\chapter{Supported events in reference implementation}
\label{app:supported-events}
\bibliographystyle{plain}
\bibliography{report}{}
% TODO
%\appendix
\end{document}
docs/rules.mk
View file @ 7c17c783
REPORT := report
SCHEME := data/server_scheme
DIAGRAM := data/diagram
PDFLATEX_FLAGS = -halt-on-error -interaction=nonstopmode \
-output-directory $(@D) -shell-escape
LATEX = TEXINPUTS=$(d):$(b): pdflatex $(PDFLATEX_FLAGS)
......@@ -19,11 +19,7 @@ $(b)%.pdf: $(d)%.tex
$(LATEX) $^; \
done
$(b)$(REPORT).pdf: $(b)$(SCHEME).pdf
$(b)$(SCHEME).pdf: $(d)$(SCHEME).tex
mkdir -p $(@D)
pdflatex $(PDFLATEX_FLAGS) $^
$(b)$(REPORT).pdf: $(d)$(DIAGRAM).tex
$(b)$(REPORT).bbl: $(d)$(REPORT).bib
BIBINPUTS=$(d) bibtex8 ${@:.bbl=.aux}
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment