Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
M
multitouch
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Taddeüs Kroes
multitouch
Commits
5252a330
Commit
5252a330
authored
Jun 20, 2012
by
Taddeüs Kroes
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Tried to bring more structure into the Related Work section.
parent
5b2f6525
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
130 additions
and
75 deletions
+130
-75
docs/report.bib
docs/report.bib
+45
-0
docs/report.tex
docs/report.tex
+85
-75
No files found.
docs/report.bib
View file @
5252a330
...
@@ -185,3 +185,48 @@
...
@@ -185,3 +185,48 @@
year = "2002"
year = "2002"
}
}
@misc{kivy,
author = "{Mathieu Virbel}, Thomas Hansen Christopher Denter Gabriel Pettier Akshay Arora",
howpublished = "http://kivy.org/",
title = "{Kivy}",
year = "2011"
}
@inproceedings{VRPN,
abstract = "dblp",
added-at = "2002-06-07T00:00:00.000+0200",
author = "Taylor, Russell M. and Hudson, Thomas C. and Seeger, Adam and Weber, Hans and Juliano, Jeffrey and Helser, Aron T.",
biburl = "http://www.bibsonomy.org/bibtex/23a77e2b392bb9adfce8b0334400bb8da/dblp",
booktitle = "{VRST}",
date = "2002-06-07",
ee = "http://doi.acm.org/10.1145/505008.505019",
interhash = "f5bb371469d96d00e725eb14bede5662",
intrahash = "3a77e2b392bb9adfce8b0334400bb8da",
keywords = "dblp",
pages = "55--61",
title = "{VRPN: a device-independent, network-transparent VR peripheral system.}",
url = "http://dblp.uni-trier.de/db/conf/vrst/vrst2001.html#TaylorHSWJH01",
x-fetchedfrom = "Bibsonomy",
year = 2001
}
@unpublished{kivygesture,
author = "organization, kivy",
note = "\url{http://kivy.org/docs/api-kivy.gesture.html}",
title = "{Gesture recognition in kivy}"
}
@manual{OpenNI2010,
added-at = "2011-06-10T23:03:19.000+0200",
author = "organization, OpenNI",
interhash = "23b4f1fbccf97a3523078da0d5c924d1",
intrahash = "d7953305373f5ce2ec6ab43e80306fdc",
keywords = "kinect",
month = "November",
organization = "OpenNI organization",
title = "{OpenNI User Guide}",
url = "http://www.openni.org/documentation; http://www.bibsonomy.org/bibtex/2d7953305373f5ce2ec6ab43e80306fdc/lightraven",
x-fetchedfrom = "Bibsonomy",
year = 2010
}
docs/report.tex
View file @
5252a330
...
@@ -84,7 +84,7 @@ detection for every new gesture-based application.
...
@@ -84,7 +84,7 @@ detection for every new gesture-based application.
multi-touch surface devices. It presents a design for a generic gesture
multi-touch surface devices. It presents a design for a generic gesture
detection architecture for use in multi-touch based applications. A
detection architecture for use in multi-touch based applications. A
reference implementation of this design is used in some test case
reference implementation of this design is used in some test case
applications, whose
goal
is to test the effectiveness of the design and
applications, whose
purpose
is to test the effectiveness of the design and
detect its shortcomings.
detect its shortcomings.
Chapter
\ref
{
chapter:related
}
describes related work that inspired a design
Chapter
\ref
{
chapter:related
}
describes related work that inspired a design
...
@@ -102,72 +102,82 @@ detection for every new gesture-based application.
...
@@ -102,72 +102,82 @@ detection for every new gesture-based application.
\chapter
{
Related work
}
\chapter
{
Related work
}
\label
{
chapter:related
}
\label
{
chapter:related
}
% TODO: herstructureren
Applications that use gesture-based interaction need a graphical user
interface (GUI) on which gestures can be performed. The creation of a GUI
\section
{
Existing application frameworks
}
is a platform-specific task. For instance, Windows and Linux support
different window managers. To create a window in a platform-independent
Application frameworks for surface-touch devices, such as Nokia's Qt
application, the application would need to include separate functionalities
\cite
{
qt
}
, do already include the detection of commonly used gestures like
for supported platforms. For this reason, GUI-based applications are often
\emph
{
pinch
}
gestures. However, this detection logic is dependent on the
built on top of an application framework that abstracts platform-specific
application framework. Consequently, an application developer who wants to
tasks. Frameworks often include a set of tools and events that help the
use multi-touch interaction in an application is forced to use an
developer to easily build advanced GUI widgets.
application framework that includes support for multi-touch gestures.
Moreover, the set of supported gestures is limited by the application
% Existing frameworks (and why they're not good enough)
framework of choice. To incorporate a custom event in an application, the
Some frameworks, such as Nokia's Qt
\cite
{
qt
}
, provide support for basic
application developer needs to extend the framework. This requires
multi-touch gestures like tapping, rotation or pinching. However, the
extensive knowledge of the framework's architecture. Also, if the same
detection of gestures is embedded in the framework code in an inseparable
gesture is needed in another application that is based on another
way. Consequently, an application developer who wants to use multi-touch
framework, the detection logic has to be translated for use in that
interaction in an application, is forced to use an application framework
framework.
that includes support for those multi-touch gestures that are required by
the application. Kivy
\cite
{
kivy
}
is a GUI framework for Python
\section
{
Gesture and Activity Recognition Toolkit
}
applications, with support for multi-touch gestures. It uses a basic
gesture detection algorithm that allows developers to define custom
The Gesture and Activity Recognition Toolkit (GART)
\cite
{
GART
}
is a
gestures to some degree
\cite
{
kivygesture
}
using a set of touch point
toolkit for the development of gesture-based applications. The toolkit
coordinates. However, these frameworks do not provide support for extension
states that the best way to classify gestures is to use machine learning.
with custom complex gestures.
The programmer trains a program to recognize using the machine learning
library from the toolkit. The toolkit contains a callback mechanism that
Many frameworks are also device-specific, meaning that they are developed
the programmer uses to execute custom code when a gesture is recognized.
for use on either a tablet, smartphone, PC or other device. OpenNI
\cite
{
OpenNI2010
}
, for example, provides API's for only natural interaction
Though multi-touch input is not directly supported by the toolkit, the
(NI) devices such as webcams and microphones. The concept of complex
level of abstraction does allow for it to be implemented in the form of a
gesture-based interaction, however, is applicable to a much wider set of
``touch'' sensor.
devices. VRPN
\cite
{
VRPN
}
provides a software library that abstracts the
output of devices, which enables it to support a wide set of devices used
The reason to use machine learning is the statement that gesture detection
in Virtual Reality (VR) interaction. The framework makes the low-level
``is likely to become increasingly complex and unmanageable'' when using a
events of these devices accessible in a client application using network
set of predefined rules to detect whether some sensor input can be seen as
communication. Gesture detection is not included in VRPN.
a specific gesture. This statement is not necessarily true. If the
programmer is given a way to separate the detection of different types of
% Methods of gesture detection
gestures and flexibility in rule definitions, over-complexity can be
The detection of high-level gestures from low-level events can be
avoided.
approached in several ways. GART
\cite
{
GART
}
is a toolkit for the
development of gesture-based applications, which states that the best way
\section
{
Gesture recognition implementation for Windows 7
}
to classify gestures is to use machine learning. The programmer trains an
application to recognize gestures using a machine learning library from the
The online article
\cite
{
win7touch
}
presents a Windows 7 application,
toolkit. Though multi-touch input is not directly supported by the toolkit,
written in Microsofts .NET. The application shows detected gestures in a
the level of abstraction does allow for it to be implemented in the form of
canvas. Gesture trackers keep track of stylus locations to detect specific
a ``touch'' sensor. The reason to use machine learning is that gesture
gestures. The event types required to track a touch stylus are ``stylus
detection ``is likely to become increasingly complex and unmanageable''
down'', ``stylus move'' and ``stylus up'' events. A
when using a predefined set of rules to detect whether some sensor input
\texttt
{
GestureTrackerManager
}
object dispatches these events to gesture
can be classified as a specific gesture.
trackers. The application supports a limited number of pre-defined
gestures.
The alternative to machine learning is to define a predefined set of rules
for each gesture. Manoj Kumar
\cite
{
win7touch
}
presents a Windows 7
An important observation in this application is that different gestures are
application, written in Microsofts .NET, which detects a set of basic
detected by different gesture trackers, thus separating gesture detection
directional gestures based the movement of a stylus. The complexity of the
code into maintainable parts.
code is managed by the separation of different gesture types in different
detection units called ``gesture trackers''. The application shows that
predefined gesture detection rules do not necessarily produce unmanageable
code.
\section
{
Analysis of related work
}
\section
{
Analysis of related work
}
The simple Processing implementation of multi-touch events provides most of
Implementations for the support of complex gesture based interaction do
the functionality that can be found in existing multi-touch applications.
already exist. However, gesture detection in these implementations is
In fact, many applications for mobile phones and tablets only use tap and
device-specific (Nokia Qt and OpenNI) or limited to use within an
scroll events. For this category of applications, using machine learning
application framework (Kivy).
seems excessive. Though the representation of a gesture using a feature
vector in a machine learning algorithm is a generic and formal way to
An abstraction of device output allows VRPN and GART to support multiple
define a gesture, a programmer-friendly architecture should also support
devices. However, VRPN does not incorporate gesture detection. GART does,
simple, ``hard-coded'' detection code. A way to separate different pieces
but only in the form of machine learning algorithms. Many applications for
of gesture detection code, thus keeping a code library manageable and
mobile phones and tablets only use simple gestures such as taps. For this
extendable, is to user different gesture trackers.
category of applications, machine learning is an excessively complex method
of gesture detection. Manoj Kumar shows that when managed well, a
predefined set of gesture detection rules is sufficient to detect simple
gestures.
This thesis explores the possibility to create an architecture that
combines support for multiple input devices with different methods of
gesture detection.
\chapter
{
Design
}
\chapter
{
Design
}
\label
{
chapter:design
}
\label
{
chapter:design
}
...
@@ -180,17 +190,17 @@ detection for every new gesture-based application.
...
@@ -180,17 +190,17 @@ detection for every new gesture-based application.
Application frameworks are a necessity when it comes to fast,
Application frameworks are a necessity when it comes to fast,
cross-platform development. A generic architecture design should aim to be
cross-platform development. A generic architecture design should aim to be
compatible with existing frameworks, and provide a way to detect and extend
compatible with existing frameworks, and provide a way to detect and extend
gestures independent of the framework.
An application framework is written
gestures independent of the framework.
Since an application framework is
in a specific programming language. To support multiple frameworks and
written in a specific programming language, the architecture should be
programming languages, the architecture should be accessible for
accessible for applications using a language-independent method of
applications using a language-independent method of communication. This
communication. This intention leads towards the concept of a dedicated
intention leads towards the concept of a dedicated gesture detection
gesture detection application that serves gestures to multiple applications
a
pplication that serves gestures to multiple applications a
t the same time.
at the same time.
This chapter describes a design for such an architecture. The architecture
This chapter describes a design for such an architecture. The architecture
is represented as diagram of relations between different components.
is represented as diagram of relations between different components.
Sections
\ref
{
sec:multipledrivers
}
to
\ref
{
sec:daemon
}
define requirements
Sections
\ref
{
sec:multipledrivers
}
to
\ref
{
sec:daemon
}
define requirements
for the architecture, and extend th
e
diagram with components that meet
for the architecture, and extend th
is
diagram with components that meet
these requirements. Section
\ref
{
sec:example
}
describes an example usage of
these requirements. Section
\ref
{
sec:example
}
describes an example usage of
the architecture in an application.
the architecture in an application.
...
@@ -255,7 +265,6 @@ detection for every new gesture-based application.
...
@@ -255,7 +265,6 @@ detection for every new gesture-based application.
\section
{
Restricting events to a screen area
}
\section
{
Restricting events to a screen area
}
\label
{
sec:areas
}
\label
{
sec:areas
}
% TODO: in introduction: gestures zijn opgebouwd uit meerdere primitieven
Touch input devices are unaware of the graphical input
Touch input devices are unaware of the graphical input
widgets
\footnote
{
``Widget'' is a name commonly used to identify an element
widgets
\footnote
{
``Widget'' is a name commonly used to identify an element
of a graphical user interface (GUI).
}
rendered by an application, and
of a graphical user interface (GUI).
}
rendered by an application, and
...
@@ -581,18 +590,20 @@ The reference implementation is written in Python and available at
...
@@ -581,18 +590,20 @@ The reference implementation is written in Python and available at
\item
Basic tracker, supports
$
point
\_
down,~point
\_
move,~point
\_
up
$
gestures.
\item
Basic tracker, supports
$
point
\_
down,~point
\_
move,~point
\_
up
$
gestures.
\item
Tap tracker, supports
$
tap,~single
\_
tap,~double
\_
tap
$
gestures.
\item
Tap tracker, supports
$
tap,~single
\_
tap,~double
\_
tap
$
gestures.
\item
Transformation tracker, supports
$
rotate,~pinch,~drag
$
gestures.
\item
Transformation tracker, supports
$
rotate,~pinch,~drag
$
gestures.
\item
Hand tracker, supports
$
hand
\_
down,~hand
\_
up
$
gestures.
\end{itemize}
\end{itemize}
\textbf
{
Event areas
}
\textbf
{
Event areas
}
\begin{itemize}
\begin{itemize}
\item
Circular area
\item
Circular area
\item
Rectangular area
\item
Rectangular area
\item
Polygon area
\item
Full screen area
\item
Full screen area
\end{itemize}
\end{itemize}
The implementation does not include a network protocol to support the daemon
The implementation does not include a network protocol to support the daemon
setup as described in section
\ref
{
sec:daemon
}
. Therefore, it is only usable in
setup as described in section
\ref
{
sec:daemon
}
. Therefore, it is only usable in
Python programs. Th
us, th
e two test programs are also written in Python.
Python programs. The two test programs are also written in Python.
The event area implementations contain some geometric functions to determine
The event area implementations contain some geometric functions to determine
whether an event should be delegated to an event area. All gesture trackers
whether an event should be delegated to an event area. All gesture trackers
...
@@ -819,8 +830,7 @@ complex objects such as fiducials, arguments like rotational position and
...
@@ -819,8 +830,7 @@ complex objects such as fiducials, arguments like rotational position and
acceleration are also included.
acceleration are also included.
ALIVE and SET messages can be combined to create ``point down'', ``point move''
ALIVE and SET messages can be combined to create ``point down'', ``point move''
and ``point up'' events (as used by the Windows 7 implementation
and ``point up'' events.
\cite
{
win7touch
}
).
TUIO coordinates range from
$
0
.
0
$
to
$
1
.
0
$
, with
$
(
0
.
0
,
0
.
0
)
$
being the left
TUIO coordinates range from
$
0
.
0
$
to
$
1
.
0
$
, with
$
(
0
.
0
,
0
.
0
)
$
being the left
top corner of the screen and
$
(
1
.
0
,
1
.
0
)
$
the right bottom corner. To focus
top corner of the screen and
$
(
1
.
0
,
1
.
0
)
$
the right bottom corner. To focus
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment