Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
M
multitouch
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Taddeüs Kroes
multitouch
Commits
b091c913
Commit
b091c913
authored
Jun 25, 2012
by
Taddeüs Kroes
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Corrected some typo's and moved a paragraph.
parent
be6d8ffd
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
53 additions
and
65 deletions
+53
-65
docs/data/diagrams.tex
docs/data/diagrams.tex
+1
-1
docs/report.tex
docs/report.tex
+52
-64
No files found.
docs/data/diagrams.tex
View file @
b091c913
...
@@ -103,7 +103,7 @@
...
@@ -103,7 +103,7 @@
\node
[block, below of=eventdriver, dashed]
(analysis)
{
Event analysis
}
\node
[block, below of=eventdriver, dashed]
(analysis)
{
Event analysis
}
edge[linefrom] (eventdriver)
edge[linefrom] (eventdriver)
edge[linefrom] node[right=5]
{
events
}
(secondeventdriver);
edge[linefrom] node[right=5
pt
]
{
events
}
(secondeventdriver);
\node
[block, below of=analysis]
{
Application
}
\node
[block, below of=analysis]
{
Application
}
edge[linefrom] node[right, near start]
{
gestures
}
(analysis);
edge[linefrom] node[right, near start]
{
gestures
}
(analysis);
...
...
docs/report.tex
View file @
b091c913
...
@@ -89,11 +89,7 @@ detection for every new gesture-based application.
...
@@ -89,11 +89,7 @@ detection for every new gesture-based application.
Chapter
\ref
{
chapter:related
}
describes related work that inspired a design
Chapter
\ref
{
chapter:related
}
describes related work that inspired a design
for the architecture. The design is presented in chapter
for the architecture. The design is presented in chapter
\ref
{
chapter:design
}
. Sections
\ref
{
sec:multipledrivers
}
to
\ref
{
chapter:design
}
. Chapter
\ref
{
chapter:testapps
}
presents a reference
\ref
{
sec:daemon
}
define requirements for the architecture, and introduce
architecture components that meet these requirements. Section
\ref
{
sec:example
}
then shows the use of the architecture in an example
application. Chapter
\ref
{
chapter:testapps
}
presents a reference
implementation of the architecture, an two test case applications that show
implementation of the architecture, an two test case applications that show
the practical use of its components as presented in chapter
the practical use of its components as presented in chapter
\ref
{
chapter:design
}
. Finally, some suggestions for future research on the
\ref
{
chapter:design
}
. Finally, some suggestions for future research on the
...
@@ -153,11 +149,11 @@ detection for every new gesture-based application.
...
@@ -153,11 +149,11 @@ detection for every new gesture-based application.
The alternative to machine learning is to define a predefined set of rules
The alternative to machine learning is to define a predefined set of rules
for each gesture. Manoj Kumar
\cite
{
win7touch
}
presents a Windows 7
for each gesture. Manoj Kumar
\cite
{
win7touch
}
presents a Windows 7
application, written in Microsofts .NET, which detects a set of basic
application, written in Microsofts .NET, which detects a set of basic
directional gestures based
the movement of a stylus. The complexity of the
directional gestures based
on the movement of a stylus. The complexity of
code is managed by the separation of different gesture types in different
the code is managed by the separation of different gesture types in
d
etection units called ``gesture trackers''. The application shows that
d
ifferent detection units called ``gesture trackers''. The application
predefined gesture detection rules do not necessarily produce unmanageabl
e
shows that predefined gesture detection rules do not necessarily produc
e
code.
unmanageable
code.
\section
{
Analysis of related work
}
\section
{
Analysis of related work
}
...
@@ -256,7 +252,7 @@ detection for every new gesture-based application.
...
@@ -256,7 +252,7 @@ detection for every new gesture-based application.
used.
used.
Because driver implementations have a common output format in the form of
Because driver implementations have a common output format in the form of
events, multiple event drivers can
run
at the same time (see figure
events, multiple event drivers can
be used
at the same time (see figure
\ref
{
fig:multipledrivers
}
). This design feature allows low-level events
\ref
{
fig:multipledrivers
}
). This design feature allows low-level events
from multiple devices to be aggregated into high-level gestures.
from multiple devices to be aggregated into high-level gestures.
...
@@ -272,8 +268,8 @@ detection for every new gesture-based application.
...
@@ -272,8 +268,8 @@ detection for every new gesture-based application.
an event takes place. User interfaces of applications that do not run in
an event takes place. User interfaces of applications that do not run in
full screen modus are contained in a window. Events which occur outside the
full screen modus are contained in a window. Events which occur outside the
application window should not be handled by the application in most cases.
application window should not be handled by the application in most cases.
What's more,
widget within the application window itself should be able to
What's more,
a widget within the application window itself should be able
respond to different gestures. E.g. a button widget may respond to a
to
respond to different gestures. E.g. a button widget may respond to a
``tap'' gesture to be activated, whereas the application window responds to
``tap'' gesture to be activated, whereas the application window responds to
a ``pinch'' gesture to be resized. In order to be able to direct a gesture
a ``pinch'' gesture to be resized. In order to be able to direct a gesture
to a particular widget in an application, a gesture must be restricted to
to a particular widget in an application, a gesture must be restricted to
...
@@ -304,9 +300,9 @@ detection for every new gesture-based application.
...
@@ -304,9 +300,9 @@ detection for every new gesture-based application.
application logic, which is a tedious and error-prone task for the
application logic, which is a tedious and error-prone task for the
developer.
developer.
A better solution is to group events that occur inside the area covered by
The architecture described here groups events that occur inside the area
a widget, before passing them on to a gesture detection component.
covered by a widget, before passing them on to a gesture detection
Different gesture detection components can then detect gestures
component.
Different gesture detection components can then detect gestures
simultaneously, based on different sets of input events. An area of the
simultaneously, based on different sets of input events. An area of the
screen surface will be represented by an
\emph
{
event area
}
. An event area
screen surface will be represented by an
\emph
{
event area
}
. An event area
filters input events based on their location, and then delegates events to
filters input events based on their location, and then delegates events to
...
@@ -318,7 +314,7 @@ detection for every new gesture-based application.
...
@@ -318,7 +314,7 @@ detection for every new gesture-based application.
represented by two event areas, each having a different rotation detection
represented by two event areas, each having a different rotation detection
component.
component.
\subsection
*
{
Callback mechanism
}
\subsection
{
Callback mechanism
}
When a gesture is detected by a gesture detection component, it must be
When a gesture is detected by a gesture detection component, it must be
handled by the client application. A common way to handle events in an
handled by the client application. A common way to handle events in an
...
@@ -333,38 +329,12 @@ detection for every new gesture-based application.
...
@@ -333,38 +329,12 @@ detection for every new gesture-based application.
\areadiagram
\areadiagram
%Note that the boundaries of an area are only used to group events, not
%gestures. A gesture could occur outside the area that contains its
%originating events, as illustrated by the example in figure \ref{fig:ex2}.
%\examplefiguretwo
A remark must be made about the use of event areas to assign events to the
detection of some gesture. The concept of an event area is based on the
assumption that the set or originating events that form a particular
gesture, can be determined based exclusively on the location of the events.
This is a reasonable assumption for simple touch objects whose only
parameter is a position, such as a pen or a human finger. However, more
complex touch objects can have additional parameters, such as rotational
orientation or color. An even more generic concept is the
\emph
{
event
filter
}
, which detects whether an event should be assigned to a particular
gesture detection component based on all available parameters. This level
of abstraction provides additional methods of interaction. For example, a
camera-based multi-touch surface could make a distinction between gestures
performed with a blue gloved hand, and gestures performed with a green
gloved hand.
As mentioned in the introduction chapter [
\ref
{
chapter:introduction
}
], the
scope of this thesis is limited to multi-touch surface based devices, for
which the
\emph
{
event area
}
concept suffices. Section
\ref
{
sec:eventfilter
}
explores the possibility of event areas to be replaced with event filters.
\subsection
{
Area tree
}
\subsection
{
Area tree
}
\label
{
sec:tree
}
\label
{
sec:tree
}
The most simple usage of event areas in the architecture would be a list of
A basic usage of event areas in the architecture would be a list of event
event areas. When the event driver delegates an event, it is accepted by
areas. When the event driver delegates an event, it is accepted by each
e
ach e
vent area that contains the event coordinates.
event area that contains the event coordinates.
If the architecture were to be used in combination with an application
If the architecture were to be used in combination with an application
framework like GTK
\cite
{
GTK
}
, each GTK widget that responds to gestures
framework like GTK
\cite
{
GTK
}
, each GTK widget that responds to gestures
...
@@ -375,15 +345,15 @@ detection for every new gesture-based application.
...
@@ -375,15 +345,15 @@ detection for every new gesture-based application.
too.
too.
This process is simplified by the arrangement of event areas in a tree
This process is simplified by the arrangement of event areas in a tree
structure.
A root event area represents the panel, containing five other
structure. A root event area represents the panel, containing five other
event areas which are positioned relative to the root area. The relative
event areas which are positioned relative to the root area. The relative
positions do not need to be updated when the panel area changes its
positions do not need to be updated when the panel area changes its
position. GUI frameworks
, like GTK, use this kind of tree structure to
position. GUI frameworks
use this kind of tree structure to manage
manage
graphical widgets.
graphical widgets.
If the GUI toolkit provides an API for requesting the position and size of
If the GUI toolkit provides an API for requesting the position and size of
a widget, a recommended first step when developing an application is to
a widget, a recommended first step when developing an application is to
create
some
subclass of the area that automatically synchronizes with the
create
a
subclass of the area that automatically synchronizes with the
position of a widget from the GUI framework.
position of a widget from the GUI framework.
\subsection
{
Event propagation
}
\subsection
{
Event propagation
}
...
@@ -430,10 +400,28 @@ detection for every new gesture-based application.
...
@@ -430,10 +400,28 @@ detection for every new gesture-based application.
When regular propagation is stopped, the event is propagated to other
When regular propagation is stopped, the event is propagated to other
gesture detection components first, before actually being stopped.
gesture detection components first, before actually being stopped.
\eventpropagationfigure
\newpage
\newpage
\eventpropagationfigure
\section
{
Detecting gestures from events
}
The concept of an event area is based on the assumption that the set of
originating events that form a particular gesture, can be determined based
exclusively on the location of the events. This is a reasonable assumption
for simple touch objects whose only parameter is a position, such as a pen
or a human finger. However, more complex touch objects can have additional
parameters, such as rotational orientation or color. An even more generic
concept is the
\emph
{
event filter
}
, which detects whether an event should
be assigned to a particular gesture detection component based on all
available parameters. This level of abstraction provides additional methods
of interaction. For example, a camera-based multi-touch surface could make
a distinction between gestures performed with a blue gloved hand, and
gestures performed with a green gloved hand.
As mentioned in the introduction chapter [
\ref
{
chapter:introduction
}
], the
scope of this thesis is limited to multi-touch surface based devices, for
which the
\emph
{
event area
}
concept suffices. Section
\ref
{
sec:eventfilter
}
explores the possibility of event areas to be replaced with event filters.
\section
{
Detecting gestures from low-level events
}
\label
{
sec:gesture-detection
}
\label
{
sec:gesture-detection
}
The low-level events that are grouped by an event area must be translated
The low-level events that are grouped by an event area must be translated
...
@@ -555,14 +543,14 @@ start the GUI main loop in the current thread
...
@@ -555,14 +543,14 @@ start the GUI main loop in the current thread
\examplediagram
\examplediagram
\chapter
{
T
est applications
}
\chapter
{
Implementation and t
est applications
}
\label
{
chapter:testapps
}
\label
{
chapter:testapps
}
A reference implementation of the design has been written in Python. Two test
A reference implementation of the design has been written in Python. Two test
applications have been created to test if the design ``works'' in a practical
applications have been created to test if the design ``works'' in a practical
application, and to detect its flaws. One application is mainly used to test
application, and to detect its flaws. One application is mainly used to test
the gesture tracker implementations. The other application uses multiple event
the gesture tracker implementations. The other application uses multiple event
areas in a tree structure, demonstrating event delegation and propagation. T
eh
areas in a tree structure, demonstrating event delegation and propagation. T
he
second application also defines a custom gesture tracker.
second application also defines a custom gesture tracker.
To test multi-touch interaction properly, a multi-touch device is required. The
To test multi-touch interaction properly, a multi-touch device is required. The
...
@@ -710,9 +698,9 @@ first.
...
@@ -710,9 +698,9 @@ first.
\testappdiagram
\testappdiagram
%\section{Discussion
}
\section
{
Results
}
%
%
\emph{TODO: Tekortkomingen aangeven die naar voren komen uit de tests}
\emph
{
TODO: Tekortkomingen aangeven die naar voren komen uit de tests
}
% Verschillende apparaten/drivers geven een ander soort primitieve events af.
% Verschillende apparaten/drivers geven een ander soort primitieve events af.
% Een vertaling van deze device-specifieke events naar een algemeen formaat van
% Een vertaling van deze device-specifieke events naar een algemeen formaat van
...
@@ -737,12 +725,12 @@ first.
...
@@ -737,12 +725,12 @@ first.
\label
{
sec:eventfilter
}
\label
{
sec:eventfilter
}
As mentioned in section
\ref
{
sec:areas
}
, the concept of an event area is based
As mentioned in section
\ref
{
sec:areas
}
, the concept of an event area is based
on the assumption that the set o
r
originating events that form a particular
on the assumption that the set o
f
originating events that form a particular
gesture, can be determined based exclusively on the location of the events.
gesture, can be determined based exclusively on the location of the events.
Since this thesis focuses on multi-touch surface based devices, and every
Since this thesis focuses on multi-touch surface based devices, and every
object on a multi-touch surface has a position, this assumption is valid.
object on a multi-touch surface has a position, this assumption is valid.
However, the design of the architecture is meant to be more generic; to provide
However, the design of the architecture is meant to be more generic; to provide
a structured design
of
managing gesture detection.
a structured design
for
managing gesture detection.
An in-air gesture detection device, such as the Microsoft Kinect
\cite
{
kinect
}
,
An in-air gesture detection device, such as the Microsoft Kinect
\cite
{
kinect
}
,
provides 3D positions. Some multi-touch tables work with a camera that can also
provides 3D positions. Some multi-touch tables work with a camera that can also
...
@@ -770,8 +758,8 @@ whether multi-touch gestures can be described in a formal way so that explicit
...
@@ -770,8 +758,8 @@ whether multi-touch gestures can be described in a formal way so that explicit
detection code can be avoided.
detection code can be avoided.
\cite
{
GART
}
and
\cite
{
conf/gw/RigollKE97
}
propose the use of machine learning
\cite
{
GART
}
and
\cite
{
conf/gw/RigollKE97
}
propose the use of machine learning
to recognize
s gestures. To use machine learning, a set of input events forming
to recognize
gestures. To use machine learning, a set of input events forming a
a
particular gesture must be represented as a feature vector. A learning set
particular gesture must be represented as a feature vector. A learning set
containing a set of feature vectors that represent some gesture ``teaches'' the
containing a set of feature vectors that represent some gesture ``teaches'' the
machine what the feature of the gesture looks like.
machine what the feature of the gesture looks like.
...
@@ -810,8 +798,8 @@ subject well worth exploring in the future.
...
@@ -810,8 +798,8 @@ subject well worth exploring in the future.
\section
{
Daemon implementation
}
\section
{
Daemon implementation
}
Section
\ref
{
sec:daemon
}
proposes the us
age of a network protocol to
Section
\ref
{
sec:daemon
}
proposes the us
e of a network protocol to communicate
communicate
between an architecture implementation and (multiple) gesture-based
between an architecture implementation and (multiple) gesture-based
applications, as illustrated in figure
\ref
{
fig:daemon
}
. The reference
applications, as illustrated in figure
\ref
{
fig:daemon
}
. The reference
implementation does not support network communication. If the architecture
implementation does not support network communication. If the architecture
design is to become successful in the future, the implementation of network
design is to become successful in the future, the implementation of network
...
@@ -823,7 +811,7 @@ the basis for its communication layer.
...
@@ -823,7 +811,7 @@ the basis for its communication layer.
If an implementation of the architecture will be released, a good idea would be
If an implementation of the architecture will be released, a good idea would be
to do so within a community of application developers. A community can
to do so within a community of application developers. A community can
contribute to a central database of gesture trackers, making the interaction
contribute to a central database of gesture trackers, making the interaction
from their applications available for use other applications.
from their applications available for use
in
other applications.
Ideally, a user can install a daemon process containing the architecture so
Ideally, a user can install a daemon process containing the architecture so
that it is usable for any gesture-based application on the device. Applications
that it is usable for any gesture-based application on the device. Applications
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment