Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
M
multitouch
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Taddeüs Kroes
multitouch
Commits
294c0773
Commit
294c0773
authored
Jun 19, 2012
by
Taddeüs Kroes
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Some final tweaks to 'Design' chapter.
parent
89d950e4
Changes
2
Hide whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
78 additions
and
69 deletions
+78
-69
docs/data/diagrams.tex
docs/data/diagrams.tex
+1
-1
docs/report.tex
docs/report.tex
+77
-68
No files found.
docs/data/diagrams.tex
View file @
294c0773
...
@@ -140,7 +140,7 @@
...
@@ -140,7 +140,7 @@
}
}
\def\trackerdiagram
{
\def\trackerdiagram
{
\begin{figure}
[h]
\begin{figure}
[h
!
]
\center
\center
\architecture
{
\architecture
{
\node
[block, below of=driver]
(eventdriver)
{
Event driver
}
\node
[block, below of=driver]
(eventdriver)
{
Event driver
}
...
...
docs/report.tex
View file @
294c0773
...
@@ -18,20 +18,19 @@
...
@@ -18,20 +18,19 @@
% Title page
% Title page
\maketitle
\maketitle
\begin{abstract}
\begin{abstract}
Applications that use
Applications that use complex gesture-based interaction need to translate
complex gesture-based interaction need to translate primitive messages from
primitive messages from low-level device drivers to complex, high-level
low-level device drivers to complex, high-level gestures, and map these
gestures, and map these gestures to elements in an application. This report
gestures to elements in an application. This report presents a generic
presents a generic architecture for the detection of complex gestures in an
architecture for the detection of complex gestures in an application. The
application. The architecture translates device driver messages to a common
architecture translates device driver messages to a common set of
set of ``events''. The events are then delegated to a tree of ``areas'',
``events''. The events are then delegated to a tree of ``areas'', which are
which are used to separate groups of events and assign these groups to an
used to separate groups of events and assign these groups to an element in
element in the application. Gesture detection is performed on a group of
the application. Gesture detection is performed on a group of events
events assigned to an area, using detection units called ``gesture
assigned to an area, using detection units called ``gesture tackers''. An
tackers''. An implementation of the architecture as a daemon process would
implementation of the architecture as a daemon process would be capable of
be capable of serving gestures to multiple applications at the same time. A
serving gestures to multiple applications at the same time. A reference
reference implementation and two test case applications have been created
implementation and two test case applications have been created to test the
to test the effectiveness of the architecture design.
effectiveness of the architecture design.
\end{abstract}
\end{abstract}
% Set paragraph indentation
% Set paragraph indentation
...
@@ -243,26 +242,25 @@ goal is to test the effectiveness of the design and detect its shortcomings.
...
@@ -243,26 +242,25 @@ goal is to test the effectiveness of the design and detect its shortcomings.
% TODO: in introduction: gestures zijn opgebouwd uit meerdere primitieven
% TODO: in introduction: gestures zijn opgebouwd uit meerdere primitieven
Touch input devices are unaware of the graphical input
Touch input devices are unaware of the graphical input
widgets
\footnote
{
``Widget'' is a name commonly used to identify an element
widgets
\footnote
{
``Widget'' is a name commonly used to identify an element
of a graphical user interface (GUI).
}
of an application, and therefore
of a graphical user interface (GUI).
}
rendered by an application, and
generate events that simply identify the screen location at which an event
therefore generate events that simply identify the screen location at which
takes place. User interfaces of applications that do not run in full scree
n
an event takes place. User interfaces of applications that do not run i
n
modus are contained in a window. Events which occur outside the application
full screen modus are contained in a window. Events which occur outside the
window should not be handled by the program in most cases. What's more,
application window should not be handled by the program in most cases.
widget within the application window itself should be able to respond
to
What's more, widget within the application window itself should be able
to
different gestures. E.g. a button widget may respond to a ``tap'' gesture
respond to different gestures. E.g. a button widget may respond to a
to be activated, whereas the application window responds to a ``pinch''
``tap'' gesture to be activated, whereas the application window responds to
gesture to be resized. In order to be able to direct a gesture to a
a ``pinch'' gesture to be resized. In order to be able to direct a gesture
particular widget in an application, a gesture must be restricted to the
to a particular widget in an application, a gesture must be restricted to
area of the screen covered by that widget. An important question is if the
the area of the screen covered by that widget. An important question is if
architecture should offer a solution to this problem, or leave the task of
the architecture should offer a solution to this problem, or leave the task
assigning gestures to application widgets to the application developer.
of
assigning gestures to application widgets to the application developer.
If the architecture does not provide a solution, the ``Event analysis''
If the architecture does not provide a solution, the ``Event analysis''
component in figure
\ref
{
fig:multipledrivers
}
receives all events that
component in figure
\ref
{
fig:multipledrivers
}
receives all events that
occur on the screen surface. The gesture detection logic thus uses all
occur on the screen surface. The gesture detection logic thus uses all
events as input to detect a gesture. This leaves no possibility for a
events as input to detect a gesture. This leaves no possibility for a
gesture to occur at multiple screen positions at the same time, unless the
gesture to occur at multiple screen positions at the same time. The problem
gesture detection logic incorporates event cluster detection. The problem
is illustrated in figure
\ref
{
fig:ex1
}
, where two widgets on the screen can
is illustrated in figure
\ref
{
fig:ex1
}
, where two widgets on the screen can
be rotated independently. The rotation detection component that detects
be rotated independently. The rotation detection component that detects
rotation gestures receives all four fingers as input. If the two groups of
rotation gestures receives all four fingers as input. If the two groups of
...
@@ -274,11 +272,12 @@ goal is to test the effectiveness of the design and detect its shortcomings.
...
@@ -274,11 +272,12 @@ goal is to test the effectiveness of the design and detect its shortcomings.
A gesture detection component could perform a heuristic way of cluster
A gesture detection component could perform a heuristic way of cluster
detection based on the distance between events. However, this method cannot
detection based on the distance between events. However, this method cannot
guarantee that a cluster of events corresponds with a particular
guarantee that a cluster of events corresponds with a particular
application widget. In short, gesture detection is difficult to implement
application widget. In short, a gesture detection component is difficult to
without awareness of the location of application widgets. Moreover, the
implement without awareness of the location of application widgets.
application developer still needs to direct gestures to a particular widget
Secondly, the application developer still needs to direct gestures to a
manually. This requires geometric calculations in the application logic,
particular widget manually. This requires geometric calculations in the
which is a tedious and error-prone task for the developer.
application logic, which is a tedious and error-prone task for the
developer.
A better solution is to group events that occur inside the area covered by
A better solution is to group events that occur inside the area covered by
a widget, before passing them on to a gesture detection component.
a widget, before passing them on to a gesture detection component.
...
@@ -301,9 +300,9 @@ goal is to test the effectiveness of the design and detect its shortcomings.
...
@@ -301,9 +300,9 @@ goal is to test the effectiveness of the design and detect its shortcomings.
application is a ``callback'' mechanism: the application developer binds a
application is a ``callback'' mechanism: the application developer binds a
function to an event, that is called when the event occurs. Because of the
function to an event, that is called when the event occurs. Because of the
familiarity of this concept with developers, the architecture uses a
familiarity of this concept with developers, the architecture uses a
callback mechanism to handle gestures in an application.
Since an area
callback mechanism to handle gestures in an application.
Callback handlers
controls the grouping of events and thus the occurrence of gestures in an
are bound to event areas, since events areas controls the grouping of
area, gesture handlers for a specific gesture type are bound to an area
.
events and thus the occurrence of gestures in an area of the screen
.
Figure
\ref
{
fig:areadiagram
}
shows the position of areas in the
Figure
\ref
{
fig:areadiagram
}
shows the position of areas in the
architecture.
architecture.
...
@@ -324,13 +323,16 @@ goal is to test the effectiveness of the design and detect its shortcomings.
...
@@ -324,13 +323,16 @@ goal is to test the effectiveness of the design and detect its shortcomings.
complex touch objects can have additional parameters, such as rotational
complex touch objects can have additional parameters, such as rotational
orientation or color. An even more generic concept is the
\emph
{
event
orientation or color. An even more generic concept is the
\emph
{
event
filter
}
, which detects whether an event should be assigned to a particular
filter
}
, which detects whether an event should be assigned to a particular
gesture detection component based on all available parameters. This level of
gesture detection component based on all available parameters. This level
abstraction allows for constraints like ``Use all blue objects within a
of abstraction provides additional methods of interaction. For example, a
widget for rotation, and green objects for dragging.''. As mentioned in the
camera-based multi-touch surface could make a distinction between gestures
introduction chapter [
\ref
{
chapter:introduction
}
], the scope of this thesis
performed with a blue gloved hand, and gestures performed with a green
is limited to multi-touch surface based devices, for which the
\emph
{
event
gloved hand.
area
}
concept suffices. Section
\ref
{
sec:eventfilter
}
explores the
possibility of event areas to be replaced with event filters.
As mentioned in the introduction chapter [
\ref
{
chapter:introduction
}
], the
scope of this thesis is limited to multi-touch surface based devices, for
which the
\emph
{
event area
}
concept suffices. Section
\ref
{
sec:eventfilter
}
explores the possibility of event areas to be replaced with event filters.
\subsection
{
Area tree
}
\subsection
{
Area tree
}
\label
{
sec:tree
}
\label
{
sec:tree
}
...
@@ -362,7 +364,7 @@ goal is to test the effectiveness of the design and detect its shortcomings.
...
@@ -362,7 +364,7 @@ goal is to test the effectiveness of the design and detect its shortcomings.
\subsection
{
Event propagation
}
\subsection
{
Event propagation
}
\label
{
sec:eventpropagation
}
\label
{
sec:eventpropagation
}
A problem occurs when event areas overlap, as shown by figure
A
nother
problem occurs when event areas overlap, as shown by figure
\ref
{
fig:eventpropagation
}
. When the white square is rotated, the gray
\ref
{
fig:eventpropagation
}
. When the white square is rotated, the gray
square should keep its current orientation. This means that events that are
square should keep its current orientation. This means that events that are
used for rotation of the white square, should not be used for rotation of
used for rotation of the white square, should not be used for rotation of
...
@@ -416,14 +418,15 @@ goal is to test the effectiveness of the design and detect its shortcomings.
...
@@ -416,14 +418,15 @@ goal is to test the effectiveness of the design and detect its shortcomings.
complex gestures, like the writing of a character from the alphabet,
complex gestures, like the writing of a character from the alphabet,
require more advanced detection algorithms.
require more advanced detection algorithms.
A way to detect complex gestures based on a sequence of input features
A way to detect these complex gestures based on a sequence of input events,
is with the use of machine learning methods, such as Hidden Markov Models
is with the use of machine learning methods, such as the Hidden Markov
\footnote
{
A Hidden Markov Model (HMM) is a statistical model without a
Models
\footnote
{
A Hidden Markov Model (HMM) is a statistical model without
memory, it can be used to detect gestures based on the current input state
a memory, it can be used to detect gestures based on the current input
alone.
}
\cite
{
conf/gw/RigollKE97
}
. A sequence of input states can be mapped
state alone.
}
used for sign language detection by
to a feature vector that is recognized as a particular gesture with a
\cite
{
conf/gw/RigollKE97
}
. A sequence of input states can be mapped to a
certain probability. An advantage of using machine learning with respect to
feature vector that is recognized as a particular gesture with a certain
an imperative programming style is that complex gestures can be described
probability. An advantage of using machine learning with respect to an
imperative programming style is that complex gestures can be described
without the use of explicit detection logic. For example, the detection of
without the use of explicit detection logic. For example, the detection of
the character `A' being written on the screen is difficult to implement
the character `A' being written on the screen is difficult to implement
using an imperative programming style, while a trained machine learning
using an imperative programming style, while a trained machine learning
...
@@ -434,25 +437,31 @@ goal is to test the effectiveness of the design and detect its shortcomings.
...
@@ -434,25 +437,31 @@ goal is to test the effectiveness of the design and detect its shortcomings.
sufficient to detect many common gestures, like rotation and dragging. The
sufficient to detect many common gestures, like rotation and dragging. The
imperative programming style is also familiar and understandable for a wide
imperative programming style is also familiar and understandable for a wide
range of application developers. Therefore, the architecture should support
range of application developers. Therefore, the architecture should support
an imperative style of gesture detection.
an imperative style of gesture detection. A problem with an imperative
programming style is that the explicit detection of different gestures
A problem with the imperative programming style is that the explicit
requires different gesture detection components. If these components are
detection of different gestures requires different gesture detection
not managed well, the detection logic is prone to become chaotic and
components. If these components is not managed well, the detection logic is
over-complex.
prone to become chaotic and over-complex.
To manage complexity and support multiple styles of gesture detection
To manage complexity and support multiple methods of gesture detection, the
logic, the architecture has adopted the tracker-based design as described
architecture has adopted the tracker-based design as described by
by
\cite
{
win7touch
}
. Different detection components are wrapped in separate
\cite
{
win7touch
}
. Different detection components are wrapped in separate
gesture tracking units, or
\emph
{
gesture trackers
}
. The input of a gesture
gesture tracking units, or
\emph
{
gesture trackers
}
. The input of a gesture
tracker is provided by an event area in the form of events. When a gesture
tracker is provided by an event area in the form of events. Each gesture
tracker detects a gesture, this gesture is triggered in the corresponding
detection component is wrapped in a gesture tracker with a fixed type of
event area. The event area then calls the callbacks which are bound to the
input and output. Internally, the gesture tracker can adopt any programming
gesture type by the application. Figure
\ref
{
fig:trackerdiagram
}
shows the
style. A character recognition component can use an HMM, whereas a tap
position of gesture trackers in the architecture.
detection component defines a simple function that compares event
coordinates.
\trackerdiagram
\trackerdiagram
When a gesture tracker detects a gesture, this gesture is triggered in the
corresponding event area. The event area then calls the callbacks which are
bound to the gesture type by the application. Figure
\ref
{
fig:trackerdiagram
}
shows the position of gesture trackers in the
architecture.
The use of gesture trackers as small detection units provides extendability
The use of gesture trackers as small detection units provides extendability
of the architecture. A developer can write a custom gesture tracker and
of the architecture. A developer can write a custom gesture tracker and
register it in the architecture. The tracker can use any type of detection
register it in the architecture. The tracker can use any type of detection
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment