Skip to content
Projects
Groups
Snippets
Help
Loading...
Help
Support
Keyboard shortcuts
?
Submit feedback
Contribute to GitLab
Sign in
Toggle navigation
M
multitouch
Project overview
Project overview
Details
Activity
Releases
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Issues
0
Issues
0
List
Boards
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Analytics
Analytics
CI / CD
Repository
Value Stream
Wiki
Wiki
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
Taddeüs Kroes
multitouch
Commits
3b799eb0
Commit
3b799eb0
authored
Jul 05, 2012
by
Taddeüs Kroes
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
Addressed some trivial feedback comments.
parent
5283801d
Changes
1
Show whitespace changes
Inline
Side-by-side
Showing
1 changed file
with
72 additions
and
77 deletions
+72
-77
docs/report.tex
docs/report.tex
+72
-77
No files found.
docs/report.tex
View file @
3b799eb0
...
@@ -31,6 +31,7 @@
...
@@ -31,6 +31,7 @@
process would be capable of serving gestures to multiple applications at
process would be capable of serving gestures to multiple applications at
the same time. A reference implementation and two test case applications
the same time. A reference implementation and two test case applications
have been created to test the effectiveness of the architecture design.
have been created to test the effectiveness of the architecture design.
% TODO: conclusions
\end{abstract}
\end{abstract}
% Set paragraph indentation
% Set paragraph indentation
...
@@ -230,21 +231,21 @@ detection for every new gesture-based application.
...
@@ -230,21 +231,21 @@ detection for every new gesture-based application.
TUIO protocol. Another driver that can keep apart rotated objects from
TUIO protocol. Another driver that can keep apart rotated objects from
simple touch points could also trigger them.
simple touch points could also trigger them.
The component that translates device-specific messages to common events,
The component that translates device-specific messages to common events,
is
will be called the
\emph
{
event driver
}
. The event driver runs in a loop,
called the
\emph
{
event driver
}
. The event driver runs in a loop, receiving
receiving and analyzing driver messages. When a sequence of messages i
s
and analyzing driver messages. When a sequence of messages is analyzed a
s
an
alyzed as an event, the event driver delegates the event to other
an
event, the event driver delegates the event to other components in the
components in the
architecture for translation to gestures.
architecture for translation to gestures.
Support for a touch driver can be added by adding an event driver
Support for a touch driver can be added by adding an event driver
implementation. The choice of event driver implementation that is used in an
implementation. The choice of event driver implementation that is used in an
application is dependent on the driver support of the touch device being
application is dependent on the driver support of the touch device being
used.
used.
Because
driver implementations have a common output format in the form of
Because
event driver implementations have a common output format in the
events, multiple event drivers can be used at the same time (see figur
e
form of events, multiple event drivers can be used at the same time (se
e
\ref
{
fig:multipledrivers
}
). This design feature allows low-level events
figure
\ref
{
fig:multipledrivers
}
). This design feature allows low-level
from multiple devices to be aggregated into high-level gestures.
events
from multiple devices to be aggregated into high-level gestures.
\multipledriversdiagram
\multipledriversdiagram
...
@@ -261,11 +262,12 @@ detection for every new gesture-based application.
...
@@ -261,11 +262,12 @@ detection for every new gesture-based application.
What's more, a widget within the application window itself should be able
What's more, a widget within the application window itself should be able
to respond to different gestures. E.g. a button widget may respond to a
to respond to different gestures. E.g. a button widget may respond to a
``tap'' gesture to be activated, whereas the application window responds to
``tap'' gesture to be activated, whereas the application window responds to
a ``pinch'' gesture to be resized. In order to be able to direct a gesture
a ``pinch'' gesture to be resized. In order to restrict the occurence of a
to a particular widget in an application, a gesture must be restricted to
gesture to a particular widget in an application, the events used for the
the area of the screen covered by that widget. An important question is if
gesture must be restricted to the area of the screen covered by that
the architecture should offer a solution to this problem, or leave the task
widget. An important question is if the architecture should offer a
of assigning gestures to application widgets to the application developer.
solution to this problem, or leave the task of assigning gestures to
application widgets to the application developer.
If the architecture does not provide a solution, the ``gesture detection''
If the architecture does not provide a solution, the ``gesture detection''
component in figure
\ref
{
fig:fulldiagram
}
receives all events that occur on
component in figure
\ref
{
fig:fulldiagram
}
receives all events that occur on
...
@@ -275,20 +277,19 @@ detection for every new gesture-based application.
...
@@ -275,20 +277,19 @@ detection for every new gesture-based application.
illustrated in figure
\ref
{
fig:ex1
}
, where two widgets on the screen can be
illustrated in figure
\ref
{
fig:ex1
}
, where two widgets on the screen can be
rotated independently. The rotation detection component that detects
rotated independently. The rotation detection component that detects
rotation gestures receives all four fingers as input. If the two groups of
rotation gestures receives all four fingers as input. If the two groups of
finger events are not separated by cluster
detection, only one rotatio
n
finger events are not separated by cluster
ing them based on the area i
n
event will occur.
which they are placed, only one rotation
event will occur.
\examplefigureone
\examplefigureone
A gesture detection component could perform a heuristic way of cluster
A gesture detection component could perform a heuristic way of clustering
detection based on the distance between events. However, this method cannot
based on the distance between events. However, this method cannot guarantee
guarantee that a cluster of events corresponds with a particular
that a cluster of events corresponds with a particular application widget.
application widget. In short, a gesture detection component is difficult to
In short, a gesture detection component is difficult to implement without
implement without awareness of the location of application widgets.
awareness of the location of application widgets. Secondly, the
Secondly, the application developer still needs to direct gestures to a
application developer still needs to direct gestures to a particular widget
particular widget manually. This requires geometric calculations in the
manually. This requires geometric calculations in the application logic,
application logic, which is a tedious and error-prone task for the
which is a tedious and error-prone task for the developer.
developer.
The architecture described here groups events that occur inside the area
The architecture described here groups events that occur inside the area
covered by a widget, before passing them on to a gesture detection
covered by a widget, before passing them on to a gesture detection
...
@@ -316,15 +317,15 @@ detection for every new gesture-based application.
...
@@ -316,15 +317,15 @@ detection for every new gesture-based application.
function to an event, that is called when the event occurs. Because of the
function to an event, that is called when the event occurs. Because of the
familiarity of this concept with developers, the architecture uses a
familiarity of this concept with developers, the architecture uses a
callback mechanism to handle gestures in an application. Callback handlers
callback mechanism to handle gestures in an application. Callback handlers
are bound to event areas, since event
s areas controls the grouping of
are bound to event areas, since event
areas control the grouping of events
events
and thus the occurrence of gestures in an area of the screen.
and thus the occurrence of gestures in an area of the screen.
\subsection
{
Area tree
}
\subsection
{
Area tree
}
\label
{
sec:tree
}
\label
{
sec:tree
}
A basic
usage of event areas in the architecture would be a list of even
t
A basic
data structure of event areas in the architecture would be a lis
t
areas. When the event driver delegates an event, it is accepted by each
of event areas. When the event driver delegates an event, it is accepted by
event area that contains the event coordinates.
e
ach e
vent area that contains the event coordinates.
If the architecture were to be used in combination with an application
If the architecture were to be used in combination with an application
framework, each widget that responds to gestures should have a mirroring
framework, each widget that responds to gestures should have a mirroring
...
@@ -394,17 +395,17 @@ detection for every new gesture-based application.
...
@@ -394,17 +395,17 @@ detection for every new gesture-based application.
gesture detection component, nor to the ancestors of the event area.
gesture detection component, nor to the ancestors of the event area.
The concept of an event area is based on the assumption that the set of
The concept of an event area is based on the assumption that the set of
originating events that form a particular gesture, can be determined
based
originating events that form a particular gesture, can be determined
exclusively
on the location of the events. This is a reasonable assumption
exclusively
based on the location of the events. This is a reasonable
for simple touch objects whose only parameter is a position, such as a pen
assumption for simple touch objects whose only parameter is a position,
or a human finger. However, more complex touch objects can have additional
such as a pen or a human finger. However, more complex touch objects can
parameters, such as rotational orientation or color. An even more generic
have additional parameters, such as rotational orientation or color. An
concept is the
\emph
{
event filter
}
, which detects whether an event should
even more generic concept is the
\emph
{
event filter
}
, which detects whether
be assigned to a particular gesture detection component based on all
an event should be assigned to a particular gesture detection component
available parameters. This level of abstraction provides additional method
s
based on all available parameters. This level of abstraction provide
s
of interaction. For example, a camera-based multi-touch surface could make
additional methods of interaction. For example, a camera-based multi-touch
a distinction between gestures performed with a blue gloved hand, and
surface could make a distinction between gestures performed with a blue
gestures performed with a green gloved hand.
g
loved hand, and g
estures performed with a green gloved hand.
As mentioned in the introduction chapter [
\ref
{
chapter:introduction
}
], the
As mentioned in the introduction chapter [
\ref
{
chapter:introduction
}
], the
scope of this thesis is limited to multi-touch surface based devices, for
scope of this thesis is limited to multi-touch surface based devices, for
...
@@ -708,16 +709,19 @@ between detected fingers to detect which fingers belong to the same hand. The
...
@@ -708,16 +709,19 @@ between detected fingers to detect which fingers belong to the same hand. The
application draws a line from each finger to the hand it belongs to, as visible
application draws a line from each finger to the hand it belongs to, as visible
in figure
\ref
{
fig:testapp
}
.
in figure
\ref
{
fig:testapp
}
.
Note that however it is a full screen event area, the overlay is not used as
Note that the overlay event area, though covering the entire screen surface, is
the root of the event area tree. Instead, the overlay is the right sibling of
not used as the root of the event area tree. Instead, the overlay is placed on
the application window area in the tree. This is needed, because the
top of the application window (being a rightmost sibling of the application
application window and its children stop the propagation of events to the root
window event area in the tree). This is necessary, because the transformation
event area. The overlay area needs all events to keep its hand tracker
trackers in the application window stop the propagation of events. The hand
up-to-date. Therefore, the tree is arranged in such a way that the overlay
tracker needs to capture all events to be able to give an accurate
event area handles an event first, before the application window can stop its
representations of all fingers touching the screen Therefore, the overlay
propagation. The event area implementation delegates events to its children in
should delegate events to the hand tracker before they are stopped by a
right-to left order, because area's that are added to the tree later are
transformation tracker. Placing the overlay over the application window forces
assumed to be positioned over their previously added siblings.
the screen event area to delegate events to the overlay event area first. The
event area implementation delegates events to its children in right-to left
order, because area's that are added to the tree later are assumed to be
positioned over their previously added siblings.
\begin{figure}
[h!]
\begin{figure}
[h!]
\center
\center
...
@@ -738,17 +742,6 @@ transformation gestures. Because the propagation of these events is stopped,
...
@@ -738,17 +742,6 @@ transformation gestures. Because the propagation of these events is stopped,
overlapping polygons do not cause a problem. Figure
\ref
{
fig:testappdiagram
}
overlapping polygons do not cause a problem. Figure
\ref
{
fig:testappdiagram
}
shows the tree structure used by the application.
shows the tree structure used by the application.
Note that the overlay event area, though covering the whole screen surface, is
not the root event area. The overlay event area is placed on top of the
application window (being a rightmost sibling of the application window event
area in the tree). This is necessary, because the transformation trackers stop
event propagation. The hand tracker needs to capture all events to be able to
give an accurate representations of all fingers touching the screen Therefore,
the overlay should delegate events to the hand tracker before they are stopped
by a transformation tracker. Placing the overlay over the application window
forces the screen event area to delegate events to the overlay event area
first.
\testappdiagram
\testappdiagram
\section
{
Conclusions
}
\section
{
Conclusions
}
...
@@ -837,7 +830,7 @@ of all requirements the set of events must meet to form the gesture.
...
@@ -837,7 +830,7 @@ of all requirements the set of events must meet to form the gesture.
A way to describe signatures on a multi-touch surface can be by the use of a
A way to describe signatures on a multi-touch surface can be by the use of a
state machine of its touch objects. The states of a simple touch point could be
state machine of its touch objects. The states of a simple touch point could be
${
down, move,
up, hold
}$
to indicate respectively that a point is put down, is
${
down, move,
hold, up
}$
to indicate respectively that a point is put down, is
being moved, is held on a position for some time, and is released. In this
being moved, is held on a position for some time, and is released. In this
case, a ``drag'' gesture can be described by the sequence
$
down
-
move
-
up
$
case, a ``drag'' gesture can be described by the sequence
$
down
-
move
-
up
$
and a ``select'' gesture by the sequence
$
down
-
hold
$
. If the set of states is
and a ``select'' gesture by the sequence
$
down
-
hold
$
. If the set of states is
...
@@ -848,13 +841,14 @@ states can be added: ${start, stop}$ to indicate that a point starts and stops
...
@@ -848,13 +841,14 @@ states can be added: ${start, stop}$ to indicate that a point starts and stops
moving. The resulting state transitions are sequences
$
down
-
start
-
move
-
moving. The resulting state transitions are sequences
$
down
-
start
-
move
-
stop
-
up
$
and
$
down
-
start
-
move
-
up
$
(the latter does not include a
$
stop
$
stop
-
up
$
and
$
down
-
start
-
move
-
up
$
(the latter does not include a
$
stop
$
to indicate that the element must keep moving after the gesture had been
to indicate that the element must keep moving after the gesture had been
performed).
performed). The two sequences distinguish a ``drag'' gesture from a ``flick''
gesture respectively.
An additional way to describe even more complex gestures is to use other
An additional way to describe even more complex gestures is to use other
gestures in a signature. An example is to combine
$
select
-
drag
$
to specify
gestures in a signature. An example is to combine
$
select
-
drag
$
to specify
that an element must be selected before it can be dragged.
that an element must be selected before it can be dragged.
The application of a state machine to describe multi-touch gestures is a
n
The application of a state machine to describe multi-touch gestures is a
subject well worth exploring in the future.
subject well worth exploring in the future.
\section
{
Daemon implementation
}
\section
{
Daemon implementation
}
...
@@ -998,9 +992,7 @@ values are normalized using division by the number of touch points $N$. A
...
@@ -998,9 +992,7 @@ values are normalized using division by the number of touch points $N$. A
the current by the previous average distance to the centroid. Any movement of
the current by the previous average distance to the centroid. Any movement of
the centroid is used for
\emph
{
drag
}
gestures. When a dragged touch point is
the centroid is used for
\emph
{
drag
}
gestures. When a dragged touch point is
released, a
\emph
{
flick
}
gesture is triggered in the direction of the
released, a
\emph
{
flick
}
gesture is triggered in the direction of the
\emph
{
drag
}
gesture. The application can use a
\emph
{
flick
}
gesture to give
\emph
{
drag
}
gesture.
momentum to a dragged widget so that it keeps moving for some time after the
dragging stops.
Figure
\ref
{
fig:transformationtracker
}
shows an example situation in which a
Figure
\ref
{
fig:transformationtracker
}
shows an example situation in which a
touch point is moved, triggering a
\emph
{
pinch
}
gesture, a
\emph
{
rotate
}
touch point is moved, triggering a
\emph
{
pinch
}
gesture, a
\emph
{
rotate
}
...
@@ -1009,16 +1001,19 @@ gesture and a \emph{drag} gesture.
...
@@ -1009,16 +1001,19 @@ gesture and a \emph{drag} gesture.
\transformationtracker
\transformationtracker
The
\emph
{
pinch
}
gesture in figure
\ref
{
fig:pinchrotate
}
uses the ratio
The
\emph
{
pinch
}
gesture in figure
\ref
{
fig:pinchrotate
}
uses the ratio
$
d
_
2
:d
_
1
$
to calculate its
$
scale
$
parameter. The difference in distance to the
$
d
_
2
:d
_
1
$
to calculate its
$
scale
$
parameter. Note that the difference in
centroid must be divided by the number of touch points (
$
N
$
) used for the
distance
$
d
_
2
-
d
_
1
$
and the difference in angle
$
\alpha
$
both relate to a
gesture, yielding the difference
$
\frac
{
d
_
2
-
d
_
1
}{
N
}$
. The
$
scale
$
parameter
single touch point. The
\emph
{
pinch
}
and
\emph
{
rotate
}
gestures that are
represents the scale relative to the previous situation, which results in the
triggered relate to all touch points, using the average of distances and
following formula:
angles. Since all except one of the touch points have not moved, their
differences in distance and angle are zero. Thus, the averages can be
calculated by dividing the differences in distance and angle of the moved touch
point by the number of touch points
$
N
$
. The
$
scale
$
parameter represents the
scale relative to the previous situation, which results in the following
formula:
$$
pinch.scale
=
\frac
{
d
_
1
+
\frac
{
d
_
2
-
d
_
1
}{
N
}}{
d
_
1
}$$
$$
pinch.scale
=
\frac
{
d
_
1
+
\frac
{
d
_
2
-
d
_
1
}{
N
}}{
d
_
1
}$$
The angle used for the
\emph
{
rotate
}
gesture is only divided by the number of
The angle used for the
\emph
{
rotate
}
gesture is also divided by the number of
touch points to obtain an average rotation of all touch points:
touch points:
$$
rotate.angle
=
\frac
{
\alpha
}{
N
}$$
$$
rotate.angle
=
\frac
{
\alpha
}{
N
}$$
\section
{
Hand tracker
}
\section
{
Hand tracker
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment