Commit 3b799eb0 authored by Taddeüs Kroes's avatar Taddeüs Kroes

Addressed some trivial feedback comments.

parent 5283801d
...@@ -31,6 +31,7 @@ ...@@ -31,6 +31,7 @@
process would be capable of serving gestures to multiple applications at process would be capable of serving gestures to multiple applications at
the same time. A reference implementation and two test case applications the same time. A reference implementation and two test case applications
have been created to test the effectiveness of the architecture design. have been created to test the effectiveness of the architecture design.
% TODO: conclusions
\end{abstract} \end{abstract}
% Set paragraph indentation % Set paragraph indentation
...@@ -230,21 +231,21 @@ detection for every new gesture-based application. ...@@ -230,21 +231,21 @@ detection for every new gesture-based application.
TUIO protocol. Another driver that can keep apart rotated objects from TUIO protocol. Another driver that can keep apart rotated objects from
simple touch points could also trigger them. simple touch points could also trigger them.
The component that translates device-specific messages to common events, The component that translates device-specific messages to common events, is
will be called the \emph{event driver}. The event driver runs in a loop, called the \emph{event driver}. The event driver runs in a loop, receiving
receiving and analyzing driver messages. When a sequence of messages is and analyzing driver messages. When a sequence of messages is analyzed as
analyzed as an event, the event driver delegates the event to other an event, the event driver delegates the event to other components in the
components in the architecture for translation to gestures. architecture for translation to gestures.
Support for a touch driver can be added by adding an event driver Support for a touch driver can be added by adding an event driver
implementation. The choice of event driver implementation that is used in an implementation. The choice of event driver implementation that is used in an
application is dependent on the driver support of the touch device being application is dependent on the driver support of the touch device being
used. used.
Because driver implementations have a common output format in the form of Because event driver implementations have a common output format in the
events, multiple event drivers can be used at the same time (see figure form of events, multiple event drivers can be used at the same time (see
\ref{fig:multipledrivers}). This design feature allows low-level events figure \ref{fig:multipledrivers}). This design feature allows low-level
from multiple devices to be aggregated into high-level gestures. events from multiple devices to be aggregated into high-level gestures.
\multipledriversdiagram \multipledriversdiagram
...@@ -261,11 +262,12 @@ detection for every new gesture-based application. ...@@ -261,11 +262,12 @@ detection for every new gesture-based application.
What's more, a widget within the application window itself should be able What's more, a widget within the application window itself should be able
to respond to different gestures. E.g. a button widget may respond to a to respond to different gestures. E.g. a button widget may respond to a
``tap'' gesture to be activated, whereas the application window responds to ``tap'' gesture to be activated, whereas the application window responds to
a ``pinch'' gesture to be resized. In order to be able to direct a gesture a ``pinch'' gesture to be resized. In order to restrict the occurence of a
to a particular widget in an application, a gesture must be restricted to gesture to a particular widget in an application, the events used for the
the area of the screen covered by that widget. An important question is if gesture must be restricted to the area of the screen covered by that
the architecture should offer a solution to this problem, or leave the task widget. An important question is if the architecture should offer a
of assigning gestures to application widgets to the application developer. solution to this problem, or leave the task of assigning gestures to
application widgets to the application developer.
If the architecture does not provide a solution, the ``gesture detection'' If the architecture does not provide a solution, the ``gesture detection''
component in figure \ref{fig:fulldiagram} receives all events that occur on component in figure \ref{fig:fulldiagram} receives all events that occur on
...@@ -275,20 +277,19 @@ detection for every new gesture-based application. ...@@ -275,20 +277,19 @@ detection for every new gesture-based application.
illustrated in figure \ref{fig:ex1}, where two widgets on the screen can be illustrated in figure \ref{fig:ex1}, where two widgets on the screen can be
rotated independently. The rotation detection component that detects rotated independently. The rotation detection component that detects
rotation gestures receives all four fingers as input. If the two groups of rotation gestures receives all four fingers as input. If the two groups of
finger events are not separated by cluster detection, only one rotation finger events are not separated by clustering them based on the area in
event will occur. which they are placed, only one rotation event will occur.
\examplefigureone \examplefigureone
A gesture detection component could perform a heuristic way of cluster A gesture detection component could perform a heuristic way of clustering
detection based on the distance between events. However, this method cannot based on the distance between events. However, this method cannot guarantee
guarantee that a cluster of events corresponds with a particular that a cluster of events corresponds with a particular application widget.
application widget. In short, a gesture detection component is difficult to In short, a gesture detection component is difficult to implement without
implement without awareness of the location of application widgets. awareness of the location of application widgets. Secondly, the
Secondly, the application developer still needs to direct gestures to a application developer still needs to direct gestures to a particular widget
particular widget manually. This requires geometric calculations in the manually. This requires geometric calculations in the application logic,
application logic, which is a tedious and error-prone task for the which is a tedious and error-prone task for the developer.
developer.
The architecture described here groups events that occur inside the area The architecture described here groups events that occur inside the area
covered by a widget, before passing them on to a gesture detection covered by a widget, before passing them on to a gesture detection
...@@ -316,15 +317,15 @@ detection for every new gesture-based application. ...@@ -316,15 +317,15 @@ detection for every new gesture-based application.
function to an event, that is called when the event occurs. Because of the function to an event, that is called when the event occurs. Because of the
familiarity of this concept with developers, the architecture uses a familiarity of this concept with developers, the architecture uses a
callback mechanism to handle gestures in an application. Callback handlers callback mechanism to handle gestures in an application. Callback handlers
are bound to event areas, since events areas controls the grouping of are bound to event areas, since event areas control the grouping of events
events and thus the occurrence of gestures in an area of the screen. and thus the occurrence of gestures in an area of the screen.
\subsection{Area tree} \subsection{Area tree}
\label{sec:tree} \label{sec:tree}
A basic usage of event areas in the architecture would be a list of event A basic data structure of event areas in the architecture would be a list
areas. When the event driver delegates an event, it is accepted by each of event areas. When the event driver delegates an event, it is accepted by
event area that contains the event coordinates. each event area that contains the event coordinates.
If the architecture were to be used in combination with an application If the architecture were to be used in combination with an application
framework, each widget that responds to gestures should have a mirroring framework, each widget that responds to gestures should have a mirroring
...@@ -394,17 +395,17 @@ detection for every new gesture-based application. ...@@ -394,17 +395,17 @@ detection for every new gesture-based application.
gesture detection component, nor to the ancestors of the event area. gesture detection component, nor to the ancestors of the event area.
The concept of an event area is based on the assumption that the set of The concept of an event area is based on the assumption that the set of
originating events that form a particular gesture, can be determined based originating events that form a particular gesture, can be determined
exclusively on the location of the events. This is a reasonable assumption exclusively based on the location of the events. This is a reasonable
for simple touch objects whose only parameter is a position, such as a pen assumption for simple touch objects whose only parameter is a position,
or a human finger. However, more complex touch objects can have additional such as a pen or a human finger. However, more complex touch objects can
parameters, such as rotational orientation or color. An even more generic have additional parameters, such as rotational orientation or color. An
concept is the \emph{event filter}, which detects whether an event should even more generic concept is the \emph{event filter}, which detects whether
be assigned to a particular gesture detection component based on all an event should be assigned to a particular gesture detection component
available parameters. This level of abstraction provides additional methods based on all available parameters. This level of abstraction provides
of interaction. For example, a camera-based multi-touch surface could make additional methods of interaction. For example, a camera-based multi-touch
a distinction between gestures performed with a blue gloved hand, and surface could make a distinction between gestures performed with a blue
gestures performed with a green gloved hand. gloved hand, and gestures performed with a green gloved hand.
As mentioned in the introduction chapter [\ref{chapter:introduction}], the As mentioned in the introduction chapter [\ref{chapter:introduction}], the
scope of this thesis is limited to multi-touch surface based devices, for scope of this thesis is limited to multi-touch surface based devices, for
...@@ -708,16 +709,19 @@ between detected fingers to detect which fingers belong to the same hand. The ...@@ -708,16 +709,19 @@ between detected fingers to detect which fingers belong to the same hand. The
application draws a line from each finger to the hand it belongs to, as visible application draws a line from each finger to the hand it belongs to, as visible
in figure \ref{fig:testapp}. in figure \ref{fig:testapp}.
Note that however it is a full screen event area, the overlay is not used as Note that the overlay event area, though covering the entire screen surface, is
the root of the event area tree. Instead, the overlay is the right sibling of not used as the root of the event area tree. Instead, the overlay is placed on
the application window area in the tree. This is needed, because the top of the application window (being a rightmost sibling of the application
application window and its children stop the propagation of events to the root window event area in the tree). This is necessary, because the transformation
event area. The overlay area needs all events to keep its hand tracker trackers in the application window stop the propagation of events. The hand
up-to-date. Therefore, the tree is arranged in such a way that the overlay tracker needs to capture all events to be able to give an accurate
event area handles an event first, before the application window can stop its representations of all fingers touching the screen Therefore, the overlay
propagation. The event area implementation delegates events to its children in should delegate events to the hand tracker before they are stopped by a
right-to left order, because area's that are added to the tree later are transformation tracker. Placing the overlay over the application window forces
assumed to be positioned over their previously added siblings. the screen event area to delegate events to the overlay event area first. The
event area implementation delegates events to its children in right-to left
order, because area's that are added to the tree later are assumed to be
positioned over their previously added siblings.
\begin{figure}[h!] \begin{figure}[h!]
\center \center
...@@ -738,17 +742,6 @@ transformation gestures. Because the propagation of these events is stopped, ...@@ -738,17 +742,6 @@ transformation gestures. Because the propagation of these events is stopped,
overlapping polygons do not cause a problem. Figure \ref{fig:testappdiagram} overlapping polygons do not cause a problem. Figure \ref{fig:testappdiagram}
shows the tree structure used by the application. shows the tree structure used by the application.
Note that the overlay event area, though covering the whole screen surface, is
not the root event area. The overlay event area is placed on top of the
application window (being a rightmost sibling of the application window event
area in the tree). This is necessary, because the transformation trackers stop
event propagation. The hand tracker needs to capture all events to be able to
give an accurate representations of all fingers touching the screen Therefore,
the overlay should delegate events to the hand tracker before they are stopped
by a transformation tracker. Placing the overlay over the application window
forces the screen event area to delegate events to the overlay event area
first.
\testappdiagram \testappdiagram
\section{Conclusions} \section{Conclusions}
...@@ -837,7 +830,7 @@ of all requirements the set of events must meet to form the gesture. ...@@ -837,7 +830,7 @@ of all requirements the set of events must meet to form the gesture.
A way to describe signatures on a multi-touch surface can be by the use of a A way to describe signatures on a multi-touch surface can be by the use of a
state machine of its touch objects. The states of a simple touch point could be state machine of its touch objects. The states of a simple touch point could be
${down, move, up, hold}$ to indicate respectively that a point is put down, is ${down, move, hold, up}$ to indicate respectively that a point is put down, is
being moved, is held on a position for some time, and is released. In this being moved, is held on a position for some time, and is released. In this
case, a ``drag'' gesture can be described by the sequence $down - move - up$ case, a ``drag'' gesture can be described by the sequence $down - move - up$
and a ``select'' gesture by the sequence $down - hold$. If the set of states is and a ``select'' gesture by the sequence $down - hold$. If the set of states is
...@@ -848,13 +841,14 @@ states can be added: ${start, stop}$ to indicate that a point starts and stops ...@@ -848,13 +841,14 @@ states can be added: ${start, stop}$ to indicate that a point starts and stops
moving. The resulting state transitions are sequences $down - start - move - moving. The resulting state transitions are sequences $down - start - move -
stop - up$ and $down - start - move - up$ (the latter does not include a $stop$ stop - up$ and $down - start - move - up$ (the latter does not include a $stop$
to indicate that the element must keep moving after the gesture had been to indicate that the element must keep moving after the gesture had been
performed). performed). The two sequences distinguish a ``drag'' gesture from a ``flick''
gesture respectively.
An additional way to describe even more complex gestures is to use other An additional way to describe even more complex gestures is to use other
gestures in a signature. An example is to combine $select - drag$ to specify gestures in a signature. An example is to combine $select - drag$ to specify
that an element must be selected before it can be dragged. that an element must be selected before it can be dragged.
The application of a state machine to describe multi-touch gestures is an The application of a state machine to describe multi-touch gestures is a
subject well worth exploring in the future. subject well worth exploring in the future.
\section{Daemon implementation} \section{Daemon implementation}
...@@ -998,9 +992,7 @@ values are normalized using division by the number of touch points $N$. A ...@@ -998,9 +992,7 @@ values are normalized using division by the number of touch points $N$. A
the current by the previous average distance to the centroid. Any movement of the current by the previous average distance to the centroid. Any movement of
the centroid is used for \emph{drag} gestures. When a dragged touch point is the centroid is used for \emph{drag} gestures. When a dragged touch point is
released, a \emph{flick} gesture is triggered in the direction of the released, a \emph{flick} gesture is triggered in the direction of the
\emph{drag} gesture. The application can use a \emph{flick} gesture to give \emph{drag} gesture.
momentum to a dragged widget so that it keeps moving for some time after the
dragging stops.
Figure \ref{fig:transformationtracker} shows an example situation in which a Figure \ref{fig:transformationtracker} shows an example situation in which a
touch point is moved, triggering a \emph{pinch} gesture, a \emph{rotate} touch point is moved, triggering a \emph{pinch} gesture, a \emph{rotate}
...@@ -1009,16 +1001,19 @@ gesture and a \emph{drag} gesture. ...@@ -1009,16 +1001,19 @@ gesture and a \emph{drag} gesture.
\transformationtracker \transformationtracker
The \emph{pinch} gesture in figure \ref{fig:pinchrotate} uses the ratio The \emph{pinch} gesture in figure \ref{fig:pinchrotate} uses the ratio
$d_2:d_1$ to calculate its $scale$ parameter. The difference in distance to the $d_2:d_1$ to calculate its $scale$ parameter. Note that the difference in
centroid must be divided by the number of touch points ($N$) used for the distance $d_2 - d_1$ and the difference in angle $\alpha$ both relate to a
gesture, yielding the difference $\frac{d_2 - d_1}{N}$. The $scale$ parameter single touch point. The \emph{pinch} and \emph{rotate} gestures that are
represents the scale relative to the previous situation, which results in the triggered relate to all touch points, using the average of distances and
following formula: angles. Since all except one of the touch points have not moved, their
differences in distance and angle are zero. Thus, the averages can be
calculated by dividing the differences in distance and angle of the moved touch
point by the number of touch points $N$. The $scale$ parameter represents the
scale relative to the previous situation, which results in the following
formula:
$$pinch.scale = \frac{d_1 + \frac{d_2 - d_1}{N}}{d_1}$$ $$pinch.scale = \frac{d_1 + \frac{d_2 - d_1}{N}}{d_1}$$
The angle used for the \emph{rotate} gesture is only divided by the number of
The angle used for the \emph{rotate} gesture is also divided by the number of touch points to obtain an average rotation of all touch points:
touch points:
$$rotate.angle = \frac{\alpha}{N}$$ $$rotate.angle = \frac{\alpha}{N}$$
\section{Hand tracker} \section{Hand tracker}
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment