Perceptually Adaptive Graphics.pdf

(3104 KB) Pobierz
EUROGRAPHICS 2004
STAR – State of The Art Report
Perceptually Adaptive Graphics
Carol O’Sullivan, Sarah Howlett, Rachel McDonnell, Yann Morvan, Keith O’Conor
Image Synthesis Group, Trinity College Dublin
Abstract
In recent years, the Graphics community has come to realise the importance of taking human perception into ac-
count when striving for realism in images, animations and Virtual Environments. In May 2001, a EUROGRAPH-
ICS/SIGGRAPH Campfire brought together a group of researchers from various fields, including computer graph-
ics and visualisation, psychology, neuroscience, eye-movements and medicine to discuss the future and current
state of the field. Since then, many researchers have been very active in furthering the field of perceptually adap-
tive graphics. In this report, we outline the state of the art as discussed at that event and the progress that has been
made since.
Categories and Subject Descriptors
(according to ACM CCS)
: I.3.7 [Computer Graphics]: Three-Dimensional
Graphics and Realism
Keywords: Graphics and Perception, Perceptual Metrics, Psychophysics
1. Introduction
At this stage in the development of the field of Computer
Graphics, great progress has been made in generating highly
realistic renderings and simulations of objects, characters,
fluid, smoke, fire and many other effects. New hardware
architectures enable users of commodity PCs to run high-
quality graphical applications heretofore only available on
costly high-performance graphics workstations. Meanwhile,
the demand for visually appealing applications on low-end
mobile devices such as PDAs and cell phones has increased
dramatically, leading to the development of new architec-
tures and libraries to achieve this purpose. This myriad of
computational and display opportunities brings with it more
than algorithmic and technical challenges. For photorealistic
images and movies, how do we know that we are not simply
producing pretty pictures and actually representing reality in
a faithful manner? For real-time rendering and simulation,
how do we make speed-accuracy trade-offs while minimis-
ing the perceptibility of any resulting anomalies? What types
of anomalies are most noticeable and when can we fake real-
ity and get away with it? How can we quantify these factors
and use them in a methodical way to adapt our graphics to
the perception of the viewer? Perceptually adaptive Graph-
ics involves the investigation of these issues and will be the
subject of this State of the Art Report.
c The Eurographics Association 2004.
Graphics researchers can learn much by studying the lit-
erature from experimental psychology, psychophysics, neu-
rophysiology and related areas. Many computational mod-
els of various perceptual functions have been developed and
used in recent years by the graphics community. However,
such models are often difficult to apply or generalise to han-
dle the complex visual tasks typical in graphical scenes and
simulations. Experiments are often designed to study quite
specific factors in isolation and in unnatural laboratory situa-
tions. Therefore, many researchers from the field of psychol-
ogy have realised the potential of using Computer Graph-
ics to devise more natural scenes and higher level tasks that
more truly reflect the situations being simulated. Therefore,
interdisciplinary studies involving Graphics and Perception
work to the benefit of both communities.
The increasing interdisciplinary interest in graphics and
perception in recent years has resulted in new insights into
both areas. Those working on the borders of both disciplines
needed a forum in which to meet like-minded people and
discuss the state of the art and, in response to this need,
a SIGGRAPH/EUROGRAPHICS Campfire on Perceptually
Adaptive Graphics [MO01] was held in Snowbird Utah.
Prior to this informal workshop, participants were asked to
submit position statements, which were subsequently cate-
gorised under 6 headings: Image Fidelity, Scene Perception,
142
O’Sullivan
et al.
/ Perceptually Adaptive Graphics
Figure 1:
Eye-tracking can be used both to evaluate metrics and for perceptually adaptive collision handling
Applications, Visualization, Interactive Graphics and finally,
Distance and Scale in Computer Graphics. The resulting po-
sition papers provide a good snap-shot of the state of the
art in perceptually adaptive graphics at that time. In this re-
port, we will start our review of recent work from that point
onwards, and provide some indicative citations of progress
in the meantime. Recent and ongoing research will be pre-
sented under the following main headings:
Interactive Graphics:
We will discuss how perceptually
driven criteria can be used to accelerate interactive render-
ing, including perceptually adaptive LOD techniques, and
approaches that take attention and eye-movements into ac-
count.
Image Fidelity:
Topics will include image fidelity met-
rics, perceptual display of high dynamic range images, and
perception-driven global illumination techniques.
Animation:
Recent work on perceptual metrics for eval-
uating physically based animation and character animation
will be presented, including methods that use eye-tracking
(see Figure
1).
Virtual Environments:
We consider the problem of ef-
fectively generating images of objects and environments that
convey an accurate sense of distance and size, along with
fidelity metrics for Virtual Environments and methods for
measuring and predicting Presence.
Visualisation and Non-Photorealistic Rendering
(NPR)
A major challenge in visualisation is the design of
algorithms that represent large, complex datasets in such a
way that people can quickly understand and grasp their main
features. An understanding of human perception is integral
to the success of such systems and some ongoing research in
this area will be presented. Recent work in the application of
perceptual principles to non-photorealistic rendering is also
discussed.
Figure 2:
One set of stimuli from Watson’s experiment:
Original (top), QSlim at 80% (middle), Vclust 80% (bottom)
(Image from [WFM01] courtesy of Ben Watson.)
2. Interactive Graphics
In interactive graphics applications, the ideal situation would
be to render a fully detailed and photorealistic scene in real-
time. However, despite recent advances in current computer
graphics technology, this is not yet a feasible option in gen-
eral. Therefore, the aim is to produce the best perceived im-
age in the time available. An important measurement of vi-
sual quality is perceptual i.e., how much does the simulated
scene look like the original. The properties of human vision
are therefore very important in the design of perceptually
based graphics. In this section, we review some of the most
recent work in this field. Many of these approaches rely on
c The Eurographics Association 2004.
O’Sullivan
et al.
/ Perceptually Adaptive Graphics
143
duction in quality has no effect on performance even if it is
noticeable.
Focus Plus Context Screens
are one result of new research
into extending gaze-contingent displays. Foveal regions of
arbitrary shape or size can be created, with peripheral re-
gions degraded by arbitrary means such as colour or con-
trast and not simply resolution. Additionally, the simulta-
neous display of multiple foveal regions is possible, which
can be used for prediction. Usually, when peripheral content
is rendered at low resolution, the display hardware is still
the same resolution as any other part of the screen surface.
However, in the case of a focus plus context screen, there
is a difference in resolution between the focus and the con-
text area. It contains a wall sized low-resolution display with
an embedded high-resolution screen. When the user moves
the mouse, the display content pans and can be brought into
high resolution focus as required. This is interesting for large
maps or chip design where certain areas need to be focused
upon.
Baudisch
et al.
also describe attentive user interface tech-
niques for directing a system’s resources towards the scene
components in real-time 3D graphics. Specifically, attentive
3D-rendering engines are discussed, which uses a viewer’s
gaze position to vary the LOD at which an object is drawn
(see Luebke
et al.,
for further details [LRC
∗
02]).
Although
similar to a gaze-contingent display, such approaches have
one main difference; objects in an attentive 3D-rendering en-
gine are simplified at the object geometry level instead of the
image level.
Finally, they describe displays in which the user’s atten-
tion is captured as opposed to followed, referred to as
Easily
Perceived Displays.
In such systems, the idea is to guide the
user’s attention and allow computer generation of aestheti-
cally pleasing images. Such systems use a perceptual model
that works from the gaze information of one user to decide
which parts of a photograph should be removed. The result
is stylized using smooth black lines and colored regions and
not just blurred, thus guiding attention to what the original
user found important.
In all cases described by Baudisch
et al.,
the intent is to
match the characteristics of human vision to the character-
istics of computer displays, namely its distinction between
foveal and peripheral vision. Gaze-contingent displays and
3D approaches improve display frame rates and responsive-
ness given certain rendering hardware; focus plus context
screens achieve better immersion and visual context given
certain display hardware; and nonphotorealistic rendering
saves the resources of the user’s attention.
In some cases, a model of visual attention has been used
to predict fixations instead of tracking the user’s gaze. How-
ever, the success of the resulting system depends strongly
on the similarity between the predicted and actual regions
foveated by the observer. Marmitt and Duchowski [MD03]
Figure 3:
A view presented in the second experiment. Here
the periphery uses the 20 x 15 LOD, while the lowest con-
trast background is used. The central area is (always) dis-
played at the highest HMD resolution. Four distractors are
shown. (Image from [WWH04] courtesy of Ben Watson.)
models of visual attention or on an explicit measurement of
gaze direction using eye-tracking.
2.1. Gaze-contingent approaches
There is an increasing demand for better display resolu-
tion with greater numbers of pixels, posing a constant chal-
lenge as large numbers of pixels consume a lot of compu-
tational resources. However, this is one scenario in which
failures of the visual system can be exploited. Baudisch
et
al.
[BDDG03] present several different approaches to take
advantage of this using
attentive displays,
discussed in the
following paragraphs.
When there is not sufficient computer power for a given
task, there is a negative effect of large-screen technology
on user productivity. As a step towards compensating for
the demand for rendering power and display resolution, the
user’s attentional focus can be taken into account. Instead
of having to add more hardware, display and computation
resources can be instead directed to where they are needed
most. A single user can only ever focus on a small portion
of the display at a time. Approaches using an eye-tracker
are examined, which aim to match the subjective quality of
a non-degraded display. Alternatively, image content can be
removed to achieve a different effect; by presenting view-
ers with only the most important information, their cognitive
load can be reduced.
Gaze-contingent displays degrade the resolution in the pe-
ripheral image regions. The high resolution area moves with
the user’s focus, so the area under scrutiny is always ren-
dered at a higher resolution. An imperceptible degradation
is difficult to achieve but often, in visual search tasks, the re-
c The Eurographics Association 2004.
144
O’Sullivan
et al.
/ Perceptually Adaptive Graphics
have developed and evaluated a new method for the com-
parison of human and artificial scanpaths recorded in virtual
reality. They use a string editing methodology for the eval-
uation of human-human or human-artificial scanpaths. They
compare the sequence of regions of interest identified using
Itti
et al’s
attentional model [IKN98] with those recorded
from a human observer. The experiment examined three dif-
ferent scenarios; a simple cube, a panorama, and a more
complex graphical environment, which participants were al-
lowed to free-view.
They showed that, for all three situations, the similarities
between the human and the artificial scanpaths are less than
expected. Although this attentional model works reasonably
well for still images, it does not accurately predict human
fixations in a virtual reality environment. They found that
the attentional model assigns attention to a wider area of the
image, whereas observers pay more attention to the central
region of the display.
Recent work by Cater
et al.
[CCW03] supports the sug-
gestion that visual attention is largely controlled by the task.
One key advantage to this approach is that attention is there-
fore only dependent upon a specific task and not on the user.
Therefore, no eye-tracker would be needed as different peo-
ple performing the same task should, the authors claim, be
using similar visual processes. They show how task seman-
tics can be used to selectively render in high quality only the
details of the scene that are attended to.
They carried out experiments involving a task on a
still image. Participants were required to count the num-
ber of teapots in a computer generated office scene, which
was rendered at three different levels of resolution; high
(3072x3072), low (1024x1024) and selective level. At the
selective level the scene was mostly rendered at a low level
except for the visual angle of the fovea (2 degrees) centered
on each teapot. All scenes were exactly the same except for
the position of the teapots.
Results showed that, when carrying out a task, partici-
pants consistently failed to notice any difference between
the high and the selective quality image. Twenty percent of
observers even failed to notice the difference between the
high and low quality images. Furthermore, when there was
no task involved, the difference almost always went unno-
ticed. This demonstrates that people primarily attend to task-
related objects and the authors postulate that such objects
can often be identified in advance, depending on the task.
They show experimentally that it is possible to render scene
objects not related to the task at lower resolution without the
viewer noticing any reduction in quality.
As a confirmation and a demonstration that this was in-
deed the effect of inattentional blindness and not peripheral
vision, they used an eye-tracker to show that attention was
fully consumed by the counting task. Even though partici-
pants fixated on non-teapot objects, these fixations were not
enough to distinguish the difference in quality. They noted
that task driven focus seems to override low-level visual at-
tention when it comes to noticing artifacts. Therefore, they
claim, intentional blindness can be exploited to significantly
reduce the rendered quality of a large portion of the scene,
without having any significant effect on the overall percep-
tion.
Figure 4:
Results from the saliency experiment depicted as
a colour-map (white representing the greatest number): the
total length of fixations on the familiar natural objects (top),
the duration of the first fixations on the man-made artifacts
(middle), the total number of fixations on the unfamiliar ob-
jects (bottom).
2.2. Perceptually guided polygonal simplification
In the recent work of Luebke
et al.
[LH01], the primary
objective was to improve interactive rendering rather than
develop offline rendering approaches such as ray tracing,
which are typically too computationally expensive to be
evaluated interactively. They demonstrate a novel approach
to reducing model complexity that is driven by perceptual
criteria. They use a psychophysical model of visual percep-
tion to create a framework that improves interactive ren-
dering and is used for multiresolution rendering techniques.
The circumstances under which simplification will be per-
ceptible are determined, and those that are deemed per-
ceptible are not carried out. Their framework is applied to
c The Eurographics Association 2004.
O’Sullivan
et al.
/ Perceptually Adaptive Graphics
145
view-dependent polygonal simplification and factors such
as imperceptible simplification, silhouette preservation and
gaze-directed rendering are taken into account. Their results
demonstrate that imperceptible simplification was achieved
with a limited reduction in polygon count when this method
was used. In their evaluation it was found that the probability
of seeing a difference was no better than chance. They claim
that models could potentially be reduced even more i.e., up
to three times further, without a degradation in perception
due to the conservative estimate of the spatial frequency at
present.
Closely related is work from Williams
et al.
[WLC
∗
03],
who describe a best-effort simplification of polygonal
meshes based on rules of visual perception. Best-effort ren-
dering is a form of time-critical computing where processing
must occur within a certain time budget. This work applies
to a wider range of models and accounts for textures and dy-
namic lighting. They use parameterized texture deviation to
measure distortion more accurately, leading to better simpli-
fications for a certain number of polygons. The simplifica-
tion of lit models is improved by accounting for both spec-
ular and diffuse effects, under both Gouraud-shaded vertex
lighting and per-pixel normal-map lighting. Here the focus
is not so much on imperceptible simplification, but on the
approach of perceptually-guided best-effort rendering to a
budget. The most obvious advantage of this approach is on
vertex-lit models, because the distortion and tessellation ar-
tifacts in specular highlights are highly perceptible. Normal
maps are used to maintain smooth highlights even at low res-
olutions. The system has the ability to simplify low-contrast
regions and to preserve high-contrast areas such as silhou-
ettes.
How can we tell if one simplification is actually bet-
ter than another? To answer this question, Watson
et al.
[WFM01] looked at techniques that experimentally and au-
tomatically measured and predicted the visual fidelity of
simplified models. A set of 36 3D polygonal models were
simplified using two different simplification methods (QS-
lim and Vclust) to two levels of detail (50% and 80% of
the original detail), as shown in Figure
2.
The stimuli were
divided into two different object categories; natural objects
and man-made artifacts. Three experimental measures were
used to measure the fidelity of these images; naming time
(i.e., time taken to verbalise the name of an object), ratings
and forced choice preferences. All measures were affected
by simplification level and type of simplification. Naming
times were longer with increasing simplification and it took
longer to name objects simplified using Vclust. When ratings
were measured, participants were sensitive to simplification
level and also rated objects simplified by QSlim as closer
to the ideal. The preference results showed that there was a
greater preference for Qslim-simplified stimuli, which was
greater for the animal models and greater for the objects at
80% detail.
c The Eurographics Association 2004.
The effect of object type was particularly interesting.
Firstly, it took longer to name the natural objects, which
was consistent with earlier results. Furthermore, the ratings
results showed that the animal models were more like the
standard when simplified using QSlim, but that the artifacts
were more like the standard when Vclust had been used dur-
ing simplification. Regarding preferences, the preference for
QSlim-simplified stimuli was greater for the animal models
than for the artifact models and for the 80% simplified mod-
els than the 50% objects.
Watson
et al.
also examined three automatic techniques
to see how well they predicted these experimental measures.
They found that they were generally good predictors of qual-
ity as judged by ratings and preferences, but were not as suc-
cessful at predicting naming times. In general the automatic
measures correlate well to preferences, less well to ratings
and poorly to differences in naming times. Correlations were
worse for animals than for artifacts.
Pojar and Schmalstieg [PS03] present a tool for user-
controlled creation of multiresolution meshes, allowing se-
lective control of simplification. The user is allowed to
identify mesh regions of high semantic or functional im-
portance. These importance weightings are supplied by the
user through a Maya plug-in interactively. In this approach,
the original Quadric Error Metric of Garland and Heckbert
[GH97] is weighted by the user input during simplification.
The resulting framework allows the user to improve the qual-
ity of a multiresolution mesh by taking semantic and func-
tional importance into account.
In a similar approach, Kho and Garland [KG03] pro-
vide another user-guided simplification system that allows
the user to interactively control an automatic simplification
method. As before, they use a weighted quadric error met-
ric. However, as well as adaptive simplification they provide
a tool that allows the user to apply geometric constraints.
These constraints preserve features by guiding the placement
of vertices on the approximation. The resulting system needs
only a short interaction time to produce better semantic or
functional models and there is no time difference between
user-guided simplification and the original automatic sim-
plification.
More recently, Howlett
et al.
[HHO04] attempted to de-
termine salient features by using an eye-tracking device to
capture human gaze data and then investigated if the visual
fidelity of simplified polygonal models can be improved by
emphasizing the detail of salient features identified in this
way (see Figure
4).
To try to evaluate the visual fidelity
of models simplified using both metrics, a set of naming
time, matching time and forced-choice preference experi-
ments were carried out.
It was found that the perceptually weighted metric led to
a significant increase in visual fidelity for the lower lev-
els of detail of the natural objects, but that for the man-
made artifacts the opposite was true. It was concluded that

Plik z chomika:

Inne pliki z tego folderu:

Inne foldery tego chomika:

Zgłoś jeśli naruszono regulamin