Design and User Evaluation of Augmented-Reality Interfaces

Spring term 2014, fourth period

582705, 4 credit units

This is the web page for the course on Design and User Evaluation of Augmented-Reality Interfaces held by Elisa Schaeffer, visiting researcher at HIIT, at the CS Dept. of the University of Helsinki on Wednesdays and Fridays 10:15-12:00 hours in room C222. The course will be held entirely in English, but personal guidance for the project work is available in other languages as well.


Usability aspects of augmented-reality interfaces differ from the conventional desktop user interfaces and also from the usability of handheld mobile devices. In this course, each participant, either individually or in small teams, selects an augmented-reality user interface, after which we review and adapt methods for usability evaluation and particularly user experiments. Then, the participants design, carry out, document, and analyze a usability evaluation on their prototype, reporting their findings.

It is recommended, although not mandatory, to participate in the seminar on Augmented Reality (58313306) in the Fall term 2013; other previous experience of AR-technologies will compensate for not taking the seminar.


The detailed session contents and schedule depend on the number of registered participants. There are twelve sessions over a total of six weeks, which is the regular setup for one-period courses.

  1. Introductions and topic selection

    We begin with some terminology definitions and general introductions of the participants and their background.

    For the course work, everyone needs to have an AR prototype to work with, either individually or in small groups (2—3 people). This may be product of a previous work (the Fall seminar, a thesis project, a research project) or something that is available online (preferably open source). During the first session, everyone will attempt to determine on which AR prototype they will be working on and with whom.

    It is entirely permitted for the participants to develop a new AR-application or to modify an existing one, but there will be no points rewarded for the implementation efforts. It is recommendable, for time-saving purposes, to work with something that already exists.

  2. Prototype design/feature presentations

    A small introductory presentation is given by the participants regarding the AR prototype they will work with: what is it for, what does it do, how does it work, from whom is it designed, in what type of a context is it to be used, etc. Also, if they propose any change or modification to it, this needs to be justified and explained.

  3. Usability-evaluation technique mapping

    Together, we will compile a list of usability-evaluation techniques (in general) that we know or at least have heard of. Each participant will prepare a short introduction to one or more methods (to present in the next session), aimed at clarifying what "input" is required to apply the method, what exactly needs to be done, what will be captured as "output", and how is this output to be analyzed.

    Ideally, within a team, at least one testing-based evaluation, one inspection technique, and one inquiry-based method should be covered. At least one method that involves users is required to be included by each team; the presentations as such are individual. It is also feasible to include one or more expert-evaluation techniques, but the user involvement is very important for the project as a whole.

  4. Usability-evaluation technique assignment

    We begin with the individual presentations regarding the techniques. Then, each team will select two or more usability-evaluation techniques based on the descriptions prepared in the previous session. The product of the session will be a plan (including a schedule) of adapting and carrying out these evaluations on the AR prototype they're working on. These plans are presented in the beginning of the next session.

    It is desirable that different teams pick different techniques and that the selection is related to the properties of their particular AR prototype in the sense that there's a reason to expect for the techniques to adapt to that prototype.

  5. Adaptation of the usability-evaluation techniques

    Each team will present their proposed adaptation of their selected techniques (at least two per team), as it is unlikely that general, traditional usability method designed for personal computers, websites, mobile devices, or physical gadgets turn out to be directly applicable as such in an AR setting. The proposal is presented in a slideshow accompanied by a written (book-chapter like) document draft, sent to the instructor in PDF before the beginning of the session. A (complete) design of any experiments is expected to be included. Also a schedule for when, where, how, and what will be done is necessary.

    During the session, each team presents their plan (with the prepared slides) and receives feedback from others. Also, the activities for the next three sessions are agreed upon among the participants attending this session.

  6. Working session #1

    The teams work on their projects. Attendance is optional but rewarded in the participation points.

  7. Working session #2

    The teams work on their projects. Attendance is optional but rewarded in the participation points.

  8. Working session #3

    The teams work on their projects. Attendance is optional but rewarded in the participation points.

  9. Preliminary results and design guidelines

    Each team will informally present whatever results they have obtained until now. In case they have some particularly problematic situations that they have been unable to solve, others may contribute by suggesting possible solutions.

    Based on the results of the evaluation, we will begin to draft a rough set of design guidelines that based on the findings seem to be applicable for AR user interfaces.

  10. Structuring the final reports and presentations

    We spend the session going over the work that each team has completed and how to represent that in a final presentation as well as in a written report. We also identify any work that still needs to be done to successfully conclude the projects for the following week.

    It is extremely important to have correct spelling and grammar (including proper punctuation) in all written work, as well as a consistent style regarding tenses, the use of upper-case initials, etc. One point will be deducted per error and one may lose all the points even if the content is technically adequate if the written presentation does not meet general expectations for scientific writing.

    Also, prepare a decent, consistently formatted, and comprehensive bibliography and cite the sources adequately within the text body; simple lists of URLs are unacceptable. The cited sources are expected to be principally esteemed textbooks, journal articles, and conference proceedings. Each inadequate or missing reference will also cost a point, as well as each obscurely formatted reference.

  11. Final presentations

    Each group presents the findings of their evaluations and the recommendations regarding the evaluated AR prototype (and possible conclusions regarding usability considerations for AR applications in general) with a slide show, accompanied hopefully by informative video clips, and turns in a (first draft of a) written final report on the work they have carried out during the course.

    You are encouraged to invite colleagues and fellow students to attend this session.

  12. Feedback and future work

    In the final session we discuss how we could have improved the projects and the course itself. We also discuss how to continue with the work we initiated, for example in terms of coauthoring a book on the topic or preparing articles for journals or conferences. Also, report grading is carried out.

    The written final reports (possibly improved after last session) are graded.

Final reports

The final reports are posted with the students' permission in PDF format. They were later edited into a single volume, Usability Evaluation for Augmented Reality.


The first four are evaluated as a team and the entire team is expected to contribute equally. Should some of the team members be more deserving of points than others, the team must include in their report a clearly written explanation of this and a suggestion of how the effort has been divided as a percentage; this is the only way in which some team members can receive more points than others on these four aspects. The last two are individually graded.

There are no examinations.

NameParticipation Project work TotalGrade
Session (12 x 1)Online (6 x 3)Talk (10)Report (15+20+25)
Lei 0+0+1+1+1+1+1+1+1+1+1+1=10 2+0+0+1+1+0=4 8 13+17+23=53 75 3
Maninder 1+1+0+1+1+1+1+1+1+1+1+1=11 3+1+2+1+1+1=9 81 4
Aki 1+1+1+1+1+1+1+0+0+1+1+1=10 1+0+0+0+0+2=3 7 13+17+22=52 72 3
Jorma 1+1+1+1+1+1+1+0+1+0+1+0=9 0+0+0+0+1+0=1 69 2
Payel 1+1+1+1+1+1+1+1+1+1+1+1=12 3+3+3+3+3+3=18 9 14+19+24=57 96 5
Héctor 1+1+1+1+1+1+1+1+1+1+1+1=12 3+2+3+2+3+3=16 94 5
Ondřej 1+1+1+0+1+0+1+0+1+1+1+1=9 1+1+0+0+1+0=3 8 15+19+22=56 76 3

The grade limits are adjusted at the end of the course and are, in terms of the accumulated total (T):

Updated on June 26, 2014.