Skip to Content

Measuring Instruments

Last modified by Xin Wan on 2022/03/02 12:06

Basics

Decide framework

DETERMINE THE GOALS

  • What are the high-level goals of the evaluation?
  • Who wants it and why? 
  • The goals influence the approach used for the study.
     

EXPLORE THE QUESTIONS

Define (sub)goals & (sub)research questions.

CHOOSE EVALUATION APPROACH & METHODS

The evaluation approach influences the methods used, and in turn,
how data is collected, analyzed and presented

IDENTIFY PRACTICAL ISSUES

For example: 

  • Select users. 
  • Stay on budget. 
  • Stay on schedule. 
  • Find participants. 
  • Select equipment.
    Perform a pilot (trial) study!

DECIDE ABOUT ETHICAL ISSUES

EVALUATE, ANALYZE, INTERPRET AND PRESENT THE DATA

The approach and methods used influence how data
is evaluated, analyzed, interpreted and presented.

IMPACT framework

  • Intention:           Clarify objectives and hypotheses/claims
  • Metrics & measures:  What, how and why
  • People:              Target group & participants
  • Activities:          Derive activities from use cases
  • Context:             Social, ethical, physical, etc. aspects
  • Technologies:        Hardware and software

Formative evaluation

 
Focuses on the various processes of the human-technology interaction.
Derive open questions from your design specification.

Summative evaluation

 
Focuses on the overall effects of the human-technology interaction.

Data

Data Type.png

Qualitative Data

• Words
• Drawings

Quantitative Data

• Numbers
• Statistics

Experiment Design: Conditions

Within Subjects

Each participant, all conditions

Between Subjects

Each participant, one condition

GOOD EVALUATION

  • Establishing convincing arguments for your design solution
  • By conducting complementary and regular evaluations at different stages
    of your design process using the appropriate evaluation methods  
    (e.g. summative, formative, expert-based, observational, ...) 
  • Evaluations should result in insights regarding possible problems and
    their causes in order to support refinement of your design specification 
  • Look at user experience in its full breadth: Effectiveness, efficiency,
    satisfaction, learnability, mood, connectedness, ...