Skip to Content

Changes for page 4. Evaluation Methods

Last modified by Mohamed Elsayed on 2023/04/11 15:15
From version 3.1
edited by Mohamed Elsayed
on 2023/04/07 19:59
Change comment: There is no comment for this version
To version 6.1
edited by Mohamed Elsayed
on 2023/04/11 14:11
Change comment: There is no comment for this version

Summary

Details

Page properties
Content
... ... @@ -1,6 +4,3 @@
1 -The design evaluation aims to test and validate the system’s design, or to discriminate between multiple design options, such that the current design can be improved upon in incremental development cycles. The SCE method describes two parts that are relevant with respect to the system evaluation: (1) the prototype and/or simulation, and (2) the evaluation that describes the evaluation method and results.
2 -
3 -
4 4  There are several frameworks available for evaluating a prototype, one of which is DECIDE (Kurniawan, 2004), which stands for:
5 5  
6 6  * **D**etermine the goals
... ... @@ -10,10 +10,20 @@
10 10  * **D**ecide about ethical issues
11 11  * **E**valuate, analyze, interpret, and present data
12 12  
13 -To begin, the high-level goals of the study and the underlying motivation behind them should be determined, as these factors can influence the approach taken. Next, the evaluation approach and methods should be selected, taking into account whether they will be based on quantitative or qualitative data, and how the data will be collected, analyzed, and presented. Any practical issues, such as participant recruitment, budget, or scheduling, should also be identified, and a pilot study may be conducted if necessary. It is crucial to follow ethical procedures to ensure that participants are aware of their rights and are protected. Finally, the data should be evaluated to determine its reliability, validity, potential biases, environmental influences, and generalizability.
10 +To begin, the high-level goals of the study and the underlying motivation behind them are determined, as these factors can influence the approach taken. Next, the evaluation approach and methods should be selected, taking into account whether they will be based on quantitative or qualitative data, and how the data will be collected, analyzed, and presented. Any practical issues, such as participant recruitment, budget, or scheduling, are identified, and a pilot study is conducted. It is crucial to follow ethical procedures to ensure that participants are aware of their rights and are protected. Finally, the data should be evaluated to determine its reliability, validity, potential biases, environmental influences, and generalizability.
14 14  
15 -
16 16  There are two types of evaluation methods: formative and summative evaluation. Formative evaluation is based on open-ended questions that focus on specific interaction processes, while summative evaluation looks at the overall effect and determines whether the objective has been achieved. Both qualitative and quantitative data can be used to measure these evaluations. Qualitative data is used to explore and identify patterns and themes, while quantitative data is used to describe, explain, and predict outcomes. Combining both types of data is often the best approach to evaluation.
17 17  
14 +What to measure to assess effects?
18 18  
16 +* Objective measurements
17 +** Efficiency: time
18 +** Effectiveness: performance outcomes (errors, restarts, ...)
19 +* Subjective measurements
20 +** Satisfaction, pleasure/well-being, mood, excitement, likability
21 +* Validated questionnaires
22 +** System Usability Scale (SUS)
23 +** Affect Button
24 +** Godspeed questionnaire
25 +
19 19  ~1. Kurniawan, S. (2004). Interaction design: Beyond human-computer interaction by Preece, Sharp, and Rogers (2001), ISBN 0471492787.