Wiki source code of Measuring Instruments

Version 4.5 by Sofia Kostakonti on 2022/04/03 15:20

Hide last authors
Veikko Saikkonen 2.1 1
Sofia Kostakonti 4.3 2 For the evaluation of a prototype, there are several frameworks that can be followed, starting with DECIDE[1]. Decide stands for:
Veikko Saikkonen 2.1 3
4 **D**etermine the goals
5 **E**xplore the questions
6 **C**hoose evaluation approach and methods
7 **I**dentify practical issues
8 **D**ecide about ethical issues
9 **E**valuate, analyze, interpret, present data
10
Sofia Kostakonti 4.3 11 First, we would have to determine the high-level goals for the study and the motivation behind them, since they can influence how we approach it. Then, we choose the evaluation approach, the methods that will be used, whether these are based on quantitative or qualitative data, and the process of data collecting, analysis, and presentation. At the same time, any practical issues, such as participants, budget, or schedule, are identified and a pilot study is performed if needed. It is important to adhere to any ethical procedures that are in place, to ensure the participant knows their rights and is protected. Finally, the evaluation of the data takes place, where it is determined whether the results are reliable, valid, without bias, unrelated to the environment and can generalize well.
12 Another framework used is IMPACT[2]:
Veikko Saikkonen 3.1 13 == IMPACT ==
Veikko Saikkonen 2.1 14
Veikko Saikkonen 3.1 15 **I**ntention: Present the objectives and claims
Sofia Kostakonti 4.3 16 **M**easures and metrics: "What, how, and why"
Veikko Saikkonen 3.1 17 **P**eople: Define the participants
Sofia Kostakonti 4.3 18 **A**ctivities: Use cases in activities
Veikko Saikkonen 3.1 19 **C**ontext: Social, ethical, physical, etc. environment definition
20 **T**echnologies: Hardware and software
21
22
Veikko Saikkonen 3.2 23 = Evaluation methods =
Veikko Saikkonen 3.1 24
Veikko Saikkonen 3.2 25 == Formative evaluation ==
26
Sofia Kostakonti 4.3 27 - Open-ended evaluation of the design
Veikko Saikkonen 3.2 28 - E.g. How will the users respond to the new design?
29
30
31 == Summative evaluation ==
32
33 - Focus on the overall effect
34 - Summarizes if the objective is reached
35 - E.g. Are the participants happier when working with design X in comparison to design Y?
36
37 == Data ==
38
39 - Qualitative: Explore, discover, instruct
40
41 - Quantitative: Describe, explain, predict
42
43 - Subjective quantitative
44
45
46 == Statistics ==
47
48 - Descriptive: Describe the dataset, e.g. mean time on task
49 - Inferential: Using a sample to infer about a population, e.g. predicted mean time on task based on user characteristics.
50
51 == Experiment Design: Conditions ==
52
53 === Within Subjects (Repeated Measures) ===
54 Each participant is subjected to all the test conditions.
55 Fewer subjects needed and reduces variance in the Can be difficult to setup due to subjects fatiguing, learning about the setup or simply not having enough time.
56
57 === Between Subjects (Between Groups) ===
58 One subject undergoes only one test. Simple to execute, but results in significant variance due to inter-subject differences in characteristics.
59
60
Veikko Saikkonen 4.1 61 == Lenses ==
62 - Lense means to take different perspectives looking at your system
63 - E.g. perspective of the stakeholders, other groups, or technical/legal
Veikko Saikkonen 3.2 64
Veikko Saikkonen 4.1 65
Sofia Kostakonti 4.3 66 [1] Kurniawan, S. (2004). Interaction design: Beyond human–computer interaction by Preece, Sharp and Rogers (2001), ISBN 0471492787.
Veikko Saikkonen 4.1 67