Skip to Content

Changes for page Measuring Instruments

Last modified by Sofia Kostakonti on 2022/04/04 12:04
From version Icon 3.1 Icon
edited by Veikko Saikkonen
on 2022/03/01 11:46
Change comment: There is no comment for this version
To version Icon 4.3 Icon
edited by Sofia Kostakonti
on 2022/04/03 15:10
Change comment: (Autosaved)

Summary

Details

Icon Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.VSaikkonen
1 +XWiki.SKostakonti
Content
... ... @@ -1,46 +1,67 @@
1 1  
2 -= Frameworks =
2 +For the evaluation of a prototype, there are several frameworks that can be followed, starting with DECIDE[1]. Decide stands for:
3 3  
4 -== DECIDE ==
5 -
6 6  **D**etermine the goals
7 -- What are the high-level goals? Who wants it and why? The goals influence the approach of the study
8 -
9 9  **E**xplore the questions
10 -- Define the goals and reseach questions
11 -
12 12  **C**hoose evaluation approach and methods
13 -- Influences the data collection, analysis and presentation
14 -- Exploratory research => qualitative data, observational study,
15 -
16 16  **I**dentify practical issues
17 -- Users, budget, schedule, equipment etc.
18 -- Pilot study important
19 -
20 20  **D**ecide about ethical issues
21 -- Adhere to ethical procedure
22 -- User rights! Explain the goals (before/after), methodology etc.
23 -
24 24  **E**valuate, analyze, interpret, present data
25 -- Reliable results: Can the results be replicated?
26 -- Validity: Is the data related to the hypothesis?
27 -- Bias: Are the results unbiased?
28 -- Scope: Can this be generalised?
29 -- Ecological validity: Is the environment influencing the results?
30 30  
11 +First, we would have to determine the high-level goals for the study and the motivation behind them, since they can influence how we approach it. Then, we choose the evaluation approach, the methods that will be used, whether these are based on quantitative or qualitative data, and the process of data collecting, analysis, and presentation. At the same time, any practical issues, such as participants, budget, or schedule, are identified and a pilot study is performed if needed. It is important to adhere to any ethical procedures that are in place, to ensure the participant knows their rights and is protected. Finally, the evaluation of the data takes place, where it is determined whether the results are reliable, valid, without bias, unrelated to the environment and can generalize well.
12 +Another framework used is IMPACT[2]:
31 31  == IMPACT ==
32 32  
33 33  **I**ntention: Present the objectives and claims
16 +**M**easures and metrics: "What, how, and why"
17 +**P**eople: Define the participants
18 +**A**ctivities: Use cases in activities
19 +**C**ontext: Social, ethical, physical, etc. environment definition
20 +**T**echnologies: Hardware and software
34 34  
35 -**M**easures and metrics: "What, how and why"
36 36  
37 -**P**eople: Define the participants
23 += Evaluation methods =
38 38  
39 -**A**ctivities: Use cases into activities
25 +== Formative evaluation ==
40 40  
41 -**C**ontext: Social, ethical, physical, etc. environment definition
27 +- Open-ended evaluation of the design
28 +- E.g. How will the users respond to the new design?
42 42  
43 -**T**echnologies: Hardware and software
44 44  
31 +== Summative evaluation ==
45 45  
33 +- Focus on the overall effect
34 +- Summarizes if the objective is reached
35 +- E.g. Are the participants happier when working with design X in comparison to design Y?
46 46  
37 +== Data ==
38 +
39 +- Qualitative: Explore, discover, instruct
40 +
41 +- Quantitative: Describe, explain, predict
42 +
43 + - Subjective quantitative
44 +
45 +
46 +== Statistics ==
47 +
48 +- Descriptive: Describe the dataset, e.g. mean time on task
49 +- Inferential: Using a sample to infer about a population, e.g. predicted mean time on task based on user characteristics.
50 +
51 +== Experiment Design: Conditions ==
52 +
53 +=== Within Subjects (Repeated Measures) ===
54 +Each participant is subjected to all the test conditions.
55 +Fewer subjects needed and reduces variance in the Can be difficult to setup due to subjects fatiguing, learning about the setup or simply not having enough time.
56 +
57 +=== Between Subjects (Between Groups) ===
58 +One subject undergoes only one test. Simple to execute, but results in significant variance due to inter-subject differences in characteristics.
59 +
60 +
61 +== Lenses ==
62 +- Lense means to take different perspectives looking at your system
63 +- E.g. perspective of the stakeholders, other groups, or technical/legal
64 +
65 +
66 +[1] Kurniawan, S. (2004). Interaction design: Beyond human–computer interaction by Preece, Sharp and Rogers (2001), ISBN 0471492787.
67 +
Icon XWiki.XWikiComments[0]
Author
... ... @@ -1,0 +1,1 @@
1 +xwiki:XWiki.FrankBroz
Comment
... ... @@ -1,0 +1,1 @@
1 +Don't forget to fill in the specifics of your study.
Date
... ... @@ -1,0 +1,1 @@
1 +2022-03-22 08:33:36.109