Changes for page Measuring Instruments

Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -1,36 +1,67 @@
1	1
2	2	For the evaluation of a prototype, there are several frameworks that can be followed, starting with DECIDE[1]. Decide stands for:
3	3
4		-* **D**etermine the goals
5		-* **E**xplore the questions
6		-* **C**hoose evaluation approach and methods
7		-* **I**dentify practical issues
8		-* **D**ecide about ethical issues
9		-* **E**valuate, analyze, interpret, present data
	4	+**D**etermine the goals
	5	+**E**xplore the questions
	6	+**C**hoose evaluation approach and methods
	7	+**I**dentify practical issues
	8	+**D**ecide about ethical issues
	9	+**E**valuate, analyze, interpret, present data
10	10
11		-First, we would have to determine the high-level goals for the study and the motivation behind them, since they can influence how we approach it. Then, we choose the evaluation approach, the methods that will be used, whether these are based on quantitative or qualitative data, and the process of data collecting, analysis, and presentation. At the same time, any practical issues, such as participants, budget, or schedule, are identified and a pilot study is performed if needed. It is important to adhere to any ethical procedures that are in place, to ensure the participant knows their rights and is protected. Finally, the evaluation of the data takes place, where it is determined whether the results are in any way unreliable, invalid, biased, related to the environment and whether they generalize well.
	11	+First, we would have to determine the high-level goals for the study and the motivation behind them, since they can influence how we approach it. Then, we choose the evaluation approach, the methods that will be used, whether these are based on quantitative or qualitative data, and the process of data collecting, analysis, and presentation. At the same time, any practical issues, such as participants, budget, or schedule, are identified and a pilot study is performed if needed. It is important to adhere to any ethical procedures that are in place, to ensure the participant knows their rights and is protected. Finally, the evaluation of the data takes place, where it is determined whether the results are reliable, valid, without bias, unrelated to the environment and can generalize well.
	12	+Another framework used is IMPACT[2]:
	13	+== IMPACT ==
12	12
13		-Another framework often used is IMPACT[2]:
	15	+**I**ntention: Present the objectives and claims
	16	+**M**easures and metrics: "What, how, and why"
	17	+**P**eople: Define the participants
	18	+**A**ctivities: Use cases in activities
	19	+**C**ontext: Social, ethical, physical, etc. environment definition
	20	+**T**echnologies: Hardware and software
14	14
15		-* **I**ntention
16		-* **M**easures and metrics
17		-* **P**eople
18		-* **A**ctivities
19		-* **C**ontext
20		-* **T**echnologies
21	21
22		-These are the several elements that need to be considered when trying to establish evaluation objectives. First, we should present the objectives and the claims of the study. Furthermore, the specific measures and metrics that will be used need to be determined, followed by the participants and the activities they will perform based on a specific use case. We should also define the context, social, ethical, physical, or environmental, and finally, we must decide on the technologies we will use, both regarding hardware and software.
	23	+= Evaluation methods =
23	23
24		-Regarding the evaluation methods, there are two types: formative and summative evaluation. The formative evaluation is based on open-ended questions that have to do with specific processes of the interaction, whereas the summative evaluation focuses on the overall effect and summarizes whether the objective has been reached. For measuring these evaluations, both qualitative and quantitative data can be examined. The goal of the qualitative data is to explore and discover patterns and themes and that of the quantitative data is to describe, explain and predict based on the outcomes. A combination of the two is often optimal.
	25	+== Formative evaluation ==
25	25
26		-Another factor that needs to be considered during an evaluation study is the experiment design. There are two types: the within-subjects design and the between-subject design. The former calls for performing all the test conditions on all participants and getting repeated measures. This design needs fewer subjects and reduces the variance, however, there is the possibility of carry-over effects from one interaction to the next and the setup can be more challenging since it requires more time. The latter is performed on different groups of participants, each undergoing only one test condition. It is much simpler to execute but there might be greater variance due to inter-subject differences in characteristics.
	27	+- Open-ended evaluation of the design
	28	+- E.g. How will the users respond to the new design?
27	27
28		-When executing an experiment, it is also interesting to examine the interaction from multiple lenses, in order to identify issues and opportunities that are not immediately obvious. That can mean from the perspective of a different stakeholder that is not the main user, other groups that might not directly interact with the system or even a more technical or legal perspective.
29	29
	31	+== Summative evaluation ==
30	30
31		-For our study, the process we followed mostly resembles the IMPACT framework since we focused on each of these elements separately and tried to combine them to build our evaluation study.
	33	+- Focus on the overall effect
	34	+- Summarizes if the objective is reached
	35	+- E.g. Are the participants happier when working with design X in comparison to design Y?
32	32
	37	+== Data ==
33	33
	39	+- Qualitative: Explore, discover, instruct
	40	+
	41	+- Quantitative: Describe, explain, predict
	42	+
	43	+ - Subjective quantitative
	44	+
	45	+
	46	+== Statistics ==
	47	+
	48	+- Descriptive: Describe the dataset, e.g. mean time on task
	49	+- Inferential: Using a sample to infer about a population, e.g. predicted mean time on task based on user characteristics.
	50	+
	51	+== Experiment Design: Conditions ==
	52	+
	53	+=== Within Subjects (Repeated Measures) ===
	54	+Each participant is subjected to all the test conditions.
	55	+Fewer subjects needed and reduces variance in the Can be difficult to setup due to subjects fatiguing, learning about the setup or simply not having enough time.
	56	+
	57	+=== Between Subjects (Between Groups) ===
	58	+One subject undergoes only one test. Simple to execute, but results in significant variance due to inter-subject differences in characteristics.
	59	+
	60	+
	61	+== Lenses ==
	62	+- Lense means to take different perspectives looking at your system
	63	+- E.g. perspective of the stakeholders, other groups, or technical/legal
	64	+
	65	+
34	34	[1] Kurniawan, S. (2004). Interaction design: Beyond human–computer interaction by Preece, Sharp and Rogers (2001), ISBN 0471492787.
35		-[2] Benyon, D., Turner, P., & Turner, S. (2005). Designing interactive systems: People, activities, contexts, technologies. Pearson Education.
36	36