Changes for page 4. Evaluation Methods

Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -1,19 +1,35 @@
1		-The design evaluation aims to test and validate the system’s design, or to discriminate between multiple design options, such that the current design can be improved upon in incremental development cycles. The SCE method describes two parts that are relevant with respect to the system evaluation: (1) the prototype and/or simulation, and (2) the evaluation that describes the evaluation method and results.
	1	+There are two types of evaluation methods: formative and summative evaluation. Formative evaluation is based on open-ended questions that focus on specific interaction processes, while summative evaluation looks at the overall effect and determines whether the objective has been achieved. Both qualitative and quantitative data can be used to measure these evaluations. Qualitative data is used to explore and identify patterns and themes, while quantitative data is used to describe, explain, and predict outcomes. Combining both types of data is often the best approach to evaluation.
2	2
	3	+What to measure to assess effects?
3	3
4		-There are several frameworks available for evaluating a prototype, one of which is DECIDE (Kurniawan, 2004), which stands for:
	5	+* Objective measurements
	6	+** Efficiency: time
	7	+** Effectiveness: performance outcomes (errors, restarts, ...)
	8	+* Subjective measurements
	9	+** Satisfaction, pleasure/well-being, mood, excitement, likability
	10	+* Validated questionnaires
	11	+** System Usability Scale (SUS)
	12	+** Affect Button
	13	+** Godspeed questionnaire
5	5
6		-* **D**etermine the goals
7		-* **E**xplore the questions
8		-* **C**hoose evaluation approach and methods
9		-* **I**dentify practical issues
10		-* **D**ecide about ethical issues
11		-* **E**valuate, analyze, interpret, and present data
	15	+Subjective measurements and questionnaires are best fit to evaluate the project. A set of questions will be formulated and used in a questionnaire that participants can fill in after the experiment. The questions can be found [[here>>doc:3\. Evaluation.b\. Test.Questionaire Questions.WebHome]].
12	12
13		-To begin, the high-level goals of the study and the underlying motivation behind them should be determined, as these factors can influence the approach taken. Next, the evaluation approach and methods should be selected, taking into account whether they will be based on quantitative or qualitative data, and how the data will be collected, analyzed, and presented. Any practical issues, such as participant recruitment, budget, or scheduling, should also be identified, and a pilot study may be conducted if necessary. It is crucial to follow ethical procedures to ensure that participants are aware of their rights and are protected. Finally, the data should be evaluated to determine its reliability, validity, potential biases, environmental influences, and generalizability.
	17	+A trust score, as described in Gutalli et al. (2019) //(Design, development and evaluation of a human-computer trust scale)//, the effect on the mood of the participant was measured using a questionnaire. The questionnaire consisted of sub-questions related to these aspects and used a 1-5 Likert Scale to capture the level of agreement and feelings towards these aspects.
14	14
	19	+According to Gulati et al. (2019), the trust people have in robots consist of 4 different factors:
15	15
16		-There are two types of evaluation methods: formative and summative evaluation. Formative evaluation is based on open-ended questions that focus on specific interaction processes, while summative evaluation looks at the overall effect and determines whether the objective has been achieved. Both qualitative and quantitative data can be used to measure these evaluations. Qualitative data is used to explore and identify patterns and themes, while quantitative data is used to describe, explain, and predict outcomes. Combining both types of data is often the best approach to evaluation.
	21	+//1) The Percieved Risk of the Robot~:// This indicates how cautious people feel they have to be around the robot, or how risky they feel it is to interact with the robot. This score inverted shows how much people trust a robot.
17	17
	23	+//2) The Benevolence of the Robot: //This score shows how much people think a robot will act in their best interests.
18	18
	25	+//3) The Competence of the Robot: //This shows how well people think the robot is fit for its job.
	26	+
	27	+//4) The Reciprocity of the Robot: //The Reciprocity score indicates how much people feel a connection with the robot.
	28	+
	29	+//Mood Score~://
	30	+
	31	+Our Mood Score is derived from the Oxford Happiness Questionnaire //(Hills et al. ,The Oxford Happiness Questionnaire: a compact scale for the measurement of psychological well-being, (2002))//. The Oxford Happiness Questionnaire correlates with personality variables like satisfaction with life, self-esteem and happiness. This score can be used to measure the effect of the interaction with Dogg0 on people's happiness.
	32	+
	33	+
	34	+
19	19	~1. Kurniawan, S. (2004). Interaction design: Beyond human-computer interaction by Preece, Sharp, and Rogers (2001), ISBN 0471492787.