Changes for page 4. Evaluation Methods

Summary

• Page properties (1 modified, 0 added, 0 removed)

Details

Page properties

Content

...	...	@@ -1,24 +1,28 @@
1		-The following steps will be used to design and evaluate the prototype proposed against the corresponding control condition:
	1	+The following steps were used to design and evaluate the prototype proposed against the corresponding control condition:
2	2
3		-~1. Confirm the prototype: The prototype for the scenario to be tested, and the control situation will first be setup, and preliminary testing will be done by the team members. This includes the robots with and without interactive storytelling which should be confirmed and working.
	3	+~1. Confirm the prototype: For the pilot study, the scenario to be tested, and the control situation were setup at the Insyght Lab at TU Delft, and preliminary testing was done by the team members. This includes the robots with and without interactive storytelling which were confirmed and working. The voice input and touch input to the robot were verified.
4	4
5		-2. Develop Questions:
	5	+2. Develop Questions: We now develop the metrics on which the robot must be evaluated. We decided to use a modified version of the Godspeed questionnaire, which each participant was made to fill after interacting with the robot. This questionnaire has been elaborated below.
6	6
7		-3. Design Methods
	7	+3. Invite participants: Due to limited time and resources, patients with dementia (the actual users) could not be used for the study. We instead use TU Delft students to test the prototype.
8	8
9		-4. Implement and adapt:
10	10
11		-5. Make decisions:
12		-
13	13	**Research Question**
14	14
15		-"Is interactive storytelling more engaging and beneficial than storytelling in the third person for persons suffering from dementia?"
	12	+"Can personalized, interactive storytelling have a positive effect on the well-being of people with dementia?"
16	16
	14	+Sub RQ1: Does it improve the patient’s mood?
	15	+Sub RQ2: Does it spark interactions with other people?
	16	+Sub RQ3: Does it motivate them to complete their daily activities?
	17	+Sub RQ4: Does it promote memory retention?
	18	+Sub RQ5: Does it improve the storytelling experience?
	19	+
17	17	Thus, our control situation is the scenario of a robot narrating a story without any involvement of the patient, and the scenario we want to evaluate is the one where the robot narrates the same story while trying to engage and take inputs from the patient. With this, we aim to find whether it is beneficial and engaging for patients with dementia.
18	18
	22	+
19	19	**The Within-Subject Design**
20	20
21		-As part of the experiment design, we chose the within subject design over between subject. This means that each participant will interact with the robot twice. This was done due to the limited number of participants, and to avoid any biases of participant preferences.
	25	+As part of the experiment design, we chose the within subject design over between subject. This means that each participant will interact with the robot twice. This was done due to the limited number of participants, and to avoid any biases of participant preferences. However, we ensure that the order of talking to each robot changes with the participant, i.e, half the participants talk to the robot in the control situation first and then the robot in the experimental situation. For the other half this order is reversed. This was done to avoid carry over effects.
22	22
23	23
24	24	**Summative Evaluation**
...	...	@@ -28,11 +28,21 @@
28	28
29	29	**Questionnaire**
30	30
31		--modified godspeed questionnaire for robot
	35	+We used a modified version of the Godspeed questionnaire for our evaluation [1]. It measures the **anthropomorphism, animacy, likeability, intelligence, and safety** of the robot. This uses a Likert scale where the user must rate questions as a number between 1 and 5; both numbers being at opposite poles. We decided to go ahead with the Godspeed questionnaire, because in dealing with patients with dementia, it seemed relevant to measure the above mentioned 5 characteristics of the robot, as they play an important role in making the patient feel more comfortable and at ease.
32	32
33		--statistical test (p value) for evaluation
	37	+To measure whether patients with dementia completed the activity they were meant to do, and to evaluate whether storytelling made a difference to their meal, we added the following questions:
34	34
	39	+1. Please rate the question according to the following attributes. - Mood of the patient after the activity. (Scale of 1 to 5)
35	35
	41	+2. Please rate the question according to the following attributes. - Patient's feedback about the story experience (Scale of 1 to 5)
	42	+
	43	+3. Please rate the question according to the following attributes. - Patient's enjoyment (Scale of 1 to 5)
	44	+
	45	+4. Did the patient complete the activity? (Yes/No)
	46	+
	47	+5. How many minutes did the patient take to complete the activity? (<10 minutes, 10-25 minutes, 25-40 minutes, >40 minutes)
	48	+
	49	+
36	36	**Prototype**
37	37
38	38	We present a low fidelity prototype of the robot, which means a simple demonstration of the initial stages of the robot, meant for formative feedback. We wizard-of-oz the approach, and for now just present one story (in interactive and non interactive modes) for purposes of the experiment. The final robot is expected to have various templates of stories.
...	...	@@ -40,7 +40,10 @@
40	40	For prototyping, we will use incremental prototyping, which means adding features one by one and testing for each. We start with the most basic feature, complete a cycle of testing, and then add on new features to create new versions of the prototype. For the robot, we will first build the non interactive storytelling robot, then add music to it, and then add gestures. With each stage, we test the working of it, and if working as expected, we will move on to adding the next feature.
41	41
42	42
	57	+**Evaluation of Results**
43	43
44		-**Since we don't have many participants, should we skip the statistical test? Can we just report average values of responses for both scenarios?**
	59	+We decided to use the **paired sampled t test** since the experiment was a **within subject** experiment. The **one tailed t test** was used since we want to find if one condition is better than the other. Though the one tailed t test is more powerful, it could be debatable whether it is better than the two tailed t test in this scenario, since with the one tailed t test, we assume already that the experimental scenario will perform better than the control scenario.
45	45
46		-**Questionnaire should be a formal one, or should we ask 4-5 questions through Pepper? Or both?**
	61	+
	62	+
	63	+[1]C. Bartneck, D. Kuli´c, E. Croft, and S. Zoghbi, “Measurement instruments for the anthropomorphism, animacy, likeability, perceived intelligence, and perceived safety of robots,” International Journal of Social Robotics, vol. 1, no. 1, p. 71–81, 2008.