Wiki source code of Test

Version 82.1 by Aleksanteri Hämäläinen on 2022/04/03 17:48

version	line-number	content
2.1	1	= Problem statement and research questions =
1.1	2
9.1	3	People with dementia often forget to eat and drink, leading to dehydration, malnutrition and decreased wellbeing in general. Our prototype engages in discourses to remind PwD to have lunch and drink water, using the Nao robot platform. The discourse aims to reming the PwD without causing any anxiety or embarrassment which a traditional "alarm" system could cause, and keep them company throughout these activities.
1.1	4
73.1	5	The four research questions studied in this evaluation are:
3.1	6
77.1	7	{{html}}
74.1	8
78.1	9	<ol>
80.1	10	<li>* Does the robot cause PwD to eat more regularly?</li>
79.1	11	<li>Does the robot remind the PwD of their hunger?</li>
	12	<li>Does the music make the eating more enjoyable for the PwD?</li>
	13	<li>Does the PwD experience less negative emotions, such as agitation, sadness, embarrassment, after the interaction with the 'intelligent' robot?</li>
78.1	14	</ol>
77.1	15
	16	{{/html}}
	17
82.1	18	\*\ This research question is difficult due to the practical limitations in designing the experimental setup and as such is left to lesser importance.
73.1	19
2.1	20	= Method =
1.1	21
9.1	22	The prototype is evaluated with an in-person experiment with multiple participants. In the experiment, the participants will be asked to pretend to be PwD and act accordingly with/without the prototype.
1.1	23
2.1	24	== Participants ==
1.1	25
9.1	26	As there are practical difficulties with conducting the experiment with actual people with dementia due to both time constraints and COVID, our participants' group will consist of peers from other groups and friends, who will act as if they are older people with dementia. We plan to gather around 20 people for our experiments.
1.1	27
2.1	28	== Experimental design ==
1.1	29
9.1	30	We will be using a within-subject design. In the experiment all of the participants will interact with both versions of the robot, with half of the participants interacting with the version 1 first and then version 2, and the other half in reverse order, to counter-balance the carryover effects. Snacks will be made available for the participants, in case they're prompted and they're hungry. The participants will be unaware of the possibility of eating snacks, to prevent disturbing the interaction with the robot. Otherwise the subjects could be primed for eating, which would bias the results and hide the effect of the robotic interaction.
1.1	31
2.1	32	== Tasks ==
1.1	33
9.1	34	The participant will have to interact with the robot, which is programmed to engage in a lunch discourse. Two versions will be implemented: the first version will ask basic questions about mealtime, mostly acting as a reminder for the PwD to have lunch (alarm clock). The second will be our original implementation of it with the more sophisticated discourse and music.
1.1	35
2.1	36	== Measures ==
1.1	37
9.1	38	We plan on measuring the effectiveness of the discourse, both physically and emotionally. Our quantitative measure is whether the person ate the lunch they were supposed to have eaten, and the qualitative measure is the emotions that the PwD experienced before, during, and after the interaction. The qualitative measures will be recorded with a simple questionnaire. Depending on the time of the experiments, we assume that people might also not be hungry enough to be prompted to have something to eat, which might disturb the results. We do plan however to measure whether the robot will remind someone of their hunger and have them eat.
1.1	39
2.1	40	== Procedure ==
1.1	41
4.1	42	* Welcome Participants and explain what they are going to be doing.
	43	* Have them sign the permission form.
	44	* Participants complete a questionnaire(A) regarding their emotional state (control).
	45	* Have version A of interaction with the robot.
	46	* Complete questionnaire(extended version).
	47	* Have a short interview during downtime (prepared questions).
	48	* Have version B of interaction with the robot.
	49	* Complete questionnaire(extended version).
	50	* Have a short interview during downtime (prepared questions).
1.1	51
2.1	52	== Material ==
1.1	53
9.1	54	For the experiments, we'll be using the NAO robot platform, as well as a laptop for the participants to complete the questionnaires on. The questionnaire will be a combination of questions regarding the emotional state of the participants, their interaction with the robot, and the music included in the interaction. Food will be made available to see and measure how much people will eat.
1.1	55
20.1	56	Questionnaires:
	57	Consent Form and Disclaimers
	58	8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire
	59	4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire
	60	3 food-related questions of our own (5-point Likert scale)
	61	2 music-related questions of our own (5-point Likert scale)
	62
11.1	63	== Practicalities ==
1.1	64
11.1	65	Beforehand:
17.1	66
11.1	67	* Do a practice round by ourselves
	68	** Film this
	69	* Contact other groups and decide on a time slot
	70	** Might be better to reserve in 10 min slots, so that people don't have to wait so much
	71	** If possible, this could be done in parallel with another groups testing
	72	* Reserve lab
	73	* Buy snacks
	74
	75	During:
17.1	76
11.1	77	1. Give starting questionnare to fill while people are waiting for the previous participant
	78	2. Guide the participant to the testing spot
	79	3. Inform the participant where the snacks are
	80	4. Run the first version
	81	5. Give the mid-questionnare
	82	6. Run the other test
13.1	83	7. Conduct the questionnare for the participant
	84	8. Give the participant the end-questionnare
11.1	85
17.1	86	Other practicalities during:
11.1	87
	88	* We will use the "Wizard of Oz" method for recognizing agreement and disagreement, to make sure that the whole process does not depend on voice recegnition being good enough
	89	** Someone will press eg. "y" and "n" on the keyboard according to the participants answers
	90	* We will change the order in which the smart and basic versions are for each participant
	91	** this way if someone doesn't show up, we don't get skewed amounts
	92
	93	After:
17.1	94
11.1	95	* Analyze results
	96
35.1	97	= Results =
1.1	98
58.1	99	The results were gathered from 19 personnel, all of whom interacted first with one version of the robot and then the other. 10 of the participants interacted first with the simple version, nine having their first interaction with the advanced version.
	100
41.1	101	== Eating ==
	102
	103	{{html}}
44.1	104	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/EatingComp.png?rev=1.1" alt="Results on the eating of the test personas" style="display:block;margin-left:auto;margin-right:auto" width=750/>
41.1	105	{{/html}}
	106
	107	(% style="text-align:center" %)
62.1	108	Figure 1: Results on the eating of the test personas during the experiment
41.1	109
59.1	110	Simple robot:
	111
	112	* 16% ate
	113	* 33% of those would not have eaten without the robot
	114
	115	Advanced robot:
	116
	117	* 32% ate
	118	* 67% of those would not have eaten without the robot
	119
66.1	120	== Music ==
	121
	122	{{html}}
	123	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/MusicEnjoyable.png?rev=1.1" alt="Effects of music on the test personnel" style="display:block;margin-left:auto;margin-right:auto" width=1250/>
	124	{{/html}}
	125
	126	(% style="text-align:center" %)
67.1	127	Figure 2: Answers of the test personas regarding music
66.1	128
	129
51.1	130	== EVEA (Mood) ==
29.1	131
	132	{{html}}
45.1	133	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/MoodChangeDumb.png?rev=1.1" alt="Measured moods and changes for the simple robot" style="display:block;margin-left:auto;margin-right:auto" width=750/>
29.1	134	{{/html}}
	135
32.2	136	(% style="text-align:center" %)
67.1	137	Figure 3: Median measured moods for the simple version of the robot
30.1	138
32.2	139	{{html}}
45.1	140	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/MoodChangeSmart.png?rev=1.1" alt="Measured moods and changes for the advanced version of the robot" style="display:block;margin-left:auto;margin-right:auto" width=750/>
32.2	141	{{/html}}
	142
	143	(% style="text-align:center" %)
67.1	144	Figure 4: Median measured moods for the advanced version of the robot
29.1	145
33.1	146	(% style="text-align:center" %)
62.1	147	Table 1: Wilcoxon signed rank test results for the hypothesis that the mood changed during the interaction with the simple robot
37.1	148
	149	\|=Mood\|=Happiness\|=Anxiety\|=Sadness\|=Anger
	150	\|Statistic\|37\|5\|4\|14
	151	\|P-value\|0.54\|0.01\|0.01\|0.45
	152
	153	(% style="text-align:center" %)
62.1	154	Table 2: Wilcoxon signed rank test results for the null hypothesis that the mood changed during the interaction with the advanced robot
37.1	155
	156	\|=Mood\|=Happiness\|=Anxiety\|=Sadness\|=Anger
49.1	157	\|Statistic\|32\|11\|2\|17
	158	\|P-value\|0.18\|0.01\|0.01\|0.45
37.1	159
	160	(% style="text-align:center" %)
62.1	161	Table 3: Wilcoxon signed rank test results for the null hypothesis that the mood decreased during the interaction with the simple robot
33.1	162
35.1	163	\|=Mood\|=Anxiety\|=Sadness\|=Anger
	164	\|Statistic\|81\|53\|29
38.1	165	\|P-value\|0.01\|0.00\|0.23
33.1	166
34.1	167	(% style="text-align:center" %)
62.1	168	Table 4: Wilcoxon signed rank test results for the null hypothesis that the mood decreased during the interaction with the advanced robot
33.1	169
35.1	170	\|=Mood\|=Anxiety\|=Sadness\|=Anger
36.1	171	\|Statistic\|32\|149\|52
38.1	172	\|P-value\|0.00\|0.01\|0.07
34.1	173
35.1	174	(% style="text-align:center" %)
62.1	175	Table 5: Wilcoxon signed rank test results for the null hypothesis that the mood increased during the interaction with the simple robot
34.1	176
35.1	177	\|=Mood\|=Happiness
	178	\|Statistic\|37
38.1	179	\|P-value\|0.27
35.1	180
	181	(% style="text-align:center" %)
62.1	182	Table 6: Wilcoxon signed rank test results for the null hypothesis that the mood increased during the interaction with the advanced robot
35.1	183
	184	\|=Mood\|=Happiness
	185	\|Statistic\|32
38.1	186	\|P-value\|0.09
35.1	187
69.2	188	(% style="text-align:center" %)
	189	Table 7: Wilcoxon signed rank test results for the hypothesis that the mood changes with the simple and advanced robots during the interaction are different
	190
	191	\|=Mood\|=Happiness\|=Anxiety\|=Sadness\|=Anger
	192	\|Statistic\|92\|49\|85\|69
	193	\|P-value\|0.92\|0.07\|0.71\|0.31
	194
52.1	195	== Godspeed ==
	196
40.1	197	{{html}}
46.1	198	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/friendly-hist.png?rev=1.1" alt="Effects of music on the test personnel" style="display:block;margin-left:auto;margin-right:auto" width=750/>
41.1	199	{{/html}}
40.1	200
41.1	201	(% style="text-align:center" %)
62.1	202	Figure 5: Answers to the statement 'I thought the robot was friendly'
41.1	203
40.1	204	{{html}}
46.1	205	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/pleasant-hist.png?rev=1.1" alt="Answers to the statement 'I thought the robot was pleasant'." style="display:block;margin-left:auto;margin-right:auto" width=750/>
40.1	206	{{/html}}
	207
	208	(% style="text-align:center" %)
62.1	209	Figure 6: Answers to the statement 'I thought the robot was pleasant'
40.1	210
54.1	211	{{html}}
	212	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/godspeed-barchart.png?rev=1.1" alt="Godspeed questionnaire median comparison'." style="display:block;margin-left:auto;margin-right:auto" width=750/>
	213	{{/html}}
1.1	214
54.1	215	(% style="text-align:center" %)
62.1	216	Figure 7: Median measured Godspeed questionnaire dimensions
41.1	217
64.1	218
	219	(% style="text-align:center" %)
71.1	220	Table 8: Wilcoxon signed rank test results for the null hypothesis that the advanced robot scored higher in the perceived dimensions
64.1	221
56.1	222	\|=Dimension\|=Likeability\|=Intelligence
55.1	223	\|Statistic\|36\|70
	224	\|P-value\|0.01\|0.17
52.1	225
72.1	226	= Discussion =
	227	Analysis the results surfaced some minor issues in the experiment, such as the lack of comparison with two robots of similar features, with and without music. Also the practical limitations in the setup, such as the lack of different food options and some participants being aware of the design goals of the prototype could have interfered with the natural flow of the intercourse. With these limitations, the research method was successful in extracting differences within the robots and brought up additional directions for future research.
	228
	229	The most interesting direction for future research would be the longer term studying of the effect of mealtime reminders on the health of the test subjects. The longer term health study would uncover the effect on eating frequency and the development of the relationship with the robot, for example would the test subjects that were first excited about the novel interaction with the robot, develop negative feelings about the supervision that the robot is conducting into their personal life.
	230
	231	Another topic to study is the differences with and without music. The effects of music could be studied with the music tailored to personal taste and all versions of the robot with and without the music playback included in the interaction. This would allow to pinpoint the effects of music, without the other features causing variance.
	232
57.1	233	= Conclusions =
54.1	234
62.2	235	From the results we can see that the more advanced robot shows advantages over the simple version in many categories. Hints of better performance in other categories can be seen, but no conclusions should be drawn from the ones that lack the statistical significance.
54.1	236
64.2	237	In improving the eating, it seems that both robots have limited success in causing the people to eat as seen in Figure 1, they could cause the patients to eat more regularly, if triggered by timers or other suitable systems. It also seems that the advanced robot is better in the reminding, by a slight margin. However, the long term effects of reminding should be researched more to conclude whether the usage of the demonstrated robot platform or similar would cause the patients to eat more regularly. It is also unclear how the test setup and the limited choice of food affected the eating.
55.1	238
68.1	239	Based on the answers of the participants regarding music seen in Figure 2, it seems that most of them were either indifferent or liked the music. Also, as the test personnel find the advanced robot more likeable with a 5% confidence limit (Table 7), and the advanced version was the only version with music, it seems likely that the music does make the interaction more pleasant for the personas. However, some of the likeability might be due to the other advanced features of the robot and thus more research is needed to conclude the effect of the music.
64.2	240
71.1	241	The EVEA and partial Godspeed result can be seen in Figures 3-7 and Tables 1-8. The results show that with reasonable confidence (5% confidence limit), both versions of the robot decreased sadness and anxiety in the test personas. Hints are shown (10% confidence limit) that the advanced robot also decreases feelings of anger and increases happiness, while the simple robot fails to show similar results. However, in Table 7 we can see that the statistical differences in the mood differences during the interactions with the different versions are not highly significant.
64.2	242
71.1	243	A Wilcoxon signed rank test for the partial Godspeed test shows in Table 8 that with high confidence (1% confidence limit), the intelligent robot is more likeable in comparison to the simple robot. With these results it is likely that the more advanced robot is slightly preferrable and the personas might experience less negative emotions after the interaction with the robots, but it is slightly unclear if the effect is more powerful with the advanced robot.
68.1	244
	245
10.1	246	= Appendix =
	247
17.1	248	== Experiment introduction for participants ==
16.1	249
17.1	250
16.1	251
17.1	252	Hi, we are <NAME> and <NAME> from the TU Delft Socio-Cognitive Engeering course Group 1, thank you for participating in our prototype evaluation experiment. The experiment is being conducted as a part of the TU Delft course on Socio-Cognitive Engineering and aims to evaluate the prototype designed as a part of the course. The evaluated prototype is based on the Nao robot-platform and is intended to improve the wellbeing of people suffering of dementia.
16.1	253
19.1	254	Consuming food and/or water can be a consequence of the interaction between you and the robot. Therefore, we would like to ask you if you have any allergies. If you have a form of Diabetes, please let us know before we start the first part of the experiment. You are strongly encouraged to share any other health conditions that can possibly be relevant to take into account when doing an experiment with robots and food with us.
	255
	256	The link between the stimuli of the Nao-robot and the triggering of epileptic seizures is yet unknown. If you have ever experienced epileptic seizures, please let us know. Then, we could see if any special precautions are needed.
	257
18.1	258	The experiment will last for approximately 15-20 minutes, and consists of two interaction sections with the Nao robot, as well as questionnaires before, between and after the sections. We kindly ask you to act naturally during the experiment and fill the questionnaires truthfully and intuitively. Remember that we are evaluating the prototypes performance, not yours. You can stop the experiment at any time.
17.1	259
	260	We will be collecting data of the questionnaires and recording some experiments, do you agree with your experiment being recorded? All data excluding the recordings will be anonymised before analysis and storage. The recordings will not be shared with third parties. After the experiment you have the right to ask for information about the collected data and revoke the right to use it. We kindly ask you not to share any information about the experiment with other participants.
16.1	261	Do you have any questions?
	262
10.1	263	== After research interview ==
	264
	265	Setup:
	266	The test subject has finished both parts of the experiment. Before leaving the test conductor(s) sit down with them and ask the following questions in a discussion about the experiment. Discussion can flow freely, but the following topics should be discussed.
	267
	268	Topics:
	269	- Emotions before / during / after the interaction with the robot
	270	- Agitation due to the robot suggesting eating
	271	- Effect of music on the general feeling of the situation
	272	- Feeling of company during eating
	273	- Effectiveness of eating/drinking suggestions
	274
12.1	275	Questions:
	276	- Did you eat or drink anything during the experiment?
	277	- Were you feeling hungry/thirsty beforehand and did the discourse change this?
	278	- On a scale of 1-10, how likely would you have eaten/drank without the robot suggesting it?
10.1	279	- What did the interaction with the robot feel like?
	280	- With the more intelligent version?
	281	- With the less intelligent version?
	282	- What did you feel like when the robot suggested you should eat/drink?