Wiki source code of Test

Version 123.5 by Sofia Kostakonti on 2022/04/05 13:40

version	line-number	content
2.1	1	= Problem statement and research questions =
1.1	2
122.1	3	People with dementia often forget to eat and drink, leading to dehydration, malnutrition, and decreased well-being in general. Our prototype engages in discourses to remind PwD to have lunch and drink water, using the Nao robot platform. The discourse aims to remind the PwD without causing any anxiety or embarrassment that a traditional "alarm" system could cause, and keep them company throughout these activities.
1.1	4
73.1	5	The four research questions studied in this evaluation are:
3.1	6
77.1	7	{{html}}
74.1	8
78.1	9	<ol>
79.1	10	<li>Does the robot remind the PwD of their hunger?</li>
	11	<li>Does the music make the eating more enjoyable for the PwD?</li>
	12	<li>Does the PwD experience less negative emotions, such as agitation, sadness, embarrassment, after the interaction with the 'intelligent' robot?</li>
84.1	13	<li>* Does the robot cause PwD to eat more regularly?</li>
78.1	14	</ol>
77.1	15
83.1	16	* This research question is difficult due to the practical limitations in designing the experimental setup and as such is left to lesser importance.
	17
77.1	18	{{/html}}
	19
2.1	20	= Method =
1.1	21
93.1	22	The prototype was evaluated with an in-person experiment with multiple participants.
1.1	23
2.1	24	== Participants ==
1.1	25
110.1	26	As there are practical difficulties with conducting the experiment with actual people with dementia, due to both time constraints and COVID, our participants' group consists of peers from other groups and friends. In total we had 19 people take part in our experiment.
1.1	27
2.1	28	== Experimental design ==
1.1	29
114.1	30	For the experiment, we used a within-subject design. All of the participants interacted with both versions of the robot, with half of the participants interacting with version 1 first and then version 2, and the other half in reverse order. This was done to counter-balance the carryover effects. Snacks were made available for the participants, in case they were prompted and were hungry. They were fully aware of them and some of the questionnaire prompts might have given them an idea of what our experiment is about (or at least that it's related to food), which might have skewed our results.
1.1	31
2.1	32	== Tasks ==
1.1	33
112.1	34	The participant interacted with the robot, which was programmed to engage in a lunch discourse. Two versions were implemented: the first version (simple interaction) asks basic questions about mealtime, mostly acting as a reminder for the PwD to have lunch (basically an alarm clock). The second (advanced interaction) is our original implementation of it with the more sophisticated discourse and music.
1.1	35
2.1	36	== Measures ==
1.1	37
122.1	38	We measured the effectiveness of the discourse, both physically and emotionally. Our quantitative measure was whether the person ate the lunch they were supposed to have eaten, and the qualitative measure was the emotions that the PwD experienced before, during, and after the interaction. The qualitative measures were recorded with a simple questionnaire. Some people were not hungry enough to be prompted to have something to eat, which disturbed the results. However, we did measure whether the robot reminded someone of their hunger and if they ate.
1.1	39
2.1	40	== Procedure ==
1.1	41
86.1	42	The procedure was conducted as follows:
	43
115.1	44	1. Welcome participants and explain what they are going to be doing.
88.1	45	1. Have them sign the permission form.
115.1	46	1. Complete questionnaire 1 regarding their emotional state and hunger scale (control).
	47	1. Have interaction with version A of the robot.
89.1	48	1. Complete questionnaire 2 (extended version).
	49	1. Have a short interview during downtime (prepared questions).
115.1	50	1. Have interaction with version B of the robot.
89.1	51	1. Complete questionnaire 3 (extended version).
	52	1. Have a short interview during downtime (prepared questions).
87.1	53
115.1	54	We used the "Wizard of Oz" method for differentiating agreement and disagreement, to make sure that the whole process did not depend on voice recognition being good enough, and to have an overall smoother interaction. In practice, this meant that someone was pressing "y" and "n" on the keyboard according to the participants' answers, in a place the participant did not see, such as behind them. The robot's responses were hardcoded, with a few different branches available to take into account the variety of answers the participants would give. The only issue encountered was some connectivity delays at times, which only slightly affected a few of the interactions.
92.1	55
2.1	56	== Material ==
1.1	57
116.1	58	For the experiments, we used the NAO robot platform, and a laptop to control it. The participants completed the questionnaires on their phones by scanning a QR code. The questionnaires are a combination of questions regarding the emotional state of the participants, their hunger levels, their interaction with the robot, and the music included in the interaction. Stroopwafels and water in a clean cup were made available to see and measure how much people ate.
1.1	59
116.1	60	During the experiments, four different types of questions were given to the participants, in addition to the Consent Form and Disclaimers they had to sign in the beginning. The four sections were:
20.1	61
116.1	62	1. 8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire for mood assessment
	63	1. 4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire to assess the pleasantness and intelligence of the robot
	64	1. 3 hunger and food-related questions of our own, to assess if they eat before or during the interaction (5-point Likert scale)
	65	1. 2 music-related questions of our own, to measure how much they enjoyed the music and what was its effect (5-point Likert scale)
97.1	66
123.1	67	Before the first interaction, the participants were asked to respond to sections 1. and 3., while right after each interaction, they were asked to respond to all four sections, with the music section only present after the advanced interaction. The full questionnaire given to the participants can be found attached.
91.1	68
11.1	69	== Practicalities ==
1.1	70
117.1	71	For actually performing the experiments, there were multiple tasks that had to be performed beforehand.
	72	We first did a practice round by ourselves, which we filmed to have a controlled performance and to be able to give an example of the experiment if needed. For the time that the experiments were going to take place, we first checked our own availability, so we would have at least one person controlling the robot and another interacting with the participant, explaining everything and keeping notes. Afterward, we contacted people from the rest of the groups and friends of ours and decided on a schedule. Then, we had to book the lab, so we accounted for 20 minutes for each participant, as calculated by the test runs we did ourselves, considering that they were going to be some delays. The last step was to buy some stroopwafels and prepare the lab on the day of the experiments.
17.1	73
35.1	74	= Results =
1.1	75
122.1	76	The results were gathered from 19 personnel, all of whom interacted first with one version of the robot and then the other. Ten of the participants interacted first with the simple version, while the other nine had their first interaction with the advanced version.
58.1	77
41.1	78	== Eating ==
123.5	79
123.4	80	[[image:EatingComp.png]]
41.1	81
	82	(% style="text-align:center" %)
62.1	83	Figure 1: Results on the eating of the test personas during the experiment
41.1	84
59.1	85	Simple robot:
	86
	87	* 16% ate
	88	* 33% of those would not have eaten without the robot
	89
	90	Advanced robot:
	91
	92	* 32% ate
	93	* 67% of those would not have eaten without the robot
	94
66.1	95	== Music ==
	96
	97	{{html}}
	98	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/MusicEnjoyable.png?rev=1.1" alt="Effects of music on the test personnel" style="display:block;margin-left:auto;margin-right:auto" width=1250/>
	99	{{/html}}
	100
	101	(% style="text-align:center" %)
67.1	102	Figure 2: Answers of the test personas regarding music
66.1	103
51.1	104	== EVEA (Mood) ==
29.1	105
	106	{{html}}
45.1	107	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/MoodChangeDumb.png?rev=1.1" alt="Measured moods and changes for the simple robot" style="display:block;margin-left:auto;margin-right:auto" width=750/>
29.1	108	{{/html}}
	109
32.2	110	(% style="text-align:center" %)
67.1	111	Figure 3: Median measured moods for the simple version of the robot
30.1	112
32.2	113	{{html}}
45.1	114	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/MoodChangeSmart.png?rev=1.1" alt="Measured moods and changes for the advanced version of the robot" style="display:block;margin-left:auto;margin-right:auto" width=750/>
32.2	115	{{/html}}
	116
	117	(% style="text-align:center" %)
67.1	118	Figure 4: Median measured moods for the advanced version of the robot
29.1	119
33.1	120	(% style="text-align:center" %)
62.1	121	Table 1: Wilcoxon signed rank test results for the hypothesis that the mood changed during the interaction with the simple robot
37.1	122
	123	\|=Mood\|=Happiness\|=Anxiety\|=Sadness\|=Anger
	124	\|Statistic\|37\|5\|4\|14
	125	\|P-value\|0.54\|0.01\|0.01\|0.45
	126
	127	(% style="text-align:center" %)
62.1	128	Table 2: Wilcoxon signed rank test results for the null hypothesis that the mood changed during the interaction with the advanced robot
37.1	129
	130	\|=Mood\|=Happiness\|=Anxiety\|=Sadness\|=Anger
49.1	131	\|Statistic\|32\|11\|2\|17
	132	\|P-value\|0.18\|0.01\|0.01\|0.45
37.1	133
	134	(% style="text-align:center" %)
62.1	135	Table 3: Wilcoxon signed rank test results for the null hypothesis that the mood decreased during the interaction with the simple robot
33.1	136
105.1	137	\|=Mood\|=Anxiety\|=Sadness\|=Anger
	138	\|Statistic\|81\|53\|29
	139	\|P-value\|0.01\|0.00\|0.23
33.1	140
34.1	141	(% style="text-align:center" %)
62.1	142	Table 4: Wilcoxon signed rank test results for the null hypothesis that the mood decreased during the interaction with the advanced robot
33.1	143
35.1	144	\|=Mood\|=Anxiety\|=Sadness\|=Anger
36.1	145	\|Statistic\|32\|149\|52
38.1	146	\|P-value\|0.00\|0.01\|0.07
34.1	147
35.1	148	(% style="text-align:center" %)
62.1	149	Table 5: Wilcoxon signed rank test results for the null hypothesis that the mood increased during the interaction with the simple robot
34.1	150
35.1	151	\|=Mood\|=Happiness
	152	\|Statistic\|37
38.1	153	\|P-value\|0.27
35.1	154
	155	(% style="text-align:center" %)
62.1	156	Table 6: Wilcoxon signed rank test results for the null hypothesis that the mood increased during the interaction with the advanced robot
35.1	157
	158	\|=Mood\|=Happiness
	159	\|Statistic\|32
38.1	160	\|P-value\|0.09
35.1	161
69.2	162	(% style="text-align:center" %)
	163	Table 7: Wilcoxon signed rank test results for the hypothesis that the mood changes with the simple and advanced robots during the interaction are different
	164
	165	\|=Mood\|=Happiness\|=Anxiety\|=Sadness\|=Anger
	166	\|Statistic\|92\|49\|85\|69
	167	\|P-value\|0.92\|0.07\|0.71\|0.31
	168
52.1	169	== Godspeed ==
	170
40.1	171	{{html}}
46.1	172	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/friendly-hist.png?rev=1.1" alt="Effects of music on the test personnel" style="display:block;margin-left:auto;margin-right:auto" width=750/>
41.1	173	{{/html}}
40.1	174
41.1	175	(% style="text-align:center" %)
62.1	176	Figure 5: Answers to the statement 'I thought the robot was friendly'
41.1	177
40.1	178	{{html}}
46.1	179	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/pleasant-hist.png?rev=1.1" alt="Answers to the statement 'I thought the robot was pleasant'." style="display:block;margin-left:auto;margin-right:auto" width=750/>
40.1	180	{{/html}}
	181
	182	(% style="text-align:center" %)
62.1	183	Figure 6: Answers to the statement 'I thought the robot was pleasant'
40.1	184
54.1	185	{{html}}
	186	<img src="https://xwiki.ewi.tudelft.nl/xwiki/wiki/sce2022group01/download/Test/WebHome/godspeed-barchart.png?rev=1.1" alt="Godspeed questionnaire median comparison'." style="display:block;margin-left:auto;margin-right:auto" width=750/>
	187	{{/html}}
1.1	188
54.1	189	(% style="text-align:center" %)
62.1	190	Figure 7: Median measured Godspeed questionnaire dimensions
41.1	191
64.1	192
	193	(% style="text-align:center" %)
71.1	194	Table 8: Wilcoxon signed rank test results for the null hypothesis that the advanced robot scored higher in the perceived dimensions
64.1	195
56.1	196	\|=Dimension\|=Likeability\|=Intelligence
55.1	197	\|Statistic\|36\|70
	198	\|P-value\|0.01\|0.17
52.1	199
94.1	200	== Qualitative Results: Quotes and observations ==
97.1	201
94.1	202	As described, during the experiment, the interaction between the participants and the robot was observed. This section will elaborate on findings from those observations and quotes from participants.
	203
123.3	204	After each interaction section, the participant was asked how the interaction with the robot felt. From the interaction with the simple version of the robot, some interesting quotes were:
97.1	205
123.2	206	* “The robot was a bit direct.”
94.1	207	* “Efficient interaction, but less friendly than the other interaction.”
	208	* “Strange, I did not catch the questions.”
	209	* “It felt short.”
95.1	210
123.3	211	Some of these quotes stress the fact that the simple prototype interaction was rather short and direct. It should be said that the sequence of the interactions seemed to have some impact on how the participants experienced the interaction. Some participants who first experienced the simple prototype were smiling and positively surprised during this interaction, while others who first experienced the advanced prototype were overall smiling less while interacting with the less intelligent robot.
94.1	212
123.3	213	From the interaction with the advanced version of the robot, some interesting quotes were:
97.1	214
94.1	215	* “I think it’s perfect, the robot is very friendly. I liked that the robot sat down with me after a while.”
	216	* “The interaction felt quite natural.”
	217	* “Nao answered pretty quickly, you don’t have to wait for an answer. It is quite a happy robot.”
	218	* “Suggestion to eat was still a bit on the side, a little subtle if I would have dementia.”
	219	* “Very nice, calming, I could have stayed longer with the music.”
122.1	220	* “It was good, natural, the robot understands what I’m saying.”
95.1	221
123.3	222	Some participants clearly expressed how friendly they found the advanced version of the robot. The sequence of the interactions did not seem to impact their feeling about the interaction as much as with the interaction with the simple version of the robot.
94.1	223	Some reported that the interaction felt natural and intuitive.
99.1	224	As for the music, some participants told us that the music was a useful and pleasant addition to the interaction with the robot.
94.1	225	As for the suggestion to eat and drink, one participant reported that the suggestions to eat and drink were perhaps too friendly and too subtle.
123.3	226	From our observations, it seemed as if participants were either smiling more during the interaction with the advanced version of the robot or concentrating on the interaction more carefully compared to the interaction with the simple version.
94.1	227
72.1	228	= Discussion =
95.2	229
122.1	230	From the results, we can see that the more advanced robot shows advantages over the simple version in multiple categories. Hints of better performance in other categories can be seen, but no conclusions should be drawn from the ones that lack statistical significance.
95.2	231
122.1	232	As for the eating, it seems that both robots have limited success in causing the people to eat as seen in Figure 1, they could cause the patients to eat more regularly if triggered by timers or other suitable systems. It also seems that the advanced robot is better in the reminding, by a slight margin. However, the long-term effects of reminding should be researched more to conclude whether the usage of the demonstrated robot platform or similar would cause the patients to eat more regularly. It is also unclear how the test setup and the limited choice of food affected the eating.
95.2	233
122.1	234	Based on the answers of the participants regarding music seen in Figure 2, it can be seen that most of them were either indifferent or liked the music. Also, as the test personnel find the advanced robot more likable with a 5% confidence limit (Table 7), and the advanced version was the only version with music, it seems likely that the music does make the interaction more pleasant for the personas. However, some of the likeability might be due to the other advanced features of the robot and thus more research is needed to conclude the effect of the music.
95.2	235
	236	The EVEA and partial Godspeed result can be seen in Figures 3-7 and Tables 1-8. The results show that with reasonable confidence (5% confidence limit), both versions of the robot decreased sadness and anxiety in the test personas. Hints are shown (10% confidence limit) that the advanced robot also decreases feelings of anger and increases happiness, while the simple robot fails to show similar results. However, in Table 7 we can see that the statistical differences in the mood differences during the interactions with the different versions are not highly significant.
	237
122.1	238	A Wilcoxon signed-rank test for the partial Godspeed test shows in Table 8 that with high confidence (1% confidence limit), the intelligent robot is more likable in comparison to the simple robot. With these results, it is likely that the more advanced robot is slightly preferable and the personas might experience less negative emotions after the interaction with the robots, but it remains yet unclear if the effect is more powerful with the advanced robot.
95.2	239
122.1	240	The observations and interviews with the participants clearly demonstrated that for now: a more friendly and intelligent robot does make the interaction with the robot more pleasant. Also, the observations do support the data from the questionnaire in terms of the likability difference between both robot types.
109.1	241
122.1	242	Analysis of the results surfaced some minor issues in the experiment, such as the lack of comparison with two robots of similar features, with and without music. Also, the practical limitations in the setup, such as the lack of different food options and some participants being aware of the design goals of the prototype could have interfered with the natural flow of the intercourse. With these limitations, the research method was successful in extracting differences within the robots and brought up additional directions for future research.
72.1	243
122.1	244	The most interesting direction for future research would be the longer-term studying of the effect of mealtime reminders on the health of the test subjects. The longer-term health study would uncover the effect on eating frequency and the development of the relationship with the robot, for example, would the test subjects that were first excited about the novel interaction with the robot, develop negative feelings about the supervision that the robot is conducting into their personal life.
72.1	245
109.1	246	Furthermore, an aspect that was not compared in this study is how many stroopwafels the participants ate while interacting with the robot. For now, the focus was to evaluate whether the claim the robot causes the PwD - in the case of the experiment: the participants - to eat or not. For future research, the amount of food consumed by the participants could also be taken into consideration.
72.1	247
109.1	248	Lastly, another topic to study is the differences with and without music. The effects of music could be studied with the music tailored to personal taste and all versions of the robot with and without the music playback included in the interaction. This would allow to pinpoint the effects of music, without the other features causing variance.
94.1	249
57.1	250	= Conclusions =
54.1	251
122.1	252	From the results, it seems that in short-term interactions, both of the robots do remind the persons of their hunger, but the test setup might have caused many people not to eat or not to be hungry when arriving. It would also seem that the music does make the entire discourse more enjoyable as people did enjoy it, but it is unclear whether the observed increases in mood caused by the advanced robot in comparison to the simple version are due to the music or other features included in the advanced version or simply due to variance. It seems that the advanced robot is slightly more enjoyable due to the observed change in anxiety, but in total the results are inconclusive.
54.1	253
122.1	254	The long-term effects of this are unclear and require further study. The short-term experiment shows promising results to further develop such solutions, but also conduct experiments to study the long-term effects of such a solution. With a longer experiment, the development of the human-robot interaction and the effect of constant mealtime reminders would likely begin to show, which could cause differences to the presented short-term results, for example, the robot becoming more enjoyable as it becomes familiar.
96.1	255
10.1	256	= Appendix =
	257
17.1	258	== Experiment introduction for participants ==
16.1	259
17.1	260
16.1	261
122.1	262	Hi, we are <NAME> and <NAME> from the TU Delft Socio-Cognitive Engineering course Group 1, thank you for participating in our prototype evaluation experiment. The experiment is being conducted as a part of the TU Delft course on Socio-Cognitive Engineering and aims to evaluate the prototype designed as a part of the course. The evaluated prototype is based on the Nao robot and is intended to improve the well-being of people suffering from dementia.
16.1	263
19.1	264	Consuming food and/or water can be a consequence of the interaction between you and the robot. Therefore, we would like to ask you if you have any allergies. If you have a form of Diabetes, please let us know before we start the first part of the experiment. You are strongly encouraged to share any other health conditions that can possibly be relevant to take into account when doing an experiment with robots and food with us.
	265
	266	The link between the stimuli of the Nao-robot and the triggering of epileptic seizures is yet unknown. If you have ever experienced epileptic seizures, please let us know. Then, we could see if any special precautions are needed.
	267
122.1	268	The experiment will last for approximately 15-20 minutes, and consists of two interaction sections with the Nao robot, as well as questionnaires before, between, and after the sections. We kindly ask you to act naturally during the experiment and fill the questionnaires truthfully and intuitively. Remember that we are evaluating the prototype's performance, not yours. You can stop the experiment at any time.
17.1	269
122.1	270	We will be collecting data from the questionnaires and recording some experiments, do you agree with your experiment being recorded? All data excluding the recordings will be anonymized before analysis and storage. The recordings will not be shared with third parties. After the experiment, you have the right to ask for information about the collected data and revoke the right to use it. We kindly ask you not to share any information about the experiment with other participants.
	271	Do you have any questions?
16.1	272
10.1	273	== After research interview ==
	274
	275	Setup:
	276	The test subject has finished both parts of the experiment. Before leaving the test conductor(s) sit down with them and ask the following questions in a discussion about the experiment. Discussion can flow freely, but the following topics should be discussed.
	277
	278	Topics:
122.1	279	- Emotions before/during/after the interaction with the robot
10.1	280	- Agitation due to the robot suggesting eating
	281	- Effect of music on the general feeling of the situation
	282	- Effectiveness of eating/drinking suggestions
	283
12.1	284	Questions:
10.1	285	- What did the interaction with the robot feel like?
	286	- With the more intelligent version?
	287	- With the less intelligent version?
	288	- What did you feel like when the robot suggested you should eat/drink?