Wiki source code of Test

Version 67.1 by Vishruty Mittal on 2022/04/02 12:57

version	line-number	content
57.1	1	Evaluation is an iterative process where the initial iterations focus on examining if the proposed idea is working as intended. Therefore, we want to first understand how realistic and convincing the provided dialogues and suggested activities are, and would they be able to prevent people from wandering. To examine this, we conduct a small pilot study with students, who role-play having dementia. We then observe their interaction with Pepper to examine the effectiveness of our dialog flow in preventing people from wandering.
2.4	2
2.5	3
1.2	4	= Problem statement and research questions =
4.1	5
1.4	6	Goal: How effective is music and dialogue in preventing people with dementia from wandering?
1.1	7
2.7	8	Research Questions (RQ):
10.4	9
1.4	10
10.3	11	1. What percentage of people are prevented from going out unsupervised? (Quantitative) (CL01, CL05)
10.5	12	1. How does the interaction change the participant's mood? (CL02)
40.1	13	1. Can the robot respond appropriately to the participant's intention? (CL03)
10.5	14	1. How do the participants react to the music? (CL04)
	15	1. Does the activity that the robot suggests prevent people from wandering/ leaving? (CL06)
13.1	16	1. Can pepper identify and catch the attention of the PwD?
1.4	17
40.1	18	//Future research questions//
10.4	19
14.1	20	1. Does the interaction with Pepper make PwD come back to reality? (CL08)
	21	1. Does the interaction with Pepper make PwD feel he/she is losing freedom? (CL09)
	22	1. Does preventing the participant from going out alone make them feel dependent? (CL10)
9.1	23
1.2	24	= Method =
4.1	25
61.1	26	A between-subject study with students who play the role of having dementia. Data will be collected with a questionnaire that participants fill out before and after interacting with Pepper. The questionnaire captures different aspects of the conversation along with their mood before and after the interaction with Pepper.
1.1	27
61.1	28	For our between-subject study, our independent variable is Pepper trying to distract the users by mentioning different activities along with the corresponding music. Through this, we want to measure the effectiveness of music and activities in preventing people from leaving the care home, which is thereby our dependent variable. So we developed 2 different prototype designs-
	29
	30	Design X - It is the full interaction flow where Pepper suggests activities and uses music to distract people from leaving.
	31	Design Y - It is the control condition where pepper simply tries to stop people from leaving by physically keeping its hand on the door.
	32
1.2	33	== Participants ==
4.1	34
62.1	35	17 students who play the role of having dementia. They will be divided into two groups. One group (11 participants) will be interacting with design X (group 1) robot while the other group (6 students) will interact with the design Y (group 2).
14.2	36	It is assumed that all participants are living at the same care center.
1.1	37
1.2	38	== Experimental design ==
4.1	39
65.1	40	Before Experiment:
4.1	41
63.1	42	We will explain to the participants the goal of this experiment and what do they need to do to prevent ambiguity. Therefore, as our participants are students and only playing the role of having dementia, we will give them a level of stubbornness/ willpower with which they are trying to leave the care home.
	43	Participants will also be given a reason to leave, from the below list:
1.1	44
67.1	45	* going to the supermarket
	46	* going to the office
	47	* going for a walk
63.1	48
	49	After this preparation, the participant fills a part of the questionnaire.
	50
66.1	51	Experiment:
63.1	52	The participant begins interacting with Pepper who is standing near the exit door. The participant and robot have an interaction where the robot is trying to convince him/her to stay inside.
	53
66.1	54	After Experiment:
63.1	55	After the participant finishes interacting with Pepper, he/she will be asked to fill out the remaining questionnaire. Almost all the questions in the questionnaire collect quantitative data, using a 5 point Likert scale. The questionnaire also used images from Self Assessment Manikin (SAM) so that user can self attest to their mood before and after their interaction with Pepper.
	56
4.1	57	== Tasks ==
2.6	58
2.7	59	Because our participants only play the role of having dementia, we will give them a level of stubbornness/ willpower with they are trying to leave. We try to detect this level with the robot.
	60	Participants from group 1 (using intelligent robot) will also be given one of the reasons to leave, listed below:
4.1	61
2.7	62	1. going to the supermarket
	63	1. going to the office
	64	1. going for a walk
	65
2.8	66	After this preparation, the participant is told to (try to) leave the building. The participant and robot have an interaction where the robot is trying to convince the participant to stay inside.
1.1	67
	68
1.2	69	== Measures ==
4.1	70
1.14	71	We will be measuring this physically and emotionally.
	72	Physically: whether the participant was stopped from leaving the building or not.
	73	Emotionally: evaluate their responses to the robot and observe their mood before and after the interaction.
1.1	74
	75
1.2	76	== Procedure ==
4.1	77
3.1	78	{{html}}
	79	<!-- Your HTML code here -->
	80	<table width='100%'>
	81	<tr>
	82	<th width='50%'>Group 1</th>
	83	<th width='50%'>Group 2</th>
	84	</tr>
	85	<tr>
	86	<td>intelligent robot</td>
	87	<td>unintelligent robot</td>
	88	</tr>
	89	<tr>
	90	<td>
4.2	91	1. Starts with a short briefing on what we expect from the participant<br>
	92	2. Let them fill out the informed consent form<br>
5.2	93	3. Tell them their level of stubbornness and reason to leave<br>
14.2	94	4. Fill out question about current mood (in their role)<br>
5.2	95	4. Let the user interact with the robot<br>
14.2	96	5. While user is interacting, we will be observing the conversation with the robot<br>
5.2	97	6. Let user fill out the questionnaire about their experience after the interaction
3.1	98	</td>
	99	<td>
4.2	100	1. Starts with a short briefing on what we expect from the participant<br>
	101	2. Let them fill out the informed consent form<br>
14.2	102	4. Fill out question about current mood (in their role)<br>
	103	5. Let the user interact with the robot<br>
	104	6. Let user fill out the questionnaire about their experience after the interaction<br>
3.1	105	</td>
	106	</tr>
	107	</table>
1.1	108
3.1	109	{{/html}}
2.6	110
1.2	111	== Material ==
4.1	112
1.16	113	Pepper, laptop, door, and music.
1.1	114
58.1	115	= Results =
1.1	116
52.3	117	{{html}}
	118	<!--=== Comparison between intelligent (cond. 1) and less intelligent (cond. 2) prototype ===
6.2	119
52.1	120	{{html}}
	121	<img src="/xwiki/wiki/sce2022group05/download/Test/WebHome/Stay_inside.svg" width="500" height="270" />
	122	{{/html}}
	123
	124	Non of the participants who interacted with the less intelligent robot was prevented from leaving. Still, 3 people assigned to condition 1 weren't convinced to stay inside. A failure rate of 27,3 % is too high for this application since people could be in danger if the system fails.
	125
	126	Mood evolution
	127	[[image:mood_only.png\|\|height="150"]]
	128
	129	{{html}}<img src="/xwiki/wiki/sce2022group05/download/Test/WebHome/mood_before.svg" width="500" height="270" /><br>
	130	<img src="/xwiki/wiki/sce2022group05/download/Test/WebHome/mood_after.svg" width="500" height="270" /><br>{{/html}}
	131
	132	Regarding the changes in mood, 4 out of 11 participants assigned to condition 1 had an increase in mood throughout the interaction. Only one participant felt less happy afterward, the rest stayed at the same level of happiness. The overall mood shifted to happier in general (as you can see in the graphic above), even though only small improvements in mood were detected (<= 2 steps on the scale).
	133	The participants from condition 2 mostly stayed at the same mood level, 2 were less happy, one participant was happier afterward. Comparing both conditions it becomes clear, that condition 1 had a more positive impact on the participant's mood.
	134
	135	Interesting is also, none of the participants was in a really bad mood at the beginning or end.
	136
42.1	137	==== Condition 1 - intelligent Prototype: ====
1.1	138
42.1	139	{{html}}
	140	<img src="/xwiki/wiki/sce2022group05/download/Test/WebHome/Music_reco.svg" width="500" height="270" /> <br>
	141	{{/html}}
40.1	142
7.2	143	8 out of 11 Participants answered, that they don't know the music that has been played. If we told them afterward the title of the song, most participants do know the song. Why didn't they recognize it during the interaction?
	144	This can have two reasons: The part of the song we pick was too short to be recognized or not the most significant part of the song. For example, the beginning of "escape - the pina colada song" is not as well known as its chorus. Another reason could be, that the participant was distracted or confused by the robot and therefore couldn't carefully listen to the music.
40.1	145
	146	{{html}}
40.2	147	<img src="/xwiki/wiki/sce2022group05/download/Test/WebHome/Music_fit.svg" width="500" height="270" /> <br>
40.1	148	{{/html}}
	149
40.2	150	Only 4 out of 11 people agreed, that the music fits the situation. One of our claims, to use music that fits the situation or place, is therefore not reached and the music didn't have the intended effect. Even though we carefully choose the music and discussed a lot about our choice, it was hard to find music that different people connect with a certain place or activity. An approach to improve this could be using an individual playlist for each participant.
40.1	151
40.2	152
42.1	153	==== Condition 2 - less intelligent prototype: ====
16.3	154
52.3	155	Participants assigned to condition 2 weren't convinced to leave. We saw, that most of them tried to continue talking to pepper when it raises its arm to block the door, even though it didn't listen. They were surprised by peppers reaction and asked for a reason why they are not allowed to leave. In order to have a natural conversation flow, the robot should provide an explanation for each scenario that tells why the person is not allowed to leave. This confirms that our approach, to give reason to stay inside, might be helpful to convince PwD to stay inside.
42.1	156
43.1	157	=== Problems that occurred during the evaluation ===
	158
6.2	159	1. lots of difficulties with speech recognition:
	160	1.1. even though the participant said one of the expected words, pepper understood it wrong and continued with a wrong path
	161	1.2. If the participant started to talk before pepper was listening (eyes turning blue), it misses a "yes" or "no" at the beginning of the sentence, which causes misunderstandings.
	162	1. problems with face detection
	163	2.1. due to bad light face was not recognized
	164	2.2. if the participant passes pepper from the side, the face was not recognized. Therefore, we told people to walk from the front towards pepper. In most cases that helped detect the face.
	165	2.3. face detection doesn't work with face masks. This could lead to huge problems in the usage of pepper in care homes.
5.5	166
7.1	167	One of the most frequent and noticeable reactions from participants was confusion. This feeling was caused by two main factors:
	168	misunderstandings from speech recognition which leads to unsuitable answers from pepper, as well as the unsuitable environment and setting of our evaluation.
6.2	169	The reasons for failure in speech recognition are listed above. An unsuitable answer can e.g. be an argument to stay inside, that doesn't fit the participant's reason to leave. Also, some people told in a long sentence that they don't like the provided activity and still want to leave. If the speech recognition fails in this case and pepper understood you would like to do the activity, it seems like it encourages you to leave, instead of doing the activity. This leads to the total opposite of our intention.
52.3	170	Furthermore, we found out, that our prototype doesn't fit in the environment of the lab. We encourage the participant to do some activities, that they can't do in the lab environment (go to the living room, have a coffee or do a puzzle). If the robot tells asks you if you want to do the activity, most people don't know how to react and are insecure about how to answer. Participants "freeze" in front of the robot or just left the room. -->
	171	{{/html}}
6.2	172
52.3	173	=== RQ1: Are people convinced not to go out unsupervised? ===
58.1	174
52.3	175	{{html}}
	176	<table style="width: 100%">
	177	<tr>
	178	<td style="width: 50%">
54.1	179	<img src="/xwiki/wiki/sce2022group05/download/Foundation/Operational%20Demands/Personas/WebHome/RQ1.jpg?height=250&rev=1.1" />
52.3	180	</td>
	181	<td>
	182	Comment on the graph
	183	</td>
	184	</tr>
	185	</table>
	186	{{/html}}
42.1	187
52.3	188	=== RQ2: How does the interaction change the participant's mood? ===
58.1	189
52.3	190	{{html}}
	191	<table style="width: 100%">
	192	<tr>
	193	<td style="width: 50%">
54.1	194	<img src="/xwiki/wiki/sce2022group05/download/Foundation/Operational%20Demands/Personas/WebHome/RQ2.jpg?height=250&rev=1.1" />
52.3	195	</td>
	196	<td>
	197	Comment on the graph
	198	</td>
	199	</tr>
	200	</table>
	201	{{/html}}
1.1	202
52.3	203	=== RQ3: Can the robot respond appropriately to the participant's intention? ===
58.1	204
52.3	205	{{html}}
	206	<table style="width: 100%">
	207	<tr>
	208	<td style="width: 50%">
54.1	209	<img src="/xwiki/wiki/sce2022group05/download/Foundation/Operational%20Demands/Personas/WebHome/RQ3.jpg?height=250&rev=1.1" />
52.3	210	</td>
	211	<td>
	212	Comment on the graph
	213	</td>
	214	</tr>
	215	</table>
	216	{{/html}}
1.1	217
52.3	218	=== RQ4: How do the participants react to the music? ===
58.1	219
52.3	220	{{html}}
	221	<table style="width: 100%">
	222	<tr>
	223	<td style="width: 50%">
54.1	224	<img src="/xwiki/wiki/sce2022group05/download/Foundation/Operational%20Demands/Personas/WebHome/RQ4.jpg?height=250&rev=1.1" />
52.3	225	</td>
	226	<td>
	227	Comment on the graph
	228	</td>
	229	</tr>
	230	</table>
	231	{{/html}}
1.1	232
52.3	233	=== RQ5: Does the activity that the robot suggests prevent people from wandering/ leaving? ===
58.1	234
52.3	235	{{html}}
	236	<table style="width: 100%">
	237	<tr>
	238	<td style="width: 50%">
54.1	239	<img src="/xwiki/wiki/sce2022group05/download/Foundation/Operational%20Demands/Personas/WebHome/RQ5.jpg?height=250&rev=1.1" />
52.3	240	</td>
	241	<td>
	242	Comment on the graph
	243	</td>
	244	</tr>
	245	</table>
	246	{{/html}}
16.3	247
52.3	248	=== RQ6: Can pepper identify and catch the attention of the PwD? ===
58.1	249
52.3	250	{{html}}
	251	<table style="width: 100%">
	252	<tr>
	253	<td style="width: 50%">
54.1	254	<img src="/xwiki/wiki/sce2022group05/download/Foundation/Operational%20Demands/Personas/WebHome/RQ6.jpg?height=250&rev=1.1" />
52.3	255	</td>
	256	<td>
	257	Comment on the graph
	258	</td>
	259	</tr>
	260	</table>
	261	{{/html}}
16.3	262
52.3	263	=== Reliabity Scores ===
58.1	264
52.3	265	{{html}}
	266	<table style="width: 100%">
	267	<tr>
	268	<td style="width: 50%">
54.1	269	<img src="/xwiki/wiki/sce2022group05/download/Foundation/Operational%20Demands/Personas/WebHome/RelScores.jpg?height=250&rev=1.1" />
52.3	270	</td>
	271	<td>
	272	Comment on the graph
	273	</td>
	274	</tr>
	275	</table>
	276	{{/html}}
	277
58.1	278	= Limitation =
16.3	279
52.3	280	* Lab Environment: The lab environment is different from a care home, the participants found it difficult to process the suggestions made by Pepper. For example, if Pepper asked someone to visit the living room, it created confusion among the participants regarding their next action.
52.2	281
52.3	282	* Role-Playing: Participants for the experiment are not actual patients suffering from dementia. Hence it is naturally difficult for them to enact the situations and replicate the mental state of an actual person suffering from dementia.
52.2	283
52.3	284	* Speech Recognition: The speech recognition module inside Pepper is not perfect. Therefore, in certain cases, Pepper misinterpreted words spoken by the participants and triggered an erroneous dialogue flow. The problems commonly occurred with words that sound similar such as "work" and "walk". Moreover, there are some additional hardware limitations that hampered the efficiency of the speech recognition system. One prominent issue is that the microphone within Pepper is only active when the speaker is turned off. A blue light in the eye of Pepper indicated the operation of the microphone. Since most of the participants are not used to interacting with Pepper found it difficult to keep this limitation in mind while trying to have a natural conversation.
	285
	286	* Face Detection: The face recognition module within Pepper is also rudimentary in nature. It can not detect half faces are when participants approach from the side. Adding to the problem, the lighting condition in the lab was not sufficient for the reliable functioning of the face recognition module. Hence Pepper failed to notice the participant in some cases and did not start the dialogue flow.
	287
1.2	288	= Conclusions =
58.1	289
52.3	290	* People who liked the activity tend to stay in
	291	* People who knew the music found it more fitting
	292	* People are more convinced to stay in with the intelligent prototype
	293	* Cannot conclude whether moods were improved
	294	* Need to experiment with the actual target user group to derive on concrete conclusion
	295	* Experiment with personalization
40.1	296
52.3	297	= Future Work =
58.1	298
52.3	299	* Personalisation: Personalize music, and activity preferences according to the person interacting with Pepper.
	300	* Robot Collaboration: Collaborate with other robots such as Miro to assist a person with dementia while going for a walk instead of the caretaker.
	301	* Recognise Person: For a personalised experience, it is essential that Pepper is able to identify each person based on an internal database.
	302	* Fine Tune Speech Recognition: Improvements are necessary for the speech recognition module before the actual deployment of the project in a care home. Additionally, support for multiple languages can be considered to engage with non-English speaking people.