Wiki source code of Test

Version 114.11 by Sneha Lodha on 2022/04/04 00:35

version	line-number	content
32.1	1	= Ideal Evaluation (actual research) =
31.1	2
29.1	3	== Problem statement and research questions ==
	4
8.1	5	This project uses a Social Cognitive Engineering (SCE) approach to guide the design and research process. The SCE method provides a systematic approach to our study of robots for PwDs. The main goal of our application is to improve the well-being of the person with dementia (PwD) and of those living with them.
6.1	6
8.1	7	For the prototype that we have designed now, these are some research questions that we want to address.
	8
29.1	9	1. Are the different stakeholders able to use our prototype smoothly?
36.1	10	1. Does the prototype allow the PwD greater autonomy in their day-to-day life?
29.1	11	1. Does the prototype improve the emotional state of the PwD and their relatives?
28.1	12
29.1	13	== Method ==
1.1	14
36.1	15	As people with dementia have very specific situations (and because our prototype is built to deal with that aspect of customizability), we do not go for an identical experiment for all of them. Instead, the global setup is very similar, although Pepper is customized for the needs of every patient. Our evaluation can therefore be assimilated (with some reserve) to a within-subject evaluation. We also do pre-test and post-test.
29.1	16	Our main evaluation method is summative evaluation: we are trying to determine whether the robot has an impact and improves the frequency of "yes" in our yes-no driving questions.
1.1	17
2.1	18	== Participants ==
1.1	19
36.1	20	The study will be conducted on approximately twenty people in the early stages of dementia. To avoid too much gender imbalance, there will be at least five men and five women. For the same reason, we hope to find at least five people around 50 or younger, although we expect most participants to be over 70 years old.
37.2	21	The participants will be selected based on a pool of PwD who live at home and need regular visits from an HCP. Only the volunteers will be kept for the experiment. The participants must have no experience of a Pepper robot helping with dementia.
	22	As our experiment involves the PwD in their home environment, relatives and healthcare professionals (HCP) will also be involved in the process: although not being the targets of the experiment, they will contribute to obtaining the measurements and results.
1.1	23
2.1	24	== Experimental design ==
1.1	25
36.1	26	Since dementia is unique to every person, it is very hard to conduct a global experiment with the same conditions for all participants. Every one of them may have different issues in their day-to-day life, while also not having the same living conditions (alone, living with husband, family, etc), this would require a different treatment. Furthermore, we want to record whether our prototype leads to an improvement in life quality. This can not be done through a short experiment, because the reliability of such an experiment would be very low.
29.1	27	This means we need to record the patient at home and measure over an extended period of time if their well-being and autonomy globally improved.
	28
2.1	29	== Tasks ==
1.1	30
36.1	31	The PwD will live their daily life, without Pepper in the beginning, then accompanied by Pepper, as if they were not under any experiment. The fact that they actually are will obviously affect their behavior. Yet, we hope that not being recorded and being under a non-invasive experiment will help them not to stress out and may make them live their life as normally as possible.
37.2	32	The tasks performed during the evaluation by Pepper, by the PwD, and by Pepper and the PwD together, will be decided in consultation with the HCP (and potentially the relatives) based on the needs of each patient.
1.1	33
2.1	34	== Measures ==
1.1	35
36.1	36	We are planning to make behavioral and emotional measurements.
37.2	37	Behavioral measurements are the actions that the PwD is going to perform during the week, so it can be considered as subjective quantitative data. This will involve the relatives, the HCP, and the PwD themselves to quantify whether the use of Pepper did actually result in an increase in autonomy for the PwD.
	38	Emotional measurements are more related to the state of mind, change of expression, and mood, so they can be considered as qualitative data. This can be measured by frequent talks with the PwD, either by the relatives or the HCP.
	39	Measures will be done by oral discussions with the PwD, HCP, and relatives.
29.1	40
2.1	41	== Procedure ==
1.1	42
29.1	43	The whole experiment process is 4 weeks long, although only weeks 1 and 4 are technically part of the experiment itself.
37.2	44	During week 1, the PwD, the HCP, and the relatives will be asked to pay increased attention. Behavioral and emotional data will be collected. This is a regular week for the PwD, i.e., in the usual situation, except that there is more attention dedicated to them.
	45	During weeks 2 and 3, the PwD, the HCP, and the relatives will be introduced to Pepper, with the goal of getting used to it.
36.1	46	During week 4, which is the actual week of the experiment, attention will be spent trying to mirror week 1 as closely as possible. Behavioral and emotional data will be collected again.
20.1	47
29.1	48	== Material ==
20.1	49
23.1	50	1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.
37.2	51	1. Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behavior for every participant. However, the input data will be entered by the HCP(and potentially the relatives).
1.1	52
29.1	53	== Results ==
1.1	54
29.1	55	Since each PwD has its own state of dementia and personal issues, it is very difficult to get uniform results, especially since they are collected orally.
36.1	56	Getting very nice, fully robust, and reliable results is merely a hope and a dream.
29.1	57	However, we can try to consider the main trends that we are interested in.
36.1	58	Thus, the results will be mainly focused on:
29.1	59	- How much autonomy did the PwD gain?
37.2	60	→ what did the HCP, relatives, and PwD report
29.1	61	→ how many tasks did they perform that they didn't do previously
	62	→ did the relatives feel they had more time for themselves
	63	- Did their emotional state improve?
	64	→ feelings from the PwD themselves
37.2	65	→ reports from relatives and HCP
29.1	66	These results will most likely never be yes-no results, but more like clues or hints that show whether some things worked on not, which will be the point of our discussion.
1.1	67
29.1	68	== Discussion ==
1.1	69
29.1	70	* Reliability: Yes. One could replicate the same experiment with other patients.
94.1	71	* Validity: TBD. (This this ideal evaluation has not been conducted, we cannot really estimate this.)
	72	* Biases: TBD.(This this ideal evaluation has not been conducted, we cannot really estimate this.)
36.1	73	* Scope: No. It would be very difficult to generalize the results since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar efforts to customize Pepper for more patients should produce similar effects.
29.1	74	* Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.
1.1	75
29.1	76	== Conclusions ==
31.1	77
95.1	78	(Since this ideal evaluation has not been conducted, we do not have conclusions to be drawn.)
31.1	79
96.1	80
	81
33.1	82	= Feasible evaluation (students) =
31.1	83
	84	== Problem statement and research questions ==
	85
94.1	86	For the prototype that we have designed now, and for a smaller scope evaluation, the research questions are smaller-scope versions of our main research questions for the previous part.
	87	We also cannot test all our claims during only a short evaluation session with students.
31.1	88
94.1	89	1. Are the participants able to use our prototype smoothly?
	90	1. Does the prototype improve the autonomy of the patient on a specific activity?
	91	1. Does the prototype improve the emotional state of the participant after completing a specific activity?
31.1	92
43.1	93	Based on these questions and our evaluation setting, we can evaluate some of our claims from our Claim page.
	94
46.1	95	1. CL01: The PwD becomes aware of Pepper's presence.
	96	During the experiment, we can observe if this happens.
	97	1. CL02: The PwD feels understood.
	98	In the questionnaire, there are questions related to this.
	99	1. CL03: The PwD performs an activity step.
	100	We can verify this claim based on how far the participant went in the activity.
	101	1. CL08: The PwD feels accomplished.
	102	After the activity is done, there are two specific questions related to accomplishment.
	103	1. CL10: The PwD feels reassured.
	104	We can evaluate this from observing the participants, and from the affect questionnaire.
	105	1. CL11: The PwD feels content.
	106	There is a question related to this in the questionnaire.
43.1	107
31.1	108	== Method ==
	109
94.1	110	In order to collect qualitative data, we prepared two questionnaires to give to the participants: the affect assessment questionnaire and the system assessment questionnaire. All questionnaire questions are expressed in the form of statements and the participant can express one out of seven levels of agreement/disagreement from strongly disagree to strongly agree. The only exception is the additional remarks field at the end of the system assessment questionnaire, where the participants can freely put any comment/remark/feedback they have on the experiment.
31.1	111
42.1	112	Affect assessment
38.1	113
94.1	114	The affect assessment questionnaire asks the participant to describe their mood. The questionnaire is given two times: once before the experiments and once right after. The purpose of this questionnaire is to compare the mood experienced by the participant before the interaction with Pepper and after. The idea is to assess the effect of the activity. The questionnaire has six questions each designed to measure the level of a certain feeling experienced at the moment of filling the questionnaire. The six feelings we take into consideration are evaluated from the following statements:
38.1	115
94.1	116	1. I feel sad.
	117	1. I feel content.
	118	1. I feel calm.
	119	1. I feel tired.
	120	1. I feel nervous.
	121	1. I feel caring.
39.1	122
42.1	123	System assessment
39.1	124
42.1	125	The system assessment questionnaire is given after the interaction with Pepper and its purpose is to assess the participant's experience with Pepper in more detail. The questionnaire has fourteen questions which are designed to answer research questions as described below:
39.1	126
40.1	127	1. I like gardening
94.1	128	1. I would have known how to do the whole task without Pepper.
40.1	129	1. I think Pepper made the task easier for me.
	130	1. Pepper was easy to understand.
	131	1. I enjoyed the task more than if I had had to do it alone.
	132	1. I am pleased that Pepper reminded me to do the activity.
	133	1. I feel like completing the task was a good accomplishment.
	134	1. I feel like I accomplished it myself.
	135	1. I felt in control of what I had to do.
	136	1. I felt annoyed by Pepper.
	137	1. I felt frustrated by the task.
	138	1. I felt pressured by Pepper.
	139	1. Pepper cared about helping me.
	140	1. I would trust Pepper with more important activities.
39.1	141
	142	Research questions:
	143
94.1	144	1. To what extent did Pepper improve the task's experience? [Questions: 3,4,5,10,11,12]
41.1	145	1. To what extent did Pepper improve the participant's autonomy and perception of control? [Questions: 6, 8, 9]
40.1	146	1. To what extent did Pepper improve the participant's sense of accomplishment? [Questions: 7]
41.1	147	1. Did the participant perceive Pepper as a social agent? [Questions: 13, 14]
40.1	148	1. How was Pepper's ability to guide the participant through the task? [Questions: 4]
39.1	149
40.1	150	Question 1 is used to take into consideration the expected baseline enjoyment of the task for each participant.
	151
31.1	152	== Participants ==
	153
94.1	154	In our situation, our participants are 24 students from TUD. Most of them come from Computer Science and/or from the SCE class. One third of them are female and two thirds male, so gender balance is decent.
	155	Roleplaying PwDs may be difficult for students, especially for those who have not followed the course. For this reason, and in order not to introduce an extra bias "knows about dementia"/"does not know about dementia", we decided to ask the students to just act as students.
31.1	156
	157	== Experimental design ==
37.1	158
94.1	159	Since our evaluating process is relatively short, we use within-subject, which means each participant goes through all conditions. In this way, our experiment tends to have more statistical power and less variability.
31.1	160
	161	== Tasks ==
	162
94.1	163	All participants will go through our designed testing process, which combines a calendar event reminder and an activity breakdown.
	164	Pepper will propose the activity to the participant, then go step by step with them through the activity.
31.1	165
	166	== Measures ==
	167
94.1	168	We measure the mood of the participants through the affect assessment questionnaires. We also measure some aspects of the interaction between Pepper and the participant through the system assessment questionnaire. Both are qualitative data, as the participant gives an answer ranging from strongly disagree to strongly agree.
31.1	169
	170	== Procedure ==
	171
94.1	172	Before the experiment, we explain to the participants the purpose of our evaluation (context of SCE class, our prototype).
	173	We tell them that we will gather data in the questionnaires, that the data are fully anonymized and will only be used for our result analysis in this course.
	174	We also briefly explain to them how Pepper works, especially that she only listens when the eyes are blue.
	175	After the explanation, the participant can fill the consent form and the first affect questionnaire.
	176	Then, the experiment starts. The experiment is conducted in autonomy between the participant and Pepper. We only intervene if asked by the participant or if something is going wrong with the experiment.
	177	After the activity is finished by the participant, they fill the second affect questionnaire and the system assessment questionnaire.
	178	After this is done, we thank them for their participation and the evaluation session is over.
31.1	179
	180	== Material ==
	181
94.1	182	1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers. The consent form is accessed online via a QR code.
	183	1. Questionnaires. There are three questionnaires to answer (mood 1, mood 2, system). They are also accessed online via a QR code.
37.2	184	1. Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behaviour for every participant. However, the input data will be entered by the HCP (and potentially the relatives).
94.1	185	1. Gardening stuff. In order to conduct the activity, we bought and brought basic stuff required for gardening (soil, pot, seeds, etc).
31.1	186
	187	== Results ==
	188
113.2	189	The following sections contains the results gather from the affect and system assessment. Apart from the printed version being included below, the results are also included as interactive graphs. To get a better version with more information (data point information on mouse hovering for instance), simply click the provided link for each graph.
67.1	190
98.1	191	=== Affect assessment ===
	192
100.1	193	[[Affect assessment, interactive version>>https://pietro99.github.io/SCE/graphs/mood_questionnaire.html]]
67.1	194	[[image:mood.svg]]
113.2	195	Figure 1: Graphical results of affect assessment before and after conducting the evaluation activity with Pepper.
114.1	196
105.1	197	{{html}}
	198	<!DOCTYPE html>
91.1	199	<html>
	200	<head>
	201	<style>
	202	table {
	203	font-family: arial, sans-serif;
	204	border-collapse: collapse;
	205	width: 100%;
	206	}
	207
	208	td, th {
	209	border: 1px solid #dddddd;
	210	text-align: left;
	211	padding: 8px;
	212	}
	213
	214	tr:nth-child(even) {
	215	background-color: #dddddd;
	216	}
	217	</style>
	218	</head>
	219	<body>
	220
114.3	221	<h6><i>H0</i>: The mood distribution before and after the interaction with Pepper is the same.</h6>
91.1	222
	223	<table>
	224	<tr>
114.5	225	<th><i>Wilcoxon Signed-Rank results</i></th>
91.1	226	<th>I feel caring </th>
	227	<th>I feel content </th>
	228	<th>I feel calm </th>
	229	<th>I feel tired </th>
	230	<th>I feel sad </th>
	231	<th>I feel nervous </th>
	232	</tr>
	233	<tr>
	234	<th>p-value</th>
	235	<td>0.512</td>
	236	<td>0.0336 </td>
	237	<td>0.674</td>
	238	<td>0.00156</td>
	239	<td>0.372</td>
	240	<td>0.371</td>
	241	</tr>
	242	<tr>
	243	<th>statistics</th>
	244	<td>42.5</td>
	245	<td>12.5</td>
	246	<td>46</td>
	247	<td>8</td>
	248	<td>33</td>
	249	<td>24</td>
	250	</tr>
	251
	252	</table>
	253
	254	</body>
105.1	255	</html>
	256	{{/html}}
114.1	257
113.2	258	Table 1: Results of wilcoxon statistical test on affect assessment
91.1	259
113.2	260	We analyzed the participants' moods before and after the interaction with Pepper in order to be able to observe positive and negative changes that are caused by the interaction with Pepper. The hypothesis H0 is that Pepper does not have any effect, which would mean that the questionnaires 1 and 2 should give the exact same values for each of the six feelings. However, the graphs and table below show that there is a slight increase regarding positive feelings, and a slight decrease as well regarding negative feelings. The Wilcoxon Signed-rank demonstrated that the only statistically significant change happened for contentness and tiredness based on a p-value threshold of 0.05.
93.1	261
113.2	262	There however exists an important source of bias in the above mentioned result. The main one that we isolated is that the activity of gardening itself could lead to the mood improvement. For this reason, we made subgroups based on whether the participants liked gardening or not.
	263
92.1	264	{{html}}
	265	<!DOCTYPE html>
	266	<html>
	267	<head>
	268	<style>
	269	table {
	270	font-family: arial, sans-serif;
	271	border-collapse: collapse;
	272	width: 100%;
	273	}
91.1	274
92.1	275	td, th {
	276	border: 1px solid #dddddd;
	277	text-align: left;
	278	padding: 8px;
	279	}
91.1	280
92.1	281	tr:nth-child(even) {
	282	background-color: #dddddd;
	283	}
	284	</style>
	285	</head>
	286	<body>
	287
114.5	288	<h6><i>H0</i>: The mood distribution after the interaction with Pepper for people who like gardening and people who do not like gardening is the same.</h6>
92.1	289
	290	<table>
	291	<tr>
114.5	292	<th><i>Wilcoxon Rank-Sum results</i> </th>
92.1	293	<th>I feel caring </th>
	294	<th>I feel content </th>
	295	<th>I feel calm </th>
	296	<th>I feel tired </th>
	297	<th>I feel sad </th>
	298	<th>I feel nervous </th>
	299	</tr>
	300	<tr>
	301	<th>p-value</th>
	302	<td>0.907</td>
	303	<td>0.0883 </td>
	304	<td>0.786</td>
	305	<td>0.510</td>
	306	<td>0.969</td>
	307	<td>0.461</td>
	308	</tr>
	309	<tr>
	310	<th>statistics</th>
	311	<td>0.116</td>
	312	<td>1.704</td>
	313	<td>0.271</td>
	314	<td>-0.658</td>
	315	<td>-0.0387</td>
	316	<td>-0.735</td>
	317	</tr>
	318
	319	</table>
	320
	321	</body>
	322	</html>
	323	{{/html}}
	324
114.5	325	Table 2: Results of Wilcoxon Rank-Sum statistical test on affect assessment for people who like vs. dislike gardening
67.1	326
114.5	327
113.2	328	In order to analyze the difference in the mood change between people who liked the activity of gardening and people who did not, we divided the assessments into two groups and performed a Wilcoxon Rank-Sum test. The results show that only the contentness mood shows a statistically significant difference between the two groups. Hence, we did not notice any relevant pattern indicating a significant difference between "like gardening" and "dislike gardening" groups. It seems to be that this is not the cause of the mood improvement.
98.1	329
113.2	330	This confirms that PwDs can potentially benefit from a boost of energy from interacting with Pepper and, if the activity is enjoyable, a general improvement in contentness as well.
	331
	332
114.7	333	=== System Assessment ===
114.9	334
114.7	335	Task Guidance Assessment
	336	[[System assessment, task guidance questions, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_1.html]]
	337	The results of the system questionnaire will be divided into four different group with similar questions in each group. This is done to improve the readability of the results and provide a more indepth analysis of various aspects of the system setup.
98.1	338
67.1	339	[[image:group1.svg]]
114.7	340	Figure 2: Graphical representation of results for task guidance subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
105.1	341
114.7	342
	343	The first group, namely task guidance assessment, contains questions aimed at measuring how easy and pleasant was the activity when being guided by Pepper. The responses for this group are around the slightly agree line, a bit higher for the "Pepper was easy to understand" statement and a bit lower for the "I enjoyed the task more than if I had had to do it alone".
	344
105.1	345	{{html}}
	346	<!DOCTYPE html>
93.1	347	<html>
	348	<head>
	349	<style>
	350	table {
	351	font-family: arial, sans-serif;
	352	border-collapse: collapse;
	353	width: 100%;
	354	}
	355
	356	td, th {
	357	border: 1px solid #dddddd;
	358	text-align: left;
	359	padding: 8px;
	360	}
	361
	362	tr:nth-child(even) {
	363	background-color: #dddddd;
	364	}
	365	</style>
	366	</head>
	367	<body>
	368
98.1	369	<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
93.1	370
	371	<table>
	372	<tr>
114.7	373	<th><i>Wilcoxon Rank-Sum results</i></th>
93.1	374	<th>I would have known how to do the whole task without Pepper.</th>
	375	<th>I think Pepper made the task easier for me.</th>
	376	<th>I enjoyed the task more than if I had had to do it alone.</th>
	377	<th>Pepper was easy to understand.</th>
	378	<th>I am pleased that Pepper reminded me to do the activity.</th>
	379	</tr>
	380	<tr>
	381	<th>p-value</th>
	382	<td>0.759</td>
	383	<td>0.462</td>
	384	<td>0.668</td>
	385	<td>0.830</td>
	386	<td>0.927</td>
	387	</tr>
	388	<tr>
	389	<th>statistics</th>
	390	<td>-0.306</td>
	391	<td>-0.734</td>
	392	<td>-0.428</td>
	393	<td>-0.214</td>
	394	<td>-0.0918</td>
	395	</tr>
	396
	397	</table>
	398
	399	</body>
105.1	400	</html>
	401	{{/html}}
114.8	402
114.7	403	Table 3: Results of Wilcoxon Rank-Sum statistical test on task guidance subset of system assessment for people who like vs. dislike gardening
93.1	404
	405
114.9	406	Accomplishment and Autonomy Assessment
	407	[[System assessment, accomplishment and autonomy subset, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_2.html]]
67.1	408	[[image:group2.svg]]
114.9	409	Figure 3: Graphical representation of results for accomplishment and autonomy subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
105.1	410
114.9	411	The second group, namely the accomplishment and autonomy subset has questions concerning the sense of control and accomplishment felt during the task by the participants. The participants on average responded between slightly agree and agree that completing the task was a good accomplishment and that they felt in control while doing it and a bit lower for the statement "I feel like I have accomplished it myself" suggestingthat it is possible for the participants to feel like Pepper is responsible, at least partially, for the accomplishment of the task.
	412
105.1	413	{{html}}
	414	<!DOCTYPE html>
93.1	415	<html>
	416	<head>
	417	<style>
	418	table {
	419	font-family: arial, sans-serif;
	420	border-collapse: collapse;
	421	width: 100%;
	422	}
	423
	424	td, th {
	425	border: 1px solid #dddddd;
	426	text-align: left;
	427	padding: 8px;
	428	}
	429
	430	tr:nth-child(even) {
	431	background-color: #dddddd;
	432	}
	433	</style>
	434	</head>
	435	<body>
	436
98.1	437	<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
93.1	438
	439	<table>
	440	<tr>
114.9	441	<th><i>Wilcoxon Rank-Sum results </i></th>
93.1	442	<th>I feel like completing the task was a good accomplishment.</th>
	443	<th>I feel like I accomplished it myself.</th>
	444	<th>I felt in control of what I had to do.</th>
	445	</tr>
	446	<tr>
	447	<th>p-value</th>
	448	<td>0.0982</td>
	449	<td>0.220</td>
	450	<td>0.581</td>
	451	</tr>
	452	<tr>
	453	<th>statistics</th>
	454	<td>-1.653</td>
	455	<td>-1.224</td>
	456	<td>0.551</td>
	457	</tr>
	458
	459	</table>
	460
	461	</body>
105.1	462	</html>
	463	{{/html}}
93.1	464
114.9	465	Table 4: Results of Wilcoxon Rank-Sum statistical test on accomplishment and autonomy subset of system assessment for people who like vs. dislike gardening
98.1	466
101.1	467
114.9	468	The sense of accomplishment is slightly higher for people who like gardening that for those who do not. It is globally around slightly agree. An interesting fact to notice is that participants who do not like gardening felt more in control of what they had to do.
	469
	470
114.10	471	Negative Experiences Assessment
	472	[[System assessment, negative experiences subset, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_3.html]]
67.1	473	[[image:group3.svg]]
114.10	474	Figure 4: Graphical representation of results for negative experiences subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
104.1	475
114.10	476	The third group, namely negative experiences subset is used to group together questions that measure negative feeling experiences with Pepper. The results show that the participants on average answered between slightly disagree and disagreed. This suggests that Pepper was not frustrating for most people but only for a small fraction of the participants.
	477
104.1	478	{{html}}
	479	<!DOCTYPE html>
93.1	480	<html>
	481	<head>
	482	<style>
	483	table {
	484	font-family: arial, sans-serif;
	485	border-collapse: collapse;
	486	width: 100%;
	487	}
	488
	489	td, th {
	490	border: 1px solid #dddddd;
	491	text-align: left;
	492	padding: 8px;
	493	}
	494
	495	tr:nth-child(even) {
	496	background-color: #dddddd;
	497	}
	498	</style>
	499	</head>
	500	<body>
	501
98.1	502	<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
93.1	503
	504	<table>
	505	<tr>
114.10	506	<th><i> Wilcoxon Rank-Sum results </i></th>
93.1	507	<th>I felt annoyed by Pepper.</th>
	508	<th>I felt frustrated by the task.</th>
	509	<th>I felt pressured by Pepper.</th>
	510	</tr>
	511	<tr>
	512	<th>p-value</th>
	513	<td>0.951</td>
	514	<td>0.358</td>
	515	<td>0.926</td>
	516	</tr>
	517	<tr>
	518	<th>statistics</th>
	519	<td>0.0612</td>
	520	<td>0.918</td>
	521	<td>-0.0918</td>
	522	</tr>
	523
	524	</table>
	525
	526	</body>
104.1	527	</html>
	528	{{/html}}
93.1	529
114.10	530	Table 5: Results of Wilcoxon Rank-Sum statistical test on negative experiences subset of system assessment for people who like vs. dislike gardening
	531
	532
98.1	533	The participants globally disagree that the presence of Pepper annoyed, frustrated or pressured them. Those who like gardening actually had a bit more negative feelings regarding the presence of Pepper than those who dislike gardening.
	534
101.1	535
114.11	536	Social Assessment
	537	[[System assessment, social subset, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_4.html]]
67.1	538	[[image:group4.svg]]
114.11	539	Figure 5: Graphical representation of results for social subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
105.1	540
114.11	541	The fourth and final group addresses a social subset and is utilized for assessing Pepper's social presence and trustworthiness as felt by the participants. The two statements used are "Pepper cared about helping me" and "I would trust Pepper with more important activities". The responses were on average slightly above the neutral level.
	542
105.1	543	{{html}}
	544	<!DOCTYPE html>
93.1	545	<html>
	546	<head>
	547	<style>
	548	table {
	549	font-family: arial, sans-serif;
	550	border-collapse: collapse;
	551	width: 100%;
	552	}
	553
	554	td, th {
	555	border: 1px solid #dddddd;
	556	text-align: left;
	557	padding: 8px;
	558	}
	559
	560	tr:nth-child(even) {
	561	background-color: #dddddd;
	562	}
	563	</style>
	564	</head>
	565	<body>
	566
98.1	567	<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
93.1	568
	569	<table>
	570	<tr>
114.11	571	<th><i>Wilcoxon Rank-Sum results</i></th>
93.1	572	<th>Pepper cared about helping me.</th>
	573	<th>I would trust Pepper with more important activities.</th>
	574	</tr>
	575	<tr>
	576	<th>p-value</th>
	577	<td>0.854</td>
	578	<td>0.0297</td>
	579	</tr>
	580	<tr>
	581	<th>statistics</th>
	582	<td>0.183</td>
	583	<td>-2.173</td>
	584	</tr>
	585
	586	</table>
	587
	588	</body>
105.1	589	</html>
	590	{{/html}}
93.1	591
114.11	592	Table 5: Results of Wilcoxon Rank-Sum statistical test on social subset of system assessment for people who like vs. dislike gardening
	593
98.1	594	This graph shows that the trust in Pepper was highly dependent on whether the participants enjoyed the activity or not.
93.1	595
31.1	596	== Discussion ==
	597
113.1	598	* Reliability: The evaluation is reliable. One could replicate the same experiment with other patients.
	599	* Validity: This evaluation is not really valid. Our feasible evaluation does not have the corresponding target group, and is of a much smaller scope compared to our ideal evaluation. We cannot test all our claims.
	600	* Biases: The evaluation has large biases. This is discussed more in detail in the limitations where the different bias factors are explained.
	601	* Scope: The evaluation can be generalized to a larger scope, although with a lot of care, since the evaluation is not fully valid.
	602	* Ecological validity: The evaluation is partially valid in terms of influence from the environment. The affect assessment questionnaire is the same before the activity and after, with the same environment, so the environment is technically not involved in this. However, the system assessment questionnaire does rely on some elements from the environment.
31.1	603
113.1	604	Affect assessment questionnaire
106.1	605
110.1	606	We analyzed the participants' moods before and after the interaction with Pepper in order to be able to observe positive and negative changes that are caused by the interaction with Pepper. the results showed that, in general, there is a slight increase in positive moods and a slight decrease in negative moods. The Wilcoxon Signed-rank demonstrated that the only statistically significant change happened for contentness and tiredness based on a p-value threshold of 0.05.
	607
107.1	608	It is often the case that PwDs have to perform tasks that are not enjoyable for them such as taking medicines or performing routine activities. In order to analyze the difference in the mood change between people who liked the activity and people who didn't we divided into two groups and performed a Wilcoxon Rank-Sum test. The results show that only the contentness mood shows a statistically significant difference between the two groups.
	609	This confirms that PwDs can potentially benefit from a boost of energy from interacting with Pepper and, if the activity is enjoyable, a general improvement in contentness as well.
	610
110.1	611	system questionnaire
107.1	612
110.1	613	We divided the results from the system questionnaire into 4 separate groups containing similar questions as illustrated by the 4 different graphs above.
107.1	614
110.1	615	The first group contains questions aimed at measuring how easy and pleasant was the activity when being guided by Pepper. The responses for this group are around the slightly agree line, a bit higher for the "Pepper was easy to understand" statement and a bit lower for the "I enjoyed the task more than if I had had to do it alone".
107.1	616
110.1	617	The second group has questions concerning the sense of control and accomplishment felt during the task by the participants. The participants on average responded between slightly agree and agree that completing the task was a good accomplishment and that they felt in control while doing it and a bit lower for the statement "I feel like I have accomplished it myself" suggesting
	618	that it is possible for the participants to feel like Pepper is responsible, at least partially, for the accomplishment of the task.
107.1	619
110.1	620	The third group is used to group together questions that measure negative experiences with Pepper. The results show that the participants on average answered between slightly disagree and disagreed. This suggests that Pepper was not frustrating for most people but only for a small fraction of the participants.
107.1	621
110.1	622	The fourth and final group is for assessing Pepper's social presence and trustworthiness as felt by the participants. The two statements used are "Pepper cared about helping me" and "I would trust Pepper with more important activities". The responses were on average slightly above the neutral level.
109.1	623
110.1	624	We performed a statistical test for the system questionnaire to see if the difference between the "like gardening" group and the "dislike gardening" group is significant in any of the questions. The results didn't show any statistical significance with the exception of the question "I would trust Pepper with more important activities" which showed that people who liked gardening were more likely to trust Pepper with more important activities.
109.1	625
110.1	626	observations
109.1	627
110.1	628	Despite having on average good results, some participants still found Pepper frustrating or annoying. In order to understand what could have caused that we analyzed the video of the interaction and the feedback from the participants. We observed that when a participant felt frustrated was often due to Pepper's limitation. For example, sometimes Pepper would start listening too late missing part of the participant's answer. It is also common for the participant to say a word that Pepper is not able to understand which can result in the participant being stuck in a loop during the conversation which can be frustrating. We notice that in most of the experiments the experience and the ease of the interaction with Pepper improved as the participant learned how to interact with Pepper.
109.1	629
110.1	630
	631
	632
	633
	634
	635
	636
	637
	638
31.1	639	== Conclusions ==
97.1	640
	641	The results from the mood questionnaire seem to support our claims CL10: the PwD feels reassured and CL11: the PwD feels content.
	642	Although there are many potential biases, there seems to be a general trend which is that the mood of the participants slightly improved thanks to the activity.
	643
	644	All participants, except one who asked to leave the experiment early, finished the whole activity we had prepared for them during the session. This means the participants were able to perform activity steps told by Pepper. This supports our claim CL03: the PwD performs an activity step.
	645
	646	No participant failed to notice Pepper or did not hear what she was saying after the experiment had started. This supports our claim CL01: the PwD becomes aware of Pepper's presence.
	647
	648	From the system assessment questionnaire, participants quite agree that completing the task was a good accomplishment for them. This supports our claim CL08: the PwD feels accomplished.
	649
112.1	650	We did not have any question explictly aimed at targeting our claim CL08. However, frustration, annoyance and pressure are often linked to a lack of understanding from the other part. We can combine these with the question about whether Pepper cared about helping the participants, and with our observations during the experiment. When aggregated together, it seems that generally speaking, the participants felt understood. This supports our claim CL08: the PwD feels understood.
	651
	652	== Limitations ==
	653
	654	Although there seems to be a slight general trend that shows that our claim are satisfied, there are many limitations to take into account that reduce the significance of the results.
	655
	656	1. Students are not the target group, and especially do not have any form of dementia. This means the effect is probably completely different on them than on actual patients.
	657	1. The evaluation is very generic and does not reflect one of our main functionalities, which is to have a customized agent for a PwD.
	658	1. Students are Master students at TUD. So they are all studying some field related to engineering. Most of them were even Computer Science students and/or students from the SCE class. This means they globally have a very different understanding and familiarity with robots compared to the rest of the population, causing a big bias in their interaction with Pepper.
	659	1. The explanation of Pepper was very short (around 1 min), so many participants had some difficulties interacting with Pepper in the beginning. It improved after some activity steps.
	660	1. Finally, our participants have good intentions and have a positive a priori on our experiment. Therefore, they are more likely to answer with positive results than if they did not know us at all. This again may cause a significant bias in their answers.