Changes for page Test

Last modified by Mathieu Jung-Muller on 2022/04/04 13:52

From 93.1 to 92.1 From 117.1 to 116.1

From version

116.1

edited by Mathieu Jung-Muller
on 2022/04/04 12:20

Change comment: There is no comment for this version

To version

93.1

edited by Pietro Piccini
on 2022/04/02 14:57

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--XWiki.Mathieu
++XWiki.PietroPiccini

Content

@@ -64,32 +64,32 @@
  → feelings from the PwD themselves
  → reports from relatives and HCP
  These results will most likely never be yes-no results, but more like clues or hints that show whether some things worked on not, which will be the point of our discussion.
++NB: This part explains what we expect as a kind of result, it will be replaced by actual results after we perform an experiment with the class. There may also be interesting points we did not think about.
  == Discussion ==
  * Reliability: Yes. One could replicate the same experiment with other patients.
--* Validity: TBD. (This this ideal evaluation has not been conducted, we cannot really estimate this.)
--* Biases: TBD.(This this ideal evaluation has not been conducted, we cannot really estimate this.)
++* Validity: TBD.
++* Biases: TBD.
  * Scope: No. It would be very difficult to generalize the results since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar efforts to customize Pepper for more patients should produce similar effects.
  * Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.
  == Conclusions ==
--(Since this ideal evaluation has not been conducted, we do not have conclusions to be drawn.)
--
  = Feasible evaluation (students) =
  == Problem statement and research questions ==
--For the prototype that we have designed now, and for a smaller scope evaluation, the research questions are smaller-scope versions of our main research questions for the previous part.
--We also cannot test all our claims during only a short evaluation session with students.
++This project uses a Social Cognitive Engineering (SCE) approach to guide the design and research process. The SCE method provides a systematic approach to our study of robots for PwDs. The main goal of our application is to improve the well-being of the person with dementia (PwD) and of those living with them.
--1. Are the participants able to use our prototype smoothly?
--1. Does the prototype improve the autonomy of the patient on a specific activity?
--1. Does the prototype improve the emotional state of the participant after completing a specific activity?
++For the prototype that we have designed now, these are some research questions that we want to address.
++1. Are the different stakeholders able to use our prototype smoothly?
++1. Does the prototype allow the PwD greater autonomy in their day-to-day life?
++1. Does the prototype improve the emotional state of the PwD and their relatives?
++
  Based on these questions and our evaluation setting, we can evaluate some of our claims from our Claim page.
 . CL01: The PwD becomes aware of Pepper's presence.
@@ -107,18 +107,22 @@
  == Method ==
--In order to collect qualitative data, we prepared two questionnaires to give to the participants: the affect assessment questionnaire and the system assessment questionnaire. All questionnaire questions are expressed in the form of statements and the participant can express one out of seven levels of agreement/disagreement from strongly disagree to strongly agree. The only exception is the additional remarks field at the end of the system assessment questionnaire, where the participants can freely put any comment/remark/feedback they have on the experiment.
++In our situation, we recruit 20 students in our class to simulate the research. Since they are not real PwD, we ask them to act as if they were in home settings and we observe their behavior and expression. These data will also be reviewed after the evaluation to obtain the data and feedback.
++==== Questionnaires ====
++
++In order to collect qualitative data, we prepared two questionnaires to give to the participants: the affect assessment questionnaire and the system assessment questionnaire. All questionnaire questions are expressed in the form of statements and the participant can express one out of seven levels of agreement/disagreement from strongly disagree to strongly agree with the only exception being the additional remarks field at the end of the system assessment questionnaire.
++
  **Affect assessment**
--The affect assessment questionnaire asks the participant to describe their mood. The questionnaire is given two times: once before the experiments and once right after. The purpose of this questionnaire is to compare the mood experienced by the participant before the interaction with Pepper and after. The idea is to assess the effect of the activity. The questionnaire has six questions each designed to measure the level of a certain feeling experienced at the moment of filling the questionnaire. The six feelings we take into consideration are evaluated from the following statements:
++The affect assessment questionnaire asks the participant to describe their mood and feelings experienced during a task. The questionnaire is given two times: once before the experiments and once right after. The purpose of this questionnaire is to compare the feelings experienced by the participant before the interaction with Pepper and after to study how Pepper influences the participant's feelings. The questionnaire has six questions each designed to measure the level of a certain feeling experienced at the moment of filling the questionnaire. The six feelings we take into consideration are evaluated from the following statements:
--1. I feel sad.
--1. I feel content.
--1. I feel calm.
--1. I feel tired.
--1. I feel nervous.
--1. I feel caring.
++1. I feel sad
++1. I feel content
++1. I feel calm
++1. I feel tired
++1. I feel nervous
++1. I feel caring
  **System assessment**
@@ -125,8 +125,8 @@
  The system assessment questionnaire is given after the interaction with Pepper and its purpose is to assess the participant's experience with Pepper in more detail. The questionnaire has fourteen questions which are designed to answer research questions as described below:
 . I like gardening
--1. I would have known how to do the whole task without Pepper.
 . I think Pepper made the task easier for me.
++1. I would have known how to do the whole task without Pepper.
 . Pepper was easy to understand.
 . I enjoyed the task more than if I had had to do it alone.
 . I am pleased that Pepper reminded me to do the activity.
@@ -141,7 +141,7 @@
  Research questions:
--1. To what extent did Pepper improve the task's experience? [Questions: 3,4,5,10,11,12]
++1. To what extent did Pepper improve the task's experience? [Questions: 2,3,4,5,10,11,12]
 . To what extent did Pepper improve the participant's autonomy and perception of control? [Questions: 6, 8, 9]
 . To what extent did Pepper improve the participant's sense of accomplishment? [Questions: 7]
 . Did the participant perceive Pepper as a social agent? [Questions: 13, 14]
@@ -151,48 +151,47 @@
  == Participants ==
--In our situation, our participants are 24 students from TUD. Most of them come from Computer Science and/or from the SCE class. One third of them are female and two thirds male, so gender balance is decent.
--Roleplaying PwDs may be difficult for students, especially for those who have not followed the course. For this reason, and in order not to introduce an extra bias "knows about dementia"/"does not know about dementia", we decided to ask the students to just act as students.
++For this study, we simulate the real research by including 24 students from TU Delft. One third of them were female and two thirds male, so gender balance was decent. The students were not asked to roleplay someone with dementia, but simply to follow the experiment and do the activity.
  == Experimental design ==
--Since our evaluating process is relatively short, we use within-subject, which means each participant goes through all conditions. In this way, our experiment tends to have more statistical power and less variability.
++Since our evaluating process is relatively short, we use within-subject, which means each participant goes through all conditions. In this way, our experiment tends to have more statistical power and less variability. Furthermore, every PwD may have different issues in their day-to-day life, while also not having the same living conditions (alone, living with husband, family, etc), thus would require a different treatment. But in our setting, we use the same setting for every participant.
  == Tasks ==
--All participants will go through our designed testing process, which combines a calendar event reminder and an activity breakdown.
--Pepper will propose the activity to the participant, then go step by step with them through the activity.
++All participants will go through our designed testing process, which includes medication/meal/activity reminder and activity breakdown.
++Medication/meal/activity reminder: The robot will remind the patient of daily activities, through which we can see the effectiveness according to their reactions.
++Activity breakdown: In this part, the robot will break down some complex tasks into a list of simple tasks and PwDs can follow the steps to accomplish complex tasks.
++
  == Measures ==
--We measure the mood of the participants through the affect assessment questionnaires. We also measure some aspects of the interaction between Pepper and the participant through the system assessment questionnaire. Both are qualitative data, as the participant gives an answer ranging from strongly disagree to strongly agree.
++We are planning to make behavioural and emotional measurements.
++Behavioral measurements are the actions that the PwD is going to perform during the week, so it can be considered as subjective quantitative data. This will involve the relatives, the HCP and the PwD themselves to quantify whether the use of Pepper did actually result in an increase in autonomy for the PwD.
++Emotional measurements are more related to state of mind, change of expression and mood, so it can be considered as qualitative data. This can be measured by frequent talks with the PwD, either by the relatives or the HCP.
++Measures will be done by oral discussions with the PwD, HCP and relatives.
  == Procedure ==
--Before the experiment, we explain to the participants the purpose of our evaluation (context of SCE class, our prototype).
--We tell them that we will gather data in the questionnaires, that the data are fully anonymized and will only be used for our result analysis in this course.
--We also briefly explain to them how Pepper works, especially that she only listens when the eyes are blue.
--After the explanation, the participant can fill the consent form and the first affect questionnaire.
--Then, the experiment starts. The experiment is conducted in autonomy between the participant and Pepper. We only intervene if asked by the participant or if something is going wrong with the experiment.
--After the activity is finished by the participant, they fill the second affect questionnaire and the system assessment questionnaire.
--After this is done, we thank them for their participation and the evaluation session is over.
++At the start of the evaluation, all participants will be gathered in the same room, together with the researchers and relatives. We will explain the whole evaluation process, provide simple instructions, explain that participation in the evaluations is voluntary, and participants are free to stop the evaluation at any time. Then, we will explain how the gathered data will be analyzed and help us to improve our prototype. All data will be kept private. We will also emphasize that if there is anything the participants dislike, they should let us know. The purpose of the study is to find out what they think of the prototype, and their honesty is greatly appreciated. Then, we will ask them to sign the consent form.
++We will simulate a home setting, which is the most common scenario for PwDs. All PwDs are going to complete the evaluation separately.
++Our robot will give corresponding prompts. For example, “It’s time to have lunch”, “Medicine time!”, “Today is your birthday. Happy Birthday!”.
++We will record the reaction and behavior of participants after they heard the prompts.
++To test the activity breakdown function, our robot will choose a relatively complex task to perform, such as making a paper plane, doing some exercise. Our robot will break it down into simple steps.
++Record the reactions of participants and evaluate how is the activity accomplished.
++
  == Material ==
--1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers. The consent form is accessed online via a QR code.
--1. Questionnaires. There are three questionnaires to answer (mood 1, mood 2, system). They are also accessed online via a QR code.
++1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.
 . Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behaviour for every participant. However, the input data will be entered by the HCP (and potentially the relatives).
--1. Gardening stuff. In order to conduct the activity, we bought and brought basic stuff required for gardening (soil, pot, seeds, etc).
  == Results ==
--The following sections contains the results gather from the affect and system assessment. Apart from the printed version being included below, the results are also included as interactive graphs. To get a better version with more information (data point information on mouse hovering for instance), simply click the provided link for each graph.
++=== mood questionnaire results ===
--=== Affect assessment ===
--
--[[Affect assessment, interactive version>>https://pietro99.github.io/SCE/graphs/mood_questionnaire.html]]
++[[interactive version>>https://pietro99.github.io/SCE/graphs/mood_questionnaire.html]]
  [[image:mood.svg]]
--Figure 1: Graphical results of affect assessment before and after conducting the evaluation activity with Pepper.
  {{html}}
  <!DOCTYPE html>
@@ -218,11 +218,12 @@
  </head>
  <body>
--<h6><i>H0</i>: The mood distribution before and after the interaction with Pepper is the same.</h6>
++<h5>Wilcoxon Signed-rank test</h5>
++<h6><i>H0</i>: The mood distribution before and after the interaction with Pepper is the same</h6>
  <table>
    <tr>
--    <th><i>Wilcoxon Signed-Rank results</i></th>
++    <th>test results </th>
      <th>I feel caring </th>
      <th>I feel content </th>
      <th>I feel calm </th>
@@ -253,14 +253,10 @@
  </body>
  </html>
++
  {{/html}}
--Table 1: Results of wilcoxon statistical test on affect assessment
--We analyzed the participants' moods before and after the interaction with Pepper in order to be able to observe positive and negative changes that are caused by the interaction with Pepper. The hypothesis H0 is that Pepper does not have any effect, which would mean that the questionnaires 1 and 2 should give the exact same values for each of the six feelings. However, the graphs and table below show that there is a slight increase regarding positive feelings, and a slight decrease as well regarding negative feelings. The Wilcoxon Signed-rank demonstrated that the only statistically significant change happened for contentness and tiredness based on a p-value threshold of 0.05.
--
--There however exists an important source of bias in the above mentioned result. The main one that we isolated is that the activity of gardening itself could lead to the mood improvement. For this reason, we made subgroups based on whether the participants liked gardening or not.
--
  {{html}}
  <!DOCTYPE html>
  <html>
@@ -285,11 +285,12 @@
  </head>
  <body>
--<h6><i>H0</i>: The mood distribution after the interaction with Pepper for people who like gardening and people who do not like gardening is the same.</h6>
++<h5>Wilcoxon Rank-Sum test</h5>
++<h6><i>H0</i>: The mood distribution after the interaction with Pepper for people who like gardening and people who don't like gardening is the same</h6>
  <table>
    <tr>
--    <th><i>Wilcoxon Rank-Sum results</i> </th>
++    <th>test results </th>
      <th>I feel caring </th>
      <th>I feel content </th>
      <th>I feel calm </th>
@@ -320,28 +320,15 @@
  </body>
  </html>
++
  {{/html}}
--Table 2: Results of Wilcoxon Rank-Sum statistical test on affect assessment for people who like vs. dislike gardening
++=== System questionnaire results ===
--
--In order to analyze the difference in the mood change between people who liked the activity of gardening and people who did not, we divided the assessments into two groups and performed a Wilcoxon Rank-Sum test. The results show that only the contentness mood shows a statistically significant difference between the two groups. Hence, we did not notice any relevant pattern indicating a significant difference between "like gardening" and "dislike gardening" groups. It seems to be that this is not the cause of the mood improvement.
--
--This confirms that PwDs can potentially benefit from a boost of energy from interacting with Pepper and, if the activity is enjoyable, a general improvement in contentness as well.
--
--
--=== System Assessment ===
--
--**Task Guidance Assessment**
--[[System assessment, task guidance questions, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_1.html]]
--The results of the system questionnaire will be divided into four different group with similar questions in each group. This is done to improve the readability of the results and provide a more indepth analysis of various aspects of the system setup.
--
++[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_1.html]]
  [[image:group1.svg]]
--Figure 2: Graphical representation of results for task guidance subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
--The first group, namely task guidance assessment, contains questions aimed at measuring how easy and pleasant was the activity when being guided by Pepper. The responses for this group are around the slightly agree line, a bit higher for the "Pepper was easy to understand" statement and a bit lower for the "I enjoyed the task more than if I had had to do it alone".
--
  {{html}}
  <!DOCTYPE html>
  <html>
@@ -366,11 +366,12 @@
  </head>
  <body>
--<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
++<h5>Wilcoxon Rank-Sum test test</h5>
++<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>
  <table>
    <tr>
--    <th><i>Wilcoxon Rank-Sum results</i></th>
++    <th>test results </th>
      <th>I would have known how to do the whole task without Pepper.</th>
      <th>I think Pepper made the task easier for me.</th>
      <th>I enjoyed the task more than if I had had to do it alone.</th>
@@ -398,17 +398,13 @@
  </body>
  </html>
++
  {{/html}}
--Table 3: Results of Wilcoxon Rank-Sum statistical test on task guidance subset of system assessment for people who like vs. dislike gardening
--
--**Accomplishment and Autonomy Assessment**
--[[System assessment, accomplishment and autonomy subset, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_2.html]]
++[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_2.html]]
  [[image:group2.svg]]
--Figure 3: Graphical representation of results for accomplishment and autonomy subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
--The second group, namely the accomplishment and autonomy subset has questions concerning the sense of control and accomplishment felt during the task by the participants. The participants on average responded between slightly agree and agree that completing the task was a good accomplishment and that they felt in control while doing it and a bit lower for the statement "I feel like I have accomplished it myself" suggestingthat it is possible for the participants to feel like Pepper is responsible, at least partially, for the accomplishment of the task.
  {{html}}
  <!DOCTYPE html>
@@ -434,11 +434,12 @@
  </head>
  <body>
--<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
++<h5>Wilcoxon Rank-Sum test test</h5>
++<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>
  <table>
    <tr>
--    <th><i>Wilcoxon Rank-Sum results </i></th>
++    <th>test results </th>
      <th>I feel like completing the task was a good accomplishment.</th>
      <th>I feel like I accomplished it myself.</th>
      <th>I felt in control of what I had to do.</th>
@@ -460,21 +460,12 @@
  </body>
  </html>
++
  {{/html}}
--Table 4: Results of Wilcoxon Rank-Sum statistical test on accomplishment and autonomy subset of system assessment for people who like vs. dislike gardening
--
--
--The sense of accomplishment is slightly higher for people who like gardening that for those who do not. It is globally around slightly agree. An interesting fact to notice is that participants who do not like gardening felt more in control of what they had to do.
--
--
--**Negative Experiences Assessment**
--[[System assessment, negative experiences subset, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_3.html]]
++[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_3.html]]
  [[image:group3.svg]]
--Figure 4: Graphical representation of results for negative experiences subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
--The third group, namely negative experiences subset is used to group together questions that measure negative feeling experiences with Pepper. The results show that the participants on average answered between slightly disagree and disagreed. This suggests that Pepper was not frustrating for most people but only for a small fraction of the participants.
--
  {{html}}
  <!DOCTYPE html>
  <html>
@@ -499,11 +499,12 @@
  </head>
  <body>
--<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
++<h5>Wilcoxon Rank-Sum test test</h5>
++<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>
  <table>
    <tr>
--    <th><i> Wilcoxon Rank-Sum results </i></th>
++    <th>test results </th>
      <th>I felt annoyed by Pepper.</th>
      <th>I felt frustrated by the task.</th>
      <th>I felt pressured by Pepper.</th>
@@ -525,20 +525,12 @@
  </body>
  </html>
++
  {{/html}}
--Table 5: Results of Wilcoxon Rank-Sum statistical test on negative experiences subset of system assessment for people who like vs. dislike gardening
--
--
--The participants globally disagree that the presence of Pepper annoyed, frustrated or pressured them. Those who like gardening actually had a bit more negative feelings regarding the presence of Pepper than those who dislike gardening.
--
--
--**Social Assessment**
--[[System assessment, social subset, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_4.html]]
++[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_4.html]]
  [[image:group4.svg]]
--Figure 5: Graphical representation of results for social subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
--The fourth and final group addresses a social subset and is utilized for assessing Pepper's social presence and trustworthiness as felt by the participants. The two statements used are "Pepper cared about helping me" and "I would trust Pepper with more important activities". The responses were on average slightly above the neutral level.
  {{html}}
  <!DOCTYPE html>
@@ -564,11 +564,12 @@
  </head>
  <body>
--<h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
++<h5>Wilcoxon Rank-Sum test test</h5>
++<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>
  <table>
    <tr>
--    <th><i>Wilcoxon Rank-Sum results</i></th>
++    <th>test results </th>
      <th>Pepper cared about helping me.</th>
      <th>I would trust Pepper with more important activities.</th>
    </tr>
@@ -587,72 +587,31 @@
  </body>
  </html>
++
  {{/html}}
--Table 5: Results of Wilcoxon Rank-Sum statistical test on social subset of system assessment for people who like vs. dislike gardening
--This graph shows that the trust in Pepper was highly dependent on whether the participants enjoyed the activity or not.
++Since each PwD has its own state of dementia and personal issues, it is very difficult to get uniform results, especially since they are collected orally.
++Getting very nice, fully robust and reliable results, is merely a hope and a dream.
++However, we can try to consider the main trends that we are interested in.
++Thus, the results wil be mainly focused on:
++- How much autonomy did the PwD gain?
++→ what did the HCP, relatives and PwD report
++→ how many tasks did they perform that they didn't do previously
++→ did the relatives feel they had more time for themselves
++- Did their emotional state improve?
++→ feelings from the PwD themselves
++→ reports from relatives and HCP
++These results will most likely never be yes-no results, but more like clues or hints that show whether some things worked on not, which will be the point of our discussion.
++NB: This part explains what we expect as kind of results, it will be replaced by actual results after we perform an experiment with the class. There may also be interesting points we did not think about.
++
  == Discussion ==
--=== Evaluation key properties ===
++* Reliability: Yes. One could replicate the same experiment with other patients.
++* Validity: TBD.
++* Biases: TBD.
++* Scope: No. It would be very difficult to generalize the results, since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar effort to customize Pepper for more patients should produce similar effects.
++* Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.
--* Reliability: The evaluation is reliable. One could replicate the exact same experiment with other participants.
--* Validity: This evaluation is not really valid. Our feasible evaluation does not have the corresponding target group, and is of a much smaller scope compared to our ideal evaluation. We cannot test all our claims.
--* Biases: The evaluation has large biases. This is discussed more in detail in the limitations where different bias factors are explained.
--* Scope: The evaluation can be generalized to a larger scope, although with a lot of care, since the evaluation is not fully valid.
--* Ecological validity: The evaluation is partially valid in terms of influence from the environment. The affect assessment questionnaire is the same before and after the activity, with the same environment, so the environment is technically not involved in this. However, the system assessment questionnaire does rely on some elements from the environment.
--
--=== Results discussion ===
--
--As detailed in the results section, the mood of the participants slightly improved between before and after the activity.
--However, based on the Wilcoxon test, the results have only a small significance. Most notably, the significant improvements are only that the participants on average felt more content and less tired. This can be explained by the fact that they enjoyed and got motivated by the activity, but it could also simply be the case of participating in an experiment and testing out the stuff we had prepared for them. Because the participants are not our target group, and although there is a slightly significant result, we cannot really conclude that our activity really is the cause for the mood improvement. More participants and, potentially, control groups would be required to validate the results.
--
--In terms of task guidance, the answers are mostly around slightly agree. This means Pepper's impact was quite positive for the participants. Furthermore, participants generally agree that Pepper was easy to understand. This is a good result, but needs to be nuanced: our participants are Masters students so they are probably more used to robots than the average person, thus giving a positive bias in this question.
--
--For the accomplishment and autonomy part, the answers are around slightly agree. Participants who like gardening have a slightly better feeling of accomplishment. However, participants who dislike gardening felt more in control. This may be explained by the fact that they are less proactive in the activity because they enjoy it less, so Pepper telling them the task is enough for them. On the other side, participants who like gardening may want to go faster and see Pepper as an unnecessary control.
--
--In terms of negative experiences, the answers are between slightly disagree and disagree. This means Pepper did not cause by herself the participants to experience negative feelings, which is already a great result. The participants who like gardening answered a bit lower than those who dislike. This may be explained by the same reason as for the previous part.
--
--Both results for the accomplishment and autonomy part and the negative experiences part are to be taken with extreme care. Since the participants do not have dementia, their attitude towards the activity is most likely very different than for people affected by dementia.
--
--Regarding social assessment, the participants barely agree that Pepper cared about helping them. This may be caused by the fact that most of them are Computer Science and/or SCE class students. Such students are very conscious that Pepper is nothing more than the behaviour we implemented. Some participants even tried to find edge cases to test the answers of Pepper. It would be very useful to conduct the same experiment on average people randomly chosen to see whether the answers are the same. If they are, then that would be a good point to improve.
--Finally, still in terms of social assessment, whether the participants would trust Pepper with more important activities greatly depends on whether they like gardening or not, which highlights the importance of having specific tasks for the specific patients.
--
--
--=== Observations ===
--
--We also made some observations while monitoring the evaluation sessions.
--Although it did not necessarily reflect in the questionnaires, some participants still found Pepper frustrating or annoying. This was often due to Pepper's speech recognition functions. For instance, a participant would say a positive answer and get into the negative loop because Pepper understood their "yep" as "nope".
--We also noticed that participants had some difficulties interacting with Pepper in the beginning. Specifically, and although we did mention it to them, they very often spoke while the eyes were not blue. Since Pepper was not listening, they often got confused and did not know what to do. Most often, they simply repeated their answer, and then it worked. Sometimes, we had to remind them about the blue eyes. Usually, after a couple of steps of the activity, they got used to it. Overall, the ease of interaction with Pepper greatly improved over time.
--
--
--
--
--
--
--
--
--
  == Conclusions ==
--
--The results from the mood questionnaire seem to support our claims CL10: the PwD feels reassured and CL11: the PwD feels content.
--Although there are many potential biases, there seems to be a general trend which is that the mood of the participants slightly improved thanks to the activity.
--
--All participants, except one who asked to leave the experiment early, finished the whole activity we had prepared for them during the session. This means the participants were able to perform activity steps told by Pepper. This supports our claim CL03: the PwD performs an activity step.
--
--No participant failed to notice Pepper or did not hear what she was saying after the experiment had started. This supports our claim CL01: the PwD becomes aware of Pepper's presence.
--
--From the system assessment questionnaire, participants quite agree that completing the task was a good accomplishment for them. This supports our claim CL08: the PwD feels accomplished.
--
--We did not have any question explictly aimed at targeting our claim CL08. However, frustration, annoyance and pressure are often linked to a lack of understanding from the other part. We can combine these with the question about whether Pepper cared about helping the participants, and with our observations during the experiment. When aggregated together, it seems that generally speaking, the participants felt understood. This supports our claim CL08: the PwD feels understood.
--
--== Limitations ==
--
--Although there seems to be a slight general trend that shows that our claim are satisfied, there are many limitations to take into account that reduce the significance of the results.
--
--1. Students are not the target group, and especially do not have any form of dementia. This means the effect is probably completely different on them than on actual patients.
--1. The evaluation is very generic and does not reflect one of our main functionalities, which is to have a customized agent for a PwD.
--1. Students are Master students at TUD. So they are all studying some field related to engineering. Most of them were even Computer Science students and/or students from the SCE class. This means they globally have a very different understanding and familiarity with robots compared to the rest of the population, causing a big bias in their interaction with Pepper.
--1. The explanation of Pepper was very short (around 1 min), so many participants had some difficulties interacting with Pepper in the beginning. It improved after some activity steps.
--1. Finally, our participants have good intentions and have a positive a priori on our experiment. Therefore, they are more likely to answer with positive results than if they did not know us at all. This again may cause a significant bias in their answers.