Changes for page Test
Last modified by Mathieu Jung-Muller on 2022/04/04 13:52
From version 
116.1 
116.1
edited by Mathieu Jung-Muller
on 2022/04/04 12:20
on 2022/04/04 12:20
Change comment:
There is no comment for this version
To version 115.1
115.1
edited by Sneha Lodha
on 2022/04/04 00:36
on 2022/04/04 00:36
Change comment:
There is no comment for this version
Summary
Details
-
Page properties
-
- Author
-
... ... @@ -1,1 +1,1 @@ 1 -XWiki. Mathieu1 +XWiki.snehalodha - Content
-
... ... @@ -595,45 +595,25 @@ 595 595 596 596 == Discussion == 597 597 598 -=== Evaluation key properties === 599 - 600 -* Reliability: The evaluation is reliable. One could replicate the exact same experiment with other participants. 598 +* Reliability: The evaluation is reliable. One could replicate the same experiment with other patients. 601 601 * Validity: This evaluation is not really valid. Our feasible evaluation does not have the corresponding target group, and is of a much smaller scope compared to our ideal evaluation. We cannot test all our claims. 602 -* Biases: The evaluation has large biases. This is discussed more in detail in the limitations where different bias factors are explained. 600 +* Biases: The evaluation has large biases. This is discussed more in detail in the limitations where the different bias factors are explained. 603 603 * Scope: The evaluation can be generalized to a larger scope, although with a lot of care, since the evaluation is not fully valid. 604 -* Ecological validity: The evaluation is partially valid in terms of influence from the environment. The affect assessment questionnaire is the same before and after the activity, with the same environment, so the environment is technically not involved in this. However, the system assessment questionnaire does rely on some elements from the environment.602 +* Ecological validity: The evaluation is partially valid in terms of influence from the environment. The affect assessment questionnaire is the same before the activity and after, with the same environment, so the environment is technically not involved in this. However, the system assessment questionnaire does rely on some elements from the environment. 605 605 606 - === Results discussion===604 +**observations** 607 607 608 -As detailed in the results section, the mood of the participants slightly improved between before and after the activity. 609 -However, based on the Wilcoxon test, the results have only a small significance. Most notably, the significant improvements are only that the participants on average felt more content and less tired. This can be explained by the fact that they enjoyed and got motivated by the activity, but it could also simply be the case of participating in an experiment and testing out the stuff we had prepared for them. Because the participants are not our target group, and although there is a slightly significant result, we cannot really conclude that our activity really is the cause for the mood improvement. More participants and, potentially, control groups would be required to validate the results. 606 +Despite having on average good results, some participants still found Pepper frustrating or annoying. In order to understand what could have caused that we analyzed the video of the interaction and the feedback from the participants. We observed that when a participant felt frustrated was often due to Pepper's limitation. For example, sometimes Pepper would start listening too late missing part of the participant's answer. It is also common for the participant to say a word that Pepper is not able to understand which can result in the participant being stuck in a loop during the conversation which can be frustrating. We notice that in most of the experiments the experience and the ease of the interaction with Pepper improved as the participant learned how to interact with Pepper. 610 610 611 -In terms of task guidance, the answers are mostly around slightly agree. This means Pepper's impact was quite positive for the participants. Furthermore, participants generally agree that Pepper was easy to understand. This is a good result, but needs to be nuanced: our participants are Masters students so they are probably more used to robots than the average person, thus giving a positive bias in this question. 612 612 613 -For the accomplishment and autonomy part, the answers are around slightly agree. Participants who like gardening have a slightly better feeling of accomplishment. However, participants who dislike gardening felt more in control. This may be explained by the fact that they are less proactive in the activity because they enjoy it less, so Pepper telling them the task is enough for them. On the other side, participants who like gardening may want to go faster and see Pepper as an unnecessary control. 614 614 615 -In terms of negative experiences, the answers are between slightly disagree and disagree. This means Pepper did not cause by herself the participants to experience negative feelings, which is already a great result. The participants who like gardening answered a bit lower than those who dislike. This may be explained by the same reason as for the previous part. 616 616 617 -Both results for the accomplishment and autonomy part and the negative experiences part are to be taken with extreme care. Since the participants do not have dementia, their attitude towards the activity is most likely very different than for people affected by dementia. 618 618 619 -Regarding social assessment, the participants barely agree that Pepper cared about helping them. This may be caused by the fact that most of them are Computer Science and/or SCE class students. Such students are very conscious that Pepper is nothing more than the behaviour we implemented. Some participants even tried to find edge cases to test the answers of Pepper. It would be very useful to conduct the same experiment on average people randomly chosen to see whether the answers are the same. If they are, then that would be a good point to improve. 620 -Finally, still in terms of social assessment, whether the participants would trust Pepper with more important activities greatly depends on whether they like gardening or not, which highlights the importance of having specific tasks for the specific patients. 621 621 622 622 623 -=== Observations === 624 624 625 -We also made some observations while monitoring the evaluation sessions. 626 -Although it did not necessarily reflect in the questionnaires, some participants still found Pepper frustrating or annoying. This was often due to Pepper's speech recognition functions. For instance, a participant would say a positive answer and get into the negative loop because Pepper understood their "yep" as "nope". 627 -We also noticed that participants had some difficulties interacting with Pepper in the beginning. Specifically, and although we did mention it to them, they very often spoke while the eyes were not blue. Since Pepper was not listening, they often got confused and did not know what to do. Most often, they simply repeated their answer, and then it worked. Sometimes, we had to remind them about the blue eyes. Usually, after a couple of steps of the activity, they got used to it. Overall, the ease of interaction with Pepper greatly improved over time. 628 628 629 629 630 - 631 - 632 - 633 - 634 - 635 - 636 - 637 637 == Conclusions == 638 638 639 639 The results from the mood questionnaire seem to support our claims CL10: the PwD feels reassured and CL11: the PwD feels content.