Changes for page Test

Last modified by Mathieu Jung-Muller on 2022/04/04 13:52

From 107.1 to 108.1 From 112.1 to 113.1

From version

108.1

edited by Pietro Piccini
on 2022/04/03 15:56

Change comment: There is no comment for this version

To version

112.1

edited by Mathieu Jung-Muller
on 2022/04/03 19:14

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--XWiki.PietroPiccini
++XWiki.Mathieu

Content

@@ -188,12 +188,6 @@
  We produced our results as interactive graphs. Only a printed version is shown below. To get a better version with more information (data point information on mouse hovering for instance), click the provided link for each graph.
--For the statistical test, we used the Wilcoxon test because we do not assume that our data is normally distributed.
--we used the Wilcoxon Signed-rank test to measure the statistical significance of the mood change before and after Pepper's interaction because the data comes from the same group of participants.
--To measure the statistical significance between two different groups (liking gardening group and disliking gardening group) we use the Wilcoxon Rank-Sum test.
--The statistics value of the test is the sum of the difference in the rank.
--the p-value is the probability of obtaining results at least as extreme as the one we obtained given that the null hypothesis is true. We decided to set a threshold of 5% to claim statistical significance.
--
  === Affect assessment ===
  [[Affect assessment, interactive version>>https://pietro99.github.io/SCE/graphs/mood_questionnaire.html]]
@@ -578,16 +578,41 @@
  * Scope: No. It would be very difficult to generalize the results, since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar effort to customize Pepper for more patients should produce similar effects.
  * Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.
--We analyzed the participants' moods before and after the interaction with Pepper in order to be able to observe positive and negative changes that are caused by Pepper. the results showed that, in general, there is a slight increase in positive moods and a slight decrease in negative moods. The Wilcoxon Signed-rank demonstrated that the only statistically significant change happened for contentness and tiredness based on a p-value threshold of 0.05.
++**mood questionnaire**
++We analyzed the participants' moods before and after the interaction with Pepper in order to be able to observe positive and negative changes that are caused by the interaction with Pepper. the results showed that, in general, there is a slight increase in positive moods and a slight decrease in negative moods. The Wilcoxon Signed-rank demonstrated that the only statistically significant change happened for contentness and tiredness based on a p-value threshold of 0.05.
++
  It is often the case that PwDs have to perform tasks that are not enjoyable for them such as taking medicines or performing routine activities. In order to analyze the difference in the mood change between people who liked the activity and people who didn't we divided into two groups and performed a Wilcoxon Rank-Sum test. The results show that only the contentness mood shows a statistically significant difference between the two groups.
  This confirms that PwDs can potentially benefit from a boost of energy from interacting with Pepper and, if the activity is enjoyable, a general improvement in contentness as well.
++**system questionnaire**
++We divided the results from the system questionnaire into 4 separate groups containing similar questions as illustrated by the 4 different graphs above.
++The first group contains questions aimed at measuring how easy and pleasant was the activity when being guided by Pepper. The responses for this group are around the slightly agree line, a bit higher for the "Pepper was easy to understand" statement and a bit lower for the "I enjoyed the task more than if I had had to do it alone".
++The second group has questions concerning the sense of control and accomplishment felt during the task by the participants. The participants on average responded between slightly agree and agree that completing the task was a good accomplishment and that they felt in control while doing it and a bit lower for the statement "I feel like I have accomplished it myself" suggesting
++that it is possible for the participants to feel like Pepper is responsible, at least partially, for the accomplishment of the task.
++The third group is used to group together questions that measure negative experiences with Pepper. The results show that the participants on average answered between slightly disagree and disagreed. This suggests that Pepper was not frustrating for most people but only for a small fraction of the participants.
++The fourth and final group is for assessing Pepper's social presence and trustworthiness as felt by the participants. The two statements used are "Pepper cared about helping me" and "I would trust Pepper with more important activities". The responses were on average slightly above the neutral level.
++
++We performed a statistical test for the system questionnaire to see if the difference between the "like gardening" group and the "dislike gardening" group is significant in any of the questions. The results didn't show any statistical significance with the exception of the question "I would trust Pepper with more important activities" which showed that people who liked gardening were more likely to trust Pepper with more important activities.
++
++**observations**
++
++Despite having on average good results, some participants still found Pepper frustrating or annoying. In order to understand what could have caused that we analyzed the video of the interaction and the feedback from the participants. We observed that when a participant felt frustrated was often due to Pepper's limitation. For example, sometimes Pepper would start listening too late missing part of the participant's answer. It is also common for the participant to say a word that Pepper is not able to understand which can result in the participant being stuck in a loop during the conversation which can be frustrating. We notice that in most of the experiments the experience and the ease of the interaction with Pepper improved as the participant learned how to interact with Pepper.
++
++
++
++
++
++
++
++
++
++
  == Conclusions ==
  The results from the mood questionnaire seem to support our claims CL10: the PwD feels reassured and CL11: the PwD feels content.
@@ -599,4 +599,14 @@
  From the system assessment questionnaire, participants quite agree that completing the task was a good accomplishment for them. This supports our claim CL08: the PwD feels accomplished.
--We did not have any question explictly aimed at targeting our claim CL08. However, frustration, annoyance and pressure are often linked to a lack of understanding from the other part. We can combine these with the question about whether Pepper cared about helping the participants, and with our observations during the experiment. When aggregated together, it seems that generally speaking, the participants felt understood. This supports our claim CL08: the PwD feels understood. However, we did notice frustration a couple of times from the participants, because of Pepper's speech recognition system.
++We did not have any question explictly aimed at targeting our claim CL08. However, frustration, annoyance and pressure are often linked to a lack of understanding from the other part. We can combine these with the question about whether Pepper cared about helping the participants, and with our observations during the experiment. When aggregated together, it seems that generally speaking, the participants felt understood. This supports our claim CL08: the PwD feels understood.
++
++== Limitations ==
++
++Although there seems to be a slight general trend that shows that our claim are satisfied, there are many limitations to take into account that reduce the significance of the results.
++
++1. Students are not the target group, and especially do not have any form of dementia. This means the effect is probably completely different on them than on actual patients.
++1. The evaluation is very generic and does not reflect one of our main functionalities, which is to have a customized agent for a PwD.
++1. Students are Master students at TUD. So they are all studying some field related to engineering. Most of them were even Computer Science students and/or students from the SCE class. This means they globally have a very different understanding and familiarity with robots compared to the rest of the population, causing a big bias in their interaction with Pepper.
++1. The explanation of Pepper was very short (around 1 min), so many participants had some difficulties interacting with Pepper in the beginning. It improved after some activity steps.
++1. Finally, our participants have good intentions and have a positive a priori on our experiment. Therefore, they are more likely to answer with positive results than if they did not know us at all. This again may cause a significant bias in their answers.