Changes for page Test

Last modified by Mathieu Jung-Muller on 2022/04/04 13:52

From 114.9 to 114.10

From version

114.10

edited by Sneha Lodha
on 2022/04/04 00:32

Change comment: There is no comment for this version

To version

117.1

edited by Mathieu Jung-Muller
on 2022/04/04 13:52

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--XWiki.snehalodha
++XWiki.Mathieu

Content

@@ -105,6 +105,7 @@
 . CL11: The PwD feels content.
  There is a question related to this in the questionnaire.
++
  == Method ==
  In order to collect qualitative data, we prepared two questionnaires to give to the participants: the affect assessment questionnaire and the system assessment questionnaire. All questionnaire questions are expressed in the form of statements and the participant can express one out of seven levels of agreement/disagreement from strongly disagree to strongly agree. The only exception is the additional remarks field at the end of the system assessment questionnaire, where the participants can freely put any comment/remark/feedback they have on the experiment.
@@ -148,7 +148,9 @@
 . How was Pepper's ability to guide the participant through the task? [Questions: 4]
  Question 1 is used to take into consideration the expected baseline enjoyment of the task for each participant.
++We also intended to use question 2 for similar subgroups, but there were not enough different answers to get a significant number of participants in each subgroup.
++
  == Participants ==
  In our situation, our participants are 24 students from TUD. Most of them come from Computer Science and/or from the SCE class. One third of them are female and two thirds male, so gender balance is decent.
@@ -158,15 +158,18 @@
  Since our evaluating process is relatively short, we use within-subject, which means each participant goes through all conditions. In this way, our experiment tends to have more statistical power and less variability.
++
  == Tasks ==
  All participants will go through our designed testing process, which combines a calendar event reminder and an activity breakdown.
  Pepper will propose the activity to the participant, then go step by step with them through the activity.
++
  == Measures ==
  We measure the mood of the participants through the affect assessment questionnaires. We also measure some aspects of the interaction between Pepper and the participant through the system assessment questionnaire. Both are qualitative data, as the participant gives an answer ranging from strongly disagree to strongly agree.
++
  == Procedure ==
  Before the experiment, we explain to the participants the purpose of our evaluation (context of SCE class, our prototype).
@@ -177,6 +177,7 @@
  After the activity is finished by the participant, they fill the second affect questionnaire and the system assessment questionnaire.
  After this is done, we thank them for their participation and the evaluation session is over.
++
  == Material ==
 . Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers. The consent form is accessed online via a QR code.
@@ -184,6 +184,7 @@
 . Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behaviour for every participant. However, the input data will be entered by the HCP (and potentially the relatives).
 . Gardening stuff. In order to conduct the activity, we bought and brought basic stuff required for gardening (soil, pot, seeds, etc).
++
  == Results ==
  The following sections contains the results gather from the affect and system assessment. Apart from the printed version being included below, the results are also included as interactive graphs. To get a better version with more information (data point information on mouse hovering for instance), simply click the provided link for each graph.
@@ -533,9 +533,13 @@
  The participants globally disagree that the presence of Pepper annoyed, frustrated or pressured them. Those who like gardening actually had a bit more negative feelings regarding the presence of Pepper than those who dislike gardening.
--[[System assessment, fourth subset of questions, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_4.html]]
++**Social Assessment**
++[[System assessment, social subset, interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_4.html]]
  [[image:group4.svg]]
++Figure 5: Graphical representation of results for social subset of the system assessment, with results shown for people who like vs. dislike gardening, along with the average of the sample.
++The fourth and final group addresses a social subset and is utilized for assessing Pepper's social presence and trustworthiness as felt by the participants. The two statements used are "Pepper cared about helping me" and "I would trust Pepper with more important activities". The responses were on average slightly above the neutral level.
++
  {{html}}
  <!DOCTYPE html>
  <html>
@@ -560,12 +560,11 @@
  </head>
  <body>
--<h5>Wilcoxon Rank-Sum test</h5>
  <h6><i>H0</i>: The distribution of answers from people who like gardening and people who do not like gardening is the same.</h6>
  <table>
    <tr>
--    <th>test results </th>
++    <th><i>Wilcoxon Rank-Sum results</i></th>
      <th>Pepper cared about helping me.</th>
      <th>I would trust Pepper with more important activities.</th>
    </tr>
@@ -586,51 +586,44 @@
  </html>
  {{/html}}
++Table 5: Results of Wilcoxon Rank-Sum statistical test on social subset of system assessment for people who like vs. dislike gardening
++
  This graph shows that the trust in Pepper was highly dependent on whether the participants enjoyed the activity or not.
++
  == Discussion ==
--* Reliability: The evaluation is reliable. One could replicate the same experiment with other patients.
++=== Evaluation key properties ===
++
++* Reliability: The evaluation is reliable. One could replicate the exact same experiment with other participants.
  * Validity: This evaluation is not really valid. Our feasible evaluation does not have the corresponding target group, and is of a much smaller scope compared to our ideal evaluation. We cannot test all our claims.
--* Biases: The evaluation has large biases. This is discussed more in detail in the limitations where the different bias factors are explained.
++* Biases: The evaluation has large biases. This is discussed more in detail in the limitations where different bias factors are explained.
  * Scope: The evaluation can be generalized to a larger scope, although with a lot of care, since the evaluation is not fully valid.
--* Ecological validity: The evaluation is partially valid in terms of influence from the environment. The affect assessment questionnaire is the same before the activity and after, with the same environment, so the environment is technically not involved in this. However, the system assessment questionnaire does rely on some elements from the environment.
++* Ecological validity: The evaluation is partially valid in terms of influence from the environment. The affect assessment questionnaire is the same before and after the activity, with the same environment, so the environment is technically not involved in this. However, the system assessment questionnaire does rely on some elements from the environment.
--**Affect assessment questionnaire**
++=== Results discussion ===
--We analyzed the participants' moods before and after the interaction with Pepper in order to be able to observe positive and negative changes that are caused by the interaction with Pepper. the results showed that, in general, there is a slight increase in positive moods and a slight decrease in negative moods. The Wilcoxon Signed-rank demonstrated that the only statistically significant change happened for contentness and tiredness based on a p-value threshold of 0.05.
++As detailed in the results section, the mood of the participants slightly improved between before and after the activity.
++However, based on the Wilcoxon test, the results have only a small significance. Most notably, the significant improvements are only that the participants on average felt more content and less tired. This can be explained by the fact that they enjoyed and got motivated by the activity, but it could also simply be the case of participating in an experiment and testing out the stuff we had prepared for them. Because the participants are not our target group, and although there is a slightly significant result, we cannot really conclude that our activity really is the cause for the mood improvement. More participants and, potentially, control groups would be required to validate the results.
--It is often the case that PwDs have to perform tasks that are not enjoyable for them such as taking medicines or performing routine activities. In order to analyze the difference in the mood change between people who liked the activity and people who didn't we divided into two groups and performed a Wilcoxon Rank-Sum test. The results show that only the contentness mood shows a statistically significant difference between the two groups.
--This confirms that PwDs can potentially benefit from a boost of energy from interacting with Pepper and, if the activity is enjoyable, a general improvement in contentness as well.
++In terms of task guidance, the answers are mostly around slightly agree. This means Pepper's impact was quite positive for the participants. Furthermore, participants generally agree that Pepper was easy to understand. This is a good result, but needs to be nuanced: our participants are Masters students so they are probably more used to robots than the average person, thus giving a positive bias in this question.
--**system questionnaire**
++For the accomplishment and autonomy part, the answers are around slightly agree. Participants who like gardening have a slightly better feeling of accomplishment. However, participants who dislike gardening felt more in control. This may be explained by the fact that they are less proactive in the activity because they enjoy it less, so Pepper telling them the task is enough for them. On the other side, participants who like gardening may want to go faster and see Pepper as an unnecessary control.
--We divided the results from the system questionnaire into 4 separate groups containing similar questions as illustrated by the 4 different graphs above.
++In terms of negative experiences, the answers are between slightly disagree and disagree. This means Pepper did not cause by herself the participants to experience negative feelings, which is already a great result. The participants who like gardening answered a bit lower than those who dislike. This may be explained by the same reason as for the previous part.
--The first group contains questions aimed at measuring how easy and pleasant was the activity when being guided by Pepper. The responses for this group are around the slightly agree line, a bit higher for the "Pepper was easy to understand" statement and a bit lower for the "I enjoyed the task more than if I had had to do it alone".
++Both results for the accomplishment and autonomy part and the negative experiences part are to be taken with extreme care. Since the participants do not have dementia, their attitude towards the activity is most likely very different than for people affected by dementia.
--The second group has questions concerning the sense of control and accomplishment felt during the task by the participants. The participants on average responded between slightly agree and agree that completing the task was a good accomplishment and that they felt in control while doing it and a bit lower for the statement "I feel like I have accomplished it myself" suggesting
--that it is possible for the participants to feel like Pepper is responsible, at least partially, for the accomplishment of the task.
++Regarding social assessment, the participants barely agree that Pepper cared about helping them. This may be caused by the fact that most of them are Computer Science and/or SCE class students. Such students are very conscious that Pepper is nothing more than the behaviour we implemented. Some participants even tried to find edge cases to test the answers of Pepper. It would be very useful to conduct the same experiment on average people randomly chosen to see whether the answers are the same. If they are, then that would be a good point to improve.
++Finally, still in terms of social assessment, whether the participants would trust Pepper with more important activities greatly depends on whether they like gardening or not, which highlights the importance of having specific tasks for the specific patients.
--The third group is used to group together questions that measure negative experiences with Pepper. The results show that the participants on average answered between slightly disagree and disagreed. This suggests that Pepper was not frustrating for most people but only for a small fraction of the participants.
++=== Observations ===
--The fourth and final group is for assessing Pepper's social presence and trustworthiness as felt by the participants. The two statements used are "Pepper cared about helping me" and "I would trust Pepper with more important activities". The responses were on average slightly above the neutral level.
++We also made some observations while monitoring the evaluation sessions.
++Although it did not necessarily reflect in the questionnaires, some participants still found Pepper frustrating or annoying. This was often due to Pepper's speech recognition functions. For instance, a participant would say a positive answer and get into the negative loop because Pepper understood their "yep" as "nope".
++We also noticed that participants had some difficulties interacting with Pepper in the beginning. Specifically, and although we did mention it to them, they very often spoke while the eyes were not blue. Since Pepper was not listening, they often got confused and did not know what to do. Most often, they simply repeated their answer, and then it worked. Sometimes, we had to remind them about the blue eyes. Usually, after a couple of steps of the activity, they got used to it. Overall, the ease of interaction with Pepper greatly improved over time.
--We performed a statistical test for the system questionnaire to see if the difference between the "like gardening" group and the "dislike gardening" group is significant in any of the questions. The results didn't show any statistical significance with the exception of the question "I would trust Pepper with more important activities" which showed that people who liked gardening were more likely to trust Pepper with more important activities.
--**observations**
--
--Despite having on average good results, some participants still found Pepper frustrating or annoying. In order to understand what could have caused that we analyzed the video of the interaction and the feedback from the participants. We observed that when a participant felt frustrated was often due to Pepper's limitation. For example, sometimes Pepper would start listening too late missing part of the participant's answer. It is also common for the participant to say a word that Pepper is not able to understand which can result in the participant being stuck in a loop during the conversation which can be frustrating. We notice that in most of the experiments the experience and the ease of the interaction with Pepper improved as the participant learned how to interact with Pepper.
--
--
--
--
--
--
--
--
--
--
  == Conclusions ==
  The results from the mood questionnaire seem to support our claims CL10: the PwD feels reassured and CL11: the PwD feels content.
@@ -644,6 +644,7 @@
  We did not have any question explictly aimed at targeting our claim CL08. However, frustration, annoyance and pressure are often linked to a lack of understanding from the other part. We can combine these with the question about whether Pepper cared about helping the participants, and with our observations during the experiment. When aggregated together, it seems that generally speaking, the participants felt understood. This supports our claim CL08: the PwD feels understood.
++
  == Limitations ==
  Although there seems to be a slight general trend that shows that our claim are satisfied, there are many limitations to take into account that reduce the significance of the results.