Skip to Content

Changes for page b. Test

Last modified by Ilinca Rentea on 2023/04/11 12:25
From version 13.1
edited by Ilinca Rentea
on 2023/04/11 12:25
Change comment: There is no comment for this version
To version 8.1
edited by Varun Singh
on 2023/04/10 19:14
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.IlincaRentea
1 +XWiki.varunsingh3000
Content
... ... @@ -3,7 +3,7 @@
3 3  (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)The aim of our project was to provide support for people with dementia in their hobby of painting. This was done to bring some joy and comfort into their lives, especially if they previously enjoyed doing it. This could be done with a human caretaker but having a humanoid robot like Pepper has certain advantages in the overall painting activity.
4 4  
5 5  (% style="line-height:1.38" %)
6 -(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Pepper can [[improve and continue the engagement of PwDs>>doc:2\. Specification.Claims.E1.WebHome]] toward painting. This would be done by gently triggering the PwD to paint and then providing support during the activity. Pepper will also as a result of the painting activity [[improve the quality and life of the PwD>>doc:2\. Specification.Claims.E4.WebHome]]. This can be done by supporting them during painting and also playing some calming, ambient music. The painting activity would also [[bring together the PwD and their family>>doc:2\. Specification.Claims.E3.WebHome]] since during the painting activity personalized recommendations could be given to the PwD to paint and once the painting is done, a picture could be taken and sent to the family. This should provide a much better connectedness and relatedness between the PwD and their family. Pepper would not do the painting activity for the PwD instead it will just guide them so it should also provide a much [[better sense of autonomy towards the task>>doc:2\. Specification.Claims.E2.WebHome]].
6 +(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Pepper can [[improve and continue the engagement of PwDs>>doc:2\. Specification.Claims.E1.WebHome]] towards painting. This would be done by gently triggering the PwD to paint and then providing support during the activity. Pepper will also as a result of the painting activity [[improve the quality and life of the PwD>>doc:2\. Specification.Claims.E4.WebHome]]. This can be done by supporting them during painting and also playing some calming, ambient music. The painting activity would also [[bring together the PwD and their family>>doc:2\. Specification.Claims.E3.WebHome]] since during the painting activity personalised recommendations could be given to the PwD to paint and once the painting is done, a picture could be taken and sent to the family. This should provide a much better connectedness and relatedness between the PwD and their family. Pepper would not do the painting activity for the PwD instead it will just guide them so it should also provide a much [[better sense of autonomy towards the task>>doc:2\. Specification.Claims.E2.WebHome]].
7 7  
8 8  (% style="line-height:1.38" %)
9 9  (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)For all of these effects, we plan on using different questionnaires geared towards capturing the required measure to effectively evaluate our claims.
... ... @@ -14,7 +14,7 @@
14 14  (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)The prototype was evaluated with an in-person experiment with multiple participants in a between-subject study.
15 15  
16 16  == 2.1 Participants
17 -(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Our colleagues from the course CS4235 SCE since the course scope does not allow us to evaluate our robot with actual PwDs.(%%) ==
17 +(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Our colleagues from the course CS4235 SCE since the scope of the course does not allow us to evaluate our robot with actual PwDs.(%%) ==
18 18  
19 19  == 2.2 Experimental Design ==
20 20  
... ... @@ -28,7 +28,7 @@
28 28  (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)For the test condition, we first provided the participant with painting and drawing utensils and a canvas. Then, we started up the robot to guide the activity. The robot was controlled through a Wizard of Oz system.
29 29  
30 30  (% style="line-height:1.38" %)
31 -(% style="color:#000000; font-family:Arial; font-size:14.6667px; white-space:pre-wrap" %)For the control condition, we provided the participant with art supplies as well, but here the robot was not involved. Instead, they received an activity sheet that guided the participants in the same flow as the robot would do. The only main guidance that couldn't be given on paper was the search function for a suitable topic or personal picture to draw/paint.
31 +(% style="color:#000000; font-family:Arial; font-size:14.6667px; white-space:pre-wrap" %)For the control condition, we provided the participant with art supplies as well, but here the robot was not involved. Instead, they received an activity sheet which guided the participants in the same flow as the robot would do. The only main guidance that couldn't be given on paper was the search function for a suitable topic or personal picture to draw/paint.
32 32  
33 33  == 2.4 Measures ==
34 34  
... ... @@ -38,7 +38,7 @@
38 38  (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)We gave each participant a questionnaire before and after the session.
39 39  
40 40  (% style="line-height:1.38" %)
41 -(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)In the before questionnaire, we asked them about their mood, painting experience, and general attitude to robots. This was done to later check if those variables have interaction effects with the main results. See [[here >>https://forms.office.com/Pages/ResponsePage.aspx?id=TVJuCSlpMECM04q0LeCIewsut64TrQ5KmT3QxEFJ2z1URElCMFZGWFFOM085MFVMOExXNVdLRjRSNC4u]]the before questionnaire.
41 +(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)In the before questionnaire, we asked them about their mood, painting experience and general attitude to robots. This was done to later check if those variables have interaction effects with the main results. See [[here >>https://forms.office.com/Pages/ResponsePage.aspx?id=TVJuCSlpMECM04q0LeCIewsut64TrQ5KmT3QxEFJ2z1URElCMFZGWFFOM085MFVMOExXNVdLRjRSNC4u]]the before questionnaire.
42 42  
43 43  (% style="line-height:1.38" %)
44 44  (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)In the after questionnaire, we measured their experience of the activity, using an adapted version of the PACE questionnaire used in [1]. We also asked two extra questions to gauge their likelihood of long-term engagement and perceived self-competence. See [[here >>https://forms.office.com/Pages/ResponsePage.aspx?id=TVJuCSlpMECM04q0LeCIewsut64TrQ5KmT3QxEFJ2z1UN1M0T1pLU1hTN1NIRkw5MjhSWldPRzJUWC4u]]the after questionnaire.
... ... @@ -64,66 +64,108 @@
64 64  
65 65  1. (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Consent form. To protect the privacy of participants and ensure that they accept that their data was analyzed by researchers, we had each participant fill in a consent form. This consent form and experiment were approved by the Ethics Commission of the TU Delft.
66 66  1. (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Pepper robot. We programmed this robot to interact with the participants as described in our earlier sections.
67 -1. (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Utensils for painting and drawing: canvas, watercolors, and pencils. We offer each participant the choice of whether they want to draw or paint.
67 +1. (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Utensils for painting and drawing: canvas, watercolours and pencils. We offer each participant the choice of whether they want to draw or paint.
68 68  1. (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)Microsoft forms. All the questionnaires were performed by Microsoft Forms, a GDPR/TU Delft approved tool for gathering data.
69 69  
70 -== 2.7 Human Research Ethics ==
70 += 3. Results =
71 71  
72 -In order to make sure that we align with TU Delft's standards regarding human research ethics, we have filled in the [[HREC form>>attach:11_HRX-checklist for human research.pdf]] and created an [[informed consent form>>attach:Informed consent form template for research with human participants_May_2018.pdf]].
72 +(% class="wikigeneratedid" %)
73 +**RQ1: How does the assistance provided by a social robot influence engagement in art-related tasks for people with dementia?**
73 73  
74 -= 3. Results =
75 75  
76 -We grouped the results of the PACE questionnaires into two partitions:  One set of questions relating more to RQ1 and one set of questions relating more to RQ2. In the following sections, we will outline the results for both research questions.
76 +○ Independent variable: with/without robot
77 +○ Dependent variable: level of engagement
77 77  
78 78  
80 +**RQ2: How does the assistance provided by a social robot influence the level of agitation in art-related tasks for people with dementia?**
81 +
82 +
83 +○ Independent variable: with/without robot
84 +○ Dependent variable: level of agitation
85 +
86 +
79 79  (% class="wikigeneratedid" %)
80 -**RQ1: How does the assistance provided by a social robot influence engagement in art-related tasks for people with dementia?**
88 +**PACES Questionnaire handpicked questions**
81 81  
82 -[[image:RQ1.png||height="297" width="812"]]
90 +(% class="wikigeneratedid" %)
91 +Q1 Did you enjoy the activity?
83 83  
84 -As shown in the figure above, the results were not exactly significant. The only question that obtained a reasonable p-score (P=0.022) was Q14. This implies that the robot did provide a more immersive activity where the urge to do something else was diminished. Due to the fact that the robot gives more active guidance than paper instructions, it could be hypothesized that the participants were more actively feeling part of the activity.
93 +(% class="wikigeneratedid" %)
94 +Q2 Did you feel interested in the activity?
95 +Q3 Did you like performing the activity?
85 85  
97 +(% class="wikigeneratedid" %)
98 +Q4 Did you find the activity pleasurable?
99 +Q5 Were you absorbed in the activity?
86 86  
87 -**RQ2: How does the assistance provided by a social robot influence the level of agitation in art-related tasks for people with dementia?**
101 +(% class="wikigeneratedid" %)
102 +Q6 Was the activity a lot of fun?
88 88  
89 -[[image:1681152536590-277.png||height="327" width="783"]]
104 +(% class="wikigeneratedid" %)
105 +Q7 Did you find the activity energizing?
90 90  
91 91  (% class="wikigeneratedid" %)
92 -For this research question, the results also proved not to be conclusive. The only result that could be viewed as remotely significant is the result for question 7 (P=0.073). This could be caused either by the positive encouragement given by the robot, and/or the added functionality that the robot could search for pictures to draw.
108 +Q8 Did you the activity make you feel happy?
93 93  
94 94  (% class="wikigeneratedid" %)
95 -**Aggregated score**
111 +Q9 Was the activity pleasant?
96 96  
113 +(% class="wikigeneratedid" %)
114 +Q10 Were you frustrated by the activity?
97 97  
98 -[[image:https://lh5.googleusercontent.com/8pM8mwBwL6UidLnOFDJ_Bjl6EA10teBaiLZe7Wseh8RXDvNImY4MiCzu-ygAUvMvUwBxlN5wV7hsHJZJHZ-x_yGbepWHWqfkLU9HL9mCyHwie8KrSlE9YVBSQuy2DaxRsLuIpRAbEISTCe4X0EFUVJOyuw=s2048||height="260" width="308"]]
116 +(% class="wikigeneratedid" %)
117 +Q11 Did you find the activity to be gratifying?
99 99  
100 -To assess the generally positive impact of the robot, we also aggregated the scores to compare both test settings. This result also proved to be non-conclusive (P= 0.228).
119 +(% class="wikigeneratedid" %)
120 +Q12 Was the activity stimulating for you?
101 101  
102 102  (% class="wikigeneratedid" %)
103 -**After Activity questions**
123 +Q13 Did the activity give you a strong sense of accomplishment?
124 +Q14 Did you feel as though there is nothing else you would rather be doing?
104 104  
126 +
105 105  (% class="wikigeneratedid" %)
106 -We also analyzed the After Activity questions. These were asked to assess the robots' positive influence on [[EF01: Continued and improved engagement in painting>>doc:2\. Specification.Claims.E1.WebHome]] (Q3) and the robots' positive influence on the competence part of the Self Determination Theory (Q4).
128 +**After Activity questions**
107 107  
108 108  (% class="wikigeneratedid" %)
109 -[[image:1681154434106-542.png||height="249" width="582"]]
131 +AQ1 I would like to paint more in the future with the robot.
110 110  
111 111  (% class="wikigeneratedid" %)
112 -The students were agreeing on average with the statement "I would like to paint more in the future with the robot". However, for question 4, the general response was ambivalent. This is understandable as confidence in certain activity often only comes after multiple repetitions and not in one encounter.
134 +AQ2 I feel more confident in painting by myself because of this activity.
113 113  
136 +(% class="wikigeneratedid" id="HRQ1:" %)
137 +**Test Results:**
138 +
139 +The result of 1 is : MannwhitneyuResult(statistic=119.0, pvalue=0.16128927639663737)
140 +The result of 2 is : MannwhitneyuResult(statistic=124.5, pvalue=0.23160773305909133)
141 +The result of 3 is : MannwhitneyuResult(statistic=128.0, pvalue=0.2624600220829313)
142 +The result of 4 is : MannwhitneyuResult(statistic=170.0, pvalue=0.8068245764795088)
143 +The result of 5 is : MannwhitneyuResult(statistic=190.0, pvalue=0.37010115387134357)
144 +The result of 6 is : MannwhitneyuResult(statistic=154.5, pvalue=0.8038002705718079)
145 +The result of 7 is : MannwhitneyuResult(statistic=217.0, pvalue=0.07321208012798906)
146 +The result of 8 is : MannwhitneyuResult(statistic=134.0, pvalue=0.3680655063054846)
147 +The result of 9 is : MannwhitneyuResult(statistic=198.5, pvalue=0.23665845074130787)
148 +The result of 10 is : MannwhitneyuResult(statistic=191.0, pvalue=0.3521959055801669)
149 +The result of 11 is : MannwhitneyuResult(statistic=176.0, pvalue=0.6532884156524246)
150 +The result of 12 is : MannwhitneyuResult(statistic=141.5, pvalue=0.5153332738584296)
151 +The result of 13 is : MannwhitneyuResult(statistic=213.5, pvalue=0.09360342686552618)
152 +The result of 14 is : MannwhitneyuResult(statistic=91.0, pvalue=0.022196094647121984)
153 +
114 114  = 4. Discussion =
115 115  
116 -The results do not show a conclusive effect on the added value of having a robot performing the activity. This is can be attributed to different possible causes.
156 +We followed a within-study approach with a control group and a test group. The control group was first instructed to perform the painting activity without Pepper and then move on to the same activity but this time with Pepper. The test group was told to do vice-versa to minimise any transfer effect.
117 117  
118 -First of all, the questions that were asked were more focused on the experience of the activity rather than the specific added value a robot might give to the activity. Painting can be generally viewed as an enjoyable and engaging activity, so when focusing on the enjoyment part, it could be understood that the robot does not make a significant impact there. However, the robot might give a significant improvement in terms of support, motivation, and structure during the activity. Especially if the test was conducted on actual PwDs. For future work, it would be interesting to explore during the design process which parts of the activity the robot might be able to do more than paper instructions and to investigate how these contributions might be measured.
158 +We used questions Q2, Q3, Q5, Q13, Q14 to answer our research question 1 and questions Q1, Q4, Q6, Q7, Q8, Q9, Q10, Q11, Q12 to answer our research question 2. Based on the results we saw in the above sections, unfortunately, the results we obtained to answer our research questions related to Engagement and Agitation are not significant. These questions were picked from the [[PACES questionnaire>>doc:Main.b\. Human Factors.Measuring Instruments.EV01\: Phsyical Activity Enjoyment Scale.WebHome]]. The aggregated score of the results based on the PACES questionnaire was 0.228 which is not significant enough for our threshold of 0.05. Although one interesting thing to note is that the raw scores for the questions were always higher for questions that were positively associated with the Pepper robot. Although this could also have been because the participants were our colleagues and they could have been biased in their responses.
119 119  
120 -Moreover, as we alluded to earlier, the activity might be experienced highly differently for PwDs than 20-year-old TU Delft students. Therefore, it is hard to draw any conclusions about the use of the robot by testing it on people who do not need much support from it.
160 +The participants were also required to answer two more questions AQ1 and AQ2 after the activity which was used to investigate whether the participants preferred the activity with a robot or without it. The mean responses for AQ1 were in favour of painting with the robot but for AQ2 no definite conclusion could be made. This could be because of the confounding factor related to the painting activity itself; participants who liked to paint preferred the activity either way with or without the robot.
121 121  
162 +Overall our results are inconclusive in terms of statistical significance although the raw scores are in favour of the robot even with the limited participant size test that was conducted. Unfortunately based on the experiment we did, the limitations of the experiment such as not having actual PwDs we are unable to conclusively evaluate the added value of a robot.
163 +
122 122  = 5. Conclusions =
123 123  
124 124  Our project sought to offer support to individuals suffering from dementia in their passion for painting, with the objective of enhancing their emotional well-being. While human caregivers could provide such assistance, utilizing a humanoid robot such as Pepper for painting activities provides several advantages. Pepper has the potential to enhance and sustain the interest of individuals with dementia in painting by encouraging them to paint and providing assistance during the activity. As a result, the quality of life of these individuals could be improved, which may be augmented by playing calming music during the painting process. In addition, this activity could bring together the individuals with their family members, as personalized painting recommendations could be provided, and photos of completed paintings could be shared. Rather than completing the painting activity for the individuals, Pepper would guide them through the process, promoting a greater sense of independence and self-sufficiency.
125 125  
126 -We did a study to evaluate the effectiveness of having a robot like Pepper assisting the PwD with painting instead of the PwD doing the activity alone. Our results based on the responses obtained from our custom PACES questionnaire were statistically insignificant and hence we could not conclusively answer our research questions related to engagement and agitation. However, the raw scores of the questions were always in favor of having the robot while performing the activity which could indicate a positive attitude towards having the robot. Due to certain practical limitations of the course, we could not conduct the experiment with actual PwD or set it up in such a way but given enough time and effort the current implementation in Pepper could be extended to an actual care home with PwDs.
168 +We did a study to evaluate the effectiveness of having a robot like Pepper assisting the PwD with painting instead of the PwD doing the activity alone. Our results based on the responses obtained from our custom PACES questionnaire were statistically insignificant and hence we could not conclusively answer our research questions related to engagement and agitation. However, the raw scores of the questions were always in favour of having the robot while performing the activity which could indicate a positive attitude towards having the robot. Due to certain practical limitations of the course, we could not conduct the experiment with actual PwD or set it up in such a way but given enough time and effort the current implementation in Pepper could be extended to an actual care home with PwDs.
127 127  
128 128  === References ===
129 129  
11_HRX-checklist for human research.pdf
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.IlincaRentea
Size
... ... @@ -1,1 +1,0 @@
1 -370.4 KB
Content
1681152536590-277.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.MarijnRoelvink
Size
... ... @@ -1,1 +1,0 @@
1 -177.5 KB
Content
1681154434106-542.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.MarijnRoelvink
Size
... ... @@ -1,1 +1,0 @@
1 -36.0 KB
Content
Informed consent form template for research with human participants_May_2018.pdf
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.IlincaRentea
Size
... ... @@ -1,1 +1,0 @@
1 -193.0 KB
Content
RQ1.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.MarijnRoelvink
Size
... ... @@ -1,1 +1,0 @@
1 -106.4 KB
Content