Skip to Content

Changes for page b. Test

Last modified by Demi Tao on 2023/04/10 10:13
From version 23.1
edited by Demi Tao
on 2023/04/10 09:57
Change comment: There is no comment for this version
To version 11.1
edited by Karthik Prakash
on 2023/04/07 20:06
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.DemiTao
1 +XWiki.KarthikPrakash
Content
... ... @@ -93,83 +93,34 @@
93 93  
94 94  = 3. Results =
95 95  
96 -=== (% style="color:inherit; font-family:inherit" %)Results of the survey:(%%) ===
96 +|=(% style="width: 199px;" %)Tasks|=(% style="width: 147px;" %)Succeded by Themselves|=(% style="width: 146px;" %)Succeded with Some Guidance|=(% style="width: 185px;" %)Succeded with Detailed Explicit Instructions|=(% style="width: 175px;" %)Average Time to Complete Task (s)
97 +|(% style="width:199px" %)Add a reminder|(% style="width:147px" %)14.29%|(% style="width:146px" %)28.57%|(% style="width:185px" %)57.14%|(% style="width:175px" %)75
98 +|(% style="width:199px" %)Check weekly remainders on the Calendar page|(% style="width:147px" %)100%|(% style="width:146px" %)0%|(% style="width:185px" %)0%|(% style="width:175px" %)10
99 +|(% style="width:199px" %)Create a personal profile|(% style="width:147px" %)7.14%|(% style="width:146px" %)50%|(% style="width:185px" %)42.86%|(% style="width:175px" %)43
100 +|(% style="width:199px" %)Verify current profiles|(% style="width:147px" %)85.71%|(% style="width:146px" %)14.29%|(% style="width:185px" %)0%|(% style="width:175px" %)10
101 +|(% style="width:199px" %)Play memory game|(% style="width:147px" %)0%|(% style="width:146px" %)42.86%|(% style="width:185px" %)57.14%|(% style="width:175px" %)200
97 97  
98 -[[Figure: //Percentage of user satisfaction and SUS score//>>image:attach:chart.png]]
103 +if possible, note down some Parts Where Users Struggled in each task.
99 99  
100 -(% class="wikigeneratedid" %)
101 -As mentioned earlier, the user evaluation incorporated two quantitative measures. The first measure evaluated the various attributes of the system, including accessibility, trustworthiness, perceivability, understandability, and empowerment. The second measure employed was the System Usability Scale (SUS).
102 -
103 -(% class="wikigeneratedid" %)
104 -The attributes-related evaluation was analyzed based on the following way: if a respondent had a minimum total score of 60% (15 out of 25) or more, he or she was considered to be satisfied with the application. 11 out of 14 (78.57%) of the users achieved a score of 15 or higher. The average score is 18. According to the standard operating protocol (Quintana et al., 2020), the feasibility test was to be considered successfully completed if at least 75% were satisfied with the use of the application. Therefore, based on this criterion, the feasibility test was considered successfully completed.
105 -
106 -The System Usability Scale (SUS) was interpreted in terms of percentile ranking. The average SUS score for the stand-alone application is 54.17 (grade D), and that of the robot is 71.86 (grade B). Based on research, a SUS score above 68 would be considered above average and anything below 68 is below average. As a result, the usability of the robot was from this point of view considered successfully completed.
107 -
108 -
109 -|(% style="width:215px" %)**Attributes**|(% style="width:211px" %)**Mean (control group)**|(% style="width:229px" %)**Mean (Experimental group)**|(% style="width:197px" %)**P-value**
110 -|(% style="width:215px" %)Accessibility|(% style="width:211px" %)2,33|(% style="width:229px" %)3,25  |(% style="width:197px" %)0,0644
111 -|(% style="width:215px" %)Trustability|(% style="width:211px" %)3,83|(% style="width:229px" %)4,125|(% style="width:197px" %)0,3165
112 -|(% style="width:215px" %)Perceivability|(% style="width:211px" %)3,33|(% style="width:229px" %)3,5|(% style="width:197px" %)0,4112
113 -|(% style="width:215px" %)Understandability|(% style="width:211px" %)3,33|(% style="width:229px" %)4,25|(% style="width:197px" %)0,1151
114 -|(% style="width:215px" %)Empowerment|(% style="width:211px" %)3,33|(% style="width:229px" %)4|(% style="width:197px" %)0,0895
115 -|(% style="width:215px" %)Usability|(% style="width:211px" %)54,16666667|(% style="width:229px" %)71,875|(% style="width:197px" %)0,0903
116 -
117 -(% class="wikigeneratedid" %)
118 -//Table: User evaluation score//
119 -
120 -=== Observation: (Total percentage sums up to 100) ===
121 -
122 -|=(% style="width: 199px;" %)Tasks|=(% style="width: 147px;" %)Succeeded by Themselves|=(% style="width: 146px;" %)Succeeded with Some Guidance|=(% style="width: 185px;" %)Succeeded with Detailed Explicit Instructions|=(% style="width: 175px;" %)Average Time to Complete Task (s)
123 -|(% style="width:199px" %)Add a reminder|(% style="width:147px" %)14.29%|(% style="width:146px" %)28.57%|(% style="width:185px" %)57.14%|(% style="width:175px" %)97
124 -|(% style="width:199px" %)Check weekly remainders on the Calendar page|(% style="width:147px" %)100%|(% style="width:146px" %)NA|(% style="width:185px" %)NA|(% style="width:175px" %)36
125 -|(% style="width:199px" %)Create a personal profile|(% style="width:147px" %)7.14%|(% style="width:146px" %)50%|(% style="width:185px" %)42.86%|(% style="width:175px" %)69
126 -|(% style="width:199px" %)Verify current profiles|(% style="width:147px" %)85.71%|(% style="width:146px" %)14.29%|(% style="width:185px" %)NA|(% style="width:175px" %)32
127 -|(% style="width:199px" %)Play memory game|(% style="width:147px" %)0%|(% style="width:146px" %)42.86%|(% style="width:185px" %)57.14%|(% style="width:175px" %)208
128 -
129 -//Table: Results of user performance of tasks //
130 -
131 -
132 -|(% style="width:330px" %)**Tasks**|(% style="width:523px" %)**Parts where people struggled**
133 -|(% style="width:330px" %)Add a reminder|(% style="width:523px" %)(((
134 -* Don't know where to start
135 -* No immediate audio feedback indicating success
136 -)))
137 -|(% style="width:330px" %)Create a personal profile|(% style="width:523px" %)(((
138 -* It's hard to type with taped fingers
139 -)))
140 -|(% style="width:330px" %)Play memory game|(% style="width:523px" %)(((
141 -* Have no idea how to play the game, sometimes even after listening to the explicit instructions
142 -* There are too many words in the text
143 -* The beta version has no right or wrong prompts, different from the instructions, making people confused
144 -)))
145 -
146 -//Table:  Difficulties that users struggled with when solving tasks//
147 -
148 148  = 4. Discussion =
149 149  
150 -(% class="wikigeneratedid" %)
151 -In light of our research question, we found no notable disparities between the stand-alone application and the robot. The results may be influenced by the experimental setup. While we aimed to emulate a real-world scenario for participants to perform the tasks, their pre-existing digital device proficiency could have played a role. The participants may have had prior experience with using similar applications or robots, which could have affected their performance and perception of the two groups. Nevertheless, given that people with dementia likely have limited knowledge of utilizing mobile devices, we maintain a positive outlook on the potential efficacy of the visual-audio aid provided by the robot to enhance the experience of PwDs utilizing the application.
107 +=== Limitations: ===
152 152  
153 -==== Limitations: ====
109 +We ran into some hiccups while creating the application and performing the experiment:
154 154  
111 +==== Implementation: ====
112 +
155 155  * We could not adapt the robot to the PwD due to time constraints. This means that we did not take into account the severity of the PwD's visual, acoustic and kinesthetic limitations while setting up Pepper.
156 156  * We could not test the full capabilities of the robot due to privacy constraints. Since we fabricated the information about relatives to protect the privacy of participants, we were not able to perform the scenarios in a realistic manner.
157 157  * Since the version of the Google Chrome browser on the Pepper tablet was outdated, we were not able to load our Flutter application onto it and simulate actual scenarios.
158 -* Participants were from a wide variety of different backgrounds and mother tongues, it was therefore not possible to adjust Pepper to the specific culture of the participant.
159 159  
160 -==== Future Improvements: ====
117 +==== Evaluation: ====
161 161  
162 -* We can make our system more realistic/adapted to PwD by incorporating human-like responses, gestures and movements to Pepper.
163 -* We can make our system fully gesture/voice controlled to enable the PwD to (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)use the system without assistance from a caregiver, increasing their autonomy.
164 -* (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)We can incorporate privacy protocols like voice authentication and gaze detection to ensure that all personal information about the PwD, relatives and caregivers are kept safe and confidential.
119 +* The participants who took part in our experiment were not affected by dementia.
120 +* The experiment was not conducted in a real-world setting.
121 +* We were not able to test the improvement in memory recall since it involves sustained interactions with a particular user over time and our experiment involved brief, one-time interactions.
165 165  
166 -= 5. Conclusions =
167 167  
168 -After performing the experiment and running various statistical tests on the results obtained, we have made the following conclusions that hopefully answer some of our research questions:
169 169  
170 -1. We believe that an information support application **DOES IMPROVE** a PwD's well-being, since it can (% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)provide them with access to important information and support, improving their overall quality of life.
171 -1. We believe that a robot assistant **DOES IMPROVE** the experience of a PwD using it. The robot(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %) can provide companionship and assistance, making them feel independent and less isolated.
172 172  
173 -(% style="color:#000000; font-family:Arial; font-size:11pt; font-style:normal; font-variant:normal; font-weight:400; text-decoration:none; white-space:pre-wrap" %)While our experiment had its limitations, we believe that it provides a foundation for future research in developing personalized memory robots for people with dementia. We also believe that our research is applicable to mobile agents which increases the accessibility of the solution.
174 -
175 -
126 += 5. Conclusions =
chart.png
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.DemiTao
Size
... ... @@ -1,1 +1,0 @@
1 -18.8 KB
Content
XWiki.XWikiComments[7]
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.DemiTao
Comment
... ... @@ -1,1 +1,0 @@
1 -Maybe add something related to the results in the discussion
Date
... ... @@ -1,1 +1,0 @@
1 -2023-04-08 01:14:56.749
XWiki.XWikiComments[8]
Author
... ... @@ -1,1 +1,0 @@
1 -XWiki.KarthikPrakash
Comment
... ... @@ -1,6 +1,0 @@
1 -If our assumptions are not statistically significant, explain why.
2 -
3 -Some of the reasons could be:
4 -Participants are young, familiar with tech and not cognitively impaired.
5 -
6 -Rephrase the conclusion and explain why and how we arrived at it
Date
... ... @@ -1,1 +1,0 @@
1 -2023-04-08 10:33:45.41
XWiki.XWikiComments[1]
Author
... ... @@ -1,0 +1,1 @@
1 +Anonymous
Comment
... ... @@ -1,0 +1,40 @@
1 +Quintana et al. (2020) designed questions based on the most important quality attributes to evaluate how the application works for people with mild cognitive impairment.
2 +
3 +(potential questions tailored to our case)
4 +
5 +* (((
6 +How satisfied are you with the robot’s possibility to support you?
7 +)))
8 +* (((
9 +How well does the robot fulfill your expectations?
10 +)))
11 +* (((
12 +Imagine a perfect robot for this task. How far away from it is the robot you are using today?
13 +)))
14 +* (((
15 +I find the robot easily accessible for people with dementia.
16 +)))
17 +* (((
18 +I feel that I can trust the robot and that it is safe to use.
19 +)))
20 +* (((
21 +I find it easy to understand how to operate the robot.
22 +)))
23 +* (((
24 +I am able to understand all the information presented by the robot.
25 +)))
26 +* (((
27 +I feel that the robot gave me better control over my daily situation.
28 +)))
29 +
30 +The alternatives the users could give to all the above were the following: 1 = Strongly disagree; 2 = Disagree.; 3 = Neither agree nor disagree; 4 = Agree; 5 = Strongly agree.
31 +
32 +
33 +The System Usability Scale (SUS) is a dependable instrument for testing usability. It comprises of 10 questions with five response options which range from strongly agree to strongly disagree for responders (Jordan et al., 1996).
34 +
35 +(see the attachment for the 10 questions)
36 +
37 +
38 +1. Quintana M, Anderberg P, Sanmartin Berglund J, Frögren J, Cano N, Cellek S, Zhang J, Garolera M. Feasibility-Usability Study of a Tablet App Adapted Specifically for Persons with Cognitive Impairment—SMART4MD (Support Monitoring and Reminder Technology for Mild Dementia). //International Journal of Environmental Research and Public Health//. 2020; 17(18):6816. [[https:~~/~~/doi.org/10.3390/ijerph17186816>>https://doi.org/10.3390/ijerph17186816]]
39 +
40 +2. Brooke, J. SUS-A quick and dirty usability scale. //Usability Eval. Ind. **1996**, 189//, 4–7.
Date
... ... @@ -1,0 +1,1 @@
1 +2023-03-24 11:18:20.296
XWiki.XWikiComments[5]
Author
... ... @@ -1,0 +1,1 @@
1 +Anonymous
Comment
... ... @@ -1,0 +1,8 @@
1 +interview with the person who interact with the tablet only:
2 +
3 +* The system takes time to learn how to use
4 +* prefer audio instructions
5 +* expect audio feedback after completing each task
6 +* prefer visualization over text
7 +
8 +Suggestion: it's better to weaken the existence of the system and make it easier to learn
Date
... ... @@ -1,0 +1,1 @@
1 +2023-04-01 10:17:27.32
XWiki.XWikiComments[6]
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.DemiTao
Comment
... ... @@ -1,0 +1,16 @@
1 +**Interpretation for user evaluation **
2 +
3 +If a respondent had a minimum total score of 60% (15 out of 25 for the matrix question) or more, he or she was considered to be satisfied with the application.
4 +
5 +**Scoring SUS**
6 +
7 +* For odd items: subtract one from the user response.
8 +* For even-numbered items: subtract the user responses from 5
9 +* This scales all values from 0 to 4 (with four being the most positive response).
10 +* Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40.
11 +
12 +**Interpreting Scores for SUS [[*>>https://measuringu.com/sus/]]**
13 +
14 +Interpreting scoring can be complex. The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100.  Though the scores are 0-100, these are not percentages and should be considered only in terms of their percentile ranking.
15 +
16 +Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average, however, the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking.
Date
... ... @@ -1,0 +1,1 @@
1 +2023-04-07 17:01:06.25