Skip to Content

Wiki source code of b. Test

Version 6.1 by Arber Demi on 2023/03/27 13:58
Hide last authors
Ruud de Jong 1.1 1 = 1. Introduction =
2
3 <include a short summary of the claims to be tested, i.e., the effects of the functions in a specfic use case>
4
Arber Demi 3.1 5 <nothing on prototype yet, we really need to get that going, but assuming this information>
Ruud de Jong 1.1 6
Arber Demi 3.1 7 In the section [[a. Prototype>>3\. Evaluation.a\. Prototype.WebHome]] two versions of the robot were presented, one with voice functionality and one without.
8
Arber Demi 2.2 9 The main claims we are looking to test with this testing procedure are related to the functionality and usability of the robot.
10
11 The participants will be other students taking the course. The participants will be placed in the shoes of a PwD and be tasked with completing several basic actions with the robot while impaired in several known ways to simulate the difficulties of a PwD.
12
13 After the experiment, the participants will fill out a survey and be asked some more open ended questions with the purpose of understanding how the interaction with the robot went, and whether they have anything that they find concerning regarding the possible use of the system and its functions in a real life setting.
14
15 On top of this, a short questionnaire will be sent to several care homes throughout the Netherlands in hopes to get a general idea whether the caretakers at the facilities think that the system would be a good fit for the proposed use case.
16
Ruud de Jong 1.1 17 = 2. Method =
18
Demi Tao 2.1 19 The prototypes are evaluated in a simulated manner, with participants pretending to be PwDs and conducting in-person experiments.
Ruud de Jong 1.1 20
21 == 2.1 Participants ==
22
Arber Demi 5.1 23 All students in CS4235 Socio-Cognitive Engineering (2022-2023) in TU Delft are invited to test the robot. In the end, 20 students are presented.
Ruud de Jong 1.1 24
25 == 2.2 Experimental design ==
26
Arber Demi 3.1 27 <Here we can do between or within, doesn't really matter, depends on the number of people we evaluate on honestly because less people means that between subject results will be much more varied and therefore more stupid, not that they will make any sense anyway>
Ruud de Jong 1.1 28
29 == 2.3 Tasks ==
30
Demi Tao 2.1 31 In the user test, the following tasks were asked of the participants:
Ruud de Jong 1.1 32
Demi Tao 2.1 33 ==== Reminders for activities ====
34
35 * (((
36 Add a reminder that a relative will pay a visit tomorrow with the format as "sb. will come at 10 am on Friday". Set the reminder to remind you 10 min before that.
37 )))
38 * (((
39 Add a reminder USING A VOICE COMMAND that today at 2 pm will have a general health checkup.
40 )))
41 * (((
42 Check the reminders you have added for today and tomorrow.
43 )))
44
45 ==== Personal profile ====
46
47 * (((
48 Add relatives as a contact in the "profile" section.
49 )))
50
51 ==== Memory games ====
52
53 * (((
54 Go to the Games section and check what is included there.
Arber Demi 2.2 55
56 <this might not be part of the experiment as it might not get implemented in time)
Demi Tao 2.1 57 )))
58
59 ==== Medicine reminders ====
60
61 (for professional caregivers, write it just in case)
62
63 * (((
64 In the section “My Health”, add a medicine reminder to take the medicine **Donepezil**, 1 time per day at 9 PM before going to bed.
65 )))
66 * (((
67 Check medicines that have been added.
68 )))
69 * (((
70 Delete medicines that have been added.
71 )))
72
73 ==== About dementia ====
74
75 * (((
76 Go to the About dementia section and check the information provided.
77 )))
78 * (((
79 Click on the different chapters and have a look at them.
80 )))
81
82 ==== General tasks ====
83
84 * (((
85 Turn on the robot.
86 )))
87
Ruud de Jong 1.1 88 == 2.4 Measures ==
89
Demi Tao 4.1 90 Quantitative measures are used in a user evaluation.
Ruud de Jong 1.1 91
Demi Tao 4.1 92 **Interpretation for user evaluation (?**
93
94 If a respondent had a minimum total score of 60% or more, he or she was considered to be satisfied with the application.
95
96 **Scoring SUS**
97
98 * For odd items: subtract one from the user response.
99 * For even-numbered items: subtract the user responses from 5
100 * This scales all values from 0 to 4 (with four being the most positive response).
101 * Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40.
102
103 **Interpreting Scores for SUS [[*>>https://measuringu.com/sus/]]**
104
105 Interpreting scoring can be complex. The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100.  Though the scores are 0-100, these are not percentages and should be considered only in terms of their percentile ranking.
106
107 Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average, however, the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking.
108
Ruud de Jong 1.1 109 == 2.5 Procedure ==
110
Demi Tao 2.1 111 The procedure was conducted as follows:
112
113 1. Welcome participants and give an introduction.
114 1. Get them to sign a consent form.
115 1. Prepare them to pretend to be a person with dementia. *
116 1. Have interaction with the robot and complete the tasks.
117 1. Complete a questionnaire.
118 1. Have a short interview with selected participants. (if possible, 2 participants)
119
120 //* Several fingers taped together (to simulate PwD's inability to control movements flexibly);//
121
122 // Wearing very dirty glasses (simulates blurred vision and degraded perception);//
123
124 // Wearing headphones that broadcast murmurs (simulating hearing degradation and the noisy environment).//
125
Ruud de Jong 1.1 126 == 2.6 Material ==
127
Arber Demi 2.2 128 1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.
129 1. Pepper robot. <not sure how to elaborate on this>
Ruud de Jong 1.1 130
131 = 3. Results =
132
133
134 = 4. Discussion =
135
136
137 = 5. Conclusions =