Skip to Content

Changes for page b. Test

Last modified by Demi Tao on 2023/04/10 10:13
From version 8.1
edited by Demi Tao
on 2023/04/07 17:00
Change comment: There is no comment for this version
To version 5.1
edited by Arber Demi
on 2023/03/27 13:58
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.DemiTao
1 +XWiki.ArberDemi
Content
... ... @@ -1,29 +1,31 @@
1 1  = 1. Introduction =
2 2  
3 -In section [[a. Prototype>>3\. Evaluation.a\. Prototype.WebHome]] two versions of the robot were presented, one with voice functionality and one without.
3 +<include a short summary of the claims to be tested, i.e., the effects of the functions in a specfic use case>
4 4  
5 +<nothing on prototype yet, we really need to get that going, but assuming this information>
6 +
7 +In the section [[a. Prototype>>3\. Evaluation.a\. Prototype.WebHome]] two versions of the robot were presented, one with voice functionality and one without.
8 +
5 5  The main claims we are looking to test with this testing procedure are related to the functionality and usability of the robot.
6 6  
7 -The participants will be other students taking this course. The participants will be placed in the shoes of a PwD and be tasked with completing several basic actions with the robot while impaired in several known ways to simulate the difficulties of a PwD.
11 +The participants will be other students taking the course. The participants will be placed in the shoes of a PwD and be tasked with completing several basic actions with the robot while impaired in several known ways to simulate the difficulties of a PwD.
8 8  
9 -After the experiment, the participants will fill out a survey and be asked some more open-ended questions with the purpose of understanding how the interaction with the robot went, and whether they have anything that they find concerning regarding the possible use of the system and its functions in a real-life setting.
13 +After the experiment, the participants will fill out a survey and be asked some more open ended questions with the purpose of understanding how the interaction with the robot went, and whether they have anything that they find concerning regarding the possible use of the system and its functions in a real life setting.
10 10  
11 -On top of this, a short questionnaire will be sent to several care homes throughout the Netherlands in hopes to get a general idea of whether the caretakers at the facilities think that the system would be a good fit for the proposed use case.
15 +On top of this, a short questionnaire will be sent to several care homes throughout the Netherlands in hopes to get a general idea whether the caretakers at the facilities think that the system would be a good fit for the proposed use case.
12 12  
13 13  = 2. Method =
14 14  
15 -The prototypes are evaluated in a simulated manner and conducted in-person experiments. Participants will be given a [[persona>>doc:Main.sdf.Persona Scenarios.WebHome]] to play act.
19 +The prototypes are evaluated in a simulated manner, with participants pretending to be PwDs and conducting in-person experiments.
16 16  
17 17  == 2.1 Participants ==
18 18  
19 -All students in CS4235 Socio-Cognitive Engineering (2022-2023) in TU Delft are invited to test the robot. In the end, 14 students are presented.
23 +All students in CS4235 Socio-Cognitive Engineering (2022-2023) in TU Delft are invited to test the robot. In the end, 20 students are presented.
20 20  
21 21  == 2.2 Experimental design ==
22 22  
23 -Prior to the experiment, participants were asked to simulate the experience of a person with dementia. Participants who wore glasses were asked to remove them, while those who did not were provided with short-sighted or far-sighted glasses to replicate the blurred vision and degraded perception that is common in PwDs. Additionally, their index and middle fingers were taped together to simulate the difficulty in controlling movements that many PwDs experience.
27 +<Here we can do between or within, doesn't really matter, depends on the number of people we evaluate on honestly because less people means that between subject results will be much more varied and therefore more stupid, not that they will make any sense anyway>
24 24  
25 -For this experiment, we used a between-subject design. The control group interacted solely with the stand-alone application, which represented the robot without voice functionality, and received guidance through a single task paper. In contrast, the experimental group engaged with the fully-functional robot, which provided audio instructions, guidance, and encouragement.
26 -
27 27  == 2.3 Tasks ==
28 28  
29 29  In the user test, the following tasks were asked of the participants:
... ... @@ -49,12 +49,14 @@
49 49  ==== Memory games ====
50 50  
51 51  * (((
52 -Go to the Games section and play the game.
54 +Go to the Games section and check what is included there.
55 +
56 +<this might not be part of the experiment as it might not get implemented in time)
53 53  )))
54 54  
55 55  ==== Medicine reminders ====
56 56  
57 -(for professional caregivers)
61 +(for professional caregivers, write it just in case)
58 58  
59 59  * (((
60 60  In the section “My Health”, add a medicine reminder to take the medicine **Donepezil**, 1 time per day at 9 PM before going to bed.
... ... @@ -66,14 +66,42 @@
66 66  Delete medicines that have been added.
67 67  )))
68 68  
73 +==== About dementia ====
74 +
75 +* (((
76 +Go to the About dementia section and check the information provided.
77 +)))
78 +* (((
79 +Click on the different chapters and have a look at them.
80 +)))
81 +
82 +==== General tasks ====
83 +
84 +* (((
85 +Turn on the robot.
86 +)))
87 +
69 69  == 2.4 Measures ==
70 70  
71 -Two quantitative measures were employed in the user evaluation. The first measure aimed to test attributes including accessibility, trustworthiness, and comprehensibility. The second measure used was the System Usability Scale (SUS), which is a widely-used scale for evaluating the usability of software.
90 +Quantitative measures are used in a user evaluation.
72 72  
73 -In addition to the quantitative measures, a structured interview with open-ended questions was conducted. The aim of this interview was to gain a deeper understanding of participants' experiences with the robot, including any concerns they may have regarding the potential use of the system and its functions in a real-life setting.
92 +**Interpretation for user evaluation (?**
74 74  
75 -By using both quantitative and qualitative measures, the user evaluation can provide a more comprehensive understanding of participants' experiences and perceptions of the system. This approach allows for a more nuanced analysis of the data and can provide valuable insights into the strengths and weaknesses of the system, as well as areas for improvement.
94 +If a respondent had a minimum total score of 60% or more, he or she was considered to be satisfied with the application.
76 76  
96 +**Scoring SUS**
97 +
98 +* For odd items: subtract one from the user response.
99 +* For even-numbered items: subtract the user responses from 5
100 +* This scales all values from 0 to 4 (with four being the most positive response).
101 +* Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40.
102 +
103 +**Interpreting Scores for SUS [[*>>https://measuringu.com/sus/]]**
104 +
105 +Interpreting scoring can be complex. The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100.  Though the scores are 0-100, these are not percentages and should be considered only in terms of their percentile ranking.
106 +
107 +Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average, however, the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking.
108 +
77 77  == 2.5 Procedure ==
78 78  
79 79  The procedure was conducted as follows:
... ... @@ -80,27 +80,25 @@
80 80  
81 81  1. Welcome participants and give an introduction.
82 82  1. Get them to sign a consent form.
83 -1. Prepare them to pretend to be a person with dementia.
115 +1. Prepare them to pretend to be a person with dementia. *
84 84  1. Have interaction with the robot and complete the tasks.
85 85  1. Complete a questionnaire.
86 -1. Have a short interview with randomly selected participants.
118 +1. Have a short interview with selected participants. (if possible, 2 participants)
87 87  
120 +//* Several fingers taped together (to simulate PwD's inability to control movements flexibly);//
121 +
122 +// Wearing very dirty glasses (simulates blurred vision and degraded perception);//
123 +
124 +// Wearing headphones that broadcast murmurs (simulating hearing degradation and the noisy environment).//
125 +
88 88  == 2.6 Material ==
89 89  
90 90  1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.
91 -1. Pepper robot.
129 +1. Pepper robot. <not sure how to elaborate on this>
92 92  
93 93  = 3. Results =
94 94  
95 -|=(% style="width: 199px;" %)Tasks|=(% style="width: 147px;" %)Succeded by Themselves|=(% style="width: 146px;" %)Succeded with Some Guidance|=(% style="width: 185px;" %)Succeded with Detailed Explicit Instructions|=(% style="width: 175px;" %)Average Time to Complete Task (s)
96 -|(% style="width:199px" %)Turning the robot on and off|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %)
97 -|(% style="width:199px" %)Add a reminder using "+" button|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %)
98 -|(% style="width:199px" %)Add a reminder using voice|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %)
99 -|(% style="width:199px" %)Check the reminder|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %)
100 -|(% style="width:199px" %)Create a personal profile|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %)
101 101  
102 -if possible, note down some Parts Where Users Struggled in each task.
103 -
104 104  = 4. Discussion =
105 105  
106 106  
XWiki.XWikiComments[5]
Author
... ... @@ -1,1 +1,0 @@
1 -Anonymous
Comment
... ... @@ -1,8 +1,0 @@
1 -interview with the person who interact with the tablet only:
2 -
3 -* The system takes time to learn how to use
4 -* prefer audio instructions
5 -* expect audio feedback after completing each task
6 -* prefer visualization over text
7 -
8 -Suggestion: it's better to weaken the existence of the system and make it easier to learn
Date
... ... @@ -1,1 +1,0 @@
1 -2023-04-01 10:17:27.32