| ... |
... |
@@ -2,8 +2,10 @@ |
| 2 |
2 |
|
| 3 |
3 |
<include a short summary of the claims to be tested, i.e., the effects of the functions in a specfic use case> |
| 4 |
4 |
|
| 5 |
|
-<nothing |
|
5 |
+<nothing on prototype yet, we really need to get that going, but assuming this information> |
| 6 |
6 |
|
|
7 |
+In the section [[a. Prototype>>3\. Evaluation.a\. Prototype.WebHome]] two versions of the robot were presented, one with voice functionality and one without. |
|
8 |
+ |
| 7 |
7 |
The main claims we are looking to test with this testing procedure are related to the functionality and usability of the robot. |
| 8 |
8 |
|
| 9 |
9 |
The participants will be other students taking the course. The participants will be placed in the shoes of a PwD and be tasked with completing several basic actions with the robot while impaired in several known ways to simulate the difficulties of a PwD. |
| ... |
... |
@@ -18,11 +18,11 @@ |
| 18 |
18 |
|
| 19 |
19 |
== 2.1 Participants == |
| 20 |
20 |
|
| 21 |
|
-All students in CS4235 Socio-Cognitive Engineering (2022-2023) in TU Delft are invited to test the robot. In the end, X students are presented. |
|
23 |
+All students in CS4235 Socio-Cognitive Engineering (2022-2023) in TU Delft are invited to test the robot. In the end, 20 students are presented. |
| 22 |
22 |
|
| 23 |
23 |
== 2.2 Experimental design == |
| 24 |
24 |
|
| 25 |
|
-For the experiment, we used a within-subject design. All of the participants interacted with both versions of the robot, with half of the participants interacting with version 1 first and then version 2, and the other half in reverse order. This was done to counter-balance the carryover effects. |
|
27 |
+<Here we can do between or within, doesn't really matter, depends on the number of people we evaluate on honestly because less people means that between subject results will be much more varied and therefore more stupid, not that they will make any sense anyway> |
| 26 |
26 |
|
| 27 |
27 |
== 2.3 Tasks == |
| 28 |
28 |
|
| ... |
... |
@@ -85,8 +85,25 @@ |
| 85 |
85 |
|
| 86 |
86 |
== 2.4 Measures == |
| 87 |
87 |
|
| 88 |
|
-Quantitative measures are used in a user evaluation. If a respondent had a minimum total score of 60% or more, he or she was considered to be satisfied with the application. |
|
90 |
+Two quantitative measures are used in the user evaluation: the first is to test different attributes including accessibility, trustworthiness, and comprehensibility; and the second is System Usability Scale (SUS). |
| 89 |
89 |
|
|
92 |
+**Interpretation for user evaluation ** |
|
93 |
+ |
|
94 |
+If a respondent had a minimum total score of 60% (15 out of 25 for the matrix question) or more, he or she was considered to be satisfied with the application. |
|
95 |
+ |
|
96 |
+**Scoring SUS** |
|
97 |
+ |
|
98 |
+* For odd items: subtract one from the user response. |
|
99 |
+* For even-numbered items: subtract the user responses from 5 |
|
100 |
+* This scales all values from 0 to 4 (with four being the most positive response). |
|
101 |
+* Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40. |
|
102 |
+ |
|
103 |
+**Interpreting Scores for SUS [[*>>https://measuringu.com/sus/]]** |
|
104 |
+ |
|
105 |
+Interpreting scoring can be complex. The participant’s scores for each question are converted to a new number, added together and then multiplied by 2.5 to convert the original scores of 0-40 to 0-100. Though the scores are 0-100, these are not percentages and should be considered only in terms of their percentile ranking. |
|
106 |
+ |
|
107 |
+Based on research, a SUS score above a 68 would be considered above average and anything below 68 is below average, however, the best way to interpret your results involves “normalizing” the scores to produce a percentile ranking. |
|
108 |
+ |
| 90 |
90 |
== 2.5 Procedure == |
| 91 |
91 |
|
| 92 |
92 |
The procedure was conducted as follows: |
| ... |
... |
@@ -111,7 +111,15 @@ |
| 111 |
111 |
|
| 112 |
112 |
= 3. Results = |
| 113 |
113 |
|
|
133 |
+|=(% style="width: 199px;" %)Tasks|=(% style="width: 147px;" %)Succeded by Themselves|=(% style="width: 146px;" %)Succeded with Some Guidance|=(% style="width: 185px;" %)Succeded with Detailed Explicit Instructions|=(% style="width: 175px;" %)Average Time to Complete Task (s) |
|
134 |
+|(% style="width:199px" %)Turning the robot on and off|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %) |
|
135 |
+|(% style="width:199px" %)Add a reminder using "+" button|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %) |
|
136 |
+|(% style="width:199px" %)Add a reminder using voice|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %) |
|
137 |
+|(% style="width:199px" %)Check the reminder|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %) |
|
138 |
+|(% style="width:199px" %)Create a personal profile|(% style="width:147px" %) |(% style="width:146px" %) |(% style="width:185px" %) |(% style="width:175px" %) |
| 114 |
114 |
|
|
140 |
+if possible, note down some Parts Where Users Struggled in each task. |
|
141 |
+ |
| 115 |
115 |
= 4. Discussion = |
| 116 |
116 |
|
| 117 |
117 |
|
