Skip to Content

Changes for page Test

Last modified by Mathieu Jung-Muller on 2022/04/04 13:52
From version Icon 38.2 Icon
edited by XWikiGuest
on 2022/03/20 17:12
Change comment: Added comment
To version Icon 42.1 Icon
edited by Mathieu Jung-Muller
on 2022/03/26 16:18
Change comment: There is no comment for this version

Summary

Details

Icon Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.XWikiGuest
1 +XWiki.Mathieu
Content
... ... @@ -94,21 +94,57 @@
94 94  
95 95  In our situation, we recruit 20 students in our class to simulate the research. Since they are not real PwD, we ask them to act as if they were in home settings and we observe their behavior and expression. These data will also be reviewed after the evaluation to obtain the data and feedback.
96 96  
97 -In order to collect qualitative data, we have prepared two questionnaires to give to the participants: the affect assessment questionnaire and the system assessment questionnaire.
97 +==== Questionnaires ====
98 98  
99 -* affect assessment
100 -the affect assessment questionnaire has the purpose of tracking changes in the participant's affect state before and after interacting with Pepper. The questionnaire consists of a list of 10 statements each representing one emotion and participants will be asked to give a score of 1 through 5 to each statement depending on the intensity of the emotion as experienced during the questionnaire. The affect questionnaire will be given twice, once before the interaction with Pepper and once after.
99 +In order to collect qualitative data, we prepared two questionnaires to give to the participants: the affect assessment questionnaire and the system assessment questionnaire. All questionnaire questions are expressed in the form of statements and the participant can express one out of seven levels of agreement/disagreement from strongly disagree to strongly agree with the only exception being the additional remarks field at the end of the system assessment questionnaire.
101 101  
102 -* system assessment
103 -for assessing the quality of Pepper's interaction we used the Subjective Assessment of Speech System Interfaces (SASSI) questionnaire. The SASSI questionnaire is wildly used in research to evaluate speech and conversational systems. The questionnaire contains 36 questions divided into six main factors: System Response Accuracy, Likeability, Cognitive Demand, Annoyance, Habitability, and Speed. The questions are short and unambiguous and the length of the questionnaires is not too long which could cause the participants to give fast and unreliable answers but it is also long enough so that a good amount of information is collected.
101 +**Affect assessment**
104 104  
103 +The affect assessment questionnaire asks the participant to describe their mood and feelings experienced during a task. The questionnaire is given two times: once before the experiments and once right after. The purpose of this questionnaire is to compare the feelings experienced by the participant before the interaction with Pepper and after to study how Pepper influences the participant's feelings. The questionnaire has six questions each designed to measure the level of a certain feeling experienced at the moment of filling the questionnaire. The six feelings we take into consideration are evaluated from the following statements:
104 +
105 +1. I feel sad
106 +1. I feel content
107 +1. I feel calm
108 +1. I feel tired
109 +1. I feel nervous
110 +1. I feel caring
111 +
112 +**System assessment**
113 +
114 +The system assessment questionnaire is given after the interaction with Pepper and its purpose is to assess the participant's experience with Pepper in more detail. The questionnaire has fourteen questions which are designed to answer research questions as described below:
115 +
116 +1. I like gardening
117 +1. I think Pepper made the task easier for me.
118 +1. I would have known how to do the whole task without Pepper.
119 +1. Pepper was easy to understand.
120 +1. I enjoyed the task more than if I had had to do it alone.
121 +1. I am pleased that Pepper reminded me to do the activity.
122 +1. I feel like completing the task was a good accomplishment.
123 +1. I feel like I accomplished it myself.
124 +1. I felt in control of what I had to do.
125 +1. I felt annoyed by Pepper.
126 +1. I felt frustrated by the task.
127 +1. I felt pressured by Pepper.
128 +1. Pepper cared about helping me.
129 +1. I would trust Pepper with more important activities.
130 +
131 +Research questions:
132 +
133 +1. To what extent did Pepper improve the task's experience? [Questions: 2,3,4,5,10,11,12]
134 +1. To what extent did Pepper improve the participant's autonomy and perception of control? [Questions: 6, 8, 9]
135 +1. To what extent did Pepper improve the participant's sense of accomplishment? [Questions: 7]
136 +1. Did the participant perceive Pepper as a social agent? [Questions: 13, 14]
137 +1. How was Pepper's ability to guide the participant through the task? [Questions: 4]
138 +
139 +Question 1 is used to take into consideration the expected baseline enjoyment of the task for each participant.
140 +
105 105  == Participants ==
106 106  
107 -For this study, we simulate the real research by including 20 students in our class. To avoid too much gender imbalance, there will be at least five men and five women. All participants will be asked to pretend to be a demented person and to make the simulation as real as possible.
143 +For this study, we simulate the real research by including 24 students from TU Delft. One third of them were female and two thirds male, so gender balance was decent. The students were not asked to roleplay someone with dementia, but simply to follow the experiment and do the activity.
108 108  
109 109  == Experimental design ==
110 110  
111 -Since our evaluating process is relatively short, we use within-subject, which means each participant goes through all conditions. In this way, our experiments tend to have more statistical power and less variability. Furthermore, every PwD may have different issues in their day-to-day life, while also not having the same living conditions (alone, living with husband, family, etc), thus would require a different treatment. But in our setting, we use the same home setting for every participant.
147 +Since our evaluating process is relatively short, we use within-subject, which means each participant goes through all conditions. In this way, our experiment tends to have more statistical power and less variability. Furthermore, every PwD may have different issues in their day-to-day life, while also not having the same living conditions (alone, living with husband, family, etc), thus would require a different treatment. But in our setting, we use the same setting for every participant.
112 112  
113 113  == Tasks ==
114 114