In evaluation, we use the control group methods to evaluate our robot. For one group, participants interact with a dummy robot and another group interacts with the intelligent robot. The only difference between these two groups is the independent variable - dummy or intelligent robot, which makes our result more reasonable. Due to the instability of NAO's speech recognition, parts of the evaluation have to be hardcoded. Therefore our evaluation ended up as a "Wizard of Oz".
Evaluation Steps
First, all participants would sign a consent form that informed them of the usage of the collected data and our goal of evaluations.
Step 1
In our prototype, users can personalize the link between music and activities based on their existing intrinsic knowledge. But due to the limited time and requiring a comparable result between groups, in evaluation, we forced 6 pieces of music and activities. Participants listened to the music and were asked the remember the associated activities.
Step 2
In the second step, participants in two groups would play with the robot for 3 minutes.
- Group A with the intelligent robot: The Intelligent robot would inform the participants of the correct answer when they gave a wrong one. This is simulating our use case 5, assuming the quiz can help PwDs enhance the memory between music and activities.
- Group B with the dummy robot: The dummy robot can not correct the answer, which is trying to simulate the daily scenario that the robot only reminded the daily task but they had no idea whether PwD did the correct task.
Step 3
In the end, the participants would take a quiz to see how much they remembered. They are also asked to fill in a questionnaire including the feeling of the robot and possible feedback.