4. Evaluation Methods

The following steps will be used to design and evaluate the prototype proposed against the corresponding control condition:

1. Confirm the prototype: The prototype for the scenario to be tested, and the control situation will first be setup, and preliminary testing will be done by the team members. This includes the robots with and without interactive storytelling which should be confirmed and working.

2. Develop Questions: 

3. Design Methods

4. Implement and adapt:

5. Make decisions:

Research Question

"Is interactive storytelling more engaging and beneficial than storytelling in the third person for persons suffering from dementia?"

Thus, our control situation is the scenario of a robot narrating a story without any involvement of the patient, and the scenario we want to evaluate is the one where the robot narrates the same story while trying to engage and take inputs from the patient. With this, we aim to find whether it is beneficial and engaging for patients with dementia. 

The Within-Subject Design

As part of the experiment design, we chose the within subject design over between subject. This means that each participant will interact with the robot twice. This was done due to the limited number of participants, and to avoid any biases of participant preferences.

Summative Evaluation

We will evaluate the prototype's effectiveness at the end of the experiment, i.e whether interactive storytelling was beneficial as compared to non interactive storytelling. Since we are comparing two robots, we follow summative evaluation. Using a questionnaire, we will try to assess the usefulness and effectiveness of the robot. Due to limited time of the course, this will be the last evaluation. However, in the absence of time constraints, we would need to do a formative evaluation to get feedback for the next versions of the robot.

Questionnaire

-modified godspeed questionnaire for robot

-statistical test (p value) for evaluation

Prototype

We present a low fidelity prototype of the robot, which means a simple demonstration of the initial stages of the robot, meant for formative feedback. We wizard-of-oz the approach, and for now just present one story (in interactive and non interactive modes) for purposes of the experiment. The final robot is expected to have various templates of stories.

For prototyping, we will use incremental prototyping, which means adding features one by one and testing for each. We start with the most basic feature, complete a cycle of testing, and then add on new features to create new versions of the prototype. For the robot, we will first build the non interactive storytelling robot, then add music to it, and then add gestures. With each stage, we test the working of it, and if working as expected, we will move on to adding the next feature.

Since we don't have many participants, should we skip the statistical test? Can we just report average values of responses for both scenarios?

Questionnaire should be a formal one, or should we ask 4-5 questions through Pepper? Or both?