Measuring Instruments

In this section, we will outline the measuring instruments used to evaluate our prototype and a motivation for using each of them. During the lecture on evaluation, we were presented with the DECIDE (Preece et al., 2015, Chapter 13) framework and the IMPACT (Benyon et al., 2005) framework. Firstly, we will briefly explain how we incorporated the DECIDE framework into our own project. Secondly, we will mostly elaborate on the intentions from the IMPACT framework as the other letters of the acronym are delineated across this XWiki. 

DECIDE

Determine the goals

Our aim is to develop a robot that can assist PwD living in a care home on their daily walks. The idea is that PwD will perceive a higher level of autonomy when they walk with a robot dog, assuming it is tailored to their needs, compared to walking with a caretaker. PwDs do not want to put the burden on their loved ones and caretakers to assist them with conducting their daily activities in general.

Explore the questions

The research questions are defined in our Test section and include a motivation. To consider different people's attitude towards walking with a robot dog, we have constructed multiple personas and set up various value stories for the direct stakeholders to help guide us in our development.

Choose evaluation approach and methods

The evaluation methods are described in our Test section. Briefly, we aim to obtain quantative data to answer our research questions. In addition, during the experiment, we will observe the participants casually and note any remarkable behavior. For the PwD who is prone to falling, we will test its effectiveness by means of dogfooding as this would be difficult if not unethical to evaluate empirically.

Identify practical issues

We already foresee due to the scope of the course and the current COVID-19 situation that it will be impossible to evaluate our prototype with our intended target group. We account for this problem in our experiment by instructing participants to act like persons with early-to-mid stage dementia. Naturally, the confounding factor that will influence the outcome is that participants, from our class, may interpret the role of the persona they enact very differently and will answer questions depending on their interpretation.  

Decide on ethical issues

Any ethical issues that may arise are described in Test section. To sum it up, we plan to let participants fill out a consent form and inform them that they are allowed to withdraw from the experiment at any time no questions asked. Moreover, we aim to debrief the participants properly, but also allow for a debriefing in textual form when the participant fills out the final questionnaire.

Evaluate, analyze, interpret and present the data

We aim to conduct an experiment that is reliable and valid. However, as will be mentioned in limitations under Future Work/Conclusions, we were able to set up a reliable experiment (albeit a bit long per participant), but not necessarily valid as the questions of the survey did not accurately capture the autonomy of the participant at each stage. We acknowledge that the experiment was biased towards people who have more affinity with technology and understand the affordances of robots in general. In addition, there is a bias for participants who have a knack for acting or at least playing the role of someone else that is not themselves. After completion of the experiment, we are prudent to draw any conclusions about generalization and increasing the scope of the findings due to the numerous limitations and cofounding factors aforementioned. 

Preferably, one would conduct this experiment as a field study in a PwD's natural environment (i.e. the care home) with multiple sessions to account for sudden mood changes in PwD. So, the fact that the participants were students and the experiment was held in a meeting room did influence the findings a little. We predict that participants would experience higher degrees of freedom in an actual care home would hallways and rooms to explore compared to a meeting room where the participant was confined between the four walls of the meeting room. Therefore, we acknowledge that the experiment may not be ecologically valid.

IMPACT

Intention

Our intention is to study the overall effects of our prototype. As such, we opted for a summative evaluation where we compare between the experience of walking with a caretaker and walking with a robot dog as independent variables.

Metrics

The detailed metrics are outlined in our Test section. We intend to use mostly subjective measures, since the main aim is to measure an increase in perceived autonomy. Most of the questions on our questionnaire are ordinal in nature. For instance, there is a clear order from no autonomy to full autonomy, but there is no relative degree of difference between these levels. 

People, Activities, Context and Technologies (PACT)

Our PACT analysis is mentioned in our Environments section.