Skip to Content

3. Evaluation Methods

Last modified by William OGrady on 2024/04/08 22:22

Overview

To ground the design rationale in practice, the prototype of the NAO for the PwD will be evaluated in a formative evaluation with the PwD and their caregivers. The evaluation will investigate the process of how interaction with the NAO can alleviate potential symptoms caused by early-stage dementia in PwDs. Three methods of evaluation will be applied to have reliable and accurate results, as Bethel. et al (2020)[1] suggests. The study will focus on the prototype's potential effects, which are based on the desired value of autonomy as a part of self-direction. 

Study Design Variables

The study will hypothesize on the following variables with regards to the system:

  1. Relatedness. Does the design positively affect the PwD's affective state?
    1. Affect. How do participants feel about using the NAO in this state?
    2. Attitude towards Technology. What do people think about using technology? Are they biased towards the NAO before the study?
  2. Competence. Is the design competent; is the design capable enough for the PwD to rely on it?
    1. Memory self-efficacy (pre-study) How good are participants at remembering information?
    2. Memory recall (post-study) Can the participant accurately retrieve information through the NAO?

For a sample size as small as 20 participants, it is most adequate to apply a within-subjects design (they require fewer participants) [1]. That means there is an approach where every PwD is experiencing all of the conditions examined. A within-subjects design might be prone to confounds such as pre-existing notions in the environment. That is why the attitude towards robots and the pre-study sense of affect and autonomy should be examined and evaluated as such.

Socio-Cognitive Engineering - Frame 1.jpg

Surveys

Affect will be measured by the Self-Assessment Manikin [2] that takes less than a minute. It will be used for both pre-experiment and post-experiment evaluation. For Attitude towards robots, we build upon the works of [reference] and create a set of two 1-minute questions. A well-suited tool for Memory self-efficacy is the mini-mental state examination [3]. We adopt it to fit to the ecological validity and domain of interest in our study.

References

[1] Bethel, C.L., Henkel, Z., Baugus, K. (2020). Conducting Studies in Human-Robot Interaction. In: Jost, C., et al. Human-Robot Interaction. Springer Series on Bio- and Neurosystems, vol 12. Springer, Cham. https://doi.org/10.1007/978-3-030-42307-0_4

[2] Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry, 25(1), 49-59.

[3] Kurlowicz, L., & Wallace, M. (1999). The mini-mental state examination (MMSE). Journal of gerontological nursing, 25(5), 8-9.