Test

Ideal evaluation (actual research)

Problem statement and research questions

This project uses a Social Cognitive Engineering (SCE) approach to guide the design and research process. The SCE method provides a systematic approach to our study of robots for PwDs. The main goal of our application is to improve the well-being of the person with dementia (PwD) and of those living with them.

For the prototype that we have designed now, these are some research questions that we want to address.

1. Are the different stakeholders able to use our prototype smoothly?
2. Does the prototype allow the PwD a greater autonomy in their day-to-day life?
3. Does the prototype improve the emotional state of the PwD and their relatives?

Method

As people with dementia have very specific situation (and because our prototype is built to deal with that aspect of customizability), we do not go for an identical experiment for all of them. Instead, the global setup is very similar, although Pepper is customized for the needs of every patient. Our evaluation can therefore be assimilated (with some reserve) to a within-subject evaluation. We also do pre-test and post-test.
Our main evaluation method is summative evaluation: we are trying to determine whether the robot has an impact and improves the frequency of "yes" in our yes-no driving questions.

Participants

The study will be conducted on approximately twenty people in early stages of dementia. To avoid too much gender imbalance, there will be at least five men and five women. For the same reason, we hope to find at least five people around 50 or younger, although we expect most participants to be over 70 years old.
The participants will be selected based on a pool of PwD who live at home and need regular visits from a HPC. Only the volunteers will be kept for the experiment. The participants must have no experience of a Pepper robot helping with dementia.
As our experiment involves the PwD in their home environment, relatives and healthcare professionals (HPC) will also be involved in the process: although not being the targets of the experiment, they will contribute to obtaining the measurements and results.

Experimental design

Since dementia is unique to every person, it is very hard to conduct a global experiment with the same conditions for all participants. Every of them may have different issues in their day-to-day life, while also not having the same living conditions (alone, living with husband, family, etc), thus would require a different treatment. Furthermore, we want to record whether our prototype leads to an improvement in life quality. This can not be done through a short experiment, because the reliability of such an experiment would be very low.
This means we need to record the patient at home and measure over an extended period of time if their well-being and autonomy globally improved.

Tasks

The PwD will live their daily life, without Pepper in the beginning, then accompanied by Pepper, as if they were not under any experiment. The fact that they actually are will obviously affect their behaviour. Yet, we hope that not being recorded and being under a non-invasive experiment will help them not to stress out and may make them live their life as normally as possible.
The tasks performed during the evaluation by Pepper, by the PwD, and by Pepper and the PwD together, will be decided in consultation with the HPC (and potentially the relatives) based on the needs of each patient.

Measures

We are planning to make behavioural and emotional measurements.
Behavioral measurements are the actions that the PwD is going to perform during the week, so it can be considered as subjective quantitative data. This will involve the relatives, the HPC and the PwD themselves to quantify whether the use of Pepper did actually result in an increase in autonomy for the PwD.
Emotional measurements are more related to state of mind, change of expression and mood, so it can be considered as qualitative data. This can be measured by frequent talks with the PwD, either by the relatives or the HPC.
Measures will be done by oral discussions with the PwD, HPC and relatives.

Procedure

The whole experiment process is 4 weeks long, although only weeks 1 and 4 are technically part of the experiment itself.
During week 1, the PwD, the HPC and the relatives will be asked to pay increased attention. Behavioural and emotional data will be collected. This is a regular week for the PwD, i.e., in the usual situation, except that there is more attention dedicated to them.
During weeks 2 and 3, the PwD, the HPC and the relatives will be introduced Pepper, with the goal of getting used to it.
During week 4, which is the actual week of the experiment, attention will be spent trying to mirror week 1 as closely as possible. Behavioural and emotional data will be collected again.

Material

1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.
2. Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behaviour for every participant. However, the input data will be entered by the HPC (and potentially the relatives).

Results

Since each PwD has its own state of dementia and personal issues, it is very difficult to get uniform results, especially since they are collected orally.
Getting very nice, fully robust and reliable results, is merely a hope and a dream.
However, we can try to consider the main trends that we are interested in.
Thus, the results wil be mainly focused on:
- How much autonomy did the PwD gain?
→ what did the HPC, relatives and PwD report
→ how many tasks did they perform that they didn't do previously
→ did the relatives feel they had more time for themselves
- Did their emotional state improve?
→ feelings from the PwD themselves
→ reports from relatives and HPC
These results will most likely never be yes-no results, but more like clues or hints that show whether some things worked on not, which will be the point of our discussion.
NB: This part explains what we expect as kind of results, it will be replaced by actual results after we perform an experiment with the class. There may also be interesting points we did not think about.

Discussion

• Reliability: Yes. One could replicate the same experiment with other patients.
• Validity: TBD.
• Biases: TBD.
• Scope: No. It would be very difficult to generalize the results, since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar effort to customize Pepper for more patients should produce similar effects.
• Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.

Conclusions

Feasible evaluation (students)

Problem statement and research questions

This project uses a Social Cognitive Engineering (SCE) approach to guide the design and research process. The SCE method provides a systematic approach to our study of robots for PwDs. The main goal of our application is to improve the well-being of the person with dementia (PwD) and of those living with them.

For the prototype that we have designed now, these are some research questions that we want to address.

1. Are the different stakeholders able to use our prototype smoothly?
2. Does the prototype allow the PwD a greater autonomy in their day-to-day life?
3. Does the prototype improve the emotional state of the PwD and their relatives?

Method

As people with dementia have very specific situation (and because our prototype is built to deal with that aspect of customizability), we do not go for an identical experiment for all of them. Instead, the global setup is very similar, although Pepper is customized for the needs of every patient. Our evaluation can therefore be assimilated (with some reserve) to a within-subject evaluation. We also do pre-test and post-test.
Our main evaluation method is summative evaluation: we are trying to determine whether the robot has an impact and improves the frequency of "yes" in our yes-no driving questions.

Participants

The study will be conducted on approximately twenty people in early stages of dementia. To avoid too much gender imbalance, there will be at least five men and five women. For the same reason, we hope to find at least five people around 50 or younger, although we expect most participants to be over 70 years old.
The participants will be selected based on a pool of PwD who live at home and need regular visits from a HPC. Only the volunteers will be kept for the experiment. The participants must have no experience of a Pepper robot helping with dementia.
As our experiment involves the PwD in their home environment, relatives and healthcare professionals (HPC) will also be involved in the process: although not being the targets of the experiment, they will contribute to obtaining the measurements and results.

Experimental design

Since dementia is unique to every person, it is very hard to conduct a global experiment with the same conditions for all participants. Every of them may have different issues in their day-to-day life, while also not having the same living conditions (alone, living with husband, family, etc), thus would require a different treatment. Furthermore, we want to record whether our prototype leads to an improvement in life quality. This can not be done through a short experiment, because the reliability of such an experiment would be very low.
This means we need to record the patient at home and measure over an extended period of time if their well-being and autonomy globally improved.

Tasks

The PwD will live their daily life, without Pepper in the beginning, then accompanied by Pepper, as if they were not under any experiment. The fact that they actually are will obviously affect their behaviour. Yet, we hope that not being recorded and being under a non-invasive experiment will help them not to stress out and may make them live their life as normally as possible.
The tasks performed during the evaluation by Pepper, by the PwD, and by Pepper and the PwD together, will be decided in consultation with the HPC (and potentially the relatives) based on the needs of each patient.

Measures

We are planning to make behavioural and emotional measurements.
Behavioral measurements are the actions that the PwD is going to perform during the week, so it can be considered as subjective quantitative data. This will involve the relatives, the HPC and the PwD themselves to quantify whether the use of Pepper did actually result in an increase in autonomy for the PwD.
Emotional measurements are more related to state of mind, change of expression and mood, so it can be considered as qualitative data. This can be measured by frequent talks with the PwD, either by the relatives or the HPC.
Measures will be done by oral discussions with the PwD, HPC and relatives.

Procedure

The whole experiment process is 4 weeks long, although only weeks 1 and 4 are technically part of the experiment itself.
During week 1, the PwD, the HPC and the relatives will be asked to pay increased attention. Behavioural and emotional data will be collected. This is a regular week for the PwD, i.e., in the usual situation, except that there is more attention dedicated to them.
During weeks 2 and 3, the PwD, the HPC and the relatives will be introduced Pepper, with the goal of getting used to it.
During week 4, which is the actual week of the experiment, attention will be spent trying to mirror week 1 as closely as possible. Behavioural and emotional data will be collected again.

Material

1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.
2. Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behaviour for every participant. However, the input data will be entered by the HPC (and potentially the relatives).

Results

Since each PwD has its own state of dementia and personal issues, it is very difficult to get uniform results, especially since they are collected orally.
Getting very nice, fully robust and reliable results, is merely a hope and a dream.
However, we can try to consider the main trends that we are interested in.
Thus, the results wil be mainly focused on:
- How much autonomy did the PwD gain?
→ what did the HPC, relatives and PwD report
→ how many tasks did they perform that they didn't do previously
→ did the relatives feel they had more time for themselves
- Did their emotional state improve?
→ feelings from the PwD themselves
→ reports from relatives and HPC
These results will most likely never be yes-no results, but more like clues or hints that show whether some things worked on not, which will be the point of our discussion.
NB: This part explains what we expect as kind of results, it will be replaced by actual results after we perform an experiment with the class. There may also be interesting points we did not think about.

Discussion

• Reliability: Yes. One could replicate the same experiment with other patients.
• Validity: TBD.
• Biases: TBD.
• Scope: No. It would be very difficult to generalize the results, since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar effort to customize Pepper for more patients should produce similar effects.
• Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.

Conclusions