XWiki - Test

= Ideal Evaluation (actual research) =

== Problem statement and research questions ==

This project uses a Social Cognitive Engineering (SCE) approach to guide the design and research process. The SCE method provides a systematic approach to our study of robots for PwDs. The main goal of our application is to improve the well-being of the person with dementia (PwD) and of those living with them.

6

7

For the prototype that we have designed now, these are some research questions that we want to address.

8

9

1. Are the different stakeholders able to use our prototype smoothly?

10

1. Does the prototype allow the PwD greater autonomy in their day-to-day life?

11

1. Does the prototype improve the emotional state of the PwD and their relatives?

== Method ==

As people with dementia have very specific situations (and because our prototype is built to deal with that aspect of customizability), we do not go for an identical experiment for all of them. Instead, the global setup is very similar, although Pepper is customized for the needs of every patient. Our evaluation can therefore be assimilated (with some reserve) to a within-subject evaluation. We also do pre-test and post-test.

16

Our main evaluation method is summative evaluation: we are trying to determine whether the robot has an impact and improves the frequency of "yes" in our yes-no driving questions.

== Participants ==

The study will be conducted on approximately twenty people in the early stages of dementia. To avoid too much gender imbalance, there will be at least five men and five women. For the same reason, we hope to find at least five people around 50 or younger, although we expect most participants to be over 70 years old.

21

The participants will be selected based on a pool of PwD who live at home and need regular visits from an HCP. Only the volunteers will be kept for the experiment. The participants must have no experience of a Pepper robot helping with dementia.

22

As our experiment involves the PwD in their home environment, relatives and healthcare professionals (HCP) will also be involved in the process: although not being the targets of the experiment, they will contribute to obtaining the measurements and results.

23

24

== Experimental design ==

25

26

Since dementia is unique to every person, it is very hard to conduct a global experiment with the same conditions for all participants. Every one of them may have different issues in their day-to-day life, while also not having the same living conditions (alone, living with husband, family, etc), this would require a different treatment. Furthermore, we want to record whether our prototype leads to an improvement in life quality. This can not be done through a short experiment, because the reliability of such an experiment would be very low.

27

This means we need to record the patient at home and measure over an extended period of time if their well-being and autonomy globally improved.

== Tasks ==

The PwD will live their daily life, without Pepper in the beginning, then accompanied by Pepper, as if they were not under any experiment. The fact that they actually are will obviously affect their behavior. Yet, we hope that not being recorded and being under a non-invasive experiment will help them not to stress out and may make them live their life as normally as possible.

32

The tasks performed during the evaluation by Pepper, by the PwD, and by Pepper and the PwD together, will be decided in consultation with the HCP (and potentially the relatives) based on the needs of each patient.

== Measures ==

We are planning to make behavioral and emotional measurements.

37

Behavioral measurements are the actions that the PwD is going to perform during the week, so it can be considered as subjective quantitative data. This will involve the relatives, the HCP, and the PwD themselves to quantify whether the use of Pepper did actually result in an increase in autonomy for the PwD.

38

Emotional measurements are more related to the state of mind, change of expression, and mood, so they can be considered as qualitative data. This can be measured by frequent talks with the PwD, either by the relatives or the HCP.

39

Measures will be done by oral discussions with the PwD, HCP, and relatives.

== Procedure ==

The whole experiment process is 4 weeks long, although only weeks 1 and 4 are technically part of the experiment itself.

44

During week 1, the PwD, the HCP, and the relatives will be asked to pay increased attention. Behavioral and emotional data will be collected. This is a regular week for the PwD, i.e., in the usual situation, except that there is more attention dedicated to them.

45

During weeks 2 and 3, the PwD, the HCP, and the relatives will be introduced to Pepper, with the goal of getting used to it.

46

During week 4, which is the actual week of the experiment, attention will be spent trying to mirror week 1 as closely as possible. Behavioral and emotional data will be collected again.

== Material ==

1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.

51

1. Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behavior for every participant. However, the input data will be entered by the HCP(and potentially the relatives).

== Results ==

Since each PwD has its own state of dementia and personal issues, it is very difficult to get uniform results, especially since they are collected orally.

56

Getting very nice, fully robust, and reliable results is merely a hope and a dream.

57

However, we can try to consider the main trends that we are interested in.

58

Thus, the results will be mainly focused on:

59

- How much autonomy did the PwD gain?

60

→ what did the HCP, relatives, and PwD report

61

→ how many tasks did they perform that they didn't do previously

62

→ did the relatives feel they had more time for themselves

63

- Did their emotional state improve?

64

→ feelings from the PwD themselves

65

→ reports from relatives and HCP

66

These results will most likely never be yes-no results, but more like clues or hints that show whether some things worked on not, which will be the point of our discussion.

== Discussion ==

* Reliability: Yes. One could replicate the same experiment with other patients.

71

* Validity: TBD. (This this ideal evaluation has not been conducted, we cannot really estimate this.)

72

* Biases: TBD.(This this ideal evaluation has not been conducted, we cannot really estimate this.)

73

* Scope: No. It would be very difficult to generalize the results since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar efforts to customize Pepper for more patients should produce similar effects.

74

* Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.

== Conclusions ==

(This this ideal evaluation has not been conducted, we do not have conclusions to be drawn.)

79

80

= Feasible evaluation (students) =

81

82

== Problem statement and research questions ==

83

84

For the prototype that we have designed now, and for a smaller scope evaluation, the research questions are smaller-scope versions of our main research questions for the previous part.

85

We also cannot test all our claims during only a short evaluation session with students.

86

87

1. Are the participants able to use our prototype smoothly?

88

1. Does the prototype improve the autonomy of the patient on a specific activity?

89

1. Does the prototype improve the emotional state of the participant after completing a specific activity?

90

91

Based on these questions and our evaluation setting, we can evaluate some of our claims from our Claim page.

92

93

1. CL01: The PwD becomes aware of Pepper's presence.

94

During the experiment, we can observe if this happens.

95

1. CL02: The PwD feels understood.

96

In the questionnaire, there are questions related to this.

97

1. CL03: The PwD performs an activity step.

98

We can verify this claim based on how far the participant went in the activity.

99

1. CL08: The PwD feels accomplished.

100

After the activity is done, there are two specific questions related to accomplishment.

101

1. CL10: The PwD feels reassured.

102

We can evaluate this from observing the participants, and from the affect questionnaire.

103

1. CL11: The PwD feels content.

104

There is a question related to this in the questionnaire.

== Method ==

In order to collect qualitative data, we prepared two questionnaires to give to the participants: the affect assessment questionnaire and the system assessment questionnaire. All questionnaire questions are expressed in the form of statements and the participant can express one out of seven levels of agreement/disagreement from strongly disagree to strongly agree. The only exception is the additional remarks field at the end of the system assessment questionnaire, where the participants can freely put any comment/remark/feedback they have on the experiment.

109

110

**Affect assessment**

111

112

The affect assessment questionnaire asks the participant to describe their mood. The questionnaire is given two times: once before the experiments and once right after. The purpose of this questionnaire is to compare the mood experienced by the participant before the interaction with Pepper and after. The idea is to assess the effect of the activity. The questionnaire has six questions each designed to measure the level of a certain feeling experienced at the moment of filling the questionnaire. The six feelings we take into consideration are evaluated from the following statements:

1. I feel sad.

1. I feel content.

1. I feel calm.

1. I feel tired.

1. I feel nervous.

1. I feel caring.

**System assessment**

122

123

The system assessment questionnaire is given after the interaction with Pepper and its purpose is to assess the participant's experience with Pepper in more detail. The questionnaire has fourteen questions which are designed to answer research questions as described below:

124

125

1. I like gardening

126

1. I would have known how to do the whole task without Pepper.

127

1. I think Pepper made the task easier for me.

128

1. Pepper was easy to understand.

129

1. I enjoyed the task more than if I had had to do it alone.

130

1. I am pleased that Pepper reminded me to do the activity.

131

1. I feel like completing the task was a good accomplishment.

132

1. I feel like I accomplished it myself.

133

1. I felt in control of what I had to do.

134

1. I felt annoyed by Pepper.

135

1. I felt frustrated by the task.

136

1. I felt pressured by Pepper.

137

1. Pepper cared about helping me.

138

1. I would trust Pepper with more important activities.

Research questions:

1. To what extent did Pepper improve the task's experience? [Questions: 3,4,5,10,11,12]

143

1. To what extent did Pepper improve the participant's autonomy and perception of control? [Questions: 6, 8, 9]

144

1. To what extent did Pepper improve the participant's sense of accomplishment? [Questions: 7]

145

1. Did the participant perceive Pepper as a social agent? [Questions: 13, 14]

146

1. How was Pepper's ability to guide the participant through the task? [Questions: 4]

147

148

Question 1 is used to take into consideration the expected baseline enjoyment of the task for each participant.

== Participants ==

In our situation, our participants are 24 students from TUD. Most of them come from Computer Science and/or from the SCE class. One third of them are female and two thirds male, so gender balance is decent.

153

Roleplaying PwDs may be difficult for students, especially for those who have not followed the course. For this reason, and in order not to introduce an extra bias "knows about dementia"/"does not know about dementia", we decided to ask the students to just act as students.

154

155

== Experimental design ==

156

157

Since our evaluating process is relatively short, we use within-subject, which means each participant goes through all conditions. In this way, our experiment tends to have more statistical power and less variability.

== Tasks ==

All participants will go through our designed testing process, which combines a calendar event reminder and an activity breakdown.

162

Pepper will propose the activity to the participant, then go step by step with them through the activity.

== Measures ==

We measure the mood of the participants through the affect assessment questionnaires. We also measure some aspects of the interaction between Pepper and the participant through the system assessment questionnaire. Both are qualitative data, as the participant gives an answer ranging from strongly disagree to strongly agree.

== Procedure ==

Before the experiment, we explain to the participants the purpose of our evaluation (context of SCE class, our prototype).

171

We tell them that we will gather data in the questionnaires, that the data are fully anonymized and will only be used for our result analysis in this course.

172

We also briefly explain to them how Pepper works, especially that she only listens when the eyes are blue.

173

After the explanation, the participant can fill the consent form and the first affect questionnaire.

174

Then, the experiment starts. The experiment is conducted in autonomy between the participant and Pepper. We only intervene if asked by the participant or if something is going wrong with the experiment.

175

After the activity is finished by the participant, they fill the second affect questionnaire and the system assessment questionnaire.

176

After this is done, we thank them for their participation and the evaluation session is over.

== Material ==

1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers. The consent form is accessed online via a QR code.

181

1. Questionnaires. There are three questionnaires to answer (mood 1, mood 2, system). They are also accessed online via a QR code.

182

1. Pepper robot. Our robot is programmed using Choregraphe. The robot will have the same behaviour for every participant. However, the input data will be entered by the HCP (and potentially the relatives).

183

1. Gardening stuff. In order to conduct the activity, we bought and brought basic stuff required for gardening (soil, pot, seeds, etc).

== Results ==

=== mood questionnaire results ===

188

189

[[interactive version>>https://pietro99.github.io/SCE/graphs/mood_questionnaire.html]]

[[image:mood.svg]]

<!DOCTYPE html>

<html>

<head>

<style>

table {

font-family: arial, sans-serif;

199

border-collapse: collapse;

width: 100%;

}

td, th {

border: 1px solid #dddddd;

text-align: left;

padding: 8px;

}

tr:nth-child(even) {

background-color: #dddddd;

}

</style>

</head>

<body>

<h5>Wilcoxon Signed-rank test</h5>

217

<h6><i>H0</i>: The mood distribution before and after the interaction with Pepper is the same</h6>

<table>

<tr>

<th>test results </th>

222

<th>I feel caring </th>

223

<th>I feel content </th>

224

225

<th>I feel tired </th>

226

227

<th>I feel nervous </th>

</tr>

<tr>

<th>p-value</th>

</tr>

<tr>

<th>statistics</th>

</tr>

</table>

</body>

</html>

<!DOCTYPE html>

<html>

<head>

<style>

table {

font-family: arial, sans-serif;

263

border-collapse: collapse;

width: 100%;

}

td, th {

border: 1px solid #dddddd;

text-align: left;

padding: 8px;

}

tr:nth-child(even) {

background-color: #dddddd;

}

</style>

</head>

<body>

<h5>Wilcoxon Rank-Sum test</h5>

281

<h6><i>H0</i>: The mood distribution after the interaction with Pepper for people who like gardening and people who don't like gardening is the same</h6>

<table>

<tr>

<th>test results </th>

286

<th>I feel caring </th>

287

<th>I feel content </th>

288

289

<th>I feel tired </th>

290

291

<th>I feel nervous </th>

</tr>

<tr>

<th>p-value</th>

</tr>

<tr>

<th>statistics</th>

</tr>

</table>

</body>

</html>

=== System questionnaire results ===

320

321

[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_1.html]]

[[image:group1.svg]]

<!DOCTYPE html>

<html>

<head>

<style>

table {

font-family: arial, sans-serif;

332

border-collapse: collapse;

width: 100%;

}

td, th {

border: 1px solid #dddddd;

text-align: left;

padding: 8px;

}

tr:nth-child(even) {

background-color: #dddddd;

}

</style>

</head>

<body>

<h5>Wilcoxon Rank-Sum test test</h5>

350

<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>

<table>

<tr>

<th>test results </th>

355

<th>I would have known how to do the whole task without Pepper.</th>

356

<th>I think Pepper made the task easier for me.</th>

357

<th>I enjoyed the task more than if I had had to do it alone.</th>

358

<th>Pepper was easy to understand.</th>

359

<th>I am pleased that Pepper reminded me to do the activity.</th>

</tr>

<tr>

<th>p-value</th>

</tr>

<tr>

<th>statistics</th>

</tr>

</table>

</body>

</html>

[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_2.html]]

[[image:group2.svg]]

<!DOCTYPE html>

<html>

<head>

<style>

table {

font-family: arial, sans-serif;

397

border-collapse: collapse;

width: 100%;

}

td, th {

border: 1px solid #dddddd;

text-align: left;

padding: 8px;

}

tr:nth-child(even) {

background-color: #dddddd;

}

</style>

</head>

<body>

<h5>Wilcoxon Rank-Sum test test</h5>

415

<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>

<table>

<tr>

<th>test results </th>

420

<th>I feel like completing the task was a good accomplishment.</th>

421

<th>I feel like I accomplished it myself.</th>

422

<th>I felt in control of what I had to do.</th>

</tr>

<tr>

<th>p-value</th>

</tr>

<tr>

<th>statistics</th>

</tr>

</table>

</body>

</html>

[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_3.html]]

[[image:group3.svg]]

<!DOCTYPE html>

<html>

<head>

<style>

table {

font-family: arial, sans-serif;

454

border-collapse: collapse;

width: 100%;

}

td, th {

border: 1px solid #dddddd;

text-align: left;

padding: 8px;

}

tr:nth-child(even) {

background-color: #dddddd;

}

</style>

</head>

<body>

<h5>Wilcoxon Rank-Sum test test</h5>

472

<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>

<table>

<tr>

<th>test results </th>

477

<th>I felt annoyed by Pepper.</th>

478

<th>I felt frustrated by the task.</th>

479

<th>I felt pressured by Pepper.</th>

</tr>

<tr>

<th>p-value</th>

</tr>

<tr>

<th>statistics</th>

</tr>

</table>

</body>

</html>

[[interactive version>>https://pietro99.github.io/SCE/graphs/first_questionnaire_4.html]]

[[image:group4.svg]]

<!DOCTYPE html>

<html>

<head>

<style>

table {

font-family: arial, sans-serif;

512

border-collapse: collapse;

width: 100%;

}

td, th {

border: 1px solid #dddddd;

text-align: left;

padding: 8px;

}

tr:nth-child(even) {

background-color: #dddddd;

}

</style>

</head>

<body>

<h5>Wilcoxon Rank-Sum test test</h5>

530

<h6><i>H0</i>: The distribution of the sytem's questionnaire answers from people who like gardening and form people who don't like gardening is the same</h6>

<table>

<tr>

<th>test results </th>

535

<th>Pepper cared about helping me.</th>

536

<th>I would trust Pepper with more important activities.</th>

</tr>

<tr>

<th>p-value</th>

</tr>

<tr>

<th>statistics</th>

</tr>

</table>

</body>

</html>

Since each PwD has its own state of dementia and personal issues, it is very difficult to get uniform results, especially since they are collected orally.

559

Getting very nice, fully robust and reliable results, is merely a hope and a dream.

560

However, we can try to consider the main trends that we are interested in.

561

Thus, the results wil be mainly focused on:

562

- How much autonomy did the PwD gain?

563

→ what did the HCP, relatives and PwD report

564

→ how many tasks did they perform that they didn't do previously

565

→ did the relatives feel they had more time for themselves

566

- Did their emotional state improve?

567

→ feelings from the PwD themselves

568

→ reports from relatives and HCP

569

These results will most likely never be yes-no results, but more like clues or hints that show whether some things worked on not, which will be the point of our discussion.

570

NB: This part explains what we expect as kind of results, it will be replaced by actual results after we perform an experiment with the class. There may also be interesting points we did not think about.

== Discussion ==

* Reliability: Yes. One could replicate the same experiment with other patients.

575

* Validity: TBD.

576

* Biases: TBD.

577

* Scope: No. It would be very difficult to generalize the results, since each prototype is built for a special patient. However, if the results conclude that the customized prototypes did improve the well-being of the people, then similar effort to customize Pepper for more patients should produce similar effects.

578

* Ecological validity: Yes. Since we compare "without Pepper" (BEFORE) and "with Pepper" (AFTER) in a similar environment (i.e., for everything but Pepper), the results are not dependent on the environment.

579

580

== Conclusions ==

Wiki source code of Test