Skip to Content

Changes for page b. Test

Last modified by Jean-Paul Smit on 2024/04/09 15:23
From version 10.1
edited by William OGrady
on 2024/03/26 11:43
Change comment: There is no comment for this version
To version 23.2
edited by Jean-Paul Smit
on 2024/04/07 00:03
Change comment: There is no comment for this version

Summary

Details

Page properties
Title
... ... @@ -1,1 +1,1 @@
1 -c. Test
1 +b. Test
Author
... ... @@ -1,1 +1,1 @@
1 -xwiki:XWiki.WilliamOGrady
1 +xwiki:XWiki.jeanpaulsmit
Content
... ... @@ -1,10 +1,16 @@
1 1  === 1. Introduction ===
2 2  
3 -The aim of this study is to evaluate the effectiveness of the Nao robot in aiding memory retrieval through the use of a between-subjects design with a test group and a control group. We will simulate conditions of early-stage dementia in student participants to test whether interactions with Nao enhance the retrieval of memories fabricated created by us, mimicking lost memories typical of individuals with dementia.
3 +This study aims to evaluate the effectiveness of the Nao robot in aiding memory retrieval through the use of a between-subjects design with a test group and a control group. We will simulate conditions of early-stage dementia in student participants to test whether interactions with Nao enhance the retrieval of memories fabricated created by us, mimicking lost memories typical of individuals with dementia.
4 4  
5 +Recall the image from the [[3. Evaluation Methods>>Main.b\. Human Factors.Measuring Instruments.WebHome]] as in the figure below. The first two concepts, that of Autonomy and Relatedness and the memory self-efficacy, will be tested by using validated surveys. The memory recall will be tested by a custom survey that was catered to be used with the robot in the domain of interest, that is of using the NAO as an encyclopedia for recalling familiar people.
5 5  
7 +[[image:Socio-Cognitive Engineering - Frame 1.jpg]]
8 +
9 +
6 6  === 2. Method ===
7 7  
12 +[[Figure: procedure of the controlled experiment>>image:Procedure.png]]
13 +
8 8  ==== 2.1 Participants ====
9 9  
10 10  As actual PwDs are not available for this research, the participants will consist of several student volunteers. These will be divided into two groups: those using the Nao robot for memory retrieval (the test group) and those using text-based prompts (the control group).
... ... @@ -18,12 +18,9 @@
18 18  
19 19  ==== 2.3 Measures ====
20 20  
21 -Memory retrieval success will be measured using a quiz based on the fabricated memories. Participant feedback and observational notes will also be collected to assess the user experience and interaction effectiveness.
22 -We will evaluate the recall and memory-efficacy of both groups and try to relate the given answers to values **(link to evaluation in foundation) **such as autonomy and relatedness.
27 +Memory retrieval success will be measured using a [[quiz>>doc:.Quiz.WebHome]] based on the fabricated memories. Participant feedback and observational notes will also be collected to assess the user experience and interaction effectiveness. We will evaluate the recall and memory-efficacy of both groups and try to relate the given answers to values** **such as autonomy and relatedness.
23 23  
24 -**TODO: More info here once we have the actual evaluation forms (knowledge quiz + emotional questionnaire)**
25 25  
26 -
27 27  ==== 2.4 Procedure ====
28 28  
29 29  Participants will receive an overview of their role as a PwD and undergo a session with either Nao or text prompts to retrieve memories. Following the session, they will complete the memory quiz. Feedback will be gathered post-interaction.
... ... @@ -34,12 +34,12 @@
34 34  1. **Recruitment and Consent**: Recruit a diverse group of student volunteers, ensuring informed consent for participation.
35 35  1. **Training**: Brief participants on their roles and the concept of simulated memory loss without revealing specific details of the fabricated memories.
36 36  
37 -//**Phase 2: Baseline Assessment (Optional)**//
40 +//**Phase 2: Baseline Assessment**//
38 38  
39 39  1. **Pre-test**: Optionally, assess the students’ ability to recall general information unrelated to the fabricated memories to establish a baseline.
40 40  1. **Documentation**: Record responses to analyze as a baseline for later comparison.
41 41  
42 -//**Phase 3: Memory Retrieval Test**//
45 +//**Phase 3: Memory Recall**//
43 43  
44 44  1. **Introduction**: Introduce participants in the test group to the NAO robot, explaining its purpose, and provide the control group with their first text-based memory prompt.
45 45  1. **Simulation**: Conduct sessions where the test group interacts with Nao aiming to recall a set of fabricated memories.
... ... @@ -60,8 +60,44 @@
60 60  1. **Staff and Observer Insights**: Analyze observations from staff and observers for additional perspectives on the effectiveness and emotional impact of the interactions.
61 61  1. **Improvement Strategies**: Based on the feedback and results, develop a plan to enhance the memory retrieval capabilities of the Nao robot and the overall test design.
62 62  
63 -== //**Ethical Considerations**// ==
66 +=== 3. Results ===
64 64  
65 -We commit to high ethical standards, respecting the sensitive nature of simulating dementia conditions, and ensuring the well-being and dignity of all participants throughout the study. To ensure this we present a form containing the ethical considerations to each participant. **TODO: put in link to ethics form?**
68 +Twenty University students participated in the controlled experiment (M= 45%, F=55%). After gaining consent through the consent form, all participants were briefed to act as if experiencing the interaction from the viewpoint of a PwD. From the sample, half experienced
66 66  
70 +[[Pie chart showing the gender distribution of our sample. / Boxplot chart showing the familiarity with robots and the attitude towards technology.>>image:gender & familiarity.png]]
71 +
72 +
73 +Most resulting p values had an alpha above 0.05 and thus were not significant enough.
74 +
75 +[[Table showing the Affect variables>>image:Table of means.png||height="214" width="737"]]
76 +
77 +=== 4. Discussion ===
78 +
79 +**Sample Group and Size impact validity of the study**
80 +
81 +The ecological validity of the study is impacted by the fact that there were no PwD in our sample.The scope of the experiment was limited to TU Delft University students. That means that future research may benefit from a closer approach to an experiment which is closer to the experience of PwD. Moreover, the controlled experiment was restricted to a cohort of 20 participants, underscoring the potential for enhancing result validity through the utilization of a larger sample size
82 +
83 +
84 +**//Participants don't know about points system so they didn't answer with "getting the most points" in mind//**
85 +
86 +//Ambiguities in the evaluation briefing has led to several aspects in the results that might misrepresent the participants' gathered knowledge. Points were awarded to the participant for certain key descriptors, each family member's role, occupation, likes, dislikes and so on. Of course, this wasn't known to the participant, so they might have omitted descriptors they deemed less important or trivial and therefore scored worse, even though they'd heard and remembered them.//
87 +
88 +
89 +//**Participants don't know which people or facts are important, so they can get stuck in spots that are unrewarded**//
90 +
91 +//The choice to create a sprawling, multi-faceted database also had the side-effect of participants finding out a lot of information that was not rewarded by the grading system in any way. For example, the user can ask the robot to elaborate about certain memories or character traits of family members. There are also people in the database that act as ancillary characters and to create a sense of realism to the database, but participants can likely get stuck on learning about them as there is no implied hiearchy of importance to the participant. //
92 +
93 +
94 +**//GPT Assistant can elaborate on any question, and therefore the user does not know what belongs to the database, and don't know where to focus//**
95 +
96 +//Another limiation of the evaluation is with the GPT Assistant's ability to consistently elaborate on any question posed by the participant.//
97 +
98 +
99 +
100 +=== 5. Conclusions ===
101 +
102 +===== //**Ethical Considerations**// =====
103 +
104 +We commit to high ethical standards, respecting the sensitive nature of simulating dementia conditions, and ensuring the well-being and dignity of all participants throughout the study. To ensure this we present a form containing the ethical considerations to each participant.
105 +
67 67  
Procedure.png
Author
... ... @@ -1,0 +1,1 @@
1 +xwiki:XWiki.jeanpaulsmit
Size
... ... @@ -1,0 +1,1 @@
1 +606.1 KB
Content
Socio-Cognitive Engineering - Frame 1.jpg
Author
... ... @@ -1,0 +1,1 @@
1 +xwiki:XWiki.jeanpaulsmit
Size
... ... @@ -1,0 +1,1 @@
1 +2.1 MB
Content
Table of means.png
Author
... ... @@ -1,0 +1,1 @@
1 +xwiki:XWiki.jeanpaulsmit
Size
... ... @@ -1,0 +1,1 @@
1 +95.5 KB
Content
gender & familiarity.png
Author
... ... @@ -1,0 +1,1 @@
1 +xwiki:XWiki.jeanpaulsmit
Size
... ... @@ -1,0 +1,1 @@
1 +47.9 KB
Content