Changes for page Test

Last modified by Sofia Kostakonti on 2022/04/05 14:08

From 108.1 to 107.1 From 117.1 to 116.1

From version

116.1

edited by Sofia Kostakonti
on 2022/04/04 19:58

Change comment: There is no comment for this version

To version

108.1

edited by Marlein Vogels
on 2022/04/04 17:11

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)
Attachments (0 modified, 0 added, 1 removed)
- NAO_evaluation.docx

Details

Page properties

Author

@@ -1,1 +1,1 @@
--XWiki.SKostakonti
++XWiki.MarleinVogels

Content

@@ -23,15 +23,15 @@
  == Participants ==
--As there are practical difficulties with conducting the experiment with actual people with dementia, due to both time constraints and COVID, our participants' group consists of peers from other groups and friends. In total we had 19 people take part in our experiment.
++As there are practical difficulties with conducting the experiment with actual people with dementia due to both time constraints and COVID, our participants' group consists of peers from other groups and friends. In total we had 19 people take part in our experiment.
  == Experimental design ==
--For the experiment, we used a within-subject design. All of the participants interacted with both versions of the robot, with half of the participants interacting with version 1 first and then version 2, and the other half in reverse order. This was done to counter-balance the carryover effects. Snacks were made available for the participants, in case they were prompted and were hungry. They were fully aware of them and some of the questionnaire prompts might have given them an idea of what our experiment is about (or at least that it's related to food), which might have skewed our results.
++For the experiment we used a within-subject design. All of the participants interacted with both versions of the robot, with half of the participants interacting with the version 1 first and then version 2, and the other half in reverse order. This was done to counter-balance the carryover effects. Snacks were made available for the participants, in case they were prompted and they ewre hungry. The participants were unaware of the possibility of eating snacks, to prevent disturbing the interaction with the robot. Otherwise the subjects would have been primed for eating, which would have biased the results and hide the effect of the robotic interaction.
  == Tasks ==
--The participant interacted with the robot, which was programmed to engage in a lunch discourse. Two versions were implemented: the first version (simple interaction) asks basic questions about mealtime, mostly acting as a reminder for the PwD to have lunch (basically an alarm clock). The second (advanced interaction) is our original implementation of it with the more sophisticated discourse and music.
++The participant interacted with the robot, which was programmed to engage in a lunch discourse. Two versions were implemented: the first version asks basic questions about mealtime, mostly acting as a reminder for the PwD to have lunch (basically an alarm clock). The second is our original implementation of it with the more sophisticated discourse and music.
  == Measures ==
@@ -41,31 +41,46 @@
  The procedure was conducted as follows:
--1. Welcome participants and explain what they are going to be doing.
++1. Welcome participant and explain what they are going to be doing.
 . Have them sign the permission form.
--1. Complete questionnaire 1 regarding their emotional state and hunger scale (control).
--1. Have interaction with version A of the robot.
++1. Complete questionnaire 1 regarding their emotional state (control).
++1. Have an interaction with version A of the robot.
 . Complete questionnaire 2 (extended version).
 . Have a short interview during downtime (prepared questions).
--1. Have interaction with version B of the robot.
++1. Have an interaction with version B of the robot.
 . Complete questionnaire 3 (extended version).
 . Have a short interview during downtime (prepared questions).
--We used the "Wizard of Oz" method for differentiating agreement and disagreement, to make sure that the whole process did not depend on voice recognition being good enough, and to have an overall smoother interaction. In practice, this meant that someone was pressing "y" and "n" on the keyboard according to the participants' answers, in a place the participant did not see, such as behind them. The robot's responses were hardcoded, with a few different branches available to take into account the variety of answers the participants would give. The only issue encountered was some connectivity delays at times, which only slightly affected a few of the interactions.
++We used the "Wizard of Oz" method for recognizing agreement and disagreement, to make sure that the whole process did not depend on voice recognition being good enough. In practice, this meant that someone was pressing "y" and "n" on the keyboard according to the participants answers in a place the participant did not see, such as behind them.
  == Material ==
--For the experiments, we used the NAO robot platform, and a laptop to control it. The participants completed the questionnaires on their phones by scanning a QR code. The questionnaires are a combination of questions regarding the emotional state of the participants, their hunger levels, their interaction with the robot, and the music included in the interaction. Stroopwafels and water in a clean cup were made available to see and measure how much people ate.
++For the experiments, we used the NAO robot platform, and a laptop to control it with. The participants completed the questionnaires on their phones by scanning a QR code. The questionnaires are a combination of questions regarding the emotional state of the participants, their interaction with the robot, and the music included in the interaction. Stroopwafels and water in a clean cup were made available to see and measure how much people ate.
--During the experiments, four different types of questions were given to the participants, in addition to the Consent Form and Disclaimers they had to sign in the beginning. The four sections were:
++Below are listed the contents of the three questionnares:
--1. 8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire for mood assessment
--1. 4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire to assess the pleasantness and intelligence of the robot
--1. 3 hunger and food-related questions of our own, to assess if they eat before or during the interaction (5-point Likert scale)
--1. 2 music-related questions of our own, to measure how much they enjoyed the music and what was its effect (5-point Likert scale)
++Questionnare 1:
--Before the first interaction, the participants were asked to respond to sections 1. and 3., while right after each interaction, they were asked to respond to all four sections, with the music section only present after the advanced interaction.
++* Consent Form and Disclaimers
++* Control for robot version A
++** 8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire
++** 4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire
++Questionnare 2:
++
++* Questions about robot version A
++** 3 food-related questions of our own (5-point Likert scale)
++** 2 music-related questions of our own (5-point Likert scale)
++* Control for robot version B
++** 8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire
++** 4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire
++
++Questionnare 3:
++
++* Questions about robot version B
++** 3 food-related questions of our own (5-point Likert scale)
++** 2 music-related questions of our own (5-point Likert scale)
++
  == Practicalities ==
  Before the experiment we:
@@ -235,25 +235,23 @@
  = Discussion =
--From the results we can see that the more advanced robot shows advantages over the simple version in multiple categories. Hints of better performance in other categories can be seen, but no conclusions should be drawn from the ones that lack the statistical significance.
++From the results we can see that the more advanced robot shows advantages over the simple version in many categories. Hints of better performance in other categories can be seen, but no conclusions should be drawn from the ones that lack the statistical significance.
--As for the eating, it seems that both robots have limited success in causing the people to eat as seen in Figure 1, they could cause the patients to eat more regularly, if triggered by timers or other suitable systems. It also seems that the advanced robot is better in the reminding, by a slight margin. However, the long term effects of reminding should be researched more to conclude whether the usage of the demonstrated robot platform or similar would cause the patients to eat more regularly. It is also unclear how the test setup and the limited choice of food affected the eating.
++In improving the eating, it seems that both robots have limited success in causing the people to eat as seen in Figure 1, they could cause the patients to eat more regularly, if triggered by timers or other suitable systems. It also seems that the advanced robot is better in the reminding, by a slight margin. However, the long term effects of reminding should be researched more to conclude whether the usage of the demonstrated robot platform or similar would cause the patients to eat more regularly. It is also unclear how the test setup and the limited choice of food affected the eating.
--Based on the answers of the participants regarding music seen in Figure 2, it can be seen that most of them were either indifferent or liked the music. Also, as the test personnel find the advanced robot more likeable with a 5% confidence limit (Table 7), and the advanced version was the only version with music, it seems likely that the music does make the interaction more pleasant for the personas. However, some of the likeability might be due to the other advanced features of the robot and thus more research is needed to conclude the effect of the music.
++Based on the answers of the participants regarding music seen in Figure 2, it seems that most of them were either indifferent or liked the music. Also, as the test personnel find the advanced robot more likeable with a 5% confidence limit (Table 7), and the advanced version was the only version with music, it seems likely that the music does make the interaction more pleasant for the personas. However, some of the likeability might be due to the other advanced features of the robot and thus more research is needed to conclude the effect of the music.
  The EVEA and partial Godspeed result can be seen in Figures 3-7 and Tables 1-8. The results show that with reasonable confidence (5% confidence limit), both versions of the robot decreased sadness and anxiety in the test personas. Hints are shown (10% confidence limit) that the advanced robot also decreases feelings of anger and increases happiness, while the simple robot fails to show similar results. However, in Table 7 we can see that the statistical differences in the mood differences during the interactions with the different versions are not highly significant.
--A Wilcoxon signed rank test for the partial Godspeed test shows in Table 8 that with high confidence (1% confidence limit), the intelligent robot is more likeable in comparison to the simple robot. With these results it is likely that the more advanced robot is slightly preferable and the personas might experience less negative emotions after the interaction with the robots, but it remains yet unclear if the effect is more powerful with the advanced robot.
++A Wilcoxon signed rank test for the partial Godspeed test shows in Table 8 that with high confidence (1% confidence limit), the intelligent robot is more likeable in comparison to the simple robot. With these results it is likely that the more advanced robot is slightly preferrable and the personas might experience less negative emotions after the interaction with the robots, but it is slightly unclear if the effect is more powerful with the advanced robot.
--The observations and interviews with the participants clearly demonstrated that for now, that a more friendly and intelligent robot does make the interaction with the robot more pleasant. Also, the observations do support the data from the questionnaire in terms of the likability difference between both robot types.
--
  Analysis of the results surfaced some minor issues in the experiment, such as the lack of comparison with two robots of similar features, with and without music. Also the practical limitations in the setup, such as the lack of different food options and some participants being aware of the design goals of the prototype could have interfered with the natural flow of the intercourse. With these limitations, the research method was successful in extracting differences within the robots and brought up additional directions for future research.
  The most interesting direction for future research would be the longer term studying of the effect of mealtime reminders on the health of the test subjects. The longer term health study would uncover the effect on eating frequency and the development of the relationship with the robot, for example would the test subjects that were first excited about the novel interaction with the robot, develop negative feelings about the supervision that the robot is conducting into their personal life.
--Furthermore, an aspect that was not compared in this study is how many stroopwafels the participants ate while interacting with the robot. For now, the focus was to evaluate whether the claim the robot causes the PwD - in the case of the experiment: the participants - to eat or not. For future research, the amount of food consumed by the participants could also be taken into consideration.
++Another topic to study is the differences with and without music. The effects of music could be studied with the music tailored to personal taste and all versions of the robot with and without the music playback included in the interaction. This would allow to pinpoint the effects of music, without the other features causing variance.
--Lastly, another topic to study is the differences with and without music. The effects of music could be studied with the music tailored to personal taste and all versions of the robot with and without the music playback included in the interaction. This would allow to pinpoint the effects of music, without the other features causing variance.
++Lastly, the observations and interviews with the participants clearly demonstrated that for now, that a more friendly and intelligent robot does make the interaction with the robot more pleasant. Also, the observations do support the data from the questionnaire in terms of the likability difference between both robot types.
  = Conclusions =

NAO_evaluation.docx

Author

...	...	@@ -1,1 +1,0 @@
1		-XWiki.SKostakonti

Size

...	...	@@ -1,1 +1,0 @@
1		-21.3 KB

Content