Changes for page Test

Last modified by Sofia Kostakonti on 2022/04/05 14:08

From 85.1 to 86.1 From 95.2 to 96.1

From version

86.1

edited by Aleksanteri Hämäläinen
on 2022/04/03 17:59

Change comment: There is no comment for this version

To version

95.2

edited by Veikko Saikkonen
on 2022/04/04 12:12

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--XWiki.ahamalainen
++XWiki.VSaikkonen

Content

@@ -19,7 +19,7 @@
  = Method =
--The prototype is evaluated with an in-person experiment with multiple participants.
++The prototype was evaluated with an in-person experiment with multiple participants.
  == Participants ==
@@ -42,60 +42,54 @@
  The procedure was conducted as follows:
 . Welcome participant and explain what they are going to be doing.
--2. Have them sign the permission form.
--* Complete questionnaire 1 regarding their emotional state (control).
--* Have an interaction with version A of the robot.
--* Complete questionnaire 2 (extended version).
--* Have a short interview during downtime (prepared questions).
--* Have an interaction with version B of the robot.
--* Complete questionnaire 3 (extended version).
--* Have a short interview during downtime (prepared questions).
++1. Have them sign the permission form.
++1. Complete questionnaire 1 regarding their emotional state (control).
++1. Have an interaction with version A of the robot.
++1. Complete questionnaire 2 (extended version).
++1. Have a short interview during downtime (prepared questions).
++1. Have an interaction with version B of the robot.
++1. Complete questionnaire 3 (extended version).
++1. Have a short interview during downtime (prepared questions).
--== Material ==
--For the experiments, we'll be using the NAO robot platform, as well as a laptop for the participants to complete the questionnaires on. The questionnaire will be a combination of questions regarding the emotional state of the participants, their interaction with the robot, and the music included in the interaction. Food will be made available to see and measure how much people will eat.
++We used the "Wizard of Oz" method for recognizing agreement and disagreement, to make sure that the whole process did not depend on voice recognition being good enough. In practice, this meant that someone was pressing "y" and "n" on the keyboard according to the participants answers in a place the participant did not see, such as behind them.
--Questionnaires:
--Consent Form and Disclaimers
--8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire
--4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire
--3 food-related questions of our own (5-point Likert scale)
--2 music-related questions of our own (5-point Likert scale)
++== Material ==
--== Practicalities ==
++For the experiments, we used the NAO robot platform, and a laptop to control it with. The participants completed the questionnaires on their phones by scanning a QR code. The questionnaires are a combination of questions regarding the emotional state of the participants, their interaction with the robot, and the music included in the interaction. Stroopwafels and water in a clean cup were made available to see and measure how much people ate.
--Beforehand:
++Below are listed the contents of the three questionnares:
--* Do a practice round by ourselves
--** Film this
--* Contact other groups and decide on a time slot
--** Might be better to reserve in 10 min slots, so that people don't have to wait so much
--** If possible, this could be done in parallel with another groups testing
--* Reserve lab
--* Buy snacks
++Questionnare 1:
++* Consent Form and Disclaimers
++* Control for robot version A
++** 8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire
++** 4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire
--During:
++Questionnare 2:
++* Questions about robot version A
++** 3 food-related questions of our own (5-point Likert scale)
++** 2 music-related questions of our own (5-point Likert scale)
++* Control for robot version B
++** 8 questions from the [[EVEA>>https://www.ucm.es/data/cont/docs/39-2013-04-19-EVEA%20-%20Datasheet.pdf]] questionnaire
++** 4 questions from the [[Godspeed>>https://www.bartneck.de/2008/03/11/the-godspeed-questionnaire-series/]] questionnaire
--1. Give starting questionnare to fill while people are waiting for the previous participant
--2. Guide the participant to the testing spot
--3. Inform the participant where the snacks are
--4. Run the first version
--5. Give the mid-questionnare
--6. Run the other test
--7. Conduct the questionnare for the participant
--8. Give the participant the end-questionnare
++Questionnare 3:
++* Questions about robot version B
++** 3 food-related questions of our own (5-point Likert scale)
++** 2 music-related questions of our own (5-point Likert scale)
--Other practicalities during:
++== Practicalities ==
--* We will use the "Wizard of Oz" method for recognizing agreement and disagreement, to make sure that the whole process does not depend on voice recegnition being good enough
--** Someone will press eg. "y" and "n" on the keyboard according to the participants answers
--* We will change the order in which the smart and basic versions are for each participant
--** this way if someone doesn't show up, we don't get skewed amounts
++Before the experiment we:
--After:
++* did a practice round by ourselves
++** This was filmed to have a controlled performance to give an example of the experiment if needed
++* contacted other groups and decide on scheduling
++** Each participant was booked a 20 min slot
++* reserved the lab
++* bought the stroopwafels
--* Analyze results
--
  = Results =
  The results were gathered from 19 personnel, all of whom interacted first with one version of the robot and then the other. 10 of the participants interacted first with the simple version, nine having their first interaction with the advanced version.
@@ -225,15 +225,33 @@
  |Statistic|36|70
  |P-value|0.01|0.17
--= Discussion =
--Analysis the results surfaced some minor issues in the experiment, such as the lack of comparison with two robots of similar features, with and without music. Also the practical limitations in the setup, such as the lack of different food options and some participants being aware of the design goals of the prototype could have interfered with the natural flow of the intercourse. With these limitations, the research method was successful in extracting differences within the robots and brought up additional directions for future research.
++== Qualitative Results: Quotes and observations ==
++As described, during the experiment, the interaction between the participants and the robot was observed. This section will elaborate on findings from those observations and quotes from participants.
--The most interesting direction for future research would be the longer term studying of the effect of mealtime reminders on the health of the test subjects. The longer term health study would uncover the effect on eating frequency and the development of the relationship with the robot, for example would the test subjects that were first excited about the novel interaction with the robot, develop negative feelings about the supervision that the robot is conducting into their personal life.
++After each interaction section, the the participant was asked how the interaction with the robot felt. From the interaction with the less intelligent version of the robot, some interesting quotes were:
++* “The robot was bit direct.”
++* “Efficient interaction, but less friendly than the other interaction.”
++* “Strange, I did not catch the questions.”
++* “It felt short.”
--Another topic to study is the differences with and without music. The effects of music could be studied with the music tailored to personal taste and all versions of the robot with and without the music playback included in the interaction. This would allow to pinpoint the effects of music, without the other features causing variance.
++Some of these quotes stress the fact that the less intelligent prototype interaction was rather short and direct. It should be said that the sequence of the interactions seemed to have some impact on how the participants experienced the interaction. Some participants who first experienced the less intelligent prototype were smiling and positively surprised during this interaction, while others who first experienced the intelligent prototype were over all smiling less while interacting with the less intelligent robot.
--= Conclusions =
++From the interaction with the intelligent version of the robot, some interesting quotes were:
++* “I think it’s perfect, the robot is very friendly. I liked that the robot sat down with me after a while.”
++* “The interaction felt quite natural.”
++* “Nao answered pretty quickly, you don’t have to wait for an answer. It is quite a happy robot.”
++* “Suggestion to eat was still a bit on the side, a little subtle if I would have dementia.”
++* “Very nice, calming, I could have stayed longer with the music.”
++* “It was good, natural, understands what I’m saying.”
++Some participants clearly expressed how friendly they found the intelligent version of the robot. The sequence of the interactions did not seem to impact their feeling about the interaction as much as with the interaction with the less intelligent version of the robot.
++Some reported that the interaction felt natural and intuitive.
++As for the music, some participants told us that the music did was a useful and pleasant addition to the interaction with the robot.
++As for the suggestion to eat and drink, one participant reported that the suggestions to eat and drink were perhaps too friendly and too subtle.
++From our observations, it seemed as if participants were either smiling more during the interaction with the intelligent version of the robot or concentrating on the interaction more carefully compared to the interaction with the less intelligent version of the robot.
++
++= Discussion =
++
  From the results we can see that the more advanced robot shows advantages over the simple version in many categories. Hints of better performance in other categories can be seen, but no conclusions should be drawn from the ones that lack the statistical significance.
  In improving the eating, it seems that both robots have limited success in causing the people to eat as seen in Figure 1, they could cause the patients to eat more regularly, if triggered by timers or other suitable systems. It also seems that the advanced robot is better in the reminding, by a slight margin. However, the long term effects of reminding should be researched more to conclude whether the usage of the demonstrated robot platform or similar would cause the patients to eat more regularly. It is also unclear how the test setup and the limited choice of food affected the eating.
@@ -244,7 +244,26 @@
  A Wilcoxon signed rank test for the partial Godspeed test shows in Table 8 that with high confidence (1% confidence limit), the intelligent robot is more likeable in comparison to the simple robot. With these results it is likely that the more advanced robot is slightly preferrable and the personas might experience less negative emotions after the interaction with the robots, but it is slightly unclear if the effect is more powerful with the advanced robot.
++Analysis the results surfaced some minor issues in the experiment, such as the lack of comparison with two robots of similar features, with and without music. Also the practical limitations in the setup, such as the lack of different food options and some participants being aware of the design goals of the prototype could have interfered with the natural flow of the intercourse. With these limitations, the research method was successful in extracting differences within the robots and brought up additional directions for future research.
++The most interesting direction for future research would be the longer term studying of the effect of mealtime reminders on the health of the test subjects. The longer term health study would uncover the effect on eating frequency and the development of the relationship with the robot, for example would the test subjects that were first excited about the novel interaction with the robot, develop negative feelings about the supervision that the robot is conducting into their personal life.
++
++Another topic to study is the differences with and without music. The effects of music could be studied with the music tailored to personal taste and all versions of the robot with and without the music playback included in the interaction. This would allow to pinpoint the effects of music, without the other features causing variance.
++
++Lastly, the observations and interviews with the participants clearly demonstrated that for now, that a more friendly and intelligent robot does make the interaction with the robot more pleasant.
++
++= Conclusions =
++
++From the results it seems that in short-term interactions, both of the robots does remind the persons of their hunger, but the test setup might have caused many people not to eat or not to be hungry when arriving
++
++<ol>
++<li>Does the robot remind the PwD of their hunger?</li>
++<li>Does the music make the eating more enjoyable for the PwD?</li>
++<li>Does the PwD experience less negative emotions, such as agitation, sadness, embarrassment, after the interaction with the 'intelligent' robot?</li>
++<li>* Does the robot cause PwD to eat more regularly?</li>
++</ol>
++
++
  = Appendix =
  == Experiment introduction for participants ==