Changes for page b. Test

Last modified by Demi Breen on 2023/04/09 15:10

From 57.1 to 58.1

From version 58.1

edited by Hugo van Dijk
on 2023/04/08 17:18

Change comment: There is no comment for this version

To version 68.1

edited by Demi Breen
on 2023/04/09 15:10

Change comment: There is no comment for this version

Raw
Rendered

Summary

Page properties (2 modified, 0 added, 0 removed)

Details

Page properties

Author

@@ -1,1 +1,1 @@
--XWiki.hjpvandijk
++XWiki.Demibreen1000

Content

@@ -20,7 +20,7 @@
  == 2.1 Participants ==
--The test will be with students from the Delft University of Technology that are also following the Socio-Cognitive course. For the results to be valid 15 participants in a study is the minimum amount as presented by researcher Marc Brysbaert [1]. Since the decision was made for in-between subject design, which will be elaborated more in the Experimental Design section, there are thus 30 participants necessary. Due to the time constraint and the number of students in the course, 30 participants will probably not be reachable for this experiment so fewer students will participate. Also, the use of a control group is not within reach because of the lack of resources.
++The test will be with students from the Delft University of Technology that are also following the Socio-Cognitive course. For the results to be valid 15 participants in a study are the minimum amount as presented by researcher Marc Brysbaert [1]. Since the decision was made for in-between subject design, which will be elaborated more in the Experimental Design section, there are thus 30 participants necessary. Due to the time constraint and the number of students in the course, 30 participants will probably not be reachable for this experiment so fewer students will participate. Also, the use of a control group is not within reach because of the lack of resources.
  The participants are all young and do not have any form of dementia. The results of the research can thus not be seen as sufficient for a real-life implementation of the prototype. The students will be familiar with the robot and thus further explanation of the working of the robot is less relevant. Also, the students will feel more comfortable with the robot from the start. This might be very different to the PwD which might be a little hesitant in interacting with the robot. The integration of that aspect will be of great importance before real implementation.
@@ -28,7 +28,7 @@
  **Methodological set-up:**
--Pepper will be turned on and will start a conversation with the participant. It will ask the participant to go on a walk and based on the answer Pepper will go on the walk immediately or will try to motivate the PwD to go on a walk with him. During the walk, Pepper will ask the participant questions to keep the participant engaged and keep continuing on the walk. So for the experiment, a Pepper robot, the Choregraphe software, and also freedom of movement are needed. The full step-by-step schedule of the experiment is given in the attachments for both the [[goal-based motivation>>attach:Ontology & robot design - goal-based.pdf]] and the [[emotion-based motivation>>attach:Ontology & robot design - emotion-based.pdf]].
++Pepper will be turned on and will start a conversation with the participant. It will ask the participant to go on a walk and based on the answer Pepper will go on the walk immediately or will try to motivate the PwD to go on a walk with him. During the walk, Pepper will ask the participant questions to keep the participant engaged and keep continuing on the walk. So for the experiment, a Pepper robot, the Choregraphe software, and also freedom of movement are needed. The full step-by-step schedule of the experiment is given in the attachments for both the [[goal-based motivation>>attach:Ontology & robot design - goal-based.pdf]] and the [[emotion-based motivation>>attach:Ontology & robot design - emotion-based.pdf]], they are also presented in [[a. Prototype>>3\. Evaluation.a\. Prototype.WebHome]]
  **Conditions**:
@@ -38,7 +38,7 @@
  **Subject design:**
--For the experiment it was chosen to do in-between subject design as the learning effect in the within-subject design might negatively impact the results. When the same participants have to perform the same routine twice, they might get bored and frustrated due to some repetitions in questioning in the different motivation methods. In the in-between-subject design, multiple experiments will be taken with different participants. The total group of participants will thus be divided in half regarding either the emotion-based or the goal-based motivation. Both are measured by a questionnaire, which will be discussed in more detail later on in the evaluation part.
++For the experiment it was chosen to do between-subject design as the learning effect in the within-subject design might negatively impact the results. When the same participants have to perform the same routine twice, they might get bored and frustrated due to some repetitions in questioning in the different motivation methods. In the between-subject design, multiple experiments will be taken with different participants. The total group of participants will thus be divided in half regarding either the emotion-based or the goal-based motivation. Both are measured by a questionnaire, which will be discussed in more detail later on in the evaluation part.
  == 2.3 Tasks ==
@@ -45,11 +45,11 @@
  The participant is expected to experiment according to the following plan:
  * Step 1: The participant needs to fill in the consent form
--* Step 2: The participant needs to read the context information en emphasise as Bob
++* Step 2: The participant needs to read the context information and step into the shoes of Bob
  * Step 3:The participant needs to stand close enough to the robot to have an interaction/conversation with Pepper
  * Step 4: The participant needs to look the robot in the eye
  * Step 5: The participant needs to actively answer the questions provided by Pepper during the experiment
--* Step 6: When the participant has been motivated to go on a walk he/she needs to walk for 1 to 2 meters with the robot
++* Step 6: When the participant has been motivated to go on a walk he/she needs to walk for 4 meters with the robot
  * Step 7: During the walk, the participant is expected to contribute in some small talk with Pepper
  * Step 8: When the walk is finished the participant needs to communicate with Pepper after which Pepper will say goodbye and the actual experiment is over
  * Step 9: After the experiment, the participant needs to fill in the questionnaire provided by Group 4
@@ -60,7 +60,7 @@
  In an optimal scenario where we can test the robot on PwD. We would have measured the number of times a person went out. We would also have measured the effect of the goal and emotion-based motivation on the long-term over the people. Whether it will be less effective over time or not. We would also measure the emotional effect on the caregivers and the functional effect. By the functional effect, we mean whether they indeed have more time to do other tasks or not. It would also have been perfect if we could measure the effect of the walks on the PwD and their health.
--The questionnaire for the feedback is in the attachment (Questionnaire (2)). The questionnaire is based on a questionnaire in the paper " Measuring acceptance of an assistive social robot: a suggested toolkit " [2]. There are also 5 questions at the end that we added ourselves because we think it fits our experiment.
++The questionnaire for the feedback is in the attachment [[Questionnaire (2).pdf>>attach:Questionnaire (2).pdf]]. The questionnaire is based on a questionnaire in the paper " Measuring acceptance of an assistive social robot: a suggested toolkit " [2]. There are also 5 questions at the end that we added ourselves because we think it fits our experiment.
  The questionnaire measures the experiment of the interaction of the students with the robot. By that we mean it measures:
@@ -79,17 +79,17 @@
  == 2.5 Procedure ==
--The procedure is as follows: we want the to test the claims mentioned above in the introduction. Therefore we programmed two routes in Choreography: one for the emotion-based motivation test and one for the goal-based motivation. To focus on only these two types of motivations, everything else in the route was kept the same. This is also to make sure that nothing else besides the motivation influences the participant's opinion on taking the walk.
++The procedure is as follows: we want to test the claims mentioned above in the introduction. Therefore we programmed two routes in Choreography: one for the emotion-based motivation test and one for the goal-based motivation. To focus on only these two types of motivations, everything else in the route was kept the same. This is also to make sure that nothing else besides the motivation influences the participant's opinion on taking the walk.
--For the exeperiment we wrote an [[orientation script>>doc:3\. Evaluation.Scenario description.WebHome]] for the participants to introduce them to our design and explain them what they should do and that they should step into the shoes of our persona Bob. Bob is a person with anger issues and dementia. However, the participants did not know whether they are tested with emotion-based or goal-based motivation walk. We also wrote a consent form to ask for their consent to take part in the experiment. One of the main points in the consent from is that they will be recorded. We wanted to record them to re-evaluate all the experiments and see if we missed something. This also helped us with the final results and the discussion. If the participant did not agree, then we of course did not record him/her.
++For the experiment, we wrote an [[orientation script>>doc:3\. Evaluation.Scenario description.WebHome]] for the participants to introduce them to our design and explain to them what they should do and that they should step into the shoes of our persona Bob. Bob is a person with anger issues and dementia. However, the participants did not know whether they are tested with an emotion-based or goal-based motivation walk. We also wrote a consent form to ask for their consent to take part in the experiment. One of the main points in the consent form is that they will be recorded. We wanted to record them to re-evaluate all the experiments and see if we missed something. This also helped us with the final results and the discussion. If the participant did not agree, then we of course did not record him/her.
--The following happend during an experiment:
++The following happened during an experiment:
  ~1. Pepper will be turned on and will scan/check his environment
--2. Pepper will look for a face and will turn to the person that he sees
++2. Pepper will look straight up, as if it is looking at the participant's face.
--3. Pepper will recognize a face and will introduce itself to the student
++3. Pepper will say hi to the participant.
 . Pepper will then ask to go for a walk after which the student can either say yes or no
@@ -99,9 +99,8 @@
 b. When the student says no, Pepper will start the motivational part of the experiment. For the first experiment Pepper will use emotion-based                       motivation and for the second experiment Pepper will use the goal-based motivation
 c. If the student then decides to say yes, then Pepper will start walking with the student and during the walk will have some small talk
--During the experiment, one of us wrote down observations of the experiment and another one recoreded the experiment if allowed. We had also prepared a questionnaire to measure our claims, which we talked about in details in the measures section. All the participants had to fill in these questionnaire after the experiment. We wanted to make sure that we had an equal amount of both types of tests to get an unbiased result. Hence, we finished the evaluation once we had a relatively good and equal amount of experiments.
++During the experiment, one of us wrote down observations of the experiment and another one recorded the experiment if allowed. We also prepared a questionnaire to measure our claims, which we talked about in detail in the measures section. All the participants had to fill in this questionnaire after the experiment. We wanted to make sure that we had an equal amount of both types of tests to get an unbiased result. Hence, we finished the evaluation once we had a relatively good and equal amount of experiments.
--
  == 2.6 Material ==
  The material needed for this experiment is of course the Pepper robot. We also need a laptop to run the robot.
@@ -140,22 +140,21 @@
  In a couple of the evaluations, it happened that the robot cut participants off mid-sentence once it had recognized a word that was spoken if they were speaking slower or elaborating on their answers. This is not ideal for a future and complete design and definitely would be something that needs to be worked on.
--
  = 4. Discussion =
  In terms of the research question, no significant differences were found between the two different motivational methods. This is however very likely influenced by the circumstances surrounding the design and the evaluation. The design is unfortunately rather limited and with limited capabilities, due to time constraints. Speech recognition didn't always work properly and was not as flexible as desired which made the interactions less realistic for the participant. Things such as elaborating, which would be something that would be a natural part of a leisurely conversation were made very difficult as the robot could not comprehend conversation to the fullest extent.  Since participants were also prompted to give shorter answers and try to keep to things like "yes" and "no" it greatly influenced how participants interacted with the robot.
--There were also other constraints to the interaction, which had to be given as instructions to the participant before testing. These things included at what distance to stay from the robot, when to join the robot's side when it's time for the walk, how long to wait to speak after a certain prompt, etc. This further made it unnatural but was necessary for the system to perform properly. Ideally, an individual would be able to join the robot's side at any given moment and the robot's movement would not be impacted by the fact that the participant stands too close. Further, one component that has a very significant effect on the results was that it was not possible to test the design with PwD. This was attempted to be resolved by providing a persona description for participants to keep in mind during the testing, but it is difficult to simulate conditions of dementia. Only one participant ended up embodying this character to the fullest extent which was very valuable for the sake of the evaluation but was not enough to explore the concept entirely.
++There were also other constraints to the interaction, which had to be given as instructions to the participant before testing. These things included at what distance to stay from the robot, when to join the robot's side when it's time for the walk, how long to wait to speak after a certain prompt, and when to repeat themselves. This further made it unnatural but was necessary for the system to perform properly. Ideally, an individual would be able to join the robot's side at any given moment and the robot's movement would not be impacted by the fact that the participant stands too close. Further, one component that has a very significant effect on the results was that it was not possible to test the design with PwD. This was attempted to be resolved by providing a persona description for participants to keep in mind during the testing, but it is difficult to simulate conditions of dementia.
--This highlights the fact that the results may have been different if participants outside of the course were used since we are all very familiar with these robots and systems. On one hand, it could be positive, since we have all researched dementia and have gained a lot of knowledge within this which could make us better at simulating appropriate behavior with the robot or testing the systems in a reasonable way. But since participants also have an idea of how the robot works prior to the evaluation, based on their own experiences of working with the robot, perhaps some mistakes or issues went undetected. For example, a completely inexperienced user could potentially show other faults in the design that appear only if the system is entirely foreign to the user, which is likely what it would be like with a PwD. Of course, knowing about dementia is not the same thing as actually suffering from the diagnosis, so many aspects have most likely gone undetected there for that reason also.
++This highlights the fact that the results may have been different if participants outside of the course were used since we are all very familiar with these robots and systems. On one hand, it could be positive, since we have all researched dementia and have gained a lot of knowledge within this which could make us better at simulating appropriate behaviour with the robot or testing the systems in a reasonable way. But since participants also have an idea of how the robot works prior to the evaluation, based on their own experiences of working with the robot, perhaps some mistakes or issues went undetected. For example, a completely inexperienced user could potentially show other faults in the design that appear only if the system is entirely foreign to the user, which is likely what it would be like with a PwD. Of course, knowing about dementia is not the same thing as actually suffering from the diagnosis, so many aspects have most likely gone undetected there for that reason also.
  The results could also be influenced by the sheer amount of participants, which concluded at 8 participants per group (8 for the goal-oriented approach, and 8 for the emotional approach). Perhaps with more participants, the results would differ to a greater extent between the two approaches. Due to time constraints, it was not possible to include more participants in this particular study. Further, participants who started the interaction with a pre-disposed idea of what they wanted to do, like the participant mentioned above in the "Observations" section, definitely influenced the outcome, since this was no longer about listening to the prompts the robot was giving but more so acting according to a pre-disposed agenda.
--It is also highly interesting to consider if participants are perhaps inclined to be positive in general, particularly because the users are other students of the course who tend to want to stay positive towards their peers and therefore feel inclined to reply positively or give positive feedback to the study overall. This could cloud the results, while it is still understandable behavior given the context.
++It is also highly interesting to consider if participants are perhaps inclined to be positive in general, particularly because the users are other students of the course who tend to want to stay positive towards their peers and therefore feel inclined to reply positively or give positive feedback to the study overall. This could cloud the results, while it is still understandable behaviour given the context.
  A rather central aspect is also of course that the robot should really take a walk outside and not inside the lab room. Preferably, the test should have been performed in an actual garden in order to be able to assess its capabilities in the appropriate terrain. This would also make it possible to make the walk more elaborate and longer since observations during the evaluation show that participants would have rather had a longer and more extensive walk, which was not possible in the lab environment.
--In future studies, the number of participants should be considered, as well as testing the design on PwD and in an actual garden or at the very least a bigger space. The walk should preferably be more extensive and perhaps incorporate aspects of the garden or the environment into the conversation to make the experience more immersing, for example by referring to the flowers that are blooming in the garden and trying to draw PwD's attention to these aspects and create conversations from this. Hopefully, adjustments like these would improve the overall quality of the walk. Further improvements to speech recognition are needed, as well as the smoothness of the walking and the aspect of the participant's distance to the robot. Perhaps if the less realistic aspects discussed above are minimized, a robot that feels more realistic would result in participants listening to the actual prompts given to a larger extent, rather than going into the experiment with a predisposed idea of what they are going to do or answer and would also perhaps deter the participants from tending to reply positively. Further, the motivational prompts were certainly customized to the persona, but further customization could have been considered. For example, it would of course be far greater if the robot has some ability to adapt to the conversation more or less "in real time" by taking in the information given by the PwD and replying in an appropriate manner. Further, intonation could be interpreted and perhaps also shape the responses and prompts of the robot. These things are rather difficult and due to limitations in the current hardware and software may be hard to implement, but it is essential to consider these aspects for future work.
++In future studies, the number of participants should be considered, as well as testing the design on PwD and in an actual garden or at the very least a bigger space. The walk should preferably be more extensive and perhaps incorporate aspects of the garden or the environment into the conversation to make the experience more immersing, for example by referring to the flowers that are blooming in the garden and trying to draw PwD's attention to these aspects and create conversations from this. Hopefully, adjustments like these would improve the overall quality of the walk. Further improvements to speech recognition are needed, as well as the smoothness of the walking and the aspect of the participant's distance to the robot. Perhaps if the less realistic aspects discussed above are minimized, a robot that feels more realistic would result in participants listening to the actual prompts given to a larger extent, rather than going into the experiment with a predisposed idea of what they are going to do or answer and would also perhaps deter the participants from tending to reply positively. Further, the motivational prompts were certainly customized to the persona, but further customization could have been considered. For example, it would of course be far greater if the robot has some ability to adapt to the conversation more or less "in real time" by taking in the information given by the PwD and replying in an appropriate manner. Further, intonation could be interpreted and perhaps also shape the responses and prompts of the robot. These things are rather difficult and due to limitations in the current hardware and software may be hard to implement, but it is essential to consider these aspects for future work. Finally, implementing route verification using GPS and adding safety monitoring features, could significantly broaden the applicability of the robot system.
  = 5. Conclusions =

Changes for page b. Test

Summary

Details

Navigation

Need help?