Skip to Content

Changes for page 4. Evaluation Methods

Last modified by Demi Breen on 2023/04/09 14:54
From version 3.2
edited by Mark Neerincx
on 2023/03/23 10:12
Change comment: Added comment
To version 7.1
edited by Hugo van Dijk
on 2023/04/07 17:40
Change comment: There is no comment for this version

Summary

Details

Page properties
Author
... ... @@ -1,1 +1,1 @@
1 -xwiki:XWiki.MarkNeerincx
1 +XWiki.hjpvandijk
Content
... ... @@ -1,21 +1,22 @@
1 -
1 +A within-subject designed experiment is when each participant is exposed to more than one experiment under testing. A between-subject design is when participants only do one experiment [1]. With within-subject design, a risk is the so-called 'demand effect', which entails that they might expect the researchers to want certain results, and will then act as such. Another thing that might happen with within-subject design is that participants might experience a learning effect, i.e. learning from the first experiment. [2]
2 2  
3 -Our research topic is:
4 -The effect of goal-based and emotion-based explanations in prompting PwD for physical activity.
5 -\\We will create two systems, both trying to motivate the PwD to go for a walk in the garden. One will use goal-based explanations and the other emotion-based explanations.  Maybe we would also need a control group (no explanation), resulting in three systems.
3 +Quite some established questionnaires exist regarding human-robot interaction. However, most are more about the usability of a system where the user has a specific goal. Examples of these questionnaires are SASSI [3], SUS [4], and APA [5]. Questionnaires also concerning the robot's perceived likeability and general interaction are GodSpeed [6] and a questionnaire proposed by Herink et al. [7], where the latter is more elaborate. [8] proposes the Self-Assessment Manikin (SAM), a non-verbal assessment based on pictures used to measure pleasure, arousal, and dominance as a reaction to some form of stimulation. Finally, [9] explains the AffectButton, an interface component that lets users enter the most appropriate expression by moving their mouse to the proper location.
6 6  
7 -Independent effect: explanation method
8 8  
9 -Dependent effect: motivation to go to the garden
6 +=== References ===
10 10  
11 -Confounding effects: Personal enjoyment of nature, weather,
8 +[1] Greenwald, A. G. (1976). Within-subjects designs: To use or not to use?. //Psychological Bulletin//, //83//(2), 314.
9 +[2] Seltman, H. J. (2012). Experimental design and analysis (pp. 340)
12 12  
11 +[3] Hone, K. S., & Graham, R. (2000). Towards a tool for the subjective assessment of speech system interfaces (SASSI). //Natural Language Engineering//, //6//(3-4), 287-303.
13 13  
14 -The between-subject study design fits with the limited time that we have. It also makes sure there's no learning effect like what could occur with a within-subject study. We do have to take into account the potential differences between the groups meaning we cannot take the results as a direct conclusion.
13 +[4] Lewis, J. R. (2018). The system usability scale: past, present, and future. //International Journal of Human–Computer Interaction//, //34//(7), 577-590.
14 +[5] Fitrianie, S., Bruijnes, M., Li, F., Abdulrahman, A., & Brinkman, W. P. (2022, September). The artificial-social-agent questionnaire: establishing the long and short questionnaire versions. In //Proceedings of the 22nd ACM International Conference on Intelligent Virtual Agents// (pp. 1-8).
15 15  
16 -After each evaluation session, the participant will be asked to fill in a questionnaire. There's quite some existing for human-robot interaction. However, they are more about the usability of the system. While we see our system just as a conversational motivator for going outside. So we don't see these questionnaires as fit:
16 +[6] Bartneck, C. (2023). Godspeed Questionnaire Series: Translations and Usage.
17 17  
18 -SASSI, SUS (System Usability scale), Godspeed questionnaire, ASA questionnaire, AttrakDiff, SUISQ
18 +[7] Heerink, M., Krose, B., Evers, V., & Wielinga, B. (2009, September). Measuring acceptance of an assistive social robot: a suggested toolkit. In RO-MAN 2009-The 18th IEEE International Symposium on Robot and Human Interactive Communication (pp. 528-533). IEEE.
19 19  
20 +[8] Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. //Journal of behavior therapy and experimental psychiatry//, //25//(1), 49-59.
20 20  
21 -All participants of the evaluation will be part of the course. So they will all be familiar with the robot in question. They will all be students at the TU Delft aged 20-25.
22 +[9] Broekens, J., & Brinkman, W. P. (2013). AffectButton: A method for reliable and valid affective self-report. //International Journal of Human-Computer Studies//, //71//(6), 641-667.
XWiki.XWikiComments[0]
Comment
... ... @@ -1,4 +1,4 @@
1 -In this page, you can provide background information ("Foundation") on evaluation methods, as discussed in the lecture. When relevant for you, available instruments (e.g. questionnaires on emotion, or something like the "affect button", could be briefly summarized).
1 +In this page, you can provide background information ("Foundation") on evaluation methods, as discussed in the lecture. When relevant for you, available instruments (e.g. questionnaires on emotion, or something like the "affect button", could be briefly summarized). The specific evaluation method is described in "3. Evaluation".
2 2  
3 3  Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: the self-assessment manikin and the semantic differential. //Journal of behavior therapy and experimental psychiatry//, //25//(1), 49-59.
4 4