Wiki source code of b. Test

Version 11.1 by Karthik Prakash on 2023/04/07 20:06

version	line-number	content
1.1	1	= 1. Introduction =
	2
8.1	3	In section [[a. Prototype>>3\. Evaluation.a\. Prototype.WebHome]] two versions of the robot were presented, one with voice functionality and one without.
1.1	4
2.2	5	The main claims we are looking to test with this testing procedure are related to the functionality and usability of the robot.
	6
8.1	7	The participants will be other students taking this course. The participants will be placed in the shoes of a PwD and be tasked with completing several basic actions with the robot while impaired in several known ways to simulate the difficulties of a PwD.
2.2	8
8.1	9	After the experiment, the participants will fill out a survey and be asked some more open-ended questions with the purpose of understanding how the interaction with the robot went, and whether they have anything that they find concerning regarding the possible use of the system and its functions in a real-life setting.
2.2	10
8.1	11	On top of this, a short questionnaire will be sent to several care homes throughout the Netherlands in hopes to get a general idea of whether the caretakers at the facilities think that the system would be a good fit for the proposed use case.
2.2	12
1.1	13	= 2. Method =
	14
8.1	15	The prototypes are evaluated in a simulated manner and conducted in-person experiments. Participants will be given a [[persona>>doc:Main.sdf.Persona Scenarios.WebHome]] to play act.
1.1	16
	17	== 2.1 Participants ==
	18
8.1	19	All students in CS4235 Socio-Cognitive Engineering (2022-2023) in TU Delft are invited to test the robot. In the end, 14 students are presented.
1.1	20
	21	== 2.2 Experimental design ==
	22
8.1	23	Prior to the experiment, participants were asked to simulate the experience of a person with dementia. Participants who wore glasses were asked to remove them, while those who did not were provided with short-sighted or far-sighted glasses to replicate the blurred vision and degraded perception that is common in PwDs. Additionally, their index and middle fingers were taped together to simulate the difficulty in controlling movements that many PwDs experience.
1.1	24
8.1	25	For this experiment, we used a between-subject design. The control group interacted solely with the stand-alone application, which represented the robot without voice functionality, and received guidance through a single task paper. In contrast, the experimental group engaged with the fully-functional robot, which provided audio instructions, guidance, and encouragement.
	26
1.1	27	== 2.3 Tasks ==
	28
2.1	29	In the user test, the following tasks were asked of the participants:
1.1	30
2.1	31	==== Reminders for activities ====
	32
	33	* (((
10.1	34	Add a reminder that a relative will pay a visit on Sunday with the format as "<relative name> will visit you on Sunday at 3 pm for some tea". Set the reminder to remind you 10 min before that.
2.1	35	)))
	36	* (((
10.1	37	Check the remainder for this week and verify the new remainder added for the event.
2.1	38	)))
	39
	40	==== Personal profile ====
	41
	42	* (((
10.1	43	Browse through relative profiles and read the info.
	44	)))
	45	* (((
2.1	46	Add relatives as a contact in the "profile" section.
	47	)))
	48
	49	==== Memory games ====
	50
	51	* (((
8.1	52	Go to the Games section and play the game.
2.1	53	)))
	54
	55	==== Medicine reminders ====
	56
10.1	57	(for professional caregivers - not part of the evaluation)
2.1	58
	59	* (((
	60	In the section “My Health”, add a medicine reminder to take the medicine Donepezil, 1 time per day at 9 PM before going to bed.
	61	)))
	62	* (((
	63	Check medicines that have been added.
	64	)))
	65	* (((
	66	Delete medicines that have been added.
	67	)))
	68
1.1	69	== 2.4 Measures ==
	70
10.1	71	Two quantitative measures were employed in the user evaluation. The first measure aimed to test attributes including accessibility, trustworthiness, and comprehensibility. The second measure used was the System Usability Scale (SUS), which is a widely-used scale for evaluating the usability of the software.
1.1	72
11.1	73	In addition to the quantitative measures, a structured interview with open-ended questions was conducted with randomly selected participants. The aim of this interview was to gain a deeper understanding of participants' experiences with the robot, including any concerns they may have regarding the potential use of the system and its functions in a real-life setting.
4.1	74
8.1	75	By using both quantitative and qualitative measures, the user evaluation can provide a more comprehensive understanding of participants' experiences and perceptions of the system. This approach allows for a more nuanced analysis of the data and can provide valuable insights into the strengths and weaknesses of the system, as well as areas for improvement.
4.1	76
1.1	77	== 2.5 Procedure ==
	78
2.1	79	The procedure was conducted as follows:
	80
	81	1. Welcome participants and give an introduction.
	82	1. Get them to sign a consent form.
8.1	83	1. Prepare them to pretend to be a person with dementia.
2.1	84	1. Have interaction with the robot and complete the tasks.
	85	1. Complete a questionnaire.
8.1	86	1. Have a short interview with randomly selected participants.
2.1	87
1.1	88	== 2.6 Material ==
	89
2.2	90	1. Consent form. To protect the privacy of participants and ensure the evaluation process goes smoothly, we will ask participants to sign a consent form, indicating they are willing to take part in the evaluation and the data gathered from the experiment will be analyzed by researchers.
9.1	91	1. [[Pepper robot>>doc:Main.c\. Technology.Humanoid Robot.WebHome]].
	92	1. [[Questionnaire>>doc:Main.b\. Human Factors.Measuring Instruments.Questionnaire\: SUS.WebHome]]. Validated questions to test the functionality and usability.
1.1	93
	94	= 3. Results =
	95
7.1	96	\|=(% style="width: 199px;" %)Tasks\|=(% style="width: 147px;" %)Succeded by Themselves\|=(% style="width: 146px;" %)Succeded with Some Guidance\|=(% style="width: 185px;" %)Succeded with Detailed Explicit Instructions\|=(% style="width: 175px;" %)Average Time to Complete Task (s)
11.1	97	\|(% style="width:199px" %)Add a reminder\|(% style="width:147px" %)14.29%\|(% style="width:146px" %)28.57%\|(% style="width:185px" %)57.14%\|(% style="width:175px" %)75
	98	\|(% style="width:199px" %)Check weekly remainders on the Calendar page\|(% style="width:147px" %)100%\|(% style="width:146px" %)0%\|(% style="width:185px" %)0%\|(% style="width:175px" %)10
	99	\|(% style="width:199px" %)Create a personal profile\|(% style="width:147px" %)7.14%\|(% style="width:146px" %)50%\|(% style="width:185px" %)42.86%\|(% style="width:175px" %)43
	100	\|(% style="width:199px" %)Verify current profiles\|(% style="width:147px" %)85.71%\|(% style="width:146px" %)14.29%\|(% style="width:185px" %)0%\|(% style="width:175px" %)10
	101	\|(% style="width:199px" %)Play memory game\|(% style="width:147px" %)0%\|(% style="width:146px" %)42.86%\|(% style="width:185px" %)57.14%\|(% style="width:175px" %)200
1.1	102
7.1	103	if possible, note down some Parts Where Users Struggled in each task.
	104
1.1	105	= 4. Discussion =
	106
11.1	107	=== Limitations: ===
1.1	108
11.1	109	We ran into some hiccups while creating the application and performing the experiment:
	110
	111	==== Implementation: ====
	112
	113	* We could not adapt the robot to the PwD due to time constraints. This means that we did not take into account the severity of the PwD's visual, acoustic and kinesthetic limitations while setting up Pepper.
	114	* We could not test the full capabilities of the robot due to privacy constraints. Since we fabricated the information about relatives to protect the privacy of participants, we were not able to perform the scenarios in a realistic manner.
	115	* Since the version of the Google Chrome browser on the Pepper tablet was outdated, we were not able to load our Flutter application onto it and simulate actual scenarios.
	116
	117	==== Evaluation: ====
	118
	119	* The participants who took part in our experiment were not affected by dementia.
	120	* The experiment was not conducted in a real-world setting.
	121	* We were not able to test the improvement in memory recall since it involves sustained interactions with a particular user over time and our experiment involved brief, one-time interactions.
	122
	123
	124
	125
1.1	126	= 5. Conclusions =