Project homeb. Test
1. Introduction
The goal of the conversational agent is to assist stakeholders The claims that we will test are if a conversational agent increases the deliberative quality between stakeholders. While the research is scarce in this field, we hypothesise that the conversational agent will improve value reflection. Kocielnik et al. (2018) and Zhang (2023) which are both focused on reflection by way of a conversational agent and using AI Machine learning models respectively showed that interactions with AI-systems improve reflection by the participants.
We also claim that the conversational agent will help stimulate deliberation as was found in the paper by Zhang (2023).
The goal of the evaluation is that we want to determine if the conversational agent is able to be used for the different stakeholder groups. So, we find usability very important. We want to identify the features in the design of the conversational agent that are working and those that are not to allow for a better system in the future. We also want to compare the design choices to help us to make decisions. We observe the effects of a system on users.
2. Method
Evaluation
The conversational agent will be used in a formative evaluation and in a set of workshop settings. The formative evaluation asked questions of the participants of their interaction with the agent as well as the usability of the agent. Lastly, we asked our participants if they were able to reflect on their values and engage in a deliberative discussion post-participation.
Research Questions
The research questions that we are considered with are as follows:
- How do participants respond to the conversational agent?
- Are the participants able to use the conversational agent?
- Do they understand the goal of the conversational agent?
- Do the participants trust the conversational agent?
- Does the conversational agent help value reflection?
- Does the conversational agent aid the participants in value deliberation with other stakeholders?
2.1 Participants
From existing research by Lee et al., (2019) where they studied the use of a caring conversational agent and had 14 participants, and Peeters et al., (2016) which had 10 participants, we are conducting two workshops with a sample size of 10 people in each to study our conversational agent. As the goal of our conversational agent is to allow various stakeholders to reflect on their values, we are conducting a participant-based evaluation rather than an expert-based evaluation.
2.2 Experimental design
The experiment was designed to take place like a workshop.
Each of the participants entered the room behind a computer where the conversational agent was set up. They were explained the procedure and the role of the conversational agent. They were told that they would be free to leave the room at any time if they felt uncomfortable and that their data would be anonymised and only used as part of the research. Furthermore, the participants were told that they could ask questions whenever they wanted. They were also told that if anything was not to their liking they could ask the researcher. The goal of the experiment is to elicit value reflection from the participants.
The participants are told that following each interaction they will be given a value and the researchers will examine if their values change over the two interactions and if the values themselves allow reflection between the participants.
Taking the example from Kocielnik and colleague’s research on workplace reflection based on a chat and a voice-based conversational agent, we asked 10 questions to allow the participants to reflect on how they felt about each interaction (2018).
2.3 Tasks
The tasks of the experiment are reflection and deliberation. Through interacting with the conversational agent, the participants will be asked to reflect on their values. Then after they have reflected on their values, the researchers will ask the participants to participate in a deliberation amongst their fellow stakeholders about a subject regarding public safety.
2.4 Measures
We will measure the experiment by asking the participants to take part in various questionnaires. There will be a system usability scale questionnaire which is focused on the usability of the conversational agent. This will include 10 questions and will be based on a 5-point likert scale. There will also be a questionnaire which is created by the researchers that is solely focused on the question of values. The question regarding values will be asked after the interaction with the conversational agent and after the deliberation with fellow participants has taken place.
2.5 Procedure
Reflection
As explained in the scenario and the prototype, as the participants sit down, they are welcomed by the agent and then they are faced with a number of open questions from the conversational agent.
Interaction 1
Examples of these questions include:
1. What does public safety make you think of?
2. How do you feel about safety in general?
3. Do you feel safe on trains in the Netherlands?
These questions are aimed at helping the user think about the their values and reflect upon them so they are able to properly explicate their values.
Once they’ve finished answering the questions in interaction 1, they are given a value that the conversational agent found most reflects them; for example, freedom.
Then they are told that they will next be presented with a scenario in which they will be shown a crowd control scenario that is a protest scenario. After they are presented with the scenario, they will be asked a number of open questions of the conversational agent again regarding the scenario they watched and they will be presented with a value once again.
Interaction 2
The participants are presented with a scenario in which they see a protest play out.
After the scenario, they are asked sample questions. Some of these are:
- How did the scenario make you feel?
- Do you think the protestors were right?
- Do you think the police were right to act this way?
- How would you have acted in the case of the protestors?
- How would you have acted in the case of the police?
After the second interaction, the participants are once again given a value that the conversational agent feels most reflects them, for example, security.
This ends the experiment which includes the conversational agent.
Deliberation
In the following part of the experiment, the participants are all placed in a room and asked questions about public safety and the scenario that they were faced in to facilitate value deliberation. While a goal of deliberation is to reach a consensus, we recognise that they may not always be feasible. So through our deliberation, we want the participants to get an understanding of each other’s values. We want them to be able to express those feelings in a civil manner (Kasirzadeh & Gabriel, 2023; Zhang et al., 2023).
This ends the procedure.
2.6 Material
The material used is simple. As there will be computers in the room and then a researcher asking the participants questions as they sit in a circle.
References
1. Kasirzadeh, A., & Gabriel, I. (2023). In conversation with Artificial Intelligence: aligning language models with human values. Philosophy & Technology, 36(2), 1-24.
2. Kocielnik, R., Avrahami, D., Marlow, J., Lu, D., & Hsieh, G. (2018, June). Designing for workplace reflection: a chat and voice-based conversational agent. In Proceedings of the 2018 designing interactive systems conference (pp. 881-894).
3. Lee, M., Ackermans, S., Van As, N., Chang, H., Lucas, E., & IJsselsteijn, W. (2019, May). Caring for Vincent: a chatbot for self-compassion. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (pp. 1-13).
4. Peeters, M. M., Harbers, M., & Neerincx, M. A. (2016). Designing a personal music assistant that enhances the social, cognitive, and affective experiences of people with dementia. Computers in Human Behavior, 63, 727-737.
5. Zhang, A., Walker, O., Nguyen, K., Dai, J., Chen, A., & Lee, M. K. (2023). Deliberating with AI: Improving Decision-Making for the Future through Participatory AI Design and Stakeholder Deliberation. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), 1-32.