Frameworks

DECIDE

Determine the goals
- What are the high-level goals? Who wants it and why? The goals influence the approach of the study

Explore the questions
- Define the goals and reseach questions

Choose evaluation approach and methods
- Influences the data collection, analysis and presentation
- Exploratory research => qualitative data, observational study, 

Identify practical issues
- Users, budget, schedule, equipment etc.
- Pilot study important

Decide about ethical issues
- Adhere to ethical procedure
- User rights! Explain the goals (before/after), methodology etc.

Evaluate, analyze, interpret, present data
- Reliable results: Can the results be replicated?
- Validity: Is the data related to the hypothesis?
- Bias: Are the results unbiased?
- Scope: Can this be generalised?
- Ecological validity: Is the environment influencing the results?

IMPACT

Intention: Present the objectives and claims

Measures and metrics: "What, how and why"

People: Define the participants

Activities: Use cases into activities

Context: Social, ethical, physical, etc. environment definition

Technologies: Hardware and software

Evaluation methods

Formative evaluation

- Open ended evaluation on the design
- E.g. How will the users respond to the new design?

Summative evaluation

- Focus on the overall effect
- Summarizes if the objective is reached
- E.g. Are the participants happier when working with design X in comparison to design Y?

Data

- Qualitative: Explore, discover, instruct

- Quantitative: Describe, explain, predict

  - Subjective quantitative

Statistics

- Descriptive: Describe the dataset, e.g. mean time on task
- Inferential: Using a sample to infer about a population, e.g. predicted mean time on task based on user characteristics.

Experiment Design: Conditions

Within Subjects (Repeated Measures)

Each participant is subjected to all the test conditions.
Fewer subjects needed and reduces variance in the Can be difficult to setup due to subjects fatiguing, learning about the setup or simply not having enough time.

Between Subjects (Between Groups)

One subject undergoes only one test. Simple to execute, but results in significant variance due to inter-subject differences in characteristics.

Lenses

- Lense means to take different perspectives looking at your system
- E.g. perspective of the stakeholders, other groups, or technical/legal