Wiki source code of Measuring Instruments
Version 4.2 by Frank Broz on 2022/03/22 08:33
Show last authors
| author | version | line-number | content |
|---|---|---|---|
| 1 | |||
| 2 | = Frameworks = | ||
| 3 | |||
| 4 | == DECIDE == | ||
| 5 | |||
| 6 | **D**etermine the goals | ||
| 7 | - What are the high-level goals? Who wants it and why? The goals influence the approach of the study | ||
| 8 | |||
| 9 | **E**xplore the questions | ||
| 10 | - Define the goals and reseach questions | ||
| 11 | |||
| 12 | **C**hoose evaluation approach and methods | ||
| 13 | - Influences the data collection, analysis and presentation | ||
| 14 | - Exploratory research => qualitative data, observational study, | ||
| 15 | |||
| 16 | **I**dentify practical issues | ||
| 17 | - Users, budget, schedule, equipment etc. | ||
| 18 | - Pilot study important | ||
| 19 | |||
| 20 | **D**ecide about ethical issues | ||
| 21 | - Adhere to ethical procedure | ||
| 22 | - User rights! Explain the goals (before/after), methodology etc. | ||
| 23 | |||
| 24 | **E**valuate, analyze, interpret, present data | ||
| 25 | - Reliable results: Can the results be replicated? | ||
| 26 | - Validity: Is the data related to the hypothesis? | ||
| 27 | - Bias: Are the results unbiased? | ||
| 28 | - Scope: Can this be generalised? | ||
| 29 | - Ecological validity: Is the environment influencing the results? | ||
| 30 | |||
| 31 | == IMPACT == | ||
| 32 | |||
| 33 | **I**ntention: Present the objectives and claims | ||
| 34 | |||
| 35 | **M**easures and metrics: "What, how and why" | ||
| 36 | |||
| 37 | **P**eople: Define the participants | ||
| 38 | |||
| 39 | **A**ctivities: Use cases into activities | ||
| 40 | |||
| 41 | **C**ontext: Social, ethical, physical, etc. environment definition | ||
| 42 | |||
| 43 | **T**echnologies: Hardware and software | ||
| 44 | |||
| 45 | |||
| 46 | = Evaluation methods = | ||
| 47 | |||
| 48 | == Formative evaluation == | ||
| 49 | |||
| 50 | - Open ended evaluation on the design | ||
| 51 | - E.g. How will the users respond to the new design? | ||
| 52 | |||
| 53 | |||
| 54 | == Summative evaluation == | ||
| 55 | |||
| 56 | - Focus on the overall effect | ||
| 57 | - Summarizes if the objective is reached | ||
| 58 | - E.g. Are the participants happier when working with design X in comparison to design Y? | ||
| 59 | |||
| 60 | == Data == | ||
| 61 | |||
| 62 | - Qualitative: Explore, discover, instruct | ||
| 63 | |||
| 64 | - Quantitative: Describe, explain, predict | ||
| 65 | |||
| 66 | - Subjective quantitative | ||
| 67 | |||
| 68 | |||
| 69 | == Statistics == | ||
| 70 | |||
| 71 | - Descriptive: Describe the dataset, e.g. mean time on task | ||
| 72 | - Inferential: Using a sample to infer about a population, e.g. predicted mean time on task based on user characteristics. | ||
| 73 | |||
| 74 | == Experiment Design: Conditions == | ||
| 75 | |||
| 76 | === Within Subjects (Repeated Measures) === | ||
| 77 | Each participant is subjected to all the test conditions. | ||
| 78 | Fewer subjects needed and reduces variance in the Can be difficult to setup due to subjects fatiguing, learning about the setup or simply not having enough time. | ||
| 79 | |||
| 80 | === Between Subjects (Between Groups) === | ||
| 81 | One subject undergoes only one test. Simple to execute, but results in significant variance due to inter-subject differences in characteristics. | ||
| 82 | |||
| 83 | |||
| 84 | == Lenses == | ||
| 85 | - Lense means to take different perspectives looking at your system | ||
| 86 | - E.g. perspective of the stakeholders, other groups, or technical/legal | ||
| 87 | |||
| 88 | |||
| 89 |