Statistics for Decision Makers - 06.04 - Research Design - Experimental Design
Goals。
- There are many ways an experiment can be designed.
- For example, subjects can all be tested under each of the treatment conditions or a different group of subjects can be used for each treatment.
- An experiment might have just one independent variable or it might have several.
- The goal is to be able to describe basic experimental designs and their advantages, disadvantages and workarounds.
Between-Subjects Designs。
- Similar to A/B design
- The various experimental treatments are given to different groups of subjects.
- Other names
Between-group Design
Web Interface Ratings Example。
- The subjects were randomly divided into two groups.
- The first group used the web application with the first interface design. The second group used the second interface design.
- The independent variable is "Condition" and has two levels (Design 1 and Design 2).
- It is a between-subjects variable because different subjects were used for the two levels of the independent variable: the "Design 1" or the "Design 2" condition.
- The comparison of the Design 1 condition with the Design 2 condition is a comparison between the subjects in one condition with the subjects in the other condition.
Web Interface Ratings Example - chance differences 。
- The two conditions were treated exactly the same except for the interface design.
- Any differences between conditions should be attributed to the interface designs themselves.
- The possibility of chance differences between the groups is ignored.
- That is, by chance, the raters in one condition might have, on average, been more lenient than the raters in the other condition.
- Randomly assigning subjects to treatments:
- ensures that all differences between conditions are chance differences;
- it does not ensure there will be no differences.
Web Interface Ratings Example - estimating chance differences 。
How to distinguish real differences from chance differences?
- Analyzing the data from this experiment reveals that the ratings in the Design 1 condition were higher than those in the Design 2 condition.
- Using inferential statistics, it can be calculated that the probability of finding a difference as large or larger than the one obtained if the treatment had no effect is only 0.018.
- Therefore it seems likely that the design had an effect and it is not the case that all differences were chance differences.
Independent variables。
Independent variables often have several levels
- Example
- A company tries to determine whether the colour of the logo changes the perception of the trustworthiness of the company.
- The independent variable is "colour" and there are four levels of this independent variable:
|
|
1. blue 2. red 3. green 4. yellow |
- Keep in mind that although there are four levels, there is only one independent variable.
- Designs with more than one independent variable are considered next.
Multi-Factor Between-Subject Designs。
- In the "Bias Against Associates of the Obese" experiment, the qualifications of potential job applicants were judged.
- Each applicant was accompanied by an associate.
- The experiment had two independent variables:
- The weight of the associate (obese or average).
- The applicant's relationship to the associate (girlfriend or acquaintance).
Multi-Factor Between-Subject Designs。
This design can be described as a
Weight (2) x Relationship (2) factorial design
- The dependent variable was a rating of the applicant's qualifications (on a 9-point scale).
- Alternatives
- If two separate experiments had been conducted to test:
- The effect of Associate's Weight and
- The effect of Associate's Relationship
- Then there would be no way to assess whether the effect of Associate's Weight depended on the Associate's Relationship.
- Interactions
- One might imagine that the Associate's Weight would have a larger effect if the associate were a girlfriend rather than merely an acquaintance.
- A factorial design allows this question to be addressed.
- When the effect of one variable does differ depending on the level of the other variable then it is said that there is an interaction between the variables.
Within-Subjects Designs。
- A within-subjects design differs from a between-subjects design in that the same subjects perform at all levels of the independent variable.
- AKA: repeated-measures designs
Counterbalancing。
- In a within-subject design it is important not to confound the order in which a task is performed with the experimental treatment.
- For example, people presented with UI Design 2 tend to asses it worse when presented with UI Design 1 first.
| First test | Second test | |
|---|---|---|
| Subject1 | Test A | Test B |
| Subject2 | Test B | Test A |
| Subject3 | Test A | Test B |
| Subject4 | Test B | Test A |
| ... | ... | ... |
Counterbalancing。
- Counterbalancing is not a satisfactory solution if there are complex dependencies between:
- which treatment precedes which
- the dependent variable.
- In these cases, it is usually better to use a between-subjects design than a within-subjects design.
Advantage of Within-Subjects Designs。
- Individual differences in subjects' overall levels of performance are controlled.
- Subjects will invariably differ greatly from one another.
- In an experiment on problem solving, some subjects will be better than others regardless of the condition they are in.
- Within-subjects designs control these individual differences by comparing the scores of a subject in one condition to the scores of the same subject in other conditions.
- Each subject serves as his or her own "control".
- Power
- Typically within-subjects designs are more powerful than between-subjects designs.
- Within-subjects designs are more able to detect an effect than are between-subjects designs.
Alternative Names of Within-Subjects Designs
- Within-subject designs are often called repeated-measures designs since repeated measurements are taken for each subject.
- Similarly, a within-subject variable can be called a repeated-measures factor.
Complex Designs。
Designs can contain combinations of between-subject and within-subject variables.
- Example
- Between-subject variable:
- Gender.
- Within-subject variables:
- The type of user interface design.
- The type of graphic design.
Control Group。
- A statistician's wife had twins.
- The statistician rang the minister
- "Bring them to church on Sunday and we'll baptize them," said the minister.
- "No," replied the statistician. "Baptize one. We'll keep the other as a control."
Control Group。
- If A occurs than B occurs (Treatment Group).
- If A doesn't occur than B doesn't occur (Control Group).
- Example
- Half of the people visiting the website will use the old user interface design, half will use the new one.
- This will prevent speculation that the time spent on the website would increase even if we did not change the user interface.
Quiz。
Quiz
<quiz display=simple>
{Fifty subjects are each tested in both a control condition and an experimental condition. This is an example of:
|type="()"} - A between-subjects design. + A within-subjects variable.
{
Answer >>
A within-subjects variable
This is a within-subjects design since each subject was tested in each condition.
}
{The time it took each of 20 subjects to name a set of coloured squares and to read a set of colour names was recorded. This is an example of:
|type="()"} - A between-subjects design. + A within-subjects variable.
{
Answer >>
A within-subjects variable
This is a within-subjects design since each subject was tested in each condition.
}
{Subjects are randomly assigned to either a drug condition or a placebo condition. This is an example of:
|type="()"} + A between-subjects design. - A within-subjects variable.
{
Answer >>
A between-subjects design
This is a between-subjects design because each subject was tested in either one condition (drug) or another (placebo).
}
{For which of the following is there an interaction?
|type="[]"} + A: The effect of the treatment was larger for women than for men. - B: For both women and men the treatment had a large effect. + C: The combination of smoking and drinking increases the chances of throat cancer more than one would expect from the effects of smoking and drinking considered alone.
{
Answer >>
A and C
}
