Threats to Internal Validity
From EvaluationWiki
Often, a primary purpose of research and/or evaluation studies is to attribute observed effects to the treatment or program. In other words, researchers want to establish a causal or correlational link between the program and outcomes. However, the program is not the only possible influence on program outcomes.
In order to logically attribute observed effects to the program or treatment, it is necessary compare experimental results to results from a controlled setting. In laboratory settings, it is possible to meet this need through tightly controlled comparison groups. In evaluation studies of educational or social programs involving human subjects, it is often not possible to establish rigorous experimental controls for a variety of ethical and practical reasons. In these cases, less rigorously controlled comparisons are possible but may be subject to various threats to internal and external validity.
An uncontrolled threat to internal validity means that a plausible, alternative explanation of program effects exists. Any conclusions drawn from the study could be challenged based on the threat. The following are the most commonly referenced threats to internal validity:
- Selection
- Mortality
- Interaction
- History
- Maturation
- Repeated Testing
- Instrumentation
- Regression to the Mean
- Selection-Maturation Interaction
- Experimenter Bias
- Testing
Contents |
Selection
The participants in groups may be unlike in some way, so they will respond in different ways to the independent variable. This is mostly a risk for quasi-experimental designs, in which non-random assignment is used.
Mortality
Participants drop put of the test, making the groups unequivalent. Also, who drops out and why? (Often it is the people who did most poorly on the test to begin with.)
Interaction
Two or more threats can interact. For example, a Selection-Maturation interaction: difference between ages of groups could cause groups to change at different rates. A group of young people may show more improvement in a test than a group of older people, but that could be because their brains are developing faster relative to their age.
History
Outside events occurring during the course of the experiment or between repeated measures of the dependent variable may have an influence on the results. This does not make the test itself any less accurate.
Maturation
Change due to aging or development, either between or within groups.
Repeated Testing
Instrumentation
The reliability of the instrument may change in calibration (if using a measuring device) or from change in human ability to measure differences (due to fatigue, experience, etc).
Regression to the Mean
Tendency to regress toward the mean makes scores higher or lower. If a measure is not extremely reliable, there will be some variation between repeated measures. The chances are that the measurements will move towards a mean instead of towards extremes. Changes in outcomes occurring when groups or some members are selected on the basis of their extreme scores (the "extreme" group will regress towards the mean, whether it has benefited from the program or not);
Selection-Maturation Interaction
Experimenter Bias
Expectations of an outcome may inadvertently influence participant or cause the experimenter to view data in a different way.
Testing
Experience of taking test has an influence on results. Experience refers either to mental or physical changes—a participant’s attitude towards a topic may change because of a survey, which could affect results, or a participant’s physiological response to a test may change after repeated measures.
Placebo Effect
Improvement due to expectation rather than the treatment itself; can occur when participants receive a treatment that they consider likely to be beneficial.
Hawthorne Effect
When members of the treatment group change in terms of the dependent variable because their participation in the study makes them feel special—so they act differently, regardless of the treatment.
Contamination
When the comparison group is in some way affected by, or affects, the treatment group, causing an increase of efforts. Also known as compensatory rivalry or the John Henry effect.
Diffusion of treatment
Respondents in one group become aware of the information intended and intended practices meant for the other group.
