Now let's take a deeper look into the common threats to internal validity. Familiarity with these threats will help guide you in choosing your evaluation design where the goal is to minimize such threats within the confines of your available resources. Show
Observed changes seen between observation points (ie. Pre-test and post-test) may be due to changes in the testing procedure. This could include changes to the content or the mode of administration and data collection. Tips include:
RegressionThe tendency of extreme pre-test scores to revert back toward the population mean, such that when individuals are selected for program participation based on extreme pretest results their posttest scores will tend to shift toward the mean score, regardless of the efficacy of the program. Tips include:
MaturationThis is a threat that is internal to the individual participant. It is the possibility that mental or physical changes occur within the participants themselves that could account for the evaluation results. In general, the longer the time from the beginning to the end of a program the greater the maturation threat. Tips include:
TestingThe administration of a pretest prior to the program may convey knowledge to the participants. This particular threat can either overstate or understate your program effect. Tips include:
HistoryObserved program results may be explained by events or experiences (external) that impact the individual between program participation and follow up. Tips include:
SelectionWhenever you have a nonequivalent comparison group and an exposure group, the difference seen in their posttest scores could be due to pre-existing differences between the groups rather than the impact of the program itself. This is of particular concern when the exposure group and the comparison group are significantly different from one another in characteristics. Tips include:
Interactions with Selection Threats
Other threats may come into play in the course of implementing your program evaluation design. Both randomized and non-randomized designs may be at risk as long as there is a control (randomized) or comparison (non-randomized) group:
Internal Validity Scenarios Below are examples of health program evaluations, each highlighting a specific threat to internal validity. For each scenario, determine the most pressing threat to internal validity. Once you have reviewed all scenarios, select Show Answers to review the correct responses. Scenario 1: A middle school has a new afterschool program for eighth graders targeted at increasing media literacy surrounding alcohol. The goal is to make them "savvy" consumers of advertising, and reduce the impact of such ads on alcohol consumption. Unfortunately, the program can only support 20 students. The evaluator decides to administer a pretest to all 200 eighth graders in the school, and take the 10% with the lowest test scores. Instrumentation, regression, testing, maturation, or history threat? Scenario 2: The evaluators administer the pre-test for an evaluation as a pen and paper survey, and then for the post-test decide to adapt the survey to an online version. Scenario 3: The Heart Healthy program is a one-day seminar targeted at education surrounding healthy food choices and cooking skills to reduce risk of major cardiac events. A pretest is given to see how knowledgeable participants are regarding heart healthy foods and how best to prepare them. The day after the seminar a posttest is circulated to discern program impact. Scenario 4: A new family planning consultation is implemented at a local community health clinic, which experiences a high percentage of women reporting unwanted pregnancies. Peer educators are used to discuss family planning and contraceptive use with women using the clinic. The evaluators follow up with the participants one year later to record contraceptive use and whether any unwanted pregnancies occurred. There is no comparison group, so evaluators are looking only at the group that received the counseling. They find that since receiving counseling the rate of birth control usage has increased significantly. However, the evaluators are aware that last year a regulation was put in place providing free birth control to women regardless of insurance status. Scenario 5: A program targeted at promoting physical fitness and healthy eating among pre-teens was conducted in 2010 in one public middle school in Charlestown. It is now 2015 and the team is interested in following up to see the current activity level and BMI of the participants and how it varies from their pre-program scores, compared to a public middle school located in the South End.
Answer the following series of True/False statements regarding internal validity. External ValidityApplicability of evaluation results to other populations, setting and time periods is often a question to be answered once internal validity threats have been eliminated or minimized. Below is a selection of external threats that can help guide your conclusions on the generalizability of your research results:
Case Study Reflection Thinking back to the Health Bucks case study:
|