FACTORS THAT CAN AFFECT THE FINDINGS OF A STUDY (EXTERNAL AND INTERNAL VALIDITY)

External validity = population validity and ecological validity.

Extent to which findings can be applied to individuals and settings beyond those that were studied.

Population validity - - extent results can be generalized from the sample to a specified larger group (the population from which the sample was drawn). Population validity is affected by sampling and by extent to which person logical variables interact with treatment effects (student ability, gender, anxiety, etc.).

Ecological validity - - extent to which the results can be generalized from conditions of the research setting to particular naturally occurring settings.

Factors affecting ecological validity include:

• Explicit description of the experimental treatment - - described in sufficient detail for other researchers to replicate
• Multiple-treatment interference - - participant is exposed to more than one treatment
• Hawthorne effect - - mere fact that individuals are aware of participating in an experiment changes performance
• Novelty and disruption effects - -treatment causes change simply because it is different and effectiveness may erode as novelty wears off; distortion may show decline in performance initially but be effective after assimilation into routine (initial tryouts not generalized to continued use settings).
• Experimenter effect - - change due to the particular individual who administers the treatment (e.g. teacher effect).
• Pretest sensitization - - pretest interacts with the treatment so that if treatment is repeated without the pretest, different results occur. Most likely to occur when retest is a self-report measure of personality or attitude. Researchers have found substantial effect of pretests on post tests.
• Post test sensitization - - results dependent on post test administration. Post test is learning experience in its own right.
• Interaction of history and treatment - - belief that generalizing should not occur beyond the time period in which the experiment was done.
• Measurement of the dependent variables - - generalize ability limited buy type of measure used e.g. multiple choice results may not be same as essay results.
• Interaction of time of measurement and treatment effects - - administration of post test at two or more points in time may result in different findings (post test versus delayed post test may show differences in effects).

Internal Validity - - extent to which extraneous variables have been controlled so that any observed effect can be attributed solely to the treatment variable.

Extraneous variables are any variables that can effect the outcome if not controlled. Extraneous variables that can affect internal validity include:

• History - - events that occur besides the treatment (events in the environment)
• Maturation - - physical or psychological changes in the participants
• Testing - - effect of experience with the pretest - - become test wise.
• Instrumentation - - learning gain might be observed from pre to post test simply due to nature of the instrument. Particularly a problem in observation studies when observers more likely to give ratings based on expectations (conscious or subconscious).
• Statistical Regression - - Tendency for participants whose scores fall at either extreme on a variable to score nearer the mean when measured a second time.
• Differential Selection -- Effect of treatment confounded with other factors because of differential selection of participants, problem in non random samples
• Experimental Mortality - - participants lost from the study, attrition
• Selection-maturation Interaction -- similar to differential selection, except maturation is the confounding variable
• Experimental Treatment Diffusion - - Treatment is perceived as highly desirable and members of control group seek access
• Compensatory Rivalry by Control Group -- (John Henry Effect) - - control group performs beyond expectations because they perceive they are in competition with experimental group.
• Compensatory Equalization of Treatments - - occurs when experimental group received goods or services perceived as desirable and control group is given similar goods and services on compensate. Not comparing treatment with no treatment but one treatment with another
• Resentful Demoralization of Control Group - - Control group becomes discouraged because they perceive experimental group is receiving a desirable treatment that is being withheld from them

Other issues in designing experiments

• Experimenter Bias -- researcher’s expectations about the outcome that are unintentionally transmitted to participants so that their subsequent behavior is affected. Not intentional.
• Treatment Fidelity - - experimental fails to follow the exact procedures specified for the experiment (failure to follow protocol).