List of Threats to External Validity
As we said, achieving external validity isn’t easy because there are several potential threats that hinder external validity.
Various situational factors, sample features, pre-and post-test effects, or testing effects are some of the potential factors that can be perceived as threats to external validity.
Thus, a great researcher always pays attention to the following factors to ensure the external validity of a study.
Pre- and post-test effects
Pre- and post-test effects are some of the most common factors that significantly affect external validity.
A pretest is a survey, questionnaire, or another type of research instrument that is used before the actual process of conducting research begins.
A post-test is an endline of research that is done after some time has passed since the research was conducted to ensure the results are still applicable.
Unfortunately, pre- and post-tests often interfere with the generalizability of the results.
The reason is that added tests eliminate the effect of a causal relationship between variables, and as a result, they pose a threat to external validity.
The degree of external validity significantly depends on the type of sampling. The sample is part of the target population.
In fact, the results should be generalized to the population based on the sample features, which means that choosing participants reasonably is one of the most important factors that affect the external validity of the study.
While non-probability sampling methods where participants aren’t chosen randomly and thus don’t represent the population pose a threat to external validity, probability sampling counters selection bias and ensures that each member of the population has the same chance of being selected in the sample.
Selection bias is another threat to external validity, which refers to the error in selecting the participants that take part in the research.
In order to ensure external validity, it’s important for the subjects to have similar features to one another and represent the larger population.
Otherwise, their results can’t be generalized. That’s why researchers try to correct the weighting of factors and control them.
But sometimes it’s hard to avoid selection bias, especially when the participants represent specific groups such as clinical patients or criminals.
In these cases, often the research methodology doesn’t allow the researchers to ensure external validity.
A wide range of situational factors affects the external validity of the study.
In particular, confounding variables such as location, time of the day, noise, temperature, and even researcher characteristics significantly affect the way participants respond to the research manipulations.
That’s why it’s important to control these situational effects as much as possible to avoid biases and ensure external validity.
One specific factor that affects external validity is the so-called Hawthorne effect.
It’s a common concept in social psychology and represents a tendency of research participants to change their behaviors in order to meet the researchers’ requirements.
The reason for this kind of behavior is the natural need to appear socially desirable. As a result, the participants act in a pleasing way, which affects the results and damages the accuracy of the findings.
Aptitude-treatment interaction
Aptitude-treatment interaction (ATI) means that interventions in the study match the attributions and characteristics of a participant.
The reason why this factor affects the external validity is that interactions between participants’ characteristics influence the dependent variable and don’t allow the researcher to effectively control or manipulate it.
Consequently, aptitude treatment is another important threat to external validity.
Factors That Improve External Validity
Considering that sometimes the accuracy of the study findings almost completely depends on the external validity, researchers often try to look for ways to increase the external validity.
Replicating the study, considering psychological realism, trying field experiments or increasing randomization are some of the factors that help improve external validity.
Consider psychological realism
At a first glance, the phrase “psychological realism” might sound a bit complicated, but actually, it means that participants in the study perceive the experimental manipulations as real events.
For this, sometimes it’s necessary to cover the story and not reveal the real purpose of the study until the research is finished.
Otherwise, participants might try to appear socially desirable and behave differently.
So, considering psychological realism is one way to improve external validity and generalize the results in real-life settings.
Do reprocessing or calibration
Researchers often use various statistical methods in order to ensure or improve external validity.
For instance, reprocessing data using calibration tables allows researchers to go back and reprocess the data in order to avoid errors while interpreting the results.
This might be the best possible option when there are some issues in the data, such as having uneven groups or having different characteristics.
Replication is one of the most frequently used methods when the findings don’t show external validity.
In order to increase validity, researchers often use this method, which means using different samples from the same target population. If they get the same results, then it means that the study has high external validity.
But if the results are different from the previous sample group after the replication, then their results can’t be generalized to the population.
Another way to ensure external validity is to conduct an experiment outside the lab setting.
Conducting a field study in a natural setting instead of a lab setting where the researcher has to manipulate all the variables themselves is a proven way to improve external validity.
Still, based on the study’s aims and specificity, it’s not always possible to try field experiments.
Use inclusion and exclusion criteria
Using inclusion and exclusion criteria while choosing participants for your sample from the target population makes sure that you have defined the population accurately and that the sample is representative of the population.
As a result, the likelihood of producing reliable and valid results increases in accordance with the degree of external validity.
Randomization is the process in which participants are assigned to trials completely by chance.
In this case, neither the participant nor the researcher knows which group of participants belongs to which.
As a result, increasing randomization might affect the external validity and improve its degree.
The reason is that randomization increases the generalizability of your results.
External Vs. Internal Validity
Internal validity is another type of validity that’s almost the complete opposite of external validity.
In fact, internal validity is the degree to which the causal relationship that is being assessed is reliable and other factors do not influence the variables.
Internal validity measures the internal consistency of a research instrument or method that is used rather than the generalizability of results.
The main difference between external and internal validity is that internal validity is related to the internal structure of the research methods, while external validity tries to prove that the findings are universal.
Furthermore, in order to achieve internal validity, researchers usually use strong research methods and strictly control extraneous factors.
On the other hand, high external validity means that the results can be applied to practical situations and different contexts.
Respectively, the findings of the studies that have high internal validity prove that a causal relationship between variables is trustworthy, but it doesn’t say anything about the possibility of generalizing the results.
On the contrary, if a study has external validity, the results can be applied to the entire population.