Nonexperimental and Quasi-Experimental Strategies
Chapter 10: the Non-Experimental, Quasi-Experimental Strategies
Study Aids and Important terms and definitions
Many of you have been inquiring about how to better prepare for the quizzes.
One way is to use the online resources that accompany the textbook.
The book website is located at:
Select the chapter you want to view.
Any item with a symbol of a lock next to it can only be viewed by instructors.
Other, non-locked, items can be viewed by students and for each chapter there is a glossary, flash cards that you can set to view either a word or its definition first, a crossword puzzle, and a practice quiz.
As I suggested in the chapter introduction, there are several research terms that are probably new to many of you. Chapter 10 vocabulary words and terms you should know the definition of include:
nonexperimental research strategy
quasi-experimental research strategy
nonequivalent group design
assignment bias
nonequivalent control group design
pre-post design
posttest-only nonequivalent control group design
pretest-posttest nonequivalent control group design
differential effects
one-group pretest-posttest design
time-series design
interrupted time-series design
single-subject or single-case design
differential research design
developmental research designs
cross-sectional developmental research design
Nonexperiemntal and Quasi-Experimental Designs
Experimental, non-experimental, and quasi-experimental research designs are similar in that all three strategies investigate relationships between variables by comparing groups of scores. There are also important differences among them.
The experimental strategy creates the groups by manipulating an independent variable.
The nonexperimental and quasi-experimental strategies define the groups with a nonmanipulated variable such as a preexisting participant characteristic (e.g., age or gender) or time (e.g., before/after, at age 10 and age 20).
Quasi-experimental and non-experimental strategies are differentiated by the fact that quasi-experimental studies include some attempt to limit or control threats to internal validity but non-experimental studies do not.
How do you know if it is an experiment or not?
In an experiment:
- The independent variable is manipulated, and the dependent variable is measured
- All other variables are controlled
Experiments are designed to show cause and effect relationships between variables.
Nonexperiemntal and quasi-experimental designs fall short of this standard. Sometimes the independent variable is measured but not amnipulated by the researcher, for example if we want to study stress levels in people who wereexposed to a tornado compared to people who were not, the researcher doesn't control the tornado. Sometimes the independent variable is characteristic like gender, or ethnicity, or education, that can't be manipulated.
If the independent variable is manipulated by the experimenter the researh is not experimental if other variables aren't controlled. For example, if a researcher is testing a treatment in a treatment group and a no-treatment control group, if there is not random assignment to the groups it is not a true experiment.
In a well-executed experiment we can say there there is likely a cause and effect relationship between the independent and dependent variable.
In a qausi-experiment we can say that there could be a cause and effect relationship between the variables, but with much less certainty than in an experiment.
In nonexperimental research we can say that the independent and dependent variable are related, bu we can't be certain what the relationship is; it could be a causal relationship and it could be some other type of relationship.
Between Subjects Nonexperimental and Quasi-experimental designs: Nonequivalent Group Designs (p. 285)
Like experiments, nonexperimental and quasiexperimental designs can be carried out as between- or within-subjects designs. Between-subjects designs are considered first.
A non-equivalent group design is one where the assignment of participants to groups is not controlled by the investigator.
When group assignment is not controlled there is a significant threat to internal validity. Since group assignment is not random, there is a chance that the groups are not similar. Assingment bias occurs when the groups have idfferent characteristics.
For example, if we wanted to study the effect of a program to improve worker motivation in a business setting and selected one business where the program is implemented and comapred it to another business where the program was not implented yu could not be certain that any change in morale was due to the program,ebcause you could not be certain that other differences in the two businesses such as location, features of the buildings, salary and compensation, management behavior, etc. accounted for any changes in morale.
Three types of nonewuivalent between-group designs are considered below.
Differential research design (p. 286)
Differential research designs aim to establish that there are difference between pre-existing groups.
The 'differences' studied are usually partcipant characters such as gender, membership in a group such as college graduates, children from single-parent homes, ethnic groups, country of origin, single Vs, married, home owners Vs. renters,Tampa Bay Buccaneers' fans Vs. Chicago Bears fans...the possibilities are endless.
The researcher usually tries to determine if the groups differ - and sometimes may hope to show that they do NOT differ - on some measured variable such as self-esteem, extraversion, intelligence, kindness, well-being, depression, IQ, and so on.
The researcher cannot manipulate group membership - it is predetermined - so the design is nonexperimental.
As your textbook authors point out, this may sound like correlational research. However, in correlational research no groups are created, and the statistical analysis of differential and correlational designs is different.
Posttest only nonequivalent control group design (p. 288)
The posttest only nonequivalent control group design is used to show that a treatment was effective in a preexisting treatment group.
Thos in the preexisting treatment group are compared to a control group of similar types of persons.
For example, if a researcher wanted to see if a treatment from anxiety was effective in a group of people who were treated, then she might compare them to a group of people with anxiety who were not treated, and see which group scores higher on a measure of anxiety,
In a true experiment partcipants would be randomly assigned to a treatment or control condition.
In a posttest only nonequivalent control group design there is no random assignment so the groups are 'nonequivalent'.
This type of research is often done to look at the effect of new treatments. And keep in mind that the word 'treatment' must be used lossely. Treatment might mean what we usually think of as in a new medication or therapy. And treatment might mean changing the speed limit or adding lights to a road and comapring the number of accidents in a road with a different speed limit or lighting conditions over the same period fo time, or looking at how a new Freshman orientation affects grades compared to a group fo freshman who recieve no such orientation, or how receiving access to the internet affects knowledge of current events compared to a group fo people who do not have internet access.
One group is measured after receiving the 'treatment', and another control group who did not receive the treatment is measured at the same time.
There is a threat of assingment bias - maybe people who got treatment for anxiety are more motivated to get better, or the school with a new freshman orientation is better managed, or the community with new traffic regualtions is more attentive to resident safety...
The threat of assignment bias means that we cannot be sure that group differences are caused by the treatment, so the design is considered nonexpeirmental.
Pretest-posttest nonequivalent control group design (p. 291)
The pretest-posttest nonequivalent control group design is a quasi-experimental design. It goes further than the posttest only design with respect to controlling variables.
QUASI means 'resembling' or 'having some of the features of' - so a quasi-experiment resembles and experiment; it has some but not all of the features of an experiment.
The pretest-posttest nonequivalent control group design controls some for group differences by measuring particpants in both groups BEFORE AND AFTER treatment. This extra measurement period makes it a big improvement over the posttest only design.
There is no random assignment, but since the groups are measured before and after treatment there can be greater confidence in the similarity of the groups and thus that the treatment alone is responsible for any differences observed posttreatment.
The advantage of the pretest-posttest nonequivalent control group design is that if the treatment and control groups have similar scores on the pretest and different scores on the posttest, then they can be more confident that changes in the scores are due to the treatment and not to time related factors or to pretest differences on the dependent variable
However, this improvement does not completely solve the problem of assignment bias, since the particpipants are not randomly assigned to groups. The pretest can show that the groups are similar with respect to the dependent variable, but not that they are similar with respect to other variables. For instance, there could be history effects where one group has different experimences than the other in between the pre and posttests, for instance one city or school may have budget cuts, or experience an event such a school shooting or tornado or other event that the other does not, so history effects remain a threat to internal validity.
Within Subjects Non-Experimental and Quasi-Experimental Designs: Pre-Post Designs (p. 292)
Many within group nonexperimental designs are studies where a series of observations of a single group of particpants are made over a period of time before and after a treatment or event.
Like all within-subjects designs time related events are the biggest threat to internal validity.
These threats were discussed in detial in chapter 9, and they include history, maturation, testing effects, instrumentation, and statistical regression.
All of these threats are related to the passage of time between earlier and later observations.
The one group pretest-posttest design (p. 293)
The most basic nonexperimental within-subjects design is one wehre there are only two observations made; one is made before the treatment or event and a second is made after the event with the aim of showing that scores on the dependent variable changed after the treatment or event occurred.
This design is similar to a between-subjects pretest-posttest design, with one very important difference: there is no control group. In the between groups design the major aim is to compare the scores on the dependent variable in one group versus another another control group. In the within-subjects design the major aim is to compare the scores on the dependent variable before and after treatment.
This simple design does not control for the many threats to internal validity and is often used for preliminary research, such as a researcher doing a small pilot study of a new treatment to see if there is a psotivie outcome in a small group of individuals before embarking on a larger, more time consuming and more expensive controlled experiemnt.
This design is also common in market research, as in a a campaigne manager measuring voters' attitudes towards a candiadate or issue before and after viewing an adverstisement, or a groups of consumers' attitudes towards a new product befor eand after viewing or reading a commercial advertisement.
The time series design (p. 294)
The time series design is classified as a quasi-experimental design because the researcher controls some of the threats to internal validity that are not controlled in a one group oretest-posttest design.
The time series design is similar to the pretest-posttest, except instead of one pretest and one posttest measurement there are several, or a series, of measurements made before and after the treatment or event of interest.
Measuring particpants several times before and after the treatment or event allows the researcher to control for history and maturation because scores, and any changes in them, are obsrved over time. More on this in the example below...
The time series design is often used to test the effectiveness of a treatment or another variable controlled by the researher. And it is often used to study the effects of events that are not controlled by the experimenter. Events not controlled by the experimenter might be predictable, for example studying the effect of a new traffic law on traffic accidents, or unpredictable as in studying the effects of a hurricane on residents' health.
In the latter example, a researcher might examine the number of visitis to doctors' offices or community mental health centers each month for thesix months before and after a hurricane. Of a university official might examine visits to the student healrh center before and after giving free flu shots, or initiang an awareness campaign abotu the health center, or any other health promotion 'treatment.'
To understand how the time series design controls for effects of time, consider a treatment study where stress level is measured in a group of university students once a week for four weeks before and after they particpate in a stress management treatment. By measuring them several times the researcher can control for history, for example, if the intervention took place just before mid-term exams stress might increase, but the researcher would capture this increase befor ethe intervnetion begins. And if final exams occur two weeks after the treatment the researcher might notice a sudden spike in stress after seeing a drop in stress just after the experiment concludes. Or in an alternate scenario, if the treatment is very good, the researcher may notice that there is no increase in stress at the time of final exams! Multiple measurements mean the researcher has a better chance of noticing, and being able to statistically control for time related events such as history, maturation, and practice effects.
Take a look at the three examples below...
(image of three graphs, one with a postive slope and two with changing clopes)
In the first example, the graph has a postive slope and the score on the depedent variable increases at each measurment time both before and after treatment. We could not say with confidence that the treatment (indicated by a vertical line through the X-axis at the time of the treatment or event of interest) caused the change in scores. A pattern such as this one would be seen when maturation or practice effects lead to improved scores on each time of measurement. Most important, notice that if you only completed one pretest and one posttest measure ment you would conclude that the treatment worked! The time series design allows the researcher to see time effects that would go unnoticed in a single pretest-posttest design.
The second example shows a result every researcher hopes for; there is no difference in scores on the dependent variable for three measurement periods before the treatment is administered, and then the sscore increases right after treatment and stays elevated. We migth conclude with some confidence that the treatment is responsible for the change in scores.
The third example would be difficult to interpret. The scores are all over the place both before and after the treatment. Graphs like this are common when the variable being measured is something that isn't very stable from day to day, such as mood, or minutes of exercise, or minutes spent studying for a class on six different days, Note also, that if we were doing a pretest-posttest design with only one measurment before and after the treatment, depeding on exactly when the measurements occurred, we might conclude that the treatment had a positive effect, a negative effect, or no effect! The time series design allows us a clearer picture of how the dependent variable changes over time.
Single subject/single case applications
Single subject designs are a special case of time series designs where, as the name implied, research is done with only one particpant, "a single subject." The researcher might study one individual or one 'case' if the case is a single entity such as a single classroom, school, or park, or business.
Many behavior therapists and behavior analysts use single case designs to see if their treatment is effective with a given client, for instance, if they are working with a shy client they might measure 'minutes of eye contact' during an hour each day, or number of times initiaing a conversation in the classroom, before and after treatment begins. If there is no change, then this is a sign for the therapist to change what they are doing and try a different intervention.
Organizational psychologists might use a single case design to see the impact of their interventions on a business or other organization. For instance, they might look at absenteeism before and after a program to boost attendance, or new grants or patents submitted before and after a prgraom to boost creativity or productivity.
The single case designs are thus a treatment as well as a research tool.
Quasi-Independent Variables (p. 304)
You may recall that in experimental research the investigator manipualtes on of the variables and we call the manipualted variable the independent variable and the non-manipulated variable the dependent variable.
So what do we call it in nonexperimetnal and quasi-experimental research, since no variable is manipualted???
The quasi-independent variable is the variable that is used to differentiate particpants groups or treatment conditions in a nonexperimental design or quasi-experimental design. The other variable, the one measured to obtain scores within each group or condition, is still called a dependet variable.
If the researcher wants to study the difference between IQ in children who have a high protein versus low protein diet than IQ is the dependent variable and diet is the quasi-independent variable.
If he wants to study differences in visits to community health centers in New York before and after The 2001 Terrorist attacks, then before/after the attack is the quasi-independent variable and number of visitis to the metal health centers is the dependent variables.
If he wants to study differences in leadership skills in boys and girls, then sex is the quasi-independent variable and leadership skills is the dependent variable.
Developmental Research Designs (p. 297)
Developmental research designs are nonexperimental designs.
What makes them different from the other designs discussed is that they are used to study changes in behavior that are related to age.
It is common for researchers to study behavior change with age. For example, developmental psychologists and health psychologists and those studying personality and psychopathology want to know how behaviors during childhood eafect individuals throughout the lifespan, or if behaviors and traits observed in childhood are stable or change over the course of an individuals' life, or if events affect people of different ages differently.
Cross-sectional designs compare people of different ages at the same time, longitudinal studies follow a group of people for a long period of time - sometimes decases. And there are also combined designs that are both cross-sectional AND longitudinal at the same time.An example of a combined design is presented at the end of this section.
The cross-sectional developmental research design (p. 297)
The cross-sectional developmental design is a nonequivalent groups, between-subjects design where each group is composed of individuals of different ages. For example, one grou pmight be all 20 year old individuals and a comparison group might be 50 year old individuals. Researchers might want to see if there are differences in stress or well-being or life satisfaction or worry about the future or health concern at different ages. You read or hear about cross-sectional research all the time. For instance, recent studies show that younger Americans are more likely than older Americans to be unemployed, have faster reaction times, support gay marriage, be vegetarian, watch reality TV shows, take psychotropic medications and are less likely than older adults to be divorced, vote in an election, have serious health problems, read books, feel that they will be well-prepared financially for retirement.
The major strength of corss-sectional designs is that a researcher doesn't ahve to wait for people to age in order to make predictions about behavior as people age -- they can merely study groups of people of different ages.
The major weakness is that particpants may differ in more ways than age alone. For example, if we find that 85 year old persons are more frugal with their money than 55 year olds, this may not merely be a difference tha tha sto do with age. It may well be because the odler person was born during the great depression and a child during World War II when many perople had finanacial difficulties, while the 55 year old was a child during the more affluent post-war expansion.
People born at differnet times have had different experiences due to event stha toccurred during thier respective life times.
Cohort effects
A cohort is a group of individuals who were born at about the same time and grew up under similar circumstances.
A cohort effect - also called a generation effect - occurs when differences between two or more different age groups are due to differences other than their age. That is, each cohort has had unique experiences that other cohorts haven't. For example, today we find that far fewer very elderly persons use computers as do younger groups of people. However there is nothing about being old per se that makes one less likely to use a computer. It is most likely a corhort effect, where older adults did not grow up using computers. In the future, when you are 'old' you will probably use computers because you've been using them most or all of your life. Differences in computer use are correlated with and not caused by age. The difference in use by age is due to familiarity with computers.
Longitudinal developmental research design (p. 300)
A longitudinal design is a one group within subjects nonexperimental design or a quasi-experimental time series design.
In nonexperimental longitudinal designs one group of individuals is measured over time and the independent variable is age, that is how does a score on some meaure change with age?
In quasi-experimental longitudinal designs, a change in the independent variable is measured before and after some event, such as a divorce, a parent's divorce, graduation, a natural disaister ushc as a major hurricane or earthquake.
The advantage of a longitudinal design is the absence of cohort effects. I f we follow a group of 20 year olds and meausr ehtme again at age 30 all of them are part of the same cohort of individuals.
Another advantage is that the researcher can observe how a single individual changes over 10 years rather than making inferences by comparing 20 year olds to 30 year olds at the same time as in a cross-sectional design.
Of course there are also disadvantages to longitudinal designs:
- Time - it takes years or decades to complete longitudianl studies
- Mondy - it is very expensive to track people over time
- Attrition/Mortaility - particpants lose interest, can't be located, or die during the course of the study
Cross-sectional longitudinal designs
In practice there are many deisngs that are not purely longitudinal or cross-sectional. Designs that have features of both cross-sectional and longitudinal designs are mixed developmental designs.
For example, a researcher is following groups people who were 10, 20 and 40 and lived in or near New Orleans at the time of Hurricane Katrina. She is studying the impact of the hurricane on their physical and mental health and careers and schooling. She has them answer a set of questions every year and will follow-up for at least 10 years.
The study is cross-sectional in that there are three cohorts and between-subjects comparisons will be made.
It is longitudinal in that within each cohort there will be a series of measurements over a long period of time and within-subject comparisons will be made.
A lengthy example
To see how a mized design might look and review other terms related to various designs consider the following example fo a mixed developmental design...
I (Dr. Bach) was a participant in a cross-sectional longitudinal study for 15 years. The example is lengthy, and it gets at several issues in designing and carrying out this sort of study.
The study was called “High School and Beyond”
It started in the 1970’s and ended in 1994. I was a participant from 1980 – 1994. I completed a series of questionnaires in 1980, 1982, 1987, 1990, and 1994, and the researchers looked at my school records in 1980 and 1982.
The study was cross-sectional in that there were several cohorts, with three of them most often compared - the classes of 1972, 1982, and 1992 formed three between subject non-equivalent groups.
The study was longitudinal in that participants in each cohort were followed for 4 to 24 years and they looked at within-subject variables in each participant.
The study was a mixed design in that both the cross-sectional/between-subject and longitudinal/within-subject data were analyzed.
To create the class of 1980 and 1982 cohorts the researchers used probability sampling techniques employing both cluster and stratified sampling techniques. The ‘clusters’ were high schools; They were stratified by (1) whether the school was public, Catholic, or private non-Catholic, (2) geographic region with 9 regions in the US represented (3) ethnic composition of the school (4) urbanization with rural, suburban and urban schools represented (5) income level of the community (6) proximity to a college (7) number of students in the school. Using a complex procedure of combining these 7 variables, all of the high schools in the US were placed into one of more than 500 substrata, and two schools were randomly selected from each.
There were a total of 1,116 schools and from each school 36 sophomores and 36 seniors were randomly selected. Additionally (and for the purpose of a different study) they asked anyone who was a twin, triplet, or other ‘multiple’ to bring along their same age siblings.
For each participant they got parental consent and collected school grades and achievement test scores and gave several questionnaires about demographics (ethnicity, family size, parents education and occupation, etc.), attitudes towards school, plans for the future, well-being, and attitudes towards a number of issues.
In the follow-up questionnaires that came after 2, 7, 10 and 14 years had passed, the questions asked about what participants were doing, and repeated some of the questionnaires about plans for the future, attitudes, and well-being.
The cross-sectional piece compared cohorts, for example how did the classes of 1972, 1982 and 1992 differ with respect to what percentage went to college (and also the percentages by strata, e.g., were there differences in college attendance among those who attended private Vs public schools, or by ethnicity, or rural Vs. urban dwellers, etc.); how did the cohorts differ with rates of marriage a few years after graduation? How did they differ in attitudes measured by responses to True/False statements such as “In my career, I believe it is more important to earn a lot of money than to help others.” Or, “keeping up with current events is important to me.”, or “I will probably attend college.” Or, “I am very likely to have children.”
The longitudinal piece looked at whether individuals actually did what they thought they would. For example, if you said in 1980 that you probably would go to college, had you graduated from college by 1990? Or, if you had high well-being in 1980, was that still true in 1982, 1987, 1990, and 1994.
There were also combined longitudinal/cross-sectional questions, for example, the researchers looked at correlations, for example, what predicted whether or not someone would go to college? Parents’ education, grades, achievement scores, living in an urban Vs. rural area, etc. And were they the same for each cohort, or were they changing over time?
What are the strengths of this study?
Each cohort had more than 35,000 participants. The sample was very representative of the United States population. Participants were followed for a long time. The data allowed educators and politicians to get a look at trends in high school graduates and determine which areas of the country and which groups of people were succeeding and failing, which could inform teacher and school administrator education, and public policy and spending.
What were the disadvantages? Mainly cost. In addition to paying researchers to run the study, they paid each of 36,000 to more than 100,000 participants $25 each to complete each follow-up survey. The study ended because they ran out of funding, not because they thought they were done with the study! They had to track each participant for up to 24 years – not as easy or inexpensive in the 1970’s and 80’s as today in the computer age. There was attrition; some participants died, and many couldn’t be located (women in particular since their names often change after marriage), and many didn’t bother to participate in the follow-up sessions.
Interesting factoid that pertains to the utility of longitudinal research. The data from this study are available in the public domain. That is, if you wanted to replicate the study with the class of 2012 or 2022, you could access all of the data from the classes of 1972,1982 and 1992 free of charge.
The National Institute of Health and other government agencies that fund much research now require researchers to make their results available in the public domain.
Sharing data is common. It is expensive to gather and report data, and once in a database it is inexpensive for another researcher to access it and use the information to answer some new research question using an old data set.