Research Strategies and Validity

Quantitative and Qualitative research

Most of the content of this class is about quantitative research. Still, you should know the difference between qualitative and quantitative research.

Qualitative research is based on observation of research participants. Data are presented as narrative descriptions of what was observed. This is in contrast to quantitative research where data are reported in terms of statistical analyses.

Qualitative research is common in anthropology or the branch of biology known as ethology. For instance, the anthropologist might observe the describe the behavior and customs of a group of people without completing any statistical analyses. A biologist might observe members of some species and describe their behavior in great detail.

In psychology some case studies of individuals are qualitative. Sometimes treatment programs, such as a new type of therapy, or a program implemented in a school or business setting might be observed and described qualitatively.

For your final paper you will develop and write a description of a quantitative research project.

Strategies for quantitative research are introduced in chapter 6. Don’t be intimidated just yet if it seems like too much information in being presented! All of the strategies introduced in chapter 6 will be described in more detail in later chapters. The aim of chapter six is to merely introduce some of the commonly used research strategies.

Quantitative Research strategies

The descriptive research strategy (p. 160)

This is the simplest research strategy. Variables are not compared or related to one another. The investigator simply explores the state of a variable or variables in a population. For instance, what is the average reading achievement score of inner city youth in the United States? What is the prevalence of alcohol abuse and the rate of depression in South Koreans? What is the rate of obesity in Native Alaskans?

Relationships among variables

Most research strategies are not merely descriptive. Investigators relate two or more variables rather than merely describing their states. To slightly modify one of the above examples - so that variables are related - an investigator might ask, how do the math and reading achievement scores of inner city youth in private schools compare to those in public schools? Does a classroom size affect reading and math achievement scores? Does a new learning strategy lead to improvements in math and reading achievement scores? Do math achievement scores predict reading achievement scores?

Relationships between variables are often reported in graphic form.

Make sure you understand the meaning of and can identify a graph of a positive linear relationship, a negative linear relationship, a curvilinear relationship, and uncorrelated variables (pp. 160 – 161)

(image of scatterplot for positive linear correlation)

(image of scattorplot for negative linear correlation)

(images of scatterplot for curvilinear relationships)

(image of scatterplor of uncorrelated variables)

Correlational strategy

In correlational research scores on two variables are compared in a single group of individuals. For example, investigators have found a modest positive correlation between math achievement scores and reading achievement scores.

You’ve probably heard the saying that ‘correlation does not equal causation.’ Correlational research only describes relationships among variables, it does not explain them.

Experimental Research Strategy (p. 163)

The aim of experimental research is to explain rather than merely describe relationships among variables. Specifically, cause and effect relationships are examined. For example, imagine that children’s reading achievement scores were evaluated, and then half of the children were randomly assigned to receive tutoring in reading for three months, and then their reading scores were measured again. This experimental design could answer a question such as, “does tutoring in reading skills lead to greater gains in reading achievement scores than no tutoring?”

Note that in an experiment there is random assignment of participants to condition.

Quasi-experimental Strategy (p. 163)

Quasi experiments are similar to experiments, however they often lack random assignment to treatment conditions so cause and effect relationships are not as certain as in true experiments. For instance, if an investigator compared the reading achievement scores of children receiving tutoring to those not receiving tutoring AND there was no random assignment, she could not determine whether the improvement was due to the tutoring, or if some other variable also effected the outcome. For instance, perhaps those who wanted tutoring were more motivated to improve their reading, or their parents were more concerned about their reading achievement. We could not have as much confidence that the tutoring caused the improvement in scores as in the experiment described above.

A common situation where quasi-experiments are done is when we want to study a phenomenon where it would be impractical or unethical to randomly assign participants to an experimental condition. For example, if you wanted to study the effects of terrorism and compare people who are in close proximity to a terrorist attack to people who are far from a terrorist attack, you could not randomly assign half of your participants to be close to a terrorist attack!

However, if a terrorist attack did occur, you could study people who were close to or far from it after the fact. It is a quasi- rather than true experiment, because participants were not randomly assigned to a ‘treatment’ condition.

Non-Experimental Strategies (p. 164)

Non-experimental designs attempt to show that variables are related without trying to explain why they are related. For instance, a study that found that children in private schools have higher reading achievement scores than children in public schools would not show that attending private school causes higher reading scores. It could be that private school DOES cause higher reading achievement, but without random assignment of participants, many other explanations are also possible. For instance, perhaps private schools are more selective, or hire better teachers, or attract students with more achievement motivation, or with more resources.

The different research strategies are summarized on pages 165 – 167.

Internal and External Validity (p. 167)

The validity of a study is the degree to which it accurately answers the question it intended to answer.

Anything that makes us question the accuracy of results or the quality of the study can be described as a threat to validity

External validity (p. 168) is the degree to which we can generalize findings. For instance, if someone conducted a treatment study comparing two treatments and reports that a group of people who received treatment A reported greater reductions in depression than people who received treatment B, we would want to know if the findings generalize, or would also be true of samples with different characteristics, if different measures of depression were used, if different clinicians delivered the treatment, if it were delivered in an inpatient versus outpatient setting, if it were delivered in German to a German speaking sample, if it were delivered to elderly adults or young adults, and so on.

Three types of generalization are noted:

Generalization from a sample to the general population. Or, is the sample really a representative sample? Is treatment A better than treatment B for all people, or was the sample not very representative?

Generalization from one research study to another. Or, can the study be replicated with a different sample of individuals?

Generalization from a research study to a real world setting. Or, will treatment A work in a busy community mental health center where the clinicians may not have as much time to spend with each client, or as much expertise or motivation as the investigators?

Internal validity (p. 169) is the degree to which we are confident that the outcome was caused by the variable of interest. For instance, that treatment A is superior to treatment B and not because treatment A is longer, or the groups who received treatment B was more depressed to begin with, or that the clinicians who delivered treatment A had superior skills, and so on.

Threats to External Validity

Category 1: Generalizing across participants p. 171

To what extent is the sample representative?

Is the sample biased?

Is the sample all college students?

Is the sample a convenience sample of volunteers? (Table 6.4 on p. 173 provides many examples of how volunteers are likely to differ from the overall population)

Participant characteristics: do participants in the sample share common characteristics, for instance are they all highly educated, or wealthy, or share the same religion, or are from the same part of the country?

Cross-species generalization: can the results of a study conducted with rats be generalized to humans, or cats, or horses?

Category 2: Generalizing Across Features of a Study (p. 174)

Can the results of the study be generalized to other procedures for conducting the study?

Novelty effect: something new is more exciting than something routine. Experiments on new treatments often have somewhat better results than later studies when the treatment is no longer novel.

Multiple treatment interference: in research there may be practice effects when the same individuals complete the same measures over and over again, or treatment interference when participants receive multiple types of treatment, or when participation in one type of treatment has an effect on outcomes in another condition.

Experimenter characteristics: will different investigator obtain the same results? Maybe treatment A isn’t better than treatment B, and instead investigator A is especially charismatic, or friendly, or is more similar to or different from the participants.

Category 3. Generalizing Across Features of the Measures (p. 175)

Can the results of the study be generalized to other ways of measuring the phenomenon of interest?

Sensitization: does some aspect of the measurement procedure make participants more sensitive to the thing being measured? Or does self-monitoring effect the thing being monitored (for instance, if someone were trying to change eating behavior or nail biting, having participants monitor these behaviors will almost always affect eating behavior or nail biting).

Generality across response measures: there are many ways to measure a phenomenon of interest. Will the results of the treatment comparison study be the same if depression is measured in a different way?

Time of measurement: will the results be similar if the phenomenon of interest is measured at a different time? For instance, maybe treatment A is better than treatment B when measured after six weeks, but perhaps the outcome is the same of B is better after 12 weeks. And as your textbook points out, many smoking cessation programs look great if smoking is measured after say two weeks or a month. But after 12 months, only about 15% of smokers are still abstaining.

Table 6.5 on p. 177 summarizes the three categories of threats to external validity

Threats to Internal Validity (p. 176)

Extraneous variables and confounding variables threaten internal validity, that is, they may explain for the observed outcomes.

An extraneous variable is one that is not specifically being measured, and that might have an effect on the phenomenon of interest. For example, suppose a new treatment for depression is being tested and the people who receive treatment B have more anxiety than those receiving treatment A, the outcome observed may be due to differences in participant anxiety (which was not measured) instead of to differences in the treatments.

A confounding variable is one that changes systematically along with the variable being studied and provides an alternate explanation for the outcome. For instance in the study comparing treatment A to treatment B, if treatment A is always delivered by therapist #1 and treatment B is always delivered by therapist #2, it could be that case that the outcome is due to therapist #1 being better than therapist #2 rather than to treatment A being better than treatment B.

Every study has extraneous variables and many have potential confounding variables. So what is an investigator to do? The best thing to do is to try and control for these variables and maximize the certainty that the variable being measured is the only one that can explain the outcome. For instance, in the depression treatment study the experiment could be designed so that therapist 1 and 2 each deliver half of the treatment B sessions and half of the treatment A sessions.

Additionally, there are statistical procedures for controlling for extraneous variables. So one could control for the possible effect of, say, years of education, or income, on reading achievement by using various statistical procedures.

Environmental Variables: a threat to internal validity in all studies

For example, suppose a new treatment for dementia is being tested and some participants are tested in the morning and others in the evening. Those tested in the morning will usually out-perform those tested in the evening. And we might incorrectly attribute higher scores to treatment condition when ‘time of day’ is a more influential variable. Time of day is an environmental variable. Others might be weather, the characteristics of the room where participants are assessed. Any differences can create the possibility of an alternate explanation for the observed outcome.

Individual differences: threats to internal validity for studies comparing different groups (p. 181)

Here the problem of assignment bias is discussed. That is, are there differences in the way participants are assigned to groups such that members of one group differ from members of another group in a manner that may account for the observed difference? For instance, are the people in one group smarter, or younger, or friendlier or less depressed than participants in another group?

Time-Related variables: threats to internal validity for studies comparing one group over time (p. 181)

Some research projects administer multiple treatments to the same group of individuals over time instead of administering different treatments to two groups at the same time. While this approach controls for individual differences, it can create a threat to internal validity due to the effects of time on the variable of interest. Your textbook describes 5 such threats. Make sure you understand the differences between them.

History

Maturation

Instrumentation

Testing effects

Regression to the mean

Table 6.6 on p. 186 provides a summary of threats to internal validity

Balancing threats to internal and external validity

Notice that the more one tries to control threats to external validity the more one introduces threats to internal validity (and vice versa). For example, if you try to control all extraneous variables (threat to internal validity) the results may not generalize outside of the experimental setting (threat to external validity). The best we can do is to try to balance these threats, and document limitations to the study. For example, it is okay to study a treatment on a sample of young adults and conclude that “one limitation fo this study is that the participants were young, and we cannot be sure if the results will generalize to older adults.” Now you have your next project!!

Artifacts: threats to internal AND external validity (p. 188)

An artifact is an external factor that can influence or distort measurement. Your textbook describes three types of artifacts: experimenter bias, demand characteristics and participant reactivity, and exaggerated variables.

Experimenter bias

This occurs when the investigator wants or hopes for a certain outcome, and behaves (often unconsciously) in subtle ways that make the desired outcome more likely. For example, being more friendly to participants in one group as compared to another group. Blind and Double blind studies are one way to minimize experimenter bias.

Demand characteristics and participant reactivity

This occurs when participants guess the purpose of the study, or are playing the role of ‘participant in a study’ and are so influenced to behave in a certain way (demand characteristic), or when monitoring their own behavior changes it (reactivity). Sometimes deception is used to minimize demand characteristics, and the investigator must balance ethical practices with a desire to conceal the true nature of the study.