Chapter 7 - Asking People about themselves: survey research

You have undoubtedly been asked to complete a survey at some time.

If you use the internet (and who doesn't...?) then you regularly get those annoying messages that intrude on the web page and say things like "your opinion matters to us - please click here to complete the survey" or "was this page helpful?" and an option to click yes or no.

While many surveys are used for consumer research - that is, to evaluate how you feel about a product, service or web page - survey research is also used to gather data about how a constituency feels about politicians or policies, or how memberships or an organization feel about the leadership and direction f the organization, or to conduct research about how a group of people (say Americans, or middle class Americans, or residents of Florida, or Florida college students, etc.) feel about any number of issues such as Florida gun laws, their financial preparation for retirement, college tuition rates, attitudes towards gay marriage or legalization of marijuana, or any number of issues. They might also ask about behaviors, e.g., voting, immunization, smoking, wearing seat belts, making various purchases, etc. And they might be interested in demographics, for instance, to determine how many students at a school speak English as a second language; how many parents have children enrolled in after-school activities; how many are interested in having their child participate in various clubs, sports or other school-sponsored activities, or how many prefer a computer versus phone as the primary means of communicating with the school.

Your textbook asks the question, 'why conduct surveys?" and there are many answers to this question.

Surveys provide a relatively inexpensive way to gather data from a large number of people relatively inexpensively.

Surveys provide a 'snapshot' of how people feel at a particular point in time and how their attitudes change over time.

Surveys allow organizations to evaluate how effectively they are meeting the needs of stakeholders.

In short, surveys are a good way to gather all kinds of data about people's characteristics, attitudes, and behaviors.

Survey Questions

Writing useful questions is the most challenging part of constructing a good survey. And writing a good question is a lot harder than you might think. In graduate school, I took an entire course on survey research (or "Field Research Methods") and we spent most of our time learning how to design good questions and how to ask them the right way to get the information you seek. If you want to learn more about this topic, I highly recommend "Survey Research Methods" by Fowler. It's a small book, but it is packed with great information on this topic. This book has been on my shelf since grad school and I still refer to it once in a while. Now back to "survey questions."

In order to construct a survey one must first

Define the research objectives: what do they want to know and how can survey questions be written to find out what they want to know most effectively?
Know characteristics of the sample related to how the survey is administered: a survey aimed at very elderly adults may be better administered face-to-face than over the internet.
Construct questions according to whether demographic information, attitudes and opinions, or behaviors is sought. For instance, are you registered to vote? seeks demographic information; "who do you prefer to be the next president, candidate A B or C" seeks an opinion, and "did you vote in the last presidential election?" inquires about behavior.

There are several different types of survey questions. Each type has pros and cons.

Open ended questions

participants respond in their own words to the topic presented. For example,

How do you think campus safety could be improved?
What do you believe are the most important problems facing the country?

There is only one advantage to using open ended questions, and it is that open ended questions have the greatest response flexibility so participants can answer the question in an infinite number of ways.

This is also its biggest weakness - the data are often difficult to summarize or interpret and compare. Specifically, it is difficult to subject open ended questions to a statistical analysis, and participants may answer in ways that are hard to compare, for example, three answers to the question about campus safety might be "It can't be improved" - does this mean that it is perfect and there is no way to make it better, or that nothing can be done to improve it? And another participant might name procedures that can be implemented, say add metal detectors, hire more police officers, install panic buttons and emergency phones. Alternately another student may emphasize working with existing structures, such as "have more training for all personnel" The variety of possible answers makes them difficult to compare and summarize.

Restricted Questions

Restricted questions have a limited number of responses.Multiple choice and True/False are types of restricted questions.

For example, the open ended questions about campus safety above might be re-written to something like:

Which of the following strategies do you think would have the biggest impact on campus safety?

a. installing metal detectors in all classrooms

b. conducting weekly random searches of campus residence halls

c. hiring more campus police officers

d. installing emergency phones

Which do you think is the most important problem facing the country?

a. unemployment

b. the federal deficit

c. partisanship in Washington

d. the global economy

The limited number of options makes it easy to analyze the data statistically and to summarize it. For example, "60% of respondents said that unemployment is the biggest problem facing the country". "No students suggested that searching residences would improve campus safety and they were about equally divided as to the effectiveness of hiring more police officers or installing emergency phones."

A weakness is that respondents are forced to endorse one of the restricted items, but might believe something else is more important. For example, someone might answer 'unemployment' to the question about the biggest problem facing the country, but might think some other item that isn't on the list is the problem, say 'tensions with North Korea.'

One solution is to add an open-ended option "other" ___________________, that participants can respond to in any way they choose. These data are fairly easily summarized by saying something like "10% of respondents selected "other", and wrote in 150 different answers. Items written by at least 1% of respondents included, tensions with China, congress, tax reform, and immigration.

Another - though more partial solution to the problem of the limited array of questions - is to have an option to 'check all that apply" For instance, which of the following options do you think would be likely to significantly improve campus safety, and provide several options where the respondent could check any or all of them.

"None of the above" also solves the problem of having answers that might create too many restrictions.

Rating scales

Rating scales allow participants to select a numeric response, or a response that can easily be converted to a numeric rating, on a scale designed by the investigator.

Likert and Likert-type scales

Likert and Likert-type scales are among the most common.

These scales usually have four or five responses assigned a value from 1 - 4 or 1 - 5, such as "1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree." In some situations, you may not want to give people the "neutral" option of neither agreeing nor disagreeing, in which case, you would use the four response format. People have a natural tendency to choose the neutral option. In reality, there are very few things on which we neither agree nor disagree. In numerous statistical tests of response options, scientists have repeatedly found that it is usually better to force people to choose rather than giving them the easy way out of choosing a neutral response. It's usually better to force a choice because people almost always lean one way or the other, and they will give you useful information if you force a choice. If you don't force a choice, every neutral answer is usually worthless data.

The instructions for a Likert scale might be along the lines of "use the following scale (numbers 1 - 5) to describe how you feel about each of the statements below."

1 = strongly disagree, 2 = disagree, 3 = neither agree nor disagree, 4 = agree, 5 = strongly agree.

1. ____ unemployment is a serious problem

2. ____ taxes are too high

3. ____ I would be pleased if an immigration reform bill were passed

4. ____ I describe myself as liberal

By the way, 'Likert' is one of the most commonly mispronounced words in psychology. It is pronounced Lick'-urt not Like'-urt

So, what is the difference between a Likert Scale and a Likert-type Scale?

Read this excellent (and short) article to find out: Link http://john-uebersax.com/stat/likert.htm

Subjective Units of Distress

Another common rating scale is called a subjective units of distress scale (or SUDS or SUD scale) and in often used in clinical research. One version of this type of scale is something like.

Rate your level of anxiety on a scale of 1 - 10 where 1 means no anxiety at all, extremely relaxed and 10 is the most anxiety you have ever felt.

The highest and lowest ratings are called anchors, and should be extreme, for example, 'strongly disagree', or 'none at all', or 'the best',

And the person making the scale can give examples or, as in the SUD scale example above have participants set their own anchors, for example the researcher set 1 as no anxiety and 10 as the most anxiety you have ever felt.

When using rating scale questions it is important not to have too many categories. The scales that have been found to be most reliable and valid use 4 or 5 categories. Respondents find it difficult to make ratings when there are too many options, and their ratings have been found to be statistically unreliable. For instance, consider how difficult it would be to rate something on a scale of 1 - 5 versus a scale of 1 - 25. The difference between a 3 and 4 is fairly clear on a scale of 1 - 5. On a scale of 1 - 25 the difference between a 3 and 4 would be harder to decide, and that's why ratings become unreliable when there are too many choices. To maximize reliability and validity, you should only use the number of choices that you can specifically define. Never, sometimes, usually, and always are popular anchors for four point scales. Or Agree, slightly agree, slightly disagree, disagree. If you have more choices than you have anchors, you have too many choices!

Nonverbal scales

These are most commonly used for children and can be helpful with adults with intellectual disabilities, or that have poor language skills.

Nonverbal scales, use a picture rating system instead of a numeric rating system.

For example, below is an image of a nonverbal scale for rating pain in children

(image of six faces ranging from big smile to neutral to frown to rate severity of pain)

Based on my description in the previous section of the optimal number of response options, how many options should there be on the face scale pictured above? If you answered six, you are correct! What do the rest of the points mean? Is there really a meaningful choice between "hurts little bit" and "hurts little more"? No, there isn't. When you analyze responses from the scale above, you find that there are no more than six meaningful responses. So this serves as a reminder. If you create your own scale to measure something, don't use more response options than you can clearly define!!!

Semantic Differential Scales

Semantic differential scales are used to rate concepts, people, objects, behaviors, ideas, on a set of bipolar adjectives (that is, opposites...) on a 5 or 7-point scale.

For example, a company testing how people like their new spicy tortilla chip might have them complete a semantic differential rating scale using a set of adjectives such as the following:

good ___ ___ ___ ___ ___ ___ ___ bad

exciting ___ ___ ___ ___ ___ ___ ___ boring

flavorful ___ ___ ___ ___ ___ ___ ___ bland

While this may seem a strange way to rate things, 60 years of research says that anything can be rated this way, and that it is especially useful to rate along three dimensions of:

evaluation (good/bad, like/dislike, etc.)
activity (exciting/boring, slow/fast, active/passive), and
potency (flavorful/bland, strong/weak, large/small).

Graphic Rating Scale

Graphic rating scales have people place a mark (or move a computer cursor) to a point on a 100 mm line with a rating from 0 to 100

For example, in rating the quality of a store:

How would you rate the quality of our customer service?

Not good at all _______________________________ excellent

How likely are you to recommend us to a friend?

Not at all likely _______________________________ very likely

When using a paper and pencil, the researcher uses a ruler and measures the number of millimeters and assigns a score from 0 to 100. A computer can do the measuring for you when this type of scale is used online.

An advantage of this type of rating scale over a numeric is that people are less likely to have a response set of answering exactly the same to each question (e.g., writing the number 4 for every item on a rating scale from 1 - 5).

Response sets

One disadvantage to rating scales is that participants may answer with what is called a response set and answer every item or almost every item with the same answer. A common response set is to answer every or almost every item with 'neither agree nor disagree.' Sometimes it is because participants are in a hurry, and other times it is because they do not want to think about it much, and sometimes it is because they do not have a strong opinion and don't want to take the time to distinguish between, say, 'disagree somewhat' and 'neither agree nor disagree.'

Yea-saying and nay-saying, that is, agreeing or disagreeing with all the questions, are two common response sets.

Yea-saying and nay-saying can be avoided, or at least detected, by changing the wording of the questions so that a single opinion requires answering different questions in different ways. For example, one question might say "overall, I am satisfied with the economy" and another might read "I have few or no concerns about the economy"

Bad Questions = Bad Research

This is one of the most important things to remember. Poorly written questions will make the results of a survey difficult to interpret and can mislead respondents so that their answers don't necessarily reflect their views. Here are some tips for writing survey questions:

Don't use...

unfamiliar technical terms
vague or imprecise terms
ungrammatical sentence structure
overly complicated phrasing
misleading information in the question
loaded questions

For example

Have any of your grandparents had a cerebrovascular accident? (unfamiliar technical term)

Do you think the candidate running for office is good? (vague, imprecise terms)

Do you or don't you find compelling Mr. Smith's decision to go to Washington? (ungrammatical)

Do you doubt or not that the real winner of the 2000 election wasn't George Bush? (overly complicated phrasing)

Do you think that the proposed tuition increase is acceptable given that faculty will be getting very large raises? (misleading information)

Do you favor eliminating the wasteful excesses in the bloated state budget that are aimed at helping losers who can't help themselves? (loaded question)

Better questions:

Have any of your grandparents ever had a stroke?

Do you think Mr. John Smith (the candidate running for congress in your congressional district) will do a good job representing your interests?

How likely are you to vote for Mr. Smith in the upcoming congressional election?

Do you believe that George W. Bush deserved to win the 2000 presidential election?

Do you think that the proposed tuition increase of 5% is acceptable?

Do you favor eliminating some services for the poor in order to balance the state budget?

Constructing a survey

After you have written some great questions, you still have to construct your survey.

Many researchers have a panel of experts review their questions and make suggestions to eliminate or re-word some of the questions.

The order of the questions can make a difference in how people respond.

A few rules of thumb for constructing a survey are:

put demographic questions at the end of the survey
put potentially sensitive or embarrassing questions in the middle
group questions about the same topic or subtopic together
format such that the page is not too cluttered and the survey appears professional
make sure the reading level is appropriate (most aim for a 4th - 8th grade reading level for a general survey, but may be higher or lower depending on target group of respondents).

Administering a survey

There are many methods for administering a survey including:

written questionnaires
personal administration to groups or individuals
mail surveys
internet surveys

Make sure you become familiar with the pros and cons of each method.

And technology keeps advancing, so text messages and other new methods for collecting survey data are being developed all the time

One of the most famous surveys became famous because the results were so WRONG.

(image of president-elect Harry Truman holding newspaper with headline "Dewey Defeats Truman")

Pollsters used a telephone survey to see if voters preferred Thomas Dewey or Harry Truman in the 1948 presidential election. Their poll said Dewey would win.

There was so much confidence in the survey that even the Chicago Tribune printed and delivered newspapers with the headline "Dewey Defeats Truman"

What went wrong?!

In 1948 many people did not have a telephone. Their telephone survey missed people that did not have a phone and it turned out that those without a phone were more likely to vote for Truman. Also problematic was that they stopped polling several weeks before the election and did not take into account that undecided voters made up their minds (and in favor of Truman) and that some that initially favored Dewey switched their allegiance to Truman.

Could this happen today? Sure -- while most have the internet, 20% of households do not, and while many US adults are on Facebook, 30% are not -- any survey administered via a technology that many people do not have risks producing a biased sample.

See below for more on sampling.

Sampling: Selecting Research Participants

Most research involves sampling participants from a population of interest. The selection of research participants is a VERY important part of planning a research project. Some of the more common strategies will be reviewed below.

Population Vs sample

A population includes every member of the group of people (or animals) you want to study.

It is important to be clear about who your population of interest is so that you can make sure you have an adequate sample of participants in your study.

Suppose you want to study depression. How will you define your population of interest? Are you interested in all people who have ever been depressed, or only people who are presently depressed?

Suppose you decide you are going to study people who are presently depressed. Are you interested in All people with depression, or only children, or adults, or elderly adults?

Suppose you are interested in elderly people who are presently depressed, and you limit to residents of the United States. Now you have to specify how you will operationalize ‘depressed’ AND ‘elderly.’

Your population of interest will be very different if you operationalize ‘elderly’ as adults over age 60 versus adults over age 70.

Similarly, your population will look different if you operationalize ‘depressed’ as ‘being diagnosed as depressed by a licensed clinical psychologist who has administered the Structured Clinical Interview for DSM-IV” versus operationalizing depressed as “obtaining a score greater than or equal to 14 on the Beck Depression Inventory”.

It is impractical to study every member of a population. Even if it were possible, the cost would be prohibitively expensive for most research. Instead most research is conducted with a sample selected from the population of interest. Your sample is the group of individuals that actually participate in your study, and a sampling technique is the procedure you use to select them.

With proper sampling, statistical theory allows us to estimate what the entire population is like based on data obtained from a sample selected from that population.

A population may be one (in single subject research) to millions or even billions of members. Determining how many participants to recruit is one aspect of sampling.

Question: is this class of students a population or a sample?

I hope you answered, “it depends” or ”that is a trick question.”

When you complete course evaluations, or if I determine the average grade on a quiz, then you are a population of interest.

If someone was comparing grades in online versus to face-to-face psychology classes at UCF, or surveying US psychology majors’ understanding of research methods in psychology, or college students’ attitudes towards gun control or marijuana legalization, then you would be considered a sample.

confidence intervals, sampling error and sample size

Since samples are not exactly the same as the entire population a research outcome based on a sample is a close estimate of the population.

That estimate can be expressed in terms of a confidence interval. The difference between the true value of the population and the estimate based on the sample is called sampling error. A confidence interval gives us some information about the amount of sampling error.

You may have heard this lingo when hearing about poll results.

For instance, a pollster that has surveyed likely voters might report that "52% of a random sample of 800 registered voters indicated that they will vote for candidate A over candidates B & C. The survey has a margin of error of 3% using a confidence level of 95% ".

Here, the population of interest is registered voters - rather than all American adults, or all Likely voters

the sample is the 800 registered voters that they contacted

the sampling error is 3%

the confidence level is 95%

the confidence interval is 49 - 55%

Put another way, the pollster is reporting that after surveying 800 registered voters, they are 95% confident that 49% to 55% (or 52% +/- 3%) of all registered voters will vote for candidate A.

A larger sample will more closely resemble the population, and the sampling error will be smaller.

However, large samples are expensive and time consuming, so researchers determine the optimal sample size balancing their need for accuracy with practical constraints like the labor and time involved in gathering data.

Sampling techniques

Your textbook describes the difference between probability and non-probability sampling. Make sure you understand this distinction, and that you can name which procedures described below are examples of probability sampling and which are non-probability sampling procedures.

Simple random sampling

In simple random sampling each member of a population has an equal chance of being selected as a participant.

It is simple random sampling when the population is well defined, all members of the population are known, and a sample is selected through a random process.

One of the most common errors in student assignments of all types is incorrectly calling a convenience sample a random sample.

If your population of interest is third graders in Orange County public schools and you randomly select 10 third graders from the Castle Creek Elementary School, this is NOT a random sample. For it to be a random sample, all third graders in Orange County must have an equal chance of being selected. In this case, students at schools other than Castle Creek have a 0% chance of being selected. It would only be a random sample if your population of interest was third graders at Castle Creek Elementary School.

Stratified Random sampling

Stratified random sampling is used to make sure that the sample includes members of selected sub-groups of the population.

For example, if I wanted to make sure that a sample of students in the population of this class includes women and men, or juniors and seniors, or is ethnically diverse and in proportion to their number in the class, I would first ‘stratify’ the class into the identified groups and then randomly select participants from each group.

I would then select, say, eight women from all of the women and 2 men from all of the men. The reason for this is that if I just selected 10 students, there may not be any men. If the population were larger -- say, all psychology students in the country, and I was selecting 2,000 students I might not worry about this. However, with smaller samples members of small demographic groups may be missed with simple random sampling.

This method is especially useful when you want to compare groups and/or make sure that members of each sub-group are represented in the sample.

An advantage of this method is that all groups can be represented. A disadvantage is that, unless the sample is very large, the sub-sample may be too small for the results to generalize.

In practice, researchers often over-sample from the smaller group in order to ensure that the results generalize to that group (in the above example I might select 5 women and 5 men). However, here I either have two random samples (one of men and one of women) or a sample is no longer either truly random or truly representative of the class as a whole.

The best option always depends on the nature of the research questions being asked; that is, what does the researcher hope to find out?

On one study it may be important to the researcher to know what the entire group thinks, and in another case there may be interest in contrasting the views of women and men, or persons with different ethnic backgrounds, or incomes, etc.

Cluster Sampling

Cluster sampling is used when a group, or ‘cluster’, of individuals is selected all at once rather than selecting individuals one at a time. After a cluster is chosen, then all individuals in the cluster are included in the sample.

Everyone in this class might be regarded as a cluster if the population of interest were “students enrolled in psychology classes this semester,” or “UCF students.”

To go back to the example above, if my population of interest was third graders in Orange County, and I randomly selected Castle Creek Elementary School and Waterbridge Elementary School from a list of all elementary schools in orange county, and surveyed all of the third graders at each school, then I would be using cluster sampling.

Non-probability sampling

The sampling procedures described above are called probability sampling procedures because we can determine the probability that any member of the population of interest will be selected.

In non-probability sampling, the chances that any one individual will be selected is not known.

Haphazard/Convenience Sampling

This is the most common type of sampling even while it has the most problems.

It is common because - as the name implies – it is convenient.

I would be using convenience sampling if I were studying, say, attitudes of Americans towards congress, and I posted a note to all of you in the class that said “you can earn extra credit if you complete my survey about your attitudes towards congress.”

It would be a convenient way to get data, and the sample would differ from a sample selected RANDOMLY from the population of ‘all Americans.’

A convenience sample selected from this class would be younger and more educated and more likely to be female than ‘the average American.’ Individuals in this convenience sample are also individuals who are more likely to respond to requests to be a research participant than is the average person, which may or may not affect the outcome of the research. In other words, there are some external validity issues whenever convenience samples are used.

If, in order to study the population 'third graders in Orange County public schools', I surveyed 10 students randomly selected from Castle Creek Elementary, this would be a convenience sample.

Purposive Sampling

In purposive sampling the researcher is trying to obtain a sample of individuals that meet some predetermined criteria.

For example, a pollster standing outside of a mall asking everyone who passed by what they think about some issue would be using haphazard sampling.

In contrast, if she only asked women or only asked adults who appeared older than 60, then she would be doing purposive sampling.

Quota Sampling

Quota sampling is a form of convenience sampling where the composition of a convenience sample is controlled by establishing quotas.

For example, a pollster would be using quota sampling if he made sure he surveyed at 50 men and 50 women rather than just stopping the first 100 individuals

Notice that this is similar to stratified sampling, except it is a form of NON-PROBABILITY SAMPLING, while stratified sampling is a form of probability sampling. The example above is not probability sampling because the individuals that make up the sample do not have the same probability of being selected as other members of the population of interest.