Measurement Concepts

Welcome to the chapter focused on two of my favorite subjects, reliability and validity! Everything we do in psychology begins and ends with reliability and validity. The four targets below provide a great introduction to this subject. Think of these targets as a measure of depression. Each hole represents the score of a depressed person on this hypothetical measure of depression.

reliability and validity.jpg

The image above of four 'bulls eye' targets with different patterns of accurate hits is a good visual illustration of reliability and validity. If the same person completes the measure of depression over and over, and the measure is reliable and valid, you would expect to see the person get a similar score each time, and you would expect those scores to be a good measure of the construct of depression. In other words, if the measure has good reliability and validity, you expect to see a near bull's eye every time as depicted in the fourth target. However, if the person gets a similar score each time but the target is not actually a good measure of depression, you have good reliability but poor validity, as depicted in the first target. If you have a good measure of depression but the person gets a substantially different score each time, you have a valid measure with poor reliability, as depicted in the second target. Worst of all is when you have a poor measure of depression and the same person gets substantially different scores each time. This would be poor reliability and poor validity, as depicted in the third target.

Visit this webpage for more info about the graphic: http://www.documentingexcellence.com/stat_tool/reliabilityvalidity.htm

It is an excellent way to compare reliability and validity in an easy to understand way.

If it doesn't make sense immediately, go back and view it again after reading chapter 5 in your textbook! As I mentioned above, reliability and validity are the foundation of everything we do in psychology, so you need to have a good understanding of these concepts. If you don't have reliable and valid measures of a particular construct, you can't measure that construct well enough to learn anything about it! I see problems with reliability and validity every day in alcohol and drug research and clinical work. I'll give you a simple example involving a real-life patient. A student clinician asked a new patient how much alcohol they drank every day, and the patient replied "two drinks." To most of us, two drinks seems very reasonable, and definitely not a problem for an adult who is 21 years of age or older. But what exactly is "two drinks?" To some people, it's two 12-ounce beers, and that would be an example of two "standard" drinks. To this patient, however, a drink involves putting a few ice cubes in a 16-ounce glass and then filling it up with 80 proof rum, and they were drinking two of those every night. For this patient, their "two drinks" were actually about 16 standard drinks. Getting back to reliability and validity, asking someone how many drinks they had last night is a reliable measure because they will probably give you the same answer over and over. However, unless you ask them to specify exactly what they drank and how much, the validity of the answer will be very low (because "2 drinks" could be 2 standard drinks or it could be 16!).

Read on for more information related to Chapter 5...

Reliability

Reliability refers to the degree of stability or consistency of a measure of behavior. Just as there are different types of validity there are different types of reliability that are reviewed in chapter 5. Make sure you can identify the different types of reliability described below.

Any measure can be thought of as including a true score + measurement error.

True score - the 'real' score on a measure
Measurement error - any part of the score that is not due to the 'true score'

The more reliable a measure, the less measurement error and the closer the obtained score is to the true score.

For instance, a reliable oven thermometer will reflect the true temperature of the oven, while an unreliable one may give an incorrect temperature that reflects the true temperature plus or minus some number of degrees that reflect measurement error in the thermometer. Measures of behavior are also subject to measurement error. For instance, some measures of depression may reflect depression but some error is introduced because some symptoms of depression overlap with symptoms of medical problems, so ones score may reflect a true score (depression) + measurement error due to symptoms of a medical problem such as poor sleep and changes in appetite.

Reliability can be obtained by carefully constructing questions, carefully calibrating equipment, carefully training people to administer the measure, and testing it on a large number of participants, and across time, and under different conditions. There are several ways to establish that a measure is reliable.

test-retest reliability - if you take a personality test or an intelligence test on one day you should obtain a similar score if you take the same test a few days later. Personality and intelligence are stable over a few days. Test retest reliability is less useful for measures of constructs that change over time, such as mood or hunger or alertness.
internal consistency reliability - if a test is measuring a single construct then your score on one item of a measure should be correlated with your score on other items on the same measure. For instance, imagine a 20 item test on your knowledge of chapter 5. Most likely, your score on the first 10 items will be similar to your score on the last 10 items. If it is not, then the test may be unreliable. Your textbook describes two different ways to compute internal consistency reliability of a measure..
inter-rater reliability - will two different raters administering the same measure obtain the same result? This is not a concern when it comes to tests such as a multiple choice questionnaire, and it is a concern on measures such as intelligence tests or diagnostic interviews where a rater rather than the examinee records a score. Training of examiners, providing very clear instructions, and standardization of test procedures (e.g., always reading the exact same instructions to the examinee) are three ways to increase inter-rater reliability.

Construct validity of measures

Construct validity is about whether or not a measure actually and accurately measures what it purports to measure. Reliability is the first step towards validity. A reliable test may not be valid, but an unreliable test will never be valid. Imagine I asked everyone in the class to take a test of college level math ability and the test included three items what is 2+2, what is 4 + 3, and what is 5-1. The test would be reliable; almost everyone would obtain the same score of 3 out of 3 every time. And it would not be valid because it does not measure what it purports to measure.

There are several indicators of construct validity

Face validity - technically, it is not truly a measure of validity, but the name has stuck…A face valid measure looks like it measures what it claims to measure. The Rorschach inkblot test has low face validity - to many skeptics it does not look like it would measure personality. The MMPI with 567 true/false questions looks more like a measure of personality and thus has high face validity. Face validity affects the attitudes of test takers and persons that may utilize (or ignore) test results.

Another example of low face validity, Mood Rings.

mood ring.jpg

The mood rings pictured above have low face validity -- they don't look like they would measure personality! In contrast, the MMPI-2 pictured below does have high face validity -- the questions look like they might evaluate personality.

Content validity -Does the content of the measure reflect the universe of content of the construct it purports to measure? This is less complicated than it sounds. For instance, a measure of depression should ask about sadness, and changes in appetite and sleep, and thinking about death, and enjoyment of activities, etc. If a measure only asked about sadness then it would not be a valid measure of depression. A test of basic math that only included addition and division questions would have poor content validity, as it would also need subtraction and multiplication questions.

Predictive validity The measure predicts behavior, for instance, the score on a paper and pencil measure of extroversion predicts behavior in a social setting. The scores on the SAT predict college grades, the scores on a leadership potential test predict management skill..

Concurrent validity - Are scores obtained on the measure consistent with already established measures of the same construct? For example, are scores on my new measure of adult intelligence consistent with scores on the Wechsler Adult Intelligence Scale?

In the examples above of mood rings and the MMPI-2, mood rings have low face validity, content validity, predictive validity and concurrent validity. In contrast, all types of validity are high with regard to the MMPI.

However, other measures may be high with regard to one type of validity and low with regard to another. For example, the "Tom Test of Malingered Memory" is a psychological test used to determine if people are faking memory problems (people sometimes fake neurological problems in committing fraud such as filing bogus personal injury lawsuits, or applications for disability benefits).

The test has HIGH face validity...it looks like a test of memory

It has LOW content and concurrent validity as a test of memory...the score on this test is not related to scores on other measures of memory.

It has HIGH predictive validity...the score is used to predict of people are faking or exaggerating their problems with memory.

Measurement is important and validity and reliability are important aspects of measurement. New measures are created all the time. Often with the aim of being shorter, or easier or less costly or less invasive to administer. Among the first thing someone creating a new test does is make sure scores on the new measure are very highly correlated with other established measures of the same construct.

convergent and discriminant validity - The measure is correlated with related measures and uncorrelated with unrelated measures. For instance, we would expect a measure of intelligence to be correlated with - but not identical to - a measure of years of education (convergent validity), and uncorrelated with a measure of extroversion (divergent validity).

Reactivity of Measures

A measure is reactive if a persons knowledge that they are being measured affects their performance. If a manager is holding a clipboard and measuring how much attention workers are paying to their job they may perform better because they are reactive to the boss' presence. A more unobtrusive/less reactive way to measure their work behavior might be to have someone other than a manger evaluate them, or to install a camera, or the manager may simply need to keep the clipboard out of sight.

Clinical psychologists and others interested in behavior change use reactivity to their advantage in order to modify behavior. Ask a dieter to monitor her calorie intake and she will eat less. Ask a smoke to record each cigarette smokes and she will smoke less. Ask a nail biter to record each instance of nail biting and he will bite his nails less.

This is good news for behavior change specialists and not so good for psychometricians. Reactivity may affect the behavior of interest, but the psychometrician wants a measure that assesses behavior accurately rather than a measure that increases or decreases behavior while it is being measured.

Test yourself -- is a child more or less likely to take a cookie that they have been told not to if:

A) they know that you are observing them

B) they do not know that you are observing them

If you answered "A", then that means observation has high reactivity when used as way to measure cookie theft!

How might you measure cookie-theft in a way that is not so reactive?

Measurement Scales

level of measurement.gif

(image of steps with descriptions of nominal, ordinal, interval and ratio measurement scales)

The scale of a measure is the set of categories that one can use the measure for as a means of classifying individuals with respect to the construct being measured. For a simple example, a measure of sex classifies individuals a male or female. A scale of temperature would be used to classify things as hot or cold, a scale of IQ score points would be used to classify intelligence, etc.

Measures can be described as having a nominal, ordinal, interval or ratio scale

Nominal - also called 'categorical' - the scale is in terms of membership in a group, for instance, the answer to the question "which political party do you belong to' has an answer that is on a nominal scale. The answer is usually a 'name' of a category or group (e.g., democrat, republican, socialist, libertarian) and it cannot easily be answered in terms of a number. Gender, ethnicity, and handedness are nominal measures.

Ordinal - categories are organized sequentially but the difference between on group and the next is not necessarily equal. For instance, if we rate the top 40 songs, we can't say for sure that the 10th song is twice as popular as the 20th song.

Interval and ratio- all categories are the same size, so we can measure the distance between them on a scale. A ratio scale has a true Zero and an interval has a more arbitrary zero (for example, take temperature -- the Kelvin scale has an absolute value of zero, while the Farenheit scale has an arbitrary value of zero).

The scale of the measures is important because it is related to how easily two or more measures can be compared, what statistical tests can or cannot be used to analyze results, and to some extent, how the results may be interpreted.