Understanding Psychometrics: Reliability, Validity, and Test Construction

Why Psychometrics Matters for Counselors

Psychometrics is the science of psychological measurement. For counselors, understanding psychometric principles is essential for selecting appropriate assessment instruments, interpreting test results accurately, and communicating findings to clients and other professionals. The NCE dedicates a significant portion of its content to assessment and testing, making psychometrics a critical study area.

Every time you administer a screening tool, interpret a standardized test score, or use an outcome measure, you are applying psychometric principles. Understanding these principles helps you distinguish between good and poor assessment instruments, interpret results within appropriate limitations, and make sound clinical decisions based on assessment data.

Reliability

Reliability refers to the consistency and stability of test scores. A reliable test produces similar results under consistent conditions. If a test is not reliable, its scores are influenced more by random error than by the construct being measured, making the results untrustworthy.

Types of Reliability

Test-Retest Reliability: This measures the stability of scores over time. The same test is administered to the same group on two occasions, and the correlation between the two sets of scores is calculated. High test-retest reliability indicates that the test produces consistent results over time. This type is most appropriate for measuring stable traits rather than fluctuating states.

Internal Consistency: This measures the degree to which items on a test measure the same construct. Cronbach's alpha is the most commonly reported measure of internal consistency. A Cronbach's alpha of .70 or higher is generally considered acceptable for research purposes, while .80 or higher is preferred for clinical decision-making.

Split-Half Reliability: The test is divided into two halves (often odd-numbered and even-numbered items), and the correlation between the halves is calculated. The Spearman-Brown formula is used to estimate the reliability of the full test from the split-half correlation.

Inter-Rater Reliability: This measures the degree of agreement between two or more raters who independently score the same responses. High inter-rater reliability is essential for assessments that involve subjective scoring, such as behavioral observations or projective tests.

Factors Affecting Reliability

Test length affects reliability: longer tests tend to be more reliable than shorter ones. Test score variability also matters: reliability is higher when there is greater variability in scores. Guessing can reduce reliability, particularly on tests with few answer choices. Testing conditions (noise, fatigue, distraction) can introduce error and reduce reliability.

Validity

Validity refers to the degree to which a test measures what it claims to measure. A test can be reliable without being valid (it consistently measures something, but not the right thing), but it cannot be valid without being reliable. Validity is the most important property of an assessment instrument.

Types of Validity

Content Validity: The extent to which test items represent the full domain of the construct being measured. Content validity is typically established through expert review of test items against a content specification or blueprint. It is particularly important for achievement tests and licensing examinations.

Criterion-Related Validity: The extent to which test scores relate to an external criterion. Two subtypes exist. Concurrent validity compares test scores to a criterion measured at the same time (for example, comparing a new depression scale to an established depression measure). Predictive validity assesses whether test scores predict future performance or behavior (for example, whether GRE scores predict graduate school success).

Construct Validity: The broadest and most fundamental type of validity, construct validity assesses whether a test actually measures the theoretical construct it claims to measure. Evidence for construct validity includes convergent validity (correlation with measures of related constructs) and discriminant validity (lack of correlation with measures of unrelated constructs).

Face Validity: Whether a test appears to measure what it claims to measure. While face validity is not a true psychometric property, it affects test-taker acceptance and motivation. A depression inventory that asks about depression symptoms has high face validity.

Standardization and Norms

Standardized tests are administered and scored according to uniform procedures. Standardization ensures that differences in scores reflect actual differences in the construct being measured rather than differences in testing conditions.

Norms provide a frame of reference for interpreting individual scores by comparing them to the performance of a normative group. The quality of norms depends on the size, representativeness, and recency of the normative sample. When using a test, consider whether the normative sample is appropriate for the client you are testing.

Standard Scores and Score Interpretation

Common score reporting formats include:

• Standard scores: Express performance in standard deviation units from the mean. Common scales include z-scores (mean = 0, SD = 1), T-scores (mean = 50, SD = 10), and deviation IQ scores (mean = 100, SD = 15).
• Percentile ranks: Indicate the percentage of scores in the normative sample that fall below a given score. A percentile rank of 75 means the individual scored higher than 75% of the normative group.
• Grade and age equivalents: Compare performance to the average performance of individuals at a specific grade or age level. These scores are widely used but frequently misinterpreted and should be used with caution.

Psychometrics on the NCE

The NCE tests your understanding of reliability types and their appropriate applications, validity types and how to evaluate them, standardization and norms, basic score interpretation, and the ethical use of assessment instruments. Be able to identify which type of reliability or validity is described in a scenario and know the relationship between reliability and validity. Understanding these concepts is fundamental to the assessment and testing content area of the exam.