Calculating Design-Corrected Standard Errors for the National Longitudinal Survey of Youth, 1997

National Longitudinal Survey of Youth - 1997 Cohort

Calculating Design-Corrected Standard Errors for the National Longitudinal Survey of Youth, 1997

Overview

The National Longitudinal Survey of Youth, 1997 (NLSY97) is a combination of two area-probability samples. The Cross-Sectional (CX) sample is an equal-probability multi-stage cluster sample of housing units for the entire United States (every housing unit in the United States in 1997 had an equal probability of being in the CX sample). The Supplemental (SU) Sample is a multi-stage cluster sample of housing units that oversamples Hispanic and non-Hispanic Black youths; it is designed so that every eligible Hispanic and non-Hispanic Black youth in the United States in a housing unit in 1997 had an equal probability of being in the SU sample.

Since these samples are cluster samples, standard errors are larger for the NLSY97 than simple random sample calculations (calculated without correction for the design) would indicate. To correctly calculate standard errors, design variables must be used in statistical software. Without these design variables, statistical software will assume a simple random sample and underestimate standard errors. For a handful of variables selected below in Table 1, we see multiplier effects on standard errors (labeled DEFT) ranging from 1.30 to 1.62.

In order to facilitate the calculation of design effects, we provide two design variables for every Round 1 NLSY97 interview: VSTRAT and VPSU.  VSTRAT is the Variance STRATum while VPSU is the Variance Primary Sampling Unit.   The combination of VSTRAT and VPSU reflect the first-stage and second-stage units selected as part of the NORC National Sampling Frame. There are two second-stage units (VPSU) for each first-stage unit (VSTRAT).

First stage units in the NLSY97 are called Primary Sampling Units (PSUs), each of which is composed of one or more counties. The largest urban areas are selected with certainty to guarantee their representation in NSLY97. Second-stage stage units in the NLSY97 are called segments, each of which is one or more Census-defined blocks. The first-stage and second-stage units are selected with probabilities proportional to size (housing units for the CX sample; minority youths for the SU sample), and the sample housing units (third-stage units) are then selected to be an equal-probability sample.

To create the variables VSTRAT and VPSU, we recode the PSUs and segments, depending on whether the PSU was selected with certainty. Certainty PSUs are considered strata, so all the segments in one certainty PSU are in one VSTRAT value, with segments divided so that half are assigned to VPSU = 1 while the other half are assigned to VPSU = 2.  Some certainty PSUs are large enough to be divided into multiple VSTRAT values with up to twenty segments in one VSTRAT value (ten in each VPSU). Non-certainty PSUs are paired into one VSTRAT value with one PSU assigned to VPSU = 1 while the other PSU is assigned to VPSU = 2. It is rare, but possible, for PSUs to be combined in one VPSU. This strategy was designed by Kirk Wolter. 

Here is sample Stata code to analyze the variable ANALYSISVAR within an NLSY97DATAFILE with the appropriate weight variable for the analysis, WTVAR:

use NLSY97DATAFILE.dta,clear

svyset [pweight=WTVAR] , strata(vstrat) psu(vpsu) singleunit(scaled)

svy: mean ANALYSISVAR                 //mean for continuous variables

svy: proportion ANALYSISVAR        //proportion for categorical variables

estat effects                                  // design effects--this generates the DEFF and DEFT

svy, subpop (if SUBGROUP==1): mean ANALYSISVAR    // mean within a subpopulation

svy: tabulate ANALYSISVAR  //one way table

The results generated by running this code on select NLSY97 variables are shown in Table 1. They report design-corrected standard errors as well as standard errors assuming simple random sampling as would be estimated in the absence of these design variables.

Table 1: Variance estimation for selected NLSY97 variables

               

 

Variable

Mean/            proportion

Sample size

Estimate

Design-corrected std error

SRS std error

DEFF

DEFT

 

Gross family income 2009 from 2010 interview [T5206900]

mean

6527

64858.5

923.831

773.64

1.728

1.314

 

ASVAB score for Math (percentile) [R9829600]   

mean

7093

50.410

0.638

0.367

3.447

1.857

 

ASVAB score for Math (percentile) for females [R9829600, R0536300]

mean

8102

51.508

0.705

0.509

2.222

1.491

 

Weeks worked in 2008 [Z9061800]   

mean

8011

40.021

0.270

0.224

1.700

1.304

 

Ever received a bachelor’s degree or higher as of 2011 interview [T6657200]

proportion

7398

0.300

0.009

0.006

2.614

1.617

 

Never received a high school diploma as of 2011 interview [T6657200]

proportion

7398

0.199

0.007

0.005

2.368

1.539

 

Lived with 2 parents (at least 1 bioparent) in Round 1 [R1205300]

proportion

8953

0.675

0.007

0.005

2.216

1.489

 

Tabulations use the Round 1 Cumulating Cases Sampling Weight [R1236101]

 

One thing to note is that the sample size for math ASVAB scores for females is greater than the sample size for math ASVAB scores for all respondents. Though the estimate uses only the 3503 observations of females with non-missing data for ASVAB scores, the variance calculation uses information from all observations in the data whether or not they have valid values for the specific variables. (See here for more information.) 

In SAS, one would use PROC SURVEYFREQ to calculate the design-corrected standard errors. SPSS is menu-driven, so no code is given here, but you can create design-corrected standard errors within SPSS using the Complex Samples add-on.

Potential Problems with Sparse Data

If you are calculating design-corrected standard errors using a subsample of the NLSY97 data set and/or using a variable that has a large number of observations with missing values, you may receive a message such as this:

STATA error handling: “missing standard error because of stratum with single sampling unit”

VSTRAT and VPSU were created so that there was a minimum of 7 NLSY97 respondents within a VSTRAT/VPSU cell.  However, if all respondents within a cell are missing on a variable, it will be impossible to calculate the standard error. If the dataset is subset (to males or females, for example), this error becomes more likely to happen.

The best workaround is to merge two VSTRATA together to eliminate this problem (the VSTRATA are ordered so that similar VSTRATA are numerically consecutive). To diagnose the problem, run a frequency of the data by VSTRAT/VPSU and look for VSTRAT values with only one VPSU with respondents. Here is an example:

VSTRAT

VPSU

# of cases

…

…

…

x-1

1

3

x-1

2

5

X

1

4

x+1

1

6

x+1

2

4

…

…

…

The error occurs because VSTRAT = x has four cases with VPSU=1 but none with VPSU=2. This prevents Stata from calculating the variance for this strata (it has nothing to compare VPSU=1 with). The cases were sorted by VSTRAT and VPSU so that the most similar VSTRATA are numbered consecutively and the most similar cases are always within the two VPSU values of one VSTRAT value. The easiest solution is therefore to make room for VSTRAT x within VSTRAT x-1 by combining the two VPSUs within VSTRAT x-1. Then, the VSTRAT x cases are moved to the other VSPU value within VSTRAT x-1. Note that VSTRAT x-1 is chosen instead of VSTRAT x+1 in this example because VSTRAT x-1 has fewer total cases (8) than VSTRAT x+1 (10). In some, but not all cases, the VPSU 1 cases in VSTRAT x are “more similar” to the VSTRAT x-1 cases than the VSTRAT x+1 cases. Here are the two programming steps:

1. If VSTRAT = x-1 and VPSU = 2 then VPSU=1

2. If VSTRAT = x then VSTRAT=x-1 and VPSU=2.

Here is the revised frequency:

VSTRAT

VPSU

# of cases

…

…

…

x-1

1

8

x-1

2

4

x+1

1

6

x+1

2

4

…

…

…

This eliminates the “stratum with a single sampling unit.”  In severe cases of data subsets, this step may be required more than once, although this may also indicate that the “clustering” has been removed (by using less than 10 percent of the total sample, for example).

Author: Steven Pedlow, NORC at the University of Chicago

Revised: October 1, 2014