I would like to merge IPUMS CPS records to unharmonized “raw” Basic Monthly Files back to 1976. There are a number of forum questions that relate to merging records starting in (say) 1989, but I couldn’t find any questions about merging all the way back to 1976.
A sequential merge does not seem possible, because the order of CPS records in the unharmonized data (available from NBER) doesn’t seem to match the order of the records in the IPUMS data, as determined by the IPUMS-supplied variables SERIAL and PERNUM.
There are a number of unharmonized variables in the IPUMS data that could be used as merging keys, and I have found that using many of them can eliminate all duplicates and thus allow for a potential one-to-one match with the raw CPS files. This method works well starting in 1978, but for 1976 and 1977, there are still some duplicates even using a long list of variables (see below).
I am wondering whether I am missing some other way of identifying individual records in the CPS data that would be more efficient. For example, is there another way besides using SERIAL and PERNUM that indicates the order of the IPUMS records on the original CPS tape?
Here is the list of variables I am using to identify individual CPS records in 1976 and 1977:
- uh_hhid_b1
- huhhnum (the harmonized version appears to be populated more often)
- uh_lineno_b1
- uh_grdhi_b1
- uh_grdcom_b1
- uh_ind_b1
- uh_occ_b1
- uh_vet_b1
- uh_untno_b1
- uh_docno_b1
- uh_marstat_b1
- wtfinl (using harmonized version – same as unharmonized version)
- statefip (using harmonized version – same as unharmonized version)
- uh_intrv_b1
- uh_day_b1
- uh_cpschk_b1
- uh_hhdseq_b1
Here are the variables I use after 1977, which all seem to work well:
- For 1978-1983: hhid hhnum lineno mis month stfip race sex age
- For 1984-1993: hhid hhnum lineno mis month
- For 1994-1995: hrhhid myhhid2 pulineno hrmis gestfips hrmonth
- For 1996-2004: hrhhid myhhid2 pulineno hrmis
- For 2005: hrhhid hrhhid2 pulineno hrmonth
- For 2006-present: hrhhid hrhhid2 pulineno hrmis
The IPUMS Working Paper, Extending Current Population Survey Linkages: Obstacles and Solutions for Linking Monthly Data from 1976 to 1988, Version 2, describes in detail the challenges that IPUMS researchers have encountered in processing and linking these early CPS samples. The working paper notes that supplement-containing files in this period often contain partially or fully duplicated records. Additionally, there are a number of records in this period where the original household ID values (HHID) will not match the IPUMS CPS unharmonized variable UH_HHID_B1.
Specifically, UH_HHID_B1 has been edited for supplement-containing files in 1977. This is a result of a change to the method for generating household identifiers in the CPS in months that had a topical supplement that makes it impossible to link records between supplement-containing and basic monthly-only files in 1977. For these months, IPUMS CPS offers data from the supplement file rather than the BMS (though BMS variables contained on the supplement record are available). This affects six months in 1977: January, May, June, August, October, and December. Since IPUMS CPS offers both the March BMS and ASEC supplements as separate files, this does not affect the March BMS data. In each of these months, there are more supplement records than BMS records. However, due to changes in how linking keys were constructed we are unable to identify which records do not overlap between the two versions of the files.
Given this information, you might try adopting the following approach. First, drop any duplicate records where the values completely match. Second, merge only records in sample months that do not include a topical supplement. This merge should be possible using UH_HHID_B1 and the person line number UH_LINENO_B1. Finally, merge records in sample months that include a topical supplement. This merge should exclude the household and line number identifiers and instead focus on other available variables such as demographics and employment information.
Thanks very much – this is hugely helpful!