# Error when reading atus data using ipumspy **URL:** \ **Category:** TIME USE\ **Created:** [June 6, 2024, 2:36am UTC](https://forum.ipums.org/t/error-when-reading-atus-data-using-ipumspy/5928 "2024-06-06T02:36:23Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Marcus\_Esteban](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Marcus\_Esteban](https://forum.ipums.org/u/Marcus_Esteban)\ **Post date:** [June 6, 2024, 2:36am UTC](https://forum.ipums.org/t/error-when-reading-atus-data-using-ipumspy/5928/1 "2024-06-06T02:36:23Z")
I tried to read an atus extract i requested on the website using ipumspy and used the suggested read code. These are my exact outputs. > codebook = readers.read\_ipums\_ddi(“/Users/marcusesteban/Documents/Data/atus\_00001\_ddi.xml”) > /opt/miniconda3/envs/default/lib/python3.11/site-packages/ipumspy/readers.py:49: CitationWarning: Use of data from IPUMS is subject to conditions including that users should cite the data appropriately. > See the `ipums_conditions` attribute of this codebook for terms of use. > See the `ipums_citation` attribute of this codebook for the appropriate citation. > warnings.warn( > atus\_df = readers.read\_microdata\_chunked(codebook, filename = “/Users/marcusesteban/Downloads/atus\_00001.dat.gz”, chunksize=1000) > ab\_df = pd.concat([ab\_df[ab\_df[‘STATEFIP’].isin([8, 30, 48])] for ab\_df in atus\_df]) > Traceback (most recent call last): > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/arrays/integer.py:51 in \_safe\_cast > return values.astype(dtype, casting=“safe”, copy=copy) > TypeError: Cannot cast array data from dtype(‘float64’) to dtype(‘int64’) according to the rule ‘safe’ > The above exception was the direct cause of the following exception: > Traceback (most recent call last): > Cell In[60], line 1 > ab\_df = pd.concat([ab\_df[ab\_df[‘STATEFIP’].isin([8, 30, 48])] for ab\_df in atus\_df]) > Cell In[60], line 1 in > ab\_df = pd.concat([ab\_df[ab\_df[‘STATEFIP’].isin([8, 30, 48])] for ab\_df in atus\_df]) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/ipumspy/readers.py:413 in read\_microdata\_chunked > yield from \_read\_microdata( > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/ipumspy/readers.py:167 in \_read\_microdata > yield from (\_fix\_decimal\_expansion(df).astype(dtype) for df in data) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/ipumspy/readers.py:167 in > yield from (\_fix\_decimal\_expansion(df).astype(dtype) for df in data) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/generic.py:6226 in astype > res\_col = col.astype(dtype=cdt, copy=copy, errors=errors) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/generic.py:6240 in astype > new\_data = self.\_mgr.astype(dtype=dtype, copy=copy, errors=errors) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/internals/managers.py:448 in astype > return self.apply(“astype”, dtype=dtype, copy=copy, errors=errors) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/internals/managers.py:352 in apply > applied = getattr(b, f)(\*\*kwargs) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/internals/blocks.py:526 in astype > new\_values = astype\_array\_safe(values, dtype, copy=copy, errors=errors) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/dtypes/astype.py:299 in astype\_array\_safe > new\_values = astype\_array(values, dtype, copy=copy) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/dtypes/astype.py:230 in astype\_array > values = astype\_nansafe(values, dtype, copy=copy) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/dtypes/astype.py:95 in astype\_nansafe > return dtype.construct\_array\_type().\_from\_sequence(arr, dtype=dtype, copy=copy) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/arrays/masked.py:132 in \_from\_sequence > values, mask = cls.\_coerce\_to\_array(scalars, dtype=dtype, copy=copy) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/arrays/numeric.py:258 in \_coerce\_to\_array > values, mask, \_, \_ = \_coerce\_to\_data\_and\_mask( > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/arrays/numeric.py:214 in \_coerce\_to\_data\_and\_mask > values = dtype\_cls.\_safe\_cast(values, dtype, copy=False) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/arrays/integer.py:57 in \_safe\_cast > raise TypeError( > TypeError: cannot safely cast non-equivalent float64 to int64 # I tried the same line again and this happened > ab\_df = pd.concat([ab\_df[ab\_df[‘STATEFIP’].isin([8, 30, 48])] for ab\_df in atus\_df]) > Traceback (most recent call last): > Cell In[61], line 1 > ab\_df = pd.concat([ab\_df[ab\_df[‘STATEFIP’].isin([8, 30, 48])] for ab\_df in atus\_df]) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/util/\_decorators.py:331 in wrapper > return func(\*args, \*\*kwargs) > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/reshape/concat.py:368 in concat > op = \_Concatenator( > File /opt/miniconda3/envs/default/lib/python3.11/site-packages/pandas/core/reshape/concat.py:425 in **init** > raise ValueError(“No objects to concatenate”) > ValueError: No objects to concatenate ---
**Author:** ![renae](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@renae](https://forum.ipums.org/u/renae)\ **Post date:** [June 6, 2024, 6:36pm UTC](https://forum.ipums.org/t/error-when-reading-atus-data-using-ipumspy/5928/2 "2024-06-06T18:36:40Z")
Hi Marcus, You’re running into a bug in `ipumspy` wherein there is some floating point data that is incorrectly designated as an integer by `ipumspy` and `pandas` doesn’t like it! This bug will be fixed in the next release of the `ipumspy` library, but in the meantime you can get around this issue by creating a dict of data types, fixing designating the `WT06` variable as floating type, and passing this dict to the `read_microdata()` method. Below is some example code that you can adapt by replacing “padded\_id” with your atus extract number: ```auto atus_ddi = readers.read_ipums_ddi(f"atus_{padded_id}.xml") dtypes_fixed = atus_ddi.get_all_types(type_format="pandas_type") # this is the offending variable dtypes_fixed.update({"WT06": pd.Float64Dtype()}) df_a = readers.read_microdata( atus_ddi, f"atus_{padded_id}.dat.gz", dtype=dtypes_fixed ) ``` Note that this example does not read the data in chunks. However, because the ATUS files are relatively small, you should not see any meaningful difference in performance by reading in your entire extract and then filtering the data frame down to your states of interest. Hope this helps! ---
**Author:** ![Marcus\_Esteban](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Marcus\_Esteban](https://forum.ipums.org/u/Marcus_Esteban)\ **Post date:** [June 6, 2024, 7:36pm UTC](https://forum.ipums.org/t/error-when-reading-atus-data-using-ipumspy/5928/3 "2024-06-06T19:36:41Z")
Hey renae, Thanks for the help. My file though is all the years so it is taking a long time to load. Do you have a chunked solution? Thanks again! ---
**Author:** ![renae](https://avatars.discourse-cdn.com/v4/letter/r/ecd19e/32.png) [@renae](https://forum.ipums.org/u/renae)\ **Post date:** [June 6, 2024, 8:03pm UTC](https://forum.ipums.org/t/error-when-reading-atus-data-using-ipumspy/5928/4 "2024-06-06T20:03:10Z")
Hi Marcus, Sure, you can pass the `dtype` kwarg to the `read_microdata_chunked()` method as well. That would look something like this: ```auto iter_atus = readers.read_microdata_chunked( atus_ddi, chunksize=1000, dtype=dtypes_fixed ) df_a = pd.concat([df[df["STATEFIP"] == 27] for df in iter_atus]) ``` Hope this helps! ---
**Author:** ![Marcus\_Esteban](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Marcus\_Esteban](https://forum.ipums.org/u/Marcus_Esteban)\ **Post date:** [June 6, 2024, 10:31pm UTC](https://forum.ipums.org/t/error-when-reading-atus-data-using-ipumspy/5928/5 "2024-06-06T22:31:23Z")
Thanks, it works but still takes a long time.