I’m new to IPUMS and am super excited to try out the new API! So far everything has gone pretty smoothly—the instructions were really clear, and it’s great that they’re written using the python requests library since I’m also using that library to pull data!
So I was following the instructions and was able to successfully create a data request, and the status endpoint also works for me and I got the status complete. I also saw the request show up in IPUMS Login, and I was able to download it manually from my browser there.
However, I have not been able to successfully download it using the link returned by the status endpoint (in my case, https://data2.nhgis.org/extracts/019fae10-d475-0135-99f0-005056a35405/6/nhgis0006_csv.zip). I tried downloading it via requests.get, both using the {'Authorization': <My key>} header and also without the header (since the instructions don’t say that a header is required for this step). I get the following 404 response when trying to load that page:
<!DOCTYPE html>
<html>
<head>
<title>The page you were looking for doesn't exist (404)</title>
<style type="text/css">
body { background-color: #fff; color: #666; text-align: center; font-family: arial, sans-serif; }
div.dialog {
width: 25em;
padding: 0 4em;
margin: 4em auto 0 auto;
border: 1px solid #ccc;
border-right-color: #999;
border-bottom-color: #999;
}
h1 { font-size: 100%; color: #f00; line-height: 1.5em; }
</style>
</head>
<body>
<!-- This file lives in public/404.html -->
<div class="dialog">
<h1>The page you were looking for doesn't exist.</h1>
<p>You may have mistyped the address or the page may have moved.</p>
</div>
</body>
</html>
Do you know why I might be getting this error?
Thanks in advance—I’m really glad to see that this forum exists!
I’m very sorry, this is a known regression that got introduced earlier this month and we just discovered a few days ago - the URL for downloads recently changed but the API is still returning the old form. With the holiday break there’s been a little delay in getting it patched up, but we hope to fix it as soon as possible after we get back in the office after the break. In the meantime, the correct link actually appears on the NHGIS website if you go to your extracts history page and look at the link for downloading the data for that extract. You can then either retrieve it manually from the website or plug that URL into your code by hand for now. Sorry for the inconvenience!
Thanks! So would the link from the download page (https://live.nhgis.datadownload.ipums.org/web/extracts/nhgis/1533119/nhgis0015_csv.zip) work, as long as I include a header {'Authorization': <my key>}? That doesn’t seem to be working right now
Unfortunately, upon fixing the initial bug we have uncovered a second, more internal bug related to API key permissions. Once we get beyond the holiday break, we’ll have more engineers available to quickly fix the issue. Thanks for your patience, I expect a fix next week. Sorry for the continued inconvenience! At the moment, the only way to successfully retrieve an NHGIS extract is via the website.
We’re working on it as I type - sorry, this bug hit at a rough time right before the long University holiday break. We’re back to full strength this week and actively working towards a resolution. Thanks for your patience!
Happy to report this is now fixed. The URLs for the download files have changed, so please refresh those (make another call to get the status of your extract, which will return the new URLs).
Also, downloading an extract is now behind an authenticated API, so you’ll have to pass in your Authorization header to download your data. The code samples on the developer docs site have been updated to reflect this.
Thanks for everyone’s patience as we worked to resolve this issue. Please report back if you have any continued issues.