Institutionalization (gqtype in the ACS from IPUMS-USA) is used as a proxy for incarceration in prisons and jails in some academic studies (i.e. Neal and Rick, 2014 http://www.nber.org/papers/w20283.pdf), however female incarceration has shown a remarkable increase over this time period. Any ideas as to why gqtype does not show this increase? It is possible that other types of institutionalization decreased enough to offset the change, but seems unlikely.
It is likely that you do not see the increase as you expect due to the changing definition of what constituted group quarters. Furthermore, without the fine level of detail available for GQTYPE after 1980, it is impossible to say which type of institution group quarters someone belonged to in each of the samples. On thing to keep in mind if you are using institution as a proxy is to calculate the rate excluding those not in group quarters.
I’m unsure of the intent of your analysis, but you may want to look into what data the Bureau of Justice Statistics has available on incarcerations.