Sood and colleagues are to be commended on the ambition of their evaluation of a government health insurance scheme for people below the poverty line (BPL) in Karnataka, India.1 With some notable exceptions,2 3 there are few opportunities to exploit randomisation in the evaluation of large-scale health financing reforms and thus quasi experimental methods must be relied upon. In this regard, the increasing number of publications in medical journals applying econometric methods to study impacts of such schemes are to be welcomed. Nevertheless, the findings of this study should be interpreted with a number of limitations in mind, particularly given the impressive magnitude of the mortality effect reported.
First, the empirical approach used in the study is more accurately described as matching, the limitations of which are well known.4 A regression discontinuity design that uses a geographical boundary as the assignment variable is based on the idea that there is a discontinuity in the treatment variable (insurance coverage) either side of a “random” boundary but no other discontinuities in relevant variables (household characteristics, community factors, policy environment) conditional on distance to the boundary.5 The study neither demonstrates the discontinuity in insurance coverage6 despite investment in primary data collection nor does it control for distance or geographical location in any way.7 8
Second, the geographical border is far from “arbitrary.” The paper suggests that the insurance scheme was implemented in such a way that the boundary in fact divides different districts. We must therefore ask “what are all the things differing between the two regions other than the treatment of interest?”5 In India, districts are the administrative unit charged with the implementation of health policy, not to mention the administration of the BPL system, and variation in such implementation (due to differences in management, leadership, budget, health personnel etc) must be expected. Yet the analysis is unable to control for such differences between districts. A more credible alternative with the available data would be a difference-in-difference approach – that is, a comparison of poor and non-poor households between treatment and control. It would provide an opportunity to control for geographical location.
Finally, it is hard to reconcile the large mortality effect with the modest (sometimes borderline significant) utilization results. Without substantive evidence of a large increase in hospitalization, it is premature to read too much into the mortality finding.
1. Sood N, Bendavid E, Mukherji A, Wagner Z, Nagpal S, Mullen P. Government health insurance for people below poverty line in India: quasi-experimental evaluation of insurance and health outcomes. BMJ 2014;349:g5114.
2. Baicker K, Taubman SL, Allen HL, Bernstein M, Gruber JH, Newhouse JP, et al. The Oregon experiment--effects of Medicaid on clinical outcomes. N Engl J Med 2013;368(18):1713-22.
3. King G, Gakidou E, Imai K, Lakin J, Moore RT, Nall C, et al. Public policy for the poor? A randomised assessment of the Mexican universal health insurance programme. Lancet 2009;373(9673):1447-54.
4. Imbens GW, Wooldridge JM. Recent Developments in the Econometrics of Program Evaluation. Journal of Economic Literature 2009;47(1):5-86.
5. Lee DS, Lemieux T. Regression Discontinuity Designs in Economics. Journal of Economic Literature 2010;48(2):281-355.
6. Card D, Dobkin C, Maestas N. Does Medicare Save Lives? Quarterly Journal of Economics 2009;124(2):597-636.
7. Black SE. Do Better Schools Matter? Parental Valuation of Elementary Education. Quarterly Journal of Economics 1999;114(2):577-99.
8. Dell M. The Persistent Effects of Peru's Mining Mita. Econometrica 2010;78(6):1863-1903.
Rapid Response:
Sood and colleagues are to be commended on the ambition of their evaluation of a government health insurance scheme for people below the poverty line (BPL) in Karnataka, India.1 With some notable exceptions,2 3 there are few opportunities to exploit randomisation in the evaluation of large-scale health financing reforms and thus quasi experimental methods must be relied upon. In this regard, the increasing number of publications in medical journals applying econometric methods to study impacts of such schemes are to be welcomed. Nevertheless, the findings of this study should be interpreted with a number of limitations in mind, particularly given the impressive magnitude of the mortality effect reported.
First, the empirical approach used in the study is more accurately described as matching, the limitations of which are well known.4 A regression discontinuity design that uses a geographical boundary as the assignment variable is based on the idea that there is a discontinuity in the treatment variable (insurance coverage) either side of a “random” boundary but no other discontinuities in relevant variables (household characteristics, community factors, policy environment) conditional on distance to the boundary.5 The study neither demonstrates the discontinuity in insurance coverage6 despite investment in primary data collection nor does it control for distance or geographical location in any way.7 8
Second, the geographical border is far from “arbitrary.” The paper suggests that the insurance scheme was implemented in such a way that the boundary in fact divides different districts. We must therefore ask “what are all the things differing between the two regions other than the treatment of interest?”5 In India, districts are the administrative unit charged with the implementation of health policy, not to mention the administration of the BPL system, and variation in such implementation (due to differences in management, leadership, budget, health personnel etc) must be expected. Yet the analysis is unable to control for such differences between districts. A more credible alternative with the available data would be a difference-in-difference approach – that is, a comparison of poor and non-poor households between treatment and control. It would provide an opportunity to control for geographical location.
Finally, it is hard to reconcile the large mortality effect with the modest (sometimes borderline significant) utilization results. Without substantive evidence of a large increase in hospitalization, it is premature to read too much into the mortality finding.
1. Sood N, Bendavid E, Mukherji A, Wagner Z, Nagpal S, Mullen P. Government health insurance for people below poverty line in India: quasi-experimental evaluation of insurance and health outcomes. BMJ 2014;349:g5114.
2. Baicker K, Taubman SL, Allen HL, Bernstein M, Gruber JH, Newhouse JP, et al. The Oregon experiment--effects of Medicaid on clinical outcomes. N Engl J Med 2013;368(18):1713-22.
3. King G, Gakidou E, Imai K, Lakin J, Moore RT, Nall C, et al. Public policy for the poor? A randomised assessment of the Mexican universal health insurance programme. Lancet 2009;373(9673):1447-54.
4. Imbens GW, Wooldridge JM. Recent Developments in the Econometrics of Program Evaluation. Journal of Economic Literature 2009;47(1):5-86.
5. Lee DS, Lemieux T. Regression Discontinuity Designs in Economics. Journal of Economic Literature 2010;48(2):281-355.
6. Card D, Dobkin C, Maestas N. Does Medicare Save Lives? Quarterly Journal of Economics 2009;124(2):597-636.
7. Black SE. Do Better Schools Matter? Parental Valuation of Elementary Education. Quarterly Journal of Economics 1999;114(2):577-99.
8. Dell M. The Persistent Effects of Peru's Mining Mita. Econometrica 2010;78(6):1863-1903.
Competing interests: No competing interests