Variable Selection and Data Quality Challenges in Impact Assessments

Authors

  • Monica Roman Professor PhD, Bucharest University of Economic Studies
  • Liliana-Olivia Lucaciu PhD candidate Bucharest University of Economic Studies, Romania

DOI:

https://doi.org/10.18662/po/12.3Sup1/348

Keywords:

impact evaluation, counterfactual impact evaluation, variables, data quality, big data, blockchain

Abstract

The research is focused on the role of two related key concepts, namely variables and data, in the impact evaluations of public projects.  A difficult task of the evaluators and researchers is to select the appropriate variables to ensure the best model of reality and satisfy the evaluation methods' needs. Therefore, the paper aims to look at the current knowledge and discuss how variables and data could be best used to connect the evaluation models, the particularities of the intervention with the potential of the advanced quantitative assessment methods.

The results emphasise that evaluations operate with data with different levels of granularity, as required by the intervention logic. Structuring data in clusters and categories, performing evaluability assessments are useful in assessing data quality and limitations and improving them. In line with the existing literature, we demonstrate that data accessibility is a key constraint and imposes adjustment of the desired evaluation model to a feasible one.

While Big Data and Open Data systems significantly improved data quality in evaluations in recent years, blockchain, as a ledger technology with default features related to decentralisation and security, is expected to bring large benefits to evaluation. For evaluators and policymakers, blockchain potential is an area of further research looking for additional advantages that could enhance the use of quantitative methods.

References

Befani, B., Barnett, C., & Stern, E. (2014). Introduction–Rethinking impact evaluation for development. IDS Bulletin, 45(6), 1-5. https://doi.org/10.1111/1759-5436.12108

Berman, E., & Wang, X. (2012). Essential statistics for public managers and policy analysts. Sage Publications.

Carli, R., Dotoli, M., Pellegrino, R., & Ranieri, L. (2013). Measuring and Managing the Smartness of Cities: A Framework for Classifying Performance Indicators. 2013 IEEE International Conference On Systems, Man, And Cybernetics. https://doi.org/10.1109/smc.2013.223

Conn, V. (2017). Don’t Rock the Analytical Boat: Correlation Is Not Causation. Western Journal Of Nursing Research, 39(6), 731-732. https://doi.org/10.1177/0193945917701090

Crato, N., & Paruolo, P. (2019). The Power of microdata. In N. Crato, & P. Paruolo (Eds.), Data-Driven Policy Impact Evaluation. How access to microdata is transforming policy design. (p. 1-14). Springer. http://doi.org/10.1007/978-3-319-78461-8

Deepa, N., Pham, Q., Nguyen, D., Bhattacharya, S., Prabadevi, B., & Gadekallu, T. et al. (2021). A Survey on Blockchain for Big Data: Approaches, Opportunities, and Future Directions. arXiv.org. https://arxiv.org/abs/2009.00858

Elia, L., Santangelo, G., & Schnepf, S. (2015). Synthesis report on the 'Pilot projects to carry out ESF related counterfactual impact evaluations. Directorate-General for Employment, Social Affairs and Inclusion (European Commission), Joint Research Centre (European Commission) https://op.europa.eu/en/publication-detail/-/publication/3d8a3354-084d-11e6-b713-01aa75ed71a1/language-en

European Commission. (2014, January). Monitoring and evaluation of European Cohesion Policy.European Regional Development Fund, European Social Fund, European Cohesion Fund. Guidance document on ex-ante evaluation. https://ec.europa.eu/regional_policy/sources/docoffic/2014/working/ex_ante_en.pdf

European Commission. (2019). Guidelines on linking planning/programming, monitoring and evaluation (2nd ed.). European Commission. https://ec.europa.eu/neighbourhood-enlargement/sites/default/files/dg_near_guidance_note_-_addressing_capacity_development_in_programming_me.pdf

European Commission. (2021). Evaluation. https://ec.europa.eu/regional_policy/en/policy/evaluations/member-states/

Ferraro, P. J. (2009). Counterfactual thinking and impact evaluation in environmental policy. New directions for evaluation, 2009(122), 75-84. https://doi.org/10.1002/ev.297

Field-Fote, E. (2019). Mediators and Moderators, Confounders and Covariates: Exploring the Variables That Illuminate or Obscure the “Active Ingredients” in Neurorehabilitation. Journal Of Neurologic Physical Therapy, 43(2), 83-84. https://doi.org/10.1097/npt.0000000000000275

Gertler, P., Martinez, S., Premand, P., Rawlings, L., & Vermeersch, C. (2016). Impact Evaluation in Practice, (2nd ed.). World Bank.

Hujer, R., Caliendo, M., & Radic, D. (2004). Methods and limitations of evaluation and impact research. Found Eval Impact Res (CEDEFOP Ref Ser 58), 30, 131-90. https://www.cedefop.europa.eu/files/BgR1_Hujer.pdf

Karafiloski, E., & Mishev, A. (2017). Blockchain solutions for big data challenges: A literature review. IEEE EUROCON 2017 – 17th International Conference On Smart Technologies. https://doi.org/10.1109/eurocon.2017.8011213

Li, Y., Li, P., Zhu, F., & Wang, R. (2017). Design of higher education quality monitoring and evaluation platform based on big data. 12Th International Conference On Computer Science And Education (ICCSE). https://doi.org/10.1109/iccse.2017.8085513

Lucaciu, L. (2018). A Look at the Evaluation Framework for Smart Growth Programmes. Revista Romaneasca Pentru Educatie Multidimensionala, 10(3), 60-76. https://doi.org/10.18662/rrem/63

Lucaciu, L., Roman, M., Rus, S., Florian, B., Vasile, M., & Mariș, C. (2020). Evaluarea retrospectivă a intervențiilor POSDRU 2007-2013 în domeniul educației. București: Ministerul Investițiilor și Proiectelor Europene.

Lucaciu, L., Roman, M., Rus, S., Florian, B., Vasile, M., & Mariș, C.(2021a). Prima evaluare a intervențiilor POCU 2014-2020 în domeniul incluziunii sociale [The first evaluation of the POCU interventions 2014-2020 in the field of social inclusion]. București: Ministerul Investițiilor și Proiectelor Europene.

Lucaciu, L., Roman, M., Savin, A., Ionescu, D., Bulai, A., Rus, S., Florian, B., Vasile, M., Maris, C., Buhaescu-Ciuca, S., Savin, A., Platon, G., Rosu, C., Perianu, E., Dalu, A.-M., Haj, M. C., Manoleli-Preda, D., Florescu, A.-E., Baciu, A., Jurj., A., Florina, T.(2021b). Prima evaluare a intervențiilor POCU 2014-2020 în domeniul educației [The first evaluation of the POCU interventions 2014-2020 in the field of education]. Bucharest: Ministerul Investițiilor și Proiectelor Europene. http://www.anc.edu.ro/wp-content/uploads/2021/04/Primul-raport-de-evaluare-a-interven%C8%9Biilor-POCU-2014-2020-%C3%AEn-domeniul-educa%C8%9Biei-1.pdf

Morris, S., Tödtling-Schönhofer, H., & Wiseman, M. (2013). Design and commissioning of counterfactual impact evaluations. https://op.europa.eu/en/publication-detail/-/publication/f879a9c1-4e50-4a7b-954c-9a88d1be369c

Popescu, M., & Roman, M. (2018). Vocational training and employability: Evaluation evidence from Romania. Evaluation And Program Planning, 67, 38-46. https://doi.org/10.1016/j.evalprogplan.2017.11.001

Rettore, E., & Trivellato, U. (2019). The Use of Administrative Data to Evaluate the Impact of Active Labor Market Policies: The Case of the Italian Liste di Mobilità. In N. Crato & P. Paruolo, Data-driven policy impact evaluation (1st ed., pp. 165-182). Springer Open

Roman, M., & Goțiu (Lucaciu), L. (2017). Non-parametric methods applied in the efficiency analysis of European structural funding in Romania. Journal Of Applied Quantitative Methods ,18(2). https://mpra.ub.uni-muenchen.de/80548/1/MPRA_paper_80548.pdf

Scott-Phillips, T., Dickins, T., & West, S. (2011). Evolutionary Theory and the Ultimate–Proximate Distinction in the Human Behavioral Sciences. Perspectives On Psychological Science, 6(1), 38-47. https://doi.org/10.1177/1745691610393528

Silverman, D. (2011). Interpreting qualitative data. A Guide to the Principles of Qualitative Research. Sage

Trivellato, U. (2019). Microdata for Social Sciences and Policy Evaluation as a Public Good. In N. Crato & P. Paruolo (Eds.), Data-Driven Policy Impact Evaluation (pp. 27-45). Springer Open.

United Nations. (2019). National Quality Assurance Frameworks Manual for Official Statistics. United Nations Department of Economic and Social Affairs Statistics Division. https://unstats.un.org/unsd/methodology/dataquality/references/1902216-UNNQAFManual-WEB.pdf.

Vaessen, J. (2011). Challenges in impact evaluation of development interventions: randomised experiments and complexity. Evaluating the complex: attribution, contribution, and beyond. comparative policy evaluation, 18, 283-313. https://medialibrary.uantwerpen.be/oldcontent/container2143/files/Publications/DP/2010/01-Vaessen.pdf

Wang, R., & Strong, D. (1996). Beyond Accuracy: What Data Quality Means to Data Consumers. Journal Of Management Information Systems, 12(4), 5-33. https://doi.org/10.1080/07421222.1996.11518099

White, H. (2014). Current challenges in impact evaluation. The European Journal of Development Research, 26(1), 18-30. https://doi.org/10.1057/ejdr.2013.45

Williams, B. (2015). Prosaic or profound? The adoption of systems ideas by impact evaluation. IDS Bulletin, 46(1), 7-16. https://doi.org/10.1111/1759-5436.12117

World Health Organization. (2017). Data quality review: module 1: framework and metrics. https://apps.who.int/iris/handle/10665/259224

Downloads

Published

2021-09-10

How to Cite

Roman, M. ., & Lucaciu, L.-O. . (2021). Variable Selection and Data Quality Challenges in Impact Assessments. Postmodern Openings, 12(3Sup1), 01-20. https://doi.org/10.18662/po/12.3Sup1/348

Issue

Section

Research Articles

Publish your work at the Scientific Publishing House LUMEN

It easy with us: publish now your work, novel, research, proceeding at Lumen Scientific Publishing House

Send your manuscript right now