Variable Selection and Data Quality Challenges in Impact Assessments


  • Monica Roman Professor PhD, Bucharest University of Economic Studies
  • Liliana-Olivia Lucaciu PhD candidate Bucharest University of Economic Studies, Romania



impact evaluation, counterfactual impact evaluation, variables, data quality, big data, blockchain


The research is focused on the role of two related key concepts, namely variables and data, in the impact evaluations of public projects.  A difficult task of the evaluators and researchers is to select the appropriate variables to ensure the best model of reality and satisfy the evaluation methods' needs. Therefore, the paper aims to look at the current knowledge and discuss how variables and data could be best used to connect the evaluation models, the particularities of the intervention with the potential of the advanced quantitative assessment methods.

The results emphasise that evaluations operate with data with different levels of granularity, as required by the intervention logic. Structuring data in clusters and categories, performing evaluability assessments are useful in assessing data quality and limitations and improving them. In line with the existing literature, we demonstrate that data accessibility is a key constraint and imposes adjustment of the desired evaluation model to a feasible one.

While Big Data and Open Data systems significantly improved data quality in evaluations in recent years, blockchain, as a ledger technology with default features related to decentralisation and security, is expected to bring large benefits to evaluation. For evaluators and policymakers, blockchain potential is an area of further research looking for additional advantages that could enhance the use of quantitative methods.


Befani, B., Barnett, C., & Stern, E. (2014). Introduction–Rethinking impact evaluation for development. IDS Bulletin, 45(6), 1-5.

Berman, E., & Wang, X. (2012). Essential statistics for public managers and policy analysts. Sage Publications.

Carli, R., Dotoli, M., Pellegrino, R., & Ranieri, L. (2013). Measuring and Managing the Smartness of Cities: A Framework for Classifying Performance Indicators. 2013 IEEE International Conference On Systems, Man, And Cybernetics.

Conn, V. (2017). Don’t Rock the Analytical Boat: Correlation Is Not Causation. Western Journal Of Nursing Research, 39(6), 731-732.

Crato, N., & Paruolo, P. (2019). The Power of microdata. In N. Crato, & P. Paruolo (Eds.), Data-Driven Policy Impact Evaluation. How access to microdata is transforming policy design. (p. 1-14). Springer.

Deepa, N., Pham, Q., Nguyen, D., Bhattacharya, S., Prabadevi, B., & Gadekallu, T. et al. (2021). A Survey on Blockchain for Big Data: Approaches, Opportunities, and Future Directions.

Elia, L., Santangelo, G., & Schnepf, S. (2015). Synthesis report on the 'Pilot projects to carry out ESF related counterfactual impact evaluations. Directorate-General for Employment, Social Affairs and Inclusion (European Commission), Joint Research Centre (European Commission)

European Commission. (2014, January). Monitoring and evaluation of European Cohesion Policy.European Regional Development Fund, European Social Fund, European Cohesion Fund. Guidance document on ex-ante evaluation.

European Commission. (2019). Guidelines on linking planning/programming, monitoring and evaluation (2nd ed.). European Commission.

European Commission. (2021). Evaluation.

Ferraro, P. J. (2009). Counterfactual thinking and impact evaluation in environmental policy. New directions for evaluation, 2009(122), 75-84.

Field-Fote, E. (2019). Mediators and Moderators, Confounders and Covariates: Exploring the Variables That Illuminate or Obscure the “Active Ingredients” in Neurorehabilitation. Journal Of Neurologic Physical Therapy, 43(2), 83-84.

Gertler, P., Martinez, S., Premand, P., Rawlings, L., & Vermeersch, C. (2016). Impact Evaluation in Practice, (2nd ed.). World Bank.

Hujer, R., Caliendo, M., & Radic, D. (2004). Methods and limitations of evaluation and impact research. Found Eval Impact Res (CEDEFOP Ref Ser 58), 30, 131-90.

Karafiloski, E., & Mishev, A. (2017). Blockchain solutions for big data challenges: A literature review. IEEE EUROCON 2017 – 17th International Conference On Smart Technologies.

Li, Y., Li, P., Zhu, F., & Wang, R. (2017). Design of higher education quality monitoring and evaluation platform based on big data. 12Th International Conference On Computer Science And Education (ICCSE).

Lucaciu, L. (2018). A Look at the Evaluation Framework for Smart Growth Programmes. Revista Romaneasca Pentru Educatie Multidimensionala, 10(3), 60-76.

Lucaciu, L., Roman, M., Rus, S., Florian, B., Vasile, M., & Mariș, C. (2020). Evaluarea retrospectivă a intervențiilor POSDRU 2007-2013 în domeniul educației. București: Ministerul Investițiilor și Proiectelor Europene.

Lucaciu, L., Roman, M., Rus, S., Florian, B., Vasile, M., & Mariș, C.(2021a). Prima evaluare a intervențiilor POCU 2014-2020 în domeniul incluziunii sociale [The first evaluation of the POCU interventions 2014-2020 in the field of social inclusion]. București: Ministerul Investițiilor și Proiectelor Europene.

Lucaciu, L., Roman, M., Savin, A., Ionescu, D., Bulai, A., Rus, S., Florian, B., Vasile, M., Maris, C., Buhaescu-Ciuca, S., Savin, A., Platon, G., Rosu, C., Perianu, E., Dalu, A.-M., Haj, M. C., Manoleli-Preda, D., Florescu, A.-E., Baciu, A., Jurj., A., Florina, T.(2021b). Prima evaluare a intervențiilor POCU 2014-2020 în domeniul educației [The first evaluation of the POCU interventions 2014-2020 in the field of education]. Bucharest: Ministerul Investițiilor și Proiectelor Europene.

Morris, S., Tödtling-Schönhofer, H., & Wiseman, M. (2013). Design and commissioning of counterfactual impact evaluations.

Popescu, M., & Roman, M. (2018). Vocational training and employability: Evaluation evidence from Romania. Evaluation And Program Planning, 67, 38-46.

Rettore, E., & Trivellato, U. (2019). The Use of Administrative Data to Evaluate the Impact of Active Labor Market Policies: The Case of the Italian Liste di Mobilità. In N. Crato & P. Paruolo, Data-driven policy impact evaluation (1st ed., pp. 165-182). Springer Open

Roman, M., & Goțiu (Lucaciu), L. (2017). Non-parametric methods applied in the efficiency analysis of European structural funding in Romania. Journal Of Applied Quantitative Methods ,18(2).

Scott-Phillips, T., Dickins, T., & West, S. (2011). Evolutionary Theory and the Ultimate–Proximate Distinction in the Human Behavioral Sciences. Perspectives On Psychological Science, 6(1), 38-47.

Silverman, D. (2011). Interpreting qualitative data. A Guide to the Principles of Qualitative Research. Sage

Trivellato, U. (2019). Microdata for Social Sciences and Policy Evaluation as a Public Good. In N. Crato & P. Paruolo (Eds.), Data-Driven Policy Impact Evaluation (pp. 27-45). Springer Open.

United Nations. (2019). National Quality Assurance Frameworks Manual for Official Statistics. United Nations Department of Economic and Social Affairs Statistics Division.

Vaessen, J. (2011). Challenges in impact evaluation of development interventions: randomised experiments and complexity. Evaluating the complex: attribution, contribution, and beyond. comparative policy evaluation, 18, 283-313.

Wang, R., & Strong, D. (1996). Beyond Accuracy: What Data Quality Means to Data Consumers. Journal Of Management Information Systems, 12(4), 5-33.

White, H. (2014). Current challenges in impact evaluation. The European Journal of Development Research, 26(1), 18-30.

Williams, B. (2015). Prosaic or profound? The adoption of systems ideas by impact evaluation. IDS Bulletin, 46(1), 7-16.

World Health Organization. (2017). Data quality review: module 1: framework and metrics.




How to Cite

Roman, M. ., & Lucaciu, L.-O. . (2021). Variable Selection and Data Quality Challenges in Impact Assessments. Postmodern Openings, 12(3Sup1), 01-20.



Research Articles

Publish your work at the Scientific Publishing House LUMEN

It easy with us: publish now your work, novel, research, proceeding at Lumen Scientific Publishing House

Send your manuscript right now