Enhancement of research and development programs evaluation methodology in the light of international experience

The article compares international and Russian techniques for the evaluation of research and development programs and develops suggestions on their enhancement. The author argues that the simplified techniques commonly used in Russia do not provide proper basis for the appraisal and monitoring of the impact of research and development programs on innovative transformation of the Russian economy. As a result of this practice the budget expenditure on science and innovation is frequently non-efficient. The enhancement of efficiency of public investment in research and development programs requires emphasis on careful appraisal and monitoring of economic results of innovators using knowledge created and disseminated within program execution as well as seed and venture capital provided by state development institutions. The author justifies a technique for the comparative evaluation of research and development projects and subprograms based on a synthesis of the logic model of transformation of inputs into outcomes and impacts and a mathematical model of data envelopment analysis. The advantage of the technique is a necessary balance between program results evaluation areas (outputs, outcomes and impacts). The author demonstrates the identification process of the most efficient projects and subprograms at the formation and execution phases of the program via a solution of the linear programming model. The model affords to reveal the project/subprogram distance from the efficiency frontier of inputs’ transformation into outputs, outcomes and impacts

Keywords: research and development programs evaluation, international experience of research evaluation, optimization of innovative projects portfolio, key performance indicators

References

  1. Fahrenkrog G. et al. RTD evaluation tool box: Assessing the socio-economic impact of RTD Policy – Seville: European Commission Joint Research Centre, 2002.
  2. Ruegg R., Feller I. A toolkit for evaluating public R&D investment. Models, methods, and findings from ATP’s first decade. – Gaithersburg: National Institute of Standards and Technology, 2003.
  3. Ruegg R., Jordan G. Overview of evaluation methods for R&D program. A directory of evaluation methods relevant to technology development programs. – US Department of Energy, Office of Energy Efficiency and Renewable Energy, 2007.
  4. Reed J., Jordan G., Vine E. Impact evaluation framework for technology deployment programs. – US Department of Energy, Office of Energy Efficiency and Renewable Energy, 2007.
  5. Marjanovic S., Hanney S., Wooding S. A historical reflection on research evaluation studies, their recurrent themes and challenges – Santa Monica: RAND Corporation, 2009.
  6. Guthrie S., Wamae W., Diepeveen S., Wooding S., Grant J. Measuring research. A guide to research evaluation frameworks and tools. – Santa Monica: RAND Corporation, 2013.
  7. Evaluating outcomes of publicly-funded research, technology and development programs: Recommendations for improving current practice. – American Evaluation Association, 2015.
  8. National report on innovation in Russia – 2015 (draft). Available at: http://www.rusventure.ru/ru/programm/analytics/docs/NROI_RVC.pdf (in Russ.)
  9. Vasetskaya N.O., Fedotov A.V. Problem of evaluation of investments in scientific research in Russia. Nauchnye trudy Severo-Zapadnogo instituta upravleniya [Research papers of North-West Institute of Management], vol.5, no.3, pp.313-318 (in Russian)
  10. Efimov A.R. Evolution of performance indicators of state support for innovation: A case of Scolkovo innovation center. Natsionalnye interesy: prioritety I bezopasnost’ [National interests: priorities and security], 2014, no.32, pp.22-34. (in Russian)
  11. Bogatyreva V.D., Gorbunov D.V. A methodology for evaluation of efficiency and effectiveness of budget expenditure on innovation. Vestnik Samarskogo gosudarstvennogo universiteta. Seriya “Ekonomika i upravleniye” [Research papers in economics and management of Samara State University], 2015, no.9/2, pp.192-203 (in Russian)
  12. Public Law 111-5 of February 17, 2009 «American Recovery and Reinvestment Act». Available at: https://www.gpo.gov/fdsys/pkg/BILLS-111hr1enr/pdf/BILLS-111hr1enr.pdf
  13. Public Law 11-352 of January 4, 2011 «GPRA Modernization Act of 2010». Available at: https://www.gpo.gov/fdsys/pkg/PLAW-111publ352/pdf/PLAW-111publ352.pdf
  14. Offices of Management and Budget and Science and Technology Policy. Memorandum for the heads of executive departments and agencies, Subject: Science and Technology Priorities for the FY2015 Budget. Available at: https://www.whitehouse.gov/sites/default/files/omb/memoranda/2013/m-13-16.pdf
  15. Science and Technology for America’s Reinvestment: Measuring the EffecTs of Research on Innovation, Competitiveness and Science. Available at: https://www.starmetrics.nih.gov/
  16. Lane J., Bertuzzi S. Measuring the results of science investments // Science. 2011. Vol.331. pp.678-680.
  17. Largent M., Lane J. STAR METRICS and the science of science policy // Review of Policy Research. 2012. Vol.29. No.3. pp.431-438.
  18. Research Excellence Framework. Available at: http://www.ref.ac.uk/ 
  19. Assessment framework and guidance on submissions. REF 02.2011 – Bristol: HEFCE, 2011. 
  20. Research Excellence Framework 2014: The results. REF 01.2014 – Bristol: HEFCE, 2014. 
  21. Order of the Government of the Russian Federation of April 15, 2014 No.301 «On approval of the state program of the Russian Federation «Development of science and technology» for 2013-2020» (in Russ.) 
  22. Order of the Government of the Russian Federation of April 15, 2014 No.316 «On approval of the state program of the Russian Federation «Economic development and innovative economy» (in Russ.) 
  23. Directions of activity of RVC joint stock company for 2014-2016. Approved by the order of RVC board of directors of 30.07.2013, protocol No.7. Available at: http://www.rusventure.ru/ru/company/legal_basis/directions_RVC_2014_2016.pdf. (in Russ.) 
  24. Charnes A., Cooper W., Rhodes E. Measuring the efficiency of decision making units // European Journal of the Operational Research. 1978. No.2. pp.429-444. 
  25. Charnes A., Cooper W., Lewin A., Seiford L. Data envelopment analysis: theory, methodology and applications. – Boston: Kluwer, 1994.

Authors