Introduction
In order to determine the contribution of innovation to economic development, it becomes important to know how successful an innovation system is.The measurement of the performance of an innovation system influences policy decisions for the improvement of such a system. Comparisons between different time periods or between different innovation systems (of, for example, different regions or countries) further necessitate the measurement of the performance of innovation systems (Edquist & Zambala, 2009:6).
The importance of innovation led to many attempts at measuring innovative performance of the innovation system. Yet, there is not general agreement on how such innovation should be measured. The question then arises, given the problems associated with measuring innovation and with the complexity of the innovation system, how feasible is the measurement of innovation system performance. The aim of this paper is to contribute to the literature on innovation system performance.
This article first provides clarity on concepts by defining an innovation system, describing the complexity of an innovation system and explaining what is meant by innovation system performance. Thereafter, innovation and the innovation system are discussed in terms of the place of these concepts in economic schools of thought to contribute further to the explanation of the complexity of the innovation system concept.
Thirdly, the difficulties and problems associated with the measurement of innovation are identified. Thereafter, the measurement techniques that are used by the different innovation studies to measure the performance of the innovation system are critically evaluated to determine the feasibility of these techniques in the measurement of the innovation system.
Research Questions
The research questions addressed in this study are the following:
What are the problems that are experienced in measuring innovation?
How should the performance of an innovation system be measured?
Research Methodology
The research methodology applied is qualitative in nature. A descriptive analysis is made of the concepts of innovation and innovation systems. Research previously undertaken in the rest of the world concerning how innovation has been measured in different studies is analysed.The advantages and disadvantages of the different measurement techniques are identified.The measurement techniques of innovation are evaluated in the context of an innovation system and the complexity of an innovation system is considered in the evaluation of the relevant measurement techniques.
Conceptual Framework
The Definition of an Innovation System
The firms, organisations and institutions that play a role in the innovative activities all interact and form a system. Such a system is called an innovation system. The word “system” here implies that there is interaction among all the different actors or participants who contribute to innovation, and that the system includes an environment within which innovation takes place.These actors or participants do not necessarily consciously interact with each other. The different participants do not necessarily have the same goal.
The entrepreneur’s main aim will probably be profit, while the aim of the academic institutions may be research and training to enhance innovation. The aim of government may, on the other hand, be the creation of a macroeconomic environment within which innovation can take place. Yet the different participants each have an impact on one another. Nelson (1996:276) explained the system concept saying, “(t)here is no presumption that the system was, in some sense, consciously designed, or even that the set of institutions involved works together smoothly and coherently. Rather, the ‘systems’ concept is that of a set of institutional actors that, together, play the major role in influencing innovative performance”.
An innovation system can be defined as follows:
An innovation system consists of the participants or actors and their activities and interactions, as well as the socio-economic environment within which these actors or participants function, that together determine the innovative performance of the system (Eggink, 2012:24).
Innovation System Performance
Literature on innovation system performance revealed that the concept is used or interpreted in different ways.It is important to distinguish between innovative “capability” or “capacity” and innovative “performance”. Gregersen & Johnson (2005:7) asked the question, “… do we focus on the number of innovations produced in a certain period or on the creation of environments and competencies capable of sustaining and increasing learning and innovation in the future?” This study will focus on innovative performance.
The innovative capability or capacity includes the availability, quality and interaction of all the role players or participants in the innovation system as well as the degree to which the socio-economic environment of the system is conducive to innovation. Edquist & Zabala (2009:4) clarify the difference between capability or capacity and performance of innovation with the “input” and “output” concepts.Therefore, the capabilities or capacities are what is needed or should be in place for innovation to take place.The performance of the system then becomes “what comes out” of the system. However, the “output” or performance should not be confused with the “impact” of the system on, for example, economic growth.
Perhaps unfortunately, it should be noted that more innovation is not always good. All innovation is not good and “more innovation is not always better” (Edquist & Zambala, 2009:7).
Measurement of Innovation
When attempting to measure innovation in an empirical study, some problems are experienced. The different ways economists found to measure innovation in their studies will be explored and the problems discussed.
Difficulties in Measuring Innovation
Unfortunately, there is no single, generally accepted definition for innovation and this makes the identification of an innovation particularly difficult. According to the OECD & Eurostat, (2005:46), “(a)n innovation is the implementation of a new or significantly improved product (good or service), or process, a new marketing method, or a new organisational method in business practices, workplace organisation or external relations”.Many authors focus on firms as the only institutions where innovation takes place, very probably due to the discipline in which the study is done. If the discipline is focused on business studies, the role of participants in the economy, other than firms, is often ignored.
Many definitions of innovation, including the OECD & Eurostat definition, have Schumpeter’s (1961:66) definition as a foundation. Schumpeter was one of the first economists to use the concept of innovation in his theories and his definition is still the one most widely used by innovation economists. Despite his groundbreaking work, his definition of innovation received much criticism, but has not yet been replaced by any other generally accepted definition. Schumpeter’s definition of innovation is as follows:
“This concept covers the following five cases: (1) The introduction of a new good — that is one with which consumers are not yet familiar — or of a new quality of a good. (2) The introduction of a new method of production, that is one not yet tested by experience in the branch of manufacture concerned, which need by no means be founded upon a discovery scientifically new, and can also exist in a new way of handling a commodity commercially. (3) The opening of a new market, that is a market into which the particular branch of manufacture of the country in question has not previously entered, whether or not this market has existed before. (4) The conquest of a new source of supply of raw materials or half-manufactured goods, again irrespective of whether this source already exists or whether it has first to be created. (5) The carrying out of the new organisation of any industry, like the creation of a monopoly position (for example through trustification) or the breaking up of a monopoly position” (Schumpeter, 1961:66).
The OECD & Eurostat’s definition adapted the constituents of innovation from Schumpeter, being the introduction of a “new product”, “new method”, “new market” and “new organisation”. For any idea, product, method or organisation to be called an innovation, it has to be applied; therefore, the OECD & Eurostat changed the concept of “introduction” of the acts, to “implementation” of these acts.
According to Smith (2005:149), the definition of innovation is about novelty, and the first challenge is to measure novelty. He reasons that even if novelty can be measured, a definition is needed for the term “new”. According to Smith, something may be old, but it can be new to the firm, or it can be a radically novel idea or it can be only an incremental change.Rogers (1995:132) states that innovation is, “… an idea, practice or object that is perceived as new to an individual or another unit of adoption”. In such a case, the firm does not actually acquire the innovation from another firm; the firm acquires only the idea or technology. It only becomes innovation when the firm that acquired the technology or idea from another firm successfully implements that product or idea in the relevant market. This kind of innovation is particularly important in the developing world.
The definition of innovation should include incremental changes, although the Oslo Manual of the Organisation for Economic Co-operation and Development (OECD & Eurostat, 2005:17) takes only, “… a significant degree of novelty…” into consideration. The purpose of the OECD & Eurostat is to measure innovation, but incremental changes are difficult to distinguish and therefore equally difficult to measure. The problem is that the perceptions of the concepts “incremental” and “significant” differ among individuals.
Apart from the lack of a common definition, there are other problems in identifying innovations. It would seem that the identification of product innovation is easier than is that of process innovation. It also is apparent that examples of marketing and organisation innovation have little prominence in historical literature on innovation. Van Duijn (1983:173) agreed that it is not easy to either list or date major innovations.The problems that Van Duijn experienced in identifying innovations include the following: innovations are heterogeneous in character; innovations are heterogeneous in area of application; innovation is heterogeneous in impact; and the question whether only seminal innovations or subsequent improvements should be mentioned.
The question then arises if the number of innovations could provide an answer to the performance of an innovation system. Should the impact that innovation has on different socio-economic aspects not be considered also? Thus it appears that more than just the identification of the kind of innovation is needed in order to measure the impact of innovation. According to Ray, (1980:12), the diffusion of innovation across the economy, as well as the speed of this diffusion, is important when measuring the impact on the economy. The diffusion of innovation is a very complex process, where a new product or process can trigger the innovation of another new product or process or can replace the existing ones. Van Duijn, (1983:175), agrees that not only the moment of innovation is important, but also the diffusion of an innovation.
The longevity or life-span of an innovation must also be considered when measuring the impact of innovation. Different and new innovations may have different lifetimes before they are replaced by other innovations. Some innovations last an indefinite time.
It is difficult to measure innovation because innovation is a continuous process and large-scale diffusion is a gradual process (OECD & Eurostat, 2005:15; Ray, 1980:12). A constantly changing situation makes an impact study more difficult to carry out. Van Duijn (1983:174) explained that the success of an innovation can only be assessed after some time (at least a decade) has passed, but, “… this implies that the lists necessarily get thinner towards the date of compilation”, and that, “(d)rop-offs in numbers of basic innovations thus do not necessarily mean reduced innovativeness”.
The classification of innovations must also be considered. According to Ray, (1980:9), an innovation that is, for example, classified as “industrial” in the first instance, may be applied in other sectors and may even have its major impact elsewhere. He used an example of the technique of delaying the clotting of blood (used in medical science) that was being applied to the Malayan rubber plantations where the techniques are applied to increase the flow of rubber from the trees. To trace an innovation in such different applications enormously complicates the tracking process.
It must further be considered that some of the impact of innovation cannot be measured in quantifiable terms such as growth in production or profit. Measuring the impacts of medication and of vaccine innovation on development mandates the inclusion of the improvement in living standards of people. Many other innovations improve living standards, for example, innovations such as access to communication via telephone, cell phones, television, internet, the availability of electricity and transport, and many other innovations that are now taken for granted. The benefit, according to Ray (1980:9), is also different across different peoples or different industries.
Different Methods of Measuring Innovation
Although there are numerous difficulties in the measuring of innovative performance, many attempts have been made in the past to quantify innovation. Becheikh, Landry & Amara (2006) researched 108 studies on innovation performance measurement and found that 24% used firm-based surveys to measure innovation, 25% used innovation count, 18% patent registrations, 6% research and development (R&D) expenditure, 15% indices, 9% other measures (for example, sales generated by innovations, the number of trademarks, the time allocated by managers to innovation related activities, etc.) while 4% did not attempt to measure innovation. It is clear that many of these studies measured innovation by means of innovative capacity and not by innovative performance.
The studies that measure the outcomes of innovation are, for example, innovation surveys and innovation counts. The studies that focus on capabilities to innovate include the R&D expenditure and many of the indices (including indicators such as education, infrastructure, patents, etc.). Yet they do not distinguish between capacity and performance.
Inputs to Innovation
Some of the more commonly used indicators, which are actually input indicators but are incorrectly used as output indicators, are examined. One of these methods is R&D expenditure by firms. This method is popular due to the comparability over time and across countries as it is measured in monetary values and so the R&D can be expressed as a ratio to GDP without the need for exchange rate adjustments when both R&D and GDP are expressed in national currencies (Archibugi & Coco, 2005:183).
The R&D expenditure by firms gives an indication of the involvement of a firm in innovating activities. However, it does not measure the innovative output of the firms. Some of the disadvantages of using R&D expenditure for the measurement of innovation that have been identified include that not all R&D expenditure will lead to innovation. This causes R&D to be a measure that overestimates innovation (Audretsch, 2004:175; Becheikh et al., 2006:649; Greenhalgh & Rogers, 2010:59). On the other hand, not all innovations are a result of R&D expenditure. For instance, some innovations can be a sudden, clever idea of the innovator.
Further, there may be a time lag between the R&D expenditure and the innovation (Greenhalgh & Rogers, 2010:59), while adequate recording and tracking of R&D expenditure is not always available in all countries (LeBel, 2008:338). Research and development expenditure as the measurement standard favours large firms over small and medium enterprises (SMEs) as the latter two may not have formal R&D structures (Becheikh et al., 2006:649).
Another measure by many studies of innovation output is patent statistics. Although patents are used in many studies as a measure of innovation output, they should rather be seen as an input measure, or as an “intermediate output”, as Audretsch (2004:175) calls them. This is because patent registration measures inventions – not innovations.
Inventions are not an indication that an innovation will take place nor yet of when it will take place and are therefore not useful in tracking innovations. Ray (1980:12) stated that there may be a long chain of other inventions that eventually lead to or help a specific invention to take place and that it may take decades for an innovation to follow that invention. Schumpeter’s (1961:89) view was that, “…innovations … need not necessarily be any inventions at all”.
It should be noted also that not all inventions are patented (Audretsch, 2004:175; Becheikh et al., 2006:649-650; Fagerberg, Srholec & Verspagen, 2009:21; Greenhalgh & Rogers, 2010:61; and LeBel, 2008:338). The quality and quantity of patents registered varies widely across countries, perhaps due to cost restraints, as well as to the different procedures for patent registrations in different countries (Archibugi & Coco, 2005:183).
There may also be differing time lags between an invention and its resulting innovation (LeBel, 2008:338). Hasan & Tucci (2010) conducted research on innovation using patent registration as the measurement for innovation. They acknowledge the disadvantages mentioned above, but indicated their reasons for using patent data as being: (1) usually, inventions are commercialised; (2) detailed statistics of patent registrations are available for many years; (3) the cost involved to obtain and defend the patent implies that a financial return is mostly present or possible.
Outputs from Innovation
One output measure that overcomes the problems of using patent data and R&D expenditure is innovation count and there are countries that keep records of innovative output. This includes data such as new product/process announcements, specialised journals, databases, and the like (Becheikh et al., 2006:650). Two examples of such records are the United States Small Business Administration’s Innovation Data Base and Germany’s Mannheim Innovation Data Base (Audretsch, 2004:175-176). Unfortunately, not all countries keep such records.
The innovation count method of measuring innovation also has some disadvantages. Becheikh et al., (2006:650), reason that innovation count favours radical innovations over incremental ones, and product innovations over process innovations. When considering scientific publications, the quality can vary widely between countries. It must be noted also that English-speaking countries are likely to be over-represented because the majority of journals monitored by the Institute for Scientific Information are published in English (Archibugi & Coco, 2005:183). Royalties and licence fees as a measure of innovation have the disadvantage that it is not clear when the fees are an indication of the creation of technology or due to the acquisition of the technology (Archibugi & Coco, 2005:183).
Composite Variables
The development of indices to represent innovation is an attempt to reduce innovative activities in the innovation system to a single number. There are economists that find it useful to make use of, or to develop, an index to overcome the problems associated with measures of innovation, such as R&D expenditure and patent registrations.These indices combine a number of indicators in a single figure, and attach weights to the relative importance of the indicators (Greenhalgh & Rogers, 2010:62). Examples of such indices follow.
The Revealed Regional Summary Innovation Index (RRSII) is an index that is based on the European Innovation Scoreboard. It consists of seven indicators: (i) population with tertiary education; (ii) participation in life-long learning; (iii) employment in medium-high and high-tech manufacturing; (iv) employment in high-tech services; (v) public R&D expenditures; (vi) business expenditure on R&D and (vii) high-tech patent application (Fraas, 2003:1-2; Howells, 2005:1222). Archibugi & Coco (2004) developed an index called the ArCo technology index.
The ArCo measure was constructed as the average of eight different indicators reflecting various aspects of technological capability, (i) patents, (ii) scientific articles, (iii) internet penetration, (iv) telephone penetration, (v) electricity consumption, (vi) tertiary, science and engineering enrolment, (vii) mean years of schooling and (viii) literacy rate.
Although Archibugi & Coco, (2005:176), readily admit they believe that, “… there is no single number that can provide comprehensive information of the whole technological capabilities of a country”, they find that “synthetic indicators” can, “…,despite the limitations, and if taken with due caution, … help to understand the reality of certain situations, and can assist in devising strategic decisions”. Archibugi & Coco (2005) compared four different technological capability indices for 49 countries (that is, countries that are included in all four indices) and found that, despite significant differences in the case of some individual countries, there is a high correlation between each pair of indices. Even so, they admit that there is still a need to improve these indicators to show more similarity and to make them more reliable.
The World Economic Forum (Porter & Schwab, 2008:6;41), in the development of a Global Competitiveness Index, indicated innovation as one of the twelve pillars of the index. The pillar in itself is an index that consists of the following indicators: (i) capacity to innovate, (ii) quality of scientific research institutions, (iii) company spending on R&D, (iv) university-industry research collaboration, (v) government procurement of advanced technology products, (vi) availability of scientists and engineers, (vii) utility patents and (viii) intellectual property protection.
The index is a weighted average of the responses to survey questions and hard data. LeBel (2008) developed an innovation index considering only per capita scientific citations and per capita net royalty ratio, each of these two indicators carrying equal weights. Although LeBel (2008:338) admits that the index may not capture all dimensions of innovation, he reasons that the index eliminates problems associated with using patent registrations and R&D expenditure as measurements, such as those previously discussed.
A method that was used to combine different measurements and that overcomes the problem of assigning weights to indicators (as is experienced with indices) was used by Fagerberg & Srholec (2008) and is called “factor analysis”. Fagerberg & Srholec used factor analysis on data for 25 indicators of development and 115 countries between 1992 and 2004. According to Fagerberg & Srholec, (2008:1421), “(t)his method is based on the very simple idea that indicators referring to the same dimension are likely to be strongly correlated, and that we may use this insight to reduce the complexity of a large set (consisting of many indicators) into a small number of composite variables, each reflecting a specific dimension of variance in the data”.
In a factor analysis applied to innovation, indicators such as the following were included: (i) patenting, (ii) scientific publications, (iii) information and communications technology (ICT) infrastructure, (iv) International Organisation for Standardisation (ISO) 9000 certifications, (v) access to finance and (vi) education (Fagerberg & Srholec, 2008; Fagerberg et al., 2009:26).
The indices mentioned, as well as the factor analysis discussed, include elements of both innovative capacity and innovative performance. Although the composite variables overcome the problem of “correctness” of a single indicator as described by Hagedoorn & Cloodt (2003:1366), the composition of the ones mentioned does not include only performance variables. Another point of criticism, as made by Hanusch & Pyka, (2007:278) is that complex systems are irreducible, that qualitative change is important also, and not only quantitative change should be considered in the analysis of the innovation system.
Conclusion
In this study, the aim was to establish how the performance of an innovation system should be measured. The concept innovation system was defined as:
An innovation system consists of the participants or actors and their activities and interactions, as well as the socio-economic environment within which these actors or participants function, that together determine the innovative performance of the system (Eggink, 2012:24).
The analysis of empirical studies revealed that there is no generally accepted method of measuring performance of an innovation system.The analysis revealed the importance of distinguishing between innovative capacity and innovative performance of the innovation system. Many studies, in an attempt to measure innovative performance included only indicators that measure capacity or potential performance of the system. Such indicators included research and development spending and patent registration. The output or outcome of the innovation system could better be measured by innovation count or sales figures related to innovative activities.
One reason why many studies used the input indicators is the difficulties that are experienced in measuring the output indicators. The difficulties that have been experienced in the measuring of innovation include, inter alia, the lack of a common definition of innovation, the heterogeneous character of innovations, the difficulty to identify innovations as compared to inventions, the lack of available statistics and the differences in importance of innovations in view of the degrees of impact it may have.
The use of composite indicators has become popular to overcome the possible problem of using an incorrect or inaccurate single indicator. Unfortunately, many indices intending to measure innovative performance are compiled of capacity and performance indicators. A clear distinction should be made between the measurement of innovative capacity and innovative performance and also of the kind of indicators that are included in the index. It should also be noted that these composite indicators do not provide for qualitative differences and that an increase in innovation is not always good.
Extensive research is still needed to find a generally accepted measure for the performance of innovation systems to provide a basis for comparison and improvement in these systems.
References
Archibugi, D. & Coco, A. (2005). “Measuring Technological Capabilities at the Country Level: A Survey and a Menu for Choice,” Research Policy, 34, 175-194.
Publisher – Google Scholar
Audretsch, D. B. (2004). “Sustaining Innovation and Growth: Public Policy Support for Entrepreneurship,” Industry and Innovation, 11(3), 167-191.
Publisher – Google Scholar – British Library Direct
Becheikh, N., Landry, R. & Amara, N. (2006). “Lessons from Innovation Empirical Studies in the Manufacturing Sector: A Systematic Review of the Literature from 1993-2003,” Technovation, 26, 644-664.
Publisher – Google Scholar
Edquist, C. & Zabala, J. M. (2009). “Outputs of Innovation Systems: A European Perspective,” [Online], Lund University,(Paper no. 2009/14), [Retrieved August 14, 2012], Available:http://www.circle.lu.se/upload/CIRCLE/workingpapers/200914_Edquist_Zabala.pdf.
Publisher
Eggink, M. E. (2012). ‘The Role of Innovation in Economic Development,’ D.Com. Thesis, University of South Africa, Pretoria.
Fagerberg, J. & Srholec, M. (2008). “National Innovation Systems, Capabilities and Economic Development,” Research Policy, 37, 1417-1435.
Publisher – Google Scholar
Fagerberg, J., Srholec, M. & Verspagen, B. (2009). “Innovation and Economic Development,” [Online], University of Oslo, (TIK Working Papers on Innovation Studies, No. 20090723), [Retrieved February 21, 2011], Available:http://ideas.repec.org/s/tik/inowpp.html.
Publisher – Google Scholar
Fraas, M. (2003). “Oslo Innovation Scoreboard — An Attempt to Develop a Comparable Set of Indicators Used in the Regional Innovation Scoreboard for the Oslo Region,” Sintef, (Report No: STF38A03815).
Publisher – Google Scholar
Greenhalgh, C. & Rogers, M. (2010). Innovation, Intellectual Property and Economic Growth, Princeton University Press, Oxford.
Publisher – Google Scholar
Gregersen, B. & Johnson, B. (2005). Performance of Innovation Systems: Towards a Capability Based Concept and Measurements, [Online] (Paper Read at the Third Globelics Conference on Innovation Systems as Framework for the Promotion of Economic Growth, Social Cohesion and Political Development, October 31, Pretoria, South Africa), Unpublished, [Retrieved August 14, 2012], Available: http://www.respaldo.fcs.edu.uy/enz/licenciaturas/sociologia/cts.modulo4_Gregersen_Johnson.pdf
Publisher – Google Scholar
Hagedoorn, J. & Cloodt, M. (2003). “Measuring Innovative Performance: Is There an Advantage in Using Multiple Indicators?,” Research Policy, 32, 1365-1379.
Publisher – Google Scholar
Hanusch, H. & Pyka, A. (2007). “Principles of Neo-Schumpeterian Economics,” Cambridge Journal of Economics, 31, 275-289.
Publisher – Google Scholar – British Library Direct
Hasan, I. & Tucci, C. L. (2010). “The Innovation-Economic Growth Nexus: Global Evidence,” Research Policy, 39, 1264-1276.
Publisher – Google Scholar
Howells, J. (2005). “Innovation and Regional Economic Development: A Matter of Perspective,” Research Policy, 34, 1220-1234.
Publisher – Google Scholar
LeBel, P. (2008). “The Role of Creative Innovation in Economic Growth: Some International Comparisons,” Journal of Asian Economics, 19, 334-347.
Publisher – Google Scholar
Nelson, R. R. (1996). The Sources of Economic Growth, Harvard University Press, London.
Publisher – Google Scholar
OECD & Eurostat See Organisation for Economic Co-Operation and Development and Statistical Office of the European Communities.
Organisation for Economic Co-Operation and Development and Statistical Office of the European Communities (2005). ‘Oslo Manual: Guidelines for Collecting and Interpreting Innovation Data,’ The Measurement of Scientific and Technological Activities, 3rd Ed., OECD/European Communities, Paris.
Google Scholar
Ray, G. F. (1980). “Innovation as the Source of Long Term Economic Growth,” Long Range Planning, 13(2), 9-19.
Publisher – Google Scholar
Rogers, E. M. (1995). Diffusion of Innovations, 4th Ed., The Free Press, London.
Publisher – Google Scholar
Schumpeter, J. A. (1961). ‘The Theory of Economic Development: An Inquiry into Profits, Capital, Credit, Interest and the Business Cycle,’ Translated by R. Opie, Harvard University Press, USA.
Google Scholar
Schwab, K. & Porter, M. E. (Eds.) (2008). The Global Competitiveness Report: 2008-2009, [Online], World Economic Forum, [Retrieved October 24, 2008], Available: http://www.weforum.org/documents/GCR0809/index.htm/.
Publisher
Smith, K. H. (2005). Measuring Innovation, In: Fagerberg, J., Mowery, DC. And Nelson, RR. (Eds.). The Oxford Handbook of Innovation, Oxford University Press, New York, 148-177.
Publisher – Google Scholar
Van Duijn, J. J. (1983). The Long Wave in Economic Life, George Allen & Unwin, London.
Publisher – Google Scholar
