Introduction

For the current research, we have used several world databases in order to seek relevant articles on methods and mechanisms for evaluating ICT investment projects at an international level. Thus, we have identified and analyzed hundreds of articles published in – more or less famous – scientific magazines in the past five years.

As with any other field, the methods employed are either analytical or interpretive.

Analytical methods can be used when the field under research is well defined and quantifiable – quantitative or qualitative – indicators characterize the current situation. Mathematic or statistical methods are used to process collected data.

Interpretive methods apply when the area under scrutiny is not clearly defined, for example, in case of analyzing the ICT impact on the socio-economic development of a country  (Prashant Palviaa, 2018). An analysis of references reveals that ICT has an impact on a multitude of socio-economic development areas. Such an analysis could never lead to unanimously accepted general results. Therefore, it is only interpretive methods that can be employed, which allows experts to decode data and information, as well as personal observations.

It is the case of Smart Cities (March, 2018) whose analysis can only be based on interpretation. In the past several years, there have been vivid debates for and against smart cities. Though, at a first glance, the impact is entirely positive, a thorough, multidisciplinary analysis reveals certain negative consequences such as: IT illiterate citizens are discriminated against, there is infringement of privacy, certain public services are vulnerable to cyber-attacks, some public services and activities (emergency services or health services) depend on technology, yet there is no technology available, etc. Therefore, setting objectives or indicators in connection with the implementation of major ICT investment projects should only be based on a complex and thorough analysis of all risks and consequences, corroborated with backup measures.

Evaluating the Impact of ICT Investment Projects – General Methods and Models

Reference Analysis

The analysis of a public transportation project, within a presumably smart city framework (P. Tafidis, 2017) is an example of a multidisciplinary approach in evaluating the impact of an ICT investment project. The analysis aims to show how an ICT public transportation project can reduce pollution. As there is no common methodology, unanimously accepted, the authors have chosen three different regions (in Romania – Bucharest, Ilfov, Spain and Portugal) trying to see how an ICT project in public transportation could reduce pollution. Comparative methods have also been employed.

A serious and relevant analysis of an ICT investment project can only be conducted if there is a clear understanding of the project users/beneficiaries, with their characteristics and dependencies. The adoption of any technology depends on multiple factors and parameters: the users’ age and level of education, their social background, their health, etc. (Mattia Cattaneo, 2016). As one may expect, beneficiaries with high level of education are more likely to adopt new technologies. If in case of young people, we take into consideration the level of the formal education, when it comes to adults, non-formal education as well as Life Long Learning are of major significance and are also considered.

As an overall conclusion, most articles are the result of scientific research conducted by one or more teams from universities in developed countries, sometimes in partnership with universities or organizations in developing or less developed countries. Such research is mostly related to projects financed by international institutions (for example, the World Bank or government development agencies), implemented in developing countries.

Another category of articles/research refers to the implementation of ICT investment projects within companies and their impact mainly on the employees’ attitude and behavior (and not on production efficiency or on delivering services to the end beneficiaries). We should point out that there are numerous articles/analyses meant to assess the end clients’ satisfaction following the implementation of such an investment project.

Apparently, such analyses may have little to do with the current research; however, a closer look will show the opposite. The articles we analyzed reveal a sort of approach that should be considered when looking into an ICT investment project. Moreover, taking into account the membership of the research teams, we see yet again that the specificity of the target group (local character or the type of project users/beneficiaries) is of utmost importance when monitoring or assessing such a project. All the analyses/research of the projects implemented in developing countries have been conducted by teams which included people/organizations from the respective countries.

The references also show that one cannot assess a technology investment project without analyzing first its impact on users and beneficiaries, including the extent to which they use the infrastructure created. To be relevant such an analysis should cover a long period of time.

An illustration of an efficient approach to monitoring and assessing an ICT investment is the analysis of a project carried out in Arua region, Uganda, whose aim was to facilitate communication between citizens and local authorities. (Guy Grossman, 2018). The project was implemented by a non-government organization (GAPP), which supported the implementation of a platform that allowed the citizens to send anonymous messages to the public administration. The technical component (the creation of the platform) was accompanied by campaigns meant to promote and raise the people’s awareness of the new facilities. Training sessions for the public administration and several needs analyses were also conducted. The main impact of the platform was supposed to target health, education and drinking water supply. The research indicated a good impact with regard to the use of the platform in education, but only on a short and medium term (especially in the first year), without indicating significant results in the other fields. Once again, it is obvious that a medium- and long-term analysis as well as a multidisciplinary approach, tailored to the needs of the users/beneficiaries, are required if we want to show the real success of a project.

In most cases, implementation of an ICT investment project is not the result of a need identified independently by the future beneficiaries or users. It is the result of the interaction / the involvement of experts/specialists or of the clients’ requests or of other external factors (civil society, media, citizens – individually or as groups of people) or it is based on comparative analyses of similar organizations.

In most cases, adopting new technologies leads to changes and both people and organizations are resistant to change. Though one might expect that people and public institutions are more resistant to change, studies show that companies are equally resistant (Consoli, 2012). When it comes to adopting changes, the leading force in a company is determined by the business productivity, while small and medium sized enterprises (SMEs) have better abilities to adapt. Studies show that very few SMEs implement ICT investment projects based on an investment plan. In most cases, it is the replacement of old equipment or managerial changes that lead to such projects.

The implementation of any ICT project involves two components: the investment and the implementation of the respective change, which can be personal, at the department/organization level, a change of behavior, etc. Both components strongly depend on the leadership of the organization. Leadership includes here the combination among the personalities of those leading the organizations, their knowledge and abilities, regulations, procedures and organizational policies as well as the organizational culture. In most cases, the implementation of a new ICT system allows the managers’ access to valuable tools and information. Yet, the leadership methods and ICT tools are not always compatible. (Montgomery Van Wart, 2017). As a result, the thorough analysis of the organization management (people involved, knowledge, abilities, organizational culture, etc.) is critical in assessing the impact of an ICT investment project in order to see if the maximum achievement has been reached.

The influence of leadership in adopting new technologies is valid both for public and private enterprises. According to studies, leadership is the most important factor that influences the implementation of an ICT investment project. (Qureshi, 2013).

Communication is as influenced as leadership by the implementation of an ICT investment project (Asta Tarutėa, 2014). Both internal and external communication are facilitated and strongly influenced by the IT system/systems employed within the organizations. To be efficient and effective, the ICT infrastructure in the organization has to be perfectly adapted to the organizational procedures and culture, to its mechanisms and functioning. That is why most major ICT investment projects involve projects/actions that target the organizational change, the development of such activities requires perfect internal and external communication.

ICT investment projects have major effects on the whole organization and its interaction with the external environment. Considering the purpose of the analysis and the factors involved, the area under scrutiny is limited; certainly, there are areas influenced by the project implementation that are not covered by the present analysis. For example, any ICT investment project in the public system has or may have a certain impact on the fight against corruption (Anol Bhattacherjee, 2018). The analysis, the area analyzed, and the interpretation of the results are – in most cases – subjective. Therefore, such analyses are not unanimously accepted and could be questioned.

 Results of the Previous Research 

As resulted from the previous research (”Evaluating national, European and international investment projects – theories and methods” and „Virtual architectural solutions for organizations. Performance indicators”), there is no national/European/international unique, generally accepted approach regarding the assessment of a major ICT investment project.

Most assessment models and methods represent adaptations or versions of the assessment conducted for any investment project. If in case of a „traditional” investment project, most benefits are obvious and easy to evaluate, the impact of an ICT project is more complex. For example, a new building offers new living or production space and – if we compare the cost of the construction to the cost of renting a similar space, it is relatively easy to see if the investment is justified financially. Similarly, in case of a production investment (equipment or production line), it is relatively easy to see if the investment is profitable by adding the investment costs and the cost of maintenance and supplies/raw materials.

At an international level, we have identified ICT investment evaluation models only for major projects, multi-annual national strategies and multi-projects. Such analyses reveal a multidisciplinary approach and try to isolate the investment impact from other external factors, for example by selecting a similar country or community where such an investment has not been implemented. The approach is efficient and justified scientifically, but it is extremely expensive. Therefore, it is worth using it only in case of projects with national impact, whereas for small projects the cost of such an approach might equal the total project costs or even worse, it may exceed them.

Major international donors (European Commission, World Bank, etc.) have not developed specific evaluation methods and practices for the assessment of the ICT investments either. For example, in Europe there are significant differences among the objectives various countries have set for the funds in this field. Some countries (like the Baltic countries) have chosen to dedicate Operational Programs to ICT, with programmes indicators relevant at a national level (e.g.: following the implementation, all national public services shall be available electronically), other countries (like Romania) have chosen less ambitious evaluation indicators (such as the number of companies that benefitted from funding within the program). Other countries did not allocate special funds for this area at all, considering that ICT investment projects should be treated like any other investment projects and that it is up to the applicant (especially in the case of private beneficiaries) to decide if they want to invest in ICT or would rather purchase other assets.

In Romania, particularly the EU funded projects reveal results and outcome indicators that try to quantify the impact of the ICT investment projects by means of specific indicators (for example: the number of infrastructure users, the equipment purchased, etc.). However, reaching such indicators does not mean that the investment has been efficient or profitable; it means that it was necessary and it may represent an example of good or bad practice. One could reach such a conclusion only based on a real comparison of the costs and benefits, whether the project is implemented or not. However, for public infrastructure, it is very difficult to make such comparisons; even if they are possible (for example by identifying information faster, costs are significantly reduced), there is no wish to do so.

A comparative approach that would look into the impact of two or more projects implemented under similar circumstances, for similar beneficiaries, generating measurable and quantifiable benefits, could be a method to establish the efficiency of an ICT investment project. Such situations have not been identified and analyzed yet. Projects which look similar at a first glance (for example, implementing two e-government projects with the local government in cities of equal size), with similar budgets (approximately 1,5 million euro), implemented at the same time under the same regulation framework (European funds, the same call for projects) do not lend themselves to such analysis due to scarce or no information at all. Moreover, in case of such projects, a comparison should consider other important factors such as: the citizens’ standard of living, the Internet use in the city, the broadband availability and quality, and the population age. Thus, the analysis becomes complex and it is impossible to assess the impact of each parameter on the overall final result.

Moreover, even under similar funding conditions, the two local authorities may have different approaches. One may decide to allocate a larger portion of the budget for the content (considering that it will bring added value, and the platform ‚hosting’ is a cost they will cover), while the other may decide that it is essential to have access to strong hardware and software infrastructure. Based on this infrastructure, if there is a need, they will develop – with their own resources – various services for citizens. To compare the two approaches, one must analyze the impact of implementing the two investments for a longer period of time, that is throughout the investment life (conventionally, 5years)

 Case Study: ICT Investment Projects In Romania, Implemented With EU Non-Reimbursable Funds

Here are the funding sources for the ICT investment projects implemented in Romania, from the 2007-2013 EU budget:

• Sectoral Operational Programme – Increasing the Economic Competitiveness 2007-2013 (SOP IEC). The program included the Priority Axis III – Information Communication Technology (ICT) for the private and public sectors. This line funded ICT investment projects exclusively (Ministry of Communication and Information Society, 2018)
• Sectoral Operational Programme – Increasing the Economic Competitiveness 2007-2013 (SOP IEC). The programme (Priority Axis I dedicated to SMEs investments) financed several investment projects developed by companies. Some projects also covered – to a larger or lesser extent – ICT investment (Ministry of Research and Innovation, 2018)
• Sectoral Operational Programme – Increasing the Economic Competitiveness 2007-2013 (SOP IEC). The Priority Axis II dedicated to supporting research and innovation funded several investment projects carried out both by companies and public organizations (mainly research institutes and universities). Some projects also covered – to a larger or lesser extent – ICT investments. There have been actions dedicated to ICT related projects, such as the GRID projects (Ministry of European Funds, 2018),
• Regional Operational Program 2007-2913 (ROP) – included a line dedicated to investments of the small-size institutions. Some of the projects implemented represent SMES investment in ICT (Regional Operational Program, 2018).
• Sectoral Operational Programme – Development of Human Resources 2007 – 2013 (SOP DHR). This funding programme funded several projects carried out by public institutions, which included the purchase or development of applications of local, regional or national interest (Ministry of European Funds, 2018)
• Operational Programme – Development of Administrative Capacity 2007 – 2013 (OPDAC). The programme financed several projects implemented by public authorities with major components, which included the development of applications (Ministry of Regional Development and Public Administration, 2018)

Besides the funding sources above, all projects funded from non-reimbursable EU funds included an ICT horizontal objective.

Unfortunately, no information about the result or the impact of the implementation is available. Moreover, for the sources that funded not only ICT investments, there is no way to separate such projects from those which did not have such components. For example, out of all projects funded from ROP dedicated to microenterprises, we do not know the number of projects that implemented ICT related investments or purchased other type of equipment or invested in construction and buildings.

Under the circumstances, besides the analyses conducted in the previous studies, we can only conduct a quantitative analysis of the projects funded from SOP IEC Priority Axis III, dedicated to ICT.

In 2016, based on Freedom of Information Law, we asked the Ministry of European Funds to release the database of all payments made within the operational programmes as part of the 2007-2013 budget. At that time, all projects funded from 2007-2013 had been completed (deadline for completion 31 December 2005), checked by the Romanian authorities in charge while the reports had been submitted to the European Commission. Thus, the database received from the Ministry of European Funds shows the final situation of the programmes financed from the respective operational programmes. Compared to the huge volume of information received, minor changes are likely to occur in future.

The actions financed from European funds and included in the database covered a long period of time (01.01.2007 – 31.12.2015). Therefore, for a clear picture, it was necessary to convert the RON data base to EUR. To secure the accuracy of the results, we have used the exchange rate of the National Bank of Romania on the day the payment was made by the recipient of funding to the supplier/service provider/the selected builder.

From the data base released by the Ministry of European Funds, we selected the information about the projects financed within SOP IEC Priority Axis III, during 2007-2013. We identified 1.966 of such projects. Their distribution between public and private beneficiaries is shown in the next table.

Table 1: Distribution (number) of projects financed by SOP IEC Priority Axis III between public and private organizations

As we can see, more than 90% of the ICT investment projects were implemented by private entities. Though it may look surprising, we should bear in mind that Priority Axis III had calls for projects dedicated to the private entities; yet, their budget was very small (e.g.: under 100 000 RON for priority 3.1.1). There were also calls for projects for the public institutions; this time the eligible value was much higher, most of them reaching approximately 1, 500 000 euro

Obviously, it was relevant to analyze the distribution of the projects between the public and the private sectors based on the eligible value accepted by the funder at the end of the project. The result is shown in the figure below.

Figure 1 : Distribution (in terms of the eligible values) of the projects funded by SOP IEC Priority Axis III between public and private organizations

As expected, as far as the eligible value is concerned, the distribution between the private and public sectors is quite the opposite compared to the number of projects financed for each sector.

Almost all types of organizations registered in Romania were eligible for this priority axis. The budgets of the eligible expenses covered by each type of organizations are shown in the next table.

Table 2:  Covering the budget spent within SOP IEC Priority Axis III according to the type of the beneficiary organization

The geographic distribution of the eligible values accepted by the funders for all the projects within SOP IEC Priority Axis III is also relevant. However, a global analysis of the projects implemented by both public and private beneficiaries would not be relevant, considering the huge discrepancy between the eligible values of the projects implemented by the two categories of beneficiaries.

The geographic distribution is illustrated in the next figures.

Figure 2 : Geographic distribution of the projects implemented by private organizations

The distribution of funds varies from region to region, with a major discrepancy between Bucharest-Ilfov Region, with 30% of the total value and South West Oltenia Region with 3% of the total value. The number of private companies active in the respective regions, as well as their dynamics, account for such a discrepancy.

To analyze the geographical distribution of the projects implemented by public institutions, we should bear in mind that we should not compare a programme implemented by a city hall to a programme implemented by a ministry. Both in terms of impact and of complexity / costs, the two types of projects are completely different.

The database released by the Ministry of European Funds includes a field referring to the type of beneficiary. Using that field to filter the data would not lead to accurate results (there are validation errors; for example, in the same category we can find both a Prefecture and the Labor Inspection Office). That is why we have chosen to filter the data according to the beneficiary’s location, eliminating those based in Bucharest. Thus, we have excluded the local authorities in Bucharest, which is a disadvantage. Yet, it is not a major downsize considering that, anyway, the local authorities in Bucharest, are – in most cases – completely different compared to those in the rest of the country.

The geographical distribution of the projects implemented by public institutions all over the country, Bucharest excluded, is shown in the next graph.

Figure 3 : Geographic distribution of the projects implemented by public institutions outside Bucharest

Evaluating E-Learning Projects

Analysis of the International Experience

“An e-learning platform is a web ecosystem used to disseminate information, communicate and transfer knowledge in education and training processes” (Wilmar Audye Cidral, 2018)

As we have seen before, an ICT platform is an instrument (not an objective or a goal) to reach indicators. From this point of view, in order to assess strictly the investment project in ICT (in our case, the implementation of an e-learning platform), we should only measure to what extent the investment has helped or failed to achieve the proposed objectives (i.e. the quality of the “tool”) separating this assessment from the quality of the supported processes or activities. The technical analysis of the e-learning platform is a relevant activity but is of particular interest to the technological field (Guseva, 2017)

In order to evaluate the project implementation of an e-learning platform, we should separate this assessment from the quality of the educational process, the quality of course materials, teachers / trainers, etc. However, such a separation is not possible. Clearly, the success of the e-learning platform implementation is also indicated by the satisfaction of its users (teachers / instructors and students / trained people). This satisfaction can only be evaluated accurately for the overall use of the platform and not for its components.

Once again, theoretically, the incremental method is a more appropriate assessment tool (i.e.: comparing the degree of satisfaction of the platform users with the satisfaction of users that have undergone the same educational / training process, but do not have access to the platform) or the benchmarking  (comparison with the same education / training process carried out without using the e-learning platform, possibly by a similar institution). Such methods, though theoretically relevant, are actually impossible to apply because there are not two identical learning / education / training processes we could compare. Without considering other perspectives and elements, this type of processes is strongly influenced by participants. And the learners are different from one teaching process to another, making the groups impossible to compare.

A comparative method (for example, using a “control group”) can be used when we want to evaluate a limited impact, a punctual or specific action of a defined target group (e.g. when analyzing the influence of adaptive e-learning components on the efficiency of children’s learning process in the primary cycle – (S. Hubalovsky, 2018)

Similarly to other complex ICT systems such as e-governing or e-health, the situation described above does not mean that there have been no credible, robust, scientific assessments of e-learning platforms. There have been attempts to verify proposed theories and methods in given situations by applying these assessment methods to one or more implementations (e.g.  the analysis presented in (Salem Alkhalaf, 2012).

One of the elements used to assess the development of an e-learning platform is the interactivity of manuals available on the platform for the students. Although at first glance we might think that an interactive manual is more effective than a static pdf, in some cases it seems that this is just a student’s perception, and, as far as test results are concerned, the situation is exactly the opposite (Cathy Weng, 2018)

Analyzing the quality of the training process carried out through an e-learning platform is extremely difficult. The research shows that there is no clearly defined and generally accepted methodology by the academic environment regarding the qualitative assessment of an e-learning platform. Such an assessment can be made from two points of view: technological and educational (Vlastimir Nikolić, 2018)

It seems, however, that it is almost universally accepted that the technological perspective is not relevant. As far as the educational perspective is concerned, it needs to be addressed with specific pedagogical tools, though they should be adapted to new ICT-specific tools. Such adaptation has not yet been achieved, with several global approaches and various proposals made by the scientific community.

A special category of e-learning platforms is the use of these tools in order to provide courses to employees. The evaluation of such platforms differs from that of a university e-learning platform, for example. The differences derive from the fact that the two categories of users (the employees and the students) attach different importance to the various facilities of an e-learning platform. For the employees, it is the flexibility (they have access to the courses at any time and do not have to leave the workplace), repeatability and the smooth adjustment to changes that matter. (Nima Jafari Navimipour, 2015). Therefore, we could conclude that the defining elements of a “quality e-learning platform” are specific to the field of activity / use

Analysis of the Main Projects Implemented in Romania

E-learning projects are a sub-category of ICT investment projects run by public institutions. Thus, this type of projects retains all the characteristics of the “mother” class, i.e. ICT investment projects carried out by public institutions while also including specific sub-class characteristics.

One of the main features of e-learning projects in Romania in recent years is the emphasis on the amount of content available at the expense of the user’s interaction complexity with the e-learning platform (e.g. interactive manuals). According to results of the analysis in the previous sub-chapter, the degree of technological complexity of digital content is not always proportionate to the efficiency and effectiveness of the knowledge transfer. However, the power and utility of complex system-user interaction systems should not be played down.

Within the SOP IEC Priority Axis III, there was a financing line (Operation 3.2.3 Supporting the implementation of e-learning applications) dedicated to such projects. For private beneficiaries (companies), there was a funding line where e-learning projects were also eligible, but that line was common to other types of projects of interest to the private sector (e.g. e-commerce), therefore, it is impossible to separate projects by their type. On the other hand, analyzing the list of projects approved for funding, the title of the project shows that the number of private firms interested in acquiring e-learning applications was extremely small, making their contribution and importance to the entire market irrelevant.

For these reasons, the analysis of e-learning projects is limited to projects financed through SOP IEC 3.2.3.

The lack of interest and evaluation criteria for the complexity of the content of an e-learning platform is also reflected in the result and outcome indicators established by the applicant’s guide, which are strictly quantitative indicators (such as the number of available courses or number of users) without taking into account qualitative elements (such as the type of courses available, their degree of complexity, the level of interaction between the platform and users, user satisfaction, etc.).

As with the general case of ICT investment projects carried out by public institutions, in case of e-learning projects, a relevant analysis is that of the categories of expenditures covered by the project. The situation is presented in the following table:

Table 4 : Eligible values for types of expenditures covered by the e-learning projects

We find that for these projects, the discrepancy between the expenditure for the acquisition of intangible assets and the expenditure for the acquisition of tangible assets is even stronger than in the general case of publicly funded ICT investment projects (63% / 21% for e-learning projects compared to 50% / 31% for all projects carried out by public institutions).

This result is not surprising given that for e-learning projects, in addition to the projects carried out by public institutions, additional modules / extra costs are expected to adapt the digital content to the new platform.

The analysis of the available information reveals a surprising result in terms of the regional distribution of e-learning projects implemented.

Figure 4  : Regional distribution of the e-learning projects

The very large share of projects implemented in the Bucharest – Ilfov Region is no surprise, given that Bucharest is the largest university center and it is supposed to be best informed and connected to financing sources. Surprisingly, however, there is a total lack of funding in the Western Region (as Timisoara is one of the largest university centers) as well as in the South Western Region of Oltenia (where a large university center is located in Craiova).

Conclusions and Recommendations

The analysis of the international approaches of assessing the impact of ICT investment projects clearly shows that this area is new and it is currently developing. Generally accepted ICT assessment methods and mechanisms have not been identified yet, not even for parts of the ICT use (public administration, health, e-learning, etc.)

One consequence is that there is no clear definition of what a successful project means and what a failure means in this area. If extremes are clear (certainly a project that is not used at all represents a failure while an information system that is used by all potential users in all components of the system is undoubtedly a success), the scale of the success of a project is still a subjective theme, depending on who makes the assessment, his/her interests and understanding.

However, there are some conclusions that are generally valid for all approaches and situations under consideration. Here are some of the most important

• Using ICT is just a tool for achieving specific goals. Achieving an ICT infrastructure should not be an objective in itself, but a resource the user needs to achieve certain goals (higher efficiency, better access to knowledge he would have been unable to reach otherwise, etc.). Consequently, the fact that at the end of the project implementation, the ICT infrastructure is functional and does not require interventions is compulsory, yet insufficient for a successful project.
• In most cases, the need of running an ICT investment project does not necessarily come first from the users or beneficiaries of that infrastructure. The need for technology can only arise when we have understood how that technology works and how / under what conditions it works. So “ICT demand” needs to be generated and developed by other stakeholders (experts, traders, observers, etc.) who must help potential users and technology users to demand, support and use such IT systems.
• In most cases, the implementation of a major ICT investment project, as well as its monitoring and evaluation, can be conducted only by multidisciplinary teams that, above all, have the knowledge and skills required by area under scrutiny; the technical knowledge regarding the technological functioning of the investment comes second.
• The impact of implementing an ICT investment project is of major importance, affecting the organization as a whole, as well as its interactions with external environment. Depending on the purpose of the preparatory analysis or evaluation, as well as on the stakeholders, the area under analysis or evaluation is only limited to certain aspects, while certain areas are not analyzed or rated.
• The analysis as such, the analyzed area, the interpretation of the results depends – in most cases – on the analyst, who is subjective. That is why, in most cases, such impact analyses are questionable and not unanimously accepted

An example proving the importance of understanding the limitations of an impact analysis is given by the outcome of a e-learning research. Although at the first glance we might think that, in some circumstances, an interactive manual is more efficient than a static pdf, it seems that this is just a perception of the student / user and that in terms of test results, the situation is exactly the opposite. At the same time, this surprising finding proves that investment in more sophisticated technologies is not always justified, and sometimes even such choices could have adverse effects.

References

1. Zheng, Y., Schachter, H. L. & Holzer, M., 2014. The impact of government form on e-participation: A study of New Jersey municipalities. Government Information Quarterly, 31(2014), p. 653–659. http://dx.doi.org/10.1016/j.giq.2014.06.004.
2. Nam, T., 2018. Examining the anti-corruption effect of e-government and the moderating effect of national culture: A cross-country study. Government Information Quarterly, 35(2018), pp. 273 – 282. https://doi.org/10.1016/j.giq.2018.01.005.
3. Gao, X. & Lee, J., 2017. E-government services and social media adoption: Experience of small local governments in Nebraska state. Government Information Quarterly, 34(2017), pp. 627-634. http://dx.doi.org/10.1016/j.giq.2017.09.005.
4. Sharma, P. N., Morgeson, F. V., Mithasc, S. & Aljazzaf, S., 2018. An empirical and comparative analysis of E-government performance measurement models: Model selection via explanation, prediction, and parsimony. Government Information Quarterly, XXX(XXX), pp. XXX – XXX. https://doi.org/10.1016/j.giq.2018.07.003.
5. Rose, J., Flak, L. S. & Sæbø, Ø., 2018. Stakeholder theory for the E-government context: Framing a value-oriented T normative core. Government Information Quarterly, 35(2018), pp. 362-374. https://doi.org/10.1016/j.giq.2018.06.005.
6. Sangki, J., 2017. Vision of future e-government via new e-government maturity model: Based on Korea’s e-government practices. Telecommunications Policy, xxx(2017), pp. 1-12. https://doi.org/10.1016/j.telpol.2017.12.002.
7. Stefanovic, D. et al., 2016. Assessing the success of e-government systems: An employee perspective. Information & Management, 53(2016), p. 717–726. http://dx.doi.org/10.1016/j.im.2016.02.007.
8. Das, A., Singh, H. & Joseph, D., 2017. A longitudinal study of e-government maturity. Information & Management, 54(2017), pp. 415-426. http://dx.doi.org/10.1016/j.im.2016.09.006.
9. Kumar, R., Sachan, A. & Mukherjee, A., 2017. Qualitative approach to determine user experience of e-government services. Computers in Human Behavior, 71(2017), pp. 299-306. http://dx.doi.org/10.1016/j.chb.2017.02.023.
10. Mahmoodi, R. K. & Nojed, S. H., 2016. Investigating the Effectiveness of E-government Establishment in Government Organizations. Dubai, 3rd International Conference on New Challenges in Management and Organization: Organization and Leadership, 2 May 2016, Dubai, UAE, pp. 136-141. Available online at www.sciencedirect.com.
11. Serra, L. C. et al., 2015. Accessibility Evaluation of E-Government Mobile Applications in Brazil. Vila Real, Portugal, 6th International Conference on Software Development and Technologies for Enhancing Accessibility and Fighting Infoexclusion (DSAI 2015), p. 348 – 357.
12. Weng, C., Otanga, S., Weng, A. & Cox, J., 2018. Effects of interactivity in E-textbooks on 7th graders science learning and cognitive load. Computers & Education, 120(2018), pp. 172-184. https://doi.org/10.1016/j.compedu.2018.02.008.
13. Cidral, W. A., Oliveira, T., Di Felice, M. & Aparicio, M., 2018. E-learning success determinants: Brazilian empirical study. Computers & Education, 122(2018), pp. 273-290. https://doi.org/10.1016/j.compedu.2017.12.001.
14. Nikolić, V. et al., 2018. Survey of quality models of e-learning systems. Physica A, 511(2018), pp. 324-330. https://doi.org/10.1016/j.physa.2018.07.058.
15. Navimipour, N. J. & Zareie, B., 2015. A model for assessing the impact of e-learning systems on employees’ satisfaction. Computers in Human Behavior, 57(2015), pp. 475-485. http://dx.doi.org/10.1016/j.chb.2015.07.026.
16. Hubalovsky, S., Hubalovska, M. & Musilek, M., 2018. Assessment of the influence of adaptive E-learning on learning effectiveness of primary school pupils. Computers in Human Behavior, XXX(XXX), pp. XXX – XXX. https://doi.org/10.1016/j.chb.2018.05.033.
17. Alkhalaf, S., Drew, S. & Alhussain, T., 2012. Assessing the impact of e-learning systems on learners: a survey study in the KSA. Procedia – Social and Behavioral Sciences, 47(2012), pp. 98-104. doi:10.1016/j.sbspro.2012.06.620.
18. Drozdova, A. A. & Guseva, A. I., 2017. Modern technologies of e-learning and its evaluation of efficiency. Almeria, Spain, 7th International Conference on Intercultural Education “Education, Health and ICT for a Transcultural World”, EDUHEM 2016, 15-17 June 2016, , p. 1032 – 1038.
19. Haryaka, U., Agus, F., Ramadiani & Azainil, 2017. User Satisfaction Model for e-Learning Using Smartphone. Bali, Indonesia, 2nd International Conference on Computer Science and Computational Intelligence 2017, ICCSCI 2017,, p. 373–380.
20. Grossman, G., Platas, M. R. & Rodden, J., 2018. Crowdsourcing accountability: ICT for service delivery. World Development, 112(2018), pp. 74-87. https://doi.org/10.1016/j.worlddev.2018.07.001.
21. Palvia, P., Baqir, N. & Nematia, H., 2018. ICT for socio-economic development: A citizens’ perspective. Information & Management, 55(2018), pp. 160-176. http://dx.doi.org/10.1016/j.im.2017.05.003 .
22. Cattaneo, M., Malighetti, P. & Spinelli, D., 2016. The impact of University of the Third Age courses on ICT adoption. Computers in Human Behavior 63, 63(2016), p. 613e619. http://dx.doi.org/10.1016/j.chb.2016.05.087 .
23. Consoli, D., 2012. Literature analysis on determinant factors and the impact of ICT in SMEs,. Procedia – Social and Behavioral Sciences, 62(2012), pp. 93-97. doi:10.1016/j.sbspro.2012.09.016.
24. Roman, A., Roman, A., Wang, X. & Liu, C., 2017. Integrating ICT adoption issues into (e-)leadership theory. Telematics and Informatics, 34(2017), p. 527–537. http://dx.doi.org/10.1016/j.tele.2016.11.003.
25. March, H., 2018. The Smart City and other ICT-led techno-imaginaries: Any room for dialogue with Degrowth?. Journal of Cleaner Production, 197(2018), pp. 1694-1703. http://dx.doi.org/10.1016/j.jclepro.2016.09.154.
26. Qureshi, A. A., 2013. Impact of Leadership on Meaningful Use of ICT. Brussels, Belgium, 3rd World Conference on Learning, Teaching and Educational Leadership – WCLTA 2012, pp. 1744 – 1748.
27. Tarutėa, A. & Gatautisa, R., 2014. ICT impact on SMEs performance. Procedia – Social and Behavioral Sciences, 110(2014), p. 1218 – 1225. doi:10.1016/j.sbspro.2013.12.968.
28. Tafidis, P. et al., 2017 . Exploring the impact of ICT on urban mobility in heterogenic regions. Budapest, Hungary, 20th EURO Working Group on Transportation Meeting, EWGT 2017, 4-6 September 2017,, p. 309–316.
29. Bhattacherjee, A. & Shrivastava, U., 2018. The effects of ICT use and ICT Laws on corruption: A general deterrence theory perspective. In Press, Corrected Proof, pp. 1-30. https://doi.org/10.1016/j.giq.2018.07.006.
30. Ministerul Cercetării și Inovării, 2018. Organismul Intermediar pentru Cercetare. [Online], Available at: http://www.poc.research.gov.ro, [Accessed 20 07 2018].
31. Ministerul Comunicațiilor și Societății Informaționale, 2018. Organismul Intermediar pentru Promovarea Societății Informațopmașe. [Online] , Available at: http://fonduri.mcsi.ro , [Accessed 20 07 2018].
32. Ministerul Dezvoltării Regionale și Administrației Publice, 2018. AM PODCA. [Online] , Available at: http://www.fonduriadministratie.ro/ro/ , [Accessed 20 07 2018].
33. Ministerul Fondurilor Europene, 2018. Ministerul Fondurilor Europene – POS DRU. [Online] , Available at: http://www.fonduri-ue.ro/posdru-2007 , [Accessed 20 07 2018].
34. Ministerul Fondurilor Europene, 2018. AM POS CCE. [Online] , Available at: http://www.fonduri-ue.ro/poscce-2007 , [Accessed 20 07 2018].
35. Ministerul Fondurilor Europene, 2018. AM POS CCE. [Online] , Available at: http://old.fonduri-ue.ro/poscce/ , [Accessed 20 07 2018].
36. Programul Operațional Regional, 2018. POR – Programul Operațional Regional 2014 – 2020. [Online] , Available at: http://www.inforegio.ro/ro/por-2014-2020.html , [Accessed 20 07 2018].