Evaluation of the Success of an Academic Portal: An Exploratory Study of the Staff’s Perspective

Introduction

With the evolution of information technology (IT), Higher Education Institutions (HEIs) have invested in the development of websites, often referred to as Academic Portals, to achieve strategic benefits, namely, greater efficiency and effectiveness in the management of students’ and employees’ information, as well as better quality services for stakeholders (Rakemane & Serema, 2018; Semeon et al., 2010). These Academic Portals are interactive tools that aim to facilitate communication among the academic community and provide helpful content and services (Al-Debei, 2014).

The role played by academic portals is increasingly crucial in HEIs, since they need integrated information systems that support all their functions and activities, which allows them to operate efficiently, make informed decisions and offer the best educational experience to students (Lupu et al., 2018). Thus, keeping these information systems up to date is essential, so the search for changes and improvements is continuous. Given this context, HEIs have invested heavily in creating and updating their Academic Portals (Mukerjee, 2012).

Given the importance and role of Secretaria Online (as an Academic Portal) in supporting the activities of the Instituto Superior de Contabilidade e Administração do Porto (ISCAP), it was considered relevant to conduct a study to assess the success of Secretaria Online adoption by the (teaching and non-teaching) staff. This paper presents the initial results of the assessment carried out by the staff of ISCAP, using DeLone and McLean’s model of information systems success.

Following this introduction section, this paper discusses the importance of information systems success and presents DeLone and McLean’s model of information systems success. Then, are presented the research approach and the main results. This paper ends with final considerations about this research.

Information Systems Success

In the last decades, Information Systems have gained particular importance for organizations, supporting most of their activities. It is a fact that organizations continue to increase spending on Information Technology and their budgets continue to rise, even in the face of potential economic downturns (Kanaracus, 2008). Given this importance, it has become increasingly necessary to evaluate its success. The literature suggests that in an organizational context, several factors may influence the success of information systems, and several studies identify these factors and include them in success models (e.g., (Davis, 1985; Venkatesh & Davis, 2000; Venkatesh et al., 2003; DeLone and McLean, 2003)). One of the literature’s most used and cited success models is the Information Systems Success model, often referred to as DeLone and McLean’s (1992, 2003, 2016). The application of DeLone and McLean’s model in various studies has shown different results; however, it has demonstrated that it can help identify the success variables of information systems used in different contexts or scenarios (Yakubu & Dasuki, 2018).

DeLone and McLean’s model (2013) considers six dimensions or constructs to measure the information systems’ success: System Quality, Information Quality, Service Quality, Intention to Use/Use, User Satisfaction and Net Benefits. System Quality refers to the desirable characteristics of an information system and measures its technical success. Information Quality considers the desirable characteristics of the system output in terms of content and the respective reports and, to some extent, measures semantic success. Quality of Service includes all the support that a system offers to users. Intent to Use and Use refers to how users of the system use the capabilities of the information system. User Satisfaction refers to how effective the user is and how satisfied they are with the results obtained from using the information system. Benefits capture the overall impact of the system on users.

Academic Portal – Secretaria Online

The Secretaria Online is an academic portal that allows the information management of the ISCAP community, being a complex information system that supports various activities of the institution. The Secretaria Online supports the management of academic activity and other areas such as human resources, accounting, instructional communication and teaching activities. This Academic Portal aims to enable access to ISCAP information and facilitate the transfer of information among the entire academic community, having as primary objectives: improving the quality of student services; improving critical periods in the schedule of services; improving the quality of academic information, and improving the working conditions of employees. Figure 1 shows an example of a screenshot of Secretaria Online.

Figure 1: Secretaria Online screenshot

Research Approach

As previously mentioned, this work aims to evaluate the success of adopting the Secretaria Online by the ISCAP’s staff (teaching and non-teaching), based on DeLone and McLean’s model. The research approach adopted involves a case study that allows interpreting and understanding the success of the environment (Walsham, 2006). This research applied a quantitative questionnaire, often used in information systems research, to determine an environment’s dependent and independent variables without controlling them (Pinsonneault and Kraemer, 1993). According to Isaac and Michael (1995), questionnaires may be helpful to: answer questions raised, resolve observed issues, assess needs and objectives in order to determine whether specific goals have been met, establish a baseline against which comparisons can be made in the future, analyze trends over time, and in general, describe what exists, how much, and in what context.

The questionnaire used in this study was built in the LimeSurvey tool and structured into three groups of questions: one group on demographic information; another group on the usage habits of the respondents; and the third group with the questions related to the evaluation of the Secretaria Online. The group of questions related to the evaluation of the Secretaria Online was based on the DeLone and McLean model, and 23 measures or questions were defined and distributed among the six constructs: five questions are related to the Information Quality construct, five to the System Quality, three to the Service Quality, three to the Intention to Use/Use, three to the User Satisfaction and four measures to the Benefits construct. These 23 measures resulted from an extensive literature review (Table 1). After identifying several measures already validated in related works, the ones that, in our opinion, best suited Secretaria Online’s evaluation were selected.

Table 1 – Measures selected for study

In order to identify and validate the 23 items of the questionnaire, a test was conducted with the participation of 25 users. Thus, participants received a detailed definition of the constructs and were instructed to indicate which construct they believed each item best represented. Therefore, this task followed Howard and Melloy (2016) suggestions. Table 2 shows the items with enough assignments for each construct. To answer to the 23 questions/items, a 5-point Likert Scale was used, with the following labels: 1 – I totally disagree; 2 – I disagree; 3 – I neither agree nor disagree; 4 – I agree; 5 – I totally agree.

After this validation, the questionnaire was sent to 246 teaching staff and 65 non-teaching staff. The questionnaire was available for completion for 38 days and was completed by 93 teaching staff (response rate: 37,8%) and 16 non-teaching staff (response rate: 24,6%), resulting in an overall response rate of 35%. Table 3 summarises the demographic information of 109 respondents. In this study, the software SPSS 26 for Windows was used for the data analysis needs. Statistics methods adopted in this paper are mainly descriptive statistics and the test score reliability coefficient Cronbach’s Alpha.

Table 2 – Measures items

 Table 3 – Demographic Information of respondents

Results and Discussion

Table 4 shows the statistical measures used in this study to validate the results. The standard deviation (Std Dev) values show that the data generally have a solid central tendency (Weisberg, 1991). The skewness (Skew) and kurtosis (Kurt) values, which measure the concentration or dispersion, also show that the distribution tends toward normality, assuming that the acceptable skewness values must be between ±3 and the appropriate kurtosis must be in the range of ±10 (Brown, 2006).

The Kaiser-Meyer-Olkin test (KMO) calculated to examine the appropriateness of factor analysis application shows that for all constructs, the value is higher than 0.6, thus fulfilling the appropriate conditions (Hair et al., 2014). Simultaneously, Bartlett’s test presented a sig value of 0.000, demonstrating the significance of the items that make up the same construct.

The Total Explained Variance (TEV) calculated for each construct presents, except for the “Use/Intention to Use” construct, values higher than 60%, which reveals that items underlying the same construct show an appropriate percentage of the explained total variance (Hair et al., 2014).

According to Souza et al. (2017), the quality of the information provided by instruments depends, in part, on their psychometric properties, highlighting the instrument’s reliability and validity as key in validating a questionnaire. Yu and Qian (2018) also refer that reliability and validity can be assessed by psychometric properties, which are measured by internal consistency and convergence. Based on this assumption, the Cronbach’s Alpha (CA) coefficient was calculated to assess the reliability of the questionnaire and its results. According to the literature, CA should present values above 0.7 (Nunnally & Bernstein, 1994); however, Carmines and Zeller (1979) suggest that values between 0.6 and 0.7 can be considered satisfactory. The CA value for almost all constructs allows high confidence in the validity of the results presented in this paper. However, there must be some caution regarding the Use/Intention to Use Construct results since the CA value is only 0.62 (despite being in a confidence interval considered satisfactory by some authors).

Table 4 – Statistical measures

Kaiser Normalization Varimax rotation method

Kaiser-Meyer-Olkin measure of sampling adequacy (KMO)= 0.916

Bartlett’s test sig. 0.000.

Total Explained Variance: 71.18%

In order to complement the data analysis, a paired-sample t-test was performed to check if there were significant differences between the responses of the items of each construct. Paired-sample t-tests are generally appropriate and widely used when there is a match (Zimmerman, 1997), in this case, on items related to the same construct. Table 5 presents the paired t-test results. Regarding the items of the Information Quality construct, QINF3 is the item that stands out for presenting statistically significant differences when compared with the other items of the construct, being the one that presents the highest mean response value. Conversely, the QINF5 item presents a lower mean response value, and the difference with the other items is also statistically significant. Concerning the items of the System Quality construct, item QSI1 is the one that shows a statistically significant difference from the other items, with a higher mean response value than the remaining items. In the items of the Quality of Service construct, item QS3 is the one that stands out the most for presenting a mean response value, statistically higher than the other items. The item USO1 is the one that stands out among the items of the Use/Intention to Use construct, presenting significant differences from the other items and being the one that presents a higher mean response value. Finally, in the User Satisfaction construct, item BEN3 has the highest mean response value, and item BEN4 has the lowest mean value, having both statistically significant differences.

Table 5 – Table paired t-test values

Conclusions

This paper presents the results of a study that evaluates an Academic Portal’s success, the case of Secretaria Online, from the perspective of ISCAP’s staff. This study applied one of the most used models in empirical studies (Semeon et al., 2010), the DeLone and McLean information systems success model, as a theoretical reference for this evaluation.

This paper presents an exploratory analysis of the data and an analysis of Cronbach’s Alpha (CA), which allows testing the lower limit of the internal consistency of a group of items related to each construct. The study also presents a comparative analysis of response items for each construct

The exploratory character of this study allowed gathering information and familiarization with the DeLone and McLean model in a school administrative context. The results obtained in this study presuppose that Secretaria Online plays a critical role in ISCAP’s activities; however, the evaluation provided by the users shows that several aspects must improve.

Acknowledgement

This work is financed by portuguese national funds through FCT – Fundação para a Ciência e Tecnologia, under the project UIDB/05422/2020.

References

• Al-Debei, M. M. (2014) ‘The quality and acceptance of websites: an empirical investigation in the context of higher education,’ International Journal of Business Information Systems, 15(2), 170-188.
• Al-Shibly, H. (2011) ‘Human resources information systems success assessment: An integrative model,’ Australian Journal of Basic and Applied Sciences, 5(5), 157-169.
• Bernroider, E. W. (2008) ‘IT governance for enterprise resource planning supported by the DeLone–McLean model of information systems success,’ Information & Management, 45(5), 257-269.
• Brown, T. (2006) “Confirmatory factor analysis for applied researchers”, Guilford Press, New York, USA.
• Carmines, E. G., & Zeller, R. A. (1979). Reliability and validity assessment. California: Sage publications, 1-72.
• Davis, F. D. (1985). A technology acceptance model for empirically testing new end-user information systems: Theory and results. Doctoral dissertation, Massachusetts Institute of Technology.
• DeLone, W. H., & McLean, E. R. (1992) ‘Information Systems Success: The Quest for the Dependent Variable,’ Information Systems Research, 3(1), 60–95.
• DeLone, W. H., & McLean, E. R. (2003) ‘The DeLone and McLean model of information systems success: A ten-year update,’ Journal of Management Information Systems, 19(4), 9-30
• DeLone, W. H., & McLean, E. R. (2016), ‘Information Systems Success Measurement: Foundations and Trends®’ Information Systems, 2(1), 1-116.
• Fan, J. C., & Fang, K. (2006) ‘ERP implementation and information systems success: a test of DeLone and McLean’s model,’ in Technology Management for the Global Future-PICMETConference, Istanbul, 3, 1272-1278.
• Hair Jr, J.F., Anderson, R.E., Tatham, R.L., & Black, W.C. (2005). Análise multivariada de dados (5 ed.), Bookman.
• Hair, Jr., Black, W., Babin, B., and Anderson, R. (2014). Multivariate data analysis (7th Ed.). London: Pearson Education Limited.
• Halonen, R., Acton, T., Golden, W. & Conboy, K. (2009) DeLone & McLean success model as a descriptive tool in evaluating the use of a virtual learning environment, in International Conference on Organizational Learning, Knowlodge and Capabilities, Amsterdam, 1-16.
• Howard, M.C., & Melloy, R.C. (2016) Evaluating Item-Sort Task Methods: The Presentation of a New Statistical Significance Formula and Methodological Best Practices” in Journal of Business and Psychology 31, 173–186. https://doi.org/10.1007/s10869-015-9404-y
• Isaac, S., & Michael, W. B. (1995). Handbook in research and evaluation: A collection of principles, methods, and strategies useful in the planning, design, and evaluation of studies in education and the behavioral sciences (3rd ed.). EdITS Publishers.
• Jaafreh, A. B. (2017) ‘Evaluation information system success: Applied Delone and McLean information system success model in context banking system in KSA,’ International Review of Management and Business Research, 6(2), 829-845.
• Kanaracus, C. (2008) Gartner: Global IT spending growth stable. InfoWorld, April 3, 2008.
• Khayun, V., & Ractham, P. (2011) Measuring e-excise tax success factors: Applying the DeLone & McLean information systems success model. In Hawaii International Conference on System Sciences, Hawai, 1-10.
• Kim, K., Trimi, S., Park, H., & Rhee, S. (2012) ‘The impact of CMS quality on the outcomes of e-learning systems in higher education: an empirical study,’ Decision Sciences Journal of Innovative Education, 10(4), 575-587.
• Lupu, A. R., Bologa, R., Sabau, G., & Muntean, M. (2008).‘Integrated information systems in higher education,’ WSEAS Transactions on Computers, 7(5), 473-482.
• Manchanda, A., & Mukherjee, S. (2014) ‘An empirical application of DeLone and McLean model in evaluating decision support system in the banking sector of Oman’, Journal of International Technology and Information Management, 23(2), 47-58.
• Mtebe, J. S., & Raisamo, R. (2014) ‘A Model for Assessing Learning Management System Success in Higher Education in Sub‐Saharan Countries,’ The Electronic Journal of Information Systems in Developing Countries, 61(1), 1-17.
• Mukerjee, S. (2012). Student information systems–implementation challenges and the road ahead. Journal of Higher Education Policy and Management, 34(1), 51-60.
• Nizamani, S., Khoumbati, K., Ismaili, I. A., Nizamani, S., Nizamani, S., & Basir, N. (2017) ‘Testing and validating the ERP success evaluation model for higher education institutes of Pakistan’, International Journal of Business Information Systems, 25(2), 165-191.
• Nunnally, J. C. & Bernstein, I. H. (1994). Psychometric theory (3rd edition). New York: McGraw-Hill, 1-752.
• Ojo, A. I. (2017) ‘Validation of the DeLone and McLean information systems success model’, Healthcare informatics research, 23(1), 60-66.
• Pilati, R., & Laros, J. A. (2007) ‘Modelos de equações estruturais em psicologia: conceitos e aplicações,’ Psicologia: teoria e pesquisa, 23(2), 205-216.
• Pinsonneault, A., & Kraemer, K. (1993) ‘Research Methodology in Management Information Systems,’ Journal of Management Information Systems – Special Section: Strategic and Competitive Information Systems Archive, 10, 75-105.
• Rai, A., Lang, S. S., & Welker, R. B. (2002) ‘Assessing the validity of IS success models: An empirical test and theoretical analysis,’ Information systems research, 13(1), 50-69.
• Rakemane D., & Serema, B. C. (2018) ‘An evaluation of the effectiveness of student management system (SMS) at Boitekanelo College, Department of health care service management, Botswana’, Lonaka Journal of Learning and Teaching, 9(1), 134-141.
• Ross, A., Willson, V.L. (2017). Paired Samples T-Test. In: Basic and Advanced Statistical Tests. SensePublishers, Rotterdam. https://doi.org/10.1007/978-94-6351-086-8_4
• Semeon, G., Negash, S., & Musa, P. F. (2010). The Success of Student Information Management System: The Case of Higher Education Institution in Ethiopia. In America Conference on Information System, Peru, 278, 1-10.
• Souza, A. C. D., Alexandre, N. M. C., & Guirardello, E. D. B. (2017) ‘Propriedades psicométricas na avaliação de instrumentos: avaliação da confiabilidade e da validade,’ Epidemiologia e Serviços de Saúde, 26 (3), 649-659.
• Stefanovic, D., Marjanovic, U., Delić, M., Culibrk, D., & Lalic, B. (2016) ‘Assessing the success of e-government systems: An employee perspective,’ Information & Management, 53(6), 717-726.
• Tam, C., & Oliveira, T. (2016) ‘Understanding the impact of m-banking on individual performance: DeLone & McLean and TTF perspective,’ Computers in Human Behavior, 61, 233-244.
• Venkatesh, V., & Davis, F. D. (2000) ‘A theoretical extension of the technology acceptance model: Four longitudinal field studies’, Management Science, 46(2), 186-204.
• Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) ‘User acceptance of information technology: Toward a unified view,’ MIS Quarterly, 27(3), 425-478.
• Walsham (2006) ‘Doing interpretive research,’ European Journal of Information Systems, 15(3), 320-330.
• Wang, Y. S., & Liao, Y. W. (2008) ‘Assessing eGovernment systems success: A validation of the DeLone and McLean model of information systems success,’ Government Information Quarterly, 25(4), 717-733.
• Wang, Y. S., Wang, H. Y., & Shee, D. Y. (2007) ‘Measuring e-learning systems success in an organizational context: Scale development and validation,’ Computers in Human Behavior, 23(4), 1792-1808.
• Wei, K. S., Loong, A. C. Y., Leong, Y. M., & Ooi, K. B. (2009) Measuring ERP system success: a respecification of the Delone and McLean’s IS success model. In Symposium on progress in information & communication technology 7-12.
• Weisberg, H., (1991) Central Tendency and Variability (Quantitative Applications in the Social Sciences) (1st Ed.) SAGE Publications, Inc.
• Wu, J. H., & Wang, Y. M. (2006) ‘Measuring KMS success: A respecification of the DeLone and McLean’s model,’ Information & Management, 43(6), 728-739.
• Yakubu, M. N., & Dasuki, S. (2018) ‘Assessing eLearning systems success in Nigeria: An application of the DeLone and McLean information systems success model,’ Journal of Information Technology Education: Research, 17, 183-203.
• Yin, R. K. (2003). Estudo de caso: Planejamento e Métodos (2a ed.),Bookman.
• Yu, P., & Qian, S. (2018) ‘Developing a theoretical model and questionnaire survey instrument to measure the success of electronic health records in residential aged care,’ PloS one, 13(1), 1-18.
• Zimmerman, D. W. (1997). A Note on Interpretation of the Paired-Samples T test. Journal of Educational and Behavioral Statistics, 22(3), 349–360. https://doi.org/10.2307/1165289

Sapf