Introduction
Information and Communication Technology (ICT) has become one of the main tools of organizational success. The rapid deployment of ICT raises concerns among public and private organizations as to how to deal with the technology in order to enhance the agencies’ services to the public, and to improve the internal progress of the organization (Atallah, 2001).
ICT allows organisations to perform activities electronically which previously had been performed using manual, paper based systems. Through proper planning and design processes, and using standard IT tools available to us today – tools like network connectivity, user friendly browser interfaces, and powerful database systems – administrative and management processes can be greatly simplified.
This can have multiple benefits, reducing costs, supporting more efficient working processes, enhancing data gathering and analysis, allowing staff to be better deployed for other activities, providing results more quickly, increasing productivity, as well as improving the quality of services and products (Lederer et al., 1998).
While both public and private organizations are spending millions on ICT projects and building new systems, the most difficult job is more basic; to convince organizational employees to use the new technology. It has been noted that users’ attitudes towards the acceptance of new information systems has a critical impact on successful information system adoption (Davis, 1989; Venkatesh and Davis, 1996; Succi and Walter, 1999). ICT experts recognized that inadequate acceptance of technology results in wasting resources (Mazhar, 2006). Therefore user acceptance has become an important issue. It is defined as “the demonstrable willingness within a user group to employ information technology for the tasks it is designed to support” (Mazhar, 2006).
Many models are used to measure user acceptance. One of the most popular models is the Technology Acceptance Model (TAM), which was developed by Davis in 1986 and then extended by him and Venkatesh in 2000. Davis wrote that “The goal of TAM is to provide an explanation of the determinants of computer acceptance that is general, capable of explaining user behavior across a broad range of end-user computing technologies and user populations, while at the same time being parsimonious and theoretical.”(Davis, 1989), (See figure1). There are other models too, for example, the Theory of Reasoned Action (TRA), upon which TAM is based.
Figure 1: The Technology Acceptance Model (TAM)
In this paper we discuss one application of this technology to simplify processes at the Sultan Qaboos University (SQU). On the other hand we investigate whether, in fact, this new technology was accepted or not and if not why not. In all organisations it is important for stakeholders to know how effective their activities are. To assess this, a number of processes and procedures are adopted whereby such educational institutions measure their effectiveness. For academic institutions probably one of the most commonly used is the course and teaching survey.
CTS can be deployed in a number of ways, however in this paper, the discussions centred on the processes and procedures used at SQU. To set the stage, and to permit readers to understand what is happening, the first sections explain how surveys are performed. This is followed by a review of the comments and feedback received from interviews with a number of different stakeholders. The final sections include discussions regarding the comments made, as well as recommendations for the deployment of more effective surveys.
The purpose of this research is to explore the usefulness and effectiveness of using the web-based surveys in contrast to the paper-based ones by obtaining the opinions of the stakeholders (students, faculty members and staff). Moreover, it aims to investigate the causes behind the low number of students attempting the web-based surveys in comparison to the paper-based.
Research Methods
The process commenced by giving stakeholders sufficient time to get familiar with the new procedures. The interview team then commenced an interview program. This comprised a series of face to face discussions with various stakeholders involved in different aspects of the survey. Interviewees included:
- Office of the Vice-Chancellor (VC) Academic Advisor, those involved in survey preparation, analysis and reporting;
- Print shop staff, who were involved in reproducing the paper questionnaires;
- Faculty and staff, involved in sorting, collating, deploying and completing the forms;
Interviews were conducted based on broad topic areas for discussion.Otherwise participants were invited to talk openly about their impressions, experiences and feelings regarding the survey. Findings were noted in reports summarising the comments made. These reports are the foundation on which this article has been built.
CTS Process
CTS at SQU are designed to be anonymous; they are run twice per academic year. Two forms are prepared, these being the “written text form” (WTF), and the “optically marked form” (OMF) (Figures 2and 3).The WTF is designed to allow the student to make an open-ended free text response to the questions asked. Such feedback goes directly to the instructor(s) but after two week of grades submission.
Figure 2: An Example of Open Ended (Written Text) Questions
The OMF lays out a set of statements to which the student must check a response box from a serious of response options (likert scale). For example:
Figure 3: An Example of Optically Marked Form based Questions
Both forms are developed cooperatively between the SQU’s Office of the VC Academic Advisor and academic units (Figure 4). As both Arabic and English are used at the SQU, the forms are bilingual.
Figure 4: Schematic Diagram Showing Demographics and Question Requirements
Once finalized the forms are deployed, completed by students, and returned. On return the OMFs are collated by the Office of the VC Academic Advisor, statistical reports are generated, and these are made available to the concerned faculty and Administration staff. The WT forms are not subjected to any special statistical reporting processes, but are passed back to the faculty concerned.
Paper Survey
The Office of the VC Academic Advisor created the master forms. For the OMF it was necessary to use special Optical Marked Reader (OMR) software. The master forms were sent to the Print shop for reproduction. About 120,000 forms were printed 60,000 WTFs and 60,000 OMFs.
The WTFs and OMFs were delivered to the Office of the VC Academic Advisor, which placed the proper number of forms for each college into envelopes, along with pre-printed identification labels (Figure 5). Envelopes were then delivered to colleges, which takes several days.
Figure 5: Schematic Diagram Showing the Handling and Processing Flow of the Paper-based CTS Forms
In colleges, the Dean’s Office sorted/collated the WTFs and OMFs, and checked the quantities delivered. CTS forms were labelled and placed in separate envelopes, for different courses/sections.The additional envelopes had to be provided by the colleges.
Once the survey packs (envelopes of forms) have been created, they were delivered to the departments, where they were passed to the concerned faculty. Faculty assigned a 15 minutes period at the end of their lectures for form completion. Forms were distributed during these sessions and, on completion, were collected by a designated student representative. The student representative returned the completed forms to the college (completed forms were not handled by the instructor).
The Dean’s Office validated the CTS forms, ensuring the correct forms were collated together. All OMFs were returned to the Office of the VC Academic Advisor for processing. WTFs were retained by the colleges.
The Office of the VC Academic Advisor processed the forms, which were read using an Optical Mark Reader (OMR). Statistical reports were generated and passed back to the Deans of the Colleges and the Director of the Language Centre. On receipt of the results the information was communicated to concerned head of departments, then faculty. WTFs were then passed to concerned faculty together with the reports.
OMFs were retained by the Office of the VC Academic Advisor for two months and then disposed.The WTFs were handled by instructors/faculty members as per their own personal needs.
Electronic Survey
Electronic forms were developed being equivalent to the WTFs and OMFs. The Office of the VC Academic Advisor and the academic units worked together cooperatively to develop questions (Figure 6).Technical support and tools were provided by the Centre for Information Systems (CIS).
Figure 6: Schematic Diagram Showing the Handling and Processing Flow of the Electronic CTS Forms
Forms are accessible through the SQU Portal. Student login to the Portal for any campus based workstation or on the campus Wi-Fi network. Students complete forms related to their courses/sections on-line. Students have time-limited windows for form completion; a process which starts at week twelve and continues through week fourteen of each semester. Responses are collected in database tables in real-time, meaning that interim figures can be accessed at any time. Similarly, reports can be generated on a rolling basis. Faculty can access the most recent reports on-line through the Portal at any time.
Database of responses is retained for an extended period and can be subject to further analysis if and when required.
Results
Stakeholders Interviews
The researcher(s) interviewed individuals and groups with a specific interest in the processing, handling, and/or completion of the student surveys. The sections below summarize responses made.
Office of the VC Academic Advisor Staff
There were a number of issues that were raised during the interviews with the VC Academic Advisor Staff regarding the paper-based CTS forms, these issues are as follows:
- From time to time it was necessary for questions to be amended. Although generating the WTF master presented no problem, producing the OMF master was less easy since it requires the use of OMR software (of which there is only one copy in SQU). The OMR software does not allow Arabic text to be used, so the form was generated in English. To make the bilingual form first OMF master was generated with the special OMR software and printed. The Arabic text was added using Arabic enabled word processor, the OMF master being placed into a printer, which was then overprinted with the Arabic text.
- Before CTS forms completion, the VC Academic Advisor had to coordinate the printing process. The collating and paper handling of forms had particular issues for the VC Academic Advisor and the colleges. The correct number of forms, and identification labels needed to be sent to colleges.This requires information about the actual courses/section in the colleges, number of students, and other information.
- After form completion the OMFs were received from colleges. It was necessary to quality assure the received forms to enable minor errors to be addressed, including paper orientation (necessary to ensure smooth OMR scanning). After resolving problems with the forms, they were scanned. This scanning was a boring, mind numbing, occupation which takes several weeks. It was very unpopular with staff.
- Analysis of the returned responses required data to be loaded into a custom application and analysis performed (Figure 5). Results were collated and sent to colleges for distribution.
Print Shop Staff
There were a number of issues that were raised during the interviews with the Print shop Staff regarding the paper-based CTS forms, these issues are as follows:
- The OMFs required special handling, since it was printed in two colours, and the optical mark timing marks (in black) needed to properly align with the check boxes (in colour). The Print shop generated up to eight potential master plates and made sample print copies for each. Samples of OMFs were validated by the Office of the Academic Advisor by testing them in the OMR. The best sample form was selected. If any changes were required then the master plates had to be re-made and re-approved by the Office of the VC Academic Advisor.
- It was necessary to coordinate printing with the units regular work program, which at some times of the year could be significant.
- Print shop costs were not high, being around 620 Omani Rials (about US $1550), including materials and manual processes.
College Administration Staff
There were a number of issues that were raised during the interviews with the Deans Office Staff regarding the paper-based CTS forms, these issues are as follows:
- Sorting/collating forms into envelopes was required; one form was required for each student in each section. The correct number (and type) of form had to be added to each envelope, then an identification label had to be added to each form. The job was tedious, but needed to be performed accurately.
- Because the OMFs had to be completed in pencil, it was necessary to ensure that these were available when the survey was taken.
- On return of the CTS forms by the student representatives, the returned items had to be rechecked, sorted and collated. Often CTS forms were placed in incorrect envelopes and/or contained significant errors (for example OMF completed in pen rather than the required pencil). Type WTFs were retained, type OMFs were returned to the Office of the VC Academic Advisor.
- The sorting and collation program is boring, mind numbing work, and was very unpopular with staff. It took several weeks.
- There were issues ensuring that forms were returned in the proper manner and were not retained, or otherwise tampered with, by faculty.
Faculty Members
There were a number of issues that were raised during the interviews with the faculty members regarding the paper-based CTS forms, these issues are as follows:
- Faculty needed to build sufficient time into their schedule for handling the form, the standard amount of time designated was 15 minutes.
- The issues of selecting a student volunteer to administer the fill-out process and to hand the CTS forms to the Dean’s Office.
- The issue of providing fill-out tools such as pencils and erasers to all students i.e. collecting them from Dean’s Office and returning them back.
Students
There were a number of issues that were raised during the interviews with the students regarding the paper-based CTS forms, these issues are as follows:
- Some students expressed concerns that since WTFs were returned to lecturers for review they might be identified from their handwriting.
- Some students commented that even though forms were completed they could see no tangible changes based on what they commented upon.
- Since the surveys were conducted near the end of the semester, during an examination period, students felt pressured to complete the forms in the period allocated.
Analysis of the On-line CTS, Statistics and Comments
The new on-line CTS give the students the freedom and flexibility to fill-out the CTS forms at any time (within the allocated time) and anywhere.
In the main, the electronic CTS received replies from about half of the students who were expected to return a response (Table 1). Notable exceptions were the Colleges of Nursing, Law, and, especially, Medicine, where responses continue to be disappointing (Table 1). Follow up with the College of Medicine suggests that the organisation of the programs offered is the root cause of the problem. It was noted that each program can be tutored by up to eight instructors, and students feel that completion of the form (and assessment of program and instructor), is difficult to perform in any meaningful way. Other than this, college feedback indicates a low level of motivation amongst students to complete the form.
Table 1: Breakdown Showing How Many College Students Eligible to Complete the Student Quality Survey Actually Did So
The chart below shows that over the last three regular semesters on-line survey sampling intervals, student responsiveness in respect of making replies has only changed slightly, there being a drop of just a few percentage point in the last period (Figure 7).
Figure 7: Trends – Showing How Many College Students Eligible to Complete the Student Quality Survey Actually Did So
Apart from Medicine, Nursing and Law, there were sufficient student responses to get valid feedback on the majority of their courses/sections, meaning the survey was successful (Table 2). To improve responsiveness, it is important for the SQU to find ways to motivate students to complete the on-line surveys. For colleges that have difficulties owing to the mapping of multiple instructors to actual programs, ways need to be found to make it easier for students to make meaningful responses, but it should be noted that the survey itself is not the root cause of this problem.
Table 2: Breakdown Showing How Many College Courses/Sections Received Sufficient Quantity of Responses to be Considered Meaningful Based on SQU Standard Validity Criteria
Figure 8 shows that while there has been some fall in questionnaire responsiveness this is not always the case.Of particular note is the improvement in the number of program surveys being returned by the College of Nursing, as opposed to the continued poor performance of Medicine when compared to the paper survey.
Figure 8: Trends – Showing How Many College Courses/Sections Received Sufficient Quantity of Responses to Be Considered Meaningful Based on SQU Standard Validity Criteria
Invalidity criteria: A survey is considered to be invalid if it received five or less responses, or and/or 30% or less of the possible number of responses.
Table 3 shows the student responsiveness to the survey in Fall 2008 (when a paper survey was last used) and in the Spring 2009 semester (when an on-line survey was first used).It can be seen that responsiveness was much higher for the paper survey; the reason for the difference is discussed later.
Table 3: Distribution of Response Rates
Figure 9 clearly graphically shows the markedly different response profiles seen when the surveys are conducted using paper forms and electronic systems.
Figure 9: A Histogram Showing the Distribution of Response Rates – Electronic (E) and Paper Forms
Comparative Analysis between Paper and Electronic Forms
There are number of advantages of using on-line CTS over the paper-based CTS, these advantages are illustrated in Table 4 and 5. The most obvious advantage is the data validation and verification (Al-Badi et al, 2009). However, both survey strategies had advantages and limitations, the tables summarise these points (Table 4, 5).
Table 4: Comparison of Development and Handling Processes
Table 5: Comparison of Completion, Analysis and Reporting Processes
Discussion
Using the SQU Portal to execute the student survey has demonstrable advantages in most areas eliminating issues relating to preparation, printing and reproduction. Accessibility is improved for all concerned, the Office of the Academic Advisor, faculty, and students. Responses can be gathered easily, analysed quickly, and distributed efficiently.
About the only problem area with the use of electronic survey gathering is the matter of student responsiveness. As can be seen from the data, the number of responses received electronically is significantly lower than would be expected through the paper system. The reasons for this are clear. With the paper survey, while not exactly mandatory students are placed in a situation where it was almost impossible not to complete the form; with the electronic form the position is almost reversed, there is almost no pressure to complete, so many do not.
When given freedom (whether to attempt the survey or not) it was clear that students did not think that responding to the “Course and Teaching Survey” was useful, and this was reflected in their on-line responses. Many stated that “even though forms were completed they could see no tangible changes based on what they commented upon”. This is why there were differences in responsiveness between the results for paper and on-line surveys.
With the on-line survey, students discovered, for the first time, that they did not have to complete the survey – they could ignore it – since there was neither reward nor punishment, whatever they chose to do. As this discovery became more widespread, the semester results kept getting lower and lower. Linking this to the findings of many researchers in regard to technology acceptance, we confirmed that it is true that if a user cannot see the benefits of using a technology he/she will not accept it, which means he/she will not use it. The following section provides some recommendations based on the authors, and other practitioners, personal opinions and judgments.
Recommendations
Keeping the above-mentioned discussion clearly in mind, is there any way that one might improve the responsiveness of students? How could they be motivated to actually complete the electronic survey form? Here are some suggestions:
Demonstrate that Completion of the Form Gets Results
Students, in commenting on the survey, observed that whatever they wrote on the form, they could not see any changes made. Although they made this comment, when issues arising were discussed with faculty, actions, where appropriate, were taken. The problem was that these actions were not immediately observable by students. The first recommendation is, then, to implement initiatives that would show that surveys accomplished things, and had some value. While some actions are, necessarily, of a sensitive nature (for example staff discipline), others might be publicized. To show value, the SQU and individual colleges could:
- Publish statistics from the OM forms, showing anonymous data, at least in aggregate.
- Promote initiatives made when these were based upon survey results.
Develop Promotional Schemes
A reasonable subset of students might be encouraged to complete the survey if they felt it was in their own interest. Taking a strictly capitalist view that would mean getting something for the effort. Since it would be impossible to give something of tangible value to every student perhaps one option would be to implement a raffle. The problem here, of course, is that the survey is meant to be anonymous. So how would one tie completion of the survey with participation in such a scheme.
Now, the electronic survey is completed using the portal, and to ensure that tutors, courses and sections are properly designated, students have to login to the system. So, when they complete the form they are, in actual fact, easy to identify; it is just that the SQU chooses not to record this data with the actual survey results. Since it is possible to identify who has responded by completing a form, it is possible to create a list of respondents, but not tie those respondents to the actual survey data they have submitted.
So, it is possible to maintain a list of respondents, but to all intents and purposes, assuring the anonymity of the individuals.
Encourage a Socially Responsive Attitude
The purpose of the survey is to detect what the SQU does well, and what the SQU does less well. Part of this involves detecting problem areas and working on ways of addressing these problems. By doing this everyone benefits. The SQU might consider developing a promotional campaign outlining how the individual, in responding to the survey, is through a cooperative efforts with other students, is working to improve the facilities and resources for everyone. Of course, there are political and social dimensions to such a scheme, so this would need to be approached with proper attention to these important matters.
Make Completion of Forms Mandatory
The SQU might consider, through the anonymous logging process mentioned earlier, tying responsiveness to the survey with academic and/or related matters. For example, each student might be required to complete at least one survey per year as a precondition for graduation. Of course, being draconian in this way might cause a negative reaction from the students themselves.
An alternative strategy, one that does not require the identification of individuals, might be to make the release of examination grading information contingent upon the completion of a set minimum number of valid survey questionnaires for individual sections as a group. This would leverage the power of peer pressure within the group to ensure that completion is performed by a larger number of individuals.
Another approach might be to make completion of the survey forms a prerequisite for course registration. Students would not be allowed to perform the registration process without their completion.
Conclusion
Information Technology, properly designed and implemented, can be used to simplify many organisations administrative, management and operational procedures. Selection of areas for deployment should be selected with care to ensure the support of all stakeholders concerned. This will normally be where existing procedures are seen to be beneficial in terms of targets, but slow, tedious, and repetitive in execution.
On-line surveys have significant advantages in terms of preparation, distribution, reporting handling and cost; they require, however, a highly developed Information Technology infrastructure to gain most benefits.
SQU experiences showed that the paper-based survey, although not truly mandatory, was at least quasi-mandatory; the on-line survey was more akin to a voluntary process.
The lower levels of participation in the survey should not, necessarily, be considered to invalidate, or undermine the effectiveness of the survey process. In considering the responses obtained, it is important to remember that data quality is as important as data quantity. The paper based survey gives quantity, the real question is, is the reduced amount of data generated by the on-line survey offset by the improved quality of the data returned?. Indeed, was the quality of the on-line data better or worse?
Students need to be motivated to participate in quality surveys. A number of different strategies might be adopted, but it is especially important that the survey process be seen to have some impact upon the services being delivered. Motivational tools can range from encouragements and inducements, to outright compulsion.
Finally, it was clear that moving to the on-line CTS will reduce the cost and personal effort exerted and hence saving time and money. Therefore, the university should be encouraged to use and keep making all the necessary improvements and innovative ways to encourage students to seriously fill it out with extreme care and patience.
Acknowledgements
The authors would like to express their appreciation and gratitude to Stephen Millmore, from CIS, SQU, for his valuable comments, suggestions during the writing of this paper. Also authors would like to thank Dr. Cengiz Ercil from the Office of the VC Academic Advisor, SQU, for his effort in obtaining the CTS data and for his valuable and continuous comments and support.
(adsbygoogle = window.adsbygoogle || []).push({});
References
Ali H. Al-Badi, Ali O. Al Majeeni, Pam J. Mayhew & Abdullah S. Al-Rashdi (2009). ‘Building a Portal to Achieve Cross-Organisational Information Sharing and Dissemination,’ Proceedings of the 12th International Business Information Management Association (IBIMA). Kuala Lumpur, Malaysia, 29- 30 June 2009
Atallah, S. (2001). ‘E-Government: Considerations for Arab States,’ United Nations Development Program: USA.
Google Scholar
Davis, F. D. (1989). “Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology,” MIS Quarterly, 13 (3). 319-340.
Publisher – Google Scholar
Lederer, A. L., Maupin, D. J., Sena, M. P. & Zhuang, Y. (1998). “The Role of Ease of Use, Usefulness and Attitude in the Prediction of World Wide Web Usage,” Proceedings of the 1998 Association for Computing Machinery Special, ISBN: 0-89791-959-9, March 26 – 28, 1998, Boston, MA, USA, 195-204.
Publisher – Google Scholar
Mazhar, N. (2006). “Technology Acceptance Model,” Ezine Articles, [online], [Retrieved 3 June 2011], available at: [http://ezinearticles.com/?Technology-Acceptance-Model&id=202354].
Publisher
Succi, M. J. & Walter, Z. D. (1999). “Theory of User Acceptance of Information Technologies: An Examination of Health Care Professionals,” Proceedings of the 32nd Hawaii International Conference on System Sciences (HICSS). 1-7.
Publisher – Google Scholar – British Library Direct
Venkatesh, V. & Davis, F. D. (1996). “A Model of the Antecedents of Perceived Ease Of Use: Development and Test*,” Decision Sciences, 27 (3). 451-81.
Publisher – Google Scholar – British Library Direct
Venkatesh, V. & Davis, F. D. (2000). “Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies,” Management Science, 46(2). 186-204.
Publisher – Google Scholar – British Library Direct
