Technology Acceptance : Course and Teaching Surveys Case Study at Sultan Qaboos University

Course and teaching surveys (CTS) are an integral part of academic life in institutions of higher education. CTS are conducted with the aim of informing the University's commitment to continuous improvement in the delivery of high quality learning experiences for students. They are used also to gauge the students’ satisfaction with the course materials and the instructor(s) and to provide feedback to the instructor(s). Paper based surveys, for historical reasons, are frequently used for this task. With the rolling out of low cost and accessible web based systems, it is now possible to deploy computer-based instead of paper-based surveys. Paper based surveys present a problem due to the extensive amount of paper handling required, and offer greater potential for abuse by certain individuals. Electronic surveys facilitate more efficient development, distribution and reporting processes, but have some issues with motivating student responses. The objective of this research is of two folds; first, to explore the usefulness and effectiveness of using the web-based surveys in contrast to the paper-based ones by obtaining the opinions of the stakeholders (students, faculty members and staff); and secondly to investigate the causes behind the low number of students attempting the web-based surveys in comparison to the paper-based. Interviews with stakeholders) give different perspectives on this matter. Experience with a paper based survey, and the implementation of an electronic replacement at Sultan Qaboos University (SQU), allows the two approaches to be compared. The paper also investigates the acceptability and willingness of the students to use the newly created web-based survey. Many researchers have written about the Technology Acceptance Model (TAM), and some have provided extensions to the initially proposed model. In this paper we present a case study confirming the Perceived Usefulness of Technology (PUT) component of TAM, and how this factor influences a customer’s willingness to use a technology. The study showed clearly that the web-based CTS have much more advantages over the paperbased ones. Furthermore, if a user does not believe a technology is useful he/she will not use it. Many students claimed that they believe that the CTS have no benefits if there are not actions to be taken by the administration. The paper concludes by providing some recommendations to encourage people to use the technology under discussion, an on-line survey service.


Information
and Communication Technology (ICT) has become one of the main tools of organizational success.The rapid deployment of ICT raises concerns among public and private organizations as to how to deal with the technology in order to enhance the agencies' services to the public, and to improve the internal progress of the organization (Atallah, 2001).
ICT allows organisations to perform activities electronically which previously had been performed using manual, paper based systems.Through proper planning and design processes, and using standard IT tools available to us today -tools like network connectivity, user friendly browser interfaces, and powerful database systems -administrative and management processes can be greatly simplified.
This can have multiple benefits, reducing costs, supporting more efficient working processes, enhancing data gathering and analysis, allowing staff to be better deployed for other activities, providing results more quickly, increasing productivity, as well as improving the quality of services and products (Lederer et al., 1998).
While both public and private organizations are spending millions on ICT projects and building new systems, the most difficult job is more basic; to convince organizational employees to use the new technology.It has been noted that users' attitudes towards the acceptance of new information systems has a critical impact on successful information system adoption (Davis, 1989;Venkatesh and Davis, 1996;Succi and Walter, 1999).ICT experts recognized that inadequate acceptance of technology results in wasting resources (Mazhar, 2006).Therefore user acceptance has become an important issue.It is defined as "the demonstrable willingness within a user group to employ information technology for the tasks it is designed to support" (Mazhar, 2006).
Many models are used to measure user acceptance.One of the most popular models is the Technology Acceptance Model (TAM), which was developed by Davis in 1986 and then extended by him and Venkatesh in 2000.Davis wrote that "The goal of TAM is to provide an explanation of the determinants of computer acceptance that is general, capable of explaining user behavior across a broad range of end-user computing technologies and user populations, while at the same time being parsimonious and theoretical."(Davis, 1989), (See figure1).There are other models too, for example, the Theory of Reasoned Action (TRA), upon which TAM is based.

Figure 1: The Technology Acceptance Model (TAM)
In this paper we discuss one application of this technology to simplify processes at the Sultan Qaboos University (SQU).On the other hand we investigate whether, in fact, this new technology was accepted or not and if not why not.In all organisations it is important for stakeholders to know how effective their activities are.To assess this, External Variables Perceived Usefulness Perceived Ease of Use Attitude Toward Using

Intention to Use
Actual use 3 IBIMA Business Review a number of processes and procedures are adopted whereby such educational institutions measure their effectiveness.
For academic institutions probably one of the most commonly used is the teaching survey.
CTS can be deployed in a number of ways, however in this paper, the centred on the processes and procedures used at SQU.The purpose of this research the usefulness and effectiveness of using the web-based surveys in contrast to the paper-based ones by obtaining the opinions of the stakeholders (students, faculty members and staff).Moreover, it aims to investigate the causes behind the low number of students attempting the web-based surveys in comparison to the paper-based.

Research Methods
The process commenced by giving stakeholders sufficient time to get with the new procedures.The team then commenced program.This comprised a series of Figure 2: An E number of processes and procedures are adopted whereby such educational institutions measure their effectiveness.
robably one of the most commonly used is the course and can be deployed in a number of ways, , the discussions the processes and procedures To set the stage, and to stand what is happening, the first sections explain how This is followed by a review of the comments and feedback received from interviews with a number of different stakeholders.The final sections discussions regarding the comments made, as well as recommendations for the deployment of research is to explore the usefulness and effectiveness of using based surveys in contrast to the based ones by obtaining the ions of the stakeholders (students, faculty members and staff).Moreover, it aims to investigate the causes behind the low number of students attempting the based surveys in comparison to the The process commenced by giving sufficient time to get familiar with the new procedures.The interview commenced an interview comprised a series of face to face discussions with various stakeholders involved in different aspects of the Interviewees included: • Office of the Vice-Chancellor Academic Advisor, those involved in survey preparation, analysis and reporting • Print shop staff, who were involved in reproducing the paper questionnaires • Faculty and staff, involve collating, deploying and completing the forms; • Students.
Interviews were conducted topic areas for discussion.Otherwise participants were invited to talk openly about their impressions, experienc feelings regarding the survey.Findings were noted in reports summarising the comments made.These reports are the foundation on which this article has been built.

CTS Process
CTS at SQU are designed to be they are run twice per academic year forms are prepared, these being the "written text form" (WTF), "optically marked form" (OMF) and 3).The WTF is design student to make an openresponse to the questions asked feedback goes directly to the but after two week of grades submission Chancellor (VC) involved in survey , analysis and reporting; , who were involved in questionnaires; involved in sorting, loying and completing the Interviews were conducted based on broad discussion.Otherwise were invited to talk openly about their impressions, experiences and feelings regarding the survey.Findings were noted in reports summarising the comments made.These reports are the foundation on which this article has been designed to be anonymous; twice per academic year.Two forms are prepared, these being the " (WTF), and the form" (OMF) (Figures 2 is designed to allow the -ended free text response to the questions asked.Such feedback goes directly to the instructor(s) after two week of grades submission.

xample of Open Ended (Written Text) Questions
The OMF lays out a set of statements t which the student must check a response

Paper Survey
The Office of the VC Academic Advisor created the master forms.For the OM IBIMA Business Review a set of statements to which the student must check a response box from a serious of response options (likert scale).For example:

xample of Optically Marked Form based Questions
Both forms are developed cooperatively Office of the VC and academic units (Figure 4).As both Arabic and English are used at the SQU, the forms are bilingual.were delivered to the Academic Advisor, which placed the proper number of forms for into envelopes, along with preidentification labels (Figure 5).delivered to colleges, Once the survey packs (envelopes of forms) have been created, they were delivered to the departments, where they were passed to the concerned faculty.Faculty assigned a 15 minutes period at the end of their lectures for form completion. Forms were distributed during these sessions and, on completion, were collected by a designated student representative.
The student representative returned the completed forms to the college (completed forms were not handled by the instructor).The Dean's Office validated the CTS forms, ensuring the correct forms were collated together.All OMFs were returned to the Office of the VC Academic Advisor for processing.WTFs were retained by the colleges.
The Office of the VC Academic Advisor processed the forms, which were read using an Optical Mark Reader (OMR).Statistical reports were generated and passed back to the Deans of the Colleges and the Director of the Language Centre.On receipt of the results the information was communicated to concerned head of departments, then faculty.WTFs were then passed to concerned faculty together with the reports.
OMFs were retained by the Office of the VC Academic Advisor for two months and then disposed.The WTFs were handled by instructors/faculty members as per their own personal needs.

Electronic Survey
Electronic forms were developed being equivalent to the WTFs and OMFs.The Office of the VC Academic Advisor and the academic units worked together cooperatively to develop questions (Figure 6).Technical support and tools were provided by the Centre for Information Systems (CIS).Database of responses is retained for an extended period and can be subject to further analysis if and when required.

Stakeholders Interviews
The researcher(s) interviewed individuals and groups with a specific interest in the processing, handling, and/or completion of the student surveys.The sections below summarize responses made.

Office of the VC Academic Advisor Staff
There were a number of issues that were raised during the interviews with the VC Academic Advisor Staff regarding the paper-based CTS forms, these issues are as follows: • From time to time it was necessary for questions to be amended.Although generating the WTF master presented no problem, producing the OMF master was less easy since it requires the use of OMR software (of which there is only one copy in SQU).The OMR software does not allow Arabic text to be used, so the form was generated in English.To make the bilingual form first OMF master was generated with the special OMR software and printed.The Arabic text was added using Arabic enabled word processor, the OMF master being placed into a printer, which was then overprinted with the Arabic text.
• Before CTS forms completion, the VC Academic Advisor had to coordinate the printing process.The collating and paper handling of forms had particular issues for the VC Academic Advisor and the colleges.
The correct number of forms, and identification labels needed to be sent to colleges.This requires information about the actual courses/section in the colleges, number of students, and other information.
• After form completion the OMFs were received from colleges.It was necessary to quality assure the received forms to enable minor errors to be addressed, including paper orientation (necessary to ensure smooth OMR scanning).After resolving problems with the forms, they were scanned.This scanning was a boring, mind numbing, occupation which takes several weeks.It was very unpopular with staff.
• Analysis of the returned responses required data to be loaded into a custom application and analysis performed (Figure 5).Results were collated and sent to colleges for distribution.

Print Shop Staff
There were a number of issues that were raised during the interviews with the Print shop Staff regarding the paper-based CTS forms, these issues are as follows: • The OMFs required special handling, since it was printed in two colours, and the optical mark timing marks (in black) needed to properly align with the check boxes (in colour).The Print shop generated up to eight potential master plates and made sample print copies for each.Samples of OMFs were validated by the Office of the Academic Advisor by testing them in the OMR.The best sample form was selected.If any changes were required then the master plates had to be re-made and re-approved by the Office of the VC Academic Advisor.
• It was necessary to coordinate printing with the units regular work program, which at some times of the year could be significant.
• Print shop costs were not high, being around 620 Omani Rials (about US $1550), including materials and manual processes.

College Administration Staff
There were a number of issues that were raised during the interviews with the Deans Office Staff regarding the paperbased CTS forms, these issues are as follows: • Sorting/collating forms into envelopes was required; one form was required for each student in each section.The correct number (and type) of form had to be added to each envelope, then an identification label had to be added to each form.The job was tedious, but needed to be performed accurately.
• Because the OMFs had to be completed in pencil, it was necessary to ensure that these were available when the survey was taken.
• On return of the CTS forms by the student representatives, the returned items had to be rechecked, sorted and collated.
Often CTS forms were placed in incorrect envelopes and/or contained significant errors (for example OMF completed in pen rather than the required pencil).Type WTFs were retained, type OMFs were returned to the Office of the VC Academic Advisor.
• The sorting and collation program is boring, mind numbing work, and was very unpopular with staff.It took several weeks.
• There were issues ensuring that forms were returned in the proper manner and were not retained, or otherwise tampered with, by faculty.

Faculty Members
There were a number of issues that were raised during the interviews with the faculty members regarding the paperbased CTS forms, these issues are as follows: • Faculty needed to build sufficient time into their schedule for handling the form, the standard amount of time designated was 15 minutes.
• The issues of selecting a student volunteer to administer the fill-out process and to hand the CTS forms to the Dean's Office.
• The issue of providing fill-out tools such as pencils and erasers to all students i.e. collecting them from Dean's Office and returning them back.

Students
There were a number of issues that were raised during the interviews with the students regarding the paper-based CTS forms, these issues are as follows: • Some students expressed concerns that since WTFs were returned to lecturers for review they might be identified from their handwriting.
• Some students commented that even though forms were completed they could see no tangible changes based on what they commented upon.
Since the surveys were conducted near the end of the semester, during an examination period, students felt pressured to complete the forms in the period allocated.

Analysis of the On-line CTS, Statistics and Comments
The new on-line CTS give the students the freedom and flexibility to fill-out the CTS forms at any time (within the allocated time) and anywhere.
In the main, the electronic CTS received replies from about half of the students who were expected to return a response (Table 1).Notable exceptions were the Colleges of Nursing, Law, and, especially, Medicine, where responses continue to be disappointing (Table 1 The chart below shows that over the last three regular semesters on-line survey sampling intervals, student responsiveness in respect of making replies has only changed slightly, there being a drop of just a few percentage point in the last period (Figure 7).

Figure 7: Trends -Showing How Many College Students Eligible to Complete the Student Quality Survey Actually Did So
Apart from Medicine, Nursing and Law, there were sufficient student responses to get valid feedback on the majority of their courses/sections, meaning the survey was successful (Table 2).
To improve responsiveness, it is important for the SQU to find ways to motivate students to complete the on-line surveys.For colleges that have difficulties owing to the mapping of multiple instructors to actual programs, ways need to be found to make it easier for students to make meaningful responses, but it should be noted that the survey itself is not the root cause of this problem.IBIMA Business Review 10 Figure 8 shows that while there has been some fall in questionnaire responsiveness this is not always the case.Of particular note is the improvement in the number of program surveys being returned by the College of Nursing, as opposed to the continued poor performance of Medicine when compared to the paper survey.Table 3 shows the student responsiveness to the survey in Fall 2008 (when a paper survey was last used) and in the Spring 2009 semester (when an on-line survey was first used).It can be seen that responsiveness was much higher for the paper survey; the reason for the difference is discussed later.

Comparative Analysis between Paper and Electronic Forms
There are number of advantages of using on-line CTS over the paper-based CTS, these advantages are illustrated in Table 4 and 5.The most obvious advantage is the data validation and verification (Al-Badi et al, 2009).However, both survey strategies had advantages and limitations, the tables summarise these points (Table 4, 5).Easy staff coordination, easy amendment, no issues with bilingual use, no printing required.Distribution Issues owing to the large volume of paper to be handled.
Available anywhere on campus through the SQU Portal.

Handling
Issues arising due to sorting and collation.
College provides envelopes.
All processing is done electronically, no handling is required.A high level of completion is achieved since the forms are completed under the management of the tutor.
OM and WT forms are completed online and immediately recorded in an on-line database.
Completion levels are lower than those achieved with the paper process, since forms are completed at a time, and under conditions, of the student's own choosing.

Marking
Forms must be returned to Office of the Academic Advisor for central processing.Time-consuming process requiring significant handling.
OM forms are marked automatically.

Analysis
Analysis must be performed by Office of the Academic Advisor using special applications.
OM forms discarded after two months.
Analysis of OM forms is performed automatically.
New types of analysis can be applied to raw data since it is retained on-line.

Reporting
Reports are distributed to colleges on paper.This requires additional handling.
On-line reports can be accessed by faculty and staff through the SQU Portal.
Information access can be controlled so that only entitled users can see the results.

Data quality
Since forms are completed under circumstances where time is limited, and tutor management is, to some extent apparent, there may be issues relating to the quality of the data.
Analysis of data does reveal a profile that suggests that student "just want to complete the form and move on".Consequently they may not address the questions with the rigour they deserve.
Since forms are completed under conditions of the student's own choosing (viz location and time) one can assume that student responses are the result of a more considered approach than the paper based system.

Discussion
Using the SQU Portal to execute the student survey has demonstrable advantages in most areas eliminating issues relating to preparation, printing and reproduction.Accessibility is improved for all concerned, the Office of the Academic Advisor, faculty, and students.Responses can be gathered easily, analysed quickly, and distributed efficiently.
About the only problem area with the use of electronic survey gathering is the matter of student responsiveness.As can be seen from the data, the number of responses received electronically is significantly lower than would be expected through the paper system.The reasons for this are clear.With the paper survey, while not exactly mandatory students are placed in a situation where it was almost impossible not to complete the form; with the IBIMA Business Review 14 electronic form the position is almost reversed, there is almost no pressure to complete, so many do not.
When given freedom (whether to attempt the survey or not) it was clear that students did not think that responding to the "Course and Teaching Survey" was useful, and this was reflected in their on-line responses.Many stated that "even though forms were completed they could see no tangible changes based on what they commented upon".This is why there were differences in responsiveness between the results for paper and on-line surveys.
With the on-line survey, students discovered, for the first time, that they did not have to complete the survey -they could ignore it -since there was neither reward nor punishment, whatever they chose to do.As this discovery became more widespread, the semester results kept getting lower and lower.Linking this to the findings of many researchers in regard to technology acceptance, we confirmed that it is true that if a user cannot see the benefits of using a technology he/she will not accept it, which means he/she will not use it.The following section provides some recommendations based on the authors, and other practitioners, personal opinions and judgments.

Recommendations
Keeping the above-mentioned discussion clearly in mind, is there any way that one might improve the responsiveness of students?How could they be motivated to actually complete the electronic survey form?Here are some suggestions:

Demonstrate that Completion of the Form Gets Results
Students, in commenting on the survey, observed that whatever they wrote on the form, they could not see any changes made.Although they made this comment, when issues arising were discussed with faculty, actions, where appropriate, were taken.The problem was that these actions were not immediately observable by students.
The first recommendation is, then, to implement initiatives that would show that surveys accomplished things, and had some value.While some actions are, necessarily, of a sensitive nature (for example staff discipline), others might be publicized.To show value, the SQU and individual colleges could: • Publish statistics from the OM forms, showing anonymous data, at least in aggregate.
• Promote initiatives made when these were based upon survey results.

Develop Promotional Schemes
A reasonable subset of students might be encouraged to complete the survey if they felt it was in their own interest.Taking a strictly capitalist view that would mean getting something for the effort.Since it would be impossible to give something of tangible value to every student perhaps one option would be to implement a raffle.The problem here, of course, is that the survey is meant to be anonymous.So how would one tie completion of the survey with participation in such a scheme.Now, the electronic survey is completed using the portal, and to ensure that tutors, courses and sections are properly designated, students have to login to the system.So, when they complete the form they are, in actual fact, easy to identify; it is just that the SQU chooses not to record this data with the actual survey results.Since it is possible to identify who has responded by completing a form, it is possible to create a list of respondents, but not tie those respondents to the actual survey data they have submitted.
So, it is possible to maintain a list of respondents, but to all intents and purposes, assuring the anonymity of the individuals.

Encourage a Socially Responsive Attitude
The purpose of the survey is to detect what the SQU does well, and what the SQU does less well.Part of this involves detecting problem areas and working on ways of addressing these problems.By doing this everyone benefits.The SQU might consider developing a promotional campaign outlining how the individual, in responding to the survey, is through a cooperative efforts with other students, is working to improve the facilities and resources for everyone.Of course, there are political and social dimensions to such a scheme, so this would need to be approached with proper attention to these important matters.

Make Completion of Forms Mandatory
The SQU might consider, through the anonymous logging process mentioned earlier, tying responsiveness to the survey with academic and/or related matters.For example, each student might be required to complete at least one survey per year as a precondition for graduation.Of course, being draconian in this way might cause a negative reaction from the students themselves.
An alternative strategy, one that does not require the identification of individuals, might be to make the release of examination grading information contingent upon the completion of a set minimum number of valid survey questionnaires for individual sections as a group.This would leverage the power of peer pressure within the group to ensure that completion is performed by a larger number of individuals.
Another approach might be to make completion of the survey forms a prerequisite for course registration.Students would not be allowed to perform the registration process without their completion.

Conclusions
Information Technology, properly designed and implemented, can be used to simplify many organisations administrative, management and operational procedures.Selection of areas for deployment should be selected with care to ensure the support of all stakeholders concerned.This will normally be where existing procedures are seen to be beneficial in terms of targets, but slow, tedious, and repetitive in execution.
On-line surveys have significant advantages in terms of preparation, distribution, reporting handling and cost; they require, however, a highly developed Information Technology infrastructure to gain most benefits.
SQU experiences showed that the paperbased survey, although not truly mandatory, was at least quasi-mandatory; the on-line survey was more akin to a voluntary process.
The lower levels of participation in the survey should not, necessarily, be considered to invalidate, or undermine the effectiveness of the survey process.In considering the responses obtained, it is important to remember that data quality is as important as data quantity.The paper based survey gives quantity, the real question is, is the reduced amount of data generated by the on-line survey offset by the improved quality of the data returned?.Indeed, was the quality of the on-line data better or worse?Students need to be motivated to participate in quality surveys.A number of different strategies might be adopted, but it is especially important that the survey process be seen to have some impact upon the services being delivered.Motivational tools can range from encouragements and inducements, to outright compulsion.
Finally, it was clear that moving to the online CTS will reduce the cost and personal effort exerted and hence saving time and money.Therefore, the university should be encouraged to use and keep making all the necessary improvements and innovative ways to encourage students to seriously fill it out with extreme care and patience.

Figure 2 :
Figure 2: An Example of Open Ended (Written Text) Questions

Figure 3 :Figure 4 :
Figure 3: An Example of Optically Marked Form based Questions Both forms are developed cooperatively between the SQU's Office of the Academic Advisor and academic units

Figure 4 :
Figure 4: Schematic Diagram Showing Demographics and Question Requirements ed the forms are deployed, by students, and returned.On are collated by the Office Academic Advisor, statistical these are made concerned faculty and staff.The WT forms are not subjected to any special statistical reporting processes, but are passed back to

Figure 4 :
Figure 4: Schematic Diagram Showing Demographics and Question Requirements was necessary to use special Optical software.The sent to the Print shop About 120,000 forms 60,000 WTFs and 60,000

Figure 5 :
Figure 5: Schematic Diagram Showing the Handling and Processing Flow of the Paperbased CTS Forms In colleges, the Dean's Office sorted/collated the WTFs and OMFs, and checked the quantities delivered.CTS forms were labelled and placed in separate envelopes, for different courses/sections.The additional envelopes had to be provided by the colleges.

Figure 6 :
Figure 6: Schematic Diagram Showing the Handling and Processing Flow of the Electronic CTS Forms Forms are accessible through the SQU Portal.Student login to the Portal for any campus based workstation or on the campus Wi-Fi network.Students complete forms related to their courses/sections online.Students have time-limited windows for form completion; a process which starts at week twelve and continues through week fourteen of each semester.Responses are collected in database tables in realtime, meaning that interim figures can be accessed at any time.Similarly, reports can be generated on a rolling basis.Faculty can access the most recent reports on-line through the Portal at any time.
i c u l t u r e A r t C o m m e r c e E d u c a t i o n E n g i n e e r i n g L a n g u a g e C e n t r e L a w M e d i c i n e N u r s i n g S c i e n c e T o t a l s College Percentage Spring 2010 Fall 2009 Spring 2009 Fall 2008 (Paper)

Figure 8 :
Figure 8: Trends -Showing How Many College Courses/Sections Received Sufficient Quantity of Responses to Be Considered Meaningful Based on SQU Standard Validity Criteria Invalidity criteria: A survey is considered to be invalid if it received five or less responses, or and/or 30% or less of the possible number of responses.

Figure 9
Figure 9 clearly graphically shows the markedly different response profiles seen

Figure 9 :
Figure 9: A Histogram showing the Distribution of Response Rates -Electronic (E) and Paper Forms