The Effect of Language Proficiency on Course Results

Globalization is leading to an increasingly multicultural and multilingual society. The efficiency of the construction industry within this context is increasingly dependent upon effective communication across linguistic and cultural boundaries. Civil Engineering education is no exception: in the globalization of higher education, a large and growing number of university students receive their instruction in a foreign language. While taught courses typically assume language fluency, students' language proficiency often falls short of it and this affects the degree to which they understand course content, achieve intended course outcomes and, ultimately, it impacts their professional performance.


Introduction
Globalization is leading to an increasingly multicultural and multilingual society.The efficiency of the construction industry within this context is increasingly dependent upon effective communication across linguistic and cultural boundaries.Civil Engineering education is no exception: in the globalization of higher education, a large and growing number of university students receive their instruction in a foreign language.While taught courses typically assume language fluency, students' language proficiency often falls short of it and this affects the degree to which they understand course content, achieve intended course outcomes and, ultimately, it impacts their professional performance.

Abstract
The number of people learning and working in foreign language contexts is increasing with globalization and this has implications for the effectiveness of communication and understanding.While it is generally acknowledged that some level of disadvantage is experienced by those with limited language proficiency, the extent of this disadvantage is context-specific and difficult to measure.Historical and current language policies at Tallinn University of Technology provide a unique opportunity for testing and measuring the effects of students' language proficiencies on their course results.In earlier related research, the significance of the language proficiency effect was established for Civil Engineering students.The research reported in this article draws on six years of students' performance data to estimate the magnitude of the effect of language proficiency on the results of a course entitled "Project Management in Construction".The findings indicate a significant performance disadvantage is associated with language proficiency levels at or below the (Common European Framework of Reference for Languages) B2 reference level in comparison with higher levels of language proficiency.__________________________________________________________________________________________________________________ ______________ Emlyn Witt and Irene Lill (2014), Journal of e-Learning and Higher Education, DOI: 10.5171/2014.411199investigate the effect of the language of instruction on students' course outcomes as a consequence of its historical language policies.Where, previously, the two main (Estonian and Russian) language-cultural groups of students were taught the same programmes but separately, each in their own languages, the adoption of Estonian as the principal language of instruction from 2001 has led to a large minority of students (from a Russian language-cultural background) being taught in Estonian.Previous research analyzing the effects of this change revealed significant differences in course outcomes between students taught in their native language and those taught in a foreign language (Lill andWitt, 2008a and2008b).
In addition to the Russian students being taught in Estonian, increasing international exchanges and staff mobility in recent years have seen a rise in foreign faculty members at the university and, consequently, in the number of courses taught in English.In this context, students from both Estonian and Russian language-cultural backgrounds find themselves being taught in a foreign language.As English is not the primary language of instruction, there is no lower limit of English proficiency for entry into these courses so that students' (English language) proficiency levels range from very limited to fluent.This paper reports a causal comparative study of the variation of course results with students' self-assessed levels of language proficiency, and attempts to quantify the extent to which levels of language proficiency affect course outcomes.Data were collected over a period of six years from 2008 -2013 for a project management in construction course taught within the Department of Building Production in the Faculty of Civil Engineering.For this course, which was delivered entirely in English, none of the students in the sample were taught in their native language.

Problem Formulation
Nearly two decades ago, Wells (1996) noted the emergence of an international market for construction labour.Since then, considerable global integration has taken place.Under globalization, the construction industry must embrace an increasingly culturally and linguistically diverse workforce, supply chain and clientele and construction-related higher education, similarly faces increasing diversity among its staff and students.Yet how does this affect its performance?Yet, globalization does not only bring difficulties to be overcome by construction teams and for mitigating recommendations to be made by researchers.Though less widely addressed in the literature, diversity in project teams may also be advantageous.Comu et al (2007) note that, while there tend to be initial performance problems, there are also potential benefits in culturally and linguistically diverse project teams which, with sustained interaction, can lead to enhanced project performance.

Linguistic and cultural diversity in engineering education
Globalization of higher education has seen a surge in the number of international students studying at many of the world's universities, and the relative performance of these students has been a subject of considerable interest to researchers.For example, evidence of the underperformance of international students compared to their local counterparts has been reported by Morrison et al. (2005) and Foster (2012) with the latter also finding that the presence of international students may prove detrimental to the performance of local students.
While proficiency in the language of instruction is obviously a major factor with regard to student performance, it is only one of many variables which have been shown to influence learning outcomes.Other factors noted in the literature include: past performance, admission qualifications, gender, attendance, ethnicity, culture, age, academic aptitude, effort / motivation, previous experience of subject matter, financial considerations and more (Eskew and Faley, 1988;Cuevas, 1984; Alfan and Othman, 2005; Hofman and van den Berg, 2003).

Background to the current research
As indicated in the introductory section above, Tallinn University of Technology offers particular opportunities to investigate the effects of language proficiency on course outcomes.
Earlier research by the authors established significant differences between the performance of different language-cultural groups and the language of instruction for particular courses delivered in the Faculty of Civil Engineering following a change from a dual language to a single teaching language policy (Lill andWitt, 2008a and2008b).The objective of this, follow up study, was to attempt to quantify the magnitude of the language proficiency effect for a particular course.

Research Approach
Having determined that language proficiency was significantly affecting learning outcomes, a data collection effort was organized in relation to a project management in construction course which was taught in English.Students' levels of English language proficiency and their learning outcomes were captured over a period of 6 years from 2008 to 2013 in order to directly investigate the effects of different levels of language proficiency and attempt to assess the magnitude of these effects.The course was taught in each of these 6 years to a new group of primarily fourth year civil engineering students who were studying within an integrated 5 year masters' degree programme.The course content and assessment regime remained largely unchanged during the 6 years covered by the study.

Assessment of language proficiency
At the start of each year's course, all participating students were asked to complete a self-assessment of their English language proficiency using the Common European Framework of Reference (CEFR) for languages table (which was provided to them in their respective native languages).
The CEFR assessment results in a level (A1, A2, B1, B2, C1, C2) being attributed to each of five categories of language use (listening, reading, spoken interaction, spoken production and writing).For ease of data analysis, these self-assessments were converted into a single, numerical equivalent as follows: for each of the five categories, the assessed level was converted into a number (A1=1; A2=2; B1=3; B2=4; C1=5; C2=6) and then the average of all five category assessments was calculated.This resulted in single, numerical values for language proficiency ranging from 1 (lowest, basic level of proficiency) to 6 (complete fluency).__________________________________________________________________________________________________________________ ______________ Emlyn Witt and Irene Lill (2014), Journal of e-Learning and Higher Education, DOI: 10.5171/2014.411199

Assessment of learning outcomes
The primary form of assessment for the course was a written exam together with a coursework assessment.Learning outcomes for the purposes of this research were taken as the first exam result achieved by each student since subsequent (repeat) exam results would reflect the benefit of previous exam experience and would therefore tend to be better than initial (first try) results.As coursework marks are subject to other factors (such as group effects, etc.), these were excluded from the analysis.

Other variables considered
In attempting to control for other factors which might have a significant influence on learning outcomes, two further variables were considered -students' overall grade averages and students' lecture attendance rates.Overall grade averages, determined on the basis of all courses taken by the students, were considered to be a suitable proxy measure for overall student performance / ability in all courses.However, these overall grade averages in most cases included the grades achieved on the project management in construction course, but, since this typically accounted for such a small proportion (approximately 2%) of the overall grade average, no adjustment to the overall grade averages was considered necessary.
Lecture attendance rates were considered a proxy measure for student motivation / effort with regard to this particular course.Language proficiency self-assessments, together with exam results and lecture attendances were compiled for a total of 393 students.
Overall grade averages, however, were generally only available for students who had successfully completed all their taught courses at the time of data compilation.For this reason, overall grade averages were collected for only 322 students.

Analysis of Results
Analysis of the data was carried out using the SPSS software package.

Correlation testing
Figures 1-3 (below) show scatter plots of students' language proficiencies, lecture attendances and overall grade averages; respectively, versus their exam results.The scatter plots suggest weak, positive correlations between language proficiencies and exam results and also between lecture attendances and exam results.The scatter plot in Figure 3 indicates a moderate, positive correlation between students' overall grade averages and exam results.

Figure2: Scatter plot of Lecture Attendance versus Exam Result Figure3: Scatter plot of Overall Grade Average versus Exam Result
These correlations were confirmed through the calculation of Pearson correlation coefficients in each case and all were found to be statistically significant at the 0,05 level (one-tailed testing) as shown in Table 1 below.

Determination of a language proficiency 'threshold'
To further analyze the effect of language proficiency on exam results, students were categorized into groups according to their language proficiency levels.Four different categorizations were tested: 2 language proficiency levels (low, high), 3 levels (low, medium, high); 4 levels (low, low-medium, high-medium, high); 6 levels (very low, low, lower medium, higher medium, high, very high).In each case, the range of language proficiency values falling into each category was chosen, so that, the number of students in each category was approximately the same.The mean exam results associated with each language proficiency level were compared using one-way ANOVA, and statistically significant differences between the categories were found in all cases (at the 0,05 significance level).Tables 2-5 (below) summarize the data input and output for the one-way ANOVA testing and Figures 4-7 (below) provide graphical illustration of the observed differences in mean exam results associated with each language proficiency category.
Journal of e-Learning and Higher Education __________________________________________________________________________________________________________________  The language proficiency level of this threshold can be seen from Figures 4-7 to correspond approximately with the categorizations: 'high' (>= 4) for 2 levels; 'high' (> 4) for 3 levels; 'high' (> 4,2) for 4 levels; and 'high' (>= 4,2) for 6 levels of language proficiency.The accuracy of the estimated level increases with the increased refinement (number of levels) of the categorization suggesting that the threshold is slightly above the B2 language proficiency level.However, it must be noted that these results refer to all the categories of language proficiency and are not limited to those above and below the threshold value.Thus they tend to underestimate the effect of being below or above the threshold language proficiency level.Additionally, for the 2-level categorization of language proficiency, the 'high' (>= 4) category overlaps the threshold value which has already been established as being slightly above B2 (4).In consideration of this, the 2-level categorization of language proficiency was amended so as to reflect a 'low' level corresponding to language proficiency below the threshold level and a 'high' level corresponding to language proficiency above the threshold level.The partial eta squared values including those for this corrected 2-level categorization are shown in Table 7.Thus, the proportion of variance in exam results attributable to language proficiency is shown to lie between 3,2% (for the 4-level categorization) and 6,4% (for the 6-level categorization).Arguably, the most accurate estimate of the effect is provided by the partial eta squared value for the corrected 2level language proficiency categorization, i.e. 5,1%.In all cases, the effect of language proficiency remains significant (at the 0,05 significance level) when lecture attendance and overall grade average are controlled for.

Figure5: Graph of Mean Exam Results for 3 levels of Language Proficiency
While lecture attendance explains a smaller proportion of the variance (approximately 2,4%) in the exam result, the overall grade average variable accounts for a far greater proportion of the variance in the dependent variable (approximately 19%) than either language proficiency or lecture attendance.

Conclusion
There is evidence of a language-based disadvantage associated with having language proficiency at or below the B2 level.This suggests that the B2 level would provide an appropriate minimum proficiency level for students enrolling in a course taught in a foreign language.However, it does not suggest that a lower level of language proficiency cannot be overcome through student effort.
The magnitude of this disadvantage, in the case of this particular course, has been shown to be approximately 5% but this should be considered in relation to attempts to mitigate the effect of language proficiency differences.With greater efforts at mitigation it probably could be reduced and, if none were in place, then it seems reasonable to assume that the magnitude of the disadvantage would be greater.
Beyond the classroom, this has considerable implications for communication and understanding in all multilingual contexts.With such contexts increasingly arising through globalization, it suggests that significant (and, therefore, measurable) performance deficits and inefficiencies are emerging both in education and in industry.While this is unlikely to amount to an argument 'against' globalization, it certainly suggests that greater efforts should be made to mitigate language proficiency effects.

Figure 6 :
Figure 6: Graph of Mean Exam Results for 4 levels of Language Proficiency