The Effect of Language Proficiency on Course Results

Introduction

Globalization is leading to an increasingly multicultural and multilingual society. The efficiency of the construction industry within this context is increasingly dependent upon effective communication across linguistic and cultural boundaries. Civil Engineering education is no exception: in the globalization of higher education, a large and growing number of university students receive their instruction in a foreign language. While taught courses typically assume language fluency, students’ language proficiency often falls short of it and this affects the degree to which they understand course content, achieve intended course outcomes and, ultimately, it impacts their professional performance.

Tallinn University of Technology in Estonia provides an exceptional opportunity to investigate the effect of the language of instruction on students’ course outcomes as a consequence of its historical language policies. Where, previously, the two main (Estonian and Russian) language-cultural groups of students were taught the same programmes but separately, each in their own languages, the adoption of Estonian as the principal language of instruction from 2001 has led to a large minority of students (from a Russian language-cultural background) being taught in Estonian. Previous research analyzing the effects of this change revealed significant differences in course outcomes between students taught in their native language and those taught in a foreign language (Lill and Witt, 2008a and 2008b).

In addition to the Russian students being taught in Estonian, increasing international exchanges and staff mobility in recent years have seen a rise in foreign faculty members at the university and, consequently, in the number of courses taught in English. In this context, students from both Estonian and Russian language-cultural backgrounds find themselves being taught in a foreign language. As English is not the primary language of instruction, there is no lower limit of English proficiency for entry into these courses so that students’ (English language) proficiency levels range from very limited to fluent.

This paper reports a causal comparative study of the variation of course results with students’ self-assessed levels of language proficiency, and attempts to quantify the extent to which levels of language proficiency affect course outcomes. Data were collected over a period of six years from 2008 – 2013 for a project management in construction course taught within the Department of Building Production in the Faculty of Civil Engineering. For this course, which was delivered entirely in English, none of the students in the sample were taught in their native language.

Problem Formulation

Nearly two decades ago, Wells (1996) noted the emergence of an international market for construction labour. Since then, considerable global integration has taken place. Under globalization, the construction industry must embrace an increasingly culturally and linguistically diverse workforce, supply chain and clientele and construction-related higher education, similarly faces increasing diversity among its staff and students. Yet how does this affect its performance?

Linguistic and cultural diversity in the construction industry

Numerous studies have investigated construction industry issues and responses in terms of cross-cultural communication and conflict resolution. For example, Ayoko (2007), Fontaine (2007), Jaselskis et al. (2008), Loosemore & Lee (2002), Rosso et al. 2007) and Ling et al. (2007).

Similarly, explorations of construction-related country comparisons and cultural differences in multicultural teams (including those of Fields et al. (2006); Fong & Lung (2007); and Ozorovskaja et al. (2007) are well-represented in the literature.

Yet, globalization does not only bring difficulties to be overcome by construction teams and for mitigating recommendations to be made by researchers. Though less widely addressed in the literature, diversity in project teams may also be advantageous. Comu et al (2007) note that, while there tend to be initial performance problems, there are also potential benefits in culturally and linguistically diverse project teams which, with sustained interaction, can lead to enhanced project performance.

Linguistic and cultural diversity in engineering education

Globalization of higher education has seen a surge in the number of international students studying at many of the world’s universities, and the relative performance of these students has been a subject of considerable interest to researchers. For example, evidence of the underperformance of international students compared to their local counterparts has been reported by Morrison et al. (2005) and Foster (2012) with the latter also finding that the presence of international students may prove detrimental to the performance of local students.

While proficiency in the language of instruction is obviously a major factor with regard to student performance, it is only one of many variables which have been shown to influence learning outcomes. Other factors noted in the literature include: past performance, admission qualifications, gender, attendance, ethnicity, culture, age, academic aptitude, effort / motivation, previous experience of subject matter, financial considerations and more (Eskew and Faley, 1988; Cuevas, 1984; Alfan and Othman, 2005; Hofman and van den Berg, 2003).

Background to the current research

As indicated in the introductory section above, Tallinn University of Technology offers particular opportunities to investigate the effects of language proficiency on course outcomes.

Earlier research by the authors established significant differences between the performance of different language-cultural groups and the language of instruction for particular courses delivered in the Faculty of Civil Engineering following a change from a dual language to a single teaching language policy (Lill and Witt, 2008a and 2008b). The objective of this, follow up study, was to attempt to quantify the magnitude of the language proficiency effect for a particular course.

Research Approach

Having determined that language proficiency was significantly affecting learning outcomes, a data collection effort was organized in relation to a project management in construction course which was taught in English. Students’ levels of English language proficiency and their learning outcomes were captured over a period of 6 years from 2008 to 2013 in order to directly investigate the effects of different levels of language proficiency and attempt to assess the magnitude of these effects. The course was taught in each of these 6 years to a new group of primarily fourth year civil engineering students who were studying within an integrated 5 year masters’ degree programme. The course content and assessment regime remained largely unchanged during the 6 years covered by the study.

Assessment of language proficiency

At the start of each year’s course, all participating students were asked to complete a self-assessment of their English language proficiency using the Common European Framework of Reference (CEFR) for languages table (which was provided to them in their respective native languages).

The CEFR assessment results in a level (A1, A2, B1, B2, C1, C2) being attributed to each of five categories of language use (listening, reading, spoken interaction, spoken production and writing). For ease of data analysis, these self-assessments were converted into a single, numerical equivalent as follows: for each of the five categories, the assessed level was converted into a number (A1=1; A2=2; B1=3; B2=4; C1=5; C2=6) and then the average of all five category assessments was calculated. This resulted in single, numerical values for language proficiency ranging from 1 (lowest, basic level of proficiency) to 6 (complete fluency).

Assessment of learning outcomes

The primary form of assessment for the course was a written exam together with a coursework assessment. Learning outcomes for the purposes of this research were taken as the first exam result achieved by each student since subsequent (repeat) exam results would reflect the benefit of previous exam experience and would therefore tend to be better than initial (first try) results. As coursework marks are subject to other factors (such as group effects, etc.), these were excluded from the analysis.

Other variables considered

In attempting to control for other factors which might have a significant influence on learning outcomes, two further variables were considered — students’ overall grade averages and students’ lecture attendance rates. Overall grade averages, determined on the basis of all courses taken by the students, were considered to be a suitable proxy measure for overall student performance / ability in all courses. However, these overall grade averages in most cases included the grades achieved on the project management in construction course, but, since this typically accounted for such a small proportion (approximately 2%) of the overall grade average, no adjustment to the overall grade averages was considered necessary.

Lecture attendance rates were considered a proxy measure for student motivation / effort with regard to this particular course.
Language proficiency self-assessments, together with exam results and lecture attendances were compiled for a total of 393 students.

Overall grade averages, however, were generally only available for students who had successfully completed all their taught courses at the time of data compilation. For this reason, overall grade averages were collected for only 322 students.

Analysis of Results

Analysis of the data was carried out using the SPSS software package.

Correlation testing

Figures 1-3 (below) show scatter plots of students’ language proficiencies, lecture attendances and overall grade averages; respectively, versus their exam results. The scatter plots suggest weak, positive correlations between language proficiencies and exam results and also between lecture attendances and exam results. The scatter plot in Figure 3 indicates a moderate, positive correlation between students’ overall grade averages and exam results. 

Figure2: Scatter plot of Lecture Attendance versus Exam Result

 
These correlations were confirmed through the calculation of Pearson correlation coefficients in each case and all were found to be statistically significant at the 0,05 level (one-tailed testing) as shown in Table 1 below.

Table 1:  Correlation of variables with Exam Results Pearson correlation testing

Determination of a language proficiency ‘threshold’

To further analyze the effect of language proficiency on exam results, students were categorized into groups according to their language proficiency levels. Four different categorizations were tested: 2 language proficiency levels (low, high), 3 levels (low, medium, high); 4 levels (low, low-medium, high-medium, high); 6 levels (very low, low, lower medium, higher medium, high, very high). In each case, the range of language proficiency values falling into each category was chosen, so that, the number of students in each category was approximately the same. The mean exam results associated with each language proficiency level were compared using one-way ANOVA, and statistically significant differences between the categories were found in all cases (at the 0,05 significance level). Tables 2-5 (below) summarize the data input and output for the one-way ANOVA testing and Figures 4-7 (below) provide graphical illustration of the observed differences in mean exam results associated with each language proficiency category. 

Table 3: Comparison of Mean Exam Results for 3 levels of Language Proficiency

Figures 4-7 (above and below) appear to indicate that a ‘threshold’ level of language proficiency exists and students in the sample with language proficiencies below this level scored significantly lower mean exam results. This phenomenon is observable for all the categorization combinations tested.

The language proficiency level of this threshold can be seen from Figures 4-7 to correspond approximately with the categorizations: ‘high’ (>= 4) for 2 levels; ‘high’ (> 4) for 3 levels; ‘high’ (> 4,2) for 4 levels; and ‘high’ (>= 4,2) for 6 levels of language proficiency. The accuracy of the estimated level increases with the increased refinement (number of levels) of the categorization suggesting that the threshold is slightly above the B2 language proficiency level.

Table 4:  Comparison of Mean Exam Results for 4 levels of Language Proficiency

Figure 6: Graph of Mean Exam Results for 4 levels of Language Proficiency

 

Table 5: Comparison of Mean Exam Results for 6 levels of Language Proficiency

Quantification of the Effect of Language Proficiency

Figures 4-7 indicate that the magnitude of the disadvantage experienced by students with language proficiencies below the threshold level is of the order of 5% – 10% of mean exam results.

In attempting to test this statistically, the data were subjected to univariate ANOVA controlling for the effects of lecture attendance (a proxy measure for student effort and commitment to the course) and overall grade point averages (a measure of the general successfulness of the students). The partial eta squared values calculated for each (2-, 3-, 4- and 6-level) categorization of language proficiency indicate that the proportion of variance in exam results attributable to each of these factors is as shown in Table 6 below. (All the results shown are significant at the 0,05 level.)

Table 6:  Proportion of variance in exam results attributable to language proficiency and other variable

However, it must be noted that these results refer to all the categories of language proficiency and are not limited to those above and below the threshold value. Thus they tend to underestimate the effect of being below or above the threshold language proficiency level. Additionally, for the 2-level categorization of language proficiency, the ‘high’ (>= 4) category overlaps the threshold value which has already been established as being slightly above B2 (4). In consideration of this, the 2-level categorization of language proficiency was amended so as to reflect a ‘low’ level corresponding to language proficiency below the threshold level and a ‘high’ level corresponding to language proficiency above the threshold level. The partial eta squared values including those for this corrected 2-level categorization are shown in Table 7.
 

Thus, the proportion of variance in exam results attributable to language proficiency is shown to lie between 3,2% (for the 4-level categorization) and 6,4% (for the 6-level categorization). Arguably, the most accurate estimate of the effect is provided by the partial eta squared value for the corrected 2-level language proficiency categorization, i.e. 5,1%. In all cases, the effect of language proficiency remains significant (at the 0,05 significance level) when lecture attendance and overall grade average are controlled for.

While lecture attendance explains a smaller proportion of the variance (approximately 2,4%) in the exam result, the overall grade average variable accounts for a far greater proportion of the variance in the dependent variable (approximately 19%) than either language proficiency or lecture attendance.

Conclusion

There is evidence of a language-based disadvantage associated with having language proficiency at or below the B2 level. This suggests that the B2 level would provide an appropriate minimum proficiency level for students enrolling in a course taught in a foreign language. However, it does not suggest that a lower level of language proficiency cannot be overcome through student effort.

The magnitude of this disadvantage, in the case of this particular course, has been shown to be approximately 5% but this should be considered in relation to attempts to mitigate the effect of language proficiency differences. With greater efforts at mitigation it probably could be reduced and, if none were in place, then it seems reasonable to assume that the magnitude of the disadvantage would be greater.

Beyond the classroom, this has considerable implications for communication and understanding in all multilingual contexts. With such contexts increasingly arising through globalization, it suggests that significant (and, therefore, measurable) performance deficits and inefficiencies are emerging both in education and in industry. While this is unlikely to amount to an argument ‘against’ globalization, it certainly suggests that greater efforts should be made to mitigate language proficiency effects.

Acknowledgement

This research was partly supported by the Digital Processes for Education and Management of Construction (DigiEdu) project funded by the Central Baltic INTERREG IV A Programme 2007-2013 of the European Regional Development Fund and the Reformation of the Curricula on Built Environment in the Eastern Neighbouring Area (CENEAST) project funded by the Tempus Programme of the European Commission. The findings and opinions reported in this paper are the sole responsibility of the authors and can under no circumstances be regarded as reflecting the position of the programme funders.

References

1.    Alfan, E., Othman, M. N.(2005) ‘Undergraduate students’ performance: the case of University of Malaya‘, Quality Assurance in Education, Vol.13, No.4, pp. 329 – 343.  Publisher – Google Scholar

2.    Ayoko, O. B. (2007) ‘Communication openness, conflict events and reactions to conflict in culturally diverse workgroups’, Cross Cultural Management: An International Journal, Vol.14, No.2, pp. 105 – 124.  Google Scholar

3.    Comu, S., Unsal, H.I, Taylor, J.E.(2007) ‘The Dual Impact of Cultural and Linguistic Diversity on Project Network Performance’, Project Network Dynamics Lab Working Paper #0705, Columbia University

4.    Cuevas, G. J.(1984) ‘Mathematics Learning in English as a Second Language’, Journal for Research in Mathematics Education, Vol.15, No.2, (Minorities and Mathematics), pp. 134 – 144.  Google Scholar

5.    Eskew, R. K., Faley, R. H.(1988) ‘Some Determinants of Student Performance in the First College-Level Financial Accounting Course’, The Accounting Review, Vol.63, No.1, 1988, pp. 137 – 147.

6.    Fields, D., Chan, A., Akhtar, S., Blum, T. C.(2006) ‘Human resource management strategies under uncertainty: How do US and Hong Kong Chinese companies differ?’ Cross Cultural Management: An International Journal, Vol.13, No.2, pp. 171 – 186.  Publisher – Google Scholar

7.    Fong, P. S. W., Lung, B. W. C.(2007) ‘Interorganizational Teamwork in the Construction Industry’, ASCE Journal of Construction Engineering and Management, February 2007, pp. 157 – 168.  Publisher – Google Scholar

8.    Fontaine, R. (2007) ‘Cross-cultural management: six perspectives’, Cross Cultural Management: An International Journal for Construction Marketing, Vol.14, No.2, pp. 125 – 135.  Publisher – Google Scholar

9.    Foster, G. (2012) ‘The impact of international students on measured learning and standards in Australian higher education’, Economics of Education Review, Vol.31, 2012, pp. 587 — 600.  Publisher – Google Scholar

10.    Hofman, A., van den Berg, M.(2003) ‘Ethnic Specific Achievements in Dutch Higher Education’, Higher Education in Europe, Vol. 28, No.3, pp. 371 – 389.  Publisher – Google Scholar

11.    Jaselskis, E. J., Strong, K. C., Aveiga, F., Canales A. R. & Jahren, C.(2008) ‘Successful multinational workforce integration program to improve construction site performance’, Safety Science, Vol.46, No.4, pp. 603 – 618.  Publisher – Google Scholar

12.    Lill, I.; Witt, E. (2008a) ‘The Effect of Language of Instruction on Course Results for Civil Engineering Students in Estonia’, in Haigh, R.; Amaratunga, D. (ed.). Proceedings of the CIB International Conference on Building Education and Research, Building Resilience, Conference Proceedings, pp. 287-299.  Google Scholar

13.    Lill, I.; Witt, E. (2008b) ‘Training in a Multi-Cultural Environment: The Effects of the Language of Instruction’, Proceedings of the CIB W112 International Conference on Multi-National Construction Projects, Securing High Performance Through Cultural Awareness and Dispute Avoidance, Tongji University, pp. 1 — 11.

14.    Ling, F. Y. Y., Ang, A. M. H.,  Lim, S. S. Y.(2007) ‘Encounters between foreigners and Chinese, Perception and management of cultural differences’, Engineering, Construction and Architectural Management, Vol.14, No.6, 2007  pp. 501 – 518.  Publisher – Google Scholar

15.    Loosemore, M., Lee, P. (2002) ‘Communication problems with ethnic minorities in the construction industry’, International Journal of Project Management, Vol.20, No.7, 2002, pp.  517 – 524.  Publisher – Google Scholar

16.    Morrison, J.,  Merrick, B., Higgs, S., Le Métais, J.(2005) ‘Researching the performance of international students in the UK’, Studies in Higher Education, Vol.30, No.3, pp. 327 — 337.  Google Scholar

17.    Ozorovskaja, R., Voordijk, J. T.,  Wilderom, C. P. M.(2007) ‘Leadership and Cultures of Lithuanian and Dutch Construction Firms’, ASCE Journal of Construction Engineering and Management, November 2007, pp. 900 – 911.  Publisher – Google Scholar

18.    Rosso, E. D., Masera, M., Casals, M.,  Xavier, R. (2007) ‘Gender and multicultural issues in the construction project management: a knowledge management approach’, in P. J. W. F. W. E. Prof. H.A.J. de Ridder (ed.), Second International Conference World of Construction Project Management, CIB, TU Delft, The Netherlands.

19.    Wells, J. (1996) ‘Labour migration and international construction‘, Habitat International, Vol.20, No.2, pp. 295-306.  Publisher – Google Scholar