WiWiKom II

Valid assessment of students’ development of professional business and economic competencies over the course of their studies - A quasi-experimental longitudinal study

Abstract

The WiWiKom II project is based on the competency model developed and tested and the test instrument used in WiWiKom I (Zlatkin-Troitschanskaia et al., 2014). The overarching aim of WiWiKom II is objective, reliable, and valid assessment of students’ development of professional business and economic competencies that allows for valid interpretations of test results. Building on the validation approach from WiWiKom I (based on AERA, AEA & NCME, 2014; Kane 2013), WiWiKom II addresses more in-depth validation questions and expands the results gained so far by including individual change measurement.

In WiWiKom II, we implement a longitudinal design over a period of four years and four measuring dates. The project follows a quasi-experimental design, examining constructs according to the multitrait-multimethod matrix approach and two group comparisons for convergent, discriminant, incremental, and predictive validation. Quasi-experimental variation is achieved through sampling by including not only two study domains (business & economics and social studies) but also different stages of training (orientation, advanced, and specialization study phases in the Bachelor degree course). In order to conduct discriminant validation, both the WiWiKom test and an abridged version of the Berliner Test zur Erfassung fluider und kristalliner Intelligenz [Berlin test for the assessment of fluid and crystalline intelligence] (BEFKI) (Schipolowski, Schroeders & Wilhelm, 2014) are implemented. The BEFKI is a general, subject-independent knowledge test that assesses general cognitive abilities (GCA) in terms of fluid and crystalline intelligence.

Aims

In WiWiKom II, professional business and economic competencies and their development over time are modeled as central dependent variables. In a longitudinal study, we examine the extent to which these dependent variables can be explained by variables related to business and economics degree courses (e.g., learning opportunities attended over the course of Bachelor studies). Further validation questions concern the differentiation in measurement between the dependent variables and theoretically related criteria (such as GCA) as well as correlations between the dependent variables and construct-relevant external criteria (such as Bachelor degree final grade). In a quasi-experimental comparative analysis, we examine further aspects of discriminant validity by analyzing the manifestation of the dependent variables among the target group and a control group in cross-sectional and longitudinal studies. Altogether, WiWiKom II focuses on four validation aspects:

(1) Cross-sectional and longitudinal convergent validation: In a longitudinal study, the development of professional business and economics competencies is assessed and analyzed in relation to learning opportunities attended during and prior to university studies. One research focus is on the influence of learning opportunities within a business and economics degree course on the dependent variables.

(2) Cross-sectional and longitudinal discriminant validation: We analyze the differentiation, that is, the empirical separability, between GCA and professional business and economic competencies. In a longitudinal study, we examine the extent to which GCA can explain the increase in the dependent variables and whether this influence remains constant over the course of studies.
(3) Incremental and predictive validation: In longitudinal studies, we gather evidence on the ability of the test to predict study success (grade and completion of a Bachelor degree course) or successful transition into a Master degree course or into professional practice. Here, we focus on two external criteria (e.g., grade and transition) that are central to study success. The WiWiKom test should be able to explain significant proportions of variance in the external criteria and should have greater explanatory power than domain-independent predictive factors, such as university entrance qualifications and intelligence tests. Furthermore, we assume that students’ degree of expertise will increase over the course of three years of studies, which, according to the competency model, should become evident in an increase in the level of competency in business and economics.
(4) Discriminant validation by means of the known-group approach: The specificity of the test to the study domain of business and economics is analyzed by means of a comparison between two groups of students with different opportunities for acquiring professional business and economic competencies. We assume that the target group of business and economics students acquire business and economic knowledge by attending courses in this field of studies. In contrast, exploratory findings from WiWiKom I suggest that the control group of students in social studies primarily relies on competencies not specific to this field and should therefore rather refer to everyday experiences or knowledge from prior education when responding to business and economic test items. By means of the group comparison (Förster et al., 2015), we also examine how sensitive to changes the WiWiKom test is by comparing the progression of competency development while controlling for other study-related influence factors.

Project Data