A meta-analysis of effects of problem-based learning (PBL) has shown consistent positive effects for skills, but non-robust effects for knowledge (Dochy et al. 2003). These findings imply that a PBL curriculum helps students to develop skills (such as applying knowledge), but is not more successful compared to more traditional curricula, in stimulating students to acquire declarative knowledge. The findings of this meta-analysis confirm conclusions of other reviews on the effects of PBL (Vernon and Blake 1993; Albanese and Mitchell 1993). These meta-analyses focus on cognitive measures: knowledge and skills. Pascarella and Terenzini (2005) describe these as the learning outcomes generally thought to be directly related to the curriculum or academic program. In their comprehensive overview of research describing the ways in which college affects students, they also pay attention to the acquisition of more generic college outcomes such as study progress and retention. Tinto’s model on student persistence (1997, 1998) provides a useful framework to examine the effects of PBL on these more generic outcomes. The central aim in the present study is to extend research on the effects of PBL by making use of Tinto’s model.

Tinto describes as central to the college experience two concepts: (1) institutional experiences, involving the educational system, and (2) academic and social integration. Tinto’s model assumes that institutional experiences impact on persistence directly as well as indirectly via social and academic integration. The indirect route considers high levels of social and academic integration to be conditional for the decision to continue studying. According to Tinto’s model, students who persist in college and graduate successfully participate in the student culture, both within and outside the immediate context of the learning environment. (See also Severiens et al. 2006; Severiens and Wolff 2008.) Those who feel at home, who take part in extra-curricular activities, and who feel connected with fellow students and teachers, are more inclined to persist in their studies. Without social integration, it is more difficult to persist, and ultimately to graduate. Integration naturally has an academic aspect as well as a social one, as one’s willingness and ability to belong to a group depends partly on one’s ability to meet the educational level. While this plainly requires certain cognitive abilities, time and effort are also required. Similarly, teachers and staff have to provide an educational context that invites students to integrate academically.

The concepts of social and academic integration also entail an additional distinction, one between formal and informal integration, each of which is important for successful integration. While formal academic integration involves the contacts related to studying and the institute itself, informal academic integration involves contacts between teachers and students outside the direct context of the learning environment, i.e. whether students and teachers consider themselves to be at more or less the same level socially, and whether they discuss personal matters with each other. Similarly, at the level of social integration, formal integration mainly involves contacts between peers on matters of learning. Such contacts often revolve around collaborative work—for example, the ways in which students experience working together on tasks. Especially in curricula where project work is a substantial part of the curriculum, the quality of co-operative work may be an important determinant of student attrition. This formal level is distinct from the informal level, which is characterised by factors such as frequent social contact and participation in student activities. Students, who have many friends at school, feel at home and enjoy going to school, undoubtedly have a larger chance of obtaining their degree.

Even though the model assumes a direct connection between institutional experiences and retention (Tinto 1997, p. 615) Tinto does not clearly describe the process by which institutional experiences affect student learning, without touching upon processes of involvement and social and academic integration. On a general level, the model shows that pre-entry attributes translate into goal commitments, that given the institutional experiences result in a certain quality of student effort and ultimately in student persistence. There is however a large body of research that does describe this direct link between institutional experience, or the learning environment, and the quality of learning. In the next section of this paper, an overview will be presented of research examining the direct as well as the indirect connections between institutional experiences, integration and student learning and persistence.

It is remarkable that even though Tinto’s model is a widely used model in higher education that describes the departure process, a literature search in the SSCI and ERIC databases on PBL in combination with Tinto’s framework does not result in any hits. This lack indicates two distinct bodies of research: on the one hand research on effectiveness of PBL, and on the other hand research on Tinto’s work on the student departure process. However, we do find some research if the more generic search terms active learning, cooperative learning, or learner centeredness are used in combination with Tinto’s model. Because PBL may be characterized as an activating and cooperative learning environment (Schmidt 1993), hypotheses regarding the effects of PBL on social and academic integration and student learning may be drawn from this body of research.

As described, the effects of PBL on knowledge and skills have been examined in several meta-analyses. The present study aims to investigate the effects of PBL on more generic college outcomes by making use of Tinto’s twin concepts social and academic integration. Our literature search focuses therefore on research examining the direct and indirect relationships between activating learning environments (instructional methods aiming to activate students), Tinto’s social and academic integration (or related concepts such as feelings of belonging, or quality of teacher-student and student–student interactions) and the generic college outcomes study progress and retention. We have limited ourselves to peer-reviewed (published) research that was done from 1997 onwards. In the next section an overview is presented of the studies we have found.

An overview of former research

In this section a description will be given of the research focusing on (1) the link between activating learning environments and study progress and retention, (2) the link between activating learning environments and social and academic integration, and (3) the link between social and academic integration and study progress and retention.

Ad (1) Most studies investigating the direct link found positive relationships between activating learning environments and study progress and retention. For example, a recent study by Zepke et al. (2006) showed how learner centred education improves retention and completion rates. This study confirmed earlier findings by Yorke and Thomas (2003). Learner centeredness is described in terms of stimulating feelings of belonging in institutional culture, good quality teaching in general, and perhaps most crucial, catering of diverse learning preferences. In other words, for the learning environment to stimulate retention, it should be adapting to the diverse backgrounds of students. Tien et al. (2002) investigated the effect of an activating instructional approach (student-centered instruction) and found a significant increase in performance, retention and attitudes.

A study by van den Berg and Hofman (2005) concluded on the basis of a large-scale quantitative study that passive forms of education (lectures) result in less study progress. In their study, this finding is related to parallel versus serial course planning: activating courses such as PBL are planned in a serial and integrated manner. It is this aspect that seems to stimulate students in their study progress.

Ad (2) The studies described above focused on the direct link between institutional experiences and student learning. Some of the studies interpret their results in terms of processes of social and academic integration. For example, Zepke et al. (2006) described the importance of feelings of belonging in institutional culture, which is considered to be an important component of academic integration. In other words, this study seems to conceptualise institutional experience as consisting of instructional methods as well as academic integration. The present paper however, follows Braxton et al. (2000) conceptualisation in the sense that active learning and other classroom activities are seen as antecedents (sources of influence) of academic and social integration rather than as classroom activities themselves (‘active learning should not be confused with academic integration’, p. 571). They have studied the relationship between active learning behavior in the classroom on the one hand, and social integration, involvement, and the decision to continue on the other hand. Their study showed that active learning behavior indeed fosters social integration. Moreover, social integration was positively related to the decision to return.

Another study that finds similar results is from Kember and Leung (2005). They conducted a large scale survey on graduates from a university on Hong Kong. This study did not take Tinto’s model as a starting point, but focused on the relationship between the teaching and learning environment and the development of a set of intellectual capabilities. Ramsden and Entwistle’s theory of student learning (1981) was their starting point. One of the main results in this study was that active approaches to teaching and learning help to develop good teacher-student relationships in a mutual process. Good quality relationships make it more comfortable for teachers to introduce forms of teaching involving active student participation. If we translate these results to Tinto’s model, activating learning environments seem to be related to academic integration indicated by teacher-student relationships, in a mutual process.

The central research question in a study by Umbach and Wawryzinski (2005) was about faculty behaviors and attitudes and their relationship with student behaviors that are linked with positive undergraduate outcomes. On the basis of an analyses of two national US data sets (NSSE survey and a parallel faculty member dataset), they concluded that institutions where faculty members use active and collaborative learning techniques, levels of engagement and student learning were higher. Student learning was defined as self-reported personal and social gains (e.g., developing a personal code, learning effectively on your own), general education gains (e.g., writing, speaking, critical thinking) and practical competences (e.g., job skills, solving complex real world problems). Prince (2004) has conducted a study that focused on the relationship between activating learning environments and integration. In this study, it is shown that active learning (that is: collaborative and cooperative learning) promoted the quality of social interaction.

Ad (3) Braxton and Lien (2000) have reviewed the empirical evidence regarding the effect of academic integration in the student departure process. They conclude that, studies involving multiple institutes ‘…provide robust empirical backing for the effect of academic integration on both subsequent institutional commitment and student departure decisions’ (p. 22). However, studies involving single institutes receive modest empirical support. A possible explanation for the modest support is the little variation in academic integration measures in ‘single institute’ studies. Zepke and Leach (2005) arrive at similar conclusions in their review of the literature on social and academic integration. Regular and meaningful contact with teachers, as an indicator of academic integration, has a positive impact on student outcomes. Besides, students perform better in institutes that promote social integration. However, Zepke and Leach also point at research showing the downside of social integration: too much social interaction may also hinder good study performance (see also Thomas 2002).

The general conclusion from the research investigating the links between activating learning environments, social and academic integration and study progress and retention can be described as follows. Activating and cooperative learning environments foster social and academic integration, and in turn, integration positively affects generic learning outcomes such as progress and retention. However, these links do not always emerge as statistically significant links, and in some cases the links are considered to be referring to a mutual relationship. Our overview of the research describes a number of studies that have established the links between activating learning environments, social and academic integration and outcomes in a variety of ways. The aim of the present study is to examine the effects of one particular learning environment (PBL), and use Tinto’s twin concepts of social and academic integration to examine effects on study progress. We chose to use study progress as the central outcome measure, as it is one of the main generic learning outcomes in the research areas as described above. The following research question informed our study: In what ways are institutional experiences, social and academic integration and progress inter-related? Institutional experiences refer to a comparison of PBL to more conventional instructional methods. Because PBL can be considered to be an activating and cooperative learning environment, and because most former research suggests positive relationships, it seems fair to expect positive effects of PBL on levels of social and academic integration, as well as positive effects from integration on progress, as well as positive effects from PBL on progress.

Method

Participants

Participants are 305 first year students from three different psychology curricula with an average age of 20 years (SD 2.5), 61 males (23%) and 271 females (73%).

Institution 1 is a full-fledged problem-based curriculum. PBL in this institution is designed according to the ‘classical’ PBL rules. Students meet in small-group tutorials twice a week throughout the curriculum, working on “problems” that form the starting point of the learning. These problems are prepared for them by their teachers. Groups are guided by a tutor whose role is to facilitate, not to teach. Lectures are sparse, emphasis is on self-directed learning. The curriculum consists of a series of eight modules per year. Each module covers one specific psychological theme. All the activities the student undertakes during a module are related to this theme. Institution 2 offers a combination of conventional and activating teaching methods such as group work on practical assignments and Institution 3 is a lecture-based curriculum. These differences are clearly reflected in students’ perception of the learning environment. For example, in Institution 1, 41% of the students state they have to work together in groups all the time, this percentage is 17% in Institution 2, and 9% in Institution 3. When asking students about large lectures, 34% of the students in Institution 1 answer that much of the instruction time consists of large lectures, versus 57% of the students in Institution 2 and 83 % in Institution 3.

Materials

Our operational definition of integration was based on an earlier qualitative study conducted in The Netherlands (Severiens et al. 2006) in which 138 students were interviewed on their social and academic experiences in different periods during their study. In order to create a valid and reliable instrument in the context of Dutch Higher Education, we used excerpts from these interviews to develop four sets of items measuring formal and informal social and academic integration. Students were asked to rate each of the items on 5-point scales (‘not true at all’ to ‘completely true’). Reliability analyses resulted in a final choice of items. Table 1 provides two examples, the means and standard deviations, and the reliabilities of each of the scales.

Table 1 Description of measures (mean scores, standard deviations, number items, and reliabilities)
Full size table

Comparing our scales to the scales as used by Nora and Cabrera (1996) and Berger and Milem (1999) shows some similarities and differences. Our social integration scale is similar to Nora and Cabrera’s (which is based on Pascarella and Terenzini’s Interactions with Peers-scale). Our academic integration scale is similar to the Berger and Milem’s scale for academic integration. The difference is that we make an additional distinction in our scales between informal and formal aspects of integration. Informal integration refers to interaction outside the classroom, whereas formal integration refers to interaction during studying (see Table 1). The reason for this additional distinction is twofold. Firstly, Tinto’s model clearly indicates this distinction. Secondly, and perhaps more importantly, the students in our interviews describe formal and informal interaction with peers and teachers as distinct issues.

Study progress was measured by the number of credits points students obtained after one year of studying. This information was obtained from the student administration offices.

Procedure

The students were invited to participate in an email sent out in the spring of 2005. The email contained a link to an online version of the questionnaire. Information on study progress was obtained from the student administration offices in the same fall. The response rate was 33%.

Analyses

The statistical analyses were conducted following a two-step approach. In the first step, descriptive statistics were calculated to provide an impression of the magnitude of differences between the three curricula. In the second step, linear structural modeling analyses between the variables in question were conducted (Jöreskog and Sörbom 1993). This method makes it possible to test specific hypotheses about the relationships between the relevant variables. To assess the overall goodness-of-fit of the tested model, three measures are used: the Chi-square test statistic (χ 2), the adjusted goodness-of-fit index (AGFI) and the root mean square error of approximation (RMSEA). It has been argued that a value of 0.08 or less for the RMSEA indicates a reasonable error of approximation (Browne and Cudeck 1992); additionally, according to Bentler (1990), when the goodness-of-fit and adjusted goodness-of-fit indexes are greater than 0.90, the analyses indicate adequate fit of the models. It has generally been accepted that χ 2 should be expressed relative to the corresponding degrees of freedom. Among others, Carmines and McIver (1981) suggested that before rejecting a model as ill fitting, χ 2 should be two or three times greater than the degrees of freedom (Punnett and van der Beek 2000).

Results

A descriptive analysis of the data shows that after one year of studying, 66.3% of the PBL students have obtained the maximum number of credit points, compared to 47.0% in the mixed curriculum and 40.7% in the conventional curriculum. This difference is statistically significant (χ 2 = 21.261, df = 6, P < 0.002). This result seems to indicate that PBL is indeed more successful in stimulating students to keep their study pace.

Table 2 shows the means and standard deviations of the four types of integration in each of the three curricula, as well the results of analyses of variance. The PBL curriculum shows the highest average scores on formal and informal academic integration and on formal social integration. No differences were observed in informal social integration. It appears that the PBL system indeed encourages social and academic integration to a larger extent compared to more conventional curricula.

Table 2 Descriptive statistics on integration in each of the three curricula
Full size table

In the second step, Lisrel analyses were conducted to determine the interrelationships between the learning environment, integration and study progress. Figure 1 present the hypothetical model. The literature has shown that activating learning environments generally stimulate students to staying focused on their academic work and obtaining their credit points. In the model this is shown by drawing an arrow between the curriculum and study progress. The same is true for integration and study progress. The line of argument is that high levels of integration foster good quality learning, resulting in good study progress. Finally, the arrows between the curriculum and integration reflect the literature stating that a positive impact may be expected from activating learning environments on social as well as academic integration.

Fig. 1
figure 1

Theoretical model

Full size image

As we are interested in the unique contribution of each of the four types of integration, we allow for the error-covariances between all four measures to co vary. This causes the model to become saturated: the number of parameters to be estimated is the same as the number of ‘known’ parameters. By excluding the path with the smallest non-significant regression weight (the path from informal social integration to the number of credit points), a degree of freedom is generated, and it becomes possible to calculate the fit of the model. The fit of this model (without the link between informal social integration and credit points) is good: Chi-square is 0.67 (P = 0.38), with 1 degree of freedom, RSMEA = 0.000 and NFI is 1) (see Fig. 2 for the accepted model and coefficients).

Fig. 2
figure 2

Accepted model with statistically significant coefficients

Full size image

First of all, there is a direct link between curriculum and study progress. This link confirms the average differences between the three curricula: students in the PBL curriculum obtain larger numbers of credits after one year of studying compared to the mixed and conventional learning environment. Second, contrary to our expectations, informal academic integration is negatively related to study progress. This means that students with low numbers of credit points, rate the quality of informal contacts with their teachers as higher. In other words, students that lag behind are more satisfied about the informal contacts with teachers. It seems that teachers more often know these students and pay more (informal) attention to these students. In other words, this negative effect seems to refer to a correlation rather than an effect from integration on number of credits (a model that includes this correlation fits equally well).

Third, there are three positive relationships between curriculum and integration. This means that the PBL curriculum has a more positive impact on integration compared to the more conventional curricula. The quality of informal as well as formal contacts between teachers and their students in PBL is higher compared to the quality of contacts in the other two curricula. The same is true for formal social integration: students in PBL are more satisfied about their cooperative work compared to students in the other two curricula.

Discussion

From an extensive body of research on the effects of PBL (e.g., Dochy et al. 2003) we know that PBL stimulates students, more than conventional programs, to develop their cognitive skills. No (stable) differences appear between PBL and conventional curricula in stimulating students to acquire knowledge. The aim of the present study was to examine the effects of PBL in wider terms, and use the generic student outcome study progress. Tinto’s model on student attrition (1998) may supply the reason for a positive effect of PBL on study progress. If the levels of social and academic integration are higher among students in a problem-based psychology curriculum compared to their counterparts in mixed and conventional curricula, then PBL may be more successful in stimulating study progress.

The present study showed higher levels of social and academic integration of students in a PBL environment. PBL students were more satisfied about the quality of formal and informal contacts with their teachers, as well as the quality of formal contacts with their peers. Our results confirm former research indicating that activating learning environments foster social integration of students (Braxton et al. 2000; Prince 2004). The higher scores on academic integration in PBL imply that teachers in a PBL environment more often make an effort to know their students, take them seriously and invite them into the profession. This undoubtedly is related to the fact that teachers—in particular tutors—and small groups of students meet several times a week in PBL.

The present study has also shown a direct effect from the learning environment on study progress. PBL seems more successful in stimulating students to obtain their credits. Our study on PBL shows similar results compared to other studies that have examined activating and cooperative learning environments. One should note here that our ‘mixed’ curriculum offers a fair amount of activating teaching methods, but the PBL system results in the largest numbers of obtained credits.

Two of the four links between integration and study progress turned out to be significant relations. Formal social integration affects study progress, which is in line with many studies that have shown that social integration is an important predictor of study success. Students that cooperate well with fellow students, make friends, feel at home in their institute, generally do better academically. Unexpectedly, the link between informal academic integration and study progress was a negative link. This negative relationship may indicate a mutual effect rather than an impact of integration on study progress. Zepke and Leach (2005) describe this mutual effect as an adaptive aspect in the learning environment. On the basis of a review of the research on Tinto’s model they describe an emerging new discourse on how institutions may improve student outcomes. The dominant discourse describes processes, by which institutions help student to fit into their culture. A new discourse challenges this view, and argues that institutions should adapt to students and cater for their needs, in order to stimulate outcomes. It appears that our result confirms this new view. It questions the sequence in the model and suggests that at least some of the ‘arrows’ should reflect reciprocal relationships.

The fact that academic integration does not bear a larger effect on study progress does not necessarily mean that it can not play an important role. It could also mean that, as described by Tinto (1998), academic integration does not play a significant role yet, because of the low levels of integration (p. 616). The average scores in our study showed how unsatisfied students are about their contacts with teachers. This finding leaves much room for improvement in educational practice. Apart from increasing the amount of time teachers (may) devote to their students, adapting learning environments to students’ needs, in stead of the other way around, may also be a good way to improve levels of academic integration. This could be done, for example, by taking students’ approaches to studying (Prosser and Trigwell 1999) as a starting point when designing curricula and assessment methods. Or, likewise, taking students’ stages of intellectual development and patterns of reasoning (Baxter Magolda 1992) into consideration in the design of curricula. These recommendations for practice are in line with Smith and Bath’s (2006) plea for the development of a comprehensive set of guidelines for the implementation of learning communities in university settings. A parallel new line of research should focus on the question whether the impact of academic integration grows when the levels increase.

A final suggestion for further research concerns a study including second and third year students, or a longitudinal study. Such a study could show the ways in which integration affects study progress in PBL curricula in the longer run. Such a study may also answer the question whether the good informal academic integration of students at risk in their first year, helps these students to improve and become more successful in the remaining study years.