Pilot studies

The University of Ghent –  the Centre for Higher Education Governance Ghent (CHEGG) prepared two pilot pan-European comparative studies into how higher education institutions respond to the needs of students and society, based on thousands of external quality assurance reports being available in our Database for External Quality Assurance Results (DEQAR).

The first comparative study (see below) was subsequently published in the journal “Quality of Higher Education”: under the title “The use of the European Standards and Guidelines by national accreditation agencies and local review panels”.

The research on which the two studies are based was undertaken in the context of the DEQAR project (Database for External Quality Assurance Results). The project was carried out with co-funding from the European Commission (ERASMUS+ KA3 grant). The content of the study and information contained therein remains the responsibility of the authors.

Find out more about the concept (research questions) and the results (extract of conclusions) of the pilot studies in the section below. The papers may be also read in full, by downloading them onto your devices.

The results of the pilot studies were also presented and discussed with participants at the European Quality Assurance Forum (EQAF) in Berlin in 2019, during one of the three parallel plenary sessions discussing perspectives on QA.

First Pilot Study - The translation of the ESG in institutional QA reports

Research questions

For the first pilot study, published in April 2019, Jeroen Huisman and Maria Manatos analysed institutional QA review reports exploring the following research questions:

  • “Whether the ESG related to programme’s design, approval and monitoring are present in institutional external quality assurance reports;
  • Whether quality assurance agencies are translating the ESG differently when performing institutional audits of HEIs. Particularly, we aim to understand how quality assurance agencies are translating the ESG, their main concepts and dimensions, and how audit panels are applying the national guidelines for institutional external quality assurance reports and how they are approaching the ESG.”
Extracts from the concluding remarks from first pilot study
  • “Globally, our research responds positively to both our main research questions. On the one hand, the research concludes that the ESG related to programme design, approval and monitoring are present in institutional external quality assurance reports. A quantitative analysis to the frequency of ESG’s concepts and dimensions confirms that the ESG ‘appear’ in the reports, though to different extents across countries.”
  • “On the other hand, quality assurance agencies interpret and translate the main concepts and dimensions from the ESG differently, when performing institutional audits of HEIs. The differences are reflected in the ‘form’ of the reports, in particular, in the length, the focus and the guidelines for the reports. Indeed, quality assurance agencies have different foci and formulate different guidelines for institutional external quality assurance.”
  • “Under an umbrella of ‘standardisation’ and ‘homogenisation’ it is then interesting to notice how the quality assurance agencies translate and interpret the ESG, developing their own national guidelines, and how the audit panels apply such guidelines in the reports.” 
Read & download "The translation of the ESG in institutional QA reports" by UGent

Second Pilot Study - The translation of the ESG in programme QA reports

Research questions

For the second pilot study, published in October 2019, Jeroen Huisman and Maria Manatos analysed programme QA accreditation exploring the following questions:

  • How do accreditation panels offer evidence for their findings with respect to:
    • encouraging students to take an active role in creating the learning process (ESG1.3)
    • and consistently applying pre-defined and published regulations covering all phases of the student “life cycle” (ESG 1.4)?
  • Are there differences – with respect to the findings between countries and types of institutions?
Extracts from the concluding remarks from the second pilot study
  • We conclude that globally the ESG 1.3 and 1.4 are ‘present’. […] However, they are subject to interpretation, first by the agency, which edits the ESG, though not radically changing them; second, by the panels, who also edit the ESG and present different evidences for the different dimensions inside the ESG
  • re ESG 1.3, “the panels offer more explicit evidences for findings related to student’s assessment (mostly assessment procedures) and also (though to a lesser extent, and sometimes less explicitly) to the adaptability of teaching and learning (according to students’ paths, labour market needs, etc); and more implicit or even no evidence for findings related to students’ motivation, complaints, engagement, student-teacher relationship.”
  • re ESG 1.4 “the panels tend to offer more evidences for findings related to the admission (mostly admission procedures) and progression (namely procedural aspects and ways to support students to progress in their studies), and certification, than related to informal learning and recognition (particularly important European references with regard to recognition). “
Read & download "The translation of the ESG in programme QA reports" by UGent

As methodology, both studies follow a content analysis of the external quality assurance reports from in order to understand how quality assurance agencies translated the ESG for external quality assurance based on concepts (not only from the standards but also from the guidelines) related ESG 1.2 (2005), ESG 1.2 and 1.9 (2015) for the first study and ESG 1.3 and ESG 1.4 for the second study.