2020: The year that Ofqual’s “comparable outcomes” system ran into the ground?

Does the government’s reported extraordinary last-minute change to the way this year’s A-level grades are awarded spell disaster, this year at least, for the carefully-calibrated statistical system through which England’s exams regulator has controversially attempted to combat “grade inflation”?

The question suggests itself after the Guardian reported a staggering move by ministers which would see students in England given the right to substitute results they had received in their mock exams earlier in the year if they were unhappy with those awarded to them by the exam boards.

The reported intervention emerged less than 36 hours before students are due to receive their results from the boards, and with the Scottish government having been forced to re-instate 124,000 downgraded results north of the border.

This seems likely to be widely viewed as a chaotic development, with the leader of the Association of School and College Leaders (ASCL) – a union which often sounds cautious in any criticism of government – stating that the move “beggared belief”.

Geoff Barton, ASCL’s general secretary, said: “The idea of introducing at the 11^th hour a system in which mock exam results trump calculated grades beggars belief. If the government wanted to change the system, it should have spent at least a few days discussing the options rather than rushing out a panicked and chaotic response.”

Although many students will no doubt welcome an additional chance to upgrade their results, if this report proves true, obvious questions were raised in the Guardian’s piece as to fairness. For example, students in schools which had yet to stage mocks when lockdown hit would be disadvantaged, while those who had not worked as hard for their mocks as they would have done for the then-cancelled real exams would also be negatively affected.

But another striking aspect of what was reported, in my view, is how it contrasts completely not only with the grade allocation system which the regulator Ofqual has presided over for the past decade, but also, seemingly, with its overarching goal for this summer’s results.

Comparable outcomes

The system which Ofqual has operated throughout the period of the Conservative-led government* has been “comparable outcomes”.

This essentially seeks to keep the proportions of pupils achieving each grade broadly constant from one year to the next, even in cases of major change to the exams system, such as happened recently with the introduction of 1 to 9 grades at GCSE, to replace the old A*-G structure.

I don’t think it is too controversial to describe comparable outcomes as a rationing of grades, with grade boundaries in individual subject exams adjusted upwards or downwards to ensure that the proportion of grades each year is kept roughly constant.

This system was an attempt to combat “grade inflation”, which had seen the proportion of good grades climbing steadily at both GCSE and A-level in the preceding years.

The core assumption behind comparable outcomes, that students’ qualities do not change much from year to year, and therefore that that grades should be held roughly constant, would actually have protected candidates during this unprecedented summer.

That is, even if their actual education has suffered greatly, as a national cohort their grades would not, because of the process of essentially equalising results form year to year.

Indeed, Ofqual has already signalled that grades overall are to be roughly in line with those of previous years, at both A-level and GCSE, despite the undoubted shock to students’ education that Covid-19 has brought.

For, as slides released for Ofqual’s “summer symposium” last month on how grades to be awarded this year despite the unprecedented challenge of exams not actually being sat, this was made clear.

Ofqual said: “Outcomes this year will be broadly similar to those of previous years, if very slightly more generous.”

But now we get to the nub of things. Ofqual said that this was because, in what is set out as seemingly its top priority for this summer’s exams its “challenge” was “Maintaining standards nationally so that the value of the qualification is the same this year, as last year and next year.”

A key worry for Ofqual, then, as these slides underline, was that the “value” of GCSEs and A-levels to this year’s pupils and to future years’, would slide if, as it saw it, too many candidates were awarded high grades.

In another slide, the regulator had said that there was an “expectation” that “results will look broadly similar to previous years – nationally and at individual centre level – because this is important for students and for public confidence in the qualifications”.

That is, the proportion of students achieving each grade each year needs to be kept broadly constant for this reason from year to year, would be the regulator’s argument.

This appears to have been a central consideration in its decision to adopt the controversial model Ofqual has gone for this summer in terms of grading, whereby in larger subjects student grades will be calculated according to the rank order their teacher puts them in within a school for that subject, taking into account how the school performed in previous years.  

An alternative might have been to base the grades on teachers’ own submitted judgments of their students.

But this was rejected by Ofqual, with the regulator, again for this symposium, having included a slide which suggested grades soaring at both GCSE and A-level, compared to last year, as the “vast majority” of schools and colleges submitted grades for their pupils which were “optimistic”, compared to the results in 2019.

For example, the slide suggested that 13.8 per cent of A-level grades would have been at A* this summer if based on teachers’ own submitted grades of their students. This compared to only 7.8 per cent gaining the grade nationally last year. Similarly, the figure of 73 per cent of pupils who gained a grade 4 and above last year would have risen to 82 per cent if based on teacher grades this year.

Over the years, comparable outcomes has been used to keep any possible rise in national average grades to a much smaller change. Indeed, back in 2012 statistically-based interventions by Ofqual into grade boundary decisions by exam boards for GCSE English in order for keep any increase in results relatively small were at the heart of possibly the last major national exam grading row.

What happens now?

Much of this recent work by the regulator has been controversial, of course, not least in that comparable outcomes appears more or less to guarantee that a certain proportion of students each year will end up doing badly, many of them therefore emerging from 11 to 13 years of education with little to show for their efforts in terms of England’s most well-known qualifications, at least.

But my point with this piece is not to get into that debate, but rather to highlight how the approach outlined in the Guardian’s piece stands so starkly at odds with the approach of the regulator over the past decade.

For if students really are to be allowed to base their overall grades in performance in their mock exams, it appears that Ofqual will have lost all control over the proportion of students achieving its grade, which appears to have been its main priority since 2010 in its overseeing of the exams system, and again its main priority this year.

That is, it would seem to me to imply there would be no attempt to use statistics to control the number of students awarded each grade.

So this government move, if indeed it is true, appears to put ministers directly in opposition to the longstanding position of their regulator.

As with so many aspects of the response to Covid-19, this crisis is proving extraordinarily revealing, provoking as it does a host of fresh questions about the competence of the government. Hopefully the extra stress on students will be minimised, but changes at this stage hardly inspire confidence.

*Comparable outcomes was at least in development towards the end of the New Labour period. I confess I need to check whether it had actually been used at a national level before 2010.