As of today hEaD space has a new home over at Wordpress .... Can I say that here? Oh, I just did.
Hope to see you there.
Friday 10 October 2008
Sunday 5 October 2008
Degree classification and grade 'inflation'
This summer has seen an awful lot of ink expended on issues of degree quality and 'grade inflation'. I don't particularly want to wade in on one side or the other, largely because the issue is incredibly complex, in a way that many of the 'commentators' in the media (and in Parliament) over the summer either could not grasp or chose to ignore. Let me just make a few observations:
I instinctively dislike the term grade inflation. It carries a lot of (deliberately disparaging) baggage. Inflation is not a neutral term. Inflation implies a reduction in value. By even engaging in the debate in the language of inflation we are therefore, from the outset, assuming something that needs to be proved; not just that we are awarding more Firsts and Upper Seconds, but that this is wrong and devalues the status of our degrees. This in turn also assumes that our assessment practices twenty or thirty years ago were superior to those in place today, and that assumption should certainly be challenged.
And this is where we have to come back to the problem of complexity. Whether we like it or not, we are not comparing like with like. When Norman Baird at qed law produces a table showing that most law schools are awarding substantially more First and 2:i's in 2007 than in 1997, then we can certainly acknowledge that is interesting (though why 16 law schools were apparently awarding less in 2007 is more counter-intuitive and possibly even more interesting), but as Norman himself acknowledges, it is just a snapshot, based on some very simple percentages. It tells us nothing about which of those changes (if any) are statistically significant; it does not indicate whether either of those years chosen was atypical for that law school, so we have no sense of the outliers, and, obviously, it tells us nothing about the underlying practices that might account for that change. And there are a lot of candidates to explain the changes, mostly well known, including:
In short, I don't think we are facing a crisis of degree standards, but I think we do have some artificial expectations of what a mass system of HE can and should achieve, and a certain amount of political dishonesty about the consequences of massification; and universities have, to some extent, been complicit with goverment in that process.
If we really wanted the 'standards' of an elite system in a mass environment, we would need to resource the environment much better, and more logically (most degrees still operate an illogical system of resource allocation, whereby, as much by default as by design, a larger pro rata share of resources are spent on the later years, rather than on the critical first year of the programme). It would also have to become politically acceptable to have higher initial failure rates. Neither of these things are likely to happen. And I don't actually have a lot of problems with that. But we surely do have to accept that, in a mass system, the purpose, scope, and value of an undergraduate degree is different from what it once was. That is not to say necessarily better or worse, just different. To some degree, the market is already doing this for us, though, of course, fairly arbitrarily, and without much sensible public debate.
As it is we have at best a hybrid (some might just say confused) system, and the teachers - the poor bloody infantry in all this - have to work out how to fix a machine that, ultimately, is constitutionally broken. What are the fixes? Here are a couple of starters for ten for this particular university challenge:
I instinctively dislike the term grade inflation. It carries a lot of (deliberately disparaging) baggage. Inflation is not a neutral term. Inflation implies a reduction in value. By even engaging in the debate in the language of inflation we are therefore, from the outset, assuming something that needs to be proved; not just that we are awarding more Firsts and Upper Seconds, but that this is wrong and devalues the status of our degrees. This in turn also assumes that our assessment practices twenty or thirty years ago were superior to those in place today, and that assumption should certainly be challenged.
And this is where we have to come back to the problem of complexity. Whether we like it or not, we are not comparing like with like. When Norman Baird at qed law produces a table showing that most law schools are awarding substantially more First and 2:i's in 2007 than in 1997, then we can certainly acknowledge that is interesting (though why 16 law schools were apparently awarding less in 2007 is more counter-intuitive and possibly even more interesting), but as Norman himself acknowledges, it is just a snapshot, based on some very simple percentages. It tells us nothing about which of those changes (if any) are statistically significant; it does not indicate whether either of those years chosen was atypical for that law school, so we have no sense of the outliers, and, obviously, it tells us nothing about the underlying practices that might account for that change. And there are a lot of candidates to explain the changes, mostly well known, including:
- changing teaching and assessment methods;
- possible changes to student motivation and prior learning;
- availability of more student support mechanisms in HE;
- possibly greater individual and institutional pressures on teachers to give higher marks (including the growing recognition that marks may need to be defensible if challenged)
- maybe even the recognition in law that the practice of marking across a narrow band of marks, relative to other disciplines, is not defensible when you are dealing with what is usually one of the most able cohorts of students in the university.
In short, I don't think we are facing a crisis of degree standards, but I think we do have some artificial expectations of what a mass system of HE can and should achieve, and a certain amount of political dishonesty about the consequences of massification; and universities have, to some extent, been complicit with goverment in that process.
If we really wanted the 'standards' of an elite system in a mass environment, we would need to resource the environment much better, and more logically (most degrees still operate an illogical system of resource allocation, whereby, as much by default as by design, a larger pro rata share of resources are spent on the later years, rather than on the critical first year of the programme). It would also have to become politically acceptable to have higher initial failure rates. Neither of these things are likely to happen. And I don't actually have a lot of problems with that. But we surely do have to accept that, in a mass system, the purpose, scope, and value of an undergraduate degree is different from what it once was. That is not to say necessarily better or worse, just different. To some degree, the market is already doing this for us, though, of course, fairly arbitrarily, and without much sensible public debate.
As it is we have at best a hybrid (some might just say confused) system, and the teachers - the poor bloody infantry in all this - have to work out how to fix a machine that, ultimately, is constitutionally broken. What are the fixes? Here are a couple of starters for ten for this particular university challenge:
- The classification system as both the Burgess Report and the QAA have pointed is well beyond its sell-by date. It should be replaced, though personally I would favour a more radical solution than Burgess's transcript; possibly a grade point system.
- Perhaps we should also consider an institutionalised move to marking on the 'bell curve' for larger modules. If this is done transparently across the sector (a job for the Committee of Heads of University law Schools perhaps?), it could act as a potential brake on creeping grade inflation, as any move of the median grade would have to be at least reported, possibly agreed, within the sector. Moreover, 'marking to the curve' seems a fairer way of accounting for marker variations (including teacher experience and expectations; year-on-year variations between papers, etc) than our current supposedly 'objective' system.
Monday 22 September 2008
Widgets
I found this browsing the blogosphere.... particularly like the rather imperial red for countries visited. Delusions of grandeur obviously.
visited 26 states (11.5%)
Create your own visited map of The World or determine the next president
visited 26 states (11.5%)
Create your own visited map of The World or determine the next president
Thursday 7 August 2008
Normal service will (finally) be resumed shortly
Finally sent off the last big batch of entries yesterday morning for a new law dictionary I've been editing. It's been a mammoth task. The project's been ongoing for over two and a half years, with a team of nearly twenty contributors. The editing process has taken up virtually all my spare time over the last six months, and then some; it certainly hasn't left much time for blogging! It will be good to have it entirely finished, but there's still a last few entries outstanding, some cross-checking and the proofs to be done. It has reached that stage now when it will just be sooo good to see it finished....
Friday 11 April 2008
Small world
So here I am sitting with a glass of wine on a hotel veranda in Barbados, musing on the occasional delights of academic life....
Actually one of the biggest oddities must be travelling all the way to somewhere like Barbados and not even making it onto the beach, but that's how it goes sometimes. I'm over here to do job (leading a Faculty Review of the Law Faculty at the University of the West Indies) and competing pressures on my time mean I arrived late last Sunday afternoon, have been working with the review team all week and fly out tomorrow (Friday) pretty much as soon as we have reported back to the Faculty. Still it has been interesting. I find these kinds of quality processes fascinating, and enjoyable - especially when the students are as engaging and articulate as they are at UWI - but challenging too. There's the whole rather forensic thing of asking the right questions of different people and working through papers to triangulate the data we're accumulating, and then there's the developmental processes of trying to articulate good practice and make viable recommendations that will support a department to make improvements where necessary. Plus sitting behind all that activity is the little voice in your own head (not so) quietly interrogating you on your and your own department's own practice!
Obviously the outcomes are confidential, but fortunately the incidental things that fall out of the process are always really interesting too. I know more about the Caribbean Single Market than I did before I started, have discussed with a Jamaican Professor of International Law whether Caribbean human rights law allows a margin of appreciation in the way of the ECHR (answer: it doesn't, or certainly not in those terms) and experienced lots of the ways in which globalization is shrinking our world. For example, meeting the head of the law school at the College of the Bahamas, who just happens to be a Warwick graduate; sitting at lunch up at the Cave Hill Campus discussing Obama's prospects for the Democrat US Presidential nomination, almost as if it were a matter of local politics; hearing law students and law teachers expressing concerns that almost exactly mirror those I hear in the UK: increasing numbers, worries about appropriate levels of skills (of the students, not the staff... mostly), worries about job prospects (students again)and the increasing work pressures on the academics too. Plus ca change....
Actually one of the biggest oddities must be travelling all the way to somewhere like Barbados and not even making it onto the beach, but that's how it goes sometimes. I'm over here to do job (leading a Faculty Review of the Law Faculty at the University of the West Indies) and competing pressures on my time mean I arrived late last Sunday afternoon, have been working with the review team all week and fly out tomorrow (Friday) pretty much as soon as we have reported back to the Faculty. Still it has been interesting. I find these kinds of quality processes fascinating, and enjoyable - especially when the students are as engaging and articulate as they are at UWI - but challenging too. There's the whole rather forensic thing of asking the right questions of different people and working through papers to triangulate the data we're accumulating, and then there's the developmental processes of trying to articulate good practice and make viable recommendations that will support a department to make improvements where necessary. Plus sitting behind all that activity is the little voice in your own head (not so) quietly interrogating you on your and your own department's own practice!
Obviously the outcomes are confidential, but fortunately the incidental things that fall out of the process are always really interesting too. I know more about the Caribbean Single Market than I did before I started, have discussed with a Jamaican Professor of International Law whether Caribbean human rights law allows a margin of appreciation in the way of the ECHR (answer: it doesn't, or certainly not in those terms) and experienced lots of the ways in which globalization is shrinking our world. For example, meeting the head of the law school at the College of the Bahamas, who just happens to be a Warwick graduate; sitting at lunch up at the Cave Hill Campus discussing Obama's prospects for the Democrat US Presidential nomination, almost as if it were a matter of local politics; hearing law students and law teachers expressing concerns that almost exactly mirror those I hear in the UK: increasing numbers, worries about appropriate levels of skills (of the students, not the staff... mostly), worries about job prospects (students again)and the increasing work pressures on the academics too. Plus ca change....
Tuesday 11 March 2008
Legal Ethics
My former colleague John Flood is a visiting professor at the University of Miami this semester, where they have got him teaching the compulsory Professional Responsibility course. In a recent blog entry John has rightly described professional responsibility and ethics as the lost territory of English law. "Why is it", John asks, "that the business schools have been able to grasp the importance of teaching ethics, but law schools haven't?" A good question - and one I've commented on in response to John's entry, but I hope its worth expanding on that response here too.
John tends to point the finger at professional disinterest, and I'm sure that plays a part. The glib answer to the English denial of legal ethics has tended to be "no Watergate" - ie there has been no major crisis of professionalism that has forced the profession (and the academy) to seriously confront the problem. But I doubt that's the whole story.
Without trying to be exhaustive about it I think there is a range of factors. The English began codifying their professional conduct standards only about 70 years after the US - so the law of lawyering itself is still not as established a part of the culture. The greater separation of academic and vocational education I'm sure is also pretty key, and with it the tendency both of apparently vocational subjects, like ethics, to be marginalised in the academic curriculum, and of the vocational courses to lack the critical, engaging approach that we see in the best professional responsibility courses in US law schools. Consequently we also still lack any kind of critical mass of scholars. There are few discrete courses on legal ethics at the undergraduate stage in the UK, and it is notable that in the 10 years since Kim Economides and I started editing our journal Legal Ethics, the numbers of British scholars regularly contributing to the field has not significantly increased - indeed the gravitational pull on the journal has been southwards, with Australia now providing the greatest number of contributions. This is hardly surprising given that ethics is now far more established as a standard part of the Australian LLB.
Is this likely to change? In some respects its quite difficult to be optimistic. The Lord Chancellor's Advisory Committee pushed for a greater emphasis on ethics in its First Report published in 1996. This created a definite flurry of activity in the mid to late 1990s (of which we were part), but as I've just observed, it has not really been sustained. The Law Society subsequently set up an Ethics Education Forum to advise it specifically on ethics education in its work on the Training Framework Review, though quite what effect that body had is now hard to ascertain. Its regulatory successor, the Solicitors Regulation Authority, has recently commissioned Economides to review the terrain and make further recommendations. The outcome of that process remains to be seen, but whatever Kim Economides recommends is likely to meet opposition. There is no desire among academics to re-open negotiations on the Joint Statement that govern the professional requirements for the LLB, and some, perhaps understandable, reluctance to see these foundation subjects extended. A deepening of ethical training at the vocational and work-based learning stages of training might prove easier to implement, though that might still seem to be too little, too late. And to introduce more ethics at any stage will mean that teaching ad training institutions will have to address a significant knowledge gap.
At the same time these arguments are surely all wearing a little thin. Our approach is now significantly out of line with most of the major Common Law jurisdictions. We have confronted knowledge gaps before, and survived, and there is already a solid foundation of academic literature on which to build. Moreover there are numerous ways in which a more intellectually satisfying legal ethics (which could be informed by moral philosophy, axiology, the history and sociology of the professions, etc) could enrich both the academic curriculum and vocational training, and help us take the study of the legal profession itself more seriously. There is probably more pedagogic literature on teaching legal ethics than on most substantive parts of the curriculum. In short, the tools are all there. in this context, for legal educators to continue to deny ethics a serious place in the law curriculum starts to look like not just a failure of our ethical imagination, but an abnegation of our own professional responsibilities.
John tends to point the finger at professional disinterest, and I'm sure that plays a part. The glib answer to the English denial of legal ethics has tended to be "no Watergate" - ie there has been no major crisis of professionalism that has forced the profession (and the academy) to seriously confront the problem. But I doubt that's the whole story.
Without trying to be exhaustive about it I think there is a range of factors. The English began codifying their professional conduct standards only about 70 years after the US - so the law of lawyering itself is still not as established a part of the culture. The greater separation of academic and vocational education I'm sure is also pretty key, and with it the tendency both of apparently vocational subjects, like ethics, to be marginalised in the academic curriculum, and of the vocational courses to lack the critical, engaging approach that we see in the best professional responsibility courses in US law schools. Consequently we also still lack any kind of critical mass of scholars. There are few discrete courses on legal ethics at the undergraduate stage in the UK, and it is notable that in the 10 years since Kim Economides and I started editing our journal Legal Ethics, the numbers of British scholars regularly contributing to the field has not significantly increased - indeed the gravitational pull on the journal has been southwards, with Australia now providing the greatest number of contributions. This is hardly surprising given that ethics is now far more established as a standard part of the Australian LLB.
Is this likely to change? In some respects its quite difficult to be optimistic. The Lord Chancellor's Advisory Committee pushed for a greater emphasis on ethics in its First Report published in 1996. This created a definite flurry of activity in the mid to late 1990s (of which we were part), but as I've just observed, it has not really been sustained. The Law Society subsequently set up an Ethics Education Forum to advise it specifically on ethics education in its work on the Training Framework Review, though quite what effect that body had is now hard to ascertain. Its regulatory successor, the Solicitors Regulation Authority, has recently commissioned Economides to review the terrain and make further recommendations. The outcome of that process remains to be seen, but whatever Kim Economides recommends is likely to meet opposition. There is no desire among academics to re-open negotiations on the Joint Statement that govern the professional requirements for the LLB, and some, perhaps understandable, reluctance to see these foundation subjects extended. A deepening of ethical training at the vocational and work-based learning stages of training might prove easier to implement, though that might still seem to be too little, too late. And to introduce more ethics at any stage will mean that teaching ad training institutions will have to address a significant knowledge gap.
At the same time these arguments are surely all wearing a little thin. Our approach is now significantly out of line with most of the major Common Law jurisdictions. We have confronted knowledge gaps before, and survived, and there is already a solid foundation of academic literature on which to build. Moreover there are numerous ways in which a more intellectually satisfying legal ethics (which could be informed by moral philosophy, axiology, the history and sociology of the professions, etc) could enrich both the academic curriculum and vocational training, and help us take the study of the legal profession itself more seriously. There is probably more pedagogic literature on teaching legal ethics than on most substantive parts of the curriculum. In short, the tools are all there. in this context, for legal educators to continue to deny ethics a serious place in the law curriculum starts to look like not just a failure of our ethical imagination, but an abnegation of our own professional responsibilities.
Monday 25 February 2008
Shoot the REF?
A hotel in Dundee on a cold rainy night... the perfect opportunity to get the blog going again after too long a gap. I've been meaning to write for a while on a topic that has occupied a lot of my attention over the last 18 months, namely research assessment. With the submission of data for RAE 2008 at the end of last November, the UK academic community has finally consigned seven years of research activity into the hands of peer review panels. Even though the results of this latest exercise will not be known before December, the funding councils are already moving to establish the parameters for a replacement for the RAE - the proposed Research Excellence Framework (REF).
Universities were recently given the opportunity to respond to a consultation paper on the new REF. While most of the paper related to outline proposals to shift the balance of assessment in the so-called STEM (science, technology, engineering and medicine) subjects from peer review to 'metrics' (ie quantitative measures of performance, including citation counting), the paper also raised a number of questions about how the methodology for the arts, social sciences and humanities should be changed. Widespread concerns over the inappropriateness of bibliometrics for these disciplines appear, to a degree, to have been accepted and the thrust of the paper focuses on the questions of the kind of 'light touch' peer review that would be appropriate for these subjects, in conjunction with a possibly greater range of metric indicators than are used at present.
There must be relatively few academics and policy-makers who would not consider that the RAE has had its day. It has, I think, had some beneficial effects, but it has also distorted certain aspects of research activity, and been a massively resource intensive process. At Warwick alone the RAE 2008 has produced a university submission comprising, so our RAE team tell us, of 2296 pages. The hours put into the exercise by university and departmental research coordinators and administrators, by internal and external peer reviewers and various committee meetings must be staggering, creating I suspect a huge (lost) opportunity cost out of the whole exercise - and that's before we factor in the centralised costs to the funding councils of the assessment process itself.
So, what about the options for 2013?
The consultation has very much focused on metrics and especially bibliometrics as the primary methodology for the STEM disciplines. Even in this context, I think there are significant problems that need to be considered and it is hard to resist the view that bibliometrics are potentially a pretty bad idea, at least not without some considerable refinement. Let me give you just a couple of quite obvious concerns. First, there is already some debate about what bibliometrics actually measure. HEPI has argued quite forcefully that metrics actually assess research impact not research quality. If funding continues to be distributed on a quality basis, this must of itself beg the question whether metrics are the appropriate primary measure. Secondly, any kind of research assessment will effect what it seeks to measure - a good methodology will maximise 'beneficial' effects (however we define them) and hopefully minimise undesirable and inefficient distortions. Bibliometrics inevitably threaten to bring in a whole new range of distortions, for example, citation counting could simply encourage departments to use co-authoring strategically to coat-tail less highly-rated researchers on the work of research stars. Similarly, will bibliometrics actually reinforce the value of star researchers and transfer market in such stars? It could work more against new and early career researchers then the existing, qualitative approach of the RAE. Work takes time to have an impact, particularly with long publication lags in many journals. How will this be taken into account? This could be of considerable longer term significance in the context of the demographic “time bomb” most universities are facing, given aging staff profiles.
The proposals for a 'light touch' peer review for the social sciences and humanities are only broadly sketched out at this stage. Even so , there are some grounds for concern, not least given the likely speed with which changes will implemented. It is hard to see how the funding councils will reconcile the ‘light touch’ ideal with their stated commitment to continue with the process of quality profiling that was introduced for RAE 2008. (That is, where each publication is rated and the department is given a research profile, showing the percentage of work at 4*, 3*, 2*, and so on). The light touch might also do more to embed or reinforce the status quo and concentrate research funding in a way that has negative consequences for the sector as a whole and for the student learning experience. The combination of detailed peer review with a range of both quantitative and qualitative inputs has facilitated recognition and reward of smaller, emergent, research cultures within institutions - essentially post-92 universities and colleges - that have not had the cultural capital or resources in the past to develop a breadth and institutional depth of research excellence. It would be unfortunate if this capacity were to be lost. Moreover, the impact of a new methodology seems very hard to assess in diversity terms at this stage. Initially at least the new methodologies are also likely to create new, or at least different, demands on institutions, both in response to the proposed greater reliance on metrics and other quantitative measures, and in the need to manage two different REF processes.
I wonder if it really is about time we all agreed enough is enough, but that's not going to happen, is it? That's the problem with the audit juggernaut, once you set it going, its very hard to stop.
Universities were recently given the opportunity to respond to a consultation paper on the new REF. While most of the paper related to outline proposals to shift the balance of assessment in the so-called STEM (science, technology, engineering and medicine) subjects from peer review to 'metrics' (ie quantitative measures of performance, including citation counting), the paper also raised a number of questions about how the methodology for the arts, social sciences and humanities should be changed. Widespread concerns over the inappropriateness of bibliometrics for these disciplines appear, to a degree, to have been accepted and the thrust of the paper focuses on the questions of the kind of 'light touch' peer review that would be appropriate for these subjects, in conjunction with a possibly greater range of metric indicators than are used at present.
There must be relatively few academics and policy-makers who would not consider that the RAE has had its day. It has, I think, had some beneficial effects, but it has also distorted certain aspects of research activity, and been a massively resource intensive process. At Warwick alone the RAE 2008 has produced a university submission comprising, so our RAE team tell us, of 2296 pages. The hours put into the exercise by university and departmental research coordinators and administrators, by internal and external peer reviewers and various committee meetings must be staggering, creating I suspect a huge (lost) opportunity cost out of the whole exercise - and that's before we factor in the centralised costs to the funding councils of the assessment process itself.
So, what about the options for 2013?
The consultation has very much focused on metrics and especially bibliometrics as the primary methodology for the STEM disciplines. Even in this context, I think there are significant problems that need to be considered and it is hard to resist the view that bibliometrics are potentially a pretty bad idea, at least not without some considerable refinement. Let me give you just a couple of quite obvious concerns. First, there is already some debate about what bibliometrics actually measure. HEPI has argued quite forcefully that metrics actually assess research impact not research quality. If funding continues to be distributed on a quality basis, this must of itself beg the question whether metrics are the appropriate primary measure. Secondly, any kind of research assessment will effect what it seeks to measure - a good methodology will maximise 'beneficial' effects (however we define them) and hopefully minimise undesirable and inefficient distortions. Bibliometrics inevitably threaten to bring in a whole new range of distortions, for example, citation counting could simply encourage departments to use co-authoring strategically to coat-tail less highly-rated researchers on the work of research stars. Similarly, will bibliometrics actually reinforce the value of star researchers and transfer market in such stars? It could work more against new and early career researchers then the existing, qualitative approach of the RAE. Work takes time to have an impact, particularly with long publication lags in many journals. How will this be taken into account? This could be of considerable longer term significance in the context of the demographic “time bomb” most universities are facing, given aging staff profiles.
The proposals for a 'light touch' peer review for the social sciences and humanities are only broadly sketched out at this stage. Even so , there are some grounds for concern, not least given the likely speed with which changes will implemented. It is hard to see how the funding councils will reconcile the ‘light touch’ ideal with their stated commitment to continue with the process of quality profiling that was introduced for RAE 2008. (That is, where each publication is rated and the department is given a research profile, showing the percentage of work at 4*, 3*, 2*, and so on). The light touch might also do more to embed or reinforce the status quo and concentrate research funding in a way that has negative consequences for the sector as a whole and for the student learning experience. The combination of detailed peer review with a range of both quantitative and qualitative inputs has facilitated recognition and reward of smaller, emergent, research cultures within institutions - essentially post-92 universities and colleges - that have not had the cultural capital or resources in the past to develop a breadth and institutional depth of research excellence. It would be unfortunate if this capacity were to be lost. Moreover, the impact of a new methodology seems very hard to assess in diversity terms at this stage. Initially at least the new methodologies are also likely to create new, or at least different, demands on institutions, both in response to the proposed greater reliance on metrics and other quantitative measures, and in the need to manage two different REF processes.
I wonder if it really is about time we all agreed enough is enough, but that's not going to happen, is it? That's the problem with the audit juggernaut, once you set it going, its very hard to stop.
Subscribe to:
Posts (Atom)
