Shoot the REF?

A hotel in Dundee on a cold rainy night... the perfect opportunity to get the blog going again after too long a gap. I've been meaning to write for a while on a topic that has occupied a lot of my attention over the last 18 months, namely research assessment. With the submission of data for RAE 2008 at the end of last November, the UK academic community has finally consigned seven years of research activity into the hands of peer review panels. Even though the results of this latest exercise will not be known before December, the funding councils are already moving to establish the parameters for a replacement for the RAE - the proposed Research Excellence Framework (REF).

Universities were recently given the opportunity to respond to a consultation paper on the new REF. While most of the paper related to outline proposals to shift the balance of assessment in the so-called STEM (science, technology, engineering and medicine) subjects from peer review to 'metrics' (ie quantitative measures of performance, including citation counting), the paper also raised a number of questions about how the methodology for the arts, social sciences and humanities should be changed. Widespread concerns over the inappropriateness of bibliometrics for these disciplines appear, to a degree, to have been accepted and the thrust of the paper focuses on the questions of the kind of 'light touch' peer review that would be appropriate for these subjects, in conjunction with a possibly greater range of metric indicators than are used at present.

There must be relatively few academics and policy-makers who would not consider that the RAE has had its day. It has, I think, had some beneficial effects, but it has also distorted certain aspects of research activity, and been a massively resource intensive process. At Warwick alone the RAE 2008 has produced a university submission comprising, so our RAE team tell us, of 2296 pages. The hours put into the exercise by university and departmental research coordinators and administrators, by internal and external peer reviewers and various committee meetings must be staggering, creating I suspect a huge (lost) opportunity cost out of the whole exercise - and that's before we factor in the centralised costs to the funding councils of the assessment process itself.

So, what about the options for 2013?

The consultation has very much focused on metrics and especially bibliometrics as the primary methodology for the STEM disciplines. Even in this context, I think there are significant problems that need to be considered and it is hard to resist the view that bibliometrics are potentially a pretty bad idea, at least not without some considerable refinement. Let me give you just a couple of quite obvious concerns. First, there is already some debate about what bibliometrics actually measure. HEPI has argued quite forcefully that metrics actually assess research impact not research quality. If funding continues to be distributed on a quality basis, this must of itself beg the question whether metrics are the appropriate primary measure. Secondly, any kind of research assessment will effect what it seeks to measure - a good methodology will maximise 'beneficial' effects (however we define them) and hopefully minimise undesirable and inefficient distortions. Bibliometrics inevitably threaten to bring in a whole new range of distortions, for example, citation counting could simply encourage departments to use co-authoring strategically to coat-tail less highly-rated researchers on the work of research stars. Similarly, will bibliometrics actually reinforce the value of star researchers and transfer market in such stars? It could work more against new and early career researchers then the existing, qualitative approach of the RAE. Work takes time to have an impact, particularly with long publication lags in many journals. How will this be taken into account? This could be of considerable longer term significance in the context of the demographic “time bomb” most universities are facing, given aging staff profiles.

The proposals for a 'light touch' peer review for the social sciences and humanities are only broadly sketched out at this stage. Even so , there are some grounds for concern, not least given the likely speed with which changes will implemented. It is hard to see how the funding councils will reconcile the ‘light touch’ ideal with their stated commitment to continue with the process of quality profiling that was introduced for RAE 2008. (That is, where each publication is rated and the department is given a research profile, showing the percentage of work at 4*, 3*, 2*, and so on). The light touch might also do more to embed or reinforce the status quo and concentrate research funding in a way that has negative consequences for the sector as a whole and for the student learning experience. The combination of detailed peer review with a range of both quantitative and qualitative inputs has facilitated recognition and reward of smaller, emergent, research cultures within institutions - essentially post-92 universities and colleges - that have not had the cultural capital or resources in the past to develop a breadth and institutional depth of research excellence. It would be unfortunate if this capacity were to be lost. Moreover, the impact of a new methodology seems very hard to assess in diversity terms at this stage. Initially at least the new methodologies are also likely to create new, or at least different, demands on institutions, both in response to the proposed greater reliance on metrics and other quantitative measures, and in the need to manage two different REF processes.

I wonder if it really is about time we all agreed enough is enough, but that's not going to happen, is it? That's the problem with the audit juggernaut, once you set it going, its very hard to stop.

We're back!

Moira and I got back from Kenya on Sunday after an incredible challenge. This was the first visit to Africa for both of us, and the first time we had ever attempted a long distance cycle ride. It was tough. Believe me, we have earned every penny of your sponsorship money! On the last day of the ride we visited the project at Mumia that the International Childcare Trust is sponsoring with a local action group. The aim of the project is to build a drop-in centre and night shelter for street children . The centre will also provide adult education and other facilities to work with local women, including widows who are being supported to take on fostering projects with AIDS orphans and others, and families in crisis. The greeting we received (pictured) was extraordinary and it was an intensely moving experience. The first floor of the centre is virtually complete, funded by some of the money raised out of last year's Cycle Cambodia challenge, and the building was opened by ICT's chairman during our visit. The Cycle Kenya team were a great group of individuals, and even though it was tough, we had a real ball. The memories from this trip - and the visit to the Mumia project in particular - will stay with us for a very long time indeed.

A big THANK YOU again to everyone who has supported us so far. Our Just Giving site is still operational, so please, if you haven't sponsored us already and would like to, there is still time to do so. We are also in the process of putting a day-by-day account of the challenge up on Moira's blog, linked here

Ere we go...!

Its reached the fateful day. We leave Warwick this afternoon for an overnight flight to Nairobi and 411kms of cycling. Feeling more than a little apprehensive about a couple of the days - days two and four - which involve over 100 kms, which looks to be mostly upwards, and then 88kms with a big 25 km ascent in the afternoon. We've done some good mileage in training, but not enough hills I fear.....

But there's no turning back - that's why they call it a challenge! Thanks again to all out fantastic supporters who have helped us raise nearly £5000 already, and our major fund-raising events still have to take place. The photo here relates to one of those - a children's charity cycle ride in Victoria Park, Leamington which is happening on the Sunday after we get back - 25th November from 10.30 am. Big thanks to Roger and hirecentres.com for sponsoring this event on our behalf and to Stephen, our Police community support officer (pictured here with some of our first participants) who has been a tremendous help with all the detailed arrangements. Come and support us if you're in the area!

Until our return.......

The numbers' game

I got copied in yesterday to an interesting puzzle that Shakeel - our Information Manager at UKCLE - had uncovered for Gary Slapper at the Open University. Recent UCAS statistics apparently indicate that just under 90% of those applying for law are getting in. So what, Gary asked, has happened, and what about that still widely held belief that law is a massively oversubscribed subject?

I don't actually know the answer, but I have no particular reason to doubt the data. The following I think is a reasonable supposition based on what I do know about the admissions system over the last 10-15 years (the older stats quoted below pretty much all come from UCAS, courtesy of the study on "Access to and Participation in Undergraduate Legal Education" (Faculty of Law Working Paper No. 2, UWE Bristol, 1996) that Vera Bermingham and I undertook for the Lord Chancellor's Advisory Committee in 1995.

1) The extent to which law was oversubscribed was probably always a bit mythical, reflecting the way students applied to the old UCCA/PCAS system pre-1993-4. Under that system students could apply through UCCA to six universities (I think) and then separately to about the same number of polytechnics (now post-92 universities) via PCAS. There were a lot of insurance applications across the two schemes as students tried to protect their position - especially in case they did worse than they expected. This would have inflated the admissions figures. In the last year of the old scheme UCCA and PCAS together handled 31,760 applications. In the first year of the combined UCAS scheme (for 1994 entry) UCAS handled 20,988 - that doesn't mean almost 11,000 less students applied to read law, it just means that the number of applications became a truer reflection of the real number of applicants across the system as a whole.

2) Since then we know that institutions have recruited significantly more students to LLB coursess, and that the number and range of courses has also expanded. In 1994 about 8,000 students were admitted to qualifying law degrees. It appears that, subsequently, that number has more than doubled.

3) In 1994 the overall ratio of UCAS applicants to admitted students was about 2.5:1 - itself way below the old UCAS/PCAS average which I believe was nearer to 12 or even 15:1. Interestingly, the number of applicants in 1994 and in the latest figures are strikingly similar. I suspect the figure may be relatively constant in the intervening period. If that is correct, then, logically, if the number of applicants remains constant, and the number of places continues to grow, we would reach the position we seem to have now: near parity between applicants and places.

That of course represents an interesting challenge, particularly as yet more law degrees are due to come on stream in the next year or so, eg, at York and Winchester, and British universities seem to be doing less well in the highly competitive market for international students. While the elite law schools may be protected by their status and popularity from the worst effects of these trends, and will, I am sure, continue to be oversubscribed, the lives of admissions tutors in some law schools may be about to become even more interesting. How long can you keep fishing at a nearly empty pool?

"Transforming legal education"

This modest little title belongs to an excellent book just published (Ashgate, 2007) by Prof Paul Maharg of Glasgow Graduate School of Law. Paul is one of the most innovative thinkers around on legal education and his work at Strathclyde on creating transactional learning environments (teaching through simulated legal transactions) is really world class. This book reflects on a lot of that experience, but does much more in terms of developing an alternative theory of legal education pedagogy - which incidently involves linking the construction of the legal realist curriculum at Columbia in the 1920s, ethics education at Edinburgh University in the eighteenth century; and the practices of the mediaeval "Glossators" of Roman Law!

Perhaps even more interesting is the experiment that goes with the book. Paul and a number of colleagues have just launched a wiki which will form the basis of a community of practice for what Paul calls the "Transforming Initiative". I've just signed up to it, though blowed if I know quite what I'm going to do with it - yet! If you're interested in the project, go to Paul's Transforming Legal Education website - linked here

Disabling education?

Earlier today I was reading through a paper by Jenkins, Healey and Zetter on the HEA website entitled “Linking teaching and research in disciplines and departments” (published in April 2007). I was struck and somewhat depressed by the fact that none of the examples they cited were of developments in UK law schools. Of course I can’t say for sure why that was (I haven’t asked them for one thing), but when I also tried to track down examples of research-based teaching on the UKCLE website, I found remarkably little there either. Of course, the good news might be that everyone in law schools has sussed out the teaching-research nexus, and its so second nature now that nobody bothers to talk about it. But somehow I don’t think so.

Before going any further with this potential rant, I ought to clarify an earlier distinction that Alan Jenkins and Mick Healey made between research-led and research-based teaching, because I think it is fairly crucial. Research-led teaching, putting it crudely, reflects the student-as-audience paradigm of teaching, where they sit back and admire our erudition as we introduce them to our latest piece of cutting-edge research. Research-based teaching, on the other hand is a form of inquiry-based learning which treats the student as a (joint) participant in the research (and hence the learning) process. My betting is that, beyond the still fairly ubiquitous and highly variable experience of writing a final year dissertation, most law undergraduates experience at least some research-led teaching but very little genuinely research-based learning.

If that’s so, then I suggest we are missing a real opportunity and perpetuating a form of what (adapting Ivan Illych’s notion of disabling professions) I am inclined to call disabling education – an education that actively disempowers our students. It manifests itself in the attitude that we can’t or shouldn’t expect too much originality or creativity of poor little undergraduates. A variation of the same attitude seems to emerge with monotonous regularity in discussions about clinical legal education, when non-participants express doubts/shock/total horror at the idea of undergraduates actually being allowed to give legal advice to real people.

For us as academics to keep students at arms length from the experience of research seems to involve

(a) an impressive example of discrepant reasoning – speaking personally it was precisely the experience of doing a research project and being encouraged to write as an undergraduate that led me to start thinking about academia as a career, and I’m sure I’m not alone in that, or
(b) pure humbug – do we really think that the skills we have developed as researchers are so advanced and so specialised that they are wholly beyond the ken of our students?

and, from the point of view of the future of the academic profession, it is surely about as smart a survival strategy as being in the front row of the charge of the Light Brigade.

I do know that there are some examples of research-based learning happening in our law schools, and I’d love to hear from other colleagues who are doing this sort of work. I know that the people leading these courses find them engaging, motivating and exciting for themselves and their students, and that developing the research-teaching nexus has direct benefits for their own research, as well as for their students’ learning. In a recent e-mail exchange, Ben Pontin, who has been doing some very interesting research-based work with his Environmental Law module at UWE Bristol, observed “Students are brilliant at gathering 'raw material' - trawling historical archives (newpapers, official records, family histories) for snippets of information; detective work. Also, there was a geography dimension to the research, and students enjoyed designing field trips to the sites of leading nuisance actions, to identify whether any trace of what claimants were seeking to protect remains today. However, where students needed input from an academic was in interpreting the significance of empirical findings. Great at treating law as a material object (a field or stately home protected against a polluting factory or sewage works), students struggled with law's ideas!” I think that last observation makes for a particularly interesting insight and suggests a useful point around which we could reconstruct the role of the teacher in a more research-based legal education process.

TGI Friday

Just back from a day in London at a one day conference organised by three enterprising PhD students at Queen Mary. The theme was "Legal academics: Spectators or Players?" and the organisers had got a really interesting group of people together to discuss the role of legal academics and their relationship with legal practice. It could all have been a bit of a non-event, but as it was it was one of those days where the synergies worked and it sparked all sorts of interesting research and policy questions, much of it around the role of legal education in this relationship. I hope it leads on to some real outputs; it would be a shame to waste all that energy!

Its just as well it was an energising day as the batteries here are running pretty low. Its been a busy summer at the end of a long year. I've been working hard on about three different writing projects, as well as trying to keep on top of my work for the Centre and the RAE, so its been pretty much six day weeks since May, with just a week off for hols in August. Not what most people think us academics do with our time, I suspect. Even so there have been good bits. the Warwick RAE is taking shape, I am getting through the writing - if slowly - and UKCLE has just had its annual Advisory Board, and got some great feedback and support from the Board, which is so valuable to the team. Now its all about gearing up for the new academic year which starts in a week's time; I'm teaching a first year undergraduate module this term, for the first time in a long time, and have also taken over the leadership of the LLM Legal Education, and there's still a fair amount to do on both those fronts. But tomorrow, no work!