The everyday blog of Richard Bartle.

RSS feeds: v0.91; v1.0 (RDF); v2.0; Atom.

Previous entry. Next entry.

5:07pm on Monday, 23rd February, 2015:

Positive Feedback


Every few years, universities are obliged to contribute to the Research Excellence Framework, in which they describe how much research they have been doing and at what quality. This is an enormous undertaking, which eats through senior researchers' time like nothing else. The point of it is to find out which universities are doing the best research so they can be given more money than those which aren't doing the best research; put another way, it's the means by which research councils determine to give money to the same universities they've always given money to.

One of the criteria upon which universities are judged is "impact". This is a rating of the wider effect research has had on society. It's dealt with through a number of "impact cases", which demonstrate how individual pieces of work have affected the world in general. There's only a limited number of impact cases allowed per department (3 or 4 for one of our size), so being selected as an impact case is therefore quite prestigious as there's a lot riding on it.

For the REF exercise that concluded at the end of last year, my own research on player types was selected to be an impact case.

It was adjudged by the university to be very strong, because it had glowing letters of support from people such as Damion Schubert, lead designer on Star Wars: The Old Republic, explaining how it's foundational to how MMORPGs are designed. However, it was rated weak by the REF committee. The reason given was that although it was clear that player types is used extensively in MMORPGs, the economic impact of this was not demonstrated in the submission to the REF committee.

Now this isn't really fair. It's not fair, because I do actually have data on this and can demonstrate its economic impact through case studies. If the REF committee wanted more data, all it had to do is ask and then we could submit it. However, it didn't tell us that it wanted this data; it merely marked my impact as low on the basis that we hadn't supplied it.

No no no! This isn't an examination! In an examination, if the student doesn't answer the question, you can't go back to them and tell them to re-answer it: the point is to test their understanding. The REF is an information-gathering exercise: if you want information you don't have, you can simply ask for it — or at the very least we should be able to appeal and supply it. It's more like a research paper submission: in those, reviewers can provisionally accept your work but require you go back and make changes. Why on earth, after such a huge undertaking, would a committee not allow missing evidence to be supplied? They're meant to be judging research, not the ability to divine what will impress the committee members.

I dare say the committee members themselves aren't happy with this, mainly because no-one involved with the REF is happy with it. The rules are decreed from on high, however, so we have to live with them.

They could have got the same ranking by simply surveying a couple of thousand parents and asking them where they sent their children to university, weighted by household income...

Latest entries.

Archived entries.

About this blog.

Copyright © 2015 Richard Bartle (richard@mud.co.uk).