British Medical Association Medical Academic Staff Committee response to the Reform of Higher Education Research Assessment and Funding
October 2006
Answers to specific questions
Introduction
Within the Association, the Medical Academic Staff Committee (MASC) has responsibility for acting on all matters of concern to medically qualified personnel holding employment contracts at higher education institutions, medical schools and other institutions that are engaged with medical research such as the medical research charities. The MASC has been actively involved in supporting two key initiatives that will influence the future of academic medicine:
- The introduction of an integrated academic training pathway as outlined in the 'Medically and dentally qualified academic staff – recommendations for training the researchers of the future'. This will establish 250 clinical fellows and 100 clinical lecturers in the first year of operation, all of whom will need to be supervised.
- The new national health research strategy 'Best Research for Best Health', which paves the way for the establishment of the National Institute for Health Research (NIHR) to support the provision of health research, especially through recognition and membership of the NIHR Faculty. MASC also welcomes the strategy’s suggested merging of the NHS Research and Development and Medical Research Council (MRC) budgets, as long as the overall level of funding for research does not fall.
We urge the Department of Education and Skills to take note of these developments and ensure that the RAE (Research Assessment Exercise) 2008 does not militate against these initiatives. In particular:
- Future research assessment should be complementary to the work of the NIHR and the revised Research and Development funding arrangements
- NHS based Faculty researchers (including those in the NIHR) will not be subjected to the RAE. As such, mechanisms need to be devised to prevent clinical academic staff being disadvantaged e.g. in career advancement and salary in comparison to their NHS counterparts
- NHS funding should be included in any metrics based system of research assessment in order to fairly evaluate researchers’ contributions to the advancement of knowledge
- A metrics based system should be no worse than the status quo and should not undermine the progress made in respect of improvements to the RAE methodology over the last ten years (although we note the argument for retaining the RAE 2008 has not been made).
- Career breaks or less than full time working needs to be taken into account into the 2008 RAE process if clinical academics, women or people with disabilities or illness are not to be disadvantaged. For example clinical medical academics work full time but 50% of their contracted hours are work for the NHS, not University work.
- Access to clinical research subjects should be safeguarded, especially in light of the increasing role of the independent sector in the delivery of healthcare
- Measured RAE outcomes in clinical medicine should reflect the value of training clinical researchers for the NHS, and research involvement in NHS health care, education and management
- The present RAE disengages research from teaching because they are assessed in different ways. Active attempts at integrating research and teaching, through the RAE, should be made.
- The loss in some academic clinical specialties as a result of previous RAE assessments should be reversed in order to provide comprehensive education in clinical medicine
- Where Universities have successfully improved their performance since the last RAE but their research outputs do not yet reach the top 10%, this progress should be identified and rewarded in the RAE process
1. Which, if any, of the RAE 2008 panels might adopt a greater or wholly metrics-based approach?
1.1 The measures of assessment for the RAE available until now have been criticised for their narrow focus and a tendency to reward laboratory based projects instead of human studies, and hence an overall failure to adequately measure the contribution of medical academics to clinical research. The use of metrics (especially research quality, bibliometric data and products from applied research) may provide an aide to judgement. However metrics they should not comprise all of the approach adopted for the two main RAE panels connected to medicine, as they have not been tested.
1.2 Research income alone does not illustrate the measures of output and does not take into consideration the cost differential in running projects in some specialties compared to others, for example by virtue of the cost of equipment.
1.3 The approach adopted in 2008 needs to support and value the full range of medical research in all disciplines, including applied/translational research – we are concerned that these may not be adequately valued by the metrics provided.
1.4 Local factors may also need consideration. For example, a new medical school may perform poorly in a wholly-metrics based system if the academics were recent appointments that focused primarily on expanding medical education, as well as research.
2. Have we identified all the important metrics? Bearing in mind the need to avoid increasing the overall burden of data collection on institutions, are there other indicators that we should consider?
2.1 NHS funding should be included in any metrics based system of research assessment in order to fairly evaluate researchers’ contributions to the advancement of knowledge. In addition, there are also time factors and different commitments of clinical academics that should be taken into consideration, e.g. the fact that clinical academics’ time is divided between University and NHS activities. The current RAE militates against medical academics with high clinical commitments that are likely (as a result) to have limited research output. This needs to be addressed as the balance of University/NHS activity forms the survival of many craft based academic disciplines such as surgery and obstetrics and gynaecology.
2.2 Metrics should measure innovation in clinical research. In clinical research, innovation or change in practise is usually the result of several – as many as four or five – complementary studies. The new system for funding must therefore give appropriate credit for the contribution of each piece of research made to the advancement of clinical practice. In addition, a “health improvement/NHS output” factor should be considered. Changes in clinical practise will include a shift towards the use of research based evidence as a means of securing improved clinical outcome and consistency of approach in the clinical environment. The effect of research on patient care may take years to come to fruition and depends on more factors than publication of research and levels of income derived from research grants. The significance of research that is directly applicable to clinical practise must therefore be recognised by the RAE.
2.3 The metric related to “income derived from research income” needs to consider funding from the NHS. This is of particular importance given the proposal to merge health research funding into a single ring-fenced budget from NHS Research and Development and the Medical Research Council funds.
2.4 The metric related to research volume does not appear to include teaching metrics such as the number of classes/tutorials, curriculum design or any overall measurement of teaching/ mentoring. Since this is directly applicable to training in clinical research that may not be identifiable with University students (e.g. academic clinical fellows introduced as part of the new integrated academic training pathway), a degree of flexibility in the RAE system is advised in the assessment of their supervisors, in order to compensate for time spent training these new clinical researchers.
2.5 In relation to research quality, esteem indicators should be considered. Clinical academics have always provided significant contributions to healthcare strategies, training and education and setting standards for healthcare provision, for example through contributions to professional societies, Royal Colleges and national and international forums.
2.6 To address the multiple facets of the clinical academic workload, consideration should be given to the use of a more balanced scorecard which recognises the medical academic contribution to the full range of their activities. This should include a wider appreciation of the relationship between teaching and research, for example through the authorship and editing of books, in addition to acknowledgement of clinical contribution.
3. Which of the alternative models described in this chapter do you consider to be the most suitable for STEM subjects? Are there alternative models or refinements of these models that you would want to propose?
3.3 In the absence of hard evidence on the impact of each model on medicine, we find ourselves unable to actively support any of the models proposed.
3.4 Each of the models, and the assumptions contained therein, should be tested to enable a decision on which model is most appropriate for medicine as a discipline that generates output relevant to clinical care and clinical academic research careers (as distinct from ‘course education’). The models should be tested out on exemplar CVs of medical academics in different specialties – we would be willing to provide sample CVs should you so wish.
3.5 None of the models work sufficiently well to justify omitting detailed peer assessment which exists at present.
4. What, in your view, would be an appropriate and workable basis for assessing and funding research in non-STEM subjects?
4.1 Given that our interests relate to academic medicine, we have restricted our comments in this consultation to STEM (sciences, technology, engineering, mathematics and medical) subjects and have no comments on this question.
5. What are the possible undesirable behavioural consequences of the different models and how might the effects be mitigated
5.1 We are of the opinion that the RAE has contributed to the further development of excellent basic science research in the UK but it has not sufficiently valued applied/translational research, seen for example, in the failure to support clinical trials or secondary research such as meta-analysis or Cochrane reviews. This has contributed to a divergence in research outcomes expected of clinical academic staff by the NHS and universities. For this reason and because of the recent policy initiatives in academic medicine, the 2008 RAE round should actively support a full range of medical research in all disciplines as well as actively support academic medicine as a career option.
5.2 This last point is particularly important as there is a new career structure developing within academic medicine. Previous RAEs have worked against the employment of training posts in Universities. In addition there is the potential that systemic disadvantage will be further entrenched unless mechanisms are devised to prevent clinical academic staff being at a disadvantage to their NHS counterparts, who will not be subject RAE assessments.
5.3 Depending on the model adopted, smaller specialties or those that are not likely to generate significant external funding but who provide important building blocks for future advances in clinical practice, may be disadvantaged by the new arrangements. Adjustments should be made for the field of endeavour that may not receive grants from research councils or produce patents.
5.4 For example, even the most outstanding research in some smaller specialties, such as geriatrics or paediatrics/child health will not attract the same level of grant income as a clinical pharmacologist or geneticist, despite delivering, in the case of geriatrics, a huge proportion of the acute medical take across the UK, and, for paediatrics small, but significant breakthroughs providing very valuable input to our understanding of more general public health issues. In another specialty, the future of academic gynaecology is threatened by the channelling of funds primarily into cancer related research, with little grant success for research that focuses on women’s health more generally. It is likely that this approach has had a detrimental effect on a major component of the public health.
5.5 There is also the prospect that metrics based on heavily research income alone would reduce the viability of early funding as there will be no benefit to the early researchers or University. The two examples below illustrate these points:
Example 1:
In 1995 using £10,000 from Diabetes UK, we identified the second imprinted gene defect ever described - that of transient neonatal diabetes (frequency 2 cases per year in UK). This breakthrough was published in Nature Genetics (Impact factor 40) with two subsequent papers in Diabetes (IF - approx 10) and Human Molecular Genetics) likewise about 10. So each IF was at a cost of £166. This work has led on to future grants worth much more, but on the basis of the original work, prior to attaining more funds, I would have been out on my ear with such a paltry sum initially although its impact has been huge (196 citations for the three papers).
Example 2:
I have a family with an inborn error of metabolism only described in three or so families worldwide. A grant application at this stage is hardly likely to attract MRC, Wellcome funding. However, they are different to the others in their lack of neurological deterioration. Fascinatingly, the deterioration in the other children is exactly the same as in Alhzeimer's disease and these children have escaped it. By understanding the fundamental basis of their metabolism we may cast light on how to treat Alhzeimer's. No cash so far and the work has taken 18-24 months on bits and pieces of money, but if we are right - the paper will be in Brain (IF enormous) and the research of fundamental importance.
My point is that if the RAE turns into a "the more money you get in, the better your research is" these quirks of nature will no longer be viable for studying as even if the impact is huge, the chances of early funding is minimal and there will be no benefit to the early researchers or the University. There may be major funding for subsequent work but that may often go to epidemiology or other researchers not associated with the discovery team. Without the primary findings (which are unlikely to get much money as will always be high risk hypotheses), no follow on work could ensue. It’s a little like meta-analysis. This is always of relevance to public health/clinical therapy but someone had to do all the small RCTs to enable a meta-analysis to be done. Intuitive and important observational work will die out and with it the chance to identify very important observations of more general value, the impact of which is much broader than the small (and un-fundable) initial group. It will kill much of the most productive paediatric research as universities will not view people in this area as being of any value if only input (i.e. grants) and not output (i.e. value of the study) is not taken into account.
5.6 Neither this nor previous RAEs take into consideration career breaks. The lack of women in senior academic positions continues to be a significant problem. Only 11% of female medical academics are professors (Clinical Academic Staffing Numbers in UK medical and dental schools: a data update by the Council of Heads of Medical Schools, June 2006) and there are several medical schools with no female professors at all. Women comprise the majority of future recruits to medical research and teaching with over 60% of medical students currently being women. The skills available to Universities are thus diminished because they are failing to attract and retain women in medical research. Any disadvantages to women that are perpetuated by the RAE or by exclusion from the RAE (since this has been identified to be more common for women than men) should be considered and eliminated as far as possible.
5.7 It is critical that medical academics continue to have access to subjects for clinical research. Consideration ought also to be given to barriers to clinical research raised by the proliferation of health care providers in the NHS and the potential effect on loss of access to patients and other subjects/population groups by medical researchers.
6. In principle, do you believe that a metrics-based approach for assessment or funding can be used across all institutions?
6.1 Yes, for medicine, as long as the modelling is appropriate (e.g. soundly based on professional worked examples) and there is adequate recognition of the differences and potential undesirable effects described in our response to question 5.
6.2 The pressures on medical academics following the expansion of medical school places and a 25% reduction in clinical academic staffing numbers should be acknowledged. At the same time, consideration may need to be given to the natural priorities of newly established medical schools toward expanding education, as well as research.
7. Should the funding bodies receive and consider institutions' research plans as part of the assessment process
7.1 Although we have no objection to this suggestion in principle, our concerns are that it could result in a further layer of information gathering, pose issues for intellectual property and impose inappropriate and artificial timeframes on research priorities.
8. How important do you feel it is for there to continue to be an independent assessment of UK higher education research quality for benchmarking purposes? Are there other ways in which this could be accomplished?
8.1 There is much merit in formally assessing the quality of medical research, both in terms of its value to the UK economy and its international standing. The RAE has repeatedly, grossly underestimated the value of UK medical research, given that health research has been demonstrated as providing significant economic advantage See Academy of Medical Sciences report 'Medical Research Assessing the Benefits to Society' and that the UK is second only to the US in terms of international medical research output.