At KTH (the Royal Institute of Technology) in Stockholm, 2.5 percent of government research funding is allocated to university activities based on the impact factor of the journals in which articles have been published. But, the Thomson Reuters JIF rating is not used without modification. The bibliometricians at KTH calculate an impact factor for every journal.

The use of impact factors in the research community

Svenska 2016-09-29

Frequently used but fiercely criticised. Curie has looked into how some Swedish research funders and universities use the disputed Journal Impact Factor, a measure that can currently affect who is granted research funds and which journals researchers choose to publish in.

The journal impact factor, JIF, is a way of ranking scientific journals on the basis of how often their articles are cited. This dimension (see definition in the information box) is calculated once every year by Thomson Reuters and is based on the citations in their database, Web of Science.

The measure emerged in the beginning of the 1960s and was originally conceived as a way of comparing scientific journals, irrespective of their size. Librarians quickly accepted JIF as an aid to managing journal stocks, but it now has a much wider area of use and greater influence – not without criticism (read more about JIF in this article in Curie).

Own method

It is not unusual in Sweden that JIF is included in the allocation model used for distributing government research funds at a higher education institution. At KTH (the Royal Institute of Technology) in Stockholm, 2.5% of government research funding is allocated to university activities based on the impact factor of the journals in which articles have been published.

But, as Per-Anders Östling, investigator at KTH Central Administration points out, the Thomson Reuters JIF rating is not used without modification.

“Since JIF does not take into account some subjects being cited far more than others, it is not a fair way of distributing funds. We have created our own method of using the measure in a more just way,” he says.

The bibliometricians at KTH calculate an impact factor for every journal. Described in brief, every journal’s citations in a given year are compared to the average for journals in the same subject area, giving a field-standardised journal impact rating.

Articles and reviews are counted, and the impact factor for a given year is calculated on publications in the previous five years. The unit’s measurement is the sum of impact factors for journals where an article has been published, calculated as an average over three years.

The publication indicator was introduced two years ago. The idea behind it was to give researchers an incentive to publish in scientific journals of high quality and increase the quality of research in the long-term.

Arne Johansson

Arne Johansson

“In financial terms, is so small that it disappears in the noise. But we believe that together with our annual bibliometric follow-up, it encourages researchers to dare to take on greater challenges when they publish. They may try to reach greater heights scientifically, publish fewer articles but achieve more quality,” says Arne Johansson, vice-president for research at KTH.

Says nothing about individual articles

The impact factor can be seen as a measure of a journal’s effect, based on how often its previous articles have been cited during a certain time period. It does not indicate how many times each article in the journal will be cited. The citation frequency for individual articles in a journal have a much wider variation.

A small number of widely cited articles have a large effect on the impact factor, while the vast majority of articles are below the average.

Even in journals with a high impact factor, a large proportion of the articles have few or no citations at all, and many articles published in journals with a low impact factor are often cited.

This phenomenon is not a problem for the KTH distribution model, says Arne Johansson.

“Certainly, JIF says nothing about how well-cited an individual article is. But on average, at a group level there is a strong correlation between citations and publishing in journals with high impact factor. On average, their articles are more often cited. We also recommend that researchers publish in good journals in their own field. If an article is published in journals with a high JIF outside their own field, it will probably not be cited so often.”

The KTH measurement of a journal’s impact is also used in a bibliometric check that is carried out once a year. It is calculated for the HEI as a whole, for schools and departments and even individuals. The personal measurement is accessible for individuals and their managers.

“At the individual level, the indicator is mostly for researchers to follow their development over time. Salary proposals are taken up in appraisal talks, where many other factors are included,” says Arne Johansson.

At KTH, the description of the measure says that it should be used with caution when interpreting values based on few publications. When employing people, the impact factor has a minor role, according to Arne Johansson, and is just one part of a deeper process in which experts examine the applicant’s research, plans and merits.

Works well at the department level

At KI, Karolinska Institutet, JIF is used in combination with two other bibliometric indicators for the distribution of government research funding to the university’s departments.

Anders Gustafsson

Anders Gustafsson

“JIF works well as one of several quality measures at the department level. The departments are large enough to give meaningful results. The idea is that the different indicators encourage quality. In general, it is more difficult and requires more work to get published in journals with a high JIF compared with weaker journals,” says Anders Gustafsson, Dean of Research at Karolinska Institutet.

He is aware that differences in publication patterns make JIF unsuitable for comparisons between different scientific areas and says that the distribution model may be unfair in fields where journals generally have lower impact factor, such as clinical areas and nursing science. For these reasons, KI is considering field-normalising part of the JIF measurement, i.e. to relate the impact factor to the average value of a journal’s field.

JIF is not formally used for employment or salaries at KI, says Anders Gustafsson.

“But you can’t escape the fact that when someone has many publications in prestigious journals, it is impressive.”

Does not use JIF

Sweden’s largest private research funder, the Knut and Alice Wallenberg Foundation, does not use JIF when they evaluate projects says Göran Sandberg, the Foundation’s executive member.

“We encourage our evaluators to focus on the content of the project, its feasibility and audacity. They must of course assess applicants’ scientific expertise and if they look at impact factors we can’t prevent that, but we don’t give the evaluators any such information.”

In formal terms, JIF is not used by the Swedish Research Council in its assessment of applications for research funds, explains Staffan Karlsson, analyst for the Council. The researchers who review applications assess them on the basis of the project the applicant wishes to carry out, as well as their qualifications.

To help them in this process, evaluators are given a battery of grading criteria, but JIF is not included in these. Within the subject area of science and technology, applications must of course include information on how many citations articles have had, but not about the journal’s impact factor.

Up to the evaluators

Ultimately though, says Staffan Karlsson, it is up to the evaluators how they view the merits in an application. When the Swedish Research Council analysed whom they granted funding to, it emerged that the sum of the JIF for the applicant’s articles varied with the person’s likelihood of being given a grant. The higher the total JIF score, the greater was the chance of receiving funds.

“In the fields of medicine and science, applicants’ JIF scores are positively correlated to funding received – in fact more so than anything else we could measure, including the age, title, number of years after a doctoral degree or the articles’ citations.”

It is not possible to conclude from this alone that evaluators judge applications on the basis of JIF, which is after all a measure designed to rank scientific journals and not to assess the quality of individual articles or researchers. On the other hand, Staffan Karlsson would not be surprised if articles in journals with high JIF ratings would influence an assessment. In many areas of science it is considered important to publish in prestigious journals and it would be rather strange if evaluators were not impressed by articles in such journals, he states.

“In my sphere, as a bibliometricist, JIF is deeply questioned and there are a number of methodological problems which make the factor rather unfortunate to use in the evaluation of research. But there is no doubt that the journals with high JIF ratings are difficult to get published in.”

May be used as a shortcut

Evaluators are often under time pressure and there is a risk that JIF is used as a short cut, according to Staffan Karlsson.

“It is easy to look at how “fine” the journals are in a publication list, instead of forming an opinion on each individual article.

There is then a risk that articles are overvalued if they are published in a journal with high impact factor, and other articles are overlooked if they are published in lower-ranked journals.

There may be very important articles in a mediocre journal. If an evaluator looks too much at JIF scores, there is a risk that it will not help the assessment,” says Staffan Karlsson.

Read more in Curie: Factors that influences research careers

Text: Sara Nilsson
Photo: Jann Lipka

Ta del av information om behandlingen av dina personuppgifter