11 universities opt out of Maclean's survey

[winner2000]

Funny how a good number of those schools rank pretty poorly on the list...


11 universities opt out of Maclean's survey

CAROLINE ALPHONSO

Globe and Mail Update

Eleven of Canada's universities are refusing to participate in the influential Maclean's university rankings, saying they find the magazine's survey methodology to be "oversimplified" and "arbitrary."

In a letter sent Monday to Maclean's, the coalition said they will not provide any data for the magazine's popular, yet controversial, annual survey of universities, coming out this fall.

The letter was signed by the presidents of the University of British Columbia, Simon Fraser University, University of Alberta, University of Calgary, University of Lethbridge, University of Manitoba, University of Toronto, McMaster University, University of Ottawa, University of Montreal and Dalhousie University.

"In various ways and for some years, many institutional spokespersons have expressed considerable reservations about the methodology used in the Maclean's university survey and the validity of some of the measures used," the presidents wrote.

"Thus far, these serious concerns have gone largely unaddressed, and there is still no evidence that Maclean's intends to respond to them."

The Maclean's university ranking, a competitor to The Globe and Mail's University Report Card, has been the subject of much debate. Universities have objected to being scored and ranked. The letter will no doubt lead to other schools pulling out of the rankings, sources said.

The letter comes on the heels of universities saying they didn't want to participate in Maclean's recent graduate survey. Despite objections from various universities, the issue, which looked at alumni satisfaction with their university experience, was still published using data from various sources.

Tony Keller, managing editor of special projects at Maclean's, said in an interview Monday that the 16th annual rankings issue will continue in its current form regardless of whether institutions willingly participate.

"This data is all available and we will be publishing it," he said.

Among their objections, the university presidents say that they are concerned that the Maclean's rankings aggregate data from a range of variables. The magazine, they say, arbitrarily assigns weights to variables that are of questionable validity. This, they say, does a disservice to students.

"The variables selected by Maclean's also fail to capture the breadth of experiences students say are important in their university education," they write.

Indira Samarasekera, president of the University of Alberta, said in an interview: "We felt that if they were not using the data in a way that we considered rigorous, given that we are academic institutions that are committed to accuracy, transparency and quality ... we really couldn't justify public institutions spending taxpayer dollars to support the exercise."

Mr. Keller, however, said the methodology used by Maclean's is fair.

"Based on 16 years of experience of doing this at Maclean's, based on extensive consultation with the universities and with experts in the field, we came up with a ranking of all the elements that make up quality in a university," he said.

[civ@uw]

Hasn't UofT done fairly well in these rankings?

[divx]

Hasn't UofT done fairly well in these rankings?

yes, ranked 2nd for eng, that's pretty good

[soulflare]

They're just pissed that Western won... and now, between the Saugeen Stripper and half the nation quitting, we're assured to defend our crown.

[jljdaigl]

Ya but alot of those schools rank pretty low, haha, i guess they dont like to see it in writting

[Chief]

Ya but alot of those schools rank pretty low, haha, i guess they dont like to see it in writting

Well looking at independant world rankings, UofT, UBC, and Mac are 3 of the 4 Canadian universities that rank in the top 100 (24th, 36th, and 88th respectively.) So if 3 of the best ranked by an independant world authority have called MacLeans rankings "oversimplified" and "arbitrary," I believe them.
O, back to school thread, so I guess I should source this information. (http://www.royaledu.net/PDF/2004_aca...ing_top500.pdf)

[Sylvestre]

The letter was signed by the presidents of the University of British Columbia, Simon Fraser University, University of Alberta, University of Calgary, University of Lethbridge, University of Manitoba, University of Toronto, McMaster University, University of Ottawa, University of Montreal and Dalhousie University.

low? UBC, UofA, UofC, UofT, Mac, OttawaU and SFU are all top schools (and that's more than 50% of the list). If anything, what's interesting is most aren't in GTA. Most aren't even in Ontario.
I agree with them. I've always found the results are biased for Ontario schools, and for schools with top post-graduate programs (which are rarely indicative of good undergrad educations)

[JimG]

Macleans gets their results by sending out surveys. How can you compare things about student life when most people have only gone to one or two universities. You can't say whether UofT or York or Mac or Waterloo is better if you haven't been to all of them.

[ynchu]

SFU wants out because they constantly got lower rankings in various areas. Remember back in the days SFU used to say it won the "most comprehensive" award? Now what? Because SFU ain't winning, it wants out?

The fact is simple and obvious. Start spending more per student and SFU will do better! Don't try to hide that fact that SFU spends almost the least per student.

[MizTEcK]

what i'd like to see is a separate ranking for undergraduate and graduate schools. UT for example is only prominent for its graduate studies and research, undergrade is horrible compared to other universities.

[nkwu]

I thought Western was rated highest for their mid year thing, part of their regular circulation, but they never fared well in the Maclean's big "Guide to Canadian Universities" that is seperate from circulation.

[Eyies]

Interesting news, thanks OP.

[elemenopee]

Doesn't sound like it's going to make any difference

from http://www.herald.ns.ca/Front/522173.html

Dalhousie president, Tom Traves — along with the leaders of universities in Ontario, British Columbia, Quebec, Manitoba and Alberta — addressed a letter Monday to the magazine’s special projects editor, Tony Keller, outlining the reason for dropping out of the 2006 survey.

....

Mr. Keller told The Canadian Press that even though some universities have refused to participate in the survey, they will still be assessed for the annual University Ranking Issue, which hits newsstands Nov. 2.

"Our (upcoming) issue . . . will contain all the data it’s always contained and it will rank every school that it’s always ranked, including every one of the universities mentioned in the letter," he said. "From a reader’s perspective, this isn’t going to change anything."

[longo]

saw this in the paper today! I'm with the universities. Down with Macleans rankings!

[Rocky1231]

Here's the actual letter they sent:


August 14, 2006


Mr. Tony Keller
Managing Editor, Special Projects
Maclean’s 11th floor
One Mount Pleasant Road
Toronto, ON M4Y 2Y5


Dear Mr. Keller:

We regret to advise you that our universities will not be participating in the 2006 Maclean’s questionnaire.

We share Maclean’s goal of providing good information for students and their families who are researching post-secondary education. We also compliment you on your editorial coverage of the post-secondary sector. Many of the articles in Maclean's have contributed to the national discussion about post-secondary education, and have helped to frame students’ choices. Our concern relates specifically to Maclean's attempts to generate a global ranking of Canadian universities.

In various ways and for some years, many institutional spokespersons have expressed considerable reservations about the methodology used in the Maclean’s university survey and the validity of some of the measures used. Thus far, these serious concerns have gone largely unaddressed, and there is still no evidence that Maclean’s intends to respond to them.

We welcome public assessment of our work, and all our institutions devote significant resources to that end. We already publish a great deal of data about our own institutions on-line and intend to publish more in future, ideally in the form of standardized datasets that will facilitate valid temporal and institutional comparisons. However, it is truly hard for us to justify the investment of public funds required to generate customized data for your survey when those data are compiled in ways that we regard as over-simplified and
arbitrary.

Our concerns about Maclean’s misuse of data in its rankings issue can be briefly recapitulated here.

To begin with, the Maclean’s rankings aggregate data from a range of variables related to the student body, class sizes, faculty, finances, library and reputation. It is inappropriate to aggregate information across a range of programs at a large and multi-dimensional research university into a single ranking number. Consider how such an approach might pervert one’s understanding of a general hospital that is ranked #1 in obstetrics and #10 in cancer care. Averaging these rankings would result in this hospital being ranked “#5 overall”. For the patient seeking care in one of these areas, such a measure would be useless at best and misleading at worst. This is, effectively, the method that Maclean’s applies to Canadian universities by its calculation of “league tables” based on the arbitrary assignment of weights to variables which, by themselves, are of questionable validity. The variables selected by Maclean’s also fail to capture the breadth of experiences students say are important in their university education such as, for example, extra-curricular activities or the opportunity for rich and diverse interactions
with peers and faculty outside the classroom.

We are also concerned by Maclean’s recent attempt to draw comparisons of student experience across incomparable surveys of student engagement, and Maclean's reliance on survey data with low response rates and all the associated response biases that arise from skewed profiles of respondents. The responsible compilation and comparison of data is a core tenet of academic research. Several universities already show student survey data, in context, on their own web sites and question Maclean's decision to pull different kinds of data out of context and compare “apples and oranges”.
Maclean’s treatment of these survey data, in our view, fails to give appropriate notice to these methodological limitations.

It is not just the Maclean’s student survey that has suffered from low response rates. Equally troubling is the fact that a clear majority of individuals who receive the Maclean's reputational survey do not respond. This is a particular concern as the results of the reputational survey not only affect rankings in a significant way, but are given prominence separately by your magazine.

This is only a partial accounting of the methodological flaws in the Maclean’s rankings. In short, the ranking methodology used by Maclean’s is oversimplified and arbitrary. We do find it ironic that universities are being asked to subsidize and legitimize this flawed methodology, when many faculty, staff and students at our institutions are dedicated in their research to ensuring that data are collected rigorously and analyzed meticulously.

We remain open to the possibility of collaborating with Maclean’s at some future date, particularly if we can agree on means to ensure that the data will be valid and the analyses truly informative. Meanwhile, we will continue to publish data on our websites to facilitate informed student and family choice.

Yours truly,