Thursday, October 1, 2020
Commentary: On the (ab)use of comparisons in recommendation letters
Comparing candidates for tenure and promotion with other researchers is lazy and fraught with bias.
The Holy Grail of a physics career is tenure: the point where one’s job is secure and one can afford to shift research focus to longer-term goals. To attain it, assistant professors compile a dossier of their accomplishments in the past five years, and their departments solicit letters from the community about the candidate’s reputation.
In theory—I am a particle theorist—those letters are crucial in that they provide an outside perspective on the candidate’s research accomplishments. But in practice the process of writing and evaluating letters can be fraught with bias. That’s in part because writers of those letters often try to compare the candidate with other scientists.
My first experience with a comparison in academia came about because a senior professor with whom I was working made an offhand comment. I reminded him, he said, of another person, someone in between us in age and whom I admired. My spirits soared. It was clear that he meant it positively, and I interpreted it as a flattering comparison.
Later that day, I shared his remark with another colleague who was supporting my application for a permanent position at her own institution. “He’d better not write that in his letter,” she said. “If he does, we will never hire you here!” I spent that night lying awake wondering if my chances for what would have been my first real job were being torpedoed by good intentions. I did not end up being offered that position.
My next experience with those kinds of comparisons came after I had transitioned to a permanent position. My department was hiring a postdoctoral scholar, and it was enlightening to see how the process looked from the employer’s point of view. As I went through the application files, I noticed that many letter writers chose to compare their subjects with other researchers in the field. Those letters stood out as memorable, for no small part because of the layer of gossip provided by the comparisons.
One particular letter compared one candidate, a freshly minted PhD, with someone I knew well, my office mate at my previous job. I vehemently disagreed with the comparison. Both the candidate and my former office mate were excellent physicists, but I knew each well enough to know that they were quite different physicists, whose interests and personalities were radically different.
After puzzling over why the comparison so troubled me, I identified the problem. The letter writer didn’t know my former office mate very well, certainly not as well as I did. Although his comparison with the candidate was not intrinsically flawed, it was incapable of conveying what he intended to someone with my foreknowledge. A comparison that seemed perfectly reasonable to him grew into a distraction for me that overshadowed everything else he had written.
On another occasion, a letter writer decided to emphasize how excellent his former student was by explaining how much better he was than a number of more senior physicists—including me. That tactic made for a somewhat tense group meeting to discuss whom we should hire. All of my colleagues had read that same letter and seen my name. “Well, I guess we had better hire this guy,” I joked when we discussed the file. No one laughed.
Now that I’m a full professor, perhaps 10 to 15 people each year ask me to write letters in support of their promotions or hires involving tenure. About half of them request that I compare the candidate with others in their field, and perhaps a quarter list specific names for whom they would like a direct comparison.
My stomach roils every time I read one of those requests.
Frequently, some of the names on the list seem inappropriate for comparison, because the person is in a different subfield, is at a different career stage (shockingly few such requests even bother to ask to renormalize for career stages), or has had such a different path through life that a comparison becomes essentially meaningless.
Even worse, I may not know the proposed comparators or be able to figure out how the department making the request considers them. Although I do my best to provide thoughtful and nuanced responses, I can never shake the feeling that my comparisons reveal more about me than about the candidate being considered. What’s more, for someone from a different field of physics, they are less than useless.
While serving on my university’s tenure and promotions committee, I’ve lost count of the times I have read “Professor X is so much better than Professors Y and Z” and thought to myself, “I don’t know what I am supposed to do with this information.”
The practice of seeking and making comparisons is dangerous. We should stop doing it. Because the human mind loves connections, comparisons feel like a way to convey a wealth of information succinctly, but that concision is precisely the problem. For the comparison of two academics to be meaningful, both the person writing the letter and the person reading it need to have a shared understanding of what the comparison means. It favors those who are “in network” and indulges in a kind of communal narcissism that perpetuates a power structure based on privilege rather than merit.
Importantly, physics thrives as a field because of fresh ideas. Directly comparing candidates presupposes a unique set of review criteria, leading hiring committees toward a roster of homogeneous candidates. It disadvantages women and underrepresented groups, thus sacrificing the vibrant perspective they bring to the field.
Furthermore, the possibility for misunderstanding is huge. Human intellect is a complicated multidimensional parameter space. The chance is small that when a reader encounters a name in a letter it aligns with the writer’s intended comparison along the same axes in that parameter space.
Forgoing comparisons with specific people need not lead to a loss of quantitative evaluation. More neutral comparisons such as “Would I support this promotion at my own institution?” or even “Would it be successful at my own institution?” allow evaluators to place a candidate within their field without the distraction or the drama of attaching a name to the comparison, especially if some context for how things work locally is provided.
More broadly, statements as to how a candidate fits within their cohort can go a long way to contextualize a sense of relative ranking and emphasize their unique contributions without getting into the weeds of a specific comparison.
A thoughtful letter of recommendation is a crucial tool for academic advancement. Avoiding the lazy route of imperfect comparison is something to which we all should aspire.
Tim Tait chairs the department of physics and astronomy at the University of California, Irvine.