| Thank you all for being here, it means a lot especially since this is a Tuesday during the exam period. A good friend and mentor told me that today is the one time I get to talk about anything I want, so I’m going to talk about myself 🙂 First the choices that led to be me becoming a researcher, what I’ve been doing as a researcher and various experiences with failure, and perhaps some lessons learned from all of this. Don’t worry, there is a reward for listening to me in about 40 minutes. |
To start with, I never wanted to be a researcher or a professor. But in retrospect there are some early patterns I recognize that might have played a part in all of this.
|
| After hearing about Mary Poppins, I had to test out the magic of umbrellas by myself. After a sample size of 3 broken umbrellas but without broken bones I decided my theory did not hold up after all. |
| Then I wanted to be a teacher, like my grandmother. On play dates with other children I made them math workbooks to fill in, so safe to say, I was not very popular. |
A few years later, I wanted to be a webdesigner. I made my first HTML website – not because I was interested in programming (and let’s not debate what is a real programming language now) – but because I was obsessed with the Spice Girls. I was of course Baby Spice 🙂
Already all great examples of female role models! |
|
My favorite subject in high school was mathematics, and we had an amazing teacher Mrs Huizinga. But when deciding what to study in university, my misconception was that mathematics is what you want to do if you want to be a statistician or a math teacher, which I decided was not for me after all.
I had heard about BSc in Artificial Intelligence, which at the point was actually more about cognitive science and linguistics rather than what tech bros want you to believe it is today. That was not possible due to various practical life considerations, but I had heard about this recent program Media and Knowledge Engineering in Delft, which was similar in some respects. This is the actual official poster of the the university, recruiting new students for that year. |
| During my studies I got involved in various student committees and truly started feeling like I was part of a community. Probably it helped that I didn’t try to give other students math work sheets, beer was much more popular. Many of the people are still great friends of mine to this day. |
| I supported my studies through web development in PHP – a “childhood dream” so to say. Altogether that did not leave a whole lot of time to study, but I did get “good enough” grades – such as a 7 out of 10 for linear algebra, since I still had no idea this would be important later. But all in all, I think all the “extra things” I did were just as valuable as the courses I was taking. |
Don’t worry though, things got better during my MSc and they let me graduate 🙂 I look a bit surprised on the picture though.
|
| During the study I had the course Pattern Recognition with Bob Duin and David Tax, who later also became one of my PhD advisors. The course was an amazing revelation that so many different problems could be approached in the same way, by embedding examples with different labels (such as positive or negative) in a feature space, studying their distributions and distances, detecting outliers and more. Some concepts like the random subspace method, seemed so impossible that I had to try and reimplement it by myself – without breaking umbrellas – and seeing that it really works. |
| I was still doubting about the PhD and tried to gather some advice from others who had done a PhD but have after left academic. The most valuable piece of advice was received was from Ela Pekalska who had graduated from Delft a few years earlier. Her advice boiled down to “think about who you want to be, not what you want to do”, referring to your values. I realized my values do not really align with those of the Big Four companies, and I also found out that the PhD contract is 4 years – so more job security – and I decided to go for it, still without any particular goal of becoming a professor. |
| I started my PhD and was doing research full-time, so perhaps this is when I could start calling myself a researcher? |
The research was on a learning paradigm called multiple instance learning. Here the examples themselves are not labeled positive or negative, but there is information about the labels of groups of these examples, which can happen in different applications where labels are difficult to obtain, such as medical imaging. I approached this problem by studying the similarities between the example groups, building on Ela’s work from some years ago.
Now that I did sign up for a PhD, I wanted to do a good job, but without a specific career goal in mind. I spent a lot of time playing around with similarities between groups of examples, writing small conference papers and having engaging discussions with other researchers and of course also did some … |
| … sightseeing. All these conferences were great for my development as a researcher, but I found out this did not count for my PhD, nor for my grant reviewers years later. |
| Then I spent time writing and revising papers. I remember a reviewer commenting that “the method is not novel, and it is too simple to have such good results”. I discovered that a positive example of a published paper is one that introduces a novel learning paradigm, then proposes a complex method to solve it, so the method by default becomes state-of-the-art for that specific paradigm. Rather than incorporating this into my paper, I instead wrote a different paper – about all the novel paradigms I could find about groups of examples, but which could all be written down in one generalized formulation. |
| By now I was thoroughly enjoying doing research in Delft, and I was hoping to do a postdoc, although it took me a while to dare to say this to others. I did not have many examples of postdocs, and definitely not ones I could see myself being – for example, one postdoc did his PhD with a Nobel Prize winner, and was moving on to a large social media company in the US. I thought that maybe people would laugh at me if I said I could see myself continuing in research. |
| I was slowly starting to think that maybe this was also a pattern recognition problem – a feature space where we could represent researchers, with features like the impact factor of where you published. And I did not feel like I could get to the positive side of the decision boundary after I would be done with the postdoc, also given the state of the academic job market. Also, people told me that staying in the same country for your postdoc was career suicide, but I had no intention of moving abroad. |
For my research visit I went to Tübingen, which I think influenced my next steps quite a lot. At a summer school there I saw a talk by Bob Williamson on how to choose research problems. I particularly remembered three things: thinking about the biggest problems in the world and whether my research related to that, thinking about what sentence in a textbook – however specific – your research might change, and not to invent yet another hammer for solving problems, as we already have enough of those. Given my focus on specific methods that was a bit depressing to here, but at the same time motivating.
One talk that particularly inspired the way I think about these things, was a talk given by Bob Williamson at a summer school in Tübingen in 2013. I was still a PhD student, feeling quite unsure about whether my research would ever be useful, and then I saw this talk on how to choose your research problems, which was just excellent. I particularly remembered three things: thinking about the biggest problems in the world and whether my research related to that, thinking about what sentence in a textbook – however specific – your research might change, and not to invent yet another hammer for solving problems, as we already have enough of those. Reflecting on my PhD this was a bit depressing but at the same time motivating. |
| Tübingen was great – I even got a diploma there. But most importantly, I met Aasa and Chloé, who were a few years ahead of me, and who I could relate to, and through their mentorship I started believing I should try to stay in research. |
| During the last year of my PhD I also briefly visited Aasa in Copenhagen, and during that visit met my now-husband Mattias. Although he soon moved to join me in The Netherlands, we would still from time to time visit Copenhagen, which played a role later on. |
| Finally I finished up the thesis – which has a cat puzzle on the cover, and defended it on this day, exactly 10 years ago – which is why I made you come here on a Tuesday. You can check it out if you are tired of listening to me talk about myself 😄 |
| And here are some things I did next! |
I found a postdoc in Rotterdam, just half an hour away from where I lived. The project was to apply methods similar to the ones I worked on to medical imaging, so it seemed like a perfect position.
How it works in short, is that we take data from existing patients (e.g. healthy vs cancer), then train a classifier (e.g. a neural network) to distinguish the patient groups, apply classifier on other patients to predict their diagnosis, and evaluate how good these predictions are compared to other methods or medical experts. |
| If this is successful, you can then publish your results. You could develop software that could be used in the clinic and there are such examples, although this happens only for a tiny fraction of all publications, and even approved software often lacks external validation. But I saw that in many cases, it was more about overhyping the results and getting attention and citations. |
The postdoc turned out to be a combination of three jobs – the project you are hired for, developing your own research line, and applying for the next thing, either research funding or a “normal job”. I also felt like I did not really belong, perhaps because I started realizing I did not want to do things the way that successful examples of researchers did.
Collaborations often translated into having co-authors you have never met, who felt that “you owe them a paper”, or would “forget” to cite your work even if the idea was yours. The research was focusing on methods, presenting these as novel, and optimizing them until the results looked good. I don’t want to highlight any examples of my research from this time because I have started to doubt the validity of what we did. |
Paper rejections already felt normal at the time, but rejections for funding – and therefore job security – were a whole different thing. Overall the situation combined with several events in my personal life led to a diagnosis of clinical depression, for which I was lucky to get some therapy, but I thought it would be a personal failure to say yes to medication.
Around this time there was this thing on the internet, where a professor at Princeton has shared his CV of failures, an idea first proposed by Melanie Stefan a few years year. I met Johannes years later and he’s a great guy, but at the time he really pissed me off because his rejections were from Harvard and Berkley, and not the kind of things I was getting rejected for, and I thought that my CV of failures would already be longer, so I had to test it out. |
Without feeling like part of a community, I tried to find one online, which was on the platform we remember as Twitter. Here I met academics from all sorts of different fields, who were were thinking critically about research, advocating for open science and reproducibility, and talking about barriers in science. This was truly a supportive place and I miss it.
At the time, after not finding positions or funding, I was almost sure I would leave academia, so I felt like there was nothing to lose by sharing my CV of failures online. The response was SO overwhelmingly positive, and it was amazing to feel that something I wrote had impact, much more so than the papers I had been writing. I just had to do more of that! Since there is a limit to how quick you can accumulate failures, I then started interviewing other academics about theirs, which gathered more and more attention. Based on my website statistics in 2020 I estimated that over 100,000 people have read at least one of my blog posts, and I’ve received hundreds of thank you messages about this. I have to admit that I got a bit lazy with this project after moving countries but maybe it is something I’ll continue in the future.
Interestingly, the Princeton professor did himself add what he called a meta-failure to his CV, writing “This darn CV of Failures has received way more attention that my entire body of academic work.“, which I would have to agree with. |
| But it was also just a fun place to be which led to all sorts of things like getting invited for talks, getting together a group of academics to write a paper about being an academic on Twitter. Perhaps my biggest claim to fame, MC Hammer followed me after I argued with him about AI. If you are too young to remember MC Hammer this is him. |
Although due to the ongoing rejections I was convinced I would be leaving academic, a tenure track position unexpectedly came up, where I could do all the things and also teach and more. Several goals regarding the volume of research, teaching, etc., were set, and I would have a midway evaluation after two years, and then after four years for the tenure decision.
Combining the tenure track realities with the housing situation in The Netherlands, and thus a long commute, I’m fairly sure that my days would be unacceptable for Danish work-life balance standards. |
| Thanks to the commute, I did get to read a lot of papers, and see more patterns in what was being researched or not. As a highlight I wrote these reviews, one on machine learning methods for data where only limited labels are available, and one, which I am probably most proud of, on transfer learning and what researchers might mean when they say that datasets are similar. The papers also started gathering citations, although I once heard that this was “unfair” because academics should not be expected to promote their research on social media. |
But I was slowly feeling part of a community again. I finally got to supervise a PhD student, Ralf, who got his own funding to pursue a PhD. And I gathered a motivated and “gezellig” group of MSc students, for the first time deciding that this was my group, and I would call it PURRlab, or at least I registered the domain name.
I also gathered a sort of academic support became friends with many of them. In particular Felienne and Daniël – our pubquiz team “Waardeloze wetenschappers” and the concept of “noodbiertje” if one of us had a rejected grant proposal. And another support group where we called our tenure track journey “flying a plane while building it”.
Here I also should mention Bettina Speckmann, who started out as my mentor in a formal mentorship program for women academics, but our meetings evolved into ranting about academia over beers and good food. |
However, my mental health was still at a fairly low point, and after more life events I did finally accept medication offered to me by the doctors. And it was amazing, because I actually started better and better! Much better – I even needed less sleep, and could put in some extra hours into writing grant applications.
I was not made aware that the medication is not suitable for people who may have undiagnosed bipolar disorder. So what I was experiencing was in fact mania, which after several nights of writing turned into a full blown psychosis. Herein I thought I was one of the Avengers, took myself off the grid, followed the voices to a village in France, and got locked up for several days without anybody having a good idea where I was. Thanks to Mattias and a good friend of mine, I did return home safely, though fairly beaten up and also gaining various other medications in my inventory. This wasn’t advertised on all the successful CVs I’ve seen, and please don’t try this at home. |
But soon I started returning to all the things, among others successfully finishing my teaching portfolio. Although I felt like I achieved the conditions laid out to me at the start of the tenure track, things did not go as expected. Around this time I also received the question, “when will you start doing real research?”, so perhaps what I was doing did not fit into the normal distribution of research paper examples of proposing novel methods.
The outcome of my midway evaluation was that that my evaluation committee did not expect I would be good enough for a permanent position by the end of my contract. As a special offer because I have been ill, I was offered an additional temporary contract of a year, where I would be “really able to prove myself”.
In retrospect I think I could have contested the decision, but I decided the institution did not align with my values and I resigned from my job, which I of course posted on the internets. |
| So next a number of exciting things happened… |
| First I decided to write one last paper about all the failures I’ve been in the machine learning field, and through Twitter found a perfect collaborator Gaël. Here we among others showed that large datasets are not enough to create methods that generalize in practice, that research overfocuses on benchmark datasets, and that the reported state-of-the-art results are often not as good as one thinks. |
| Then my post about leaving the tenure track also gathered some attention online, and Barbara Plank got in touch with me advising me to apply for positions at ITU. I was not at all searching for positions abroad, but since I already had all the documents prepared, I thought I might as well send them in. This was during lock down, and by now I was focusing on developing my own thing by giving workshops about failure. I kind of forgot about the whole ITU application, but half a year later, much to my surprise I received news that I was qualified for a position as associate professor. I gave a job talk about all the failures in machine learning, and had an interview where I probably told most of this story and started crying, but was still offered the position. And this was worth moving countries for, especially given our earlier connection with Copenhagen. |
| I can’t stress enough how ITU has been a transformative experience. My “not real research” was suddenly valued – I was trusted enough to decide what it is what I wanted to do. And what I wanted to do was not to develop novel methods, but focus on data, evaluation and meta-research. I even finally got funding to do so, and was able to hire PhDs and postdocs – PURRlab 2.0 – and what an amazing time this has been. |
| We pointed out several problems in how researchers use data – often not questioning why they use that dataset, duplicating datasets in non-transparent ways, and not citing them. At least nowadays it is more rare to start with a problem first, and then create a dataset to address it, which we did recently. |
We also showed that shortcuts, or the Clever Hans effect, are a huge problem in medical imaging. The effect is called after a horse called Hans, who lived in the start of the last century and could count, so the trainer would ask how much is 2+2, and Hans would move his hoof four times. Of course this was later debunked because Hans was simply moving his hoof until his trainer looks satisfied. But a similar thing is happening in machine learning, where the classifier learns to have good predictions by relying on different feature.
For example, we showed that it is possible to classify lung diseases accurately even if you remove the lungs from the images. This is happening in a lot of applications, so it is important that we find a way to keep track of all this evidence about the data quality. |
On a positive note, we found that non-experts can create annotations that can be helpful to avoid these shortcuts, and to create more robust methods.
By now my team members are much more qualified to talk about these works, so keep an eye on when they might be presenting! |
| With collaborators from Heidelberg and many others, we have also worked on evaluating whether the evaluation itself is done in a way that can justify the conclusions – TLDR it is often not. |
| So you can see why I am a fairly skeptical of what we are doing as a field. The quality and diversity of the datasets we work on is decreasing, while every day hundreds of methods are posted on arXiV, so it is unclear how we can generalize from this as a field. This is also leading to a lot of overoptimistic results and hype, and wastes resources. Finally I’m concerned about the opportunity cost of overfocusing on inventing methods that do well on benchmarks – what actual world problems are not getting researched? |
| When I present my research I often hear that I am only pointing out problems. Others agree pointing out problems is important, but they cannot do anything about the incentives. If you recognize yourself in this, I highly recommend to read this blog post by Tal Yarkoni and to critically review the motivations for your research. |
| So here are some of my proposed solutions – firstly, don’t forget to look at your data, and read, and write. I think we should also be critical of how we evaluate things and not only present the best results. Finally I think it is important to study the systems within research functions, which is something I am planning to do in a project starting this fall. |
|
| Firstly, it was a fallacy of mine to see research as a pattern recognition problem where the examples that you see, can be used to predict what is going to happen in the future. But it’s not quite that simple, because a lot of the data is missing. |
Secondly, there are of course other features involved. The number of positions and funding might be different than a few years ago. Your gender, country where you were born, any disability or anything that influences how might create barriers for you, also play a role, and these researchers might disappear through the leaky pipeline.
Interestingly, these properties might not be taken into account when creating the rules that lead to “it’s good for your CV” type of advice. And even if you try to do everything right, you might get an unexpected output, because there is randomness involved.
Finally, I believe that “making it” to the next level of research is a proxy for being happy with what you are achieved . Sure, it is probably easier to make an impact, but it’s not a necessary condition as I have experienced. |
Why do I find it important to talk about this? I already mentioned that it is important to look at the data, and we should do that critically with the data of the system we are in. The data we collect is in many ways biased, and your dataset might only consist of the examples that “made it”, or the survivors.
If you have not seen this picture before, it comes from a story about World War II where many planes were missing in action, while others returned, although damaged. A statistician looking at these damaged planes then proposed to reinforce the areas where the planes were hit — but there was no data on the planes that were hit even worse, and did not manage to return. I am part of this biased dataset, so probably you should not be taking career advice from me.
|
Reflecting on all this, I think about patterns in what you between research and in other areas of life. The most recent pattern I am enjoying in this one – where you get to choose your alignment which is related to your values – this is my chaotic good half-orc barbarian called Barbro Alestorm 😄. And then of course, gather your party before venturing forth.
|
ANAnd what a party it has been!
I can’t stress how much of a community feeling I have felt at ITU. Thanks Peter for the leadership in this, and all the colleagues who became collaborators, mentors, and friends. In particular DASYA for giving me at home and especially Pinar for continued support and our Gilmore Girls sessions, Yvonne, Andrzej, Stefan, and everybody who’s been joining Friday beers, social dinners and of course Copenhell, the women researchers group, everybody in the DHL “Slow Science” teams, and many others. |
| Of course my lab! Next to doing not real research we have had some great times together at conferences, social events, watching each others cats and more. I’m sad that you are moving on to other places this year but so happy for what we’ve been doing together and looking forward to collaborations in the future. |
| Lastly thank you to all friends and family who have been doing the work behind the scenes for getting my plane to come back. |
