Where she begins the talk made me laugh a lot. I research machine learning and one thing I disagree with a lot of my peers with is the speed in which ML is progressing. I don't disagree that things have changed rapidly, but the underlying work and innovation is rather slow. There's a lot of similarities in frameworks and underlying connections that mean one should be able to transfer a lot of knowledge from one niche to another. (e.g. I was focused on normalizing flows in the beginning of my PhD, while everyone was on GANs, and when diffusion came out I instantly understood it. And now flows are becoming a thing but I still am unable to publish in this area because I cannot obtain a reviewer that understands the base concepts nor the history and context of the papers. It doesn't work if I have to spend 10 pages in background contextualizing everything...).
I think with funding, the thing is you need to embrace the noise. I don't want to dissuade people from trying to optimize this process. But I think if we don't recognize there is a high amount of noise, that it is doomed to fail. There's things you know you know, there's things you know you don't know, there are things you don't know that you don't know, and there are things you think you know and don't know. Due to the latter two, you have to embrace noise into the system or else you will be limited to the first two. You have to embrace the outside, wild, and crazy. It can be extremely difficult to differentiate genius from craziness, and often there is none. There's always a price, so the question is which is more costly. She picks universities as the great breakthrough place, and to be fair, I mostly agree, but I think there's a lot of metric hacking that she isn't thinking about. That this has been baked in for years. The people that are successful in the system breed that system. Nor does it account for the dark horses, the noise. It reinforces fast timelines, as we're rewarding predictability in domains where it may take several decades for those predictions to be proven accurate or not. There's a certain irony in her mentioning Higgs, as his original paper was in 1964 but we didn't discover the particle until 2012.
Part of the issue she is talking about is solved more easily. Lots of research is rejected for lack of novelty. Why this is even a metric to judge a paper's worth is beyond me, as the foundation of science is about reproducibility. It also encourages obscurification as so much is obvious once you've been told, but not prior. If we go back to the form of publishing where we publish works void of major errors and plagiarism, I bet we would see an increase in innovation again. Another part of the issue is in evaluation itself (similarly solved). We're all under lots of time pressure and so few spend time reviewing carefully (you'd be shocked at some reviews I've gotten and similarly how generic they are: "paper is well written and easy to understand. Not enough experiments, not novel. There are too many writing errors" (points out easily solvable thing like a broken cross reference or an incorrect issue like saying 'these data')). You can see part of this in HN comments too btw. Where people look at benchmarks and read them as answers instead of hints. Which there is an extra problem in that anything sufficiently novel will be unlikely to be state of the art on all things in the initial go. So it is easy to reject. Then the research is not pursued. But this is ludicrous when we're talking about research, which is at a much lower level. The problem is people read research papers as if they are reviewing products. This can only stifle innovation. In machine learning there is the "gpu rich" and the "gpu poor." I can with high confidence say that the "gpu poor" aren't any worse at innovating, but rather that their works are just more likely to be rejected. How can you compete when others can spend 2-5x the money to tune parameters when what the paper is about is an architectural change or a change to optimization methods. We aren't holding variables equal here and very few want to admit it.
I think with funding, the thing is you need to embrace the noise. I don't want to dissuade people from trying to optimize this process. But I think if we don't recognize there is a high amount of noise, that it is doomed to fail. There's things you know you know, there's things you know you don't know, there are things you don't know that you don't know, and there are things you think you know and don't know. Due to the latter two, you have to embrace noise into the system or else you will be limited to the first two. You have to embrace the outside, wild, and crazy. It can be extremely difficult to differentiate genius from craziness, and often there is none. There's always a price, so the question is which is more costly. She picks universities as the great breakthrough place, and to be fair, I mostly agree, but I think there's a lot of metric hacking that she isn't thinking about. That this has been baked in for years. The people that are successful in the system breed that system. Nor does it account for the dark horses, the noise. It reinforces fast timelines, as we're rewarding predictability in domains where it may take several decades for those predictions to be proven accurate or not. There's a certain irony in her mentioning Higgs, as his original paper was in 1964 but we didn't discover the particle until 2012.
Part of the issue she is talking about is solved more easily. Lots of research is rejected for lack of novelty. Why this is even a metric to judge a paper's worth is beyond me, as the foundation of science is about reproducibility. It also encourages obscurification as so much is obvious once you've been told, but not prior. If we go back to the form of publishing where we publish works void of major errors and plagiarism, I bet we would see an increase in innovation again. Another part of the issue is in evaluation itself (similarly solved). We're all under lots of time pressure and so few spend time reviewing carefully (you'd be shocked at some reviews I've gotten and similarly how generic they are: "paper is well written and easy to understand. Not enough experiments, not novel. There are too many writing errors" (points out easily solvable thing like a broken cross reference or an incorrect issue like saying 'these data')). You can see part of this in HN comments too btw. Where people look at benchmarks and read them as answers instead of hints. Which there is an extra problem in that anything sufficiently novel will be unlikely to be state of the art on all things in the initial go. So it is easy to reject. Then the research is not pursued. But this is ludicrous when we're talking about research, which is at a much lower level. The problem is people read research papers as if they are reviewing products. This can only stifle innovation. In machine learning there is the "gpu rich" and the "gpu poor." I can with high confidence say that the "gpu poor" aren't any worse at innovating, but rather that their works are just more likely to be rejected. How can you compete when others can spend 2-5x the money to tune parameters when what the paper is about is an architectural change or a change to optimization methods. We aren't holding variables equal here and very few want to admit it.