erm as an ex technical programmer and research assistant for a world leading rnd organization not sure I buy this for all experiments - an experiment needs to be reproducible yes but…
Most science is based on physical observation of the experiment the code is just a offshoot of the test equipment.
In the case where you are modelling some thing you do experiments to prove your mathematical model. I once spent a sweltering afternoon in a bunny suit and rubber gloves and mask helping prepare a dummy fuel rod from a Breeder Reactor so that we would do experiments to see if our model of two-phase flow was valid.
And surly saying you can reproduce my experiment but only using my code can everyone not see the danger here - you would want to repeat the experiment and implement ones own version of the maths behind it.
Exactly what I was thinking. It is important that the original code is vetted - for example during a peer review process - but to say that the entire code source is necessary to reproduce the result seems self contradictory.
Surely it should be reproducible especially without the original source code.
(EDIT: that's not to say that you shouldn't provide source code - it probably depends on the experiment)
> Most science is based on physical observation of the experiment the code is just a offshoot of the test equipment.
Even if we accept this as true, I don't see why it's an argument against publishing code for that science which does directly depend on the simulations you run.
> you would want to repeat the experiment and implement ones own version of the maths behind it.
That's a good point, but only valid if the exact mathematics and methods are clearly explained elsewhere. But as the article states, usually there's ambiguity. And if I try to reproduce your simulation and get different results, it's very difficult for me to get enough confidence to call you out on it (perhaps I'm the one who screwed up). If I find a bug in your code, it's easy.
This argument quickly tends towards absurdum. Even with all of the materials, equipment and original experimenters, you cannot reproduce the context of the original results. But we can do our best to ensure that we can try to reproduce the original results.
So, while I might not be able to reproduce your raw data, I can at least make sure your analysis is error-free (we can argue about its correctness). Going further, if I have the necessary equipement, you can send me the raw materials (especially in the life sciences) so that I can try to reproduce your raw data in my lab. Etc as remains practical (obviously I can't ask you to send me samples from your breeder cores).
Most science is based on physical observation of the experiment the code is just a offshoot of the test equipment.
In the case where you are modelling some thing you do experiments to prove your mathematical model. I once spent a sweltering afternoon in a bunny suit and rubber gloves and mask helping prepare a dummy fuel rod from a Breeder Reactor so that we would do experiments to see if our model of two-phase flow was valid.
And surly saying you can reproduce my experiment but only using my code can everyone not see the danger here - you would want to repeat the experiment and implement ones own version of the maths behind it.