[D] Google AI refuses to share dataset fields for a dataset paper (ACL'18) and associated challenge (at CVPR'19)

Let's say you publish a method to implant a device into a mouse at a specific location. You do it, you kill the mouse, and cut the corpse open to verify the location. Reproducibility here means that you take a similar mouse and perform the method and then observe similar results. You should not use the same mouse for two reasons: (1) the published mouse is already cut in pieces. (2) maybe the published mouse was somehow special which made the method work; the experiment should be done with a different mouse to reproduce the result scientifically.

Translated to data science that is, we should observe similar results on similar data sets. That's reproducibility. What you are talking about is double-checking the calculation. I'd trust google's engineers in that regard. Thanks to them for sharing their findings and software.

/r/MachineLearning Thread