I tend to agree that the biology papers often lack proper documentation of procedures and methodologies. This is a wonderful effort to reproduce some of the key experiments. That being said, I think it's also very important to look at the quality and qualifications of the labs doing the reproductions.
I don't have any direct link to cancer research, so I can't speak with authority on the subject, but I have been involved in the past with a company working in the Preimplantation Genetic Diagnosis field.
The basics of their procedure is to create one or more human embryos via IVF, incubate the embryos for up to 6 days, than either freeze them, or transplant them into the prospective mother. On day 3 or 5 of incubations the embryo is biopsied, and the genetic material is tested to make sure there are no aneuploidy defects. We were also able to test for some other types of genetic abnormalities. This is for people who are having problem becoming pregnant.
In any case, some time in the mid 2000s there were 3 papers published in Europe claiming that performing biopsy on Day 3 is extremely detrimental to the embryo, and their conclusion was that PGD with Day 3 should not be performed. The experiments were conducted by people who were unskilled in micro manipulation.
They did follow proper protocols, and I am sure they did their best to replicate proper procedures. But micro manipulation is as much skill as it is knowledge. For instance, I can write a detailed procedure on how to shoot a compound bow, and you can follow that procedure exactly. But, without practice, you are not going to hit the bullseye on the first try.
Because we were in the business of providing services to doctors, not publishing papers, we constantly tracked our embryo mortality rates, birth rates, and accuracy of testing. The better our results were, the more business we would get. And we couldn't fake the results, because clinics ordering the test would be the once recording all of those statistics for us.
Any way, long story short, none of our data agreed with the papers claiming that Day 3 biopsy was detrimental to the embryo. In fact, quite the contrary, many of our statistics suggested that Day 3 biopsy and Day 4 or Day 5 transfer would result in better implantation rates. But, the papers were published, and referenced, and then it became "common knowledge" that Day 3 biopsy is bad, and the medical industry moved on to Day 5 biopsy and embryo cryopreservation, and so has the company I worked with.
To the company I worked for it's all the same, money is money. Day 3 or Day 5 biopsy, they make money all the same. But the patients are not more limited. From the stats we have seen, it doesn't look like Day 4 or 5 biopsy is worse for the embryo, but being frozen isn't a walk in the park. With Day 5 biopsy you have to freeze the embryo, in order to allow time for the test results to come back.
Any way, it's my 2 cents. Reproducibility is important, but I think it's just as important to change the incentives of those who publish papers. If you goal is to be published, then of course your research will suffer. It's the publish or parish mentality in academia that is the problem, I think.
Why didn't your company publish their data. If day 3 biopsy is better and you have the data to show it get it out. This is not some meaningless result - this is a matter of life and death.
Because things in medical science function a lot differently from the rest of science. Take a look at this video to see what I mean:
https://www.youtube.com/watch?v=VArT6Kj_x_8
I don't have any direct link to cancer research, so I can't speak with authority on the subject, but I have been involved in the past with a company working in the Preimplantation Genetic Diagnosis field.
The basics of their procedure is to create one or more human embryos via IVF, incubate the embryos for up to 6 days, than either freeze them, or transplant them into the prospective mother. On day 3 or 5 of incubations the embryo is biopsied, and the genetic material is tested to make sure there are no aneuploidy defects. We were also able to test for some other types of genetic abnormalities. This is for people who are having problem becoming pregnant.
In any case, some time in the mid 2000s there were 3 papers published in Europe claiming that performing biopsy on Day 3 is extremely detrimental to the embryo, and their conclusion was that PGD with Day 3 should not be performed. The experiments were conducted by people who were unskilled in micro manipulation.
They did follow proper protocols, and I am sure they did their best to replicate proper procedures. But micro manipulation is as much skill as it is knowledge. For instance, I can write a detailed procedure on how to shoot a compound bow, and you can follow that procedure exactly. But, without practice, you are not going to hit the bullseye on the first try.
Because we were in the business of providing services to doctors, not publishing papers, we constantly tracked our embryo mortality rates, birth rates, and accuracy of testing. The better our results were, the more business we would get. And we couldn't fake the results, because clinics ordering the test would be the once recording all of those statistics for us.
Any way, long story short, none of our data agreed with the papers claiming that Day 3 biopsy was detrimental to the embryo. In fact, quite the contrary, many of our statistics suggested that Day 3 biopsy and Day 4 or Day 5 transfer would result in better implantation rates. But, the papers were published, and referenced, and then it became "common knowledge" that Day 3 biopsy is bad, and the medical industry moved on to Day 5 biopsy and embryo cryopreservation, and so has the company I worked with.
To the company I worked for it's all the same, money is money. Day 3 or Day 5 biopsy, they make money all the same. But the patients are not more limited. From the stats we have seen, it doesn't look like Day 4 or 5 biopsy is worse for the embryo, but being frozen isn't a walk in the park. With Day 5 biopsy you have to freeze the embryo, in order to allow time for the test results to come back.
Any way, it's my 2 cents. Reproducibility is important, but I think it's just as important to change the incentives of those who publish papers. If you goal is to be published, then of course your research will suffer. It's the publish or parish mentality in academia that is the problem, I think.