News about research data management

New TU Wien study on improving reproducibility

Tomasz Miksa recently presented the results at the eScience 2024 Conference in Osaka, Japan.

The photo shows a male person at a lectern next to a screen with a presentation.

© Carlos Cuevas

Tomasz Miksa at the eScience conference in Osaka in September 2024.

The paper “Mission Reproducibility: An Investigation on Reproducibility Issues in Machine Learning and Information Retrieval Research” investigates reproducibility challenges in Information Retrieval (IR) and machine learning research. The study analysed 45 reproducibility reports from 17 papers published at major IR conferences, identifying three core problem areas: code availability, data limitations, and incomplete experimental processes, which were further classified into 13 categories. The paper offers recommendations aligned with the FAIR principles to address these issues.

The study is a collaborative effort involving Moritz Staudinger, Bettina M. J. Kern, Tomasz Miksa, Lukas Arnhold, Peter Knees, Andreas Rauber, and Allan Hanbury from the Faculty of Informatics and the Center for Research Data Management at TU Wien. Students from the Experiment Design course contributed by publishing their exercise reports as open-access documents on Zenodo, opens an external URL in a new window, providing data for analysis. These reports helped identify practical challenges in reproducing experiments, making them a key part of the study.

The recommendations will be integrated into the review process for submissions to the TU Wien Research Data repository, opens an external URL in a new window. Additionally, findings from this research will be incorporated into the TRS-course 058.005 Introduction into Research Data Management, opens an external URL in a new window.

To the paper

The paper was presented at the 20th IEEE International Conference on e-Science (eScience 2024, opens an external URL in a new window) and can be found in the IEEE Xplore.

M. Staudinger et al., "Mission Reproducibility: An Investigation on Reproducibility Issues in Machine Learning and Information Retrieval Research," 2024 IEEE 20th International Conference on e-Science (e-Science), Osaka, Japan, 2024, pp. 1-9, doi: 10.1109/e-Science62913.2024.10678657, opens an external URL in a new window.

Contact

TU Wien
Center for Research Data Management
Favoritenstraße 16 (top floor), 1040 Vienna

research.data@tuwien.ac.at