“I thought I used the same parameters but I’m getting different results”
“I can’t remember which version of the code I used to generate ﬁgure 6”
“The new student wants to reuse that model I published three years ago but he can’t reproduce the ﬁgures”
“It worked yesterday”
“Why did I do that?”
We would like to announce the release of version 0.1 of Sumatra, a tool for tracking computational experiments and analyses so as to be able to easily replicate them at a later date.
Replication of computational experiments or analyses ought to be easy, given that computers don't suffer from the problems of inter-subject and trial-to-trial variability that make reproduction of biological experiments so challenging. In general, however, it is not easy, perhaps due to the complexity of our code and our computing environments, and the difficulty of capturing every essential piece of information needed to reproduce a computational experiment using existing tools such as spreadsheets, version control systems and paper notebooks.
The aim of Sumatra is to record as much as possible of the experimental context (software versions, parameters, dependencies, platform information, what files were produced, etc.) automatically, and make it easy to annotate the record with information that cannot be obtained automatically (why the simulation or analysis was performed, tags for later searching, etc.).
Given the large differences in the workflows of different researchers (command line, GUI, batch-jobs (e.g. in supercomputer environments), or any combination of these for different components (simulation, analysis, graphing, etc.) and phases of a project), it is difficult to provide a one-tool-fits-all solution, therefore Sumatra provides the core functionality as a Python package on top of which various different interfaces can be built.
Sumatra currently provides a command-line interface and a rudimentary web interface; we hope that people will also be interested in incorporating Sumatra's functionality within their own tools.
For more information and documentation, check out https://neuralensemble.org/trac/sumatra/.