autosubmit Documentation
Release 3.1.6
Domingo Manubens - Javier Vegas
October 27, 2015
CONTENTS
1 Introduction 1
1.1 What is Autosubmit ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Why is Autosubmit needed ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.3 How does Autosubmit work ? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2 Tutorial 5
2.1 Quick start guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 Installation 15
3.1 How to install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
3.2 How to configure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
4 Usage 17
4.1 Command list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 How to create an experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.3 How to create a copy of an experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.4 How to create a dummy experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.5 How to configure the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
4.6 How to check the experiment configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.7 How to run the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.8 How to test the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
4.9 How to monitor the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.10 How to monitor job statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.11 How to stop the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.12 How to restart the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
4.13 How to rerun a part of the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
4.14 How to clean the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.15 How to refresh the experiment project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
4.16 How to delete the experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.17 How to add a new job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
4.18 How to add a new platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
4.19 How to refresh an experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.20 How to archive an experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
4.21 How to unarchive an experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
5 Troubleshooting 31
5.1 How to change the job status stopping autosubmit . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5.2 How to change the job status without stopping autosubmit . . . . . . . . . . . . . . . . . . . . . . . 32
6 Developing a project 35
i
7 Variables reference 37
7.1 Job variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
7.2 Platform variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
7.3 Project variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
8 Module documentation 39
8.1 autosubmit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
8.2 autosubmit.config . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
8.3 autosubmit.database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
8.4 autosubmit.date . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
8.5 autosubmit.git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8.6 autosubmit.job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
8.7 autosubmit.monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
8.8 autosubmit.platform . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
Python Module Index 89
Index 91
ii
CHAPTER
ONE
INTRODUCTION
1.1 What is Autosubmit ?
Autosubmit is a python-based tool to create, manage and monitor experiments by using Computing Clusters, HPC’s
and Supercomputers remotely via ssh. It has support for experiments running in more than one HPC and for different
workflow configurations.
Autosubmit is currently used at Barcelona Supercomputing Centre (BSC) to run EC-Earth, NEMO and NMMB air
quality model.
Autosubmit has been used to manage models running at supercomputers in IC3, BSC, ECMWF, EPCC, PDC and
OLCF.
Autosubmit 3.0 version is now available via PyPi package under the terms of GNU General Public License.
1.2 Why is Autosubmit needed ?
Autosubmit is the only existing tool that satisfies the following requirements from the weather and climate community:
• Automatisation: Job submission to machines and dependencies between jobs are managed by Autosubmit. No
user intervention is needed.
• Data provenance: Assigns unique identifiers for each experiment and stores information about model version,
experiment configuration and computing facilities used in the whole process.
• Failure tolerance: Automatic retrials and ability to rerun chunks in case of corrupted or missing data.
• Resource management: Autosubmit manages supercomputer particularities, allowing users to run their experi-
ments in the available machine without having to adapt the code. Autosubmit also allows to submit tasks from
the same experiment to different platforms.
1.3 How does Autosubmit work ?
You can find help about how to use autosubmit and a list of available commands, just executing:
autosubmit -h
Execute autosubmit <command> -h for detailed help for each command:
autosubmit expid -h
1
