Python - Subprocess
How can I run another program?
Run a command in a subprocess.
check_outputand when to use each of these.
Read it’s output.
Time estimation: 45 minutesLevel: Intermediate IntermediateSupporting Materials:Last modification: Feb 13, 2023License: Tutorial Content is licensed under Creative Commons Attribution 4.0 International License. The GTN Framework is licensed under MITpurl PURL: https://gxy.io/GTN:T00097version Revision: 8
Best viewed in a Jupyter Notebook
This tutorial is best viewed in a Jupyter notebook! You can load this notebook one of the following ways
Run on the GTN with JupyterLite (in-browser computations)
Launching the notebook in Jupyter in Galaxy
- Instructions to Launch JupyterLab
- Open a Terminal in JupyterLab with File -> New -> Terminal
- Select the notebook that appears in the list of files on the left.
Downloading the notebook
Sometimes you need to run other tools in Python, like maybe you want to Here we’ll give a quick tutorial on how to read and write files within Python.
In this tutorial, we will cover:
Programs can run other programs, and in Python we do this via the
subprocess module. It lets you run any other command on the system, just like you could at the terminal.
The first step is importing the module
You’ll primarily use two functions:
Which executes a command and checks if it was successful (or it raises an exception), and
Check Call: Downloading Files
Which executes a command returns the output of that command. This is really useful if you’re running a subprocess that writes something to stdout, like a report you need to parse. We’ll learn how to use these by running two gene callers, augustus and glimmer. You can install both from Conda if you do not have them already.
conda create -n subprocess augustus glimmer3
Additionally you’ll need two files, you generally should not do this, but you can use a subprocess to download the file! We’ll use
subprocess.check_call for this which simply executes the program, and continues on. If there is an error in the execution, it will raise an exception and stop execution.
url = "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/836/945/GCF_000836945.1_ViralProj14044/GCF_000836945.1_ViralProj14044_genomic.fna.gz" genome = 'Escherichia virus T4.fna.gz' subprocess.check_call(['wget', url, '-O', genome]) subprocess.check_call(['gzip', '-d', genome])
wget https://ftp.ncbi.nlm.nih.gov/.... -O "Escherichia virus T4.fna.gz" gzip -d "Escherichia virus T4.fna.gz"
The above segment
- sets a url variable
- sets an output filename,
Escherichia virus T4.fna.gz
check_callwith a single argument: a list
wgeta tool we use to download files
- the URL
-Oindicating the next argument will be the ‘output name’
- what we want the output filename to be called
check_callwith a single argument: a list
gzipa tool to decompress files -
-dindicating we want to decompress
- and the filename.
This list is especially important. When you run commands on the command line, normally you just type in a really bit of text by yourself. It’s one big string, and you’re responsible for making sure quotation marks appear in the right place. For instance, if you have spaces in your filenames, you have to quote the filename. Python requires you specify a list of arguments, and then handles the quoting for you! Which, honestly, is easier and safer.
Here we manually quote the argument
glimmer3 "bow genome.txt"Output: Python
Here python handles that for us
subprocess.check_call(['glimmer3', 'bow genome.txt'])
This is one of the major reasons we don’t use
os.systemor older Python interfaces for running commands. If you’re processing files, and a user supplies a file with a space, if your program isn’t expecting that space in that filename, then it could do something dangerous! Like exploit your system!
So, always use
subprocessif you run to commands, never any other module, despite what you see on the internet!
There are more functions in the module, but the vast majority of the time, those are sufficient.
Check Output: Gene Calling with Augustus
gff3 = subprocess.check_output([ 'augustus', # Our command '--species=E_coli_K12', # the first argument, the species, we're using a phage so we call genes based on it's host organism 'Escherichia virus T4.fna', # The path to our genome file, without the .gz because we decompressed it. '--gff3=on' # We would like gff3 formatted output (it's easier to parse!) ]) gff3 = gff3.decode('utf-8') gff3 = gff3.split('\n')
augustus --species=E_coli_K12 'Escherichia virus T4.fna' --gff3=on
If you’re using
subprocess.check_output() python doesn’t return plain text
str to you, instead it returns a
bytes object. We can decode that into text with
.decode('utf-8'), a phrase you should memorise as going next to
check_output(), for 99% of use cases.
Let’s look at the results!
It’s a lot of comment lines, starting with
#. Let’s remove those
cleaned_gff3 =  for line in gff3: if line.startswith('#'): continue cleaned_gff3.append(line) print(cleaned_gff3[0:20])
And now you’ve got a set of gff3 formatted gene calls! You can use all of your loop processing skills to slice and dice this data into something great!
All unix processes have three default file handles that are available to them:
stdin, where data is passed to the program via a pipe. E.g.
generate-data | my-program, there the program would read the output of
generate-datafrom the pipe.
stdout, the default place where things are written. E.g. if you
print()in python, it goes to
stdout. People often redirect
stdoutto a file, like
my-program > output.txtto save the output.
stderr, generally if your program produces output on
stdout, you might still want to log messages (errors, % done, etc.) If you write to
stdout, it might get mixed in with the user’s outputs, so we write to
stderr, which also gets printed to the screen, and looks identical as any print statement, but it’s coming from a separate pipe.
One of the more complicated cases, however, is when you need pipes.
url = "https://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/001/721/125/GCF_001721125.1_ASM172112v1/GCF_001721125.1_ASM172112v1_cds_from_genomic.fna.gz" cds = 'E. Coli CDSs.fna.gz' subprocess.check_call(['wget', url, '-O', cds]) subprocess.check_call(['gzip', '-d', cds])
With subprocesses, you can control the stdin, and stdout of the process by using file handles.
Here we pipe a file to a process named
build-icmwhich takes one argument, the output name. It reads sequences from stdin.
cat seq.fa | build-icm test.icm # OR build-icm test.icm < seq.faOutput: Python
Here we need to do a bit more.
- Open a file handle
- Pass that file handle to
check_output. This determines where stdin comes from.
with open('seq.fa', 'r') as handle: subprocess.check_call(['build-icm', 'test.icm'], stdin=handle)
We’ll do that now:
with open('E. Coli CDSs.fna', 'r') as handle: subprocess.check_call(['build-icm', 'test.icm'], stdin=handle)
build-icm test.icm < 'E. Coli CDSs.fna'
Here we build a model, based on the sequences of E. Coli K-12, that Glimmer3 can use.
output = subprocess.check_output([ 'glimmer3', # Our program 'Escherichia virus T4.fna', # The input genome 'test.icm', # The model we just built 't4-genes' # The base name for output files. It'll produce t4-genes.detail and t4-genes.predict. ]).decode('utf-8') # And of course we decode as utf-8 print(output)
glimmer3 'Escherichia virus T4.fna' test.icm t4-genes
What happened here? The output of the program was written to
stdout, so Python may print that out to your screen, but
output will be empty. To solve this common problem we can re-run the program and collect both
output = subprocess.check_output([ 'glimmer3', # Our program 'Escherichia virus T4.fna', # The input genome 'test.icm', # The model we just built 't4-genes' # The base name for output files. It'll produce t4-genes.detail and t4-genes.predict. ], stderr=subprocess.STDOUT).decode('utf-8') # And of course we decode as utf-8 print(output)
Here we’ve re-directed the
stdout and mixed both of them together. This isn’t always what we want, but here the program produces no output, and we can do that safely, and now we can parse it or do any other computations we need with it! Our Glimmer3 gene calls are in
t4-genes.predict if we want to open and process those as well.