Prerequisites for building software/conda packages

</span></div>

</span></div>

---

# Prerequisites for building software/conda packages

<div class="contributors-line">
		Authors: 
<a href="/training-material/hall-of-fame/abretaud/" class="contributor-badge contributor-abretaud"><img src="/training-material/assets/images/orcid.png" alt="orcid logo"/><img src="https://avatars.githubusercontent.com/abretaud?s=27" alt="Avatar">Anthony Bretaudeau</a>

<a href="/training-material/hall-of-fame/cmonjeau/" class="contributor-badge contributor-cmonjeau"><img src="https://avatars.githubusercontent.com/cmonjeau?s=27" alt="Avatar">Cyril Monjeaud</a>

</div>

<div class="footnote" style="bottom: 4 em;"><i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated: May 27, 2021</div>

<div class="footnote" style="bottom: 2.5em;"><i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a></div>
<div class="footnote" style="bottom: 1em;"><strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

### <i class="far fa-question-circle" aria-hidden="true"></i><span class="visually-hidden">question</span> Questions

- What does 'installing a software' means on a Linux architecture?

- Why my compilations always fail?

- How to solve common compiling and installation issues?

---

### <i class="fas fa-bullseye" aria-hidden="true"></i><span class="visually-hidden">objectives</span> Objectives

- Learn how to compile and install tools using standard procedures.

- Learn needed tricks to write conda build.sh files.

---

### Environment variables

```sh
$ MY_NAME="Bobby"
$ echo $MY_NAME
Bobby
```

```sh
$ MY_DATE=$(date)
$ echo $MY_DATE
Wed Feb 14 12:12:21 CET 2018
```

Use export to make sure the variable is accessible to any script/program you run from the current shell.

```sh
$ export MY_DATE=$(date)
$ echo $MY_DATE
Wed Feb 14 12:12:21 CET 2018
$ bash some_script.sh # some_script.sh will have access to $MY_DATE
```

Many environment variables predefined in a shell: PATH, HOSTNAME, HOME, LANG, USER, ...

---

### Show me the PATH

```sh
$ the_binary --help
```

How does the system knows where to find the binary?

PATH is an environment variable defining possible locations of binaries.

```sh
$ echo $PATH
/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
```

Paths separated by `:`, ordered list (highest priority first).

As all environment variable, you can redefine it:

```sh
$ export PATH="/opt/xxx/bin/:$PATH"
$ echo $PATH
/opt/xxx/bin/:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin
```

---

### Show me the PATH

The command `which` lets you know which binary will be used by your shell.

```sh
$ which the_binary
/opt/xxx/bin/the_binary
```

It throws an error if the binary is not "in the PATH".

```sh
$ export PATH="/usr/bin/"
$ echo $PATH
/usr/bin
```
```sh
$ which the_binary
/usr/bin/which: no the_binary in (/usr/bin)
```

---

### Thinking outside the PATH

What if you want to run a binary only located in the current working directory?

```sh
$ ls
my_binary
$ my_binary --help
bash: my_binary: command not found...
```

You need to add `./` to explicitly tell the shell to run the file from working directory:

```sh
$ ./my_binary --help
It works!
```

`./` is just the relative path to `my_binary`, it could be a more complicated one, or an absolute path:

```sh
$ ../somewhere/my_binary --help
It works!
$ /home/someone/womewhere/my_binary --help
It works!
```

---

### Installing a single binary

Just a single executable file (binary, script).

You only need to copy it to a `bin` directory and make sure permissions are set:

```sh
$ cp the_binary /usr/local/bin
$ chmod a+x /usr/local/bin/the_binary
```

Other possible `bin` directories:

```sh
/bin            = reserved for system
/usr/bin        = installed by package manager (apt, yum, ...)
/usr/local/bin  = manually installed binaries
/opt/xxx/bin    = manually installed binaries (xxx=path)
/yyy/bin        = a bin directory wherever you like (yyy=path)
```

.center[.footnote[If you install a precompiled binary, make sure that it was compiled for the same architecture as your system (e.g. x86_64)]]

---

### Compiling a single binary

In the following slides we consider a program written in `C`.

Usually, compilation is done with `make`.

```sh
$ ls
Makefile    my_program.c
```
```sh
$ make
```
```sh
$ ls
Makefile    my_program.c    my_binary
```

`make` will read the instructions defined in the file `Makefile` and run the compiler automatically to produce the binary.

You can then copy the binary to a bin dir.

Some exotic tools come with other scripts or methods for compiling, read the README or INSTALL files.

---

### Compiling a library

Libraries are reusable chunks of code. They are not executable as is. They are compiled similarly as binaries.

```sh
$ ls
Makefile    my_lib.c    my_lib.h
```
```sh
$ make
```
```sh
$ ls
Makefile    my_lib.c    my_lib.h    libmy.so.1.2.8
```

`.h` files are code files defining which functions are provided by the library.

`.so` are compiled binary code. Their file name structure is important: `libXXXX.so.version`.

---

### Installing a library

```sh
$ cp libmy.so.1.2.8 /usr/local/lib/
```

Some symbolic links need to be created:

```sh
$ ln -s /usr/local/lib/libmy.so.1.2.8 /usr/local/lib/libmy.so
$ ln -s /usr/local/lib/libmy.so.1.2.8 /usr/local/lib/libmy.so.1
```

You also need to install `headers` in a `include` dir:

```sh
$ cp my_lib.h /usr/local/include/
```

Other possible `lib` (or `lib64`) directories (same principle for `include`):

```sh
/lib(64)            = reserved for system
/usr/lib(64)        = installed by package manager (apt, yum, ...)
/usr/local/lib(64)  = manually installed
/opt/xxx/lib(64)    = manually installed (xxx=path)
/yyy/lib(64)        = a lib directory wherever you like (yyy=path)
```

---

### make install

Copying/symlinking manually each file is painful.

Running `make install` usually install everything automatically for you.

```sh
$ make
$ make install
```

By default, installs to `/usr/local`. We will see how to change this later.

---

### Compiling a binary that uses an external library

For example, if your program relies on `zlib` to read or create gzipped files.

`make` needs to know where are the `.h` and `.so` files to compile your program properly.

Usually, a script named 'configure' is distributed with the program sources.

```sh
$ ls
configure   Makefile    my_program.c
```

This script explores the filesystem to find the needed `.h` and `.so` files. You can then run `make` and `make install` as usual.

```sh
$ ./configure
$ make
$ make install
```

Sometimes, `configure` is replaced by another software like cmake.

---

### Playing with ./configure

You can pass many options to the `configure` script.

A very common one is `--prefix` which allows to change the installation directory.

```sh
$ ./configure --prefix=/home/somewhere
$ make
$ make install
```

The binary will be installed in `/home/somewhere/bin/`

If you are compiling a library, files will be installed in `/home/somewhere/include/` and `/home/somewhere/lib(64)/`

---

### Playing with ./configure

You can pass many options to the `configure` script.

You can often disable or enable some software features this way.

```sh
$ ./configure --disable-gpu --enable-greedy-algorithm
$ make
$ make install
```

---

### Playing with ./configure

By default, `configure` only searches for `.h` and `.so` files in standard directories (`/usr/`, `/usr/local`).

If your program depends on a library installed in an exotic location, you need to specify it.

There might be a specific `configure` option.

```sh
$ ./configure --zlib-dir=/home/somewhere/zlib/
```

Or, you can define some standard environment variables:

```sh
$ export CFLAGS="-I/home/somewhere/zlib/include $CFLAGS"
$ export LDFLAGS="-L/home/somewhere/zlib/lib $LDFLAGS"
```

In some cases, you might need to define additional variables:

```sh
$ export CPATH="/home/somewhere/zlib/include:$CPATH"
$ export LIBRARY_PATH="/home/somewhere/zlib/lib:$LIBRARY_PATH"
```

---

### Playing with ./configure

The `CFLAGS` environment variable can also be used for other purposes like enabling some compiler optimisation, predefining some C macro or compiling with debugging symbols.

```sh
$ export CFLAGS="-I/home/somewhere/zlib/include -O2 -DDEBUG -g"
```

When you are compiling C++ code, you need to use `CXXFLAGS` instead of `CFLAGS`.

---

### Shebang

The first line of script is called the [shebang](https://en.wikipedia.org/wiki/Shebang_%28Unix%29).

```sh
#!/bin/bash
```

It determines how your script will be run when called in a shell.

```sh
$ my_script.sh
$ # is interpreted as
$ /bin/bash /usr/bin/my_script.sh
```

---

### Shebang

You need to write an absolute path in the shebang. But never do this:

```sh
#!/usr/bin/python
```

Because you are not sure /usr/bin/python will always be at this location. Preferred solution:

```sh
#!/usr/bin/env python
```

This will ensure to use the `python` found using the PATH environment variable.

`/bin/bash` or `/usr/bin/env` are considered to always be present.

---

### Python/Perl/R packages

Installing Python modules consists in placing the source files in the correct path inside the Python installation.

For example BioPython code will be installed in `/usr/lib/python3.6/site-packages/Bio/`

Some Python modules consists both of Python code and C code that need to be compiled.

Python modules are usually installed with specific setup mechanisms that take care of it all for you.

```sh
$ pip install my_module
```

```sh
$ cd my_module_src/
$ python setup.py install
```

Perl or R modules are installed in similar ways.

---

### Common error: Undefined symbol

Symptom: you get an `Undefined symbol` while running an installed program.

Cause: the program uses a library which is not available in the expected location.

To list all the libraries needed by the program and identify the one causing the problem:

```sh
$ ldd `which nano`
	linux-vdso.so.1 (0x00007ffdfb48c000)
	libmagic.so.1 => not found
	libncursesw.so.6 => /lib64/libncursesw.so.6 (0x00007f16cbb54000)
```

Make sure the library is correctly installed. If it is installed in an exotic location, use the `LD_LIBRARY_PATH` environment.

```sh
$ export LD_LIBRARY_PATH="/home/somewhere/magic/:$LD_LIBRARY_PATH"
$ ldd `which nano`
	linux-vdso.so.1 (0x00007ffdfb48c000)
	libmagic.so.1 => /home/somewhere/magic/lib/libmagic.so.1
	libncursesw.so.6 => /lib64/libncursesw.so.6 (0x00007f16cbb54000)
```

---
### <i class="fas fa-key" aria-hidden="true"></i><span class="visually-hidden">keypoints</span> Key points

- There is a common procedure to compile and install many tools: `./configure && make && make install`

- Some exotic tools require adjustements to compile or install properly

- Pay attention to INSTALL and README files, and to documentation

---

## Thank You!

This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!

<a href="/training-material/hall-of-fame/cmonjeau/" class="contributor-badge contributor-cmonjeau"><img src="https://avatars.githubusercontent.com/cmonjeau?s=27" alt="Avatar">Cyril Monjeaud</a>

</div>

</div>

<a rel="license" href="https://creativecommons.org/licenses/by/4.0/">
This material is licensed under the Creative Commons Attribution 4.0 International License</a>.