Installing Pulsar

Tip

This documentation covers installing Pulsar by hand. The tutorial Running Jobs on Remote Resources with Pulsar in the Galaxy Training Network contains a step-by-step guide for installing Pulsar using Ansible.

There are two primary ways to deploy Pulsar. The newer and preferred method is to install Pulsar from PyPI using the standard pip and venv Python tools.

The older method also requires these tools to install Pulsar’s dependencies but Pulsar itself is served directly from a clone of the Pulsar source tree - this mirrors how Galaxy is most typically deployed. This may be beneficial during Pulsar development and is required for certain experimental features such as Mesos support.

Both methods presented here require a Python 3.5 (or later) runtime for either Windows, Linux, or macOS. Python 2 is no longer supported as of the 0.14.0 release of Pulsar.

These instructions also require venv. Open a console on your machine and type python3 -m venv - if the module is missing you will need to install it. It is part of any full Python installation, but some Linux distributions (such as Debian and its derivatives) package it separately. On Debian systems, you can

From PyPI

Start by creating a directory for the Pulsar configuration files and setting up a venv to install Pulsar into using the following three commands.:

$ mkdir pulsar
$ cd pulsar
$ python3 -m venv venv

Next, activate this newly created venv. From a Linux or macOS terminal, this can be done with the command . venv/bin/activate and in Windows you can type venv\Scripts\activate.

Next install Pulsar using pip.:

$ pip install 'pulsar-app[web]'

If you are planning to use Pulsar in the optional message queue mode rather than traditional web mode (see the configuration docs), you can drop [web] from the install command (this extra qualifier causes web serving packages to be installed).

Next, create the required configuration files for use with Pulsar in this directory by running the following command.:

$ pulsar-config

The pulsar-config script can bootstrap various Pulsar deployment options, run pulsar-config --help for full details. For instance, Pulsar can be configured to monitor a message queue and skip the web server configuration - enable this by passing --mq to pulsar-config. Another useful option is --supervisor which will generate a Supervisord configuration for this directory and install Supervisord.

pulsar-config installs a few files into this directory. app.yml contains Pulsar configuration options and server.ini contains web server related information (it will not exist if configured --mq).

Launching Pulsar

The Pulsar server can be started by running:

$ pulsar [--daemon]

Under Linux and macOS the --daemon argument can be supplied to run Pulsar as a daemon, and stopped with pulsar --stop-daemon. If --daemon is not supplied, Pulsar will just run in the foreground (the only option for Windows).

The Pulsar deployment can be tested by running the following command, which will submit an example job and wait for its completion.:

$ pulsar-check

If Pulsar is not running on the default port 8913, pulsar-check should be called with an explicit URL using the argument --url=http://localhost:8913. Likewise if a private token has been configured it can be supplied using --private_token=<token>.

From Source

Alternatively, Pulsar can be obtained from GitHub using the following command and ran directly from the source tree (like Galaxy is traditionally deployed):

$ git clone https://github.com/galaxyproject/pulsar

The following steps assume your current working directory is the newly created pulsar directory.:

$ cd pulsar

Create a new Python virtual environment called .venv in the pulsar root directory:

$ python3 -m venv .venv

Activate environment (varies by OS). From a Linux or macOS terminal:

$ . .venv/bin/activate

Or from a Windows terminal:

$ .venv\Scripts\activate.bat

Finally, install Pulsar’s required dependencies into the virtual environment:

$ pip install -r requirements.txt

If using the standard webserver, it can be installed with:

$ pip install Paste PasteScript

Launching Pulsar

Before launching Pulsar, it may make sense to copy over the sample configuration files. server.ini is used to describe web server related properties and app.yml is used for Pulsar application-related configuration files:

$ cp server.ini.sample server.ini
$ cp app.yml.sample app.yml

Pulsar should now be launchable via the run.sh script under Linux or macOS or using the run.bat script under Windows. So under Linux or macOS, Pulsar can be launched with:

$ ./run.sh [--daemon]

This daemon can be stopped using ./run.sh --stop-daemon. When run as a daemon, Pulsar will log to the file paster.log. If --daemon is not supplied, Pulsar will just run in the foreground.

Under Windows, Pulsar can be started using:

$ run.bat

and will run as long as that process is alive and log to standard output. A test job can be submitted using the command:

$ python run_client_tests.py

If Pulsar’s server.ini has been modified and it is not running on the default port 8913, run_client_tests.py should be called with an explicit URL using the argument --url=http://localhost:8913. Likewise if a private token has been configured it can be supplied using --private_token=<token>.

Pulsar Webservers

Pulsar’s default webserver (if web dependencies are installed) is Paste. However, uWSGI or circus will be used instead, if found.

A precompiled version of uWSGI can be installed with:

$ pip install pyuwsgi

Or compiled from source with:

$ pip install uWSGI

Pulsar Dependencies

Several Python packages must be installed to run the Pulsar server. The core set of required dependencies were installed during the Pulsar installation in the previous section. Additional dependencies are required for features such submitting to a cluster (drmaa), communicating via message queue (kombu), etc…. Most of the time these can just be installed with pip install <dependency_name>.