Metadata-Version: 2.1
Name: biomaj_process
Version: 3.0.19
Summary: BioMAJ process service
Home-page: http://biomaj.genouest.org
Download-URL: http://biomaj.genouest.org
Author: Olivier Sallou
Author-email: olivier.sallou@irisa.fr
Classifier: Development Status :: 5 - Production/Stable
Classifier: Environment :: Console
Classifier: Natural Language :: English
Classifier: Operating System :: POSIX :: Linux
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.4
License-File: LICENSE
Requires-Dist: biomaj_core
Requires-Dist: biomaj_zipkin
Requires-Dist: pika==0.13.0
Requires-Dist: redis
Requires-Dist: PyYAML
Requires-Dist: flask
Requires-Dist: python-consul
Requires-Dist: prometheus_client>=0.0.18
Requires-Dist: protobuf
Requires-Dist: requests

# About

[![PyPI version](https://badge.fury.io/py/biomaj-process.svg)](https://badge.fury.io/py/biomaj-process)

Microservice to manage the process execution of biomaj.

A protobuf interface is available in biomaj_process/message/message_pb2.py to exchange messages between BioMAJ and the download service.
Messages go through RabbitMQ (to be installed).

# Protobuf

To compile protobuf, in biomaj_process/message:

    protoc --python_out=. message.proto

# Development

    flake8  biomaj_process

    CONFIG=tests/config.yml python -m pytest -v tests/biomaj_tests.py

# Run

## Message consumer:

export BIOMAJ_CONFIG=path_to_config.yml
python bin/biomaj_process_consumer.py

## Web server

If package is installed via pip, you need a file named *gunicorn_conf.py* containing somehwhere on local server:

    def worker_exit(server, worker):
        from prometheus_client import multiprocess
        multiprocess.mark_process_dead(worker.pid)

If you cloned the repository and installed it via python setup.py install, just refer to the *gunicorn_conf.py* in the cloned repository.


    export BIOMAJ_CONFIG=path_to_config.yml
    rm -rf ..path_to/prometheus-multiproc
    mkdir -p ..path_to/prometheus-multiproc
    export prometheus_multiproc_dir=..path_to/prometheus-multiproc
    gunicorn -c ..path_to/gunicorn_conf.py biomaj_download.biomaj_process_web:app

Web processes should be behind a proxy/load balancer, API base url /api/process

A prometheus /metrics endpoint is exposed on web server


3.0.19
  Replace nose by pytest
3.0.18:
  Normalize yaml setup with other biomaj components
3.0.17:
  Fix yaml load warnings
3.0.16:
  In case of proxy comm failure, sleep a few seconds before retry
3.0.15:
  Add proc name in logs for waiting and result
3.0.14:
  Add env var BIOMAJ_HOST_DATA_DIR to use with *docker* process executor to specify data.dir location if different between host and container (if BIOMAJ_DATA_DIR not a subdirectory of BIOMAJ_DIR for example)
    If variable is not set, container wil mount data.dir between host and container,
    else will mount BIOMAJ_HOST_DATA_DIR:data.dir
3.0.13:
  Fix traefik prefix
3.0.12:
  Update pika dependency release
  Add tags for traefik support
3.0.11:
  Catch exception for microservices if remote server is not reachable
  Rename package for protobuf to avoid collisions
3.0.10:
  README doc fixes
  Add dependency banks additional volumes for Docker
3.0.9:
  Allow local_endpoint definition per service
  Fix volumes for docker with dependency banks
3.0.8:
  Fix #1, DRMAA needs colon in front of output and error path (mandatory depending on versions)
3.0.7:
  fix case if process desc or type is missing in bank properties
3.0.6:
  disable web thread logging
3.0.5:
  retry in case of contact failure with proxy
3.0.4:
  for docker, restrict directory to bank directory and dependencies
  add host in prometheus stats
3.0.3:
  fix prometheus + gunicorn multi process
  add consul supervision
3.0.2:
  add logging info
  execute docker processes in biomaj micro service
3.0.1:
  bug fixes
3.0.0:
  move process management out of biomaj main package
