tmserver.extensions.gc3pie package

Module contents

class tmserver.extensions.gc3pie.GC3Pie(app=None)

Bases: object

An extension that exposes a GC3Pie engine to manage computational tasks.

Parameters:

app: flask.Flask, optional

flask application (default: None)

Examples

gc3pie = GC3Pie() gc3pie.init_app(app)

continue_task(task)

Continues interrupted task.

Parameters:

task: gc3libs.Task

computational task

get_id_of_most_recent_submission(experiment_id, program)

Gets the ID of the most recent Submission.

Parameters:

experiment_id: int

ID of the processed experiment

program: str

name of the program that submitted the task

Returns:

int

get_id_of_most_recent_task(experiment_id, program)

Gets the ID of the top level task for the given experiment that was most recently submitted by program.

Parameters:

experiment_id: int

ID of the processed Experiment

program: str

name of the program that submitted the task, e.g. "workflow"

Returns:

int

get_task_status(task_id, recursion_depth=None)

Gets the status of submitted task.

Parameters:

task_id: int

ID of the Task

recursion_depth: int, optional

recursion depth for querying subtasks

Returns:

dict

status of task

See also

tmlib.workflow.utils.get_task_status_recursively()

init_app(app)

Initializes the extension for a flask application. This will create a GC3Pie engine and start it in the background using the “gevent” scheduler.

Parameters:

app: flask.Flask

flask application

See also

tmserver.extensions.gc3pie.engine.BGEngine

kill_task(task)

Kills submitted task.

Parameters:

task: gc3libs.Task

computational task

resubmit_task(task, index=0)

Resubmits a task.

Parameters:

task: gc3libs.Task

computational task

index: int, optional

index of an individual task within a sequential collection of tasks from where all subsequent tasks should be resubmitted

retrieve_most_recent_task(experiment_id, program)

Retrieves the top level task for the given experiment from the store that was most recently submitted by program.

Parameters:

experiment_id: int

ID of the processed Experiment

program: str

name of the program that submitted the task, e.g. "workflow"

Returns:

gc3libs.Task

retrieve_task(task_id)

Retrieves a task from the store.

Parameters:

task_id: int

persistent task ID

Returns:

gc3libs.Task

computational task

store_task(task)

Stores task in the database.

Parameters:

task: gc3libs.Task

computational task or collection of computational tasks

submit_task(task)

Submits task.

Parameters:

task: gc3libs.Task

computational task

Submodules

tmserver.extensions.gc3pie.engine module

class tmserver.extensions.gc3pie.engine.BgEngine(lib, *args, **kwargs)

Bases: object

A GC3Pie Engine instance that runs in the background.

A BgEngine exposes the same interface as a regular Engine class, but proxies all operations for asynchronous execution by the wrapped Engine instance. In practice, this means that all invocations of Engine operations on a BgEngine always succeed: errors will only be visible in the background thread of execution.

Users can define a custom callback function that gets invokes after each Engine.progress() call and applied to each task managed by the engine. To this end, set the attribute progress_callback.

Parameters:

lib: str

library for scheduler, either "threading" or "gevent"

args: list, optional

additional arguments as array, the first and only element must be an instance of gc3lib.core.Engine

kwargs: dict, optional

additional arguments that can be parsed to the Engine instance as a mapping of key-value pairs

add(task)

close()

fetch_output(task, output_dir=None, overwrite=False, changed_only=True, **extra_args)

find_task_by_id(task_id)

free(task, **extra_args)

get_backend(name)

get_resources()

iter_tasks()

Iterate over all tasks managed by the Engine.

kill(task, **extra_args)

peek(task, what='stdout', offset=0, size=None, **extra_args)

progress()

Proxy to Engine.progress.

If the background thread is already running, this is a no-op, as progressing tasks is already taken care of by the background thread. Otherwise, just forward the call to the wrapped engine.

redo(task, index)

remove(task)

select_resource(match)

start(interval)

Starts triggering the main loop every interval seconds.

Parameters:

interval: int

looping interval for the scheduler

stats(only=None)

stop(wait=False)

Stops background execution of the main loop.

Parameters:

wait: bool

to wait until all submitted jobs have been executed

submit(task, resubmit=False, targets=None, **extra_args)

update_job_state(*tasks, **extra_args)

tmserver.extensions.gc3pie.engine.at_most_once_per_cycle(fn)

Ensures the decorated function is not executed more than once per each poll interval.

Cached results are returned instead, if Engine.progress() has not been called in between two separate invocations of the wrapped function.