tmlib.workflow.illuminati package

Module contents

Workflow step for creation of pyramid images.

To achieve efficient zoomable visualization of terabyte-size microscopy image dataasets accross multiple resolution levels, images need to be represented in pyramid <https://en.wikipedia.org/wiki/Pyramid_(image_processing)> _ format. To this end, the illuminati step casts images to 8-bit and tiles them up according to available positional information. Users further have the option to correct images for illumination artifacts and align them between acquisitions based on pre-calculated statistics (if available).

Submodules

tmlib.workflow.illuminati.api module

class tmlib.workflow.illuminati.api.PyramidBuilder(experiment_id)

Bases: tmlib.workflow.api.WorkflowStepAPI

Parameters:

experiment_id: int

ID of the processed experiment
collect_job_output(batch)

Creates MapobjectType instances for Site, Well, and Plate types and creates for each object of these classes an instance of Mapobject and MapobjectSegmentation. This allows visualizing these objects on the map and using them for efficient spatial queries.

Parameters:

batch: dict

job description
create_run_batches(args)

Creates job descriptions for parallel computing.

Parameters:

args: tmlib.workflow.illuminati.args.IlluminatiBatchArguments

step-specific arguments
Returns:

generator

job descriptions
create_run_jobs(user_name, job_collection, verbosity, duration, memory, cores)

Creates jobs for the parallel “run” phase of the step. The illuminati step is special in the sense that it implements multiple sequential runs within the “run” phase to build one pyramid zoom level after another.

Parameters:

user_name: str

name of the submitting user

job_collection: tmlib.workflow.jobs.RunPhase

emtpy collection for “run” jobs

verbosity: int

logging verbosity for jobs

duration: str

computational time that should be allocated for a single job; in HH:MM:SS format

memory: int

amount of memory in Megabyte that should be allocated for a single

cores: int

number of CPU cores that should be allocated for a single job
Returns:

tmlib.workflow.jobs.RunPhase

collection of jobs
create_run_phase(submission_id, parent_id)

Creates a job collection for the “run” phase of the step.

Parameters:

submission_id: int

ID of the corresponding Submission

parent_id: int

ID of the parent WorkflowStep
Returns:

tmlib.workflow.job.MultiRunPhase

collection of “run” jobs
delete_previous_job_output()

Deletes all instances of ChannelLayer and ChannelLayerTile as well as instances of MapobjectType, Mapobject and MapobjectSegmentation for each Plate, Well and Site.

has_collect_phase = True
run_job(batch, assume_clean_state=False)

Creates 8-bit grayscale JPEG layer tiles.

Parameters:

batch: dict

batches element

assume_clean_state: bool, optional

assume that output of previous runs has already been cleaned up

tmlib.workflow.illuminati.args module

class tmlib.workflow.illuminati.args.IlluminatiBatchArguments(**kwargs)

Bases: tmlib.workflow.args.BatchArguments

Parameters:

**kwargs: dict, optional

keyword arguments to overwrite
align

bool: whether images should be aligned between multiplexing cycles

batch_size

int: number of image files that should be processed per job

clip

bool: whether images intensities should be clipped

clip_percent

float: threshold percentile at which image intensities should be clipped

clip_value

int: threshold value at which image intensities should be clipped (defaults to 99.99th percentile; the set value overwrites calculated percentile)

illumcorr

bool: wether images should be corrected for illumination artifacts

class tmlib.workflow.illuminati.args.IlluminatiSubmissionArguments(**kwargs)

Bases: tmlib.workflow.args.SubmissionArguments

Parameters:

**kwargs: dict, optional

keyword arguments to overwrite

tmlib.workflow.illuminati.cli module

class tmlib.workflow.illuminati.cli.Illuminati(api_instance, verbosity)

Bases: tmlib.workflow.cli.WorkflowStepCLI

Creation of pyramids for interactive, web-based visualization of images.

Parameters:

api_instance: tmlib.workflow.illuminati.api.PyramidBuilder

instance of API class to which processing is delegated

verbosity: int

logging level

illuminati

Creation of pyramids for interactive, web-based visualization of images.
experiment_id

ID of the experiment that should be processed

-h, --help

show this help message and exit

--verbosity, -v

increase logging verbosity

illuminati experiment_id cleanup

cleans up the output of a previous submission, i.e. removes files and database entries created by previously submitted jobs
-h, --help

show this help message and exit

illuminati experiment_id collect

collects the output of run jobs, i.e. performs a post-processing operation that either cannot be parallelized or needs to be performed afterwards
-h, --help

show this help message and exit

illuminati experiment_id info

prints the description of a given batch job to the console
-h, --help

show this help message and exit

--job, -j

ID of the job for which information should be displayed

--phase {collect,run}, -p {collect,run}

phase of the workflow step to which the job belongs

illuminati experiment_id init

creates batches for parallel processing and thereby defines how the computational task should be distrubuted over the cluster (also cleans up the output of previous submissions)
-h, --help

show this help message and exit

--align, -a

whether images should be aligned between multiplexing cycles

--batch-size <batch_size>, -b <batch_size>

number of image files that should be processed per job (default: 100)

--clip, -c

whether images intensities should be clipped

--clip-percent <clip_percent>

threshold percentile at which image intensities should be clipped (default: 99.9)

--clip-value <clip_value>

threshold value at which image intensities should be clipped (defaults to 99.99th percentile; the set value overwrites calculated percentile)

--illumcorr, -i

wether images should be corrected for illumination artifacts

illuminati experiment_id log

prints the log output of a given batch job to the console
-h, --help

show this help message and exit

--job, -j

ID of the job for which log output should be shown

--phase {collect,init,run}, -p {collect,init,run}

phase of the workflow step to which the job belongs

illuminati experiment_id resubmit

resubmits previously created jobs for “run” and “collect” phases to the cluster and monitors their status upon processing
-h, --help

show this help message and exit

--depth <index>, -d <index>

number of child tasks that should be monitored (default: 1)

--interval <seconds>, -i <seconds>

seconds to wait between monitoring iterations (default: 10)

illuminati experiment_id run

runs an invidiual batch job on the local machine
-h, --help

show this help message and exit

--assume-clean-state

assume that previous outputs have been cleaned up

--job <job_id>, -j <job_id>

ID of the job that should be run

illuminati experiment_id submit

creates batch jobs for the “run” and “collect” phases, submits them to the cluster and monitors their status upon processing (requires a prior “init”)
-h, --help

show this help message and exit

--depth <index>, -d <index>

number of child tasks that should be monitored (default: 1)

--interval <seconds>, -i <seconds>

seconds to wait between monitoring iterations (default: 10)

--cores <number>

number of cores that should be allocated to each “run” job (may be increased in case memory requirements of a job exceed resources of a single core) (default: 1)

--duration <hh:mm:ss>

walltime that should be allocated to a each “run” job in the format “HH:MM:SS” (may need to be adapted depending on the choice of batch size) (default: 06:00:00)

--memory <mb>

amount of memory in megabytes that should be allocated to each “run” job (default: 2147)

tmlib.workflow.illuminati.stitch module

tmlib.workflow.illuminati.stitch.calc_grid_coordinates_from_layout(stitch_dims, stitch_layout)

Determines the position of each image in the stitched mosaic image.

Parameters:

stitch_dims: Dict[str, int]

number of rows (“n_rows”) and number of columns (“n_cols”) of the stitched mosaic image

stitch_layout: str

“horizontal”, “zigzag_horizontal”, “vertical”, or “zigzag_vertical”
Returns:

List[Tuple[int]]

pair of zero-based “row” (y) and “column” (x) position indices for each individual image in the stitched mosaic image in the order of acquisition, i.e. sorted according to image site
Raises:

TypeError

when stitch_layout doesn’t have type string

ValueError

when stitch_layout is not in the set of the possible options
tmlib.workflow.illuminati.stitch.calc_grid_coordinates_from_positions(stage_positions, n, reverse_rows=False, reverse_columns=False)

Calculates the relative position of each image within the acquisition grid. The coordinates are one-based to be consistent with the OME data model.

Parameters:

stage_positions: List[Tuple[float]]

absolute microscope stage positions

n: int

number of expected grid coordinates

reverse_rows: bool, optional

sort positions along row dimension in descending order

reverse_columns: bool, optional

sort positions along column dimension in descending order
Returns:

List[Tuple[int]]

relative positions (zero-based coordinates) within the grid
tmlib.workflow.illuminati.stitch.calc_stitch_dimensions(stage_positions)

Determines stitch dimensions from stage positions.

Parameters:

stage_positions: List[Tuple[float]]

stage positions in row (x) and column (y) direction
Returns:

Tuple[int]

number of rows and columns of the stitched mosaic image
tmlib.workflow.illuminati.stitch.calc_stitch_layout(stitch_dims, stage_positions)

Determines the stitch layout of the mosaic image, i.e. in which order individual images need to be stitched together.

Parameters:

stitch_dims: Dict[str, int]

number of rows and columns of the stitched mosaic image

stage_positions: Tuple[float]

stage positions in row (x) and column (y) direction
Returns:

str

stitch layout: “horizontal”, “zigzag_horizontal”, “vertical”, or “zigzag_vertical”
Raises:

ValueError

when stitch layout can’t be determined
tmlib.workflow.illuminati.stitch.guess_stitch_dimensions(n_sites, stitch_major_axis='vertical')

Guesses dimensions of a stitched mosaic image. In case n_sites is not divisible by two, a larger mosaic will be selected that fits n_sites.

Parameters:

n_sites: int

total number of sites (individual images) in the stitched mosaic image

stitch_major_axis: str, optional

"horizontal" if there are more columns than rows and "vertical" otherwise (default: "vertical")
Returns:

Tuple[int]

number of rows and columns of the stitched mosaic image
Raises:

TypeError

when value of n_sites is not an integer

ValueError

when value of stitch_major_axis is neither "horizontal" nor "vertical" or when value of n_sites is not a positive integer

IndexError

when dimensions cannot be determined