Parallel processing class

This module helps use multiple processes (cores) to process multiple images simultaneously.

class library.image_manipulation.parallel_manager.ParallelManager

Bases: object

Methods to support processing any part of pipeline (discreet function) using multiple cores

get_nworkers()

Get the number of cores to use per workstation. The same number is used for both downsampled and full resolution images. There is little point in setting two different levels in one host. Much effort was used to set these numbers. The number must be 2 or greater

run_commands_concurrently(function, file_keys, workers)

This method uses the ProcessPoolExecutor library to run multiple processes at the same time. It also has a debug option. This is helpful to show errors on stdout.

Parameters:

• function – the function to run

• file_keys – tuple of file information

• workers – integer number of workers to use

run_commands_with_threads(function, file_keys, workers)

This method uses the ThreadPoolExecutor library to run multiple processes at the same time. It also has a debug option. This is helpful to show errors on stdout.

Parameters:

• function – the function to run

• file_keys – tuple of file information

• workers – integer number of workers to use