Pipeline config files

Pipeline config files#

PyOPIA makes use of TOML for configuration files, which can used loaded and given to pyopia.pipeline.Pipeline, or passed directly from command line, like this: pyopia process config.toml (‘Commmand line tools’ page).

Config files can be loaded for use in scripts or notebooks using:

toml_settings = pyopia.io.load_toml('config.toml')

Main components of the config#

The [general] section contains information that applies generally to the dataset, or to several steps within a pipeline.

The [steps] section contains sub-steps describing the pyopia.pipeline class and input arguments that perform each step of the processing.

Examples#

SilCam#

Below is a typical example for a SilCam processing pipeline. This setup could be used for other standard images by adapting the load function.

[general] # general settings independent of processing steps
raw_files = 'raw_data/*.silc' # string used to obtain list of raw data files for processing
pixel_size = 24 # pixel size of imaging system in microns
log_level = 'INFO' # (defaults to INFO) sets the level of printed output or details in the log (see python logging library for details)
log_file = 'pyopia.log' # (optional) path to logfile - logfile not written if not defined here

[steps] # setup of analysis pipeline order, functions, and parameters

    [steps.classifier]
    pipeline_class = 'pyopia.classify.Classify'
    model_path = 'keras_model.keras' # path to trained nn model

    [steps.load]
    pipeline_class = 'pyopia.instrument.silcam.SilCamLoad'

    [steps.correctbackground]
    pipeline_class = 'pyopia.background.CorrectBackgroundAccurate'
    average_window = 10 # number of images used to create background
    bgshift_function = 'accurate' # optional 'fast' or 'accurate' method for moving backgrounds. For static background use 'pass' or comment this line.

    [steps.imageprep]
    pipeline_class = 'pyopia.instrument.silcam.ImagePrep'
    image_level = 'imraw' # the level of processing for further analysis. Either 'imraw' for ignoring background or 'im_corrected' for using the backgroun-corrected image. Defaults to 'imc' if not defined.

    [steps.segmentation]
    pipeline_class = 'pyopia.process.Segment'
    threshold = 0.85 # threshold used for segmentation
    segment_source = "im_minimum" # the image used for segmentation

    [steps.statextract]
    pipeline_class = 'pyopia.process.CalculateStats'
    propnames = ['major_axis_length', 'minor_axis_length', 'equivalent_diameter', 'solidity'] # optional parameters to request from skimage.regionprops when calculating particle geometries. Defaults to ['major_axis_length', 'minor_axis_length', 'equivalent_diameter']. 'solidity' is needed for oil and gas analysis to help remove occluded particles.
    export_outputpath = 'exported_rois' # Path to folder to put extracted particle ROIs (in h5 files). Required for making montages later. Leave this option out if you don't want to export ROIs
    roi_source = "imref" # the image used to extract ROIs, and give to the classifier

    [steps.output]
    pipeline_class = 'pyopia.io.StatsToDisc'
    output_datafile = 'proc/test' # prefix path for output nc file

Holo#

Here is a typical configuration of a holographic reconstruction pipeline.

[general] # general setting independent of processing steps
raw_files = 'holo_test_data_01/*.pgm' # string used to obtain list of raw data files for processing
pixel_size = 4.4 # pixel size of imaging system in microns'wavelength = 658, # laser wavelength in nm

[steps] # setup of analysis pipeline order, functions, and parameters
    [steps.initial]
    pipeline_class = 'pyopia.instrument.holo.Initial'
    # hologram reconstruction settings
    wavelength = 658 # laser wavelength in nm
    n = 1.33 # index of refraction of sample volume medium (1.33 for water)
    offset = 27 # offset to start of sample volume in mm
    minZ = 0 # minimum reconstruction distance within sample volume in mm
    maxZ = 50 # maximum reconstruction distance within sample volume in mm
    stepZ = 0.5 #step size in mm

    [steps.load]
    pipeline_class = 'pyopia.instrument.holo.Load'

    [steps.correctbackground]
    pipeline_class = 'pyopia.background.CorrectBackgroundAccurate'
    average_window = 1 # number of images used to create background
    bgshift_function = 'accurate' # optional 'fast' or 'accurate' method for moving backgrounds. For static background use 'pass' or comment this line.

    [steps.reconstruct]
    pipeline_class = 'pyopia.instrument.holo.Reconstruct'
    stack_clean = 0.02

    [steps.focus]
    pipeline_class = 'pyopia.instrument.holo.Focus'
    stacksummary_function = 'max_map'
    threshold = 1
    increase_depth_of_field = true
    focus_function = 'find_focus_sobel'
    merge_adjacent_particles = 2

    [steps.segmentation]
    pipeline_class = 'pyopia.process.Segment'
    threshold = 0.99 # threshold used for segmentation
    segment_source = 'im_focussed' # the image used for segmentation

    [steps.statextract]
    pipeline_class = 'pyopia.process.CalculateStats'
    propnames = ['major_axis_length', 'minor_axis_length', 'equivalent_diameter', 
                              'feret_diameter_max', 'equivalent_diameter_area']
    roi_source = 'im_focussed'

    [steps.output]
    pipeline_class = 'pyopia.io.StatsToDisc'
    output_datafile = 'proc-holo-singleimage/test' # prefix path for output nc file

UVP#

Here is a typical configuration for basic analysis of raw UVP data.

[general]
raw_files = "uvp_data/*.png"
pixel_size = 80

[steps]

    [steps.classifier]
    pipeline_class = "pyopia.classify.Classify"
    model_path = "model/silcam-classification_database_20240822-200-20240829T091048-best-epoch.keras"

    [steps.load]
    pipeline_class = "pyopia.instrument.uvp.UVPLoad"

    [steps.segmentation]
    pipeline_class = "pyopia.process.Segment"
    threshold = 0.95
    segment_source = "imraw"

    [steps.statextract]
    pipeline_class = "pyopia.process.CalculateStats"
    roi_source = "imraw"
    export_outputpath = "uvp_rois"

    [steps.output]
    pipeline_class = "pyopia.io.StatsToDisc"
    output_datafile = "proc/uvp-test"
Contents