Semantic Segmentation Beta Version Business Processing Definition

1. Business Outline
1.1. Purpose
"An unknown image is segmented semantically by a semantic segmentation model learned using arbitrary image data."

The following shall be satisfied in order to achieve the purpose.

 The user can easily perform the area division with minimum parameter input.

 The user can check the learning result of the model (including the model accuracy) and perform the model tuning.

 The function is implemented in the form of connecting to the API on the APMI PGEditor.

1.2. Operational processing diagram

*From the viewpoint of licenses, etc., it is necessary to separately confirm whether all models existing in timm can be used.

1.3. Perspectives of licenses, etc.

It is necessary to eliminate imagenet, etc., which are prohibited for commercial use.

 Imagenet is prohibited for commercial use.

 There is no clear information on the use of parameters (pre-learned weights) learned by imagenet by third parties, so expert advice is

required.

 In addition to the legal point of view, imagenet terms of use exist.

2. Business Flow Details
2.1. User Input of Parameters (Learning)

2.1.1. Processing Contents

User sets parameters (Data paths, data extensions, learning conditions, etc.) on JupyterNotebook

No. Input Processing Output

1. User enters parameters on JupyterNotebook Input parameters

2. Input parameters Generate input.zip from input parameters and input dataset, including input.zip

Input dataset execution conditions and data

2.1.2. I/O List

 Params

No. File name Description

1. Params(class) User-specified parameters on JupyterNotebook

2. ├ input_dir(str) Have the following run-time configuration information

3. ├ epochs(int) Number of study sessions

4. ├ batch_size(int) Batch size in batch gradient descent

5. ├ learning_rate Initial (base) learning rate

(float)

6. ├ nfolds(int) Number of folds when doing cv

7. ├ random_crop Parameters for random cropping

(dict)

8. ├ flip(dict) Flip Parameters

9. ├ shift_scale_rotate Shift, Scale Change, and Image Rotation Parameters

(dict)

10. ├ bright_contrast Luminance Transformation Parameters

(dict)

11. ├ noise(dict) Parameters related to noise addition and blur

12. ├ class_def(dict) Mask pixel value (RGB)

13. ⎿ exp_name(str) Experiment name

 input.zip

row File name Description

number

1. input.zip A zip file containing a collection of input files.

For the analysis facility container, download and use from Amazon S3 before running the analysis.

2. ├ input.json (dict) Have runtime configuration information corresponding to:

3. │├ ml_params(dict) parameter list at learning

4. │├ aug_params(dict) data extension parameter list

5. │⎿ class_def(dict) mask pixel value (RGB)

6. │⎿ exp_name(str) experimental name

7. ⎿ input_files.zip zip file of the input dataset

8. ├ image Folder containing training images

9. │{input image n} .png source image file

10. ⎿ mask Folder containing mask image

11. {mask image n} .png Mask image file

 input.json > ml_params, aug_params,class_def,exp_name

No. File name Description

1. ml_params (dict) Learning Parameter List

2. ├ epochs (int) Number of Learning Times

3. ├ batch_size (int) Batch Size in Batch Gradient Descent

4. ├ learning_rate (float) Initial (Basis) Learning Rate

5. ⎿ nfolds (int) Number of folds for cv

6. aug_params (dict) Data extended parameter list

7. ├ random_crop (dict) Parameters for random cropping

8. │├ scale(list) Minimum/maximum ratio of cropped size to original image size

9. │├ p(float) Probability of random cropping being applied

10. ├ flip (dict) Flip Parameters

11. │├ p(float) Probability that the flip will be applied

12. ├shift_scale_rotate (dict) Parameters for shift, scale change, and image rotation

13. │├ scale_limit(float) Maximum scale ratio

14. │├ rotate_limit(int) Maximum rotation angle

15. │├ p(float) Probability that shift_scale_rotate will be applied

16. ├ bright_contrast(dict) Luminance transformation parameters

17. │├ p(float) Probability that luminance transformation is applied

18. ├ noise (dict) Parameters related to noise addition and blur

19. ⎿ p(float) Probability that noise is applied

20. class_def (dict) Mask pixel value (RGB)

21. ⎿ {class-n_name}(list) Contains the RGB value of class n

22. exp_name(str) Experiment name

*Only user-settable parameters are included in the job processing definition.

*Other parameters are included in the design document.

2.2. Semantic segmentation model learning

2.2.1. Processing content

Learning a segmentation model based on input parameters

The output destination area is shown below.

 Local: APMI PGEditor area

 System: APMI system area

 Push: AWS ECR area

No. Input Processing Output

 1. input.zip 1. Read input information, apply preprocessing (padding, splitting) (Push)

2. Create class_def.yml (mask pixel value), config.yml (model input input.json

parameter) from input.json image

3. Store pre-processed image and mask in splited_image, splited_mask mask

4. Store split image and mask information in dataframe class_def.yml

config.yml

splited_image

splited_mask

dataframe

2. class_def.yml Run GroupKFold for any Fold (Push)

config.yml Use the file name of the original image as the group learning_curve

splited_image 1. Split the dataset for training and validation models

splited_mask 2. Create NN model

dataframe 3. Learn NN model

4. Save NN model and learning curve

3. 1. Save Prediction Mask (Push)

2. Save Indicator Values for Each Image (csv) learning_curve

3. Save Feature Extraction Results from Mask Images for Each Image (csv) models.zip

4. Save Execution Information (txt) predicted

5. Save Preprocessing Information (pth, csv) result_table.csv

6. Convert models folder to zip file with password extracted_features.csv

7. Record experiment success or failure (json) summary.txt

8. Combine processing result files in output.zip output.json

output.zip

(Local)

output.zip

2.2.2. I/O Files

 Input Files

Field No. File name Description

1. input.zip A zip file containing a collection of input files.

For the analysis facility container, download and use from Amazon S3 before running the analysis.

2. ├ input.json(dict) Have the following run-time configuration information

3. │├ ml_params(dict) Learning-time parameter list

4. │├ aug_params(dict) Data-extended parameter list

5. │├ class_def(dict) Mask pixel value (RGB)

6. │⎿ exp_name(str) Experiment name

7. ⎿ input_files.zip Input dataset zip file

8. ├ image Folder containing training images

9. │{input image n} .png Original image file

10. ⎿ mask Folder containing mask image

 11. {mask image n} .png Mask image file

 Output file

No. File name Description

1. output.zip A zip file containing the entire output file

2. ├ log The folder containing the log file for learning execution

3. │├ stderr.log Log file during learning

4. │⎿ stdout.log Log file at the end of learning

5. ⎿ training Folder for storing learning results

6. ⎿ {exp_name} Folder to store learning results

7. ⎿ {yyyy_mmdd_hhmm_name} Folder to store learning results

8. ├ learning_curve Directory to store learning curve

9. │⎿ {name}.png Learning curve image (.png)

Present per Fold for cross-validation

10. ├ models.zip zip file containing learned weights, etc.

11. │├ {model_name}.pth Model weights

Present per Fold for cross-validation

12. │├ executed_config.yml learning condition

13. │⎿ preprocess_info.pth preprocessing information

14. ├ predicted Directory containing the image of the prediction mask

15. │⎿ {image_id}.png Prediction result mask image for original image {image_id} .png (.png)

16. ├ extracted_features.csv Feature amount extraction result from mask image for each image

Hereinafter, each feature amount

File name column File name

total_area Total pixels

npixcels_class_{name} Total number of pixels in the class

coverage_ratio_class_{name} Class coverage

areas_class_{name} Number of pixels per object in the

class (array)

coordinates_class_mudrock Coordinates of the class object

(array)

narea_class_{name} Number of class objects

17. ├ result_table.csv Evaluation index calculation result for each image (with evaluation only)

18. ├ summary.txt Validation result

19. ⎿ output.json Summary of execution information

2.3. Model evaluation

2.3.1. Details of processing

User considers model tuning based on validation results
2.4. Semantic segmentation inference

2.4.1. Processing Contents

Object region detection of unknown image using learned semantic segmentation model

Similar to learning, user inputs and executes parameters on JupyterNotebook

The output destination area is shown below

 Local: APMI PGEditor area

 System: APMI system area

 Push: AWS ECR Area

No. Input Processing Output

1. User enters parameters on JupyterNotebook Input parameters

2. Input Parameters Generate input.zip from input parameters and input dataset, including execution input.zip

Input Dataset conditions and data

3. input.zip 1. Read input information and apply preprocessing (padding, splitting) (Push)

2. Store preprocessed image and mask in splited_image, splited_mask input.json

3. Store split image and mask information in dataframe splited_image

splited_mask

dataframe

4. input.json 1. inference conditions, setting up models (Push)

splited_image 2. Inference execution (average output per Fold if cross-validation is predicted (prediction mask)

splited_mask performed)

dataframe

5. Predicted value 1. Calculation of the evaluation index (csv, with evaluation only): (Push)

2. Mask Pixel Value Storage (yml) predicted

3. Feature Extraction (csv) class_def.yml

4. Save Execution Results (txt) extracted_features.csv

5. Record Experiment Success/Failure (json) summary.txt

6. Combine processing result files into output.zip result_table.csv

output.json

output.zip

2.4.2. Input/output list

 Params

row File name Description

number

1. Params User-specified parameters on JupyterNotebook

2. ├ input_dir(str) Path of the input dataset

3. ├ model_dir(str) Path to zip containing learned model

4. ├ eval(int) Accuracy evaluation flag with correct answer mask

5. ⎿ exp_name(str) Experiment name

 input.zip

row name description

number

1. input.zip A zip file containing a collection of input files.

For the analysis facility container, download and use from Amazon S3 before running the

analysis.

2. ├ input.json (dict) Dictionary containing experimental conditions

3. │├ eval (int) Accuracy evaluation flag by correct answer mask

4. │⎿ exp_name (str) Experiment name

7. ⎿ input_files.zip Input dataset zip file

8. │├ image Folder containing inferred images

9. ││{input image n} .png Original image file

10. │⎿ mask *With evaluation

11. │ {Mask Image n} .png Mask image file # with evaluation

12. ⎿ models.zip Zip file containing learned weights, etc.

13. ├ {model_name}.pth Weights of the learning model

Present per Fold in the case of cross-validation

14. ├ executed_config.yml learning condition

15. ⎿ preprocess_info.pth pre-processing information during learning

*Only parameters that can be set by the user are included in the job processing definition.

*Other parameters are included in the design document.

 Output file

No. Name Description

1. output.zip A zip file containing the entire output file

2. ├ log The folder containing the log file for inference execution

3. │├ stderr.log The log file for inference execution

4. │⎿ stdout.log Log file at the end of inference

5. ⎿ prediction Folder in which inference results are stored

6. ⎿ {exp_name} Folder in which inference results are stored

7. ⎿ {yyyy_mmdd_hhmm_name} Folder to store inference results

8. ├ predicted Directory to store prediction mask images

9. │⎿ {image_id}.png Prediction result mask image (.png) for original image {image_id} .png

10. ├ extracted_features.csv Feature amount extraction result from mask image for each image

Hereinafter, each feature amount

File name column File name

total_area Total pixels

npixcels_class_{name} Total pixels of the class

coverage_ratio_class_{name} Class coverage

areas_class_{name} Number of pixels per object in the

class (array)
 coordinates_class_mudrock Coordinates of the class object

(array)

narea_class_{name} Number of class objects

11. ├ result_table.csv Evaluation index calculation result for each image

*With evaluation

12. ├ summary.txt Validation results

13. ⎿ output.json Summary of execution information