INFO

Latest updates

v1.46: Added automatic cropping to common area. The code is contributed by Jean-
Marc Lugrin.
v1.45: Fix to weight calculation for older PixInsight versions, HDRMLT and hue shift
updates, Use _DBE instead of _BE, Collapsible section status saved, Added status
text while processing images, Update for PixInsight 1.8.9

Overview

AutoIntegrate is a PixInsight script to process FITS and other image files and run a
basic image processing workflow to create a final image. Script has a GUI interface
where some processing options can be selected. AutoIntegrate can do the whole
image processing workflow including image calibration.

After running the script there will be integrated light images and automatically
processed final image. LRGB, color/OSC/DSLR and narrowband files are accepted.

AutoIntegrate is best suited for:

 When you are starting out with PixInsight and want to do a quick image
processing before learning all the details.
 For quick calibration, alignment and integration (but not a replacement for
WBPP), especially useful if you use remote observatories or other existing,
pre-calibrated images or prepared masters.
 Quick generation of a reasonably good image, especially with filter
combinations that may be difficult to manage when starting out.
 The possibility to have a quick look at a result and manually process the
problematic steps if needed (so you do not spend a day processing an image
whose color were badly balanced o start with). This is especially useful to see
if your calibration/integration is good enough or leave artifacts/gradients/...
that should be removed early.

Below is a sample output from AutoIntegrate script with otherwise default options but
Crop to common area was enabled.
Data for this image is from Telescope Live remote telescope.

Files accepted by the script

Script accepts all files that are supported by PixInsight. It can be used with FITS files
from telescopes, or RAW files from DSLR. It can be used with both calibrated and
non-calibrated files. For non-calibrated files the script can do image calibration.

Calibrated files from remote telescope systems like Telescope Live, iTelescope,
Slooh and Deep Sky West have been tested and they work fine.

When using RAW or OSC files, the script by default does debayering of the files. It is
recommended to use the Pure RAW setting in PixInsight. If the files are already
debayered you should choose None in the Debayer list.

How the script automatically recognizes different files

1. First script checks if the file has a FILTER keyword that specifies the image as
Luminance, Red, Green, Blue, Ha, SII, OIII or Color data. The following
FILTER values are recognized: Luminance, Clear, L, Red, R, Green, G, Blue,
B, SII, S, Halpha, Ha, H, OIII, O, Color. If there is a FILTER keyword but it is
not one of the recognized values, the file is treated as a color file. Only the
first letter of the FILTER keyword is used to detect the filter type.
2. If the FILTER keyword is not found then files that end with _L, _R, _G, _B,
_H, _S, _O or _C are treated as filter files. For example ngc6514_R.fit is
treated as a Red filter file.
3. Otherwise files are assumed to be color files.
If IMAGETYP keyword is set on files script can automatically detect bias, dark and
flat calibration files when adding light files. Also flat dark files are recognized with a
special value 'flat dark'.

Adding filter files manually

If automatic detection of the FILTER keyword does not correctly recognize the filter
type, it is possible to manually add files for each filter. This can be enabled by
checking 'Add manually' or 'Do not use FILTER keyword’ boxes.

There are separate buttons for each filter type recognized by the script. Also
color/OSC/DSLR files can be added manually. Manual adding is useful if the FILTER
keyword is missing or just for overriding filter keyword.

Downloading the script

Using PixInsight update system

It is possible to automatically install and update AutoIntegrate script by adding it to

the PixInsight update repository. Whenever PixInsight is started it will then check for
updates to AutoIntegate.js. To enable automatic updates you need to add the
following Url to the PixInsight Resources/Updates/Manage Repositories.

Note that automatic updates may not have the very latest version of the script.

https://ruuth.xyz/autointegrate

AutoIntegrate is added to the Script/Batch Processing menu.

Manual download

Right click the link AutoIntegrate.js and use 'Save link as...' or similar option in the
browser to save the script on your local machine.

In a page GitHub releases you can find the latest release and download it from there.
Releases page has some detailed information of changes in each release.
You can also go to GitHub AutoIntegrate main page to find the latest changes and
download it from there. GitHub master branch may contain test changes that are not
yet available as release.

Running the script

When using PixInsight update system

Click AutoIntegrate in the Script/Batch Processing menu.

Running the script from the script editor

Steps to run the script from the script editor

1. Download the script

2. Open Script Editor in PixInsight
3. Open file AutoIntegrate.js
4. Press F9 to run the script in the editor
5. Add files with Add buttons
6. Click Run and wait until the script completes

AutoIntegrate dialog screen

Below is the script dialog screen. Often it is best to start with default options, add
files and hit the Run button.

Script remembers the last open or closed states of collapsible sections in the dialog.
It is useful for example with smaller screens where it is possible to start with only
minimal sections open.
YouTube videos

I have created short YouTube videos that show the workflow with AutoIntegrate
script. These are created with an older version of AutoIntegrate script so dialog looks
a bit different. Basic steps are still the same.

Processing RGB files

Script supports LRGB, RGB and Color/OSC/DSLR image workflow.

Basic RGB workflow using AutoIntegrate is described here: PixInsight Image

Processing Workflow

Processing narrowband files

Narrowband images are supported by the script. They are processed mostly the
same way as other images. Main difference is that you can choose the color palette
that is used when mapping narrowband images to RGB channels.

Basic narrowband workflow using AutoIntegrate is described here: PixInsight

Narrowband Processing Workflow
Below is a sample output from AutoIntegrate script when using narrowband data and
SHO color palette. Otherwise it was run with default settings but Remove green cast
and Fix star colors options were checked.

Thanks to Erik Westermann for providing narrowband data.

Image calibration

Script can do image calibration if needed. Two basic workflows are supported:

1. Calibrating with bias, darks and flats

2. Calibrating with darks, flat darks (or dark flats) and flats

If any of the calibration files are not available the script tries to run calibration with
the remaining files.

If there is only one file for bias, dark or flat dark it is assumed to be a master file. If
you have multiple master files for example for different binning you can give multiple
master files but in that case option "Master files" must be checked. When multiple
master files are available, the script picks the one with the same resolution.

YouTube video of image calibration

AutoIntegrate output directories

AutoIntegrate will by default put output files into subdirectories.

 AutoOutput contains intermediate files generated during processing

 AutoMaster contains generated master calibration files
 AutoCalibrated contains calibrated light files
 AutoProcessed contains processed final images. Also integrated images and
log output is here.

By default output files are written into the same directory as the first light file in the
file list. It is possible to give an absolute or relative path to the output directory.

Crop to common area

It is possible to automatically crop possible bad pixels on the sides of the images.
This works by integrating all images, finding an area where all images are
contributing and cropping channel images to that area.

Automatic cropping makes it easier to get an image with only good data in it. It also
helps automatic processes such as stretching because only real data is included in
the image.

To check how much cropping was an image file LowRejectionMap_ALL is left

minimized on the screen.

Automatic cropping can be used also during AutoContinue. Cropping requires that
LowRejectionMap_ALL is open in the PixInsight desktop. Cropping can be done only
if starting from Integration_[LRGBHSO] or Integration_RGBcolor files.

Below is an example that shows the difference between non-cropped image (left)
and cropped image (middle). On the right is a LowRejectionMap_ALL file showing
the cropped area.
Sometimes the cropping helps also with automatic processing. Below is an example
that shows the processing difference between non-cropped image (left) and cropped
image (right). Since bad pixels on side are cropped automatic processing can better
stretch the images since there are only real data in the image.

Automatic processing of all Narrowband palettes

Narrowband palette option All automatically processes all HSO palettes it knows.
Every image is named with the palette option and automatically saved to disk as a
.xisf file. Images are also left open on screen. With Save final image files selection it
is possible to save them all for example as 16 bit TIFF files. You can also use extra
options in images, then images with name Auto_+palette-name+_extra are created.
All narrowband palettes known by AutoIntegrate, and two SHO palettes after extra
processing.

Extracting color channels from OSC files

When processing Color/OSC/DSLR files it is possible extract separate color

channels from color data. Channels can be extracted as LRGB, HSO or HOS data.
After channels are extracted, processing continues as in mono filter processing.

Channels are extracted right after cosmetic correction and debayering. Channel files
are saved to disk so they can be used for processing later.

Using separate channels enables different narrowband mappings when collecting

narrowband data with OSC cameras. Also LRGB data can benefit from separate
processing.
Thanks to René Bondo Hoffmann for providing OSC camera data using Optolong L-
eXtreme filter. Image is processed to emulate the Hubble (SHO) color palette using
'L-eXtreme SHO' palette option. Additional color processing was done after the
AutoIntegrate script run.

Fixing linear defects

It is possible to automatically fix linear column and row defects by using linear defect
detection algorithm from PixInsight LinearDefectDetection.js script. Defect
information is used by CosmeticCorrection to fix the defects.

AutoContinue

It is possible to run manually background extraction, histogram transformations or

other steps on the integrated images and then continue automatic processing from
there.

Basic RGB workflow using AutoIntegrate includes also the use of AutoContinue and
is described here: PixInsight Image Processing Workflow

Extra processing
The following extra processing can be applied to the image

 StarNet, StarNet2 or StarXTerminator to separate stars and background and

then combine them with selected option
 Clip shadows
 Automatic Background Extraction
 Darker background with a luminance mask
 ExponentialTransformation with a luminance mask
 HDRMultiscaleTransform with a luminance mask
 LocalHistogramEqualization with a luminance mask
 Add contrast
 Noise reduction
 ACDNR noise reduction
 Color noise reduction
 Sharpening
 Smaller stars with a star mask. Number of iterations can be given. More
iterations will generate smaller stars.

If multiple options are selected they are executed in the order listed above.

In case of Run or AutoContinue, extra processing steps are applied to a copy of the
final image. A new image is created with _extra added to the name. For example if
the final image is AutoLRGB then a new image AutoLRGB_extra is created.
AutoContinue can be used to apply extra processing after the final image is created.

There are also narrowband specific extra processing options in the Narrowband
processing section.

 Hue shift for more orange color, useful in SHO images

 Remove green cast, useful in SHO images
 Remove part of the green cast but leave some green color, useful in SHO
images
 Fix magenta star colors often visible in SHO images

From a drop down list it is possible to select an image into which extra processing is
applied. With the Apply button extra processing is run directly on the selected image
without making a copy of it.

Batch mode for mosaic images

Batch mode is intended to be used with mosaic images. In mosaics there are several
sets of images and typically the same basic processing options are used for all
images. In Batch mode AutoIntegrate script automatically asks for files for the next
mosaic panel. You can freely choose the directories for images. Script creates a final
processed image which has the same name as the directory where images were
read. So if images are from directory P1 then the image name will be P1. At the end
of the script all batch mode images are left open and all intermediate images are
closed.

When batch files are ready for further processing is it possible to save all files by
clicking a button in the section “Save final image files”. Batch mode adds a keyword
to each image and it is used to find files for saving. It is possible to save images in
PixInsight .xisf format, 16-bit TIFF format or 8-bit TIFF format. If you want to
assemble the final image in Photoshop, save images in 16-bit TIFF format.

Batch mode keyword is saved on disk with .xisf format but not with TIFF format. If
you want to process images later and use batch save then you should save files also
in .xisf format.

Window name prefix

It is possible to give the user specified prefix to image window names. This makes all
generated window names unique and allows results from multiple processing runs to
be open at the same time. These images are also iconized at different columns on
the screen.

By default the icon column for a new window prefix is automatically managed by the
script. Script tries to track used icon column positions and assign free ones for new
runs.

Sometimes, like when using multiple workspaces, this automatic management does
not work optimally. In that case it is possible to manually set the icon column for a
window prefix. This can be enabled by setting option Manual icon column control.
The change will be effective when restarting the script.

When manual icon control is enabled there is a new control Icon column. With that
control it is possible to manually set the column where icons are placed. The default
for the Icon column is Auto where the icon column is chosen automatically. Last icon
column is remembered and automatically restored when the script starts.

Binning
Software 2x2 binning option is available for light files. It is done as the first operation
for all light files or only for color channel files. It uses the IntegerResample routine for
binning. For very noisy images binning should help to reduce noise at the costs of
decreased resolution. If luminance data is good then it can be useful to do binning
only for color data.

Blink in AutoIntegrate

AutoIntegrate includes a simple Blink functionality. It is possible to see a stretched

image by clicking through the file list files. Blink window can be zoomed to 1:1.
Currently the focus is locked on the script so the blink window is in a fixed position
and size.

Files in the file list have a checked flag. Using blink functionality bad files can be
unchecked. Only checked files are used for processing.

Blink view

Saving settings to persistent module settings

It is possible to save all settings into PixInsight persistent module settings. These
settings are automatically restored on startup.

All saved parameters, persistent module settings, process icon and setup file, will
include only those values that are different from default values.

There is also a button to reset all parameters to default values.

Saving settings to script icon

It is possible to save your own default options to a script icon. It can then be used to
recall default settings. Script icons can be saved to a file.

Parameter values are restored from a process icon they are restored after persistent
module settings.

Settings are saved by dragging the lower left corner triangle icon to the desktop. By
double clicking the icon a script window opens. Clicking the Apply global button
starts the script with those settings that are saved to the icon. PixInsight also saves
MD5 checksum to the script icon. When using the script icon with a modified script
you need to clear the checksum field first.

Save and restore script setup

Current parameter settings and image file lists loaded into light and calibrate image
pages can be saved to an setup file. That file can later be restored back to
AutoIntegrate. Saved file keeps checked/unchecked status for files. The file must
have a .json extension. Loading the setup from a Json file works in all file dialogs but
there is also a separate button for it in the GUI.

Optionally it is possible to save just file lists and not current parameter settings.
There is a separete button to save just file lists.

File paths that are in a subdirectory of the setup file are saved as relative paths. This
makes it possible to move or share setup and images files and it still works in a new
location. Relative paths are automatically restored as full paths.

Setup file saves only those parameters that have a non-default value. It is good to
note that other parameters settings are not changed when restoring the setup. Also
window prefix and output directory are saved to a setup file.

Sort and filter file lists

Filtering option is used to filter images using current weighting and filtering settings.
If no filtering is set then files are just sorted based on weight calculation. For large
data sets it may be useful to use multiple filterings. For example, sort first by PSF
Signal and then by Eccentricity and after each sort uncheck bad files that are at the
end of the list.
Automatic filtering of bad files

AutoIntegrate has a few ways to automatically filter out files based on image
statistics calculated by the SubframeSelector process.

The simplest filtering uses a limit value for SSWEIGHT. Only files with calculated
SSWEIGHT above the limit value are included. This is a good way to filter files if the
expected SSWEIGHT values are known. Calculated SSWEIGHT values can be
checked from the AutoIntegrate.log file after processing.

There are also a few ways to filter files by detecting outliers in the data set. This
method does not require previous knowledge of the data. It can be useful in cases
with huge data sets where manual filtering gets too complicated.

The following variables can be used for filtering outliers: SSWEIGHT, FWHM,
Eccentricity, SNR, Star count, PSF Signal and PSF Power. Each variable is used
independently to detect and filter outliers.

There are also three ways to calculate outlier filtering threshold values:

 Two sigma that filters out values that are two sigmas away from mean value.
 One sigma that filters out values that are one sigmas away from mean value.
 Interquartile range (IQR) measurement that is based on median calculations.

By default FWHM and Eccentricity are filtered for too high values, and all others are
filtered for too low values. It is also possible to select filtering for both too low and
high values.

Narrowband to RGB mapping

A special processing is used for narrowband to (L)RGB image mapping. It is used to

enhance (L)RGB channels with narrowband data. It cannot be used without RGB
filters. This mapping is similar to NBRGBCombination script in PixInsight or as
described in Light Vortex Astronomy tutorial Combining LRGB with Narrowband. You
can find more details on parameters from those sources.

If narrowband RGB mapping is used then narrowband Color palette is not used.
With narrowband RGB mapping you can choose:
- Mapping of narrowband channels to (L)RGB channels
- Boost for (L)RGB channels
- Bandwidth for each filter
- Test the mapping with a test button

Processing Hubble data

A YouTube video shows how to process Hubble data using AutoIntegrate script. It
includes basic steps to download Hubble data and shows how to do basic
processing using AutoIntegrate script.

Some tips for using AutoIntegrate script

 Very often you get good results by running the script with default settings and
then continue processing in PixInsight.
 Always before running the script you should check the quality of the data
using a Blink tool. Remove all files that have movement like wind effect or
clouds.
 With extra processing it is possible to do basic enhancements like remove
stars using StarNet, StarNet2 or StarXTerminator, increase contrast, run
HDRMultiscaleTransform, LocalHistogramEqualization and make smaller
stars automatically.
 If you want to integrate only a single channel, you can do it by clicking
Monochrome and Integrate only options, and then selecting the correct files.
 When processing comets it can be useful to keep the green color. This can be
done using 'No SCNR' option. Also with comets the core may get
overexposed. Choosing 'None' in image integration rejection can help with
this.
 If you want to test with local normalization or with different image integration
rejection settings you can do the following steps:
1. Run normal processing to generate registered image files. You may
also want to check option 'Integrate only'.
2. Check option 'Start from ImageIntegration' and select registered *_r.xisf
files as Light files.
3. Choose local normalization or image integration settings you want to
test.
 4. Click 'Run' button. Result files will be saved into subdirectory
AutoProcessed under the lights files directory.
 If you are getting satellite trails in you integrated image you can try to use
Winsorised sigma rejection and lower High sigma value. Values like 2.2 or
even 1.8 could be a good starting point. In general a lower High sigma value
can be helpful if rejection is not good and you are using one of the Sigma
rejection algorithms.

Tips for processing One Shot Color (OSC) or RAW files

 Default options are typically a pretty good start for color images. If there is a
strong color cast and/or vignetting it is worth trying with 'Use ABE on
combined images' and 'Use BackgroundNeutralization' options. Sometimes
also choosing a different setting in the Link RGB channel option helps.
 For OSC/DSLR files PixInsight should be put into Pure RAW mode. It can be
set in Format Explorer by double clicking RAW format. AutoIntegrate does
debayering of images so no automatic debayering should be done.

Other resources

Erik Westermann has a web site Remote Astrophotography. He talks about

AutoIntegrate in his blog post PixInsight AutoIntegrate.js Processing Script. It has a
lot of useful information so please check it out.

Glenn Newell discusses his latest (not so) secret PixInsight weapon, the
AutoIntegrate script in the San Jose Astronomical Association (SJAA) meeting. Link
to the YouTube presentation is: SJAA Imaging Special Interest Group 05 18 2021

List of processes used by AutoIntegrate

Below are listed PixInsight processes that may used by AutoIntegrate script with
default LRGB processing. AutoIntegrate writes processing output to the Process
Console window and to AutoIntegrate.log file. Details of processing steps can be
found from the header block of the source code.

 ImageCalibration
 IntegerResample
 CosmeticCorrection
 Debayer
 ChannelExtraction
 SubframeSelector
 StarAlignment
 LocalNormalization
 ImageIntegration
 PixelMath
 LinearFit
 ScreenTransferFunction
 HistogramTransformation
 MultiscaleLinearTransform
 ChannelCombination
  ColorCalibration
 CurvesTransformation
 LRGBCombination
 ACDNR
 AutomaticBackgroundExtractor
 SCNR
 TGVDenoise
 ExponentialTransformation
 HDRMultiscaleTransform
 LocalHistogramEqualization
 StarNet
 StarNet2
 StarXTerminator

Script processing settings

History of AutoIntegrate script

I started my astrophotography hobby with remote telescopes in 2017. I almost

immediately got PixInsight and started working with FITS files. From the start I was
happy with the results but also found that there are a lot of repetitive and complex
tasks to be done. After a few months I even stopped my new hobby for a while.
When I started astrophotography again after the summer 2018 I realized that there is
this wonderful scripting capability in PixInsight. So I started developing my script.
FITS files from remote telescopes were already calibrated so I did some first steps
around ImageIntegration. This is where the script name comes from. Since then I
have expanded the script into both directions, from image calibration to post
processing.