ProcMT Gui

Testing Version

make a backup of your data!

Chapter

Table of Contents
1. Pr e fac e..................................................................................................... 3

2. Pr o cM T Gu i.............................................................................................5

Dir ector ie s........................................................................................................ 5

ts..................................................................................................................... 6
config............................................................................................................. 7
ca l........................................................................................................................ 9
pr oces sings ....................................................................................................... 9
jobs ...................................................................................................................... 9
db.................................................................................................................. 1 1
3. Pr o ce ssin g ............................................................................................. 13

Abo ut Pr oces sing Mod ules .........................................................................1 3

Pr oce ssing Pa ra m eter s ................................................................................1 5
C r ea te a Pr ocessi ng .....................................................................................1 6
AT S Fil e Rea der ........................................................................................1 7
Sca le, Detr end, Ha nning ........................................................................1 8
FFT............................................................................................................... 1 8
Ca libra te S pectr a .....................................................................................1 9
Autocr o ss Par z eni ng ................................................................................2 0
Sta ck All ...................................................................................................... 2 2
Coh er ency T hr eshold ...............................................................................2 2
Media n Pr oces sing ...................................................................................2 2
R un a Pr ocessi ng ........................................................................................... 2 4
pr ocm t_cm ds.................................................................................................. 2 4
4. C SAM T.................................................................................................... 25

G ener a l............................................................................................................ 2 5
R un a CS AMT Pr ocessi ng ............................................................................2 5
5. Lin u x....................................................................................................... 29

6. Ter m s ...................................................................................................... 31

7. A ppe n dix ................................................................................................ 33

Exa m ple XM L config files ...........................................................................3 3

2 ADU EMAP.............................................................................................. 3 3
S cr ipts.............................................................................................................. 3 5
 1. Preface

ProcMT (process MT) is a collection of libraries and

executables. They are bundled together in several GUIs
(graphical user interfaces).

ProcMT works on a very specific directory structure

where the software reads XML data and ats data
(advanced time series) as well as calibration files.

The philosophy is to configure all first and then run the

program. If configured you can run many jobs – the
computer does wait wait on mouse clicks.

All actions and joblists are stored in XML files and will
be executed on request from the GUI or shell.

The LINUX user will be able to run procmt_main from

script files, in case you have to process hundreds of
sites. The script interface can also run on the ADU
systems.

Activating all parallel options may lead to memory

exhaustion on smaller 32bit systems.
 2. ProcMT Gui

Directories

ProcMT expects this directory structure:

Fig 2.1: Directory tree

Start ProcMTGui, go to File Create Project, then

create the new folder in your filemanager
Chapter 2.

Figure 2.1 Create Folder

select the new directory and press ok. You also can
populate the directory with help of a shell script.

You can create additional folders if you want (for

example processings_old or documents). This will
not harm ProcMT.
Chapter 2.

Default Folders

ts

The timeseries data has to be copied into the

/ts/Site_nnn directories. Create Site_1 (manually)
for site 1 and so on ( ref. Fig 2.1); for multi channel
CSAMT create for example Site_140_142_144. Site
and site numbers have to be separated by
underscore ( _ ). The site will be processed in the
given order as the directory name.

config

sk ip t his for 4- 5 cha nnel MT, y ou don 't need it.

For C SAMT y ou MUS T hav e a config ur a tion

The configuration describes the XML channel

number, the channel type you want (Ex, Ey, Hx, Hy,
Hz). In the simple single site MT case Site_1 with 5
channels the relation will be Ex = 0 .. Hz = 4.

In a multi site setup you create for example:

Ex, Ey, Hx, Hy = 7, 8, 2, 3 (1st site → no. 140)

Ex, Ey, Hx, Hy = 0, 1, 2, 3 (2nd site → no. 142)
Ex, Ey, Hx, Hy = 5, 6, 2, 3 (3rd site → no. 144)
Chapter 2.
5.6 0.1 7.8 5.6 0.1 7.8

150 148 146 ← 144 142 140

ADU-07e ADU-07e

The order of your channels is linked to the directory

name.

Fig 2.2: Create Channel Configurations

Chapter 2.

In Mappings Configuration you insert the

numbers of sites you want to have and press
Apply.

Configure the channels. Hence: if you process a

directory with name Site_140_142_144 you can't
configure more than 3 sites!

Un-check “off ” to activate the channel.

cal

Put your MFS06nnn.txt and MFS07nnn.txt fil es

here.

processings

You find and create your processings here. All

processing files are XML files. You can edit them
manually. In this case make sure that you supply
valid parameters.
Chapter 2.

jobs

A job file contains the processing instructions and

the reference to the time series data to be used.

This directory contains temporary jobs, manually

configured and saved jobs. You can run one or more
jobs from the GUI or from the shell "procmt_main
jobs/jobfile.xml".

Figure 2.2 Select jobs from GUI

For batch processing.

Chapter 2.

db

Contains a SQLite database. All tensors and spectra

are stored here. After creating a new survey the
database must be created by hand.

In the database all results ever calculated will be

stored.
The database can be disabled in the procmt_cmds
module (command line options) with de-activate
database true.
 3. Processing

About Processing Modules

Each module contains one or more processing

instructions and will call the corresponding library
which will be linked in realtime to ProcMT.

You create them in the same order you want to process

the data, e.g. you

• read

• scale

• fft (spectra)

• calibration

• auto- cross spectra / parzening

• one or more MT processings

After scheduling an MT processing ProcMT will

generate EDI files and, if requested, insert the result
into a database for later usage.
Chapter 3.

ats read scale, detrend pre-whitening

FFT calibration spectra OP

auto- cross
MT module 1 MT module N
spectra

SQL database EDI files ts filter

spc filter ats file

Chapter 3.

Figure 3.1 Processing Scheme

Processing Parameters

Some processing parameters appear in almost any

MT module:

 start index: 0 … N, first tensor to process;

default 0 (s hould no t be set by t he user )

 stop index: 0 … N, last tensor to process;

default 0 ( first = last = 0 = take all) (s hould no t be
set by the u ser )

 start frequency: limit the frequency range (sho uld

no t be set by th e user )

 stop frequency: limit the frequency range (s hould

no t be set by th e user )

 upper reject rho: reject ρ if ρ x y and ρ y x exceed this

limit
 lower reject rho: reject ρ if ρ x y and ρ y x fall below
this limit
 upper reject phi reject: φ if φ x y and φ y x exceed
this limit
 lower reject phi: reject φ if φ x y and φ y x fall below
this limit
Hence: the simple cut-off method works on the xy
and yx elements only. Data is not rotated, fixed
Chapter 3.

rotated or Swift rotated. It can be used to cut-

off segments where distortions can be identified
by ρ or φ. This can happen if cables where
broken and have been re-connected later.
 use n stdev for rejection, n=0=no calculation :
reject tensors outside n ⊗ the standard
deviation. E.g. having a Gauss distribution 3
times would be definitively outside the
confidence interval. If 2 for example z x y mean is
5 ± 0.75 and z y x 9 ± 1.2 all tensors with z x y
greater than 3.5 and smaller 6.5 and z y x greater
7.8 and smaller 10.2 will be selected. if zxy is 5.2
and z y x is 12.1 this element will NOT be
selected.
 recalculate stddev after reject: if above is
selected take 1 here to recalculate the standard
deviation only from the selected tensors (again)
Should be 1 if rejection by standard deviation is
used.

The order is: rotation (if applies) → reject rho →

reject phi → processing method → reject by standard
deviation → re-calculate the standard deviation on
selected data.
Chapter 3.

Create a Processing

The filename of the EDI file will be made from the

site number and the processing comment. If your
processing comment is mt_std_sharp and coherency
threshold is selected for site 99 the resulting EDI
file will be: 99_mt_std_sharp_mt_processing_ct.edi

ATS File Reader

Controls the data input.

The ATS file reader will control your fft (window)

length.

 skip_samples: skip samples at the beginning of

the file, default 0
 skip_last: ship samples at the end of the file,
default 0.
 nslice: for CEA sliced files: n t h slice of the ats file
to process. Only active if sliced files are read.
Use 1024 if you want to process the last
available slice.
 nslices: for CEA sliced files: use n slices for
processing starting with nslice
 move_ahead: move n samples ahead after each
reading, default 0
Chapter 3.

 move_back: move back n samples after reading,

default 0
 read_samples: how many samples to read at once
(e.g. your fft window length) into one block
 skip_marked_parts: skip marked parts – this
information is stored in the “atm” files as a
Boolean 0/1 list.
 overlapping: since a Hanning window is used you
can set default 0 or 0.4. Do not use for CSAMT
and sliced files.
 prestacks: pre-stack time segments in the time
domain n times before calculating the FFT,
default 0
 dipole_txtime: n/a
 skip_tx_bds: n/a

Scale, Detrend, Hanning

Prepares the data for FFT.

 scale: scale the electric field to mV/km, default 1

 detrend: trend removal from the segment,
default 1
 hanning: apply a Hanning window to the data,
default 1
Chapter 3.

FFT

Executes the FFT with the window length given in

the ATS file reader.

 cut upper spectra: default 0.05 - removes upper

5%; r ef. pa ge 33

 cut lower spectra: default 0.001 - removes lower

0.1% r ef. pa ge 33

 dump raw spectra: control the raw spectra of

each channel (stacked). Result will be found in
the dump directory and can be opened for
example with xmgrace or qtgrace, default off

Cut upper and cut lower can be used to limit the

output spectra. This may only be used together with
the various dump options.

Calibrate Spectra

 calibration: auto: uses calibration from the xml

file, if not found from the cal directory and
finally built in (default). Builtin: uses a master
calibration function which fits excellent for
sampling rates up to 4 kHz and good above.
Theoretical: uses a theoretical function.
Chapter 3.

 dump calibration: dumps athe calibration

function to the dump directory.
 dump cal spectra: dumps the calibrated spectra
to the dump directory.

Autocross Parzening

This module controls the generation of auto- cross-

spectra and the parzening (MT) as well as the
target frequencies (from list or calculated =
CSAMT).

 xml MT site config file (empty for default) : You

MUST give a configuration file in case of
CSAMT. In case of standard MT (2, 4, 5
channels) ProcMT can guess the configuration.
 cut upper spectra: limit upper frequency for
processing by n (default 0); 0.1 would remove
upper 10% ref. page 33
 cut lower spectra: limit lower frequency for
processing by n (default 0.001) ref. page 33
 parzen radius: 0.05 to 0.8 combines frequencies
in the spectral domain. higher values (0.25, 0.5)
should create more smooth data. Lower values
can be used for more stable MT data as well to
get points near interfering lines (e.g. close to
50/60 Hz)
Chapter 3.

 activate MT/CSAMT: Switch to MT, MT scalar,

CSAMT, CSAMT scalar. Hence that scalar data is
mostly useless.
 Frequency List: Select one of the pre-defined
lists (default, txmsmall, generate ( for CSAMT) or
select a filename freq_list_[1-7], CEA_TX or
CSAM_TX (for CSAMT) or VLF for VLF.
These files (ending with .dat) are to be placed in
the config directory and are plane ASCII files
containing the frequencies you want to use.
the default list contains 8 frequencies per
decade and is compatible with older processings
from ADU-06.

If the CSAMT mode is activated following items will

be evaluated by ProcMT:

 base TX frequency: the transmitter frequency –

for example 8 Hz
 n odd harmonics : e.g.1, 2, 3 will generate a
frequency list of 8 Hz; 8 Hz, 24 Hz and 8 Hz,
24 Hz 40 Hz and so on.
 dipole 1 TX time[s]: time in seconds. Take even
times like 8, 16, 32, 64 and so on in order to fit
most possible FFT windows
Chapter 3.

 dipole 2 TX time[s]: must have the same value as

above. Dipole transmission times must be allways
symmetric.

 activate online processing : n/a yet

 dump CSAMT spectra infos: dump CSAMT spectra
infos.
 dump parzen spectra: dump the stacked parzened
spectra.

After finishing auto- and cross spectra the MT

modules are activated. You can select one or more
modules at the same time. Depending on the
memory and CPU of your computer you can choose
the cat all option in the command line module. This
will lead to a vast amount of memory usage and
parallel execution of the MT module s. Spectra from
several bands will be processed overlapped.

Stack All

Stack all is the most simple method. In most cases

this method is used to compare the statistical
method against the base. Hence: in case of
excellent data all methods will converge. In case of
noisy data you will see different results for any
Chapter 3.

method. It is the responsibility of the geophysicist

to decide which result is the most reasonable.

This module consist of the basic parameters only, as

described in Processing Parameters , page 15

Coherency Threshold

Coherency threshold is a classical method to sort

out data. If Ex and Hy are correlated the transfer
function z x y will have a good estimation (small
error). In some cases of correlated spikes or other
correlated noise you may want to limit the upper
threshold in order to sort out what is normally the
“best” data. In other words: if Ex and Hy are
outliers at the same time, the data seems to be good
but is an outlier.

ref. Processing Parameters , page 15, and:

 upper threshold: min 0.02, max=1.0
(default=1.0). 1.0 corresponds to perfect
correlated data. Data with “better” coherency
will be rejected.
 lower threshold: min 0.0001, max=0.99
(default=0.8). In case of 0.8 data with a
coherency less than 0.8 for z x y and z y x will be
rejected
Chapter 3.

 upper threshold %: adaptive coherency – n/a yet

 lower threshold %: adaptive coherency – n/a yet

Median Processing

The median processing calculates the median of the

the z x y and z y x component of your tensor array.

Choosing upper and lower bound median values will

be rejected if outside. Hence: you will not get the
median itself if upper & lower = 1 (it will likely
crash). Reason: The processing evaluates xy and yx.
In general the median is not biased by outliers as
we have it using the mean ( Stack All ).

ref. Processing Parameters , page 15, and:

 upper bound median: min=1.05 max=100, default
1.2
 upper bound median: min=1.05 max=100, default
1.2
Values from lower 1.2 to upper 1.2 will include
tensors where z x y and z y x are inside this range
(z / 1.2 and z * 1.2). If Z is Gauss distributed this is
a valid assumption and will remove outliers. If Z has
a different distribution the result can be biased.
The same applies if you select 2.2 and 1.6 – which is
Chapter 3.

not symmetric as above. Here you give preference

to lower resistivity values.

Fig 3.1: Create processing

Make sure that Window Length (top right) and

Window Length FFT are same.

1024 = last slice even if less available

Chapter 3.

All processing modules will be processed in the

order as they appear.

Run a Processing

Select a processing file and select a meas directory

and press Run Job.

procmt_cmds

You can set parameters using the command line

shell, for example -queue_len 8. With additional
disable -procmt_cmds false the procmt_cmd XML
module will be completely ignored if existing.
Otherwise (if existing) values will be taken from
XML file.
 4. CSAMT

General

Using CSAMT data the processing must fit the

transmitter data. You can NOT FILTER the data. The
amount of samples for each dipole transmitted is
calculated by the dipole TX time [s]. The TX time must
be the same for each dipole.

All CSAMT processings for each TX dipole must contain

the same MT/CSAMT processing types. If you
configure a median processing, median processing must
appear in all configured jobs.

The CSAMT processings also must contain the same

processing comments in each XML in order that
ProcMT combines all timeseries from all transmissions
and all sampling rates to a single EDI file.

Use the “Merge inputs” only if you make EMAP. In this

case your site directory may contain more than one xml
meas file.
Chapter 4.

Run a CSAMT Processing

A good naming convention for your CSAMT

processing is: csamt_2048Hz_16s_16s. That is
processing_TXfreq_dipoletime1_dipoletime2. For
tensor CSAMT dipole time 1 must be equal dipole
time 2.

Figure 4.1 same processing comments, different

processings
Chapter 4.

If the processing was already configured during

data recording (a name like mt_std_1024 or
csamt_1Hz_64s_64s) and a corresponding XML file
with same name is found in the processings
directory you simply double click either on the Site
or on a specific run.
 5. Linux

On a shell

procmt_createjob ts/Site_132_134_136_138/*/*xml >jobs/132.xml

 6 . Te r m s

Anisotropy: Zyx/Zxy , assumed do be greater 1. An

anisotropy limit of 1.8 prevents Swift or rotation of
tensors with an anisotropy less than 1.8. This
prevents from unnecessary rotation if required.
EMAP: is a configuration where multiple electric fields
are mapped to one magnetic field.
Limit Frequency ranges: In general the frequency
spectra wants to be limited. Especially at the lower
end spectral density is low and may not deliver
good results. In this case it can be better to limit
the lower end and to continue with the next lower
band (sampling rate).
If you filter only by 8 x instead of 32 x and
additionally limit the fft range 0.8 to 0.2 you get
almost constant bandwidth similar to cascade
decimation.
odd harmonics: when transmitting rectangular waves
you see in the resulting spectra odd harmonics of
the base frequency. They can be used to CSAMT
processing as well. However: the 3 r d harmonic (=
Chapter 6.

2 n d odd) may be already too small to deliver

good results in the processing.
 7. Appendix

Example XML config files

2 ADU EMAP

Here you see 2 ADU recording in EMAP mode, 71 is for

E only, 69 for E & H. This config file needs channel
number and serial number.

<?xml version= "1.0" encoding= "utf-8" ?>

<!-- this is an emap file for a multiple ADU (many systems in
use-->
<!-- this file contains serial numbers! -->
<!-- the "Merge Inputs" must be active -->
<!-- the file contains TWO mt_site_config so directory name
must be Site_1_2-->
<channel_map>
<mt_site_config>
<Ex> 5</Ex>
<Ex_acin> -1 </Ex_acin>
<Ey> 6</Ey>
<Ey_acin> -1 </Ey_acin>
<Hx> 7</Hx>
<Hx_acin> -1 </Hx_acin>
<Hy> 8</Hy>
<Hy_acin> -1 </Hy_acin>
<Hz> -1 </Hz>
<Hz_acin> -1 </Hz_acin>
<REx> -1 </REx>
<REx_acin> -1 </REx_acin>
<REy> -1 </REy>
<REy_acin> -1 </REy_acin>
<RHx> -1 </RHx>
<RHx_acin> -1 </RHx_acin>
<RHy> -1 </RHy>
<RHy_acin> -1 </RHy_acin>
<RHz> -1 </RHz>
Chapter 7.
<RHz_acin> -1 </RHz_acin>
<local_adu_serial> 0</local_adu_serial>
<local_site_no> 0</local_site_no>
<remote_adu_serial> 0</remote_adu_serial>
<ser_Ex> 71 </ser_Ex>
<ser_Ey> 71 </ser_Ey>
<ser_Hx> 69 </ser_Hx>
<ser_Hy> 69 </ser_Hy>
<ser_Hz> 0</ser_Hz>
<ser_REx> 0</ser_REx>
<ser_REy> 0</ser_REy>
<ser_RHx> 0</ser_RHx>
<ser_RHy> 0</ser_RHy>
<ser_RHz> 0</ser_RHz>
<site_no> -1 </site_no>
<sitedoc_channel_no> 0</sitedoc_channel_no>
</mt_site_config>
<mt_site_config>
<Ex> 5</Ex>
<Ex_acin> -1 </Ex_acin>
<Ey> 6</Ey>
<Ey_acin> -1 </Ey_acin>
<Hx> 7</Hx>
<Hx_acin> -1 </Hx_acin>
<Hy> 8</Hy>
<Hy_acin> -1 </Hy_acin>
<Hz> -1 </Hz>
<Hz_acin> -1 </Hz_acin>
<REx> -1 </REx>
<REx_acin> -1 </REx_acin>
<REy> -1 </REy>
<REy_acin> -1 </REy_acin>
<RHx> -1 </RHx>
<RHx_acin> -1 </RHx_acin>
<RHy> -1 </RHy>
<RHy_acin> -1 </RHy_acin>
<RHz> -1 </RHz>
<RHz_acin> -1 </RHz_acin>
<local_adu_serial> 0</local_adu_serial>
<local_site_no> 0</local_site_no>
<remote_adu_serial> 0</remote_adu_serial>
<ser_Ex> 69 </ser_Ex>
<ser_Ey> 69 </ser_Ey>
<ser_Hx> 69 </ser_Hx>
<ser_Hy> 69 </ser_Hy>
<ser_Hz> 0</ser_Hz>
<ser_REx> 0</ser_REx>
<ser_REy> 0</ser_REy>
<ser_RHx> 0</ser_RHx>
<ser_RHy> 0</ser_RHy>
<ser_RHz> 0</ser_RHz>
<site_no> -1 </site_no>
<sitedoc_channel_no> 0</sitedoc_channel_no>
</mt_site_config>
</channel_map>
Chapter 7.

Scripts

#!/bin/bash
mpwd =`pwd `;
echo $mpwd
cd jobs
for pdir in `ls `;
do
echo 'preparing' $pdir
cd $mpwd
procmt_main . /jobs /$pdir
done
Chapter 7.