NCPS DV Reference Manual

Consumer Insights
Application
Development
NCPS
DATA VALIDATION
Version 22.12.0
Reference Manual for Backend Jobs
Document Revision History:

Created 28-Sept-2006
Last revision: 23-Nov-2022
Version number: 2.30
Author App Dev C.I.
© 2022 The Nielsen Company NCPS DV Reference Manual

Page 1 from 698
Contents
EHDV21000P- Backdata Loader ................................................................................................................................................................................................. 5

EHDV21001P – Standard loader ............................................................................................................................................................................................... 14
EHDV21002P – Family Activity Loader .................................................................................................................................................................................... 26
EHDV21007P – Advanced cross coding check – report ......................................................................................................................................................... 31
EHDV21008P – Cross Coding ................................................................................................................................................................................................... 35
EHDV21014P – Check Promotion Codes ................................................................................................................................................................................. 40
EHDV21015P – Formal Price Validation ................................................................................................................................................................................... 44
EHDV21016P- Purchase Inspection ......................................................................................................................................................................................... 51
EHDV21018P- Multipack Validation .......................................................................................................................................................................................... 54
EHDV21019P- Private Brand – Extraction................................................................................................................................................................................ 57
EHDV21020P- Private Brand Validation – process ................................................................................................................................................................. 60
EHDV21022P – Rawda Loader .................................................................................................................................................................................................. 65
EHDV21023P – Trip Data Alignment......................................................................................................................................................................................... 70
EHDV21024P – Trip Data Validation ......................................................................................................................................................................................... 76
EHDV21025P – Lower Bound limit computation ..................................................................................................................................................................... 81
EHDV21026P – Update proc-keys ............................................................................................................................................................................................ 84
EHDV21028P – Weekly Family Usability computation............................................................................................................................................................ 89
EHDV21029P – Common Sample Family Usability computation ........................................................................................................................................... 96
EHDV21034P – Load Household Characteristics .................................................................................................................................................................... 99
EHDV21035P – Load Family Monitor...................................................................................................................................................................................... 103
EHDV21036P – Load Shop File ............................................................................................................................................................................................... 109
EHDV21037P – Load Shop Characteristics ........................................................................................................................................................................... 114
EHDV21038P – Compute median price deviation ( MAD) ..................................................................................................................................................... 119
EHDV21039P- Compute standard price ................................................................................................................................................................................. 124
EHDV21040P- Load standard price ........................................................................................................................................................................................ 130
EHDV21041P – Rawda extraction for rawda correction ....................................................................................................................................................... 135
EHDV21042P – Rawda Correction .......................................................................................................................................................................................... 142
EHDV21043P- Data file extraction for special analysis ........................................................................................................................................................ 147
EHDV21048P- DB cleaning...................................................................................................................................................................................................... 152
EHDV21050P- Load Supplier Prices ....................................................................................................................................................................................... 154
EHDV21051P- Copy Supplier Price ........................................................................................................................................................................................ 158
EHDV21052P – Pharma Price Loader (Italy) .......................................................................................................................................................................... 162
EHDV21053P- Copy NRSP Prices ........................................................................................................................................................................................... 167
EHDV21054P- Compute & Load Median Price NRSP ............................................................................................................................................................ 172
EHDV21055P- Compute Median Price CLUSTER .................................................................................................................................................................. 176
EHDV21056P- Copy Median Price CLUSTER ........................................................................................................................................................................ 180
EHDV21057P- Copy standard price from Cycle .................................................................................................................................................................... 184
EHDV21060P – Shop Weight computation ............................................................................................................................................................................ 187
EHDV21061P – Nan reclassification by X-Coding (all tables) .............................................................................................................................................. 192
EHDV21062P - Load OPUS Prices .......................................................................................................................................................................................... 197
EHDV21063P- Copy OPUS Prices .......................................................................................................................................................................................... 201
EHDV21065L – Housekeeping Joblog and Msglog ............................................................................................................................................................... 204
EHDV21065U – Housekeeping ShopUsabilityT40 and FamilyUsabilityT41 ........................................................................................................................ 206
EHDV21065M – Housekeeping HHParamT06 & FamilyMonitorT42 ..................................................................................................................................... 208
EHDV21065P – Housekeeping CnanPrice, ClusterPrices, Shop_Prices, Supplier_Prices ................................................................................................ 210
EHDV21066P – Check Family Code on T-ALL ....................................................................................................................................................................... 213
EHDV21067P- Check Shop Code on T-ALL ........................................................................................................................................................................... 216
EHDV21068P- Data Selection .................................................................................................................................................................................................. 219
EHDV21069P- Copy total basket Finland ............................................................................................................................................................................... 225
EHDV21070P- OGRDS LOADER ............................................................................................................................................................................................. 228
EHDV21073P- Active Items extraction ................................................................................................................................................................................... 234
EHDV21074P- NRSP Price Loader .......................................................................................................................................................................................... 236
EHDV21075P- HBE – Automatic Purch. Inspection .............................................................................................................................................................. 241
EHDV21076P- Automatic price validation on Rawda ............................................................................................................................................................ 247
EHDV21078P – Re-Xcoding by exception (based on barcode list) ...................................................................................................................................... 254
EHDV21079P Loyalty card usage extraction ......................................................................................................................................................................... 258
EHDV21082P – Extraction from Table ALL (ALLT11) ........................................................................................................................................................... 262

Page 2 from 698
EHDV21083P – Extract Family Monitor (Italy) ....................................................................................................................................................................... 267
EHDV21084P – Variable weight volume validation ............................................................................................................................................................... 272
EHDV21085P- HCE – Heavy Contributor Edit ........................................................................................................................................................................ 279
EHDV21086P- Copy Median Price CLUSTER HH Geo /SH Chain......................................................................................................................................... 285
EHDV21088P – Cross Coding based on barcodes list ......................................................................................................................................................... 290
EHDV21089P – Intended User Alignment .............................................................................................................................................................................. 296
EHDV21090P – Total Spent validation ................................................................................................................................................................................... 300
EHDV21091P – Assortment Validation................................................................................................................................................................................... 304
EHDV21092P – DV KPI computation and extraction ............................................................................................................................................................. 311
EHDV21093P – Extract for panelist compliance ................................................................................................................................................................... 316
EHDV21094P- Basket Validation ............................................................................................................................................................................................ 321
EHDV21095P – Price & Total Value validation ...................................................................................................................................................................... 327
EHDV21097P – Populate Shop_priceT99 from cps data (UK) .............................................................................................................................................. 331
EHDV21098P – Spain fresh food only basket validation ...................................................................................................................................................... 335
EHDV21099P–Variable weight barcodes management ........................................................................................................................................................ 338
EHDV21120P – Purchase File Extraction to CASE for LATAM ............................................................................................................................................ 341
EHDV21121P – UPC File Extraction to CASE for LATAM ..................................................................................................................................................... 347
EHDV21122P – Trip File Extraction to CASE for LATAM...................................................................................................................................................... 351
EHDV21123P – Additional Trip Data Facts ............................................................................................................................................................................ 355
EHDV21131P – Formal Telecom Price Validation ................................................................................................................................................................. 360
EHDV21132P – Validate Telecom Prices against Catalogue prices .................................................................................................................................... 364
EHDV21133P – Automatic Telecom Price Validation ............................................................................................................................................................ 367
EHDV21134P – Median Telecom Price computation............................................................................................................................................................. 373
EHDV21135P- Data Selection by exception (based on barcode list) ................................................................................................................................... 376
EHDV21136P- Family Usability Extraction from Sample DB ................................................................................................................................................ 381
EHDV21137P – Variable Weight Volume Validation / IMDB Max .......................................................................................................................................... 384
EHDV21137S – Variable Weight Volume Validation / Computed Max ................................................................................................................................. 391
EHDV21138P- Private Brand Validation ................................................................................................................................................................................. 399
EHDV21139P- Beverages Pack Detection.............................................................................................................................................................................. 406
EHDV21140P- Table ALL Cleaning ......................................................................................................................................................................................... 413
EHDV21141P- COMPUTATION OF EAN WEIGHT, FR HARD DISCOUNTERS ..................................................................................................................... 415
EHDV21142P – GENERIC SHOP EDIT .................................................................................................................................................................................... 420
EHDV21143P – Country Rawda Copy .................................................................................................................................................................................... 424
EHDV21144P – Cycle Rawda Copy......................................................................................................................................................................................... 429
EHDV21147P – Cycle Price Copy ........................................................................................................................................................................................... 434
PRODUCTION FLOW ............................................................................................................................................................................................................... 439
EHDV21148P – EXTERNAL PRICE LOADER ......................................................................................................................................................................... 444
EHDV21149P – FRESH FOOD RECOVERY TOOL ................................................................................................................................................................. 451
EHDV21150P- US Data Loader................................................................................................................................................................................................ 458
EHDV21152P- US Data Edit ..................................................................................................................................................................................................... 474
EHDV21153P- Duplicate Transmission Detection ................................................................................................................................................................. 488
EHDV21154P- US xCoding Shrink .......................................................................................................................................................................................... 493
EHDV21155P- US xCoding ...................................................................................................................................................................................................... 497
EHDV21156P- US Reclassification ......................................................................................................................................................................................... 501
EHDV21157P- DV File Merger ................................................................................................................................................................................................. 506
EHDV21158P- Brand Exception .............................................................................................................................................................................................. 509
EHDV21159P- US Threshold Computation ............................................................................................................................................................................ 514
EHDV21160P- Uncoded Extraction......................................................................................................................................................................................... 519
EHDV21161P- PGI Xcoding by Default ................................................................................................................................................................................... 526
EHDV21162P- PRICE COMPUTATION BY CHARACTERISTICS ........................................................................................................................................... 530
EHDV21163P- PGI DATA SELECTION .................................................................................................................................................................................... 538
EHDV21164P- LOYALTY CARD USAGE RATE COMPUTATION .......................................................................................................................................... 543
EHDV21165P- NEW PRODUCT ALERT .................................................................................................................................................................................. 548
EHDV21166P- IBD RESOLUTION PERMANENT STORAGE ................................................................................................................................................. 552
EHDV21167P- CA Data Loader ............................................................................................................................................................................................... 556
EHDV21168P- CA Data Edit..................................................................................................................................................................................................... 571
EHDV21169P- UNCODED TRACKING .................................................................................................................................................................................... 586
EHDV21170P- Price Management for ReXCoding and Recovering .................................................................................................................................... 591
EHDV21171P- New Standard Price and MAD Computation ................................................................................................................................................ 596
EHDV21172P- Converted Unit Outlier Threshold ................................................................................................................................................................. 601
EHDV21173P- Converted Unit Outlier Edit ........................................................................................................................................................................... 606
EHDV21174P- Load Consolidated Items Weight .................................................................................................................................................................. 610

Page 3 from 698
EHDV21175P- Copy Consolidated Items Weight.................................................................................................................................................................. 613
EHDV21176P- Price Load using OGRDS Char ..................................................................................................................................................................... 616
EHDV21177P- Assortment Items Loader ............................................................................................................................................................................... 618
EHDV21178P – Additional ASIN Xcoding .............................................................................................................................................................................. 621
EHDV21179P – PCP Correction Factor computation ............................................................................................................................................................ 625
EHDV21180P – PCP Distribution Factor computation .......................................................................................................................................................... 628
EHDV21181P – Automatic Price Validation Standard .......................................................................................................................................................... 631
EHDV21182P – Price and MAD Computation Standard ........................................................................................................................................................ 641
EHDV21183P - Total Value Normalization.............................................................................................................................................................................. 651
EHDV21301P – EHDV21309P – FLAT FILE IMDB LOADER .................................................................................................................................................. 654
Surgery Tracking ..................................................................................................................................................................................................................... 659
Document Revision History: ................................................................................................................................................................................................... 676
Appendix .................................................................................................................................................................................................................................. 691

Page 4 from 698
EHDV21000P- Backdata Loader
Phase:
PHASE 5 – DATA MANIPULATION
Process Description:
Backdata Loader is used to load data in a batch mode directly to the output tables:
RAWDAILYT60, TBSKT13, PROCKEYST29, from backdata input ASCII files, prepared by the
legacy system. This process is used by countries to initialise their DV environment with already
validated data.
Note : For DT5084 used for loading receipt Id - The alphanumeric receiptId mentioned in input file
will not be directly loaded into tbskt13. Instead, the receipt IDs will be loaded into tdv_receipt
table along with a unique sequence id. This sequence id is loaded into tbskt13's DT5084 column.
Also note that only DT5084 must be used for loading receipt Id and no other datacode must be
used for this purpose.
Note : For python version Hive data refresh is done at the beginning and oracle data refresh is
done at the end of this process in terminateJob for these three tables:
• Prockeyst29
• Rawdailyt60
• Tbskt13
Recommendations:
Constraints:
Input files should come in pairs, one file for the purchase records (rawdaily records) and one for
total basket records (tbsk records). In case the job is submitted for N periods > 1 then these
ASCII files should contain all the records for all the periods submitted.
Single file submission is not allowed in NORMAL Mode, is allowed only in REWORK mode.
Prerequisites:
Before initiating a Backdata Loader job, the following tables should be already loaded with data:
The IMDB table (nan_itemst73)

The Family Table (familyt43)
The Shop Table (shopt44)
The data codes for the specific job (EHDV21000P) should be defined in Data Codes
Table (datacodet20)
Source of Information:

Page 5 from 698
Oracle Tables Involved:
Table Description
CAL_FACTORT05 Static Parameters table for submission of job
FAMILYT43 It contains the list of the family codes loaded into family usability table.
Updated automatically by the system
SHOPT44 List of Shops
NAN_ITEMST73 List of NAN_KEY loaded from EIMDB
DATACODET20 Purchase datatype layout definition (DT Parameter Screen)
DATATYPEST07 The table contains the list of the datatypes and the position in the record.
RAWDAILYT60 Purchase RAWDA
PROCKEYST29 List of Rawda Processing keys (used here to check if data loaded are
already present in output tables)
TBSKT13 Total basket Rawda
TDV_RECEIPT List of TICKET ID loaded in DV

Page 6 from 698
Files:
Input Files:
Rawda file structure:
Position Length Name Description

1 4 ac_audit AUDIT
5 8 dt_pur_date DATE
13 4 ac_time TIME
17 10 ac_hhold HOUSE HOLD
27 10 ac_shop SHOP
37 35 ac_barcode BARCODE
72 16 nc_nan_key NAN KEY
88 16 DataType Value1
…
…
16 DataType Valuen
TBSK file structure:

1 4 ac_audit AUDIT
5 8 dt_pur_date DATE
13 4 ac_time TIME
17 10 ac_hhold HOUSE HOLD
27 10 ac_shop SHOP
37 35 ac_barcode BARCODE
72 16 nc_nan_key NAN KEY
88 16 DataType Value1
…
…
16 DataType Valuen
Table DATATYPEST07 will be used to identify the DTYPEs valid for each of the output tables
(RAWDAILYT60, TBSKT13) based on the ac_destination values.
Having identified the eligible DTYPEs, then the DATACODET20 table will be used to find out the
exact ASCII file position of each DTYPE as well as its length.
Output Files :
Two output ASCII files will be created by the job, one for RAWDATA records and one for TBSK
records.
The layout of these files will be the same as the layout of the Input ASCII files with one additional
column at the end of each record stating the rejected reason (e.g Missing NAN, Missing HHOLD,
Missing SHOP, Zero PRICE, Zero QUANTITY etc)

Page 7 from 698
Parameters :
Static Parameters:
Parameter Type Mode Description

DECIMAL_SHIFT List Optional • When set to OFF, data type values coming from
ASCII file are expected to have the decimal point in
the value.
• When set to ON, data type values coming from

ASCII file are expected NOT to have the decimal
point in the value, therefore decimal shifting should
be applied based on the DECIMAl definition for
each Data Type.
• Valid values are

ON
OFF
Default value if parameter is not defined is OFF

LOCK List Optional LOCK. When LOCK is set to ‘N’ then records loaded in
PROCKEYST29 are loaded with DT5090=0. When LOCK
is set to ‘Y’ then records loaded in PROCKEYST29 are
loaded with DT5090=1
Valid values are

‘Y’ – YES
‘N’ - NO
Default value if parameter is not defined is Y

REWK_RAWDA List Optional Valid values are
ON
OFF
This parameter is used only when the job’s process mode

is REWORK.
When job’s process mode in NORMAL mode this
parameter is ignored and both file are loaded by the
process.
By setting the parameter to ON only the specific modified

ASCII input file will be loaded by the process

Page 8 from 698
REWK_TBSK List Optional Valid values are
ON
OFF
This parameter is used only when the job’s process mode

is REWORK.
When job’s process mode in NORMAL mode this
parameter is ignored and both file are loaded by the
process.
By setting the parameter to ON only the specific modified

ASCII input file will be loaded by the process.
HH_CHECK List Optional When set to OFF, household values in the ASCII file are
not checked if they exist in the familyt43 table
When set to ON, household values in the ASCII file are

checked if they exist in the familyt43 table
Valid values are

ON
OFF
Default value if parameter is not defined is ON

NAN_CHECK List Optional When set to OFF, nan key values in the ASCII file are not
checked if they exist in the NAN_ITEMST73 table
When set to ON, nan key values in the ASCII file are
checked if they exist in the NAN_ITEMST73 table
Valid values are

ON
OFF

SHOP_CHECK List Optional When set to OFF, shop values in the ASCII file are not
checked if they exist in the shopt44 table
When set to ON, shop values in the ASCII file are checked
if they exist in the shopt44 table
Valid values are

ON
OFF

Page 9 from 698
Dynamic Parameters:
Parameter Used? Comments

Name (Yes/No)
Period Yes Submission with Period 0 is not allowed for this job
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management :
Proc.TYPE CYCLE PERIOD option

Process XXXX YYYY.PPP normal Allowed
Process XXXX YYYY.PPP F# Not allowed
Process XXXX YYYY.PPP PC Not allowed
Rework XXXX YYYY.PPP normal Allowed
Rework XXXX YYYY.PPP F# Not Allowed
Rework XXXX YYYY.PPP PC Not allowed
Other Screen Parameters:

Page 10 from 698
Error messages managed by the job:
Message Description Tag Type Explanation Action

code
P0020308 Error while getting File System Error S Generic error while open a QLS
Lock accessing a file
P9900245 Rawda tables could not System error S RAWDAILYT60 Open a Ticket
be locked could not be locked
S00C0050 Column not in structure Column not S Checks that all Abort process
in struct datatypes defined in
table
DATACODET20
actually
P0020125 File Name is blank File Name is S Purchase Data Input
blank File is missing
P0020126 File Name not present FileName S Purchase Data Input
not present File is missing
P0020125 File Name is blank File Name is S TBSK Data Input
P0020126 File Name not present FileName S TBSK Data Input
P0020125 File Name is blank File Name is S OUTPUT File is
blank missing
P0020126 File Name not present FileName S OUTPUT File is
not present missing
P0020505 Static Value not present Stat.Val not S REWK – Load Mode Check if all
present is OFF for both input required Static
files parameter are
defined for
submission
cycle
P9900222 Error while validating Error while W Invalid records Check for the
records in the file Checking identified in input validity of
File files records
P9900214 Can NOT RELEASE Can NOT S Can NOT RELEASE Open a QLS
Lock. RELEASE Lock.
Lock
P9900261 Invalid Character in Input Invalid S Audit:
file Character in
P9900232 Input file format not Input file S Input RAWDA ASCII Check input file
correct format no file does not exist/or layout
format is not correct
P9900232 Input file format not Input file S Input TBSK ASCII Check input file
correct format no file does not exist/or layout
P9900193 Full Xcoding Required Full Xcoding W Xcoding Required. Check the
Require Non-linked items cross coding
were found in whether all the
RAWDA ASCII file values are
crosscoded
P0010035 Oracle Error Oracle Error S Oracle Error Check for
oracle syntax

Page 11 from 698
List of job Counters in MsgLog :
Message code Description Tag Comments

P0000000 Submission period Date Range Get Date range for submission period
date range
P0020608 Static Value Static Value RECORDS LOCKED: Information message
only. No action required
P0020608 Static Value Static Value LOAD RAWDA: Information message only. No
action required
P0020608 Static Value Static Value LOAD TBSK: Information message only. No
action required
P9900123 Total Number of Record Count # RAWDA Records in input file: Information
Records Present in message only. No action required
the input file
P9900123 Total Number of Record Count # TBSK Records in input file: Information
the input file
P9900123 Total Number of Record Count # TBSK Records inserted: Information
the input file
P9900123 Total Number of Record Count # RAWDA Records inserted: Information
the input file
P9900123 Total Number of Record Count # PROCKEYS Records inserted: Information
the input file
P9900321 Number of purchase #Keys ALL RAWDA records from input file validated
keys validated OK validated OK OK
P9900321 Number of purchase #Keys # RAWDA records from input file not validated
P9900321 Number of purchase #Keys ALL TBSK records from input file validated OK
keys validated OK validated OK
P9900321 Number of purchase #Keys # TBSK records from input file not validated
P1023020 Number of records delete Cnt- Information message only. No action required
deleted from TBSKT13
TBSKT13 table
P1049031 Number of Records Delete Cnt – # RAWDATA Records deleted:
deleted from RAWDA
RAWDAILYT60
deleted from PRKEYS
PROCKEYST29 table
P9900263 Table # Partition * Created Information message only. No action required
Subpartition % Partition
created.
P9900263 Create Dummy Created Information message only. No action required
Partition: Partition
P9900045 Duplicate Key Found Dup Record Information message. Duplicated records
in RAWDA ASCII File in File should be removed.
P9900045 Duplicate Key Found Dup Record Information message. Duplicated records
in TBSK ASCII File in File should be removed.

Page 12 from 698
P9900123 # TBSK Records Already Information.Already loaded records in
already loaded loaded TBSKT13 is mentioned here
records
P9900123 # RAWDA Records Already Information.Already loaded records in
already loaded loaded RAWDAILYT60 is mentioned here
records
P9900123 # RAWDA Records Already Information.Already loaded records in
without related basket loaded RAWDAILYT60 without basket information
records

Page 13 from 698
EHDV21001P – Standard loader
Phase :
PHASE 0 – INPUT AREA
This program is loading the purchase data into the Data Validation system. During the loading
phase, quality controls are applied. This will be used to process home scanner data and
homepanel data. Customized rules, by type of panel, will be applied.
This program can be included in a group containing, for example, standard loader, check Family
codes, check shop codes, cross coding and cross coding report.
This program requires a Purchase file and an Info file as input file.
Parquet Refresh [Before the job]

* Data in oracle table is refreshed to parquet for Allt11 and Prockeyst29 tables.
* Data will be refreshed for all the periods for the give audits in input file.
*Error in Refresh parquet is logged using the error number - P9900605
Oracle Refresh [End of the job]

* Data from Allt11 table is refreshed back to oracle table.
* All the affected partitions and sub partitions are refreshed in Oracle refresh.
*Error in Oracle Refresh is logged using the error number - P9900627
Recommendations :
It is recommended that local reformatting program check basic errors in Purchase File (shop
empty, Household empty…).
Constrains :
The 01P process has been enhanced such that when any records with zero units (DT5650) is
available in input file then these records are excluded from loading any DT type in ALLT11
It is not allowed to re-load purchase key already present in table All, in such a case Rework
option should be used. In rework mode, we delete from T-ALL all the family baskets present in
the input file.
Prerequisites :
The Purchase file and Info file have to be prepared in the standard layout format by the
local reformatting program.
Source of Information :
Oracle Tables Involved :

Page 14 from 698
Table Description
ALLT11 All the data collected by the families are loaded inside T ALL.
During the load phase, all the single fields of the record are separated
in datatypes and loaded in the table ALL as separate records
NAN_ITEMST73 List of Nan_keys extracted from EIMD (used there only in case of Rework PC
for Homepannel only)
MODULEST31 List of Modules extracted from EIMD (used there only in case of Rework PC
PROCKEYST29 List of Rawda Processing keys (used here to check if data loaded are already
present in output tables)
FAMILY_LOGT67 Log of changes applied by DV automatic validations processes (used here to
check if data loaded are already present in output tables)
FILE_LOGT59 List of Files # loaded in DV (can be access in screen Services → File log)
TDV_RECEIPT List of TICKET ID loaded in DV
Files:
Input Files:
Purchase file (File id : FILEI1)
The purchase file should contain both total basket and detailed purchase records.
It should be ordered by AUDIT/ YEAR/ MONTH/ DAY/ TIME/ HHNUMBER/ SHOP/ SCAN PC/
BARCODE
Purchase data record layout :

Positio Length Name Description
n
1 4 AUDIT
5 4 YEAR YYYY
9 2 MONTH MM
11 2 DAY DD
13 4 TIME HHMM
17 10 HHNUMBER
27 10 SHOP
37 4 SCAN PC 0000
41 30 BARCODE Left Justified
71 10 LAPTOP_CODE 0000000000
81 DT # 1 Refer to DatacodeT20 (DT Parametes screen)
DT # 2 Refer to DatacodeT20 (DT Parametes screen)
Basket data record layout :

n
1 4 AUDIT
5 4 YEAR
9 2 MONTH
11 2 DAY
13 4 TIME
17 10 HHNUMBER

Page 15 from 698
27 10 SHOP
37 4 SCAN PC 0000
41 30 BARCODE 0 Left Justified
71 10 LAPTOP_CODE 0000000000
81 DT # 1 Refer to DatacodeT20 (DT Parametes screen)

Page 16 from 698
Info file (File id : FILEI2)
n
1 4 AUDIT
5 7 PERIOD YYYYPPP
12 80 FILENAME Purchase File name
92 12 TRANSMISSION yyymmddhhmm(This field can also start from 96th position
DATE/TIME with length of 12 )
Output Files (File id : FILEO1) :
Output file containing list of basket keys deleted from ALLT11 because present in
purchase input file and already present in ALLT11 (logic only applied in case of REWORK mode).

n
1 4 AUDIT
5 4 YEAR YYYY
9 2 MONTH MM
11 2 DAY DD
13 4 TIME HHMM
17 10 HHNUMBER
27 10 SHOP
TICKET file (File id : FILEI3)

n
1 4 AUDIT
5 8 DATE YYYYMMDD
13 4 TIME HHMM
17 10 HHNUMBER
27 10 SHOP
37 36 TICKETID
Parameters :
Static Parameters
Parameter Type Value Name (example) Description

Type
PARM TBSK ON / OFF Available only in oracle version
PARM CHECKOUT Y/N If CHECKOUT is not present or set to Y then
CheckOutputTables is applied, if set to N this
check is not applied
PARM PRICE_INC AVG/MAX/NULL To manage price in case duplicates records
to be merged

Page 17 from 698
PARM TOTAL_TILL ON / OFF To apply specific check for Total Gross and
Total Discount
ON – Step is to be performed
OFF – Step is not to be performed
Target is to check the presence of Gross and

Discount Total in input and 18anellists a Total
Ticket value.
PARM MULTI_AUDI ON / OFF To be able to load multiaudit input file.
TS
Default value =OFF if NOT defined explicitly
PARM SHIFTING ON / OFF To be able to do shifting of purchase dates
PARM SHIFT_LATE 7 Define the number of days to use for
_DAYS detection of late transmissions
PARM SHIFT_MOV 14 Indicate the number of days to add to
E_DAYS purchase data for shifting operation
PARM SHIFT_OUT 14 Define the number of days to use for
_DAYS detection of out of scope transmissions
PARM SHIFT_OUT 21.5 Define the maximum rate allowed for out of
_RATE scope transmissions , if value higher job
would fail
PARAM RECEIPT_FI ON / OFF To be able to load ticket input file.
LE
TBSK parameter is used in case the standard loader is submitted as part of a group to manage only
purchase keys present in input file (this using File #).
PRICE_INC parameter is used to drive cases of duplicated records with minimum one record price=0 and
minimum one record price>0.
If PRICE_INC=AVG, merge record will have a price computed as sum(pricexqty)/sum(qty)

if PRICE_INC=MAX, merge record will have a price computed as MAX(price)
if PRICE_INC is not present of different from MAX or AVG, then job will end in severe error.
Remark : PRICE_INC has also impact on duplicated records will all prices>0
if PRICE_INC=AVG or null, merge record will have a price computed as sum(pricexqty)/sum(qty)
If PRICE_INC=MAX, merge record will have a price computed as MAX (price)
AVG option should be used to if it is considered that anellist correctly reported a promo (buy one get one
free, buy one get one 50% discount.
MAX option should be used if is considered that anellist wrongly report price=0
If TOTAL_TILL=ON
The Total Basket (DT5941) is computed when missing using to the 2 new collected Data Types:
● Total Gross as datatype DT5898
● Total Discount as datatype DT5899
In the input file either the Total Spent (DT5941) only will be populated with a value different to 0,
Either the Total Gross (DT5898) and/or Total Discount (DT5899)

Page 18 from 698
The 3 possible different situations are the following:
Input Field Value Records Loaded in

Action
DT5941 DT5898 DT5899 ALLT11
>0 0 0 DT5941 only No action to perform
0 >0 0 DT5898 only Set DT5941 to DT5898
Remove DT5898 record
0 >0 >0 DT5898 and DT5899 Set DT5941 to DT5898-DT58991
Remove DT5898 DT5899 records
- Missing partition check will be performed before ALLT11 insertion.

- This procedure will create the missing partitions and align the table if partitions are missing
- If Shifting is ’ON’
● Using first day of submission period shift date limit (using first day and SHIFT_LATE_DAYS)
and out of scope date limit (using first day and SHIFT_OUT_DAYS) is calculated
● For Purchase date less than limit date and greater than or equal to out of scope date shifting is
applied
○ Set DT5888=purchase date before shifting
○ Set DT5889=submission period
○ Set purchase date = purchase date + SHIFT_MOVE_DAYS
● For Purchase date less than out of scope date shifting is not applied
○ Set DT5889=submission period
● For duplicate trips generated during this shift ,ac_time is incremented by one .
● Shift rate is computed as ( # trip records out of scope / # total trip records in file) ,if it’s greater
than SHIFT_OUT_RATE,then the process ends in failure.
- When Shifting is ’OFF’ ,01P works as usual
Dynamic Parameters
Homescanner
Parameter Name Used? Comments

(Yes/No)
Period Yes Submission with Period 0 is NOT allowed for this job
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes file names can be customized by the users
Homepanel
Parameter Name Used? Comments

(Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
1
When Total Gross > Total Discount only.

Page 19 from 698
PC Selection Yes Option by PC code (s) used for rework option only
Before rework process, PC (one or more) must be loaded together in an unique
file for the same run
F# Selection No
DTSelection No
File Selection Yes file names can be customized by the users
Rework Management :
HomeScanner

Process XXXX 0000.000 F# Not allowed, run like normal process
Process XXXX 0000.000 PC Not allowed, run like normal process
Rework XXXX 0000.000 normal Delete in TALL same baskets into file,
then normal process
Rework XXXX 0000.000 F# Not allowed, then normal rework
Rework XXXX 0000.000 PC Not allowed, then normal rework
Homepanel
Proc.TYPE CYCL PERIOD option

E
Process XXXX 0000.000 normal Load data if same basket not present in T-ALL
If same basket found, stop job & rollback
Rework XXXX 0000.000 normal Delete in T-ALL same baskets into file,
Then normal process
Rework XXXX 0000.000 F# Not allowed, run like normal rework
DT Parameter screen is used to define purchase file layout.

Remark : Three Datatypes are mandatory in purchase file layout : 5650 (quantity), 5675 (price), 5674
(purchase value). If price & qty are available, Purchase value is computed. If purchase value & qty are
available, price is computed by the system.

code
P0020125 File Name is blank File Name is S File name is not Enter a valid file
blank specified name
P0020126 File Name not present FileName S File name is not Enter a valid file
not present available name
P0020564 Error while getting Position System S Check
and Precision Info Error dtparamters and
DT definition
screen
P0020576 Error while generating System S Check
DTInfo for the given Tag- Error dtparamters and

Page 20 from 698
DataCode Info List DT definition
screen
P0020563 DataCode Info Not Present DTCode S Datacode info not Enter the value
(Position,Precision) Info absent available for Datacode
Info
P0020580 Error while populating the System S Open a QLS
MsgLog Array about Invalid Error
Tags for DTInfo
P1001027 Error while setting DT System S Open a QLS
Pointers Error.
P1001001 Position and length Data Code S Values not found for Enter the values
information missing for not in T20 position and length for position and
datacode in T20 fields in Datacodet20 length fields in
Datacodet20
P1001002 Error in Ehfn_Ident_DT System S Generic error while Open a QLS
Error. identifying datatypes
P9900211 Audit Code is NOT correct Audit Code S Audit code in input Update input file
is NOT co file is not consistent with Valid Audit
with submission audit code
This is valid only if

job is aubmitted with
MULTI_AUDITS=OF
F.
P9900047 Invalid Date Invalid Date S Date is not correct Check date in
(can be File input file
Transmission Date or
purchase file date)
P0020599 Error while checking Date System S Error while validating Check date field
Validity Error date from Purchase file in input file
P9900001 File Handling-Invalid Wrong file S Invalid operations are Check input
Operation operation done with file filenames in
Msglog, check if
files are present
on FileSystem
with correct
grants
P9900002 File Handling-Read Error File Read S Error while reading Check input
Error input file filenames in
Msglog, check if
files are present
on FileSystem
with correct
grants
P9900004 Value Error Value Error S Error in value Check reported
record in input
file
P0020587 Error while Merging Values System S Open a QLS
of the arrays Error
P9900014 Empty File Empty File S Input file is empty Check input file
P1001009 Error in Ehfn_CheckFile System S Generic error while Open a QLS
Error. checking input file
P1001010 Initialisation of file table not Insert F# S Open a QLS
done. Insert file number 0 Zero

Page 21 from 698
P9900083 Blank Record Blank S INFO file Record is Check INFO file
Record` empty content
P0020609 Error while Checking Long System S Error in file Check
Date Error transmission date from transmission
INFO file date from INFO
file
P1001011 Error in Ehfn_LoadFileLog System S Generic error while
Error. managing File#
P1001012 Error in System S Generic error while Open a QLS
Ehfn_ProcessForReworkAu Error. deleting basket keys
dit form table All
Ehfn_ProcessForReworkTB Error. deleting basket keys
SK form table All
Ehfn_ProcessForReworkPC Error. reworking with rework
by PC mode
P1001017 Error in System S Generic error while in Open a QLS
Ehfn_LoadTBSKData Error. loading the tbskt
records from input file
P1001018 Units = 0, removing record Purch Units W Any record with Check purchase
=0 DT5650=0 is not input file
loaded
P1001019 Error in System S Generic error while in Open a QLS
Ehfn_LoadPurchaseData Error. loading the purchase
records from input file
Ehfn_ComputeDatatype Error. computing datatype
P1001023 Error in Ehfn_LoadOutput System S Generic error while Open a QLS
Error. inserting in table ALL
Ehfn_CheckOutputTables Error. checking output tables
P1001003 Basket key already exists in Duplicate S Reported record in Delete reported
T11 Basket already present in table record from
ALL and we are not in input file or
Rework mode submit with
Rework mode
P1001029 Time is blank Time is S Time value is empty in Correct input file
blank Purchase file for for given record
givent record
P9900039 Missing Laptop Code Missing S Laptop code is empty Correct input file
Laptop Code in Purchase file for for given record
givent record
P9900040 Missing Family Code Missing S Family code is empty Correct input file
Family Code in Purchase file for for given record
givent record
P9900051 Shop Code Missing Shop Code S Shop code is empty in Correct input file
Missing Purchase file for for given record
givent record
P1001028 Scan PC is blank Blank S Scan PC is empty in Correct input file
ScanPC Purchase file for for given record
givent record
P1001004 Invalid combination of ScanPC!=0- S Scan PC and barcode Enter correct

Page 22 from 698
scanpc and barcode BarCode=0 combination is not combination of
correct in Purchase file Scan PC and
for givent record Barcode in
Purchase file for
givent record
P1001005 Total Basket Record Duplicate S Duplicate basket key Correct input file
duplicated TBSK rec in purchase file for for given record
given key to skip
duplicated key
P1001030 Exception while validating System S Not trapped rrror faced Check purchase
the file contents Error. while validating a file file, open a QLS
content
P0020521 Data Code not present for DT not S Some missing DT are DT5650,5675
the given Tag present missing in DT and 5674 are
parameter screen mandatory
P0010039 Out of Scope high rate Out of Scope S Out of Scope records Please check the
high rate are in high rate input file
P0010042 Receipt file is empty Empty W Ticket file is empty Please check the
receipt file input ticket file
P0010043 Dup Key in Receipt File Dup Key S Duplicate key in Please check the
RCPTFILE Ticket file input ticket file
P0010044 Job ID (Receipt) missing Missing Job W TicketID is not Please check the
ID available in TICKET input ticket file
file
P0010045 Job ID (Receipt) already Known Job W TicketID already Please check the
existing ID loaded in input ticket file
TDV_RECEIPT_JOBI with reference
D taken from
TDV_RECEIPT
_JOBID

Page 23 from 698

P9900125 Number of Records Deleted In case of Rework, this counters provides number of
Deleted Record records deleted from Table All
Count
P1001033 Number of records Outside Information message only.
outside the date range DTRAN Program is loading records even if it’s outside date.
of the selected period GE Cnt
P9900123 Total Number of Record Information message only. No action required
Records Present in Count
the input file
P9900117 Number of Warnings Warning Information message only. No action required
s
P9900116 Number of Severe Severe Information message only. No action required
Errors Error
P1001034 Number of records Wrong Information message only. No action required
with Blank or Zero Laptop
laptop code cnt
P1001035 Number of distinct Basket Information message only. No action required
Baskets present in the cnt
input file
P1001036 Number of distinct Distinct Information message only. No action required
Families present in Family
the input file cnt
P1001037 Number of Total TBSK Information message only. No action required
basket Records Records
Cnt
P1001038 Number of Records Insert Information message only. No action required
inserted into ALLT11 Cnt
P1009039 Proc Mode TBSK TBSK Information message only. No action required
forced
P0020611 # of NRSP Shops in NRSP Information message only. No action required
SHOPUSABILITYT Counter
40
P9900108 File Number File Information message only. No action required
Number
P1001015 Key deleted as scan Scan PC Information message only. No action required. For
pc is equal to the pc = PC homepannel only.
code selected for
rework
P1001014 Rework by PC Reclass. Information message only. No action required. . For
requested – Record Rec homepannel only.
reclassified found- deleted
Records deleted
P1001024 Data already present Rework Some keys present in table ALL are already present in
in output tables – required Rawda or FamilyLog. So applying DataSelection in on
Rwk option required those data in normal mode, data will not be processed.
for data processing Data should be deleted from Rawda+FamilyLog or
rework should be used to process data.
P0710028 Total Number of Trip # Total Number of Trip Records Incremented to a new
Records Incremented Records time,to avoid duplicate trips and purchases
to a new time Updated

Page 24 from 698
P0010036 # Records Shifted # Records shifted when shifting is ON
Records
Shifted
P0010037 # Records Out of # Records out of scope when shifting is ON
Scope Records
Out of
Scope
P0010038 % Records Out of % Rate of out of scope when shifting is ON
Scope Records
Out of
Scope
P0010040 # Records from # Records read from receipt file
Receipt File Records Information message only. No action required
Receipt
P0010041 # Records loaded in # Records loaded into TDV_RECEIPT_JOBID
JobID table Records Information message only. No action required
JobID
P9900264 P9900264 P990026 P9900264
4
Output Table #: Output Table #: Output Output Table #:
Table #:
Partition * Partition * Partition Partition * Subpartition
Subpartition Subpartition *
Subpartit
ion
P9900635. Parquet Partition Parquet Parquet Partition creation
creation Partition
creation

Page 25 from 698
EHDV21002P – Family Activity Loader
Phase :
It will be used to process Homescanner data and Homepanel data.This program is

loading the family information into the TDV_FAMILY_ACTIVITY table. We insert here all the
information about the collaboration of the family like
➔ Family in holiday (DT 5643)
➔ No shopping in the data-collection period (DT 5684)
➔ Other information
We use this information, together with the data, to generate an information system for the family
usability process or for the control of the collaboration of the family.
Recommendations :
Constrains :
Family code should not be 0 and should be present in FAMILYT43

Data-type code should not be 0 and should be linked to TDV_FAMILY_ACTIVITY
Prerequisites :
Loading family in holiday information, we require that a single record for each day of
holiday has to be loaded inside the system
Table Description
TDV_FAMILY_ACTIVITY It contains the Family activity information
Files:
Input Files:
File id : FILEI1
1 4 AUDIT Xxxx
5 8 DATE Yyyymmdd
13 4 TIME Hhmm
17 10 FAMILY 00xxxxxxxx
27 10 SHOP 0000000000
37 4 SCAN PC 0000
41 30 BAR CODE 0000000000…..
71 4 DATACODE Xxxx
75 16.5 DATAVALUE 1 ( xxxxxxxxxx.xxxxx )

Page 26 from 698
- Record deletion: when data value field is equal to 0 in file, it indicates to delete
corresponding keys from table TDV_FAMILY_ACTIVITY. If key does not exist, record will be
rejected and listed to the user.
- Records are rejected if DATE value is invalid, if the audit value is different from the job audit
submission value,
- Records with FAMILY code not present in FAMILYT43 table are not rejected but are reported
with WARNING message in the message log.
Output Files :
File id : FILEO1

1 4 AUDIT Xxxx
15 60 Warning Panelist with Holidays Date From < Period start date
message or
Panelist with Holidays Date To – Holidays Date From >= 7
days
65 8 Date from Day of holidays from
73 8 Date to Day of holidays to
81 3 Number of Number of days between holidays date from and holidays
Days date to
In this output file we report:

➢ all households with Min(Date Holiday) < Date from of submission period (computed
based on submission period + number of periods)
➢ all households with Max (Date Holiday) - Min(Date Holiday) >7
File id : FILEO2

1 4 AUDIT Xxxx
5 8 DATE Yyyymmdd
13 4 TIME Hhmm
27 10 SHOP 0000000000
37 4 SCAN PC 0000
41 30 BAR CODE 0000000000…..
71 4 DATACODE Xxxx
75 16.5 DATAVALUE 1 ( xxxxxxxxxx.xxxxx )
This output file contains the HHOLDS records (as they appear in the INPUT file) which are
rejected from loading due to the dates reported are out of range of the loaded period.
The HHOLD reported

• declare more days than MAX_DAYS

Page 27 from 698
or
• declare dates far from period range with less/more than MAX_DAYS
Date from Input File < Period Submission datefrom – MAX_DAYS

or
Period Submission dateto + MAX_DAYS < Date from Input File
Parameters :
Static Parameters

Type
PARM MAX_DAYS 35 Set the detection level with MAX_DAYS
(Expressed in Days)
PARM FORCE NO If no value is selected, default value is set to

NO.
If YES is selected then all records from Input

file is loaded.
If NO is selected then only HHOLDS records

with dates within the date range:
Period Submission datefrom – MAX_DAYS

< Date from Input File <
Period Submission dateto + MAX_DAYS
Otherwise HHOLDS records which

• declare more than MAX_DAYS
or
• declare dates far from period range
with less/more than MAX_DAYS
are rejected and NOT loaded into table
TDV_FAMILY_ACTIVITY
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :

Page 28 from 698
Process XXXX YYYY.PPP normal Load data
Process XXXX YYYY.PPP F# Not allowed, run like normal process
Process XXXX YYYY.PPP PC Not allowed, run like normal process
Rework XXXX YYYY.PPP normal Delete common key data between INPUT file and
TDV_FAMILY_ACTIVITY table and then load
Rework XXXX YYYY.PPP F# Not allowed, then normal rework
Rework XXXX YYYY.PPP PC Not allowed, then normal rework
No other screen parameters

code
P9900014 Empty File Empty File S File is empty without Check for the
records file whether it
has valid
records
P9900001 File Handling-Invalid Wrong file S Invalid operations Check input
Operation operation are done with file filenames in
Msglog, check
if files are
present on
FileSystem
with correct
grants
Msglog, check
if files are
present on
FileSystem
with correct
grants
P9900232 Input ASCII file does not Wrong File S Error while reading Check the
exist/or format is not Format or input file FORMAT /
correct File does LAYOUT of the
not exist ASCII file.
P9900091 Date Outside Date Range Date not in W Date in input file is
DTRANGE outside the
submission date
range
P1002004 Family from HHLog file is Family not W Family not present in Load Family in
not present in T43 in T43 T43 table T43 table using
Load HH char
job
P9900155 No records found for No records S Check if input
processing found file is empty

Page 29 from 698
P9900424 Invalid Value Invalid S Invalid values either Check input file
Value for audit, date or for invalid
datacode values
P1038014 Key Already Present- Key S Common keys are Submit job in
Rework Option Required Already found between the REWK mode
Present input file and data in
TDV_FAMILY_ACTI
VITY
P9900421 Key does not exist in Key does W Keys in the input file
database not exist with datavalue=0 do
not exist in table
TDV_FAMILY_ACTI
VITY
P9900433 HHOLD with more than HHOLD W
MAX_DAYS Code
P9900434 HHOLD with dates far HHOLD W
from period range with Code
more than MAX_DAYS

P9900123 # Records read Record Information message only. No action required
from Input File: Count
P9900420 # Records Deleted # Information message only. No action required
from Records
TDV_FAMILY_ACT Deleted
IVITY:
P1003009 # Records already Records Information message only. No action required
present in present-
TDV_FAMILY_ACT Cnt
IVITY:
P9900141 # Records Inserted Number Information message only. No action required
into of
TDV_FAMILY_ACT records
IVITY: in
P0020001 HOLIDAY IN THE # Information message only. No action required
WEEK (DT5643) Records
inserted
P0020002 NO PURCHASE # Information message only. No action required
DECLARED Records
(DT5684) inserted
P0020003 TRANSMISSION # Information message only. No action required
DONE (DT5645) Records
inserted
P0020004 TRANSMISSION # Information message only. No action required
WITH PURCHASE Records
DONE (DT5646) inserted

Page 30 from 698
EHDV21007P – Advanced cross coding check – report
Phase :
This program will be used to process Homescanner data. It’s checking the status of the
links in table ALLT11 and it is reporting missing links in cross coding in one output file.
Criteria for prioritisation are reported in the report (number of purchase transactions by
Group_ID/Barcode).
Xcoding model there is similar to the one of Xcoding job (08P), so we have 3 Xcoding logics :
➔Cross coding by exception (by shop)
➔Cross coding by barcode type
➔Standard cross coding
Recommendations :
This job can be used after Xcoding job to report non Xcoded barcodes to EIMDB coding
team. The counter of purchase acts can be used for prioritise the coding.
Constraints :
Prerequisites :
Table Description
CROSSCODING_INSTRUCTIO Crosscoding links information extracted from EIMDB
NST49
CROSS_CODE_GROUPST26 List of Xcode Group to be used for Xcoding (Xcode Group screen)
SHOPCOLLID_LINKT19 Instructions for Xcoding by shop
BARCODE_TYPET23 Instructions for Xcoding by barcode type
NAN_ITEMST73 List of Nan_keys extracted from EIMDB
Files:
Input Files:
No input file.

Page 31 from 698
Output Files :
File with not existing links (File id : FILEO1)

1 Not fixed Group-id 0 for standard Xcoding
Xcode group in case of Xcoding by shop (with flag to
Xcode general is set to ‘No’)
Xcode group in case of Xcoding by Barcode Type
Not fixed Not fixed Barcode
Not fixed Not fixed Counter
Counter: we report the number of purchase transactions associated to the same Group_ID &
barcode
File Manual Exceptions, contains barcodes without any link for EIMDB load. (File id :
FILEO2)
This file is a cvs file (comma separated file, no fixed layout)

1 Not fixed IMDB Retailer Group-id
Code
Not fixed Not fixed Barcode Barcode
Group-id :
Contains EIMDB Group if Xcoding by retailer or by barcode type
Contains value for static parameter MIS_GR_ID in case of standard Xcoding
Parameters :
Static Parameters

Type
XCOD MODE FULL/PART
PARM MIS_GR_ID FR01 Group-ID in EIMDB manual exception file in
case of standard Xcoding
PARM SCRIPT_CF Check Appendix for
T Information
In case of option PART :

Only purchase transactions in ALLT11, that are not yet X-Coded (Identified by Bar code >
0 and NAN key = 0) will be processed.
In case of option FULL :

All the purchase transactions in ALLT11, including those purchase transactions, which
are already X-Coded, will be processed.
Dynamic Parameters

Name (Yes/No)

Page 32 from 698
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :

Rework XXXX YYYY.PPP normal Not allowed, then normal rework
Xcoding instructions are the same as the one used for Xcoding (08P) and standard Re-Xcoding
(61P).
Those instructions are defined in GUI screens:
Services → X-Coding →XcodingGroup : list of Xcode group to be used for Xcoding
Services → X-Coding → Bar Code-Type: set up of Xcoding instruction by Barcode Type
Services → X-Coding → Shop-Group Id: set up of Xcoding instruction by Shop code

code
P9900155 No records found for No records S No record in ALLT11
processing found table for submission
cycle/date_range
P0020125 File Name is blank File Name S File name not Enter valid
is blank specified filename
P0020126 File Name not present FileName S File name not found Enter valid file
not present name
P0020504 Static Value Blank Static S Missing static
Value parameter value
Blank
P9900156 Collaborator Id missing Collaborato S if collaborator ID is Check for
r Id miss blank in the table collaborator id
where we link the in table
barcode type and
the collaborator id ,
then an error
message is issued

Page 33 from 698
P1008005 Purchase date is not Outside S ALLT11 purchase Check EIMDB
inside valid pc-item date IMDB date is not inside content for
range of IMDB-Link not Dtrange NAN_ITEMST73 reported
applied NAN_KEY validity NAN_KEY
date range

P9900005 Required No. of Periods Information message only. No action required
Periods not Present not
enough
P0020608 Static Value Static Information message only. No action required
Value
P9900212 Number of Number Information message only. No action required
barcodes without of
any link barcode
sw
barcodes with of
many links barcode
sw
P1007009 Number of Many Information message only. No action required
purchase acts with Link Cnt
many links
List of Messages for External Job Submission :
Message Description Type Tag Comments

code
P9900394 Automation Job I External Information message only. No action
submitted with the JOB name required
following name:
ended JOB required
SUCCESSFULLY SUCCEED
P9900392 Automation Job S External Additional information is given within the
FAILED JOB message. In case there is no additional
FAILED information the following message is
displayed:
NO ADDITIONAL EXTERNAL JOB
INFORMATION AVAILABLE
displayed:

Page 34 from 698
EHDV21008P – Cross Coding
Phase :
This program is adding the NAN_KEY code, from EIMDB, to the original barcode delivered from
the household.
The NAN_KEY code is required as all the data validation system (Instructions / category
definition) is based on the EIMDB Crosscoding instruction table.
Any purchase transaction can be x-coded with 3 logics:

o Standard Xcoding ( same link for all shops )
o Xcoding by exception on shop (customized links by chain or shops)
o Xcoding by exception on Barcode Type
14. Standard Xcoding :
This Xcoding should be used for EAN / UPC codes and for I codebook products
If the standard x coding (same link for all shops) is requested, the search for the link will be based
on :
BARCODE || DATE FROM & DATE TO
Only collaborator id’s present in the system with Xflag value = 1 in Services → X-Coding
→XcodingGroup screen will be used.
The COLLABORATOR-ID’s belonging to the “x coding by exception” area will be excluded from
the search
Only the links with the right date range will be selected
DT_EFFECTIVEFROM <= date of the purchase transaction <= DT_EFFECTIVETO
In case more than one NAN_KEY is found we select the first NAN_KEY by ordering as follows :
DT_EFFECTIVEFROM DESC, DT_UPDATE DESC, NC_NAN_KEY DESC
2. Xcoding by exception on shop
We can have 2 kinds of situations :

- cross coding retailer : all the shops of that retailer are using the same barcodes for the
same products (many shops – one collaborator ID)
- cross coding by shop : a shop is using a dedicated set of links
Exception table contains the link of all the “shops || collaborator-id“ requiring a dedicated set of
instructions.
This table is filled using Services → X-Coding → Shop-Group Id screen
when the x coding by exception is running,
First the system is searching for a link in CrosscodingInstruction table from EIMDB using :
BARCODE || GROUP_ID || DATE FROM & DATE TO

Page 35 from 698
If more than one Xcoding instruction found: Xcoding is not applied, record in written in output file
FILEO2.
If no link found :
Then if the flag parameter in Shop-Group Id screen is set to ‘Y’, then we search the missing
link among the collaborator-id’s belonging to the area of the standard x coding.
3. Xcoding by Barcode Type
The screen Services → X-Coding → Bar Code-Type is setting up Xcoding instruction by Barcode
Type
Any purchase act in ALLT11 table with DT5098<>0 will be Xcoded using this logic.
It is mandatory to have a Xcode group id linked to for each possible value of DT5098 in ALLT11
The system is searching for a link in CrosscodingInstruction table from EIMDB using :
BARCODE || GROUP_ID || DATE FROM & DATE TO
If more than one Xcoding instruction found: Xcoding is not applied, record in written in output file
FILEO2.
-This file should be in CSV format (separated by ;)
- This file contains 4 columns : GroupId, Barcode, Description GroupId and Number of Barcodes
by GroupId
Recommendations :
This job can be run only in an Home-scanner environment
Constraints :
Prerequisites :
The EIMDB should be refreshed in advance (job Load EIMDB).

Services → X-Coding →XcodingGroup screen should be updated to flag all Xcode group to be
used for Xcoding
Services → X-Coding → Shop-Group Id screen to be filled in advance , if required.
Services → X-Coding → Bar Code-Type to be filled in advance , if required
Table Description
CROSSCODING_INSTRUCTIO Crosscoding links information extracted from EIMDB
NST49
CROSS_CODE_GROUPST26 List of Xcodes group to be used for Xcoding (Xcode Group screen)
NAN_ITEMST73 List of Nan_keys extracted from EIMD

Page 36 from 698
FILE_LOGT59 List of Files # loaded in DV (can be access in screen Services → File
log)
Files:
Input Files:
No input file.
Output Files :
List of non Xcoded purchase acts (File id: FILEO1) :

1 4 AUDIT Xxxx
5 8 DAY Yyyymmdd
13 4 TIME Hhmm
27 10 SHOP 0000000000
37 Not fixed BARCODE 0000000000…..
Not fixed 8 Error code
Not fixed 20 Error tag
List of multiLink barcode (File id: FILEO2)

List of barcodes with more than one record in Xcoding Instruction table
Column Name Description

1 GROUP ID Xxxx
2 BARCODE 0000000000…..
3 Group id Description XXX
4 Number of barcodes 12…
Parameters :
Static Parameters
Paramete Type Name(example) Description

r Type Value
XCOD MODE FULL/PART
In case of option PART :

Only purchase transactions in ALLT11, that is not yet X-Coded (Identified by Bar code >
0 and NAN key = 0) will be processed.
In case of option FULL :

All the purchase transactions in ALLT11, including those purchase transactions, which
are already X-Coded, will be processed.
Dynamic Parameters

Page 37 from 698
Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :

Rework XXXX 0000.000 normal Not allowed, run like normal process
Rework XXXX 0000.000 F# Not allowed, run like normal process
Rework XXXX 0000.000 PC Not allowed, run like normal process
Xcoding instructions are the same as the one used for Xcoding report (07P) and standard Re-
Xcoding (61P).
Those instructions are defined in GUI screens :
Services → X-Coding → Bar Code-Type : set up of Xcoding instruction by Barcode Type
Services → X-Coding → Shop-Group Id : set up of Xcoding instruction by Shop code

code
cycle/date_range
not present name
P0020504 Static Value Blank Static S Missing static
Value parameter value
Blank
barcode type and
then an error
message is issued

Page 38 from 698
P1008005 Purchase date is not Outside S ALLT11 purchase Check EIMDB
inside valid pc-item date IMDB date is not inside content for
range of IMDB-Link not Dtrange NAN_ITEMST73 reported
applied NAN_KEY validity NAN_KEY
date range
P1008018 Error while updating File System S
Log Error

enough
P0020608 Static Value Static Information message only. No action required
Value
P1008011 Total Number of Purchas Information message only. No action required
Purchase acts to be eacts to
linked link
P9900186 Total Number of Total Information message only. No action required
Purchase with Number
NAN=0 of Purc
purchase acts with Link Cnt
many links
Purchase acts e acts
linked during cross linked
coding
transaction NOT Number
Linked after of tran
Xcoding
Purchase acts with Link
more than one link Count
P1008015 Number of No Link Information message only. No action required
Purchase acts Count
without any link
P1008016 Number of Wrong Information message only. No action required
Purchase acts with CNAN
PC – Items not count
present in IMDB
P1008017 Number of Wrong Information message only. No action required
Purchase acts with CNAN
PC-Item in wrong Date cnt
date range
P1007007 Number of No Link Information message only. No action required
Purchase acts Count
without any link

Page 39 from 698
EHDV21014P – Check Promotion Codes
Phase :
PHASE 1 – FORMAL DATA VALIDATION
The collection of the promotions is a part of the purchase transactions we receive from
the families. We need to verify that all the promotion values in the data files are known to the
NCPS system. Promotion values are used in the data processing system and in the PFW special
analysis.
We can define in the system a list of promotion codes. The list is flexible; any country can
add any promotion code and value. Only the codes in the list will be verified by the program.
We compare the list of the promotion codes and their values in T-359 by timestamp with
the promotion codes and values in the promotion table.
For each promotion code (datatype code), we select from T-359 by timestamp all the values of
the promotion.
If the promotion value in the T-359 is not inside the list of the promo values in the table, a
condition code 16 is issued.
We can have 2 types of errors:
➢ The promotion value is missing from the promotion table and the user has to update the
promotion table
➢ The promotion table is wrong and the user is allowed to correct the T-359 by the on line
correction screen applying the reclassification of the promotion value.
After the online correction or the updating of the promotion table, it is requested to run again the
validation of the promotions to complete the validation cycle.
The online correction is used to modify the value of a promotion and not to delete a promotion (by
value = 0).
If you want to eliminate a promotion from a purchase transaction you have to use the job “rawda
correction”
Recommendations :
Constraints :
Prerequisites :

Page 40 from 698
Table T-359 should be loaded (Data Selection job) and promotion values should be instructed
(Validation -> Occasion screen).
Table Description
OCCASIONT04 Contains list of possible promotion code / values by cycle
P359_1T12 DV validation area data
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters :
Static Parameters
No static parameter.
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

Process XXXX YYYY.PPP normal Validate the bonus pack instructions
Process XXXX YYYY.PPP F# Not allowed , run “normal” option x process
Process XXXX YYYY.PPP PC Not allowed , run “normal” option x process
Rework XXXX YYYY.PPP normal Not allowed , run “normal” option x process
Rework XXXX YYYY.PPP F# Not allowed , run “normal” option x process
Rework XXXX YYYY.PPP PC Not allowed , run “normal” option x process

Page 41 from 698
Validations->Occasion screen should be updated with allowed promotion values by promotion
codes for submission cycle.

code
P1014004 Invalid Promotion Value Invalid W Promotional codes Enter valid
Promo Val are invalid promotional
codes into
P359_1T12
table
P0020501 Array List Blank Array List S Datacode is empty Open QLS
Blank
P0020552 Error in Sorting Char System S open a QLS
Arrays Error
P0020554 Error while checking for System S open a QLS
existence in Char Arrays Error
P1014001 Error while retrieving System S Open a QLS
Distinct Family Count Error
P1014002 Error while retrieving System S
Count of Distinct Families Error
with Promotions
P1014003 Error while retrieving the System S
Count of Unique Baskets Error
with Promotions
P0020115 Data Value not found in DataValue
359 table not in T12
P0020121 Error in getting the System S
DataValue from359 table Error
P0020101 Error Type is blank Error Type S
is blank
P0020102 Error Type not present Err No not S
present
P0020103 Error while retrieveing the System S
Error Type Error
P0020146 Error while writing record System S Open QLS
into Datalog Full Record Error
Array
P1014007 Error while checking System S Open QLS
Promotional Codes Error

Page 42 from 698
P1014006 Total number of Distinct Information message only. No action required
distinct families Family Cnt
validated for
Timestamp
P1014009 Total number of Promotion Information message only. No action required
distinct Baskets Basket Cnt
with promotion
P1014010 Total number of Promotion Information message only. No action required

purchase acts with PurAct Cnt
promotion
P1014011 Total number of Wrong Information message only. No action required

purchase acts with Promo Cnt
Wrong promotion
information

Page 43 from 698
EHDV21015P – Formal Price Validation
Phase :
In some countries, prices are collected from the families. The procedure for the price collection is
established from the country.
In some countries, the price can be collected only for some categories or some type of shops.
The missing prices will then be filled with RMS prices or by ad hoc estimation procedures.
The price validation program is running the following steps:

➢ Control of the price of the purchase transaction by the standard price
➢ Replace the price value 0 by the standard price
The formal price validation is driven by the “price group” concept: the prices are validated within a
set of shops with similar characteristics for the price fixing. Shop characteristic used for Price
group is instructed in static parameter IDCH01.
The purchase data of the period P and timestamp T needs to be validated.
We have 2 controls based on the standard price of the period T-1.
Control 1 :
We control the price reported by the family by an interval around the standard price of period P-1
using the lower and upper limits [PVPL and PVPU are static parameters driving this check].
If the price is inside the limits ( i.e. > PVPL x STD PRICE and < PVPU x STD PRICE), the
inspection is completed successfully.
If the test is not satisfied, we control the absolute difference between the price and the standard
price with a reference value stored in static parameter table (MINDIF parameter).
If the absolute difference is less or equal than the reference value, the inspection is completed
successfully
If the above tests are not satisfied : the price of the purchase act will be written in to Data log. The
price value will be proposed to the user for the manual inspection and the on line correction.
Control 2 :
If the price of the purchase transaction is missing, FPV process is imputing a price using following
steps :

Page 44 from 698
Source of STND PRICE Product group level
NAN level
Price group XXXX Step 1 Step 3
Std Price for period P-1 by price look for std price look for stnd price
group (shop characteristic) by nan || price group by product group || price group
[optional, in order to complete the

FPV validation and not to stop the
process to fill the stnd price for the
product (mainly new products)]
Price group XXXX Step 1 bis
Latest std Price for period range look for std price
[P-1- N ; P-2] by price group by nan || price group
(shop characteristic)
N is defined as static parameter

Price group 0 Step 2 Step 4
Std Price for period P-1 for all look for stnd price look for stnd price
shops by nan || price group = 0 by product group || price group=0
[ optional]
Price group XXXX Step 2 bis
Latest std Price for period range

[P-1- N ; P-2] by price group look for stnd price
(shop characteristic) by nan || price group = 0
A price correction flag datatype (5904) is updated with specific value based on imputation step
(see appendix for datatype values)
Before comiting the updates, program is performing following check
Compute ratio R : (number of purchase acts imputed step 3 + number of purchase acts imputed
step 4 ) / total number of purchase acts selected by FPV
If R > LIM_STEP34 (static parameter)

Raise Severe Error : Number of prices imputed at product group level too high, rollback
Once all the above steps finished, some items have a price, others have standard price equal
to zero these items have records with temporary datacode 9998. These items should be
Imputed with DEF_STD_PRICE in the following condions :
- If number of disnct nan non assigned with price < NB_ITEMS_MAX and each nan
has the purchase < NB_PURCHASE_MAX then the price should be assigned with
DEF_STD_PRICE. These items should be updated into the GT_STDPT999 with the

Page 45 from 698
datacode DT5675 (datacode related to the price instead of temporary 9998). As the
price is changed, these items should be flagged with the price correcon flag
DT5904 with a new value 20.
. Once these steps are done, insert into the

P359_1T12 is done using GT_STDPT999
- Else nothing to do
- Log all items being assigned with DEF_STD_PRICE
- Report number of items assigned with the DEF_STD_PRICE
- End with warning in case default price assigned but no more records with price= 0
Recommendations :
Constraints :
Any missing price after Step1/2/3/4 should be loaded using load Std price.
Prerequisites :
Shop usability table updated with the shop characteristic file from the sample by the “cycle/period
to be validated”. The price-group must be one of the shop characteristics.
The computation of the standard price must be prepared before the validation starts.
The standard price is computed on the data of the past period(s) already validated and loaded in
the rawda.
Table Description
SHOPUSABILITYT40 It contains the characteristic identifier and the characteristic value by
shop by cycle and period (0 or YYYYPPP)
To be updated by cycle/period at each validation cycle (if requested)
CNANPRICET34 Table containing standard price and MAD by cycle, price Group
DATALOGT55 The table contains the outliers of the quality controls system candidate
for the user inspection and on line selection.
Files:
Input Files:
No input file.
Output Files :
File containing values with no standard price, this file can be amended manually and reloaded
with Load Standard Price job (File id: FILEO1):

1 4 CYCLE Submission cycle
5 7 PERIOD Submission period -1
12 16 AC-ITEM NAN_KEY

Page 46 from 698
25 10 PRICE Fixed to 0
GROUP
35 4 DATATYPE Fixed to ‘5653’ (standard price)
CODE
39 16.5 PRICE VALUE Fixed to 0, can be updated
File containing more details about missing prices (barcode, shop), this file can be used to retrieve
prices from other system (File id: FILEO2) :

5 7 PERIOD Submission period -1
12 16 ITEM NAN_KEY
28 35 BARCODE BARCODE
63 10 SHOP SHOP
Parameters :
Static Parameters
Parameter Type Value Name Description

Type
(example)
PARM PVPL 0.30 Price value Parameter Lower limit
PARM PVPU 1.30 Price value Parameter Upper limit
PARM MINDIF 2 Minimum absolute difference price and the std price to
report warning in DatalogT55
MODL STEP3 Y Option to use or not Step3 of FPV logic
CHID IDCH01 PRCGRP Shop Characteristic used for “Price Group”
MODL STEP4 Y Option to use or not Step4 of FPV logic
PARM NUM_PER_STD 12 Number of periods back considered in steps 1 bis and 2
bis
PARM LIM_STEP34 0.3 Limit for proportion of purchase acts imputed by step3
and 4
Eg : if 100 purchase acts are validated by FPV, if
LIM_STEP34=0.3, FPV will fail if #records imputed by
step 3 + #records imputed by step 3 is greater than 30
PARM DEF_STD_PRICE 1 Default price to be assigned
PARM NB_ITEMS_MAX 5 Max number of items to be assigned by default 5
PARM NB_PURCHASE_ 5 Max nb of purchase by nan by default 5
MAX
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No

Page 47 from 698
DTSelection No
File Selection Yes
Rework Management :


code
P0020105 Shop Char value not Shop Char S
present val absent
P1015001 No Transaction available NoData for S
for the Selected Timestamp
TimeStamp
P1040001 Missing Price Group Missing S
Price
Group
P0020589 Error while fetching Std. System S Open QLS
Price for Price Group Error
P0020562 Standard Price not Std.Price W Open QLS

present absent
P1015002 Price reported by the Price W Price can be
family is outside the outside limit amended using
deviation limit online
correction
screen
P1015003 No Purchase No S
Transactions available. Purchase
Transact
P9900204 Number of distinct PC – Zero Price W
Item – Price group Count
combination without std.
price
P0020115 Data Value not found in DataValue S
P0020121 Error in getting the System S Open QLS
DataValue from359 table Error
P0020101 Error Type is blank Error Type S
is blank

Page 48 from 698
P0020102 Error Type not present Err No not S
present
P0020103 Error while retrieveing the System S
Error Type Error
P0020146 Error while writing record System S
into Datalog Full Record Error
Array
P1015004 Error while calculation System S
Standard Price Error

P9900216 Number of Number of Information message only. No action required
Purchase acts with Purchase a
Price=0 after all
steps of Formal
Price Validation
P9900217 Number of distinct Number of Information message only. No action required
NANs with Price=0 distinct N
after all steps of
Formal Price
Validation
P9900117 Number of Warnings Information message only. No action required
Warnings Count
Purchase acts with Purchase a
Price=0 before
Formal Price
Validation
P9900215 Number of distinct Number of Information message only. No action required
NANs with Price=0 distinct N
before Formal Price
Validation
P9900116 Number of Severe Severe Information message only. No action required

Errors Error Count
P9900119 Total Number of Validated Information message only. No action required

Purchase Acts Record Cnt
Validated
P1015006 Number of ShpTyp Information message only. No action required
purchase acts for Missing Cnt
which Shop Type is
missing
P9900205 Number of distinct Zero Price Information message only. No action required
PC – Item – Price Count
group combination
without std. price
P0150001 Impute by Price Information message only. No action required
nankey/pricegroup> Imputation
0 current

Page 49 from 698
nankey/pricegroup> Imputation
0 historic
nankey/pricegroup Imputation
0 current
nankey/pricegroup Imputation
0 historic
P0150005 Impute by product Price Information message only. No action required
group/pricegroup>0 Imputation
current
P0150007 Impute with Default FPV Information message only. No action required
Std Price in FPV
P0150006 Impute by product Price Information message only. No action required

group/pricegroup 0 Imputation
current
P0150008 Items assigned with FPV Information message only. No action required
DEF_STD_PRICE
P0150009 Default price FPV Warning message only. No action required
assigned,no more
records with
price=0

Page 50 from 698
EHDV21016P- Purchase Inspection
Phase :
The purchase inspection is detecting the unusual behaviour of the family based on the
purchase units compared with the behaviour of all the families purchasing the same product.
The outliers of the purchase transactions are proposed to the user for the manual inspection and
the correction by the on line correction screen.
Only purchase units greater than 0 are validated.
Logic applied to detect outliers is the following :
By NAN_KEY,
14. Select the T observations who have the largest volumes x (among x>0; if less then T
available, we just select the ones we have available).
The L observations we obtain in this way are sorted and stored in an array x
2. Determine the critical limit as

Crit.value = Mn+(Md-Mn)*log(alpha)/log(0.5)
Where Mn = Minimum(x)-1
Md = Median(x)
Recommendations :
Constrains :
Prerequisites :
Table Description
DATALOGT55 The table contains the outliers of the quality controls system candidate for
the user inspection and on line selection.
Files:
Input Files:
No input file.
Output Files :
No output file.

Page 51 from 698
Parameters :
Static Parameters
Parameter Type Name(example) Description

Type Value
PARM ALPHA 0.10 critical limit we want to apply to the tail (suggested value
0.10)
PARM T 5 number of observations we select from the tail (suggested
value 10% of sample size)
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


code
P1016001 No Transaction available NoData for S Check the
for the Selected Timestamp selected
TimeStamp Timestamp
P1016006 Purchase Value > Critical PurchVal>Cr W This value can
Value iticalVal be updated
using Online
correction
P0020115 Data Value not found in DataValue S Open QLS
P0020121 Error in getting the System Error S Open QLS
DataValue from359 table

Page 52 from 698
P0020101 Error Type is blank Error Type is S Open QLS
blank
P0020102 Error Type not present Err No not S Open QLS
present
P0020103 Error while retrieveing the System Error S Open QLS
Error Type
P0020146 Error while writing record System Error S Open QLS
into Datalog Full Record
Array
P1016002 Error while Checking System Error S Open QLS
Purchase Value
P0020501 Array List Blank Array List S Open QLS
Blank
P0020502 Error while calculating System Error S Open QLS
Median
P0020551 Error in Sorting Number System Error S Open QLS
Arrays
P1016005 Error while calculating System Error S Open QLS
Critical Value

P1016008 Total Number of PC – PC – Item Information message only. No action required
items in KEYS table Cnt
for the Timestamp
Validated
P9900117 Number of Warnings Warnings Information message only. No action required
Count

Page 53 from 698
EHDV21018P- Multipack Validation
Phase :
This program is used to validate the “multipack information” reported by the families using a
purchase unit inspection process.
We select Multipack product by selecting all items with multi-pack factor EIMDB characteristic > 1
(this characteristic is defined in static parameter MLPK).
We check quantity reported by the family using following logic :
➢ check if the purchase unit is a multiple of the multi-pack factor in EIMDB
➢ check if the purchase unit is greater than a given “parameter x” ( by product group )
➢ if “parameter x” is missing for the selected “product group” ,
get the parameter value for the product group 0
➢ if both the conditions are satisfied, the purchase unit reported by the family will be
corrected by
purchase unit = purchase unit / multi-pack factor in IMDB
Recommendations :
Constrains :
Prerequisites :
➢ Table T-359 should be loaded (Data Selection)

➢ Define the default char id in EIMDB to identify multipack products (static parameter
MLPK)
➢ Define the customized “parameter x” by product group for inspection (screen
Validation→Conversion for job EHDV21018P)
➢ Define default “parameter x” by product group = 0 for inspection
Table Description
FAMILY_LOGT67 List of changes applied by DV automatic validation jobs

Page 54 from 698
NAN_CHAR_VALUEST75 Item characteristics values id extracted from EIMDB
CHAR_VALUEST24 Definition characteristic values id extracted from EIMDB
PRDGR_FACTORT94 Instruction coming from screen Validation→Conversion
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters :
Static Parameters
Parameter Type Name (example) Description

Type Value
CHID MLPK 1323 EIMD characteristic code used as Multipack
factor
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

Process XXXX YYYY.PPP F# Not allowed, run “normal” option x process
Process XXXX YYYY.PPP PC Not allowed, run “normal” option x process
Rework XXXX YYYY.PPP normal Not allowed, run “normal” option x process
Rework XXXX YYYY.PPP F# Not allowed, run “normal” option x process
Rework XXXX YYYY.PPP PC Not allowed, run “normal” option x process
Screen Validation→Conversion should be updated for job EHDV21018P

Page 55 from 698

code
P0020522 Error in retrieving Data System Error S Open a QLS
Code for Tag
P0020575 Error while generating System Error S Open a QLS
DataCode List for the
given Tag List
P0020579 Error while populating the System Error S Open a QLS
MsgLog Array about
Invalid Tag-DataCode
Static Values
P1027007 Error while Preparing the System Error S Open a QLS
data
P0020505 Static Value not present Stat.Val not S Check if all
present required Static
parameter are
defined for
submission
cycle
P9900034 Invalid DataCode Invalid S Open a QLS
DataCode
P0020523 No Data Codes present in No DT for S Open a QLS
Destination Table dest.Table
P0020534 Error while retrieving Data System Error S Open a QLS
Code for Destination List
P0020501 Array List Blank Array List S Open a QLS
Blank
P0020552 Error in Sorting Char System Error S Open a QLS
Arrays
P0020554 Error while checking for System Error S Open a QLS
existence in Char Arrays
P1027006 Error in setting DT System Error S Open a QLS
Pointers

P0180001 Number of Records # Records Information message only. No action required
Updated P359_1T12 Updated
P0180002 Number of Records # Records Information message only. No action required
Updated Updated
S0000004 Record added Record Information message only. No action required
successfully added
success

Page 56 from 698
EHDV21019P- Private Brand – Extraction
Phase :
The “private brand” validation is checking the consistency between the “retailer” where the family
went shopping with the manufacturer of the products included in the shopping basket.
Whether we find an inconsistency and we identify a “private brand product” associated to the
wrong retailer, we apply an automatic correction and we re-classify the purchase transaction into
a new basket where we build the consistency between the “retailer” and the “manufacturer” of the
product (retailer itself).
The procedure is built by two programs:

➢ Based on the product characteristics from the EIMDB, we extract all private brands
products present in data to be validated, (job EHDV21019P)
➢ Validation process of the baskets (job EHDV21020P)
EHDV21019P process
We extract all items present in KEYST14 table for selected cycle/timestamp.

From this, we filter the private brand items. This is done by selecting from EIMD all products with
PL1C characteristic with values in (CHVAL01, CHVAL02…). PL1C, CHVAL01, CHVAL02… are
instructed as static parameters.
Then we get the private brand value from EIMDB for all selected products (EIMDB char to be
used is instructed as static parameter PL2C).
All information are loaded in PRIVLAB_PCITEMT66 table.
The manufacturer of the private brands is stored as product characteristic in IMDB.
We need define the rules for the recognition of the private brand product.
Recommendations :
Constrains :
Prerequisites :
➢ EIMDB contains the private label identification.

➢ Shop usability table contains the chain identifier of the retailers.
➢ Instructions for the PB validation available (screens : Validation → Private Brand )

Page 57 from 698
Table Description
CHAR_TYPEST17 List of product characteristic codes from EIMDB
CHAR_VALUES_TRANST Description of product characteristic codes from EIMDB
25
PRIVLAB_PCITEMT66 List of private brands products for cycle/timestamp in validation
KEYST14
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters :
Static Parameters

Type Value
PARM LANG IT Language to extract char value from EIMDB
CHID PL1C 3828 Characteristic code of the private label
identifier into EIMDB
CHID PL2C 3824 Characteristic code of the private label brand
in EIMDB
PL1C CHVL01 2291240 First characteristic value of the private label
identifier in EIMDB
PL1C CHVL02 2291239 Second characteristic value of the private
label identifier in EIMDB
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No

Page 58 from 698
Rework Management :

Process XXXX YYYY.PPP normal allowed

code
P9900163 PL1CID without the char PL1CID – no W
value into IMDB charval
P1019007 Error while executing System Error S
Ehsp_ProcessPBExtractio
n
P1019003 Error while retrieving data System Error S
from T37
P1019009 Private Labels not Found PrivateLbl S
not found
P1019001 Error while retrieving data System Error S
from T05

P1019010 Number of private PB product Information message only. No action required
brand products found count
P1019008 No Private brand No PB Information message only. No action required
products found and no Products
records Inserted

Page 59 from 698
EHDV21020P- Private Brand Validation – process
Phase :
EHDV21020P process
The validation of the Private Brands will be applied to a limited set of purchase transactions by
the following criteria:
➢ we will inspect only the shop types selected by the user
➢ we will inspect only the private brands selected by the user
shop types :
User selects the list of the shop types by an on line screen (Validation → Private Brand
→ Shop Type)
Shop/Shop Type instruction are stored in shop usability table
one or more shop types can be selected.
Private brands
➢ user can select the list of the private brands to be inspected by an on line screen
(as instructed in Validation → Private Brand → PB)
➢ one or more PB’s can be selected
➢ if a PB present in the IMDB is not selected and included in the list of the on line
instructions , we will ignore all the products of that retailer and we will treat the
products of that PB as standard products belonging to a manufacturer (no
checks for error, no reclassification in case of errors)
when we identify an inconsistency between a product and the characteristics of the shops to
which the product has been assigned, we reclassify the purchase transaction by the following
rules :
if
➢ a shopping basket doesn’t contain retailer products , we stop the inspection
➢ a shopping basket contains a product ( or more than one ) belonging to the
same retailer and the chain code of the shop of the shopping basket is not
representing the retailer of the products , we change the shop code of the
whole basket selecting a new shop code having the same shop type and the
shop chain of the retailer owning the PB products.
The shop code used for reclassification is instructed in following online screen :
Validation → Private Brand → PB Shop.
➢ shopping basket contains more than one PB products belonging to different
retailers ( more than one ),
➢ we open a new shopping basket ,

Page 60 from 698
one for each retailer ( the new shop has the same shop type and the shop
chain of the retailer) and we move to this basket only the purchase
transactions belonging to that retailer
The new shop code used is instructed in following online screen : Validation
→ Private Brand → PB Shop.
We leave all the original purchase transactions ( manufacturer only) into the
original basket with the same original shop.
➢ the purchase transaction re-classified into new baskets are deleted

from the original baskets
➢ the total basket information are not modified and they will be adjusted
later in the rawda by a dedicated procedure ( basket adjustment ,
basket validation )
All changes applied are logged in FAMILYLOGT67 table and can be checked using FamilyLog
online screen.
Recommendations :
Constrains :
EHDV21019P should be completed first.
Prerequisites :

Table Description
PRIVLAB_PCITEMT66
KEYST14 Keys present in DV validation area
PRIVATE_LABELT21 Data as defined in screen :Validation → Private Brand → PB
PRIVATELABEL_SHOPT22 Data as defined in screen :Validation → Private Brand → Shop

Page 61 from 698
PRIVATELABEL_BASKETT Data as defined in screen :Validation → Private Brand → Shop type
72
SHOPUSABILITYT40 It contains the characteristic identifier and the characteristic value by shop
by cycle and period (0 or YYYYPPP)
PRIVLAB_PCITEMT66 List of changes applied by DV automatic validation jobs
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters :
Static Parameters

Type Value
CHID SHPTYP TYPE
CHID SHCHAN CHAN
CHID PL2CID 3824
PARM LANG IT Language to extract char value from EIMDB
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


Page 62 from 698

code
P1020025 Error while executing System Error S
Ehsp_ProcessPBValidatio
P0020104 Shop Char Value is blank Shop Char S
Val blank
P0020105 Shop Char value not Shop Char S
present val absent
P0020116 Error in retrieving the System Error S
Shop Characteristic value
P9900155 No records in table No records S There is no data in Check
char_values_transt25 for found char_values_transT Language
Selected Language: 25 table for the selected in
selected Static Static
Parameter Parameters
Language
P1020008 Error checking if the shop System Error S
belonging to the selected
basket has shop type
value
P1020011 Error loading CNAN data System Error S
P1020012 Private label without chain PLbl – No W
information chain info
P1020013 Error loading IMDB data System Error S Open QLS
P1020024 Error while executing System Error S Open QLS
Ehfn_controlChain
P1020014 No Shop code found in No Shop S
the shop info table. code found
P1020023 Error while changing shop System Error S Open QLS
codes
P1020022 Error while identifying System Error S Open QLS
shop codes
P1020021 Error while sorting data System Error S Open QLS
array
P1020015 Error while deleting old System Error S Open QLS
data
P1020006 Error retrieving proc id System Error S Open QLS
from T14
P1020016 Error while inserting data System Error S Open QLS
into EH14V01 to adjust
keys
P1020017 Error while inserting new System Error S Open QLS
purchase acts
P1020026 Data value not found in System Error S Open QLS
EH12V01
P1020018 Error while updating HH- System Error S Open QLS
Log

Page 63 from 698
P1020020 Error while adding System Error S Open QLS
purchase acts and
updating HH-log
P1020019 Error while checking if System Error S Open QLS
atleast one shop with
given type is present
P1020002 Shop type code not Shptyp not in S Open QLS
available for validation in T40
T40
P1020001 No instrns given by the No PB S
user for validation – check Instrns
requested shop type
codes
P1020003 For the selected shop No Shop to S
types, no shop codes validate
available for validation in
T359
P1020028 Error While deleting the System Error S Open QLS
data from Table T66
P9900052 CharID Missing CharID S
missing

P1020029 Number of purchase Reclassificati Information message only. No action required
acts reclassified on Cnt
P1020012 Private label without PLbl – No Private Label ID and description provided.
chain information chain info

Page 64 from 698
EHDV21022P – Rawda Loader
Phase :
PHASE 2 – AUTOMATIC DATA VALIDATION
The rawda loader is moving data validated from P359_1T12 table to the final rawda tables :
➢ RAWDAILYT60
➢ TBSKT13
➢ PROCKEYST29
Rawda loader system is composed in 2 sections :

1. Loading phase
2. Deletion of the data and instructions just loaded from the working tables
1. Loading phase
We check of the key is already present in Rawda :

➔ If key coming from table P359 is not already present in Rawda we insert in Rawda
➔ If key coming from table P359 is already present in Rawda, the key is not inserted and
if File# is >0 the job is failing
The key loaded in tables RAWDAILYT60 ,TBSKT13, PROCKEYST29 is the following

▪ audit
▪ date
▪ time
▪ Household code
▪ shop code
▪ barcode
In table RAWDAILYT60 we load detailed purchase data coming from table P359 (keys with
Barcode>0).
When loading purchase data in RAWDAILYT60, a control on price value is applied :

if price = 0 → a severe error is raised and the job fails
In table TBSKT13 we load total basket data coming from table P359 (keys with Barcode=0).
All keys loaded in both RAWDAILYT60 and TBSKT13 are loaded also in PROCKEYST29
When RERUN=Y
o Only the data for filenumber = -1 are taken in consideration as part of process
o ALLT11 data removal will use the initial file number instead of the -1 stored in validation
area using the basket key
Datatypes present in PROCKEYST29 are following :

➢ the file number [ F# ] indicating the origin file of the purch.transaction
▪ Any data correction applied manually by the user will reset the value of F# to 0
(zero)

Page 65 from 698
➢ the “lock flag” indicating whether the cycle/period of the data validation has been already
completed . Those purchase transactions key marked by “1” are available for the
data processing.
▪ All the purchase transaction loaded into rawda after the “lock flag” process set up
will not be used for the data processing having the “lock flag = 0” (unless you run
the lock flag process in rework mode).
2. Deletion of the data and instructions just loaded from the working tables
The same purchase transactions will be deleted from the intermediate tables of the system where
we perform the validation of the data.
➔ delete intermediate table P359_1T12 for all keys related to the current Timestamp
➔ delete intermediate table KEYST14 for all keys related to the current Timestamp
➔ delete intermediate table DATALOGT55 for all keys related to the current Timestamp
➔ delete keys moved to the rawda from the initial loaded table (ALLT11 )
Remark : for Homescanner system, in table ALL now only un-coded purchase
transactions remain
done at the end of this process in terminateJob for these four tables:
1.Allt11
2.Prockeyst29
3.Rawdailyt60
4.Tbskt13
Recommendations :
Constrains :
Prerequisites :
Table Description
FILE_LOGT59 List of Files # loaded in DV (can be access in screen Services → File log)
DATALOGT55 The table contains the outliers of the quality controls system candidate for
the user inspection and online selection.
FAMILY_LOGT67 Table containing all information related to HH collaboration at daily level.
PROCKEYST29 List of Rawda Processing keys

Page 66 from 698
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters :
Static Parameters
Parameter Type Name Mandatory Description

Type Value (example)
PARM RERUN Y or N NO Rerun mode ‘Y’ or ‘N’
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


code
S00C0050 Column not in structure Column not S open a QLS
in struct

Page 67 from 698
P9900214 Can NOT RELEASE Can NOT S open a QLS
Lock. RELEASE
Lock
P0020150 No data found in Pr-Keys No PRKEYS S
for datacode data
P0020135 Error while 68anellist68 System Error S
the DataCode Info for T60
P0020149 Error while retrieving System Error S
DataValue from PrKeys
P0020137 Error while 68anellist68 System Error S
the DataCode Info for T16
P0020138 Error while Loading Key System Error S
for RawDa Loader
P0020154 Audit Not Present in No Audit in S
AUDITDEFT01 T01
P0020155 DataCode Not Present in DataCode S
T07 Not Present
P0020156 Pc-Item Not Present in Pc-Item Not S
T31 Present
P0020157 Family Code Not Present Family Not S
in T43 In T43
P0020158 Shop Code Not Present in Shop Not In S
T44 T44
P0020151 Error in inserting into System Error S
destinations
P0020141 Error in LoadOutput – System Error S
RawDa (in Ehpk_RawDa)
P0020602 Position of DataCode not Position of S
found DataCode
P0020601 Source Value is null Source S
Value is null
P0020600 Error while getting Source System Error S
Position
P0020556 Cell value is not present Cell not in S
in Type Table type tab
P0020553 Error while checking for System Error S
existence in Number
Arrays
P1022002 Insert Key already present System Error S
in T13
P1022006 Price=0 found during Price=0 S
RAWDA Loading Phase found
P1022004 Transaction in T359 and Diff F#- S
RAWDA with different File T359-
No. not updated RAWDA
P1022013 Error in DeleteTAll System Error S

Page 68 from 698

P1022021 Number of Records P359T12 – Information message only. No action required
present in P359T12 Rec cnt
P1022022 Number of records Insert Cnt – Information message only. No action required
inserted in to Rawda
RAWDAILYT60
inserted in to TBSK
TBSKT13
P1022026 Number of Records Delete Cnt – Information message only. No action required
Records deleted from ALLT11
ALLT11
deleted from T359
P359_T12
deleted from KEYS KEYS
table – KEYST14
P1047005 Number of records Delete Cnt- Information message only. No action required
deleted from T55
DataLogT55
families for the Family Cnt
Timestamp
P9900165 Transaction in T359 Diff F#- Information message only. No action required
and RAWDA with T359-
different File No. other RAWDA
than 0
P1022001 No. of Products not System Error Information message only. No action required
Updated for RAWDA
due to wrong File No.

Page 69 from 698
EHDV21023P – Trip Data Alignment
Phase:
In CPS purchase data collected from families we can have some errors like:
➢ assignment of a private brand product to a generic shop or a shop of a different
retailer
➢ assignment of a product to a wrong shop type
During the data validation process, DV will correct these errors by dedicated routines:
➢ private brand validation
➢ basket validation…
DV applies in such case a reclassification of the whole basket or part of it by changing shop code
(or family code or the time of the shopping trip in other cases).
The new basket generated contains purchase transactions only. The information of the shopping
trip is not duplicated from the original basket to the new basket during the validation steps.
Trip data alignment is used to align the content of the RAWDA and the content of the TBSK
after the loading phase of the final tables by copying the following information into the new
basket:
Primary shopper sex

Primary shopper year born
secondary shopper sex
secondary shopper year born
payment type
Fidelity card
Others …….
List of DT to be copied are instructed as DATATYPEST07 entry.
Copy of tbskt information for reclassified keys
When a family introduces an error, like a product into a wrong shop type, we assume that the
total basket information reported by the family are correct as copied from the cash-lip of that
shopping trip.
When we correct the error, by the basket validation, and we generate a new basket, we accept
the concept that all the information of the original basket are correct.
For the new basket we suggest the following procedure:

➢ copy the status information (shoppers, type of payment, fidelity card.)
➢ do not copy the quantitative information (value, units, #-transactions.)
We will compute the quantitative information of the new basket using DV “basket validation
program” (24P), This 24P should be so submitted after “basket adjustment program” and before
the “lock of the production cycle”.

Page 70 from 698
The quantitative information will be based on the purchase transactions associated to the new
basket (example: total value spent =  Pricei x unitsi, where I is the single purchase transaction)
The trip data alignment program doesn’t substitute the set of quality control routines for the total
basket transaction finalized to prepare the information for application using total basket
information (like PFW – trip data file …)
Delete of Tbskt record without purchase data
Trip data alignment is checking all total basket record without any purchase data
For all those records program is checking if any purchase date is remaining in table ALL (un-
xcoded data)
In case that the job is submitted with the static parameter EMPTY_BASKET=N or without defining
an EMPTY_BASKET static parameter then any total basket records without any purchase
transaction in Rawda or in Table ALL will be deleted from TBSKT13 and PROCKEYST29.
Submitting the job with this option will delete not only the reclassified records but also records in
TBSKT13 and PROCKEYST29 which have only total basket information coming from input data
itself.
Otherwise if the job is submitted with the static parameter EMPTY_BASKET=Y then only the
basket records which have been reclassified and lead to empty baskets will be deleted.
At end of job the orphan baskets in T13 are deleted and count is informed to user via error
message number.
Note : For python version Hive data refresh is done at the beginning for these four tables:
1.Allt11
2.Prockeyst29
3.Rawdailyt60
4.Tbskt13
For python version oracle data refresh is done at the end of this process in terminateJob for these
three tables:
1.Prockeyst29
2.Rawdailyt60
3.Tbskt13
Recommendations:
Trip data alignment and trip data validation are managing alignment between purchase
keys & tbskt keys. This alignment is mandatory for aggregation system computing Tbskt
facts (DP09 onward). For this reason 23P & 24P are highly recommended to be included in
standard DV flow.
➢ Submit before the job locking production cycle [EHDV21026P]

➢ Don’t use this job if your “cycle/audit” contains total basket data only
➢ Should be submitted before Trip Data Validation
Constraints:
Prerequisites:

Page 71 from 698
Rawda should be loaded
Table Description
ALLT11 All the data collected by the families are loaded inside TALL
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters:
Static Parameters

Type (example)
PARM EMPTY_BASKET Y/N IF ‘Y’ THEN
➢ All the empty baskets due to
reclassification will be deleted
ELSIF ‘N’ or not defined THEN
➢ All the empty baskets due to
reclassification and also all empty
basket coming from input data will be
deleted
Dynamic Parameters

Name (Yes/No)
Period Yes Submission with Period 0 is allowed for this job
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 72 from 698
Rework Management:
Process Type Cycle Period Option

Rework XXXX YYYY.PPP F# Not allowed
By visiting Services -> Job DT layout - > JDT1 - Services - job DT layout page user can view
the list of data types instructed in the screen.
Below is the list of mandatory DT’s which job will copy from source to destination basket
DT code DT description
5695 Points Voucher Used
5071 Day of the week
5944 Primary Shopper Sex
5945 Primary Shopper Year Born
5946 Secondary Shopper Sex
5694 Home delivery code
5943 Payment Type
5949 Fidelity Card
5084 Receipt ID
5082 Conversion Rate
5693 internet order
5947 Secondary Shopper Year Born
5892 Historical Total Shop Expenses
Additional if there is a need to instruct any other datatype please use the GUI Services -> Job
DT layout - > JDT1 - Services - job DT layout page and insert in a new position

code
P9900114 Production cycle already Rework S Resubmit in
completed-Rework mode requested rework mode
requested

Page 73 from 698
P1029002 DataCode not present in DataCode S Check static
the selected DataCode not present parameter
List
P1023008 DataType Codes missing No DT in S No Data codes are Check static
for TBSK alignment in staticparam defined in static parameter
Static Parameter Table parameter table.
P1023009 DataType Code not found DT not in S The data code Check static
in DT catalog T07 defined in static parameter
parameter table is
not present in the
master list of data
codes.
P1023010 Error in getting STATIC System Error S Some error occurred Check static
VALUE while retrieving the parameter
data codes from
static parameter.
P1023002 Total Basket Adjustment TBSK Adj- W No Total Basket It is only an
not Applicable- No not allowed records are present warning
records type 0 available for the given Audit message. No
and Period. action required.
P9900174 Cycle with TBSK only. Cycle with S No purchase data
Total Basket Adjustment TBSK only available
not applicable
P1023016 No Purchase acts present No Purchase W No records present It is only a
in Final Tables for Trip Acts in RAWDA for the warning
Data Adjustment given Audit – Period. message. No
required
P1023007 Error in Rawda Baskets System Error S Some error occurred Open a QLS
during the Rawda
Baskets.
P1023012 No Data found in T29 System Error S Data inconsistency Open a QLS
P1023011 Error while checking System Error S Some error occurred Open a QLS
HHLOG in the inner block of
Check HH Log
P1023006 Error while checking HH System Error S Some error occurred Open a QLS
LOG while checking the
HH Log.
P1023013 Error while Inserting into System Error S Duplicate key found Open a QLS
T13 – Duplicate Value in TBSK table during
Found insertion.
P1023015 Error while inserting into System Error S Duplicate key found Open a QLS
PR-Key – Duplicate Value in T29 table during
Found insertion.
P1023014 Error while generating System Error S Some error occurred Open a QLS
New TBSK during Generate
New TBSK.
P1023001 Error while Deleting System Error S Some error occurred Open a QLS
Records while deleting TBSK.
P1023005 Error while checking in System Error S Some error occurred Open a QLS
PRKeys while checking in
PRKeys

Page 74 from 698
P1023004 TBSK of the RAWDA not System Error S Some error occurred Open a QLS
found in TBSK while checking
TBSK of the
RAWDA
List of job Counters in MsgLog:

P1023018 Number of records Insert Count- Information message only. No action required
inserted into TBSKT13
TBSKT13 table
deleted from PRKEYS
PROCKEYST29 table
TBSKT13 table
P1023003 Main Basket Key not Main Bskt Information message only. There is purchase
found not found data without total basket information
P1023022 Number of Orphan delete Cnt- Information message only. No action required
Basket records TBSKT13
deleted from
TBSKT13

Page 75 from 698
EHDV21024P – Trip Data Validation
Phase:
The data validation system includes some routines for the validation / preparation of the total
basket data.
This program is the part of the project is to provide a “set of tools” in order to fill into the “total
basket table“and “ready to use” information required by the business applications.
The total basket data are used for the following business purposes:
➢ Trip data file for PFW
➢ Infact DB with projected data by the Standard Processor ( early indicator for retailers )
➢ Shopper track service for retailers
The information for the trip data file can be generated by different sources:
➢ from DV standard flow, as information coming from panellists ( barcode = 0 )
➢ By a process, located into the data validation system, deriving the data from the purchase
data rawda (RAWDAILYT60).
Sometimes, not all the information are present into the 359 file or the data validation system can
modify the original values loaded inside the database and therefore a subsequent batch process
is required in order to generate the missing / valid data into the total basket table.
This program is proceeding with following logic :
➢ Select all basket keys in RAWDAILYT60 (AUDIT/DATE/TIME/HHOLD/SHOP)
➢ For each key compute total basket datatypes based on instructed rule.
➢ If basket key is present in TBSKT13 table => updating computed datatype values for
existing key
➢ If basket key is not present in TBSKT13 table => insert key with computed datatype
Selection by TBSK and File Number not used in this program as we can have the same basket
composed by data coming from different file numbers:
• Original basket file
• data manipulation activity
• rework from a new file number
If file number is applied, we can have not correct results.

Page 76 from 698
List of datatypes that are computed by the job(fixed list)
DT code DT description Routine Routine description
5017 Any Promo V010 Any Deal – 5017

5666 # Units Purchased V001 Sum Purchase Qty(5650) by BSKT –
5666
5950 Number of Different V002 Number of Items purchased by BSKT
Items computed on – 5950
Rawda
5748 Total Value Spent on V012 Tot val spent PROMO-5748-
Promo- by hh Max(5749,5748)
5749 Outlet Value on Promo V011 Total outlet value on PROMO-sum
PVp-5749
5755 Total Outlet value V003 Total Outlet value by BSKT
Computed as sum[DT5650xDT5675]
5756 # Transactions on Deal V005 # Transactions on Deal – 5756
5757 Activity Deal 1 V006 Specific Promotion active in the
Basket
Basket
Basket
Basket
5941 Total Shopping V004 Total Value Spent – 5941-
Expenses by hh MAX(sum[DT5650xDT5675],5892
5676 Total number of items V013 Total number of items collected 5676
collected – MAX(DT5676, DT5950)
5951 Number of units V014 Number of units purchased under
purchased under deal/promotion – DT5951
deal/promotion
The routines for the validation can be viewed online via screen Validations→TBSK
Recommendations:
Trip data alignment and trip data validation are managing alignment between purchase keys &
tbskt keys. This alignment is mandatory for aggregation system computing Tbskt facts (DP09
onward). For this reason, 23P & 24P are highly recommended to be included in standard DV flow.
Constraints:
Prerequisites:
➢ Complete data validation for the current CYCLE / TRIP

➢ Run the TRIP DATA ALIGNEMENT procedure
o To validate basket data and
o To align the output tables with the PR-KEY tables
➢ Select the routines for the validation by the on line screen

Page 77 from 698
Table Description
TBSKCOMPT70 Contains DT computation instruction, defined in GUI TBSKT screen
OCCASIONT04 Contains list of possible promotion code / values by cycle
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters:
Static Parameters
Parameter Type Name Description

Type Value
PARM BCDTYPE 1 It represents the barcode type , usually the
hhp collaborator ( not packaged goods)
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 78 from 698
Rework Management:
Process Cycle Period Option

Type
Process XXXX YYYY.PPP Normal Allowed
Process XXXX YYYY.PPP TBSK Not allowed
Rework XXXX YYYY.PPP Normal Allowed
Rework XXXX YYYY.PPP TBSK Not allowed
The routines for the validation can be viewed on line screen Validations→TBSK

code
P1024034 Period is locked – Rework Rework S
option required Required
in struct
record in input
file
P1024016 Error in System Error S
Ehsp_GetExceptionPCs
Ehsp_TripDataValidation
P1024022 Datatype requested for DataCode S
computation is not defined not in T07
in T07
P1024019 No Routines for Active S
Computation No Routines
Ehsp_LoadRoutine
Ehsp_Load_Basket_Purc
hase
Ehsp_ComputeDT
P1024023 Source Datatypes are not SourceDT S
correctly defined in T07 not Defined
P1024007 Error in Ehfn_V001 System Error S

Page 79 from 698
P1024021 Data Validation not Bskt Lock- S
Completed and Basket Cycle open
Locked
Ehsp_Update_Basket
Ehsp_Prepare_Output
Ehsp_Update_Table
Ehsp_AlignPRKeys
P0020597 Error while getting the System Error S
position in Char Array

P1024030 Processing cycle is Period Information message only. No action required
closed closed
P1024031 Processing cycle is Period not Information message only. No action required
not closed closed
P1024026 Number of T60 Bskt Information message only. No action required
RAWDAILYT60 Process Cnt
baskets to be
processed
inserted in to TBSK
TBSKT13
P9900241 Number of records Update Cnt Information message only. No action required
updated in TBSKT13 – TBSK
P1024027 Number of baskets Open and Information message only. No action required
open and computed computed
P1024028 Number of baskets Close-not Information message only. No action required
closed and not computed
computed
P1024029 Number of Baskets Closed and Information message only. No action required
Closed and Computed Computed
P1024032 No PC Exception No PC Information message only. No action required
required for fresh food Exception
P1024020 Total Value Spent < Totvalue<Pr Information message only. No action required
Total Value Spent on omoValue
Promo – Data
Adjusted
P1024033 Number of Routines Information message only. No action required
active for Active
Computation Routines

Page 80 from 698
EHDV21025P – Lower Bound limit computation
Phase:
PHASE 3 – HH INFORMATION SYSTEM
This job is computing Lower Bound limit for Weekly Usability Computation
Lower bound is the minimum value that a family should have reported to be considered as usable
for Expansion factor computation.
Lower bound may depend from the size of the family, so this job has been designed to computed
a Lower bound by family size.
Lower bound is computed for each period, based on the purchase acts values present in the DV
Rawda.
Result of this computation is stored in table HHPARAMT06 table:
AC_CYCLE Cycle code

NC_PERIOD Period code
AC_CHAR_ID Char code for family size
AC_CHARVAL Char value for family size
AC_DATACODE 5091 for lower bound
NC_DATAVALUE Lower bound value
Logic of the job
Get the list of families present in FamilyusabilityT41 for submission period & cycle
Delete date present in HHPARAMT06 table submission period & cycle
Compute total spent (DT5650 x DT5675) by family for submission period & cycle
Get Family size
For each family size value

Order purchase value in ascending order
Compute median ( q2 )
Compute third quartile (q3 )
Apply mathematical formula
LB = lower bound = q2 / Exp ( z * s )
Where
z = z value of the normal distribution = 3 (defined as static parameter)
s = 2 * ln ( q3 / q2 ) / c
c = constant = 1.349 (defined as static parameter)
Update the HH parameter table T06 by

CYCLE / period / char id / char value / datacode / datavalue
Recommendations:

Page 81 from 698
Constraints:
➢ Job locking production cycle [EHDV21026P] has to be submitted before launching 25P
Prerequisites:
➢ Rawda should be loaded and period locked
Table Description
RAWDAILYT60 Purchase Rawda
FAMILYUSABILITYT41 Family characteristics table
HHPARAMT06 Family usability parameters by cycle/period/HH_Char_id/HH_Char_Val
Files:
Input Files:
No input file.
Output Files:
No output file.
Parameters:
Static Parameters

Type Value
PARM ZZND 3 Statistical parameter
PARM CC01 1.349 Statistical parameter
CHID HHSIZE 0000000001 Characteristic code for family Size
MODL BOUNDL 2006020 To be set to BOUNDL
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No

Page 82 from 698
DT Selection No
File Selection No
Rework Management:

Rework XXXX YYYY.PPP normal Not allowed

code
P1025002 Model Identifier not MODEL!=BO S Check Static
present in T05 UNDL parameter
Error while deleting from Open a QLS
P1025006 HH Parameter. System Error S
Characteristic requested Check Static
not found in Family HHSIZE not parameter
P1025004 Usability table. in T41 S
Error while checking Open a QLS
P1025011 Family Usability table System Error S
Error during retrieval of Open a QLS
P1025007 Purchase values System Error S
Error during retrieval of Open a QLS
P1025008 Family Char System Error S
Error during computation Open a QLS
P1025009 of Median and 3rd Quartile System Error S
Error during insertion of Open a QLS
P1025010 lower value bound System Error S

P1025012 Number of distinct Cnt of T41- Number of values for Size
char values in T41 charvalue

Page 83 from 698
EHDV21026P – Update proc-keys
Phase :
In the data validation, we have decided to have the possibility to move to the rawda
➢ the family baskets received after the data processing completion
➢ the purchase transactions linked after the data processing completion
in addition we have decided to guarantee the stability of the information delivered by the
“tracking” service.
In order to conciliate these two requirements, we will mark all the purchase transactions validated
and ready for the “data processing” for a selected cycle/period.
The mark is a flag assigned by the “update processing keys” process.
All the purchase transactions moved to the output tables after the above process will not be used
by the processing system. In case of rework we are able to produce the same information for the
clients.
In case we need for the “data processing” the data loaded in the final table after the “update
processing keys” process, we give to the user the possibility to run again the “update
processing keys” process in rework mode and to define a new set of purchase transactions for
the processing system.
Data for the client will be changed under the responsibility of the Operation Department.
➢ when we run the job in processing mode , we take all the purchase transactions in the
selected cycle-period
➢ the lock-flag is assigned to
▪ the purchase transactions

▪ the total basket information
➢ when we run the job in rework mode , we take all the purchase transactions in the
selected cycle-period
➢ when we run the job in rework mode by F# , we take only the purchase transactions
identified by F# selected in the job submission screen for the selected
cycle-period
➢ when we run the job in rework mode by PC(s) , we take only the purchase transactions
identified by the PC code(s) selected in the job submission screen for the selected cycle-
period
➢ the lock flag can be assigned to any purchase transaction when we load the information
by the “rawda correction” job.
Process Logic:
• Initial disabling of trigger

Page 84 from 698
• Direct update of the 3(T13,T60,T29) tables one by one
• In rework mode consider to update unlocked data only
• When RERUN=Y
o Set back DT5095 file number to initial value instead of the -1 stored in prockeys
table using the basket key
• Implementing parallel queries
• Final enabling of trigger
• Proper use of Rollback and trigger enabling commands in termination step for the
failure cases
At the end of 26P, if the job completes succesfully or completes with warning, the value for
COPY_ON_CLOUD in Setup_Param will be checked. If it is 'N', no action performed.
If it is 'Y', a python module will be launched which will copy the data from DV server(source) to
DP server(destination).
Setup_Param Content Details:
• COPY_ON_CLOUD : This param can be set for the country to have one of the two
values: 'Y' or 'N' - By default, the param_id ‘COPY_ON_CLOUD’ in Setup_param is set to
‘N’.Whenever the user wants to initiate a copy, the user must set ‘COPY_ON_CLOUD’ to
‘Y’
• ADLS_CONTAINER : azure ADLS container
• AZCOPY_APPLICATION_ID : azure application id
• AZCOPY_SPA_CLIENT_SECRET : azure client secret
• AZCOPY_TENANT_ID : azure tenant id
• AZURE_PORTAL_URL : azure portal url
• DP_HIVE_REMOTE_ADDRESS: remote(destination) DP server db address
• DP_REMOTE_DB_NAME: remote(destination) DP server dbname
• DP_TABLES: List of tables that needs to be refreshed to cloud separated by ',' Eg:
rawdailyt60,tbskt13
Recommendations :
Constrains :
Prerequisites :
To be run after rawda loader, at the end of the validation phase.
Table Description
TDV_ITEM_EVENT_TRACKING Item event tracking
Files:
Input Files:

Page 85 from 698
No input file.
Output Files :
No output file.

Page 86 from 698
Parameters :
Static Parameters
Parameter Type Name Mandatory Description

Type Value (example)
PARM RERUN Y or N NO Rerun mode ‘Y’ or ‘N’
PARAM ONLY_RE Y or N NO ONLY_REFRESH_PA
FRESH_P RQUET mode ‘Y’ or ‘N’
ARQUET
Dynamic Parameters

Name (Yes/No)
# Periods Yes Selection not required (leave field blank).
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection Yes
DTSelection No
File Selection No
Rework Management :

Rework XXXX YYYY.PPP F# Allowed
Rework XXXX YYYY.PPP PC Allowed

code
P9900115 Production cycle already Optn not S
completed-Processing accepted
options not accepted
P1026005 Processing option not Rework not S
Correct – Rework not required
required
P1026001 Error while updating T29 System Error S

Page 87 from 698
P9900005 Required No. of Periods not Information message only. No action required
Periods not Present enough
P1045012 Number of Records Update Information message only. No action required
updated in count-
PROCKEYST29 PRKEYS
P1026003 No Records Present No PrKeys Information message only. No action required
in PROCKEYST29 record
table

Page 88 from 698
EHDV21028P – Weekly Family Usability computation
Phase:
This job is computing a weekly usability flag at Audit / Period / Household level.
A Family is declared usable if one of following condition is verified:

- Holiday flag is present in FAMILYLOGT67
- No Shopping flag is present in FAMILYLOGT67
- The Total Spent value at HH/Period is OK based on HHCELL or BOUNDL logic
The weekly usability flag is stored in FAMILYMONITORT42, structure for this table is following:
AC_CYCLE Cycle code
AC_HHOLD Panellist code
AC_DATACODE Datatype code
NC_DATAVALUE Datatype value
DT_UPDATE Update date
The usability is replicated on Rawda tables in specific DT (5096).
Logic of the job
Select all families for which usability is to be computed
Job is selecting from T41 all Household present for submission cycle/period.
Reset FamilyMonitoT42
All records for DT 5900 are deleted from T42 for submission cycle/period range
Reset forced flag in FamilyLog
All force activity flags log are deleted from T67 for submission cycle/period range
Check all families in Holidays or no Shopping
From FAMILYLOGT67 job is selecting all HH with Holidays datatype (5643) or No

shopping datatype (5684) present for the period range.
Compute total spent on period using content of TBSKT13
By Household job is computing sum of TBSKT DT instructed as static parameter for

period range
Check if total spent in inside or outside limit
We have two logics to compute usability:

Page 89 from 698
For HHCELL model:
- From T41 we get HHCELL char value for the 90anellist,

- From T06 we get the
DT5092 value [ minimum purch. Value bound ]
DT5097 value [ max purch. Value bound ]
For submission period (if no instruction found for submission period job is using
most recent one
Family is declared usable if: DT5097 >= Total spent value on period >= DT5092
For BOUNDL model:
- From T41 we get HHZISE char value for the 90anellist,

- From T06 we get the DT5091 value [ Low bound ] for submission cycle/period (if no
instruction found for submission period job is failing)
Family is declared usable if: Total spent value on period >= DT5091
Write report
For all families with Holidays flag or No Shopping flag or total spent>0, we produce a flat
file containing usability flag and total spent value.
Insert activity flags in T42
For all families selected for weekly usability computation we load DT5900 value (1 if
family usable, 0 if not)
Update in Prockeyst29
Usability is replicated on T29/T13/T60 tables.
Recommendations:
➢ 28P is resetting all 5900 flags in T42, which means that any manual change on 5900 by
35P should be re-applied in case 28P is re-submitted
Constraints:
Prerequisites:

➢ Instructions in T06 should be present:
-BOUND Instructions present in the HH parameter table T06 ( by batch )
-CELL LIMITS Instructions present in the HH parameter table T06 ( by on line )
The instructions for a given cycle and period are valid for all the following
periods.

Page 90 from 698
Table Description
FAMILYMONITORT42 Table containing weekly usability information
FAMILYLOGT67 Log of Families collaboration information
HHPARAMT06 Family usability parameters by cycle/period/HH_Char_id/HH_Char_Val
Files:
Input Files:
No input file.
Output Files:
Report file (File id: FILE01)

n
1 4 CYCLE Cycle
5 7 PERIOD YYYYPPP
12 10 HH CODE HouseHold code
22 4 DT CODE Datatype code
26 16 LYT_FLAG Value of usability flag as loaded in T42 (xxxxxxxxxx . xxxxx)
42 20 Model Model used for validation (HHCELL or BOUNDL)
62 10 HH-char value Value of HH char (HHSIZE of HHCELL depends on model)
72 16 Tot purch value Total spent (xxxxxxxxxx . xxxxx)
Special families file (File id: FILE02)
This file is containing list of households that declared holidays or No shopping for this period

n
Purval outside limit file (File id: FILE03)
This file is containing details about households not OK based on HHCELL model.

Page 91 from 698
n
11 10 HH-char value Value of HH char (HHSIZE of HHCELL depends on model)
21 4 DT CODE Datatype code for total spent
25 16 Tot purch value Total spent (xxxxxxxxxx . xxxxx)

Page 92 from 698
Parameters:
Static Parameters

Type Value
MODL BOUNDL logic to use to compute usability
or HHCELL
DTDT IDDT01 5951 DT for purchase value in T13
DTDT IDDT02 5900 DT for weekly usability flag in T42
DTDT IDDT03 5096 DT for family usability flag in 29
CHID HHSIZE SIZE HH Char_id to use in case MODL=BOUNDL
CHID HHCELL CELL HH Char_id to use in case MODL= HHCELL
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

Instruction of limits by for purchase values by period, char value => Services -> HH Parameters

code
P1027004 Error in checking System error S Open a QLS
purchase value bound
Error in checking Open a QLS
P1027005 purchase value cell System error S
Error in setting DT Open a QLS
P1027006 Pointers System Error S

Page 93 from 698
Error while Preparing the Open a QLS
P1027007 data System Error S
Family code not Check T41,
present in T41 for load missing
Family-char id not in Family not in cycle/period with char
P1027011 FamilyUsability table. T41 S correct Char_id
Error in retrieving the data Open a QLS
from Rawda –
Ehsp_getHHPurchaseVal
P1027014 ue System Error S
Error in retrieving the data Open a QLS
from TBSK –
Ehsp_getTBSKHHPurcha
P1027015 seValue System Error S
P1027016 Error in get All families System Error S Open a QLS
Error in retrieving the Open a QLS
P1027017 Family Char System Error S
Error in checking HH Log Open a QLS
P1027018 – Ehfn_chkHHLog System Error S
Error in checking Loyalty- Open a QLS
P1027019 Ehfn_CheckLoyalty System Error S
Error in Writinfg to output- Open a QLS
P1027020 Ehsp_Writeoutput System Error S
Error in getting Wrong Open a QLS
P1027021 Families System Error S
Error in Perform Family Open a QLS
P1027022 Usability System Error S
Family Usability is Lock the period
Production Cycle not Period not running only on (26P)
P1027025 closed closed S locked period
Error in checking Open a QLS
P1027005 purchase value cell System error S
Error in setting DT Open a QLS
P1027006 Pointers System Error S
No data No instruction for Check screen
found in T06 family usability HHParameter
P9900026 No data found in T06 S S
DT instructed as Check DT
static parameter not instructed as
No Data Codes present in No DT for defined in DT static
P0020523 Destination Table dest.Table S S definition parameter

P1027012 Total number of family Family Cnt Total number of HH present in T41 for
inspected: cycle/period
P9900267 Total Number of # Families in Total number of HH that declared holidays or
Families in “holidays” “holid No shopping for this period
or “no purchase”

Page 94 from 698
P9900268 Total Number of # Families Total number of HH with at least on shopping
Families tot spent > 0 tot spent trip for the period with Tot spent>0
Families tot spent > 0 tot spent trip for the period with Tot spent>0, and for
ok based TBSKT which Tot spent is OK based on usability rule
usability rule
Families tot spent > 0 tot spent trip for the period with Tot spent>0, and for
NOT ok based TBSKT which Tot spent is not OK based on usability
usability rule rule
P9900271 Total Number of # Families Number of HH computed as usable for
Families set usable in set usable cycle/period
T42
P9900272 Total Number of # Families Number of HH computed as non usable for
Families set non set non u cycle/period
usable in T42

Page 95 from 698
EHDV21029P – Common Sample Family Usability computation
Phase:
This job is computing usability of the family for the computation of the expansion factor
by index/period/common sample. It creates a flat file that can be then loaded in Sample DB using
job: JES137 – Load Production Condition.
The usability of the family for the expansion is based on the weekly usability flags in the
FamilyMonitor table computed by job 28P.
The following parameters are required to be instructed as static parameter:

Sample Source Type code
Sample DB index code,
Sample DB period code
Common Sample codes (can be more then one),
For each Common Sample code
Total number of DV periods
Minimum number of DV periods to consider family as usable.
Logic of the job
Get the list of families present in FamilymonitorT42 with period <= submission period and
cycle=submission cycle
Get period definition for submission cycle
For each common sample get number of periods, min number of periods for usability
for each family
compute number of weekly usability flag = 1 for list of common sample periods
if >= to minimum instructed then common sample usability flag set to 1
for this family/cs
else common sample usability flag set to 1 for this family/cs
Results of this computation are then written in an output file.
Recommendations:
Constraints:
Prerequisites:

➢ Weekly usability computation (28P) should be completed

Page 96 from 698
Table Description
CYCLEPERIODT03 Periods definition
Files:
Input Files:
No input file.
Output Files:
This file is intended to be loaded in Sample DB using JES137 – Load Production

Condition job.
1 4 AC_INDEX Index Code
6 2 AC_SRTY Source Type
9 10 AC_COLL_ID Collaborator Code
20 3 AC_EXPF Common sample code
24 7 NC_PER Period Code
32 1 FL_CSMPL_BEL Family Usability flag
ONG
Parameters:
Static Parameters

Type Value
DTDT IDDT02 5900 DT for weekly usability flag in T42
SRTY SRTY01 10 Source type code in Sample DB
IXIX IXIX01 0090 Index code in Sample DB
IXPE IXPE01 2006020 Period code in Sample DB
IXCS IXCS01 004 Common Sample code #1
IXCT 004 4 Total number of periods in CS 004
IXCM 004 3 Min number of periods for usability in CS 004
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 97 from 698
Rework Management:


code
P1029002 DataCode not present in DataCode S Check static
the selected DataCode not present parameter
Check static
Record in Family Monitor parameter and
for cycle,period and data No Record in if 28P was
P1027005 code T42 S submitted
Error while checking for Open a QLS
record in Family Monitor
P1029004 table System Error S
Check periods
definition and
static
P1029005 Insufficient no. of periods Less periods S parameter
Error in Perform Sample Open a QLS
P1029006 Link System Error S
Tot Check static
Total Periods < Minimum Pers<Min parameters
P1029008 Period Pers S
P9900004 S Value Error Value Error S Open a QLS

P1029011 Number of Records o/p file rec
written in to the o/p count
file
P1029012 Total Number of CS Count
Common Samples

Page 98 from 698
EHDV21034P – Load Household Characteristics
Phase :
PHASE 4 – SERVICES
We load the list of families / characteristics for each cycle/period.
The family characteristics are extracted from the Sample DB or from any other repository present
in the company and they are loaded into the FAMILY USABILITY TABLE.
We use the family characteristics in data validation system in the following areas:
➢ For the management information system area where we determine the family usability
by
o The “total spent” by family by CELL
o The “minimum spent” by family by HH SIZE (the “lower bound” criteria)
➢ For the alignment of the family codes from the data collection area and the list of families
in the “family usability table” (job Check Family Code Ehdv21066P)
➢ To supply the family characteristics to PFW file
➢ For some Data Validation routines (Heavy Buyer Edit)
Household characteristics can be loaded in DV using two modes : DBLINK or FILE.
➔ In case FILE mode is used, hh chars info should be providing in a flat file (see details bellow).
➔ In case DBLINK mode is used, data are retrieved by DBLink from SampleDB system (this
option is valid for countries using NCPS Sample DB).
Logic to retrieve data from Sample DB by DBLINK is similar to the logic used by Sample DB job
JE3030. Sample DB source type and index should be instructed as static parameter of 37P. Then
data will be extracted from Sample DB T55 tables using list of chars, length and tags instructed
as JE3030 static parameter in Sample DB for submission Index.
In sample DB T55 table, chars values with date_from < DATE1 and date_to > DATE1
If DV period = 0, DATE1=system date
If DV period <>0, DATE1= end date of DV submission period
Recommendations :
If a family code in a purchase transaction is not associated to a selected family

characteristic, request to update the Sample DB and to reload into DV the file of the family
characteristics (otherwise purchase transaction will not be moved by DataSelection job).
Constrains :
The family codes are not shared across DV audits.
Prerequisites :
File with HH characteristics available in case of FILE option.

Page 99 from 698
In case of DBLINK option, DBLINK should be existing, JE3030 static parameter should
be present and data in T55 table of Sample DB should be up to date.
The input file can be extracted from European Sample DB using Sample job JE3030.
In case of DBLINK option, data are retrieved from Sample DB Oracle database.
Table Description
FAMILYUSABILITYT41 It contains the characteristic identifier and the characteristic value by Family
by cycle and period (YYYYPPP)
To be updated by cycle/period at each validation cycle
Files:
Input Files:
Family characteristics (File id: FILEI1) :
Required only if FILE mode is used.

n
1 2 SRCTYPE Not used
3 10 HH CODE Household Code
13 7 PERIOD Period
20 3 NO_CHAR Number. Of Characteristics
23 16 CHAR_ARRAY Array of (Characteristic Code (char 6), Characteristic Value
(200) (char 10)) of maximum 200.
Output Files :
No output file.
Parameters :
Static Parameters

Type
PARM SOURCE DBLINK Mode used to load data : DBLINK or FILE.
If not present, mode will be set to FILE by
default.
PARM SRCTYPE 03 Sample DB SourceType code for retrieving
data from SampleDB T55 table (used only in
case of DBLINK option)

Page 100 from 698
PARM INDEX 9000 Sample DB Index code for retrieving data
from SampleDB T55 table (used only in case
of DBLINK option)
PARM EXTRTYPE ALL/ACTIVE If set to ALL char for all collab in T51 will be
extracted, if set to ACTIVE only chars for
collabs belonging to submission index code
are extracted
PARM LAYOUT_SET e.g 1000 / 1005 etc Allignment with the SMP12.2/12.3
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :

Rework XXXX YYYY.PPP F# Not allowed, run “normal” option x rework
Rework XXXX YYYY.PPP PC Not allowed, run “normal” option x rework

code
not present name
P9900014 Empty File Empty File S File is Empty Enter file with
values
P1034002 Error while deleting from System S
T41-Rework Mode Error
P0020552 Error in Sorting Char System S Open a QLS
Arrays Error
P0020554 Error while checking for System S Open a QLS

Page 101 from 698
P9900001 File Handling-Invalid Wrong file S
Operation operation
P9900002 File Handling-Read Error File Read S
Error
P9900004 Value Error Value Error S
P1034001 Error while identifying System S
CharIds Error
T41-Rework by ChId Error
Mode
P1034005 Error while loading Family System S
Char File Error
P9900060 Blank Hhold Blank S Hhold is blank Enter Hhold
Hhold with values
P9900084 Hhold is zero Hhold is S Hhold should not be Give positive
zero zero values for
Hhold
P9900086 Blank CharId Blank S CharId field is Blank Enter Value for
CharId CharID
P1034004 Error while validating System S Family char file is Check for
Family Char File Error Invalid errors in Family
Char File
P9900087 Blank CharValue Blank S Char Value is Blank Enter value for
CharValue Char Value

the input file
P9900124 Number of incorrect Wrong Information message only. No action required
Records Present in Record
the input file Count
P1034006 Number of new New Information message only. No action required
families inserted Family
into FAMILYT43 Count
P1034007 Number of records Insert Information message only. No action required
inserted into Count –
FAMILYUSABILITY T41
T41
P1034008 Number of records Update Information message only. No action required
updated into Count –
FAMILYUSABILITY T41
T41
P1034009 Number of records Delete Information message only. No action required
deleted from Count –
FAMILYUSABILITY T41
41

Page 102 from 698
EHDV21035P – Load Family Monitor
Phase :
This job is to be used to update the HH Monitor table by exception. Update is done based on an
input file prepared by the user
This is a single period job. Multiple periods inside input file not allowed
Logic for FAMILYMONITORT42 update :
Any datatype belonging to FAMILYMONITORT42 table can be loaded in this table, structure for
this table is following :
AC_CYCLE Cycle code

AC_HHOLD Panellist code
AC_DATACODE Datatype code
NC_DATAVALUE Datatype value
DT_UPDATE Update date
One datatype has a special management: datatype defined by static parameter IDDT02: [HH
activity flag in HH Monitor]. Standard value for this static parameter is – 5900
For each record in input file we check if there is already information in T42:
➔ If there is no record present for this Cycle/Period/HH/datatype, the record is inserted
➔ If there is already a record present for this Cycle/Period/HH/datatype, the datatype value is
updated
In case the updated datatype is the datatype [HH activity flag in HH Monitor] a record is
inserted in FAMILYLOGT67 to keep a trace of this change.
Logic for PROCKEYST29 update :
In case static parameters IDDT01 and IDDT02 are both different from 0, the update of
PROCKEYST29 family activity is applied.
Step 1 :
Program is selecting from HH monitor the list of Households for submission Cycle/Period with
IDDT02 [5900] = 1
➔ we update PROCKEYST29 with IDDT01 [5096] =1 for the submission
Cycle/Period
Step 2:
Program is selecting from PROCKEYST29 all families present for job submission Cycle/Period
From this list we get all Households that are not present in FAMILYMONITORT42 with
IDDT02 [5900] = 1

Page 103 from 698
➔ All Households present in PROCKEYST29 and not set as usable in
FAMILYMONITORT42 are updated with IDDT01 [5096] =0 for the submission Cycle/Period

Page 104 from 698
Rework option :
This is a full rework by period. If this option is selected:

➔ we delete all the information for the data-types present in the file in table
FAMILYMONITORT42 for submission Cycle/Period
➔ we delete all the information for the data-types present in the file in table
FAMILYLOGT67 submission Cycle/Period for job = EHDV21035P
Recommendations:
This job should be used to perform corrections to the family usability computed by Family
Usability program.
Constrains:
Following checks are performed on input file, if one check is not OK, job is ending with severe
error :
- Household code should not be blank and should not be 0
Household code should be present in table T43 (any Household with one char
loaded in DV is automatically loaded in T43)
1 Datatype code should be defined in DV and should belong to

FAMILYMONITORT42 table
The cycle & period code in input file are not used and not checked by the program. The
submission 105anel & period will be used to insert in FAMILYMONITORT42.
Prerequisites:
This job should be used after Family usability computation and after Rawda load of data (to
ensure that family usability change are applied on Prockeyst29).
Table Description
FAMILYMONITORT42 Table containing family usability flag by period
FAMILY_LOGT67 List of changes applied by DV automatic validation jobs and family
collaboration data
PROCKEYST29 Table containing processing keys for purchase and tbskt records
FAMILYT43 List of Households present in FAMILYUSABILITYT41 table

Page 105 from 698
Files:
Input Files:
Family characteristics (File id: FILEI1) :

n
1 4 CYCLE Cycle
5 7 PERIOD Period
12 10 HH CODE Household Code
22 4 DATATYPE Datatype code (only datatype belonging to T42 can be
CODE loaded)
26 16 DATATYPE Datatype value xxxxxxxxxx.xxxxx
VALUE (5 decimal)
input file to be sorted by “cycle / period / family code”
Output Files :
No output file.
Parameters :
Static Parameters

Type Value
DTDT IDDT01 5096 If this static parameter is defined, update of
ProckeysT29 datatype for family usability is
updated.
DTDT IDDT02 5900 This static parameter is mandatory. It defines
the Hhmonitor activity flag DT code.
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :


Page 106 from 698

code
P1035002 DTCode not Found in DTCode S In input file record Correct
DATATYPES table for not in T07. with non valid datatype code
destination datatype was found in input file
FAMILYMONITOR
P1035004 Error while deleting from System S System error Open a QLS
P1035006 Error while loading Family System S System error Open a QLS
Info File Error
P1035007 Error while validating System S System error Open a QLS
Family Info File Error
P9900040 Missing Family Code Missing S Household present Check
Family in input file is not in Household
Code DV T43 table code in input
file, if correct
load char for
this Household
P9900060 Blank Hhold Blank S Blanck household Check record
Hhold was found in input file and
correct HH
code
P9900084 Hhold is zero Hhold is S household = zero Check record
zero in input file and
correct HH
code
P9900120 Number of Records with Wrong S In input file record Change input
Wrong Cycle Cycle Cnt with wrong cycle file with correct
were found cycle or submit
for another
cycle
P9900121 Number of Records with Wrong S In input file record Change input
Wrong Period Period Cnt with wrong period file with correct
were found period or
submit for
another period
P9900122 Number of Records with Wrong S In input file record Correct
Wrong Data Codes Data Code with non valid datatype code
Cnt datatype was found in input file

Page 107 from 698
P1035008 Number of records Inserted
inserted in to Record
Family Monitor Cnt
Table
P1035009 Number of records Updated
updated in Family Record
Monitor Table Cnt
P1035010 No records Deleted No Rework option was selected but no record were
from Family Monitor Records deleted
Table Deleted
P1035011 Number of records Deleted Rework option was selected and some records
deleted from Family Record were deleted
Monitor Table Cnt
P9900123 Total Number of Record
the input file
P9900124 Number of incorrect Wrong

Page 108 from 698
EHDV21036P – Load Shop File
Phase :
We keep inside the system a directory of all the shops used within the CPS system.
These shops are the “source of purchases” declared by the families when they report the
information of the shopping trip. The description is used to facilitate the activities of the user
during the preparation of the on-line instructions.
List of shops codes with their description can be loaded in DV using two modes : DBLINK or
FILE.
➔ In case FILE mode is used, list of shops should be providing in a flat file (see details bellow).
option is valid for countries using NCPS Sample DB). No input file is required when this option is
used.
JES023. Sample DB source type should be instructed as static parameter of 36P.
In sample DB T51 table, only shop codes with date_from < DATE1 and date_to > DATE1
Recommendations :
This job should be submitted before Load Shop Characteristic. Recommendation is to

submit this job using period 0 in case of generic shop code, so that list of shops should be
available for all periods.
Constrains :
The list of shops codes is unique in a country for all DV Audits.
Prerequisites :
File with shop description available in case of FILE option.

In case of DBLINK option, DBLINK should be existing and data in T51 table of Sample
DB should be up to date.
The input file can be extracted from European Sample DB using Sample job JES023.

Page 109 from 698
Table Description
SHOPST44 It contains the shop code and the shop description

Page 110 from 698
Files:
Input Files:
Shop list (File id: FILEI1):

1 10 AC_SHOP Shop code
11 80 TX_DESCR Description
Output Files :
No output file.
Parameters :
Static Parameters

Type
default.
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :
Proc.TYPE CYCLE PERIOD Option


Page 111 from 698

code
P9900014 Empty File Empty File S The file is empty Replace a file
without any values with values
P9900001 File Handling-Invalid Wrong file S Error is manipulation
Operation operation of the file
P9900002 File Handling-Read Error File Read S Error in reading the
Error file
P1036002 Error while Loading Shop System S Unhandled
File Error exception.
Check the
Oracle Error in
key field.
Report the
error to the
local support
team or CPS-
CC.
P9900061 Blank Shop Code Blank Shop S Shop code is empty Enter a valid
Code shop code
P9900074 Invalid Shop Invalid S Shop details Enter a valid
Shop provided is not valid shop detail
P1036001 Error while Validating System S Unhandled
Shop File Error exception.
Check the
Oracle Error in
key field.
Report the
error to the
local support
team or CPS-
CC.

the input file
P1036003 Number of Records Insert Information message only. No action required
inserted into Count-
ShopT44 table ShopT4
4

Page 112 from 698
P1036004 Number of Records Update Information message only. No action required
updated into count-
ShopT44 table ShopT4
4

Page 113 from 698
EHDV21037P – Load Shop Characteristics
Phase :
We load the list of shop / characteristics by cycle/period
The shop characteristics are extracted from the Sample DB or from any other repository present
in the company and they are loaded into the SHOP USABILITY TABLE.
We use the shop characteristics in data validation system in the following areas:
➢ For the data validation routines (basket validation, private brand validation, Formal
price validation, Automatic Price Validation)
➢ For price imputation
➢ For the alignment of the shop codes from the data collection area and the list of
shops in the “shop usability table (job Check Shop Code Ehdv21067P)
➢ To supply the shop characteristics to CASE file
Shops characteristic can be loaded in DV using two modes : DBLINK or FILE.
➔ In case FILE mode is used, shop chars info should be providing in a flat file (see details
bellow).
option is valid for countries using NCPS Sample DB).
When the mode is DBLINK, get the list of back periods PERIOD_REC_ARRAY according to the
Period and #Periods parameters. 1 means submitted period only. O is not possible.
JE3030. Sample DB source type and index should be instructed as static parameter of 37P. Then
data will be extracted from Sample DB T55 tables using list of chars, length and tags instructed
as JE3030 static parameter in Sample DB for submission Index.
In sample DB T55 table, chars values with date_from < DATE1 and date_to > DATE1
Option to refresh, instructions for Xcoding by shop
DV is managing a logic to Xcode barcodes by shop code. This logic is driven by link between
shop code and Xcoding group stored in table SHOPCOLLID_LINKT19.
In case of countries where individual shop codes are used, maintenance of shop code/Xcode
group can be quite huge workload. To manage this case, 37P has an option to refresh shop
code/Xcode group link automatically using :
Link between Shop Chain and Xcoding group stored in CHAINCOLLID_LINKT110 (this
info should be stable)
Link between Shop Chain and Shop code stored in SHOPUSABILITYT40
In case FULL refresh option is used, data in SHOPCOLLID_LINKT19 are deleted for submission
cycle, then full list of Shop code / XcodeGroup links are refreshed.

Page 114 from 698
In case INC option is used, only shop code present managed by current run of 37P will be
considered. If shop code is present in T19 for submitted audit, information for group_id/flag will be
updated. If shop code is not present in T19 for submitted audit, new record will be inserted in T19.
All the shop group related to the submitted cycle will be resolved and its respective shop contents
will be stored in TDV_SHOP_GROUP_CONTENTS and its respective resolved entry is added in
TDV_SG_RESOL_TRACKING.
Recommendations :
If a shop code in a purchase transaction is not associated to a selected shop

characteristic, it is requested to update the Sample DB and to reload into DV the file of the shop
characteristics (otherwise purchase transaction will not be moved by DataSelection job).
Recommendation is to submit this job using period 0 in case of generic shop code, so
that shops chars should be available for all periods.
NCPS DP PFW extractor is using shop characteristics for period=0, so if shop

characteristics are used in PFW it is indeed mandatory to keep shop char for period 0 up to date.
Constrains :
The list of shops codes is unique in a country for all DV Audits, but characteristic can be
different by Cycle.
Prerequisites :
All shop codes present in Shop char file should have been loaded before in DV using
Load Shop Description.
File with Shop characteristics available in case of FILE option.
In case of DBLINK option, DBLINK should be existing, JE3030 static parameter should
be present and data in T55 table of Sample DB should be up to date.
The input file can be extracted from European Sample DB using Sample job JE3030.
Table Description
SHOPST44 It contains the shop code and the shop description
SHOPUSABILITYT40 It contains the characteristic identifier and the characteristic value by shop by
cycle and period (0 or YYYYPPP)
CHAINCOLLID_LINKT110 Instruction for Xcoding by chain

Page 115 from 698
Files:
Input Files:
Shop characteristics (File id: FILEI1) :

1 2 SRCTYPE Not used
3 10 SHOP CODE Shop Cod
13 7 PERIOD Period
20 3 NO_CHAR Number. Of Characteristics
23 16 CHAR_ARRAY Array of (Characteristic Code (char 6), Characteristic Value
(200) (char 10)) of maximum 200.
Output Files :
No output file.
Parameters :
Static Parameters

Type
default.
PARM INDEX 9000 Sample DB Index code for retrieving data
from SampleDB T55 table (used only in case
of DBLINK option)
PARM EXTRTYPE ALL/ACTIVE If set to ALL char for all collab in T51 will be
extracted, if set to ACTIVE only chars for
collabs belonging to submission index code
are extracted
PARM XLINK FULL or INC Option to populate T19 using T110 instruction
and shop chars values that are loaded by
27P. If not present, T19 is not updated.
PARM LAYOUT_SET e.g 1000 / 1005 etc Allignment with the SMP12.2/12.3

Page 116 from 698
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :


code
P9900014 Empty File Empty File S The file is empty Replace a file
without any values with values
P9900002 File Handling-Read Error File Read S Error in reading the
Error file
P0020552 Error in Sorting Char System S
Arrays Error
P0020554 Error while checking for System S
P1037001 Error while identifying Error while S
CharIds identify
T40-Rework by ChId Error
Mode

Page 117 from 698
P1037005 Error while loading Shop System S
Char File Error
P1037006 Shop Code not present in Shop not W Shop code
Shop Catalog List present should loaded
by Load Shop
List

the input file
P1037007 Number of records No Information message only. No action required
for which the shop Shop
is not present in count
Shop List table
P1037008 Number of records Insert Information message only. No action required
inserted into Shop Count
Usability Table
P1037009 Number of records Update Information message only. No action required
updated into Shop Count
Usability Table
P1037010 No Records deleted No Information message only. No action required

from Shop Usability Deletion
Table
P1037011 Number of records Delete Information message only. No action required
deleted from Shop Count
Usability Table

Page 118 from 698
EHDV21038P – Compute median price deviation ( MAD)
Phase:
This program is computing Median Absolute Deviation (MAD) of prices present in Rawda.
This MAD factor will then be used by APV process to validate prices delivered by the panellists.
The datatype used to compute MAD is DT5667 (historical price, before APV process).
MAD is computed by Price Group (price group is a shop characteristic used to build group of
shops congruent in term of price). Price group characteristic to be used are defined as static
parameter PRPRG.
MAD is computed using historical price stored in Rawda (original price loaded in Rawda before
any correction by Price Validation).
- If FPV_3_4_EXCLUDE is ‘YES’ then in the prices are extracted from the RAWDAILYT60 to
store into GT_PROCKEYST29, exclude the prices coming from FPV by applying the filter
DT5904 <> 3 and DT5904 <> 11 and DT5904 <> 20.
- Else let the current behaviour
MAD is computed using Rawda purchase information of submission period + a number of periods
backward defined in static parameter NUM_PER (number of periods in Job submission screen is
not used here).
Computed MAD values are stored in CNANPRICET34 table (same table where is stored
Standard Price)
Logic to compute MAD is the following:
• For each cycle / period / NAN_KEY / price group, following logic is applied :
• median price
• we compute price_deviation = ABS ( price – median_price )
• compute median of price deviation
• if number of observations is less than MINOBS (Static parameter value), median
price and MAD are not computed
• we compute also MAD for price group 0 (all the observations by NAN_KEY )
On top of MAD at NAN_KEY level, a deviation factor at Product Group level is computed using
following logic:
• For each cycle / period / product group / price group,

• compute RT_PRGR = [ median of price deviation / median price ]
for each NAN_KEY inside the Product Group for same price group
• cumulate  RT_PRGR at Product Group/ price group level
• count n as # of NAN_KEY by at Product Group/ price group
• compute CVpc =  RT_PRGR / n
• if number of items is less than MINITM (Static parameter value), CVpc is not
computed

Page 119 from 698
• we compute by default the CVpc for price group 0, (all the observations by Product Group )
Recommendations:
This job should be submitted after all Families are loaded in Rawda and before APV.

Page 120 from 698
Constraints:
Prerequisites:
• Availability of the data in rawda for the submission period

• Price Group information available in ShopUsability table
Table Description
RAWDAILYT60 Purchase rawda
CNANPRICET34 Table containing standard price and MAD by NAN_KEY,cycle, period, price
Group
PROCKEYST29 List of RAWDA processing keys (purchase + Tbskt)
MODULEST31 Definition of Modules, Product Groups, Super Groups extracted from EIMD
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters:
Static Parameters

Type (example)
CHID PRGRP SOUR Shop Characteristic to be used for Price
Group
PARM MINOBS 5 Minimum number of purchase acts available
by NAN_KEY required to allow computation
of MAD factor
PARM MINITM 5 Minimum number of NAN_KEY by Product
Group required to allow computation of CVpc
PARM NUM_PER 2 Number of Rawda periods backward used to
compute MAD
PARM SCOPE 1 A new parameter SCOPE has been added in
order to limit the scope of products that
“Median Price” is computed based on the IBD
scope of products.
PARM FPV_3_4_EXCLUDE ‘Y’ or ‘N’ Default value is ‘NO’

Page 121 from 698
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection No
DTSelection No
File Selection No
Rework Management :
Proc.TYPE CYCLE PERIOD option Tag

Process XXXX YYYY.PPP Normal allowed
Process XXXX YYYY.PPP PC allowed
Rework XXXX YYYY.PPP Normal allowed
Rework XXXX YYYY.PPP PC allowed

code
P0020505 Static Value not present Stat.Val not S Reported Static Add an entry in
present parameter is missing Static
for submission cycle Parameter
screen
P0020564 Error while getting System Error S Check DT
Position and Precision parameters
Info and DT
definition
screen
P0020563 DataCode Info Not DTCode Info S Datacode info not Enter the value
Present absent available for Datacode
(Position,Precision) Info
P0020576 Error while generating System Error S Check DT
DTInfo for the given Tag- parameters
DataCode Info List and DT
definition
screen
MsgLog Array about
Invalid Tags for DTInfo
P1038003 Error in System Error S Open a QLS
Ehfn_Set_Dt_Pointers

Page 122 from 698
Ehfn_ProcessForRework
PC
P1038015 No Price Group Char No S The Price Group Check the
available in T40-Check PriceGroup Char value is not SHOPUSABILI
Table Contents available in TYT40 table
SHOPUSABILITYT4 content and
0 table for reported enter valid
Shop code. Price Group
Char value
P1038011 Error while checking for System Error S Unable to check the Check the
Price Group Price Group Char SHOPUSABILI
value in TYT40 table
SHOPUSABILITYT4 content and
0 table SELECT query
P1038012 Error while checking for System Error S Open a QLS
Missing Shops
P1038017 No Transaction available No S No purchase Could be that
for MAD Computation Transaction transaction available Rawda is
empty for
selected
period, of no
shop with
selected
P1038006 Error while extracting System Error S Open a QLS
Rawda
P1038007 Error while computing System Error S Open a QLS
Median
P1038008 Error while computing System Error S Open a QLS
MAD
P1038010 Error while inserting into System Error S Open a QLS
T34
P0020628 Error while Dropping System Error S open a QLS
Tables
Tables

P1038022 Mad Will be computed Price group List of Price Group for which MAD is computed
for Price group
P1038016 CV not available for CV not List of Product Group/ Price Group for which
PC present CVpc is not computed (# NAN_KEY < MINITM)
P1038020 Total Number of Purchase act Total number of purchase acts used for MAD
purchase acts count computation
P1038021 Total Number of Family Total number of purchase acts used for MAD
Families Count computation

Page 123 from 698
EHDV21039P- Compute standard price
Phase :
This program is computing Standard Price based on prices present in RAWDA.

The datatype used to compute MAD is DT5667 (historical price, before APV process).
This Standard Price factor will then be used by APV process to validate prices delivered by the
panellists. It can be used also to fill missing prices in FPV process.
Standard Price is computed by Price Group (price group is a shop characteristic used to build
group of shops congruent in term of price). Price group characteristic to be used are defined as
static parameter PRPRG.
Standard Price is computed using historical price stored in RAWDA (original price loaded in
RAWDA before any correction by Price Validation).
- If FPV_3_4_EXCLUDE is ‘YES’ then in the prices are extracted from the RAWDAILYT60 to
store into GT_PROCKEYST29, exclude the prices coming from FPV by applying the filter
DT5904 <> 3 and DT5904 <> 11 and DT5904 <> 20.
- Else let the current behaviour
Standard Price is computed using RAWDA purchase information of submission period + a

number of periods backward defined in static parameter NUM_PER (number of periods in Job
submission screen is not used here).
Computed Standard Price values are written in an output file.
Logic to compute Standard Prices is the following:
For each audit / period / NAN / price group, the following formula is applied :
std_price ( t ) = w * median_price( [t – NUM_PER +1 ; t] ) + ( 1-w ) * std_price ( t-1 )
w=n/(n+k)
n is the number of observations by NAN/price group
K is a parameter given to the program by the static parameter table
• In case std_price ( t-1 ) is null, we apply :

std_price ( t ) = median_price( [t – NUM_PER +1 ; t] )
• if the number of observations is less than MINOBS (defined as static parameter), the
standard price is not computed
• we compute also by default the standard price for the price group 0
(all the observations by CNAN )
It will be used in the price validation if the requested NAN/price group std price is missing.
• we compute also standard price at Product Group / price group level, using same logic

Page 124 from 698
• we compute also standard price at Product Group / price group 0, using same logic (this is an
option driven by static parameter PG_PRG0)
Recommendations :
This program can be included in same group as Load Std price job (40P)
This job should be submitted after all Families are loaded in Rawda and before APV.

Page 125 from 698
Constraints:
Prerequisites:
• Availability of the data in rawda for the submission period

• Price Group information available in ShopUsability table
Table Description
CNANPRICET34 Table containing standard price and MAD by NAN_KEY,cycle, period, price
Group
Files:
Input Files:
No input file.
Output Files :
Standard price (File id: FILEO1) :

1 4 CYCLE CHAR(4)
5 7 PERIOD NUMBER
12 4 PRODUCT_GROUP CHAR (product group or 0)
16 16 NAN NUMBER (nan_key or 0)
32 10 PRICE_GROUP CHAR (0 or shop characteristic value)
42 4 DT_CODE Fixed value 5653
46 16 PRICE_VAL NUMBER (XXXXXXXXXX.XXXXX)
Parameters :
Static Parameters

Type
CHID PRGRP SOUR Shop Characteristic to be used for Price
Group
PARM K 5 Factor user in Std price formula
PARM MINOBS 5 Minimum number of purchase act by
NAN_KEY/ Price Group required to allow
computation of Sdt price

Page 126 from 698
PARM NUM_PER 3 Number of Rawda periods backward used to
compute MAD
PARM PG_PRG0 1 Option to compute standard price at Product
Group & Price Group=0
PARM SCOPE 1 A new parameter SCOPE has been added in
order to limit the scope of products that
“Standard Price” is computed based on the
IBD scope of products.
PARM FPV_3_4_E ‘Y’ or ‘N’ Default value is ‘NO’
XCLUDE
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management:

Rework XXXX YYYY.PPP Normal Not allowed

code
screen
in struct

Page 127 from 698
P9900202 No Shops Selected from No Records S The Shops is not Check content
PROCKEYST29 Table Selected selected from of
PROCKEYST29 PROCKEYST2
Table 9 Table and
select valid
Shops
P1039005 Missing Shop Missing S The Shops is not Check content
Shop selected from of
PROCKEYST29 PROCKEYST2
Table 9 Table and
select valid
Shops
P1039001 Production Cycle is not Period not S Production Cycle Lock
locked locked Period is not locked Production
Cycle period
Msglog, check
if files are
present on
FileSystem
with correct
grants
P9900003 File Handling-Write Error File Write S Error is manipulation Open QLS
Error of the file

record in input
file
P1039003 Error while computing System Error S Standard Price is not Compute valid
Standard Price compute Standard Price
P0020589 Error while fetching Std. System Error S Standard price for Select valid
Price for Price Group Price Group is not Standard price
retrieving. for Price Group
P0020582 Invalid Format Invalid S Reported
Format Datatype is not
present in input
file with correct
format
P0020583 Error while formatting the System Error S open a QLS
number
P1039004 Error in Standard Price System Error S The Standard Price Select valid
is not valid Standard Price
P0020628 Error while Dropping System Error S Open QLS
Tables
Tables

Page 128 from 698

P1039008 Total number of Record Information message only. No action required
records written in to count
the file
Shops Shop Count
P1039010 Number of Shops not Missing Information message only. No action required
present in Shop Shop Cnt
Usability table
P1039011 Total number of Purchase act Information message only. No action required
Purchase acts count
P1039012 Total number of Distinct PC Information message only. No action required
Distinct PCs count
P1039013 Total number of Distinct Information message only. No action required
Distinct Families Family cnt

Page 129 from 698
EHDV21040P- Load standard price
Phase :
This program is used to load standard price into CNANPRICET34 table from external file.
If the period in Input File is defined as 0 then consider that it is not mismatching with
the job submission period otherwise if Input File period is defined different from 0 or
the job submission period the job does not accept it and terminates with error.
Input standard price file can be :

• Created by Compute Standard Price job (39P)
• Created by FPV job (15P) and amended manually by users
• Created from scratch manually by users
Only standard price related to price groups instructed in static parameters are loaded.
Recommendations:
This program can be used along with Compute Standard Price program
Constrains:
Prerequisites:
Availability of the file containing the standard price values
Table Description
CNANPRICET34 Table containing standard price and MAD by NAN_KEY, cycle, period, price
Group
Files:
Input Files:
Standard price (File id: FILEI1) :
1 4 CYCLE CHAR(4)
5 7 PERIOD NUMBER
12 4 PRODUCT_GROUP CHAR (product group or 0)
16 16 NAN NUMBER (nan_key or 0)

Page 130 from 698
32 10 PRICE_GROUP CHAR (0 or shop characteristic value)
42 4 DT_CODE Fixed value 5653
46 16 PRICE_VAL NUMBER (XXXXXXXXXX.XXXXX)
Output Files:
No output file
Parameters :
Static Parameters

Type Value
CHID PG01 SOUR Shop Characteristic to be used for Price
Group
PG01 CHVL01 0 Value of price groups allowed for load
(optional)
PG01 CHVL02 0000000001 (optional)
Rem : if no CHVLXX are available, all price group values present in flat file are loaded
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection No
DTSelection No
File Selection Yes
Rework Management:

Process XXXX YYYY.PPP PC allowed
Rework XXXX YYYY.PPP PC allowed

Page 131 from 698
code
P1040001 Missing Price Group Missing S Price Group is Correct Price
Price Group empty Group in input
file
P1040007 Price Group Code Wrong Pr.Gr S Price Group is Correct Price
Wrong. 0000000000 Group in input
file
P9900034 Invalid DataCode Invalid S DataCode is invalid Correct
DataCode (0 or blank) DataCode in
input file
P9900017 Missing DT Value Missing DT S The DT value is not Correct DT
Value available value in input
file
P9900025 Item is Blank / Zero Item is Blank S Both NAN_KEY and Correct PC or
/ Zero PC values are empty NAN_KEY
values in input
file
P9900095 Mismatch in Cycle Mismatch in S The Cycle is not Check Cycle in
Cycle matching input file
P9900096 Mismatch in Period Mismatch in S The Period value is Check Period
Period not matching in input file
Msglog, check
if files are
present on
FileSystem
with correct
grants
record in input
file
Msglog, check
if files are
present on
FileSystem
with correct
grants
P0020501 Array List Blank Array List S Datacode in the Select valid
Blank selected datacode datacode list
list is empty
P0020552 Error in Sorting Char System Error S open a QLS
Arrays
P0020554 Error while checking for System Error S open a QLS
P1040003 Error in Ehfn_CheckFile System Error S open a QLS
P1040004 Error in System Error S open a QLS

Page 132 from 698
PC
P0020112 PC-Item not found PC-Item not S NAN_KEYS not Enter valid PC-
found fount in Item
NAN_ITEMST73
P0020118 Error in checking for the System Error S open a QLS
presence of PC – Item in
CNAN table
P0020578 Error while generating System Error S Unable to get the open a QLS
DataCode List for the precision for Data
given Tag List Code in Input file
Ehfn_LoadIntoTable

Page 133 from 698

P9900010 Missing PC Missing PC Information message only. No action required
the input file
P1040008 Total Number of Distinct PC Information message only. No action required
distinct PCs Count
P9900018 PC Code is 0 PC Code is Information message only. No action required
0
P9900011 Skipped PC Skipped PC Information message only. No action required
P1040009 Total Number of Update Information message only. No action required
records updated in Count
CNAN Price table
P1040010 Total Number of Insert Count Information message only. No action required
records inserted in to
CNAN Price table
P1040011 Total Number of Selected Information message only. No action required
records for the Rec Cnt
selected PC
P1040012 Total Number of PC – Missing Information message only. No action required
Items missing in CNAN Cnt
CNAN table
P0020608 Static Value Static Value Information message only. No action required
P1044021 Period 0 record(s) will Instruction Information message only. No action required
be replaced with
submission period

Page 134 from 698
EHDV21041P – Rawda extraction for rawda correction
Phase :
This program is an utility to extract information from the rawda to an output file.
The file extracted can be used for any purpose :

1. special analysis
2. data control
3. preparation of input file for Rawda correction
4. other
The job can run using different options :

1. extract one or more periods
2. extract one or more datatype
3. extract data for some selected PC only or for all PC
4. extract data by F# selection
Datatype selection :
Datatypes to be extracted are defined when submitting the job in DT Selection screen.
It’s mandatory to have at least one datatype selected.
This program can use only datatypes present in the output tables. If other datatypes are selected
during the job submission, these datatypes will be ignored.
This program can only extract datatype from following final tables :
1. RAWDAILYT60
2. TBSKT13
3. PROCKEYST29
Note:For python version, Hive data refresh is done at the beginning of this process for the
submission audit, period (back-periods included) for the following tables:
1. RAWDAILYT60
2. TBSKT13
3. PROCKEYST29
Logic of extraction :
Filetering by shop/hhold/ product
On top of extraction parameters described above, extraction of a given list of HHOLD/SHOP/NAN

or BARCODE can be applied.
Those new input files are mandatory to submit the job. For each of those 3 files:
- if input file is empty or does not have correct header the filtering is not applied
- if header is correct and at list of record is present only instructed HHOLD/SHOP/NAN or
BARCODE will be extracted.

Page 135 from 698
DT_ZERO=Y DT_ZERO=N DT_ZERO=N DT_ZERO=N
IBD = None IBD = None IBD = IBD ID IBD = IBD ID
One or more We extract all We extract all records in We extract only the We extract only the nankey
T60 DT records in T60 T60 with DT value>0 nankey in T60 for the given IBD
in T60 for the given with DT value>0
IBD
One or more We extract all We extract all records in We extract only the We extract only the nankey
T29 DT records in T29 T29 with DT value>0 nankey in T29 for the given IBD
in T29 for the given with DT value>0
IBD
One or more We extract all We extract all records in We extract all records We extract all records in
T13 DT records in T13 T13 in T13 T13
with DT value>0 with DT value>0
One or more We extract: We extract We extract: We extract
T13 DT all records in T13 all records in T13 with DT all records in T13 all records in T13 with DT
+ + value>0 + value>0
One or more all records from T60 + only the nankey +
all records from T60 with in T60 for the given only the nankey
T60 DT DT value>0 IBD in T60 for the given IBD
with DT value>0
T60 DT all records in T29 all records in T29 with DT only the nankey only the nankey
+ + value>0 in T29 for the given in T60 for the given IBD
One or more all records from T60 + IBD with DT value>0
all records from T60 with + +
T29 DT DT value>0 only the nankey only the nankey
in T60 for the given in T60 for the given IBD
IBD with DT value>0
all records from T13 with + +
T13 DT DT value>0 all records from T13 all records from T13 with
DT value>0
One or more We extract: We extract: We extract: We extract:
+ all records from T13 with + +
T13 DT all records from T60 DT value>0 all records from T13 all records from T13 with
+ + + DT value>0
One or more all records from T60 with only the nankey +
T60 DT DT value>0 in T60 for the given only the nankey
IBD in T60 for the given IBD
with DT value>0

Page 136 from 698
Here are 3 examples of extractions:
Case 1 :
Shop file is containing:
SHOP
0000000016
Hhold file is containing:

HHOLD
0000003173
0000003472
Product file is empty
DT5650 is instructed
➔All DT5650 datatypes values for selected period range for shop=0000000016,
HHOLD=0000003173 or =0000003472, for all products will be extracted from RAWDAILYT60
Case 2:
SHOP
0000000016

HHOLD
0000003173
0000003472
Product file is containing:

BCD
6408430004324
DT5650 is instructed
➔ All DT5650 datatypes values for selected period range for shop=0000000016,
HHOLD=0000003173 or =0000003472, barcode=6408430004324 will be extracted
Case 3:
SHOP
0000000016

HHOLD
0000003173
0000003472
Product file is containing:

NAN
402254
839947
DT5650 and DT5941 are instructed

Page 137 from 698
➔All RAWDAILYT60 datatypes values for DT5650 for selected period range for
shop=0000000016,
HHOLD=0000003173 or =0000003472, NAN_KEY= 402254 or =839947 will be extracted
➔All TBSKT13 datatypes values for DT5941 for selected period range for shop=0000000016,
HHOLD =0000003173 and 0000003472 will be extracted
Recommendations :
Constraints :
Prerequisites :
Rawda should be loaded (Rawda Loader job or backdata loader).
Table Description
MODULEST31 List of Modules extracted from EIMDB
Files:
Input Files:
• Input file 1 : HHOLD selection file (File id: FILEI1)
Header (first record)

1 5 FILE Fixed value: ‘HHOLD’
IDENTIFIER
Main

1 10 HHOLD Panelist code to be extracted
• Input file 2 : SHOP selection file (File id: FILEI2)

1 4 FILE Fixed value : ‘SHOP’
IDENTIFIER
Main

1 10 SHOP Shop code to be extracted

Page 138 from 698
• Input file 3 : PRODUCT selection file (File id: FILEI3)

1 3 FILE Fixed value : ‘NAN ‘ or ‘BCD ‘ with 2 blanks char at the
IDENTIFIER end
NAN if list of Nan_Key is provided
BCD if a list of Barcode is provided
Main

1 35 PRODUCT Nan_Key of Barcode code to be extracted (depending on
CODE header)
Output Files :
Rawda extraction by DT (File id: FILEO1) :

1 4 AUDIT Xxxx
5 4 YEAR YYYY
9 2 MONTH MM
11 2 DAY DD
13 4 TIME HHMM
17 10 HHNUMBER 00||xxxxxxxx
27 10 SHOP
37 35 BARCODE LEFT ALIGNED
72 16 NAN_KEY
88 4 DATATYPE Xxxx
CODE
92 16 DATATYPE xxxxxxxxxx.xxxxx
VALUE
Parameters :
Static Parameters

Type Value
PARM DTZERO N Option to extract datatype values zero
If N, datatype values=0 are not extracted
If Y, datatype values=0 are extrated
PARM IBD xx Where xx can be IBD ID
If IBD is None or Unset
all the nankeys are extracted
If IBD is Not None or set
only the nankeys for the given IBD id is
extracted.
IBD filtration is applicable only for T60 and T29

DT’s

Page 139 from 698
DTDT DTxxxx yyyy Where xx can be any number and yyyy is the
datatype number.
Example: DTDT (type)

DT5650 (type value)
5650 (name)
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection Yes
DTSelection No
File Selection Yes
Rework Management :


code
in struct
P9900221 No Records Selected No Records W Rawda is empty for Check if rawda
from PROCKEYST29 Selected selected parameters has been
loaded for
selected period
P9900237 No Records Selected No Records S Inconsistency open a QLS
from PROCKEYST29 Selected between
PROCKEYST29 and
RAWDAILYT60
P1041004 Error in Extract Data System Error S Generic error while open a QLS
extracting data
P1041002 Error in Check DataCodes System Error S Generic error while open a QLS
checking datatypes

Page 140 from 698
P1041003 Error in Count Destination System Error S Generic error while open a QLS
checking datatypes
destination
P0020582 Invalid Format Invalid S open a QLS
Format
P0020583 Error while formatting the System Error S open a QLS
number
Tables
Tables
P0020505 No DTs have been System Error S No datatypes have Define DTDT
defined as static been defined as static
parameters static parameters parameter(s)

P1041008 Total number of Selected Information message only. No action required
distinct Data Codes DTCode Cnt
selected for Extraction
P1041007 Total number of Distinct PC Information message only. No action required
distinct PCs Extracted Count
P1041009 Total number of Record Cnt Information message only. No action required
records Extracted
P1041010 Total number of Rawda Information message only. No action required
Rawda records Record Cnt
Extracted
P1041011 Total number of TBSK TBSK Information message only. No action required
records Extracted Record Cnt
P1041013 Total number of PR- PRKeys Information message only. No action required
Keys records Record Cnt
Extracted
P1041001 Datatype will not be DT not Datatypes not belonging to T60/T13/T29 are
extracted as it is not a present ignored.
part of O/P Table

Page 141 from 698
EHDV21042P – Rawda Correction
Phase :
After the rawda loader, the user can run any customized quality control on the data. After the
inspection, a data correction may be requested.
This program is applying data correction to the data in the final tables. Performing some controls
in order to guarantee the final consistency of the data.
Rawda Correction is driven by an external flat file. Input file layout is same as Rawda Extraction
(one record per Datatype)
The program is working at:

1. total basket level
2. purchase transaction level
The program allows the following function

1. delete single purchase transaction (by pc-item)
2. insert a new tbsk or purchase transaction
3. update a tbsk or purchase datatype
Rawda correction can be used to:

- Insert new records in TotalBasket/Purchase Rawda
- Delete some records from TotalBasket/Purchase Rawda
- Update some Datatypes values in TotalBasket/ Purchase Rawda
Following table is summarising possible options:
NAN_ DATATYPE DATATYPE

BASKET KEY BARCODE KEY CODE VALUE ACTION
AUD/ DATE/TIME/HH/SH
0 0 0000 ---- Delete TBSK + PRK
AUD/ DATE/TIME/HH/SH
0 0 xxxx Xxxx Insert / upd. TBSK
AUD/ DATE/TIME/HH/SH Xxxxxx
xxxxxx…. …. 0000 ---- Delete RAWDA+ PRK
AUD/ DATE/TIME/HH/SH xxxxxx
xxxxxx… … xxxx Xxxx Insert / upd. RW + PRK
➢ we apply controls on the purchase transaction data
We cannot delete a whole basket by a single instruction (basket key)
Recommendations :

Page 142 from 698
Constraints :
Checks performed by Rawda Correction:

 Cycle in input file must be consistent with submission one
 Any purchase date in input file should be inside submission date range
 Any Panelist code present in input file must be present in Family Char table
 Any Shop code present in input file must be present in Shop Char table
 Datatype code must be defined
 Any Barcode/Nan_key must refer to existing Xcoding link in case of Insert
➔ If any of those cases is faced, an error will be raised
On top of this, following checks are performs on Qty + Price datatypes :
Input file content INSERT in T60 UPDATE in T60

Two records : one with DT5650>0 and Check OK, no Warning
one with DT5675>0 message Check OK, no Warning message
Two records : one with DT5650=0 and
one with DT5675=0 Check not OK Warning message Check not OK Warning message
Two records : one with DT5650=0 and
one with DT5675>0 Check not OK Warning message Check not OK Warning message
Two records : one with DT5650>0 and
one with DT5675=0 Check not OK Warning message Check not OK Warning message
One record only , DT5650=0 Check not OK Warning message Check not OK Warning message
One record only , DT5675=0 Check not OK Warning message Check not OK Warning message
One record only , DT5650>0 Check not OK Warning message Check OK, no Warning message
One record only , DT5675>0 Check not OK Warning message Check OK, no Warning message
One record only other than DT5675
and DT5650 (eg DT5662) Check not OK Warning message Check OK, no Warning message
Prerequisites :
Input file should be available with correct format.
Table Description
CROSSCODING_INSTRU Crosscoding links information extracted from EIMDB
CTIONST49

Page 143 from 698
CROSS_CODE_GROUPS List of Xcode Group ID 144lignme to be used for Xcoding
T26
Files:
Input Files:
Rawda correction (File id: FILEI1):

1 4 AUDIT Xxxx
5 4 YEAR YYYY
9 2 MONTH MM
11 2 DAY DD
13 4 TIME HHMM
27 10 SHOP
72 16 NAN_KEY
88 4 DATATYPE Xxxx
CODE
VALUE
Output Files:
No output file.
Parameters :
Static Parameters

Type
PARM CHK_BSKT Y Option to check if TBSKT key is existing
before inserting new purchase act in T60
- if the parameter is set to Y : purchase act is
not inserted in T60 if basket key is not
already present in T13
- if the parameter is set to N or is not present
: the check is not applied, purchase acts are
inserted in T60 even if not basket key is not
existing in T13
PARM HH_CHECK ON/OFF Optional : To set the check mode for family
code
PARM SHOP_CHE ON/OFF Optional : To set the check mode for shop
CK code
Dynamic Parameters

Name (Yes/No)

Page 144 from 698
# Periods Yes
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection No
DTSelection No
File Selection Yes
Rework Management :

Process XXXX YYYY.PPP PC Allowed
Rework XXXX YYYY.PPP PC Not allowed, run “PC” option x process

code
Lock. RELEASE
Lock
P9900234 Duplicate record present Duplicate S Check for
in file: record pre duplicated
records in input
file
P9900072 Audit Code in the file != File S Check audit
Audit Code linked with Aud!=Cycle code in input
Cycle Aud file
P9900084 Hhold is zero Hhold is zero S Hhold should not be Give positive
zero values for
Hhold
P0020526 Data Code Info not DT Info not S Information about Check if DT in
present for the Data Code in T07 datacode not input file are
present in Datacode present in DT
Definition
screen
P1042004 Family Code is not Family S Family data not Load Familyt43
consistent in I/P File missing found in Familyt43 table with
table family data
P9900101 Shop is Zero Shop is Zero S Shop code is empty Enter a valid
shop code
P1042005 Shop Code is not Shop S Shop code is Load ShopT44
consistent in I/P File Missing missing table with shop
code

Page 145 from 698
P1042029 Error in processing pc, System Error S Open QLS
item when checking for
record validity
P9900229 Purchase Date outside Purchase S Check
the date range of the Date outside consistency
selected period range between date
in input file and
date range
P9900230 Invalid NAN Invalid NAN S Check NAN in
input file
P9900232 Input file format not Input file S Format of input file is Check input file
correct format no not correct layout
P0020625 Error while getting the System Error S Open QLS
period for date
P9900231 Not Valid NAN for Not Valid S Check
Barcode NAN for Ba NAN_KEY
barcode link
validity in
EIMDB
P9900233 Data not consistent , price Data not W Record not loaded Check price +
or units = 0 consistent Qty
P9900266 Purchase act not loaded Purchase W Record not loaded Insert first
as Tbskt is not existing not loaded TBSKT record
or disable
static param
CHK_BSKT
P0420001 Unknown Job ID (Receipt) Unknown S Receipt ID missing Check the
Job ID in input file
TDV_RECEIPT_JO receipt ID
BID

P1048002 Records Deleted Records Information message only. No action required
Deleted
S0000004 Record added Record Information message only. No action required
successfully added
success
S0000006 Record updated Record Information message only. No action required
successfully updated
succe

Page 146 from 698
EHDV21043P- Data file extraction for special analysis
Phase :
We have developed a software utility to extract the information already validated into a sequential
file. This program extracts the data from the output tables, according to the user selection, and
generates a flat file to be used for any kind of application.
The extraction is driven by a set of instructions. Only the data present in the output tables can be
extracted.
Extraction process is modied such that only the validated records are extracted I,e the locked
flag(DT5090=1)
No data-type selection is required in the job submission system as we take the filedefinition and
its data-types from the table T20 (DT parameter screen) by the audit code.
DT parameter screen can contain :
one layout for purchase records (dt of RawdailyT60 and/or ProckeysT29)
one layout for total basket records (dt of TbsktT13 and/or ProckeysT29)
The date range of this program is determined by the period code and the number of periods
selected by the user
Selection by Product Group is available :

In case no PG is selected all keys for purchase acts + total baskets are extracted
In case some PG are selected all keys for purchase acts linked to selected PG are
extracted + total baskets are extracted if required by static parameter BSKT_PG.
The output file, at the end of the job, is sorted by the key of the file
Recommendations :
Constraints :
Prerequisites :
➢ To be run after rawda loader, at the end of the validation phase.
➢ Parameter table by audit available ( DT parameter screen where the file layout is defined
)

Page 147 from 698
Table Description

Page 148 from 698
Files:
Input Files:

IDENTIFIER
Main


IDENTIFIER
Main


IDENTIFIER end
Main

CODE header)
Output Files :
Rawda extraction special analyses (File id: FILEIO) :

1 4 AUDIT Xxxx
5 4 YEAR YYYY
9 2 MONTH MM
11 2 DAY DD
13 4 TIME HHMM

Page 149 from 698
27 10 SHOP
37 4 SCAN PC PG x Hscanner
pc code x Hpanel
41 35 BARCODE Barcode x Hscaner
Item x Hpanel (12)
76 10 LAPTOP_CODE Nan_Key x Hscaner
86 DT # 1 Refer to DatacodeT20 (DT Parameters screen)
DT # 2 Refer to DatacodeT20 (DT Parameters screen)
DT # 3 Refer to DatacodeT20 (DT Parameters screen)
Parameters :
Static Parameters

Type Value
PARM BSKT_PG N If value is ‘N’, basket records are not
extracted when extraction by PG is required
If value is ‘Y’, basket records are extracted

when extraction by PG is required
PARM FILTER_D YES If value ‘YES’,logically deleted records are not
T5126 extracted.The default value for this is ‘YES’
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection Yes
DTSelection No
File Selection Yes
Rework Management :

Process XXXX YYYY.PPP F# Allowed
Rework XXXX YYYY.PPP F# Not allowed, run “F #” option x process

Page 150 from 698
Message Description Tag Typ Explanation Action

code e
S00C0050 Column not in structure Column not S Open QLS
in struct
P1043001 No records present for No records W Check if Rawda
Extraction in the final present is loaded for
tables submission
period
P9900001 File Handling-Invalid Wrong file S Error is manipulation Open QLS
P9900237 No Records Selected No Records S Inconsistency Open QLS
from PROCKEYST29 Selected between
PROCKEYST29 and
RAWDAILYT60
P1043002 Value Error occurred in System Error S Open QLS
function
Ehfn_extractRawda
P1043003 Unhandled Exception in System Error S Open QLS
function
Ehfn_extractRawda
P1043005 Unhandled Exception in System Error S Open QLS
function getFormat Info
record in input
file
P9900003 File Handling-Write Error File Write S Error is manipulation Open QLS
Error of the file

Tables
Tables

records extracted into Count
the o/p file
P1043008 Total Number of Distinct Information message only. No action required
distinct families Family Cnt
extracted into the o/p
file

Page 151 from 698
EHDV21048P- DB cleaning
Phase :
PHASE 7 – Online cleaning
This program cleans the data from DB based on the user static parameter selection or via the
online cleaning
The process submitted via job submission or production flow will expect the static parmater
TABLE_NAME to be filled mandatory. Values can be RAWDA , TOTAL BASKET, FAMILY LOG,
BEVERAGES OR MODULE PRICES.
Based on its value the required cleaning of tables are done.
If the job is submitted via the online cleaning, the below tables can be cleaned
Process steps:
If table name read from dynamic parameter is null

o consider to check Static parameter value
o consider period range using submission period and NUM_PER
o inform user on it thru message log
o Based on Static parameter value the required cleaning of tables are done.
Parameters:

Page 152 from 698
Static Parameters
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection Yes
DTSelection No
File Selection Yes
Cleaning Matrix:

Page 153 from 698
EHDV21050P- Load Supplier Prices
Phase:
This program loads the prices provided from retailer (for example “Picard” in France) into “price
by supplier” table : SUPPLIER_PRICEST98.
Those prices are fixed prices (same for all shops) for a given periods.
The prices are loaded at barcode level (not NAN_KEY).
Structure of the “price by suppliers” table is :
Source Source of price

Price group Retailer collabor ID
Barcode External barcode
Date from Price validity from
Data-type code 5651
Price Price value
Date to Price validity to
Source is defined as static parameter

Price group defined as static parameter CHVL01
In case of submission rework, all existing price for submission source/chain are deleted before
load of data.
Any price with date_from lower than submission date – number of days defined as static
parameter DAYS are automatically deleted
Job is also providing functionality to copy in supplier price table prices instructed in €imdb as
characteristics (these are prices collected from manufacturer (by features…) )
done at the end of this process in terminateJob for these three tables:
• Prockeyst29
• Rawdailyt60
• Tbskt13
Recommendations:
Constraints:
Price in input file should not be 0

EAN in input file should not be 0
Prerequisites:
• Availability of the file containing the price values.

Page 154 from 698
Table Description
SUPPLIER_PRICEST98 Table containing catalogue prices provided by retailer
CROSSCODING_INSTRU Xcoding instruction table
CTIONST49
Files:
Input Files:
Supplier price (File id: FILEI1) :
Input file is not fixed layout file, but file with ; delimiter
YEARMONTH Converted to date_from, date_to

PRODUCTCODE Not used
PRODUCTDESCR Not used
CATEGORY Not used
PRICEEUR Price
PRICEFRF Not used
PROMOTION Not used
WEIGHT Not used
EANCODE Barcode
BRAND Not used
DESCRIPTION0 Not used
PROMOTIONDESCR Not used
TARIFDESCR Not used
Output Files:
No output file
Parameters:
Static Parameters

Type
CHID IDCH01 ENS Identify the characteristic ID for retailer chain
PARM SOURCE 9933 Identify the Source ( country or audit )
CHVL CHVL01 0000000081 Identify the characteristic value for retailer chain
PARM DAYS 365 Identify the number of days to keep in the table for
history
PARM CHAR_1 28338 Identify the characteristic ID 1 for the PRICE in €imdb
PARM XCODE_G FRAA Identify Xcode group characteristic ID 1 for the PRICE
ROUP1 in €imdb

Page 155 from 698
Remark : Price is stored in characteristic value description
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection Yes
Rework Management:


code
P0020505 Static Value not present Stat.Val not S No transactions Do the
present available for the necessary
selected timestamp transaction for
selected
timestamp
P9900176 Date Range – Overlap Date Range S
– Overlap
P1042020 Error while Checking File System Error S
P9900175 Missing Value Missing W
Value
P0020111 Same barcode for same Duplicate S There are more Check with
XcodeGroupId, date key insert than one Xcode IMDB team
range has been identified
instructions having
overlap in validity
dates for more than
1 nankey and same
barcode.

Page 156 from 698
P9900110 Date Range Date Range Information message only. No action required
P9900142 Number of records Number of Information message only. No action required
read records re
inserted records in
skipped records sk

Page 157 from 698
EHDV21051P- Copy Supplier Price
Phase:
This is program copies the prices of retailer “Picard” from “price by supplier” table into Table ALL.
So, we copy the prices by using price group, only for price values = 0
The general assumptions are

• We process all the purchase transactions by the cycle / period concept
• The Picard price file is delivered once a month ( calendar )
• Table of the prices from Picard will contain historical data for 4 months
Model
Shop selection
Selection of shops to be included in this process using price_group char and char value
instructed as static parameter
Copy of price on purchase acts
Selection of all price=0 in ALLT11 for selected shops
Update of price=0 with price selected from SUPPLIER_PRICEST98 for selected source,
price_group for T98.dt_datefrom<=purchase date < T98.dt_dateto;
If Price is not found for the barcode within the period date range, then search again
without date range filtering and update ALLT11 with most recent price.
In case no supplier price is found for any date, following logic is applied based on static
parameter MODL :
If MODL=DELETE, purchase act is deleted from ALLT11
If MODL=DEFAULT, a default price is copied, default value is defined by static
parameter DETAULT
If MODL=NULL, no update
Rem : All purchase acts for which price has been imputed, are flagged with DT5904= 9 (price
correction flag).
Remark : In REWORK mode all purchase acts are updated in ALLT11 with price in
SUPPLIER_PRICEST98 (even if price>0 in ALLT11)
Recommendations:
Constraints:
• The prices delivered by Picard are copied only
Prerequisites:

Page 158 from 698
• Availability of the file containing the price values
• File to be delivered by retailer
Table Description
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters:
Static Parameters

Type Value
PARM SOURCE 9933 Identify the characteristic VAL for the source
CHID PRGRP ENS Identify the characteristic ID for the PRICE
GROUP to access the shop usability table
CHVL SOUR 0000000081 Identify the characteristic VAL for the PRICE
GROUP value
PARM DEFAULT 3.3 Identify the default price value is mode
DEFAULT selected
MODL DEFAULT null MODL : represents the action to be taken
when the price is not found
• DELETE
• DEFAULT ( ac_name contains the
default price to be inserted )
• NULL
Dynamic Parameters

Name (Yes/No)
# Periods Yes

Page 159 from 698
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 160 from 698

code
P0020504 Static Value Blank Static Value S Missing static
Blank parameter value
P1021015 Shop not Found in T40 Shop not in S
T40

succe

Page 161 from 698
EHDV21052P – Pharma Price Loader (Italy)
Phase:
The objective of this procedure is to load Pharma prices from Pharma price flat file to table
SUPPLIER_PRICEST98
Structure of this table is following:

Field name Description
AC_SOURCE Identifier for the source of price (eg 01 for Pharma prices)
AC_PRICE_GROUP Price Group
AC_BARCODE Barcode code
DT_DATEFROM Price validity date from
DT_DATETO Price validity date to
AC_DATACODE 5651 for supplier price
NC_PRICE Price value
In the PriceBySupplierT98 will be kept the historical average price values for each EAN/APO.
To manage historical values a ‘validity period’ (DateFrom/DateTo) will be define following these
rules:
DateFrom = InsertDate ( as defined by static parameter)
DateTo = 99991231 for new EANCode

InsertDate-1 for already exist EANCode
The average price values are computed based on the Pharma prices input file and calculated by
EAN/APO code.
Input file can contain 2 distinct external codes: minsan and ean8_13, those cases are managed
as follows:
If pr_minsan null and pr_ean8_13 null

No insert
If pr_minsan not null and pr_ean8_13 null

Insert one record : pr_minsan in AC_BARCODE + Price
If pr_minsan null and pr_ean8_13 not null

Insert one record : pr_ean8_13 in AC_BARCODE + Price
If pr_minsan not null and pr_ean8_13 not null

Insert 2 records :
pr_minsan in AC_BARCODE + Price
pr_ean8_13 in AC_BARCODE + Price
On top of this, if price=0, no insert will be done
Logic of process mode
➢ Program is checking if following condition is true

Page 162 from 698
o InsertDate > max(DateFrom) in the PriceBySupplierT98 table fro submission Source
In case the check is not OK, job is ending with code 16
➢ For each barcode in input file, average price is computed
➢ For all barcode present in input file, program is checking if there is a record already present in
T98 and if exists selects the latest one
• In case barcode is not present in T98: a new record will be inserted. The
‘validity period’ for this new record will be set in this way:
DateFrom = InsertDate and DateTo = 99991231
• In case barcode is already present in T98:

➔ Existing record will be updated changing the ‘validity period’ in this way:
DateTo = InsertDate – 1
➔ A new record will be inserted with DateFrom = InsertDate and DateTo = 99991231
Rework mode:
➢ Program is checking if following condition is true

o InsertDate = max(DateFrom) in the PriceBySupplierT98 table fro submission Source
In case the check is not OK, job is ending with code 16
➢ For each barcode in input file, average price is computed
➢ For all barcode present in input file, program is checking if there is a record already present in
T98 and if exists selects the latest one
• In case barcode is not present in T98: a new record will be inserted. The
‘validity period’ for this new record will be set in this way:
DateFrom = InsertDate and DateTo = 99991231
• In case barcode is already present in T98: the median price will be updated
with the newest value.
➢ For all barcode present in T98 input file, but doesn’t exist in the pharma input file
The record with DateFrom= InsertDate will be deleted and the record with DateTo=
InsertDate-1 will be updated setting the DateTo field equals at 99993112.
Constraints:
Prerequisites:
Pharma input file should be available
1 Pharma prices from external
Table Description
SUPPLIER_PRICEST98 Table containing fixed prices provided by supplier and Pharma prices

Page 163 from 698
Files :
Input Files :
Pharma Price file (FILEI1)

The pharma prices will be taken from the DNIELSEN.DBF input file. (See Error!
Reference source not found. ). This file has got many fields but we will use only these 3
(three):

161 09 pr_minsan Codice MinSan
250 13 pr_ean8_13 Codice EAN8 o EAN13
353 08 pr_preuro C Prezzo in euro
Output Files:
ReportFilename (FILEO1)
1 20 EAN Code
21 7 Old Pharma xxxx.xx
Price
28 7 New Pharma xxxx.xx
Price
This file will be filled only in case of REWORK mode and will contain:
-All the newest pharma price values that dit not already exist in PriceBySupplierT98 table
-All the oldest pharma price values that exist in PriceBySupplierT98 table but don’t exist
into the input file
DiscardFilename (FILEO2)
This file will contains all records in input file with pr_minsan = 0 or pr_ean8_13 or pr_preuro C
BadtFilename (FILEO3)
1 160 NOTUSED1
170 80 NOTUSED2
263 90 NOTUSED3
This file will contains all records with un-correct format that can not be loaded.
Parameters:

Page 164 from 698
Static Parameters

Type Value
CHVL CHVL01 PR_GR Shop char for price group
PARM INSDT 20070628 Insert date,
PARM SOURCE 01 Source of prices inserted in T98
DTDT IDDT01 5651 DataType Code of PHARMA Prices

Page 165 from 698
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


code
P9900146 Invalid InsertDate Static Invalid S Check
Paramete InsertDate InsertDate
static
parameter
P0020504 Static Value Blank Static Value One static is
Blank missing

S0000004 Number of Records Insreted in Information message only. No action required
Insreted in T98 T98

Page 166 from 698
EHDV21053P- Copy NRSP Prices
Phase:
This program copies the RMS prices stored in table SHOP_PRICEST99 into table P359_1T12.
The general assumptions of this job are

• The NRSP prices delivered once a week
• Table of the NRSP prices will contain historical data for 4 months
The table structure of the input SHOP_PRICEST99 is:
Source Identifier for source of price (eg 01 for NRSP prices)

Shop code CPS shop code
Nan_Key Product identifier shared by NCPS/NRSP
P_Date_from Price validity date from
Data-type code 5651 for RMS price
Price Price value
P_Date_to Price validity date from
Shop code local RMS shop code
Shop code local: filled only if shop reclassification has been required in Load NRSP price job.
Imputation Model :
Select all keys to be included in this job:
We select in P359_1T12 only shop codes for which price should be imputed from RMS.
Selection is done in SHOPUSABILITYT40 using static parameter char + value.
We select all the keys in table P359 = shop || NAN || purchase date for shops to be
imputed form RMS for select date range
Retrieve prices from RMS:
Check all prices available in SHOP_PRICEST99 table with

P359.shop= SHOP_PRICEST99.shop
Nan_key = SHOP_PRICEST99.NAN_KEY
P_date_from <= purchasing date <= P_date_to
If no price is found and if static parameter DAYS>0 we check if a price is available in past
periods :
Check all prices available in SHOP_PRICEST99 table with
P359.shop= SHOP_PRICEST99.shop
Nan_key = SHOP_PRICEST99.NAN_KEY
P_date_from-DAYS <= purchasing date <= P_date_to
If more than one price value, we take the most recent one

Page 167 from 698
Apply correction on P359 :
If price in P359=0
If RMS price found = we impute RMS price
If RMS price not found,
if static parameter MODL=’DELETE’, we delete purchase act in P359, if
MODL=’NULL’ do nothing
If price in P359>0
If RMS price found, if static parameter MINMAX=’YES’, we check if price in inside
min/max range of price for this nan computed for all RMS shops, if not replace existing
price by RMS price
Rem : All purchase acts for which price has been imputed, are flagged with DT5904= 4 (price
correction flag).
Recommendations:
It is recommended to submit this job before other RMS price imputation jobs (copy cluster price,
FPV)
Constraints:
• The prices delivered by NRSP are copied only
• The NRSP’s prices must not be changed by any validation routine
Prerequisites:
• Availability of the file containing the price values from RMS

• File loaded in DV
• Data present in P359
Table Description
SHOP_PRICEST99 Table containing prices by period, shop, NAN_KEY
Files:
Input Files:
No input file.
Output Files:
No output file

Page 168 from 698
Parameters:
Static Parameters

Type
CHID CH01 NRSP Identify the characteristic ID for the PRICE
GROUP to access the shop usability table
CH01 CHVLxx 01 Identify the characteristic VAL for the PRICE
GROUP value
PARM SOURCE 01 Identify the characteristic values for the source
MODL DELETE MODL : represents the action to be taken when
the price is not found
• DELETE
• NULL
PARM DAYS 28 Identify the number of days to search back in the
history for a price same product
PARM MINMAX NO Option to validate prices already present in P359
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


.

code
P1021015 Shop not Found in T40 Shop not in S Shop codes are not
T40 present in T40

Page 169 from 698
Page 170 from 698

P0020611 # of NRSP Shops in NRSP Information message only. No action required
SHOPUSABILITYT40 Counter
P0020612 # of Recs in UPD Information message only. No action required
P3591_T12 Updated Counter
P0020613 # of Recs in DEL Counter Information message only. No action required
P3591_T12 Deleted

Page 171 from 698
EHDV21054P- Compute & Load Median Price NRSP
Phase:
This program is used to compute median price on prices loaded from RMS. Computation
is done using RMS prices loaded in SHOPPRICET99 table by job Load price NRSP (74P).
Computed standard prices are stored in CNANPRICET34 table. They will be used by FPV
process (15P) to impute missing prices.
Model for median price computation:
- We process all the NRSP prices of a week in a single run. It is a weekly process (or cycle /
period / # periods). Selection in SHOPPRICET99 is : select dt_from inside submission date
range.
- Median price is computed using Price Group concept (shop characteristic used to group
shops with congruent price policy). Price group are defined in SHOPUSABILITYT40 table.
Price group to be used for this process are defined as static parameter (PG01,CHVL01,
CHVL02…)
- Median price is computed also for Price group 0 (all shops)
- Median price at Product group level is also computed.
- Median price at Product group level & Price Group 0is also computed.
Then FPV process will use following logic :

Step Period Product group nan Price group source of the information
1 Yyyyppp 0 xxxxxx X from the RMS prices
2 Yyyyppp 0 Xxxxxx 0 from the RMS prices
3 Yyyyppp xxxx 0 X from the RMS prices
4 Yyyyppp xxxx 0 0 from the RMS prices
Remark: Steps 3 is in used only in case Static parameter Price Group 0 is enabled, Step 4 is in
used only if static parameter PG_PRG0 is set to 1
- Median price is computed only if number of observation is greater than minimum number of
observations required to compute median price ( # shops by NAN_KEY or product group) set
by static parameter NOBS.
- Prices are extracted from SHOPPRICET99 using SOURCE as defined in static parameter
SOURCE
- Median Price are loaded in CNANPRICET34 with period computed as the previous as the
submission period (this because FPV process is checking std price for previous period).
Note:
CNANPRICET34 table can be loaded by several processes :
o Compute/Load std price
o Compute Median price NRSP
o Load Std price based on output file of FPV
In case for example std price computation based on Rawda has been submitted already on
same cycle/period, median price from RMS will substitute to the one already present
Recommendations:

Page 172 from 698
This job can be included in same group as the load prices from NRSP (74P)
Constraints:
Only shops defined in SHOPUSABILITYT40 with expected Price Group are selected to compute
Median Price.
Prerequisites:
Price group defined in SHOPUSABILITYT40 table.
Prices to be delivered by NRSP and loaded in SHOPPRICET99 table (74P).
Table Description
SHOP_PRICEST99 Contains prices from RMS at NAN_KEY/ Shop code level
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters :
Static Parameters

Type Value
CHID PG01 SOUR Shop characteristic to be used as price group
PARM NOBS 2 Minimum number of observation to compute
median price
PARM SOURCE 0039 Source to be used in SHOP_PRICEST99
table
PG01 CHVL01 0000000001 Characteristic value #1 to be used for median
price computation
price computation
price computation
PARM PRG0 1 Option to compute standard price at Nan level
for Price Group=0 (all shops)

Page 173 from 698
PARM PG_PRG 1 Option to compute standard price at product
0 group level for Price Group=0 (all shops)
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management:
Proc.TYPE CYCLE PERIOD option tag

Process CYCLE PERIOD Normal Allowed
Process CYCLE PERIOD TBSK F # (mandatory) Not allowed
Process CYCLE PERIOD PC Not allowed
Rework CYCLE PERIOD Normal Not allowed
Rework CYCLE PERIOD TBSK F # (mandatory) Not allowed
Rework CYCLE PERIOD PC Not allowed

code
P1021015 Shop not Found in T40 Shop not in S Shop values is not Check the
T40 available in content of
SHOPUSABILITYT4 SHOPUSABILI
0 table TYT40 table

P9900005 Required No. of Periods not Periods not Information message only. No action
Present enough required
P0540001 NAN KEY/PRGRP: # of Merge Cnt- Information message only. No action
Record Inserted/Updated CNANPRICE required
for PRGRP>0
P0540002 PRODUCT GROUP: # of Merge Cnt- Information message only. No action
for PRGRP>0

Page 174 from 698
P0540003 NAN KEY/PRGRP: # of Merge Cnt- Information message only. No action
for PRGRP=0:
P0540004 PRODUCT GROUP: # of Merge Cnt- Information message only. No action
for PRGRP=0
P9900159 There are Shops in Shops T99 Not Some shops present in
Shop_PricesT99 not in In T40 SHOPPRICET99 are excluded for
ShopUsabilityT40 median price computation because
they are not present in
SHOPUSABILYT40
P9900236 All shops in All shops in Information message only. No action
SHOP_PRICEST99 SHOP_P required
present in
SHOPUSABILYT40

Page 175 from 698
EHDV21055P- Compute Median Price CLUSTER
Phase:
This program computes the median price by CLUSTER (geo / chain) for RMS prices.
Prices used for computation are selected from “scantack price table” (SHOP_PRICEST99)
The table structure of the input SHOP_PRICEST99 is:

Price Price value
Computed median prices are stored in dedicated table CLUSTER_PRICEST100
Structure of the output CLUSTER_PRICEST100 table:

CYCLE 55P submission cycle
PERIOD CODE 55P submission period
Nan NAN_KEY
CHAIN Chain code – Char from shop usability table
GEO Geography code – Char from shop usability table
0 for all GEO
Data-type code 5652
MEDIAN Price Median price value
The assumption of this job is we process all the NRSP prices of a week in a single run.
The job allow to compute cluster price for different sources in T99.
Model
Shop selection
Shops instructed to be included for this process are selected in SHOPUSABILITYT40 table
job using price_group characteristic and char values instructed as static parameter
Prices selection
Prices to be used for this computation are selected from T99 using SOURCE static
parameter and T99 dt_from computed using PERIODS_BX parameter (defined by
SOURCE)
Median prices computation

Page 176 from 698
Shop cells resolution is performed using shops characteristics from SHOPUSABILITYT40.
Shop characteristic for GEO and CHAIN are defined as static parameter.
A median price is computed by NAN_KEY/ CHAIN/GEO

If number for observation available > MINOBS static parameter (defined by source),
media price is not computed
Then median price is computed by NAN_KEY/ CHAIN/ 0
The price allocation by copy cluster price will be then be performed according to the following
hierarchy:
Period nan chain geo source of the information

Yyyyppp xxxxxx x X T100
Yyyyppp xxxxxx x O from the median price file
Recommendations:
This job can be instructed in a group as successor of NRSP price loader job
Constraints:
• Minimum number of observations required to compute median price ( # shops by nan or

product group)
Prerequisites:
• Prices to be delivered by NRSP

• Prices to be loaded in SHOP_PRICEST99 (job 73P)
• Shop usability table filled by shop characteristics from the sample
Table Description
CLUSTER_PRICEST100 Table containing RMS median prices computed at CHAIN/GEO level
Files:
Input Files:
No input file.
Output Files:
No output file

Page 177 from 698
Parameters:
Static Parameters

Type Value
CHID PG01 PGGRP Identify the characteristic ID for the PRICE GROUP to
access the shop usability table
PG01 CHVLxx 0000000001 Identify the characteristic values for the PRICE GROUP
CHID CHAN CHAN Identify the characteristic ID for the CHAIN

CHID GEO GEO Identify the characteristic ID for the GEOGRAPHY
PARM PERIODS 52 Identify the number of Periods to keep in the table for
history
PARM SOURCE1 01 Identify the source #1
PARM PERIODS_ 1 Identify the number of PERIODS BACK to compute cluster
B1 price for source #1
PARM MIN_OBS1 5 Identify the MIN NUM OF OBSERVATION FOR THE
MEDIAN COMPUTATION for source#1
PARM SOURCE2 02 Identify the source #2
B2 price for source #2
PARM MIN_OBS2 5 Identify the MIN NUM OF OBSERVATION FOR THE
MEDIAN COMPUTATION for source#2
PARM SOURCEX 01 Identify the source #X
BX price for source #X
PARM MIN_OBS 5 Identify the MIN NUM OF OBSERVATION FOR THE
X MEDIAN COMPUTATION for source#X
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

Type

Page 178 from 698
Message Description Tag Typ Explanation Action

code e
T40
P9900160 Some Characteristic CharVals S
Values are missing from missing T40
SHOPUSABILITYT40
P1021014 No Shops in T99 are also T99 S
in T40 INTERSECT
T40
Message Description Tag Comments

code
P9900159 There are Shops in Shops T99 Information message only. No action required
Shop_PricesT99 not Not In T40
in ShopUsabilityT40
P1049060 Number of Records Merge Cnt- Information message only. No action required
Merged(INS/UPD) CNANPRIC
into CNANPRICET34 E
P0550001 CHAIN/GEO: # of Merge Cnt- Information message only. No action required

Record Inserted for all CNANPRIC
SOURCES E

Page 179 from 698
EHDV21056P- Copy Median Price CLUSTER
Phase:
This program copies the prices of retailer organized by “CLUSTER” into table P359_1T12.
The assumptions of this job are

• Table of the CLUSTER prices will contain historical data for 4 months
Model :
Shop selection
Selection of shops to be included in this process using price_group char and char values
Cluster resolution
Assignment of shops to cells (GEO/CHAIN)
Geo and chain are shop characteristics. We do not use family char to supply the GEO
information
Selection of all price=0 in P359_2T12
Update of price=0 using following logic
prod type Period Nan chain geo source of the information

Nan_key yyyyppp Xxxxxx x X from the median price file
Nan_key yyyyppp xxxxxx x 0 from the median price file
For each purchase transaction with the price value = 0, we search first for the best detail:
chain x and geo x If missing, we search by chain x only.
Rem : All purchase acts for which price has been imputed are flagged :
with DT5904= 6 (price of cluster Chain/Geo).
With DT5904= 7 (price of cluster Chain/0).
Recommendations:
This job can be set-up as a successor of NRSP price copy job (53P)
Constraints:
Prerequisites:

Page 180 from 698
• File to be delivered by NRSP
• Median price by chain and geo computed (55P)

Page 181 from 698
Table Description
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters:
Static Parameters

Type Value
CHID PG01 PRGRP Identify the characteristic ID for the PRICE GROUP
to access the shop usability table
PG01 CHVLxx XXXXXXXXXX Identify the characteristic values for the PRICE
GROUP
CHID GEO GEO Identify the characteristic ID for the GEOGRAPHY
CHID CHAN CHAIN Identify the characteristic ID for the CHAIN
PARM SOURCE 01 Identify the source to be used in T100
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 182 from 698

code
T40
Missing

succe

Page 183 from 698
EHDV21057P- Copy standard price from Cycle
Phase:
PHASE 6 – AD HOC BY COUNTRY
The main concept of this program is to copy standard price in CNANPRICET34 from a source
cycle to a target cycle for a particular range of periods.
This job is to be submitted using destination cycle (where std price should be copied). The source
cycle is defined as static parameter.
Users can instruct as static parameters the price group to be copied

If no price group codes are declared in static parameter table, all price groups to be loaded during
loading phase.
If a price group card selection is present, it is necessary to declare also the price group 0.
Recommendations:
This job can be used alone or in a group (for example with compute and load standard prices)
Constraints:
All the periods defined in the target cycle are equal to the periods in the source cycle
i. Same label
ii. Same definition
Prerequisites:
CNAN PRICE GROUP table filled for source Audit-Cycle

Source Cycle and Target Cycle existing and belonging to same Audit
Table Description
CNANPRICET34 Std price table
Files:
Input Files:
No input file.
Output Files:
No output file

Page 184 from 698
Parameters:
Static Parameters

Type Value
CHID PG01 PRGRP Shop Characteristic to be used for Price
Group
PG01 CHVLXX Shop characteristic values for Price group
PARM CYCLESO 0050 Source Cycle
Dynamic Parameters

Name (Yes/No)
# Periods yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


code
P9900220 Source Cycle and source Source S
period not available Cycle and
sou
P9900218 NAN not present in the NAN not W
NAN_ITEMST73 Table present in t
P9900219 Source Cycle and source Source S
period without std price Cycle and
sou

Page 185 from 698
P0020564 Error while getting System Error S Check
Position and Precision dtparamters
Info and DT
definition
screen
P0020576 Error while generating System Error S Check
DTInfo for the given Tag- dtparamters
DataCode Info List and DT
definition
screen
MsgLog Array about
Invalid Tags for DTInfo
P1001027 Error while setting DT System Error S Open a QLS
Pointers

read records re
P1049038 Number of Records Insert Cnt- Information message only. No action required
inserted into CNANPRIC
CNANPRICET34 E
skipped records sk

Page 186 from 698
EHDV21060P – Shop Weight computation
Phase:
Shop weight computation job is computing a total FMCG at basket level based on basket
purchase acts.
This information can be then used for panellist usability or expansion factor computation.
The shop weight is computhe from the original purchase transaction basket filtering the purchase
transactions by
➢ List of shop type and/or shop chain
➢ Specific scope of products (based on IBD screen)hop weight will be
identified as a “total basket information” and stored into TBSK table
➢ with a dedicated “datatype code”with the same key of the purchase
basket of the family
Logic of the job
Resolution of scope of product

Retrieve from IBD Group id table instructions related to the IBD selected
Resolve instructions to get list of Nan_key to consider in Shop Weight computation
Resolution of scope of shop

Retrieve from Shop Usability table all shop codes that verify Shop char + Shop char
values conditions instructed as static parameter
Shop weight computation
From RAWDAILYT60 job is selecting purchase records based on date range, list of
nan_key, list of shops.
Then job is computing value spent at basket level (AUDIT/DATE/TIME/HH/SHOP) by
DT5650 x DT5675
Reset of shop weight computation

DT Shop Weight is reset to 0 for submission audit & date range
Insert/update in TBSKT13
For each basket key computed
-If the key is existing already in TBSK13 datatype shop weight is updated with computed value
(DT code instructed as static parameter, recommended value is 5128)
-If the key is not existing TBSK13 a new key is inserted with only shop weight DT >0
Also a Prockey record is inserted with
DT5090 = MAX(DT5090 of purchase acts)
DT5096 = MAX(DT5096 of purchase acts)
Other DT as default values
Note : For python version, Hive data refresh is done at the beginning of this process for the

Page 187 from 698
• Prockeyst29
• Rawdailyt60
• Tbskt13
• Shopusabilityt40
oracle data refresh is done at the end of this process for the submission audit, period
(back-periods included) for following tables:
• Prockeyst29
• Tbskt13
Recommendations:
➢ Submit before the job locking production cycle [EHDV21026P]

➢ Should be submitted after Trip Data Validation
Constraints:
Following check is performed :
LOCK FLAG = 0 LOCK FLAG = 1

by CYCLE / PERIOD by CYCLE / PERIOD
( into CYCLE-PERIOD ( into CYCLE-PERIOD TABLE )
TABLE )
Late family arrival,
BASKET DATA CYCLE validation NOT CYCLE validation already
completed completed, family not usable for
MAX DT-LOCK VALUE = 0 production
( into PR-KEYS ) PROC option: compute shop
weight data RWRK option is mandatory
CYCLE validation already

BASKET DATA completed
MAX DT-LOCK VALUE = 1 CASE NOT VALID RWRK option is mandatory (data
( into PR-KEYS ) validated + late arrival)
Prerequisites:
Rawda should be loaded
Table Description

Page 188 from 698
SHOPUSABILITYT40 Shop characteristics table
HBE_INSTRT102 Instruction for group of product
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters:
Static Parameters

Type Value
DTDT IDDT02 5128 TBSKT datatype to store shop weigth
PARM BCDTYP 1 Barcode type (optional)
CHID CH01 SHOPT Shop characteristic code 1
CH01 CHVL01 HYP Shop characteristic value 1 for shop char
code 1
CH01 CHVLXX SUP Shop characteristic value XX for shop char
code 1
CHID CH02 CHAIN Shop characteristic code 2
CH02 CHVL02 TESCO Shop characteristic value 1 for shop char
code 2
CH02 CHVLXX ASDA Shop characteristic value XX for shop char
code 2
PARM IBD-ID 1 IBD id to be used
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

Page 189 from 698
Instruction of scope of products to be used is retrieved from IBD instruction screens.

code
P9900114 Production cycle already Rework S Resubmit in
completed-Rework mode requested rework mode
requested
P9900177 Shop Not present in Shop Not S Check shop
Shopusability table present in chars for shop
code in error
P1027025 Production Cycle not Period not S Job has been Submit update
closed closed submitted in RWK Proc keys or
mode but period is resubmit in
not closed PROC mode
P9900178 Cycle already closed Cycle S Job has been resubmit in
already submitted in PROC RWK mode
closed mode but period is
closed
P9900179 Data Validation Not Data S Inconsistency
Completed and Basket Validation between period lock
Locked Not and lock flag in T29
P9900184 No Basket Keys selected Basket Keys W No record found in Check
selec T29 for submission submission
period/audit parameter
P9900196 No Records Selected for No Records S No NANS selected Check
processing Selected instruction in
HBE screen

P9900181 Number of shops Number of Number of distinct shops found in
selected for period shops sele Shopusability table for submission char
P9900182 Number of Basket Number of Number of Basket Keys selected from T29
Keys selected Basket Key
S0000006 Number of Records Record Total number of record updated in T13
updated in TBSK updated
succe
S0000004 Number of Records Record Total number of record inserted in T13
inserted in TBSK added
success

Page 190 from 698
S0000004 Number of Records Record Total number of record inserted in T29
inserted in added
PROCKEYS success
P9900206 Number of NANS Number of Number of distinct Nan_key selected based on
selected NANS selec instructed scope of product in HBE instruction
screen

Page 191 from 698
EHDV21061P – Nan reclassification by X-Coding (all tables)
Phase :
This job is re-Xcoding based on EIMDB Xcoding instruction the purchase records present in:
Output rawda (tables RAWDAILYT60 and PROCKEYST29)
FamilyLog table (table FAMILYLOGT67)
The Xcoding logic used here is the same as the one used in Crosscoding process (job 08P), so
Xcoding models are :
- by barcode Type
- by Shop/Xcode Group
- standard Xcoding
-Zero coded
Logic available to select barcodes to be reclassified, selection of the logic is done by value of
Static parameter MODL:
• If MODL=PUBL, job is applying full Re-Xcoding based on Published Copy #
- Select all barcodes present in Rawda for submission Date Range (based on submission
period + # of periods defined in Job submission screen)
- Xcode all barcodes based on given Published Copy. Static parameter PARM PUBL is
defining Published copy to be used.
In case PARM PUBL=0, the latest full published copy loaded in DV is selected
For Barcodes that can not be Xcoded, Nan_Key is set to 0
-A new static parameter LAC_EXCLUDE parameter is introduced to add an extra filter to
select all AC_BCTYPE_MNEM different from LAC or not.
-the below picture describes the LAC_EXCLUDE parameter usage.
Rem: Using Xcoding onDate, only new items and Re-classified barcodes should be selected.

Page 192 from 698
But if EAN XcodeGroup selected all barcodes will be Xcoded (EAN Xcoding are rebuilt each time
a EIMDB published copy is produced).
The Xcoding is using the purchase date to look for correct link to be applied: it is looking for a link
with EIMDB_date_from < Purchase date < EIMDB_date_to
Example of EIMDB item reclassification, based on Date XcodingLink instruction table :
GROUPID BARCODE BC_SUFFIX DT_EFFFROM XCODE_KEY BT_MNEM NAN_KEY NAN_CODE MODULE P_G DT_EFF_TO
PUB_NUM CURRENT_REC DT_CREATED DT_UPDATE
FR17 3059941004407 0 19000101 00:00:00 31069647 EAN 116797 8222000761 2006030500

913 2003101000 2006030209
FR17 3059941004407 0 20060306 00:00:00 66313833 EAN 5581680 8222008681 2525122500
913 C 2006030209 2006030209
This EAN 3059941004407 will be Xcoded to NAN_KEY 116797 for any purchase date before
2006/03/05 and to 5581680 for any purchase date greater than 2006/03/05
At the end of the process TDV_REX_LOG table is filled using RAWDAILYT60 and the process
temp table.The process will insert record only when there is a nankey change and insert job
submission time as timestamp.
The structure of table is as follows:
Recommendations:
Constraints:

Page 193 from 698
Prerequisites:
Xcoding instructions should be set-up
EIMDB should be refreshed
Table Description
CTIONST49
LAST_PUB_EVENTT30 List of loaded published copy
CROSS_CODE_GROUPS List of Xcodes group to be used for Xcoding (Xcode Group screen)
T26
TDV_REX_LOG Stores the list of transactions and nankey change detaiils
Files:
Input Files:
No input file.
Output Files:
List of changed NAN_KEYS (File id: FILEO1) :

1 35 BARCODE VARCHAR2
36 16 NC_OLDNAN NUMBER
52 16 NC_NEWNAN NUMBER
68 3 TRANSACTION NUMBER
List of non Xcoded barcodes (File id: FILEO2) :

1 35 AC_BARCODE VARCHAR2
36 16 NC_OLDNAN NUMBER

Page 194 from 698
Parameters :
Static Parameters
Case of Xcoding by published copy:

Parameter Type Type Name (example) Description
Value
MODL PUBL
PARM PUBL 1550 Publication number
LAC_EXCLUDE YES/NO YES LAC to exclude “YES” or “NO”,default “NO”
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Xcoding instructions are the same as the one used for Xcoding (08P). Those instructions are
defined in GUI screens:
Services → X-Coding → Bar Code-Type: set up of Xcoding instruction by Barcode Type
Services → X-Coding → Shop-Group Id: set up of Xcoding instruction by Shop code

code
S0000109 Missing Link by Exception Missing Link W
in T49
S0000108 Multiple Links Exist in Multiple W
Xcoding Instructions table Links T49
P1008005 Purchase date is not Outside W
inside valid pc-item date IMDB
range of IMDB-Link not Dtrange
applied
P9900194 Nan Key=0 Found in T- Nan Key=0 W
359, transactions Found in T
DELETED.
P9900193 Full Xcoding Required Full Xcoding W Some
Require barcodes are
remained not
Xcoded after
this job

Page 195 from 698
P0020125 File Name is blank File Name is S File name not Enter valid
blank specified filename
not present name

P9900199 Number of Number of Number of purchase transaction selected for
Transactions Selected Transactio re-Xcoding on RAWDAILYT60
for Xcoding
Transactions Selected Transactio re-Xcoding on PROCKEYST29
for Xcoding
Transactions Selected Transactio re-Xcoding on RAWDAILYT60 with
for Xcoding Nan_key=0 before re-Xcoding
P0610001 Number of Number of Number of Transactions re-Xcoded on tables

Transactions Xcoded Transactio PROCKEYST29, RAWDAILYT60.
Number of purchase transition in T60,T29
where Nan_key has been updated
P0610002 Total Number of Total Number Number of Transactions Zero re-Xcoded on
transaction NOT of tran tables PROCKEYST29,RAWDAILYT60.
Linked after Xcoding For those purchase acts Nan_key has been
updated to 0.
P9900185 Total Number of Total Number Number of Transactions Not re-Xcoded on
transaction NOT of tran tables PROCKEYST29,RAWDAILYT60, Due
Linked after Xcoding to Xcode Mode.
In case of Re-Xcoding on Increamental or
Date we do not update any Nan_key to 0
even if no valid link was found for selected
Xcoding Innstruction.
P9900185 Total Number of Total Number Number of Transactions Not re-Xcoded on
transaction NOT of tran tables PROCKEYST29,RAWDAILYT60, Due
Linked after Xcoding to No Change.
Here a Nan_key was found but it was the
same as existing one so update was not
perfomed.
Number of purchase transaction selected for re-Xcoding on RAWDAILYT60

=
Number of Transactions re-Xcoded on tables PROCKEYST29, RAWDAILYT60.
+
Number of Transactions Zero re-Xcoded on tables PROCKEYST29,RAWDAILYT60.
+
Number of Transactions Not re-Xcoded on tables PROCKEYST29,RAWDAILYT60, Due to Xcode
Mode.
+
Number of Transactions Not re-Xcoded on tables PROCKEYST29,RAWDAILYT60, Due to No
Change.
Same logic is applied on FAMILY_LOGT67

Page 196 from 698
EHDV21062P - Load OPUS Prices
Phase:
• This program loads load prices supplied by an In Store Observation Service (OPUS –
France) into the OPUS_PRICEST93 table.
• The Prices in the Input File are at Barcode level but are loaded in the OPUS_PRICEST93
table at NAN_KEY level (after Xcoding).
Structure of the OPUS_PRICEST93 table is :
Source Identifier for source of price values

Data-type code 5651 for OPUS price
Price Price value
Source is defined as static parameter
Process Flow
• Get Static Parameters
• Read Input File
• Perform XCoding
IF IDCH01 is defined
Reclassify RMS shops to CPS shops
• Perform XCoding
• Load records in OPUS_PRICEST93 table as shown below:
Source SOURCE value from Static parameter

Shop code CPS shop code (after shop reclassification if any otherwise
RMS shop code)
Nan_Key NAN_KEY
P_Date_from Date_From (based on job submission period and Number of
Periods back)
Data-type code 5651
Price Price value / 100
P_Date_to Date_To
(based on job submission period)
Recommendations:
Constraints:
Prerequisites:
• Availability of the file containing the price values.

Page 197 from 698
Table Description
OPUS_PRICEST93 OPUS_PRICEST93 table contains item prices only for a month
data. No history retained in this table.
CROSS_CODE_GROUPST26 List of Xcode Group ID for the country
CROSSCODING_INSTRUCTIONST49 Crosscoding links information extracted from OGRDS
SHOPUSABILITYT40 It contains the characteristic identifier and the characteristic
value by shop by cycle and period (0 or YYYYPPP)
To be updated by cycle/period at each validation cycle (if
requested)
Files:
Input Files:
OPUS prices (File id: FILEI1) .
The Input file is a fixed layout file.
Field Field Name Type Width Sample Values

1 Opus id Character 6 AU0301
2 Shop Character 10 5900000256
3 Barcode Character 13 3011360030481
4 Price Character 6 0 decimals
The price is represented in cents of Euro

Loading the price value inside the price table the price has to be divided by 100 (and keep the
decimals)
Output Files:
No output file
Parameters:
Static Parameters

Type
CHID IDCH01 ENS Identify the characteristic ID for the shop code
reclassification into shopusability table
PARM SOURCE 9933 Identify the Source ( country or audit )

Page 198 from 698
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection Yes
Rework Management:


code
P0020505 Static Value not present Stat.Val not S No transactions Do the
present available for the necessary
selected timestamp transaction for
selected
timestamp
P9900176 Date Range – Overlap Date Range S
– Overlap
P9900187 Shop Reclassification Stat.Val W No Shop
NOT requested. 'IDCH01' not reclassification is
present requested
P0020125 OPUS File is Blank or S Check if the file
NOT present is present in
the Input
directory OR if
content is
BLANK
(EMPTY)
P9900187 Shop Reclassification I Check if there are
NOT Required RMS shop codes
that have to be re-
classified to CPS
shop codes
P9900005 Required No. of Periods Periods not I
not Present enough
P0020608 Static Value Static Value I
P9900110 Date Range Date Range I

Page 199 from 698

P9900142 # of Input Records: Number of Information message only. No action required
records re
P9900141 # of Inserted Records: Number of Information message only. No action required
records in
P9900143 # of Skip Records: Number of Information message only. No action required
records sk

Page 200 from 698
EHDV21063P- Copy OPUS Prices
Phase:
This program copies the OPUS prices stored in table OPUS_PRICEST93 into table P359_1T12.
The general assumptions of this job are:

• All the purchase transactions are processed by the cycle / period concept
• The OPUS prices delivered once a month
• OPUS_PRICEST93 table contains item prices only for a month data. No history retained
in this table.
The table structure of the input OPUS_PRICEST93 is:
Source Identifier for source of price values

Data-type code 5651 for OPUS price
Price Price value
Shop code local: local shop list not belonging to RMS standards..
Imputation Model :
Select all keys to be included in this job:
• Check the existence of records in the KEYST14 table for the submitted Audit and Timestamp.
• Select all records from the KEYST14 for the submitted Audit and Timestamp where ac_item
<> 0 nan_key > 0)
Retrieve prices from OPUS:
• Check all prices available in OPUS_PRICEST93 table with

P359.shop= OPUS_PRICEST93.shop
Nan_key = OPUS_PRICEST93.NAN_KEY
• For the purchase keys selected from the KEYST14, select the corresponding keys from
p359_1t12 where Price = 0
• Check if matching records found in OPUS_PRICEST93 (for the records with shop and
Nankey)

Page 201 from 698
• Apply Price correction on P359
• All purchase acts for which price has been imputed, are flagged with DT5904= 8 (price
correction flag).
Recommendations:
The NRSP price copy must be run before this job (OPUS Price copier)
Constraints:
• Only the OPUS prices are copied
• The OPUS prices must not be changed by any validation routine
Prerequisites:
• Data present in the validation area (tables P359_1t12, KEYST14)

• attention : the same shop code can be present in the NRSP data and in OPUS data
Table Description
KEYST14 DV validation area keys
OPUS_PRICEST93 Table containing prices by shop, NAN_KEY
Files:
Input Files:
No input file.
Output Files:
No output file

Page 202 from 698
Parameters:
Static Parameters

Type
PARM SOURCE 01 Identify the characteristic values for the source
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


.

code

S0000006 # of Recs in UPD Information message only. No action required
P3591_T12 Updated Counter

Page 203 from 698
EHDV21065L – Housekeeping Joblog and Msglog
Phase :
PHASE 7 – ONLINE CLEANING
This program is responsible for the cleaning of Joblogt52 & MsglogT53 tables.
The cleaning is based on the jobstart time which is present in both tables. The user defines for
each table separately in the static parameter table the number of periods to retain in the
database. This is a physical delete in database.
Logic of the deletion :

➔ Job is retrieving the max period locked in CycleperiodT03 for the submission cycle
➔ Job is retrieving number periods to retain in T52 (defined by user as static param T52)
➔ Based on Cycle/ Period definition (CycleperiodT03), program is computing the first
period to retain
➔ Job is computing date_from of the first period to retain: v_ac_dtfrom
➔ Job is deleting from T52 all records for submission cycle
with jobstarttime < v_ac_dtfrom
Then same logic is applied to T53.
Recommendations :
This job can be submitted inside a group containing other Housekeeping jobs.
Constraints :
Prerequisites :
Production cycle should be completed and locked.
Table Description
CYCLEPERIODT03 Definition of periods (date from, date to, lock flag) by Cycle
JOBLOGT52 Table containing a Log of all submitted jobs (one record by job submission).
Table displayed in Job Submission → Job Log screen
MSGLOGT53 Table containing detailed report of each job submission (table accessed by
Job Log → View Messages)
Files:
Input Files:
No input file.

Page 204 from 698
Output Files :
No input file.
Parameters :
Static Parameters

Type Value
PARM T52 156 Number of periods to retain in T52 table
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


Page 205 from 698
EHDV21065U – Housekeeping ShopUsabilityT40 and FamilyUsabilityT41
Phase :
This procedure is responsible for the cleaning of ShopUsabilityT40 & FamilyUsabilityT41 tables.
The cleaning is based on the period which is present in both tables. The user defines for each
table separately in the static parameter table the number of periods to retain in the database. This
is a physical delete in database.
period to retain : p_from
with period < p_from
Recommendations :
Constraints :
Prerequisites :
Table Description
Files:
Input Files:
No input file.

Page 206 from 698
Output Files :
No input file.
Parameters :
Static Parameters

Type Value
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


Page 207 from 698
EHDV21065M – Housekeeping HHParamT06 & FamilyMonitorT42
Phase :
This procedure is responsible for the cleaning of HHParamT06 & FamilyMonitorT42 tables.
table separately in the static parameter table the number of periods to retain in the database.
This is a physical delete in database.
Recommendations :
Constraints :
Prerequisites :
Table Description
HHPARAMT06
FAMILYMONITORT42 Store panellist collaboration by week
Files:
Input Files:
No input file.

Page 208 from 698
Output Files :
No input file.
Parameters :
Static Parameters

Type Value
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


Page 209 from 698
EHDV21065P – Housekeeping CnanPrice, ClusterPrices, Shop_Prices,
Supplier_Prices
Phase :
This procedure is responsible for the cleaning of CnanPriceT34, ClusterPricesT100,

Shop_PricesT99, Supplier_PricesT98 tables.
Functionality for tables CnanPriceT34 and ClusterPricesT100 is similar to as in EHDV21065U.

table separately in the static parameter table the number of periods to retain in the database.
This is a physical delete in database.

Functionality for Supplier_PricesT98 is similar to EHDV21065U job. The cleaning is based on the
date_from.
period to retain
with dt_datefrom < v_ac_dtfrom
Functionality for Shop_PricesT99 is specific as this table is can contain huge amount of data and
is a partitioned table.
The cleaning is based on the partition name.
The user defines in the static parameter table the number of periods to retain in the database.
period to retain
➔The starting date of periods to be retained is then converted to following format
‘P’YYYYMMDD (format of partition name of T99).
with dt_datefrom < v_ac_dtfrom

Page 210 from 698
➔The list of partitions to be dropped is retrieved by selecting list of partitions of T99 with
partition name lower than date_from
➔ all drop partition statements are submitted.
Recommendations :
As volume in T99 can growth to huge amount of data, it is recommended to not retain
more than 12 weeks in this table.
Constraints :
Prerequisites :
Table Description
CNANPRICET34 Contains standard prices and MAD by cycle, period, price group, product
group, NAN_KEY
CLUSTERPRICEST100 Table containing cluster prices computed by 55P job
SHOP_PRICEST99 Table containing RMS prices by period, shop, NAN_KEY
Files:
Input Files:
No input file.
Output Files :
No input file.
Parameters :
Static Parameters

Type Value
Dynamic Parameters

Page 211 from 698
Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


Page 212 from 698
EHDV21066P – Check Family Code on T-ALL
Phase :
This program validates the family codes inside the purchase data loaded in table ALLT11.
It Checks whether the families inside the purchase data are present in the family list (T41 table)
HH present in HH not present in

T-ALL table T-ALL table
HH present in OK
HH LIST table ----------
HH not present in WARNING

HH LIST table CC 08 ----------
If the condition code 08 is issued, we can have 2 types of errors :

• The family code is missing from the HH LIST table and the user has to update the HH
usability table
• The family code is wrong and the user needs to correct the input file (run again the standard
loader)
Recommendations:
It is recommended to submit this job before Data Selection and adjust situation if needed, as any
purchase record with HH not present in Family List will remain in table ALLT11 and will not be
moved to the validation area.
As families’ characteristics are loaded by period, it is recommended to submit this job at real
period level (so not using period 0)
Constraints:
Prerequisites:
Load of purchase transaction.

Loading of families into family usability
Table Description

Page 213 from 698
ALLT11 Parking place of all the purchase transactions before validation.
Area of the cross coding or new item classification.
Files:
Input Files:
No input file.
Output Files:
No input file.
Parameters :
Static Parameters
Dynamic Parameters

Name (Yes/No)
Period Yes Submission with Period 0 is allowed for this job, in that case the whole
table will be checked
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection Yes
DTSelection No
File Selection No
Rework Management:


Page 214 from 698
code
P9900188 No transactions available No S Table ALLT11 is Check if
for current selection transactions empty for purchase data
avai submission audit + have been
date range loaded
P9900189 Family not present in T41 Family not W Family details are Load family
but purchase transactions present in missing in T41 table details into T41
available T41 for the available table or
transactionsa correct& reload
purchase file
P1012005 Error while checking System Error S Generic Error Open QLS
Family Codes

P1012008 Number of distinct Distinct Number of families present in ALLT11
families Validated family Cnt
P1012009 Number of Active Active Number of families present in ALLT11 and in
Families Family Cnt FAMILY list
P1012010 Number of Families Missing Number of families present in ALLT11 and not
missing in Family Family Cnt in FAMILY list
Usability Table

Page 215 from 698
EHDV21067P- Check Shop Code on T-ALL
Phase :
This program validates the shop codes inside the purchase data file. It Checks whether the shops
inside the purchase data are present in the shop list table / shop usability table (SHOPT44)
SH present in T-ALL SH not present In

table T-ALL table
SH present in
SH usability tab OK ----------
SH not present in
SH usability tab WARNING ----------
CC 08
If the condition code 08 is issued,

We can have 2 types of errors:
➢ The shop code is missing from the usability table and the user has to update the Shop
usability table.
➢ The shop code is wrong and the user has to reload the current standard input file. (run
again the standard loader)
Recommendations :
It is recommended to submit this job before Data Selection and adjust situation if needed, as any
purchase record with Shop not present in Shop List will remain in table ALLT11 and will not be
moved to the validation area.
Constraints :
Prerequisites :
Load of purchase transaction

Loading of shops in to shop usability
Table Description
ALLT11 Parking place of all the purchase transactions before validation.
Area of the cross coding or new item classification.

Page 216 from 698
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters :
Static Parameters
Dynamic Parameters

Name (Yes/No)
Period Yes Submission with Period 0 is allowed for this job, in that case the whole
table will be checked
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection Yes
DTSelection No
File Selection No
Rework Management :


code

Page 217 from 698
P9900188 No transactions available No S Table ALLT11 is Check if
for current selection 218uthorisa empty for purchase data
ti available submission audit + have been
date range loaded
P9900191 Shop Code Missing from Shop Code W Shop code is not a Check the
T40 table Missing Available in T40 missing shop
table code
P9900192 Shop Description Missing Shop W Description of shop Please enter a
from T44 table Description is missing valid
Missing description for
shop
P9900190 Shop not present in T44 Shop not W shop not available in
but purchase transactions present in SHOPT44 for the
available T44 available transaction
P1020022 Error while identifying System S Generic error open a QLS
shop codes Error

Shops Validated Shop Cnt

Page 218 from 698
EHDV21068P- Data Selection
Phase:
This is the starting point of the data validation. This program selects the set of data to be
processed by the criteria defined by the user and it distributes the other data to the related tables.
Only valid purchase data are moved to P359_1T12 and KEYST14, this means:
• keys Xcoded
• keys not present in the new item table
• date of the purchase transaction consistent with the activity dates of the pc-item
inside the IMDB ( dt_val_from <= key date <= dt_val_to )
• panelist code present in Familyusability table
• shop code present in Shopusability table
• keys not present in validation area
If the key is already in validation area, we do not move any data of the same basket in T-359 until
the validation has been completed, the rawda loaded and the data already validated deleted from
T-359, T- ALL and KEY table
File number concept :
When we load a basket (all purchase transactions of a family in a shopping trip), we associate to
all purchase transactions a “file number” to identify the origin of the data and to control the flow
and the consistency of the operations inside DV system.
The file number is a unique identifier, generated automatically by the system during the loading
phase of the data in T-ALL (01P), and it will be associated to all the purchase transactions within
the same input file.
This file number will control the processing system and the rework system in order to prevent
data disruption due to wrong processing options selected by the user.
The File log screen is displaying information about loaded files with details about number of
records loaded, also information flag if data have been loaded or not in Final tables.
If a file number is loaded in to final table, the field DT_RAWDALOAD in File log table will not be
null for that file number. If this value is null, it indicates that the file number is not loaded yet.
Standard processing mode :
We always load the full basket of the purchase transactions for each family in Table ALL, for
standard load process there is no distinction between:
• standard loading,
• loading by File#,
• loading by pc
In case of normal processing mode (no Rework), it is not allowed to have in DataSelection
records belonging to a basket already present in final with a file# not already loaded in
final table.

Page 219 from 698
To manage purchase data on already loaded basket is it mandatory to have in ALLT11 data with
the same File# as in Rawda.
To manage new File# on already loaded data, Rework mode is required (take care that rework
mode will delete data in Rawda).
➔ In case of process mode 1, 2 following control is applied:
The job is checking if basket key selected by DataSelection are already present in output tables
(PR-KEYS of HHLOG) with different File#
If this is the case, then all purchase keys & tbskt keys related to this basket key are excluded
from Dataselection
➔ In case of process mode 3 (submission for a list of PC) following control is applied:
The job is checking if basket key-PC combination selected by DataSelection are already present
in output tables (PR-KEYS of HHLOG) with different File#
If this is the case, then all purchase keys & tbskt keys related to this basket key are excluded
from Dataselection
Rework mode:
Following table is summarising check performed in case of standard mode :
One of the file #

associated to the basket key
in KEY table has date null in Check OK
file log Rework allowed
( never loaded in rawda )
All the file #’s

associated to the basket key
in KEY table Check NOK
have date not null in file log Rework not allowed
( all them loaded in rawda )
If the data present in ALLT11 table is from a file number, which is already present in Final tables,
then rework option is not allowed.
The option is not allowed, because, this case will happen only under the following circumstances:
A file is loaded in to ALLT11. It is cross coded. Part of the file is not cross coded.
The part, which is cross coded is moved into Final tables.
Now the remaining part is cross coded and the user is trying to move it to final tables.

Page 220 from 698
If rework option is allowed for the above condition, we will delete the earlier records, which are
already present from the same file. This is not correct.
If rework option is accepted
In case or procession option 4 (Rework mode, not option)
All data in final tables related to submission cycle/ data range are deleted.
In case or procession option 5 (Rework mode, by file#)
All keys related to basket keys selected by DataSelection are deleted from final tables
Recommendations:
Constraints:
Prerequisites:
Data should be loaded in ALLT11 and Xcoded

HH + SHOP information should be loaded
Table Description

Page 221 from 698
Files:
Input Files:
No input file.
Output Files :
No output file.
Parameters :
Static Parameters
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No This job is creating a Timestamp
PC Selection Yes
F# Selection Yes
DTSelection No
File Selection No
Rework Management :


code
Info and DT
definition
screen

Page 222 from 698
P1009009 Rework by period and Old data in S
audit requested-T11 still T11
contains old data
P1009005 Exception occurred in System Error S
Rework Audit Period
P1009030 Key found during Rework System Error S
– Not Correct
P1009010 Data linked to file # not F# not W
processed-already in Processed
validation/final tables-rwk
reqd
Control Process Option
P1009015 File no. requested for rwk F# in final S
present in final tables- tables
process option allowed
only
P1009013 Exception occurred in Rewk Aud- S
Rework Audit Period File Per-F# err
P1009021 Rework by pc requested – System Error S
pc not present in T-ALL
P1009022 Rework by pc requested – PC in more S
pc inside more than a file than 1 F#
– not allowed
P1009031 Selected PCs are not System Error S
present in any files
P1009024 Rework by pc requested- PCs in S
pc codes requested are multiple F#
not inside an unique file
number
P1009025 Rework by pc not allowed Rewk-PC S
if other data is still in not allowed
validation phase
P1009018 Exception occurred in System Error S Open QLS
Rework Audit Period PC
DeleteSpecial1
P1009029 Error in Reset of lock System Error S Open QLS
flags
P1009002 No records selected from No records S
table ALL for the Audit + selected
Period
P1009003 No keys moved from T- No keys S
ALL to T-359 moved
P1009032 Rework by PC not Rwk-PC not S
allowed for Home allowed
Scanner
P1009004 No keys available for No keys S
processing available
P0020578 Error while generating System Error S Open QLS
given Tag List

Page 223 from 698
Process Audit Period
P1009034 Error While deleting the System Error S Open QLS
selected PCs from final
tables

P1009036 Total Number of Purchase Information message only. No action required
Purch acts/TBSK Keys count
Records loaded in to
Keys table
P1009037 Total number Baskets Basket count Information message only. No action required
loaded in to Keys
table
P1009038 Total Number of PCs PC Count Information message only. No action required
loaded in the Keys
table

Page 224 from 698
EHDV21069P- Copy total basket Finland
Phase:
PHASE 5 – AD HOC BY COUNTRY
This program is used to copy total basket records in TBSKT13 table into purchase records in
RAWDAILYT60.
This functionality was requested by Finland.
The purchase records copied have same key as Tbasket records except Barcode that is
instructed as dummy barcode by static parameter. The Nan_Key is also instructed as static
parameter.
The keys are inserted also in PROCKEYST29.
One or several datatypes values can be copied from T13 to T60. Source and destination
datatypes are instructed as static parameter.
Recommendations:
Constraints:
Prerequisites:
This copy should be possible when validation is completed, after the Update Processing Key
submitted
Table Description
Files:
Input Files:
No input file.
Output Files:
No output file

Page 225 from 698
Parameters:
Static Parameters

Type Value
PARM BCODE Fixed dummy barcode copied in T60
PARM BNAN Fixed dummy Nan_Key copied in T60
DTDT DT01 Source datatype 1 (from T13)
DT01 DTTG Desctination datatype in T60 for source
datatype 1
DTDT DTXX XXXX
DTXX DTTG XXXX
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


code
P9900032 Production cycle not Prod.cycle S
completed yet open
Lock. RELEASE
Lock

Page 226 from 698
P0020574 Error while populating the System Error S
MsgLog Array about
Invalid Static Values
in struct

P9900031 Production cycle Prod.cycle Information message only. No action required
already completed Closed
read records re
inserted in to Rawda
RAWDAILYT60
P1049035 Number of Records Insert Cnt – Information message only. No action required
inserted into PRKEYS
PROCKEYST29

Page 227 from 698
EHDV21070P- OGRDS LOADER
Phase:
This process loads/refreshes the OGRDS product referential tables in NCPS DV from the
published copy of the OGRDS system through a DB Link interface. Those tables are used by
many other DV jobs to retrieve Xcoding instructions, product characteristics, weight, units etc
The OGRDS FULL published copy is a set of tables containing a full consistent product
referential. This is a full snapshot of OGRDS. This set is published every week-end.
Each published copy is identified by a unique identifier, the OGRDS “Publication number”.
Recommendations:
It is recommended to update OGRDS before any production.
Constraints:
All OGRDS tables are unique across audits (one product referential for all audits).
Prerequisites:
OGRDS production system in MOP
Table Description
CROSS_CODE_GROUPST26 List of Xcode Group ID for the country
CROSSCODING_INSTRUCTIONST49 Crosscoding links information extracted from
OGRDS
CHAR_TYPEST17 List of product characteristic codes from OGRDS
CHAR_TYPES_TRANST18 Description of product characteristic codes from
OGRDS
CHAR_VALUEST24 List of product characteristic values from OGRDS
CHAR_VALUES_TRANST25 Description of product characteristic values from
OGRDS
CVALS_IN_USET27 List of characteristic values in use
IMDB_LANGUAGEST16 List of languages in use in OGRDS
MODULEST31 List of Modules (link to Product group, SuperGroup
MODULES_TRANST32 Description of modules
MOD_CTYPE_ASGNST77 List of characteristic assigned to Modules
MOD_WEIGHT_ASGNST78 List of weight codes assigned to Modules
NAN_CHAR_VALUEST75 Nan Characteristic values

Page 228 from 698
NAN_ITEMST73 List of Nan_keys extracted from EIMD, link to
Module
NAN_ITEMS_TRANST74 Description of Nan_keys
NAN_ITEM_LINKST76 Table containing all linked Nan_key (eg banded
packs items)
NAN_WEIGHT_VALUEST79 Weight of Nan_keys
PRODUCT_GROUPST45 List of Product group
PRODUCT_GROUPS_TRANST46 Description of product groups
SUPER_GROUPST47 List of super groups
SUPER_GROUPS_TRANST48 Description of super group
UNITST85 List of units
UNIT_CONVERSIONST86
WEIGHT_CODEST68 List of Weight codes
WEIGHT_CODES_TRANST69 Description of Weight codes
WEIGHT_UNIT_ASGNST80
WT_VALS_IN_USET33 List of Weight codes in use
IMDB_HISTORICAL_EVENT_LOGT10 List of all publications (Full and Inc) published by
OGRDS
LAST_PUB_EVENTT30 List of published copy loaded in NCPS DV
NAN2CNAN_FROM_IMDB Link between Nan and legacy CNAN, extracted
from OGRDS as a Nan local characteristic
NAN_VAR_WEIGHT_FROM_IMDB Table containing list of variable weight Nan_Keys
X_CODE_GROUPST26A List of Xcode groups for the country with flag to use
or not for Xcoding
TDV_ITEM_TYPE_SETUP Setup of how the NAN_KEY values will be loaded
in CROSSCODING_INSTRUCTIONST49 table
based on NAN Item Type.
TDV_OGRDS_DESCR_DECODER AMAZON ASIN Code Handling
Files:
Input Files:
No input file.
Output Files:
No output file
Setup_Param:
OGRDS_HEADINGS can be set for the country to have one of the two values:
ONLINE or HHT.
• When ONLINE is set the NAN descriptions are taken from the OGRDS
Imdb_Pub_F_Nan_Items_Alt_Descs_cty_v view.
• When ONLINE is set the NAN descriptions are taken from the OGRDS
Imdb_Pub_F_Nan_Items_cty_v view.

Page 229 from 698
Parameters:
Static Parameters:

Type
PARM CNT01 FR Country code
PARM EAN EAN Option to load EAN Xcode group
PARM PGI_OPT Option to load PGI links
If this static parameter is missing OR value is

NULL, all the nan key values are loaded in
the CROSSCODING_INSTRUCTIONST49
table based on the NAN Item_Type and the
setup of the TDV_ITEM_TYPE_SETUP.
If this static parameter is present as valid IBD

id, all the nan key values are loaded in the
CROSSCODING_INSTRUCTIONST49 table
except of the nan keys which are in scope of
the IBD, based on the NAN Item_Type and
the setup of the TDV_ITEM_TYPE_SETUP.
PGI and multilinks are loaded for nan in
scope of defined IBD
PARM GRID0 ZR0 To load or not NAN 0
CHID CNAN 17320 OGRDS characteristic id for legacy CNAN
value
If this static parameter is not present table
NAN2CNAN_FROM_IMDB is not populated
CHID NAN_VAR_WGT 22456 OGRDS characteristic id for variable weight
flag.
If this static parameter is not present, table
NAN_VAR_WEIGHT_FROM_IMDB is not
populated
NANV CHVL01 14951548 OGRDS characteristic value
Further explanation of the job available Static Parameters
Country code (static parameter PARM CNT01)
The country (eg GB) is used to filter xcoding instructions, only information with Xcode group
starting with country code.
EAN Xcode Group (static parameter PARM EAN)
Using this static parameter the EAN Xcode group is loaded on top of country Xcode Group. This
Xcode group is built by program in OGRDS, this is containing all EAN external codes present in
any country and Xcoded with same NAN_KEY in all countries.
This Xcode group is built from scratch (no history, all update date are set to the publication date).

Page 230 from 698
NAN Zero
Using this static parameter the dummy nan_key zero is loaded in NAN_ITEMST73 table. This
isZero NanKey value is used by the DV logic.
PGI option
This option can be used to include or no include non fully coded items and multilink items in
Xcoding instruction.
PGI items are Product Group Item, these are generic nan_key that represent one category, for
such nan_key some characteristics are not filled (like Brand…). In some country when a new
barcode is codified, it is in some cases first linked to a PGI item for a temporary period and then
linked to a full coded nan_key.
Multilink Ean are generic items that are linked to more than one barcode.
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:
Message code Description Tag Type Explanation Action

P0010035 Oracle error Oracle S In case the oracle
error error you are
getting is the
following:
“ORA-08103:
object no longer
exists”
then you have to
wait for OGRDS
publish copy to be
completed before
running OGRDS
load again.

Page 231 from 698
P0020505 Static Value not Stat.Val S
present not
present
P9900171 Publication Number Invalid S
available is older Publication
than the one
already loaded
P9900172 Publication Already Publication S
Present Present

code
P0020608 Static Value Static Value Information message only.
No action required
P9900173 Country Selection Country Information message only.
Selection No action required
P9900169 Publication Number Publication Information message only.
Number No action required
P9900397 Reloading Publication Number: Reloading Information message only.
Pub# No action required
P9900601 OGRDS_HEADINGS Setup_Param Value Setup_Param Valid values are: ONLINE
Value or HHT
P0710001 Record(s) Inserted Successfully in # Records
X_CODE_GROUPST26A Table Inserted
CHAR_TYPEST17 Table Inserted
CHAR_TYPES_TRANST18 Table Inserted
CHAR_VALUEST24 Table Inserted
CHAR_VALUES_TRANST25 Table Inserted
CVALS_IN_USET27 Table Inserted
MODULEST31 Table Inserted
MODULES_TRANST32 Table Inserted
MOD_CTYPE_ASGNST77 Table Inserted
MOD_WEIGHT_ASGNST78 Table Inserted
PRODUCT_GROUPS_TRANST46 Table Inserted
PRODUCT_GROUPST45 Table Inserted
SUPER_GROUPST47 Table Inserted

Page 232 from 698
UNITST85 Table Inserted
UNIT_CONVERSIONST86 Table Inserted
WEIGHT_CODEST68 Table Inserted
WEIGHT_UNIT_ASGNST80 Table Inserted
WEIGHT_CODES_TRANST69 Table Inserted
WT_VALS_IN_USET33 Table Inserted
NAN_ITEMST73 Table Inserted
NAN_ITEMS_TRANST74 Table Inserted
NAN_ITEM_LINKST76 Table Inserted
NAN_WEIGHT_VALUEST79 Table Inserted
NAN_CHAR_VALUEST75 Table (DVD Inserted
chars)
CROSSCODING_INSTRUCTIONST49 Inserted
Table.
Nan_Var_Weight_From_IMDB Table Inserted
SUPER_GROUPS_TRANST48 Table Inserted
P0710028 Number of NanKeys reset to 0 in # Records
CROSSCODING_INSTRUCTIONST49: Updated
P0710029 Number of NanKeys set to negative in # Records
CROSSCODING_INSTRUCTIONST49: Updated
TDV_OGRDS_DESCR_DECODER Table Inserted

Page 233 from 698
EHDV21073P- Active Items extraction
Phase:
This program extracts flat file with NAN_KEY activity for the NRSP factory.
This program will be used to generate the list of items to be included in the item-list resolution
process.
This list of the items will be assigned to the production unit named COUNTRY / CHANNEL
Active item list generation
The NRSP system requires a full updating of the list of the active items before starting
each production cycle.
We need to identify the list of active items present in the whole RAWDA.
This list of items will replace the list of active items already present in the NRSP repository.
All child banded pack items are extracted in addition to all NAN_KEY present in Rawda.
This process extracts the nankeys from PROCKEYST29 table and active dates based on the
ITEM_ACTIV_DATE parameter in SETUP_PARAM table. Value could be either REAL_DATES or
PERIOD_END, where PERIOD_END represents current functionality and REAL_DATES skips
date transformations.
This process has a functionality to submit an external shell script than can send the extracted file
to NRSP system.
Extraction module will use it to detect all cycles/audits within same RMS file to be extracted at
once.
In case the job is submitted with period 0, the job will replace period 0 with the latest locked
period in DV for submission audit. This period will be used for output filename and also for end
date inside output file.
Recommendations:
Constraints:
Prerequisites:
Rawda should be loaded with expected periods.
Table Description
NAN_ITEM_LINKST76 Link between banded pack items (parents) and banded back components
(childs)

Page 234 from 698
Files:
Input Files:
No input file.
Output Files:
Active Nan list (File id: FILEO1) :
Output file is using following naming convention
<ExternalCountry>.<ExternalChannel>.<YYYY>WK<0ww>.<YYYY>WK<0ww>.SGALL.DAT
where :
ExternalCountry is Country Code : eg “GB”
ExternalChannel is Channel Code : eg “NCPS”
YYYY = year such as 2006 : is the submission year
ww = week number such as 52, : is the submission period
Output file layout :

1 20 NC_NAN_KEY NUMBER
21 1 M_AC_SEP ;
22 8 M_AC_YEAR_FROM YYYYMMDD
Date_From is date_from of job submission period
30 1 M_AC_SEP ;
32 8 M_AC_YEAR_TO YYYYMMDD
Date_To is date_to of job submission period,
same for all records
Parameters:
Static Parameters

Type Value
PARM COUNTRY FR Country Code
PARM SCRIPT_C Check Appendix for External shell to be submitted
FT Information
PARM CHANNEL NCPS Channel Code
PARM BNDP BPACK_CONSTIT Banded pack char to be considered
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No

Page 235 from 698
Rework Management :


P1041009 Total number of Record Cnt Information message only. No action required
records Extracted

code
following name:
ended JOB required
displayed:
displayed:
EHDV21074P- NRSP Price Loader

Phase:

Page 236 from 698
This program loads the RMS prices extracted from SIRVAL. These prices are used by other DV
process to impute prices on purchase data where price is missing.
A facility is available in SIRVAL system to extracte the required shop price file in CSV file format.
In DV, prices are loaded in table SHOP_PRICEST99, structure for this table is the following :

Price Price value
P_Date_to Price validity date to
The job is reporting all records with:
price =0
shop code=0
NAN_KEY=0
The “ Price validity date to “ inserted is the date_to of submission period.

The “ Price validity date from “ inserted is the date_from of submission period.
(The date in input file header record is not used)
When rework mode is selected, program is deleting all the records already loaded for the number
of selected weeks ( # periods).
Shop reclassification:
In some countries, shop codes in RMS and CPS are different. In that case, if a copy of prices at
shop code level is required, it is necessary to “reclassify” the shop code coming from RMS to the
correct shop code present in NCPS DP.
This program has included a functionality to do this.
To use this functionality, RMS shop code should have been loaded previously in Shop Usability
table (as all other shop char).
In shop usability we have following situation :

the key of the shop usability table represents the shop codes inside CPS
the value of the char represent the shop code outside ( RMS shop )
Then, if static parameter CHID is filled with correct shop characteristic code.
For each shop code in file coming from RMS, program is checking if this shop code is present in
shop_usability as 237uthor shop char value and if it is present, then RMS shop code is
substituted to the shop code of the key of the shop usability table (CPS shop code).

Page 237 from 698
We can have special cases where, 2 distinct shops have a correspondence between their codes:
CPS shop code of the first one is equal to the RMS shop code of the second one.
Example :
CPS shop code RMS shop code
1100000354 1100110354
No cps code 1100000354
In this case RMS shop code 1100000354 is ignored by the program to avoid to have 2 shop
codes in T99 for same period with 238lignment prices.
Recommendations:
Amount of data loaded by this program in SHOP_PRICEST99 table can be really huge.
To avoid lack of space issue, it is recommended to use Housekeeping process on a regular way
to delete in SHOP_PRICEST99 data related to old periods.
Constraints:
RMS prices loaded in SHOP_PRICEST99 are common to all audits inside the country.
Prerequisites:
Availability of the file containing the price values
Table Description
Files:
Input Files:
RMS price in CSV format (File id: FILEI1) :
 File header Section

Name Description
Flag ‘H’
ChainCode Chain Code
Date DD/MM/YYYY
 File Data Section

Name Description
Flag ‘D’

Page 238 from 698
Shop
Product Code NAN
Stock 2 decimals
Purchases 2 decimals
Sales 2 decimals
Price 2 decimals
Product Name Can be empty
GrpCode Blank
GrpName Blank
CaseSize Def Val 1 - 4 decimals
Manufacturer Blank
Output Files:
No output files
Parameters:
Static Parameters

Type Value
PARM SOURCE 01 Source of Price in SHOP_PRICEST99
CHID IDCH01 RMS_CODE Shop characteristic to be used for shop
reclassification. If missing, no shop
reclassification applied.
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management:


Page 239 from 698

code
P0020125 File Name is blank File Name is S File name not Enter valid
blank specified filename
not present name
P1042020 Error while Checking File System Error S Ending Date Missing Check header
from Input File of input file
P9900195 Data already present in Data already S Week (Start DT – Select Rework
output tables – Rwk present End DT) already option for Data
option required for data Present in processing
processing SHOP_PRICEST99
table. Only
REWORK Option is
allowed
P9900222 Error while Checking File Error while W Problem while If needed input
Checking checking Shop file should be
Code, Price Value modified.
and Nankey Value
P9900391 Can not create Partition Error S Problem while
creating Partition in
SHOP_PRICEST99
table

P9900187 Shop Shop Information message only. No action required
Reclassification Reclassification
NOT Required
read records re
inserted records in
skipped records sk
P9900263 SHOP_PRICEST9 Created Information message only. No action required
9 Dummy Partition: Partition
P9900248 Loading Loading data in Information message only. No action required
SHOP_PRICEST9 the specific
9 Partition: partition

Page 240 from 698
EHDV21075P- HBE – Automatic Purch. Inspection
Phase:
This program validates automatically the purchase units reported by the families.
Validation is applied on Rawda, quantity used to check if a correction should be applied is

historical quantity datatype (DT5642).
If quantity is to be corrected this job is automatically updating current quantity datatype (DT5650),
historical quantity remains as before validation.
All changes are logged in FAMILYLOGT67 table and can be displayed using FamilyLog screen.
Product group instruction
HBE validation is applied at product group level. Scope of items to be considered are
defined using the concept of IBD. The IBD are defined in IBD Definition, IBD Char Selection and
IBD Rule Definition screens.
IBD to be used by HBE process are defined by static parameter IBD-ID_1 & IBD-ID_2 (if IBD-
ID_2 is not present only IBD-ID_1 will be used).
IBD-ID_2 should be used in case a country would like to have 2 set of parameters ADLT, CHLD,
ADLTPARM, CHLDPARM, CUP for 2 different IBD. This for example to apply standard CUP
(ADLTPARM=0.7, CHLDPARM=0.5) for one IBD, and fix CUP=1 (ADLTPARM=0,
CHLDPARM=0) for another one.
Note: there should not be any overlapping between the 2 IBD Groups.
If ALT_GROUPING parameter is defined ,Characteristic values are got using ALT_GROUPING

parameter value for grouping else process works as usual by grouping at product group level.
HBE is using CUP “consumption unit parameter”
The CUP can be :

-extracted from FAMILYMONITOR if static parameter CUP_1 (CUP_2) is present.
-computed based on FAMILY characteristics using following formula:
CUP = 1 + [# of adults – 1] * ADLTPARM + [# of children] * CHLDPARM
Where for each HBE GROUPID :

# of adults is defined by static parameter ADLT_1 (ADLT_2)
# of children characteristic is defined by static parameter CHLD_1 (CHLD_2)
ADLTPARM_1 (ADLTPARM_2) and CHLDPARM_1 (CHLDPARM_2) are defined also by
static parameter
Two periods are considered in HBE process:

- Inspection period is the period(s) selected by the user during the job submission and the
number of periods. This is the Rawda period on which data are corrected.
- Benchmark period is the Rawda period used to represent the behaviour of the families
in the past weeks. Number of periods back to be considered for benchmark is defined by static
parameter PERBENCH.

Page 241 from 698
Statistical model used for validation:
Here are HBE main steps:

➢ Compute CUP(Consumption Unit Parameter) for all the families
➢ Determination of upper bounds for HH-Product Group totals
➢ Detection of heavy buyers by comparing individual (normalized) HH-PG totals with
boundaries.
➢ Once they have been identified, a “final” adjustment factor is computed to be applied to
each of the related data points (at transaction level) on inspection period:
➢ A non outlying pair (HH-PG) will have a unitary adjustment factor
➢ If there are no sales in inspection period, then we can do nothing: adjustment factor
should therefore be set to 1 as well
➢ If there are sales in current week but if they are less than the gap between boundary
and aggregated HH-PG purchases (over 13 weeks), then it is clear that gap cannot
be narrowed completely. In no case we will delete transactions (in order to keep
unchanged distributions, penetrations, loyalties and suchlike), so solution is to adjust
those sales in order to end up with unitary purchases VALUE at HH-PG level for
current week.
➢ Finally, update adjustment factors in raw data for current period.
➢ Adjustment factor applied is stored in RAWDAILYT60 table in datatype dt5099, dt5939

datatype (flag for qty correction) is updated also
In case rounding option is selected, dt5650 and dt5099 are adjusted as follows :
Final corrected quantity is rounded to integer value (and dt5099 accordingly), for all
nan_key not present in NAN_VAR_WEIGHT_FROM_IMDB table
Final corrected quantity is not rounded, for all nan_key present in
NAN_VAR_WEIGHT_FROM_IMDB table
Recommendations:
It is recommended to submit this job in sequence, period one after each other in ascending order
as HBE run on previous period can impact current period HBE run.
Constraints:
If production cycle has been locked, it is required to submit this job in Rework mode.
It’s a single run process, so to be re-submitted the job should be submitted with “Force single run”
option on.
Prerequisites:
• Availability of the data in for the submission period T in Rawda

• Family characteristics used to compute CUP available for submission cycle/period
• HBE instructions should be set-up in GUI
Table Description

Page 242 from 698
PROCKEYST29 List of RAWDA processing keys (purchase + Tbsk)
IBD_DEFINITIONT104 Table containing the Item Break Down (IBD) definition per cycle
IBD_CHARST105 Table containing the Item Break Down characteristics per cycle, IBD
IBD_RULEST106 Table containing the Item Break Down (IBD) rules per cycle, IBD
FAMILYUSABILITYT41 It contains the characteristic identifier and the characteristic value by
Family by cycle and period (YYYYPPP)
FAMILYMONITORT42 Table containing family usability details by period
CHAR_TYPEST17 List of product characteristic codes from OGRDS
CHAR_TYPES_TRANST18 Description of product characteristic codes from OGRDS
Files:
Input Files:
No input file.
Output Files :
No output file
Parameters :
Static Parameters
CUP computed based on #adults + #children:

Type
PARM IBD-ID_1 01 IBD #1 to be used
PARM IBD-ID_2 03 IBD #2 to be used, if not present only IBD #1
will be used
CHID ADLT_1 NBADL Family char for number of adults in HH for
group id #1
CHID CHLD_1 NBCHL Family char for number of children in HH for
group id #1
PARM ADLTPARM_1 1 Weight of #adults in CUP formula for group id
#1
PARM CHLDPARM_1 0.7 Weight of #children in CUP formula for group
id #1
CHID ADLT_2 NBADL Family char for number of adults in HH for
group id #2
CHID CHLD_2 NBCHL Family char for number of children in HH for
group id #2
PARM ADLTPARM_2 0 Weight of #adults in CUP formula for group id
#2
PARM CHLDPARM_2 0 Weight of #children in CUP formula for group
id #2
PARM PERBENCH 12 Number of periods of benchmark
PARM HBEINSPLEVEL 0.05 Statistical parameter

Page 243 from 698
PARM RESET Y In case RESET=Y all HBE factor are set to 1
before HBE process
PARM ROUNDING Y If ROUNDING is set to Y, rounding of nan
variable weight is applied
PARM CUP_ZERO FORBIDDEN Valid values:
ALLOWED
FORBIDDEN
MISSING
If parameter is not defined the default value is
FORBIDDEN
PARM ALT_GROUPING XXXXX Characteristic ID to use for Value
CUP already computed:

Type
PARM IBD-ID_1 01 IBD #1 to be used
PARM IBD-ID_2 03 IBD #2 to be used, if not present only IBD #1
will be used
CHID CUP_1 1 Flag to use CUP DT from
FAMILYMONITORT42
CHID CUP_2 1 Flag to use CUP DT from
FAMILYMONITORT42
PARM HBEINSPLEVEL 0.05 Statistical parameter
PARM RESET Y In case RESET=Y all HBE factors are set to 1
before HBE process
PARM ALT_GROUPING XXXXX Characteristic ID to use for Value
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job Yes
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management:
Process type CYCLE PERIOD Option


Page 244 from 698
IBD Definition, IBD Char Selection and IBD Rule Definition screens.

code
P9900225 Processing option and Processing S If production
Period Lock: NOT OK option an cycle if locked,
it is required to
submit the job
in REWORK
mode
P9900196 No Records Selected for No Records S There are no Check if
processing Selected records in Rawda has
PROCKEYST29 been loaded
table for processing for submission
cycle/period
P9900178 Cycle already closed Cycle S Processing cycle is If production
already already closed cycle if locked,
closed it is required to
submit the job
in REWORK
mode
P0020518 Period not present Period not S Period doesn’t exist Enter valid
present in period(both
CYCLEPERIODT03 dt_from, dt_to)
tables in
CYCLEPERIO
DT03 period
P1025004 Characteristic requested HHSIZE not S ac_hhold field value Enter valid
not found in Family in T41 not found in ac_hhold filed
Usability table. FAMILYUSABILITY value in
T41 table FAMILYUSABI
LITYT41 table
P9900223 No Records Selected for No Records W Records are not Check if
processing Selected selected for Rawda has
processing been loaded
for submission
cycle/period
P9900210 Hhold in Rawdata without Hhold in S Requested char are Load HH char
CUP Rawdata wit missing for for HHOLD
submission period listed for
for displayed submission
HHOLD period
P9900429 HHOLD having Cup value CUP value 0 I The hhold has been
0: identified as having
CUP value equals to
0

Page 245 from 698
P9900427 Families having Cup Cup Value 0 W This is message is
value 0 is allowed, value allowed displayed when
set to 1 by default CUP_ZERO
parameter is set to
ALLOWED and
records with CUP
value equals to 0
have been identified
P9900428 Families having Cup Cup Value 0 S This is message is
value 0 is not allowed not allo displayed when
CUP_ZERO
parameter is set to
FORBIDDEN and
records with CUP
value equals to 0
have been identified
P1043001 No Records Selected for No Records W There are no Check if
processing Selected records Selected Rawda has
for step 6 been loaded
processing for submission
cycle/period

P9900224 Processing option and Processing Information message only. No action required
Period Lock: OK option an
not closed closed
Transactions adjusted Transactio
P1049030 Number of Records Update Cnt – Information message only. No action required
updated into RAWDA
RAWDAILYT60
P9900206 Number of NANS Number of Information message only. No action required
selected NANS selec
P9900207 Number of HHOLDS Number of Information message only. No action required
selected HHOLDS sel
P9900208 Number of DataPoints Number of Information message only. No action required
selected DataPoints
P9900209 Number of Outliers Number of Information message only. No action required
selected Outliers s
P0750001 Number of # Information message only. No action required
Transactions adjusted Transactions
– IBD 1
P0750002 Number of # Information message only. No action required
Transactions adjusted Transactions
– IBD 2

Page 246 from 698
EHDV21076P- Automatic price validation on Rawda
Phase:
This program validates the prices delivered by the families.
Validation is applied on Rawda, comparing historical price datatype (DT5667) with some price
limits computed based on Standard price and MAD (median price deviation) of the current period.
If price is out of limits, this job is automatically updating current price datatype (DT5675),
historical price remains the same.
If PROMOTION_STEP parameter is set to ON, then PRICE value of records with any of the
promotion datatypes defined by the static parameter PROMO_DTxx is rescaled by the
PROMO_RESCALE factor before comparing with the usual boundaries.
Rawda date range to be validated is defined by submission period and number of periods back
defined by static parameter NUM_PER.
Validation is using the price group concept, shop characteristic to be used for price group is
defined as static parameter PG01.
If the RERUN Mode is ‘Y’ then

keys are selected for Price Validation from RAWDAILYT60. When the option is ‘Y’,
select only keys from RAWDAILYT60 where filenumber is -1 in PROCKEYST29. The
filenumber is the column DT5095 in PROCKEYST29. Basket key is obtained using same
set of AC_AUDIT,
DT_PUR_DATE, AC_TIME, AC_HHOLD, AC_SHOP and AC_BARCODE=’0’ in
PROCKEYST29.
Scope of purchase record to be validated by APV:
We have different combinations:
➔ no price group value /barcode type value instructed : all purchase acts are validated
➔ one or several “price group” code are instructed [ no barcode type instructed ] : only
purchases for shops belonging to instructed shop type are validated
➔ one or several “price group” code are instructed, for each of them a list of barcode type is
instructed, for each price group, only barcode belonging to instructed barcode type
(DT5098) are validated by APV
Price group and barcode type to be considered will be defined as static parameter (see example
bellow).
Step 1: Transform observed prices into standardized form by “pc-item / price group”

Page 247 from 698
sequence
Get sp = sp (prgr-x)
1 If sp (prgr-x) not available then take sp (prgr-0)
2 If sp (prgr-0) not available then take sp = std_price(prgr-x)* CVpc(prgr-x)
3 If Std_price(prgr-x) not available or if CVpc(prgr-x) is sp = std_price(prgr-0) CVpc(prgr-0)

not available then take
4 sp = max (MINMAD, sp )
5 sp = sp * MADADJ
6 zp=(p-pa)/sp
Where :
p Price of the purchase transaction under inspection in Rawda (dt5667)
pa standard price of the NAN_KEY of submission period stored in CNANPRICET34

I std price is not available for Price group x, we use std price for Price Group 0
sp (prgr-x) MAD of the NAN_KEY for price group x
sp (prgr-0) MAD of the NAN_KEY for price group 0
std_price(prgr-x) Std price of the NAN_KEY for price group x
std_price(prgr-0) Std price of the NAN_KEY for price group 0
CVpc(prgr-x) price-cv of the product class to which item k belongs for price group x
CVpc(prgr-0) price-cv of the product class to which item k belongs for price group 0
If LOWP_CHK_VAL1 or LOWP_CHK_VAL2 is missing nothing to do

Else depending original historical price1
if original historical price < L OWP_CHK_VAL1 t hen sp2= L OWP_STD_VAL1
If Original historical price >= LOWP_CHK_VAL1
and Original (historical) price < LOWP_CHK_VAL2 then
sp= LOWP_STD_VAL2
Else sp = max (MINMAD, s p ) * MADADJ
MAD adjustment factor (MADADJ) and minimum value of MAD (MINMAD) are defined as static
parameters.
o A new parameter ‘USE_DEF_MAD’ has been added in SETUP_PARAM

table.The accepted values for this are ‘Y’ or ’N’ or any cycle
value(Example:0004).Default value is given as ‘N’.
o lf country setup this parameter as either ‘Y’ or any cycle value ,the process will
take the MINMAD static parameter value as MAD for those items where standard
price exists but no MAD exists at Product Group level and Price group 0 level in
T34.

Page 248 from 698
Step 2: Computation corrected standardized prices zc
zc(i)=0 if zp(i) >= c

zc(i)=a [c- zp(i)]/(c-b) if b <= zp(i) < c
zc(i)=a if a <= zp(i) < b
zc(i)= zp(i) if –a < zp(i) < a
zc(i)=-a if –b < zp(i) <= -a
zc(i)= -a [c+ zp(i)]/(c-b) if –c < zp(i) <= -b
zc(i)=0 if zp(i) <= -c
a,b,c parameters are defined by static parameters AF, BF, CF
Step 3: Transform corrected standardized prices back into un-standardized prices.

Pc= zc * sp + pa
Step 4 : If the “new computed price” pc is not equal to the “original purchase price” reported by
the family (DT5667), then current price value (DT5675) of the purchase transaction is updated
with the new computed price.
Remark : PGI Nan_key are excluded from APV process (as many different physical products can
be linked to this Nan_Key)
Recommendations:
Constraints:
Prerequisites:
➢ Availability of the data in for the submission period T in Rawda

➢ Availability of statistical parameters MAD (median price deviation) for the period T
➢ Availability of standard price in T
Table Description
group, NAN_KEY
FAMILY_LOGT67 Log of changes applied by DV automatic validations processes

Page 249 from 698
Files:
Input Files:
No Input files
Output Files:
File containing all purchase acts with price=0 (File id: FILEO1) :

1 500 STR VARCHAR2
Parameters:
Static Parameters

Type
CHID PG01 Shop char to be used as price group
PARM AF Statistical parameter
PARM BF Statistical parameter
PARM CF Statistical parameter
PARM MADADJ Statistical parameter
PARM MINMAD Statistical parameter
PARM PROMOTION_STEP ON / OFF(will be initialize by DEFAULT)
When set to ON the price of records with
promotion datatypes are set to 1, their price is
rescaled by the rescaling factor
(DT5667*PROMO_RESCALE)
PARM PROMO_DTxx To set promotion datatypes list to check
PARM PROMO_RESCALE To set rescaling factor for promotion purchase
acts.
PARM RERUN Rerun mode ‘Y’ or ‘N’, by default ‘N’
PARM LOWP_CHK_VAL1 Optional Value to check against historical
price
PARM LOWP_CHK_VAL2 Optional Value to check against historical
price
PARM LOWP_STD_VAL1 Optional Value of SP to set for the
LOWP_CHK_VAL1
PARM LOWP_STD_VAL2 Optional Value of SP to set for the
LOWP_CHK_VAL2
Optional static parameters defining scope of Price Group / Barcode Type

Type Value
PG01 PV01 0000000001 Price group value 1 to consider in scope of
APV
PV01 BCT01 1 Barcode type 1 to consider in scope of APV

Page 250 from 698
for price group 1
APV
for price group 2
for price group 2
APV
In this example APV will validate:

➔ purchase acts with shop code of price group 0000000001, and barcode type (DT5098) = 1
➔ purchase acts with shop code of price group 0000000002, and barcode type (DT5098) =1 or 2
➔ purchase acts with shop code of price group 0000000003, all barcode types
Dynamic Parameters

Name (Yes/No)
# Periods No Not used for this job
Single Run Job Yes
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management:


code
P0020505 Static Value not present Stat.Val not S Static value is not Enter valid
present present in Static value
CAL_FACTORT05 into
table CAL_FACTOR
T05 table

Page 251 from 698
it is required to
submit the job
in REWORK
mode
P9900196 No Records Selected for No Records S Records are not Check if
processing Selected selected from Rawda has
cycle/period
P1021015 Shop not Found in T4 Shop not in S No Shops Present in Check content
T40 the of
SHOPUSABILITYT4 SHOPUSABILI
0 table TYT40 table
P9900235 There are Shops that they There are W Shops are Found in Check content
are not in Shops that RAWDAILYT60 but of
ShopUsabilityT40 not in available in SHOPUSABILI
SHOPUSABILITYT4 TYT40 table
0 table
P1021001 Std. Price not identified Std.Price not W Standard Price is not No std price
for the PC-Item found found for PC items available for
this NAN, so
no price
validation
P1021004 No Data in T14 No Data in S No records present
T14 in KEYST14 table
P1021009 Error in Price Validation System Error S Unable to execute Open a QLS
Ehsp_ValidatePrice
procedure
P1021008 Error in getting Standard System Error S Problem for getting Open a QLS
Price Standard Price using
Ehsp_Get_Standard
_Price procedure
P1021006 Error in Identifying System Error S Unable to identifies Open a QLS
Standard Price the Standard Price
P9900015 Divide By Zero Error Divide By S Divide By Zero Error Open a QLS
Zero Error
Arrays
P1021005 Error while Computing System Error S Problem when Open a QLS
New Price executing the
Ehsp_Compute_Ne
w_Price procedure
P1021003 Price with value 0 found Price = 0 S The value of Price
found found 0 or blank
P9900003 File Handling-Write Error File Write S Error is manipulation
Error of the file

Page 252 from 698
P1021011 Error while Writing to System Error S Unable to write the Open a QLS
Output File Output file
P1021013 Error while Updating System Error S Hhlog cannot Open a QLS
HHLOG Updating
P9900550 Low Price Calculation can System S Not all the static Review the
not performed.Missing parameters for the static
Static parameters. LOW price validation parameters for
have been defined. Low price
validation

P1021002 MAD not found for MAD not Information message only. No action required
PC-Item found
Validated
P1021016 Number of Distinct Distinct Information message only. No action required
Families Validated Family Cnt
P1021020 Number of Items No MAD – Information message only. No action required
without MAD Item count
P1021021 Number of Purchase No MAD-Pur. Information message only. No action required
acts without MAD and Act cnt
not inspected
P1021017 # of Records for which Corrected Information message only. No action required
price is corrected Record Cnt
Automatically
P9900117 Number of Warnings Warnings Information message only. No action required
Count

Page 253 from 698
EHDV21078P – Re-Xcoding by exception (based on barcode list)
Phase :
This job is re-Xcoding based on EIMDB Xcoding instruction the purchase records present in :
Output rawda (tables RAWDAILYT60 and PROCKEYST29)
FamilyLog table (table FAMILYLOGT67)
The Xcoding logic used here is the same as the one used in standard Re-classification process
(job 61P).
Only change is the way purchase records to be re-xcoded are selected. This job is providing
functionality filter manually barcodes and date ranges for which reXcoding should be applied.
If the SHOP_GROUP is defined ,purchases from PROCKEYST29 and FAMILY_LOGT67 only

for the shops belonging to the shop group is selected and then rexcoding is performed.
If the SHOP_GROUP is not defined process works as usual.
In standard reclassification process (61P):

data purchase range of date to be Re-Xcoded is fixed by submission Date_To and
Date_From,
barcodes to be re-xcoded are defined by list of barcode present in published copy (in
case MODL=PUBL) or Xcoding update link date range (in case MODL=DATE)
Whereas in current job (78P):
list of barcodes to be re-xcoded is instructed in a
(i)user input file:

purchase date range to be re-Xcoded is instructed also in same input file. The date range
can be different from barcode to barcode.
(ii)OGRDS mode:
Here the user defines the data range DATE01 & DATE02 through which the barcodes list
is fetched.
In OGRDS mode barcodes are fetched from CROSSCODING_INSTRUCTIONST49 (with
CROSS_CODE_GROUPST26) for the dates DATE01 and DATE02 and processed
further.
Also LAC_EXCLUDE parameter is introduced to add an extra filter to select all
AC_BCTYPE_MNEM different from LAC or not for this OGRDS mode.
At the end of the process TDV_REX_LOG table is filled using RAWDAILYT60 and the process
temp table. The process will insert record only when there is a nankey change and insert job
submission time as timestamp.
Recommendations :
Constrains :
Prerequisites :

Page 254 from 698
Xcoding instructions should be set-up
EIMD should be refreshed
File with barcode list and dates should be available.
Table Description
CROSS_CODE_GROUP List of Xcode Group ID 255lignme to be used for Xcoding
ST26
CROSSCODING_INSTR Xcoding instruction loaded from EIMDB
UCTIONST49
DATALOGT55 The table contains the outliers of the quality controls system candidate for the
user inspection and on line selection.
Files:
Input Files:
List of barcodes to be re-Xcoded (File id: FILEI1) :

1 35 BARCODE BARCODE left justified
36 8 DATE_FROM YYYYMMDD
44 8 DATE_TO YYYYMMDD
Remarks :
We check input file validity, job will fail in following cases :
invalid date_from or date_to
duplicate barcode information
barcodes in input file having no valid Xcoding instruction
Output Files :
Changed nan_key file

36 16 NAN OLD NUMBER
52 16 NAN NEW NUMBER
68 3 TRANSACTION NUMBER
Non Xcoded barcode file

Page 255 from 698
36 16 NAN OLD OLD NAN_KEY VALUE
Parameters :
Static Parameters
MODE LIST Optional Mode to be used FILE or OGRDS (FILE by default)
DATE01 DATE Optional Update Date from
DATE02 DATE Optional Update date to
LAC_EXCLUDE LIST Optional LAC to exclude Yes or No(No by default)
SHOP_GROUP SQL Optional Shop group id to be defined ,by default it is empty
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes
Rework Management :

Xcoding instructions are the same as the one used for Xcoding (08P) and standard Re-Xcoding
(61P).
Those instructions are defined in GUI screens :
Services → X-Coding → Bar Code-Type : set up of Xcoding instruction by Barcode Type
Services → X-Coding → Shop-Group Id : set up of Xcoding instruction by Shop code
Message Description Tag Ty Explanation Action

code pe
P0020125 File Name is blank File Name S File name is not Enter Valid file
is blank empty name
P0020126 File Name not present FileName S File name is not Enter Valid file
not present specified name

Page 256 from 698
P1003006 Error while validating records System S Validation error Check for the
in the file Error while accessing the validity of records
records
P9900242 Duplicate Barcode Instruction Duplicate S Barcode instruction Use unique
Barcode In value is repeated Barcode
Instruction Value
P9900243 No Instruction for Barcode No W Barcode instruction Entered Barcode
Instruction value is empty has no nankey
for B instruction value
greater than
zero
P9900193 Full Xcoding Required Full W Complete cross Check the cross
Xcoding coding is required coding whether
Require all the values are
crosscoded
P9900194 Nan Key=0 Found in T-359, Nan Key=0 W
transactions DELETED. Found in T
P1010006 Error while Validating System S
Instructions Error
P0010035 Oracle Error Oracle S Oracle Error Check for oracle
Error syntax
S0000109 Missing Link by Exception Missing W
Link in T49
S0000108 Multiple Links Exist in Multiple W
Xcoding Instructions table Links T49
P1008005 Purchase date is not inside Outside W
valid pc-item date range of IMDB
IMDB-Link not applied Dtrange

enough
Transactions of
Selected for Transac
Xcoding tion
Transactions of
Xcoded Transac
tio
transactions NOT Number
Xcoding
Changed Nans of
Change
d Na

Page 257 from 698
EHDV21079P Loyalty card usage extraction
Phase:
France requested to extract from Tbskt13 table information related to the usage of a Fidelity card
by Shop Chain. This extract includes:
• the families reporting loyalty card on the last N weeks (e.g N=12) .
• the families/chains being there for the previous X weeks (e.g X=24) and not present in
current extraction. For each Family/chain, write a record with value 0 in the output file
Detailed logic of validation
Main steps are

• Compute number of records from Tbskt13 with selected dt value by Shop Chain
• To write records in the output file according to the layout.
• Submit external unix script
Recommendations:
This job should be submitted after all validation jobs.
Rework:
No rework option
Constraints:
Prerequisites:
• Rawda should be loaded
Table Description
TBSKT13 Total basket table
SHOPUSABILITYT40 Shop characteristics
Files:
Input Files:

Page 258 from 698
No input file
Output Files:

1 10 AC_HHOLD Household ID
11 6 SHOP_CHAIN filled with the Shop Chain (shop characteristic value), left
aligned with leading 0
17 10 NC_NBR filled with the number of times fidelity card was used, left
aligned with leading 0; set to MAX_COUNT in case it is
more than MAX_COUNT
Parameters:
Static Parameters

Type Value
DTDT DT01 DT5949 Datatype for fidelity card
DT01 DTVL 1 Datatype value of “fidelity card used”
PARM MAX_COU 99 Maximum value for number of times fidelity
NT card was used
PARM SCRIPT_C /datafs_fr/sharedArea/cpsScr Name of shell script to be executed after job
FT ipt/script_79P.cmd completion
CHID PG01 ENS Shop char to identify the chain
PG01 CHVL01 0000000001 Shop char value 1
PG01 CHTN01 114 Shop chain in output file
PG01 CHVL02 0000000002 Shop char value 2
PG01 CHTN02 111 Shop chain in output file
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 259 from 698
Message
Description Tag Type Explanation Comments
code
File Name is Define valid file
P0020125 File Name is blank S File Name is blank
blank name
File Name not File Name not
P0020126 File Name not present S Define a file name
present present
Static Value not Static Value Static Value not Define static
P0020505 S
present not present present parameter
Shop Not present
Shop Not present in Shop Not No shop(s)
P9900177 S in Shopusability
Shopusability table present in available
table
Define valid
DT column not
Given dt_column not Given DTDT dt_column in
P9900628 S present in
found in TBSKT13 not found DTDT static
TBSKT13
parameter
Message code Description Tag comments
Periods not
Periods not enough
P9900005 Required No. of Periods not Present enough
Get date range for submitted
P0000000 Submission period date range Date range period
P0020608 PG01 Static Value Static parameter PG01
P0020608 CHVLXX Static Value Static parameter CHVLXX
P0020608 DT01 Static Value Static parameter DT01
P0020608 DTVL Static Value Static parameter DTVL
P0020608 MAX_COUNT Static Value Static parameter MAX_COUNT
P0020608 PG01 Static Value Static parameter PG01
P0020608 SCRIPT_CFT Static Value Static parameter SCRIPT_CFT
P0020608 CHTNXX Static Value Static parameter CHTNXX

Page 260 from 698
Number of shops selected for Number of Number of shops selected for
P9900181 period shops sele period
Total Number of records extracted Total Number of records
P1043007 into the o/p file Record Count extracted into the o/p file
List of messages for External Job submission:
Message code Description Type Tag Comments
Automation job
submitted with External Job Information message only, No action
P9900394 I
the following name required
name
Automation job
External job Information message only, No action
P9900393 ended I
succeeded required
successfully
Additional information is given within
the message. In case there is no
Automation job External Job additional message then the following
P9900392 S
failed Failed message will be displayed: No
Additional external job information
available
Additional information is given within
the message. In case there is no
Automation job External Job additional message then the following
P9900395 S
failed Failed message will be displayed: No
Additional external job information
available

Page 261 from 698
EHDV21082P – Extraction from Table ALL (ALLT11)
Phase :
This program is an utility to extract information from Xcoding table (ALLT11) to an output file.
The job can run using different options :

➢ extract one or more periods
➢ extract one or more datatype
➢ extract data for some selected PC only or for all PC
➢ extract data by F# selection
Datatype selection :
Datatypes to be extracted are defined when submitting the job in DT Selection screen. In case no
DT is selected all DT will be extracted.
PC selection :
PC to be extracted are defined when submitting the job in PC Selection screen.
If no PC is selected all PC will be extracted.
F# selection :
When we choose the selection by F#, we will extract from the rawda all the basket data (trip data
+ purchase data) associated to all the baskets loaded in the same file identified by the selected
file number.
Only one file number can be selected, file number 0 cannot be used.
Filtering by shop/hhold/ product
Logic here is similar to the one used in Rawda extraction (see 41P description for details and
examples).
82P:
Note : For python version Hive data refresh is done at the beginning of this process for the
submission audit, period (back-periods included) for the following table:
• ALLT11
Recommendations :
Constraints :
Prerequisites :
ALLT11 should be loaded (01P).

Page 262 from 698
Table Description
Files:
Input Files:

IDENTIFIER
Main


IDENTIFIER
Main


IDENTIFIER end
Main

CODE header)

Page 263 from 698
Output Files :
Rawda extraction by DT (File id: FILEO1) :

1 4 AUDIT Xxxx
5 4 YEAR YYYY
9 2 MONTH MM
11 2 DAY DD
13 4 TIME HHMM
27 10 SHOP
67 16 NAN_KEY
83 4 DATATYPE Xxxx
CODE
VALUE
Parameters :
Static Parameters

Type Value
DTDT DTxxxx yyyy Where xx can be any number and yyyy is the
datatype number.
Example: DTDT (type)

DT5650 (type value)
5650 (name)
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection Yes
F# Selection Yes
DTSelection No
File Selection Yes
Rework Management :


Page 264 from 698

code
Header Tag too long in Header Tag Check input file
P9900260 file: too long S
Error while Dropping Open a QLS
P1038001 Tables System Error S
Error while Dropping Open a QLS
P0020628 Tables System Error S
Check
No Records Selected for No Records submission
P9900196 processing Selected S parameters
P1041002 Error in Check DataCodes System Error S Generic error while open a QLS
checking datatypes
P1041003 Error in Count Destination System Error S Generic error while open a QLS
checking datatypes
destination
P0020505 No DTs have been System Error S No datatypes have Define DTDT
defined as static been defined as static
parameters static parameters parameter(s)

Selection by
BCD/NAN not applied,
P9900253 header not correct
Selection by Selection by
P9900256 BCD/NAN applied BCD/NAN
Number of distinct # distinct
P9900259 BCD/NAN selected BCD/NAN
Selection by SHOP
not applied, header
P9900252 not correct
Selection by SHOP Selection by
P9900255 applied SHOP a
P9900258 SHOP selected SHOP
Selection by HHOLD
not applied, header
P9900251 not correct
Selection by HHOLD Selection by
P9900254 applied HHOLD a

Page 265 from 698
P9900257 HHOLD selected HHOLD
Total number of
records written in to Record
P1039008 the file count
Total number of TBSK TBSK
P1041011 records Extracted Record Cnt
Total number of
Rawda records Rawda
P1041010 Extracted Record Cnt

Page 266 from 698
EHDV21083P – Extract Family Monitor (Italy)
Phase:
This job is extracting from DV a set of information related to weekly family usability & family
collaboration. Information is written to an output flat file.
➢ Logic to build output file
Lit of HH:
As reference we use list of families present in FAMILYUSABILITY table for submission
AUDIT/ PERIOD
DT computation
USABILITY
For all those HH, job is selecting DT5900 from T42
FORCED USABILITY
For all selected HH, job is checking if there is at least one record present in T67 for
datacode=DT5900, datavalue=1, proc_id is null and date in date range and audit is submission
audit
If at least one record is found with this logic value is set to 1, if not value is set to 0.
NO PURCHASE DECLARED
datacode=DT5684, datavalue=1, date in date range and audit is submission audit
TRANSMISSION DONE
datacode=DT5645, datavalue=1, date in date range and audit is submission audit
TRANSMISSION WITH PURCHASE DONE

datacode=DT5646 datavalue=1, date in date range and audit is submission audit
HOLIDAY IN THE WEEK

datacode=DT5643 datavalue=1, date in date range and audit is submission audit
TOTAL BASKET SPENT (DECLARED)

For all selected HH, job is computing sum of DT5941 in TBSKT13 for date in date range
and audit is submission audit
TOTAL BASKET SPENT (COMPUTED)

For all selected HH, job is computing sum of DT5755 in TBSKT13 for date in date range
and audit is submission audit

Page 267 from 698
➢ Multi period Logic
In case static parameter NUM_PER is greater than one. Data are extracted by period.
The output file is sorted by HH ascending and period descending, see example bellow
HH x 2008040
HH x 2008039
HH x 2008038
HH x 2008037
HH y 2008040
HH y 2008039
HH y 2008038
HH y 2008037
Recommendations:
This job should be submitted after all other validations.
Rework:
Extraction job, Rework has no impact.
Constraints:
Prerequisites:
➢ Family char should be loaded

➢ Family log should be loaded
➢ Weekly family usability should be computed
Table Description
FAMILY_LOGT67 Table containing all information related to HH collaboration at daily level
Files:
Input Files:
No input file.
Output Files:
Extracted file (FILEO1)

Page 268 from 698
1 10 PANELIST NUMBER
CODE
11 7 PERIOD Yyyywww
18 1 USABILITY DT5900 from T52
19 1 FORCED to be extracted from T67
USABILITY
20 1 NO PURCHASE DT5684 from T67
DECLARED
21 1 TRANSMISSION DT5645 from T67
DONE
22 1 TRANSMISSION DT5646 from T67
WITH
PURCHASE
DONE
23 1 HOLIDAY IN DT5643 from T67
THE WEEK
24 16 TOTAL BASKET sum of DT5941 in T13 by period/HH
SPENT
(DECLARED)
40 16 TOTAL BASKET sum of DT5755 in T13 by period/HH
SPENT
(COMPUTED)
Parameters:
Static Parameters

Type Value
PARM SCRIPT_C Check Appendix for External shell to be submitted
FT information
PARM NUM_PER Number of periods
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 269 from 698

code
P0020505 Number of periods: Stat.Val not S NUM_PER static Define
present parameter has not NUM_PER
been defined static
parameter
P0020624 Error in retrieving Period System Error S
List
P9900274 No HHOLDs selected No HHOLDs S No records selected
from FAMILUSABILITY41 selected from
FAMILYUSABILITY
T41 table

date range
P0020608 Number of periods: Static Value Number of periods defined for static
parameter NUM_PER
P9900123 Total Number of Record Count # RAWDA Records in input file: Information
the input file

code
following name:
ended JOB required
displayed:

Page 270 from 698
displayed:

Page 271 from 698
EHDV21084P – Variable weight volume validation
Phase:
Objective of this procedure is to validate units for variable weight items and apply automatic
correction.
1 For fixed weight barcode products, 272anellist272 are provide units as integer,
value can be in following range: [1,2,3…]
1 For variable weight products situation is different as 272anellist272 are required

to provide a volume (in litre or kilogram), value can be decimal in following
range of value [0.001 , 99.999].
Due to this difference, variable weight products are requiring a dedicated validation logic to
correct keyboarding error from 272anellist.
This validation routine is detecting both outliers for low quantity & high quantity and applies an
automatic correction.
- Low quantity are detected using a fixed minimum volume instructed in IMDB at Nan_key
level (minimum volume is stored in char value description)
- High quantity are detected using a dynamic maximum volume limit computed by the
program on a logic closed to the one used in Purchase Inspection job (16P). This
validation is applied only if there is more than T observations at Nan_Key level
- Corrected value is computed based on a standard volume computed at Nan_key level.
If there is less than T observations, above high volume validation is not applied, but an additional
validation can be applied (optional, based on static parameter). This additional validation is
detecting high quantities using a fixed maximum volume instructed in IMDB at Nan_key level
(minimum volume is stored in char value description).
Prices for variable weight are validated separately by standard APV routine (76P)
The period range for purchase data to be validated is defined using submission period & static
parameter NUM_PERIOD1 (number of period in submission screen is not used in this job).
IF job is submitted in REWORK mode then Reset_Quantities
Reset_Quantities: Revert back changes made by previous run of the job
 It checks for presence of records for the scope of Variable Weight items in the
FAMILY_LOGT67 table. If no records found then the job fails with the severe error ‘Initial
values not found in FAMILY_LOGT67’.
 Otherwise the DT5650 and DT5942 values are retrieved from the FAMILY_LOGT67 for
the scope of Variable Weight items.
 Delete FAMILY_LOGT67 records.

Page 272 from 698
 Update DT5939=0
STEP 1 : Select all Nan_key variable weight present in Rawda for submission period
List of variable weight nan is retrieved from table NAN_VAR_WEIGHT_FROM_IMDB
STEP 2 : Select minimum volume value by Nan_key from EIMDB
Nan_key minimum volume is retrieved from NAN_CHAR_VALUEST75 ( in IMDB char

value is instructed in g or ml, whereas in DV rawda volume are in kg or l, so retrieved value is
divided by 1000)
STEP 3: Compute maximum volume limit by Nan_key
1. Select Nan_key with at least T observations.

For each selected Nan_key,
select the max(T, CEIL(number of observations x RATIO)) which have the largest
volumes x
Example: T=5 and RATIO=0.1

if number of observation = 20
5 higher volumes will be selected
If number of observation = 150
14. Determine the critical limit as

max_volume = [MIN_VAL + (MEDIAN_VALUE – MIN_VAL) * Log (ALPHA) /
Log (0.5)]
Where MIN_VAL = Minimum(x) volume of L observations selected

MEDIAN_VALUE = Median(x) volume of L observations selected
STEP4: Compute standard volume by Nan_key
For each Nan_key compute:
std_volume ( t ) =
w * median_volume[sub_per + 1 - NUM_PER1,sub_per ] + (1-w ) * median_volume [sub_per –
NUM_PER1 – NUM_PER2, sub_per – NUM_PER1 ]
➔w=n/(n+k)
n is the number of observations of period range [sub_per + 1 – NUM_PER1,sub_per ]
k is a parameter given to the program by the static parameter table
➔NUM_PER1 and NUM_PER2 are static parameters.

Page 273 from 698
For example, consider a case where job is submitted for period 2007034, NUM_PER1 =1
NUM_PER2 =4
[sub_per + 1 – NUM_PER1,sub_per ] is date range with

start date = date from of period 2007034
end date = date to of period 2007034
[sub_per + 1 - NUM_PER1 – NUM_PER2, sub_per – NUM_PER1 ] is date range with

start date = date from of period 2007034 – 4 = date from of 2007030
end date = date to of period before 2007034 = date to of 2007033
➔ Exception is that if median_volume [sub_per + 1 - NUM_PER1 – NUM_PER2, sub_per –

NUM_PER1] is null
we compute std_volume ( t ) = median_volume[sub_per + 1 - NUM_PER1,sub_per ]
STEP5: Check volume outliers
Select all purchase act with DT5642 < min_volume or DT5642 > max_volume
Compute low value correction ratio at Nan_level
Select all Nan_key with:

total number of records in gt_table2 for Nan X > MIN_OBS1 (static parameter)
and with
(number of records with qty < min_volume for Nan X) / (total number of records in
gt_table2 for Nan X) > MAX_LCOR (static parameter)
Rem : For 5% max of correction MAX_LCOR should be 0.05
For all selected Nan : raise an warning (CC 08 )
STEP6: Apply correction on outliers
Update both 5642 & 5650 with corrected value
➢ If 5642 < min_volume
Corrected_quantity=
( std _ volume − min_ volume)

std _ volume − ( * volume _ before _ validation 2 )
min_ volume 2
➢ If 5642 > max_volume
std _ volume
Corrected_quantity=
max_ volume * (max_ volume − std _ volume)
1−
volume _ before _ validation 2
All updates are stored in Family_logT67 table

Update Qty correction flag (DT5939) = 10

Page 274 from 698
STEP7: Validation of high volumes using fixed IMDB high volume limit from IMDB, for nan_key
without enough observations. This optional step is applied only if static parameter
CHAR_MAX_VOL is instructed
We select here all nan_key not corrected due to volume too low, and with max_volume null (due
to not enough information)
For each record with Volume > imdb_max_volume
If imdb_min_volume < Volume / 10 < imdb_max_volume

Then corrected volume= Volume / 10
else
if imdb_min_volume < Volume / 100 < imdb_max_volume
else
else
corrected volume= (imdb_min_volume + imdb_max_volume) /2

Recommendations:
This job should be submitted before HBE
Rework:
Rework option available
Constraints:
Prerequisites:
- Rawda should be loaded

- Minimum volume should be instructed in IMDB ( for 0.08 kg , should be instructed 80)
1 IMDB for minimum volume
Table Description
NAN_CHAR_VALUET75 Item characteristics values id extracted from EIMDB
NAN_VAR_WEIGHT_FROM_IMDB Items with Variable Weights
CHAR_VALUES_TRANST25 Description of product characteristic codes from EIMDB

CNANPRICET34 Contains standard prices and MAD by cycle, period, price group,
product group, NAN_KEY

Page 275 from 698
Files:
Input Files:
No output file
Output Files:
No output file
Parameters:
Static Parameters

Type Value
PARM CHAR_MI 1111 EIMDB char code for minimum volume (by
N_VOL Nan_key)
PARM T 7 Minimum number of observation to compute
max_volume
PARM RATIO 0.1 Define size of right tail to consider in
max_volume computation in case of high
number of observation
PARM ALPHA 0.5 Statistical parameter to compute max_volume
PARM K 10 Statistical parameter to weight contribution of
median volume of previous periods
PARM NUM_PER 1 Number of previous period to consider to
IOD1 compute median volume (current period)
PARM NUM_PER 4 Number of previous period to consider to
IOD2 compute median volume (previous period)
PARM MAX_LCO 0.05 Limit for ratio of variable weight purchase acts
R corrected due to low volume (at Nan level)
PARM MIN_OBS1 10 Minimum observation (at Nan level) to apply
check of ratio of variable weight purchase
acts corrected due to low volume
PARM CHAR_MA 2222 EIMDB char code for maximum volume (by
X_VOL Nan_key), optional, if not present step 7 is not
applied
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job Yes
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

Page 276 from 698

code
P9900277 Not enough records for Not enough W
NAN, max volume records f
validation not applied
P9900278 Correction Ratio higher Correction W % of records Verify the
than MAX_LCOR for NAN Ratio higher corrected for volume minimum vol
than lower than min_vol instructed in
is 277lignm than EIMDB
limit for given NAN
P9900283 Initial values not found in S No records are Submit the job
FAMILY_LOGT67 found for variable in NORMAL
weight items in mode
FAMILY_LOGT67
for this job, audit,
period.

P9900276 Number of variable Number Number of distinct variable weight items present in
weight items of prockeys table for selected period range
present in prockeys variable
w
P1038020 Number of variable Purchas Number of distinct variable weight purchase acts
weight purchase e act records present in prockeys table for selected
records present in count period range
rawdata
P9900279 Number of Records Number Number of purchase records corrected due to
with qty < of volume lower than minimum volume instructed in
min_volume Records IMDB
wi
with qty > of volume higher than computed max volume
max_volume Records
wi
P1049047 #Records selected Number of records found for variable weight in
from FAMILY_LOGT67 for this job, audit, period.
FAMILY_LOGT67:

Page 277 from 698
P1045010 #Records in Number of purchase acts in RAWDAILYT60 where
RAWDAILYT60 the values of DT5650 and DT5642 have been reset
where DT5650 and to initial values.
DT5642 reset to
initial values:
P1047008 #Records deleted Number of records deleted from FAMILY_LOGT67
from
FAMILY_LOGT67:

Page 278 from 698
EHDV21085P- HCE – Heavy Contributor Edit
Phase:
This program validates automatically the expended purchase units reported by the families. HCE
uses so expansion factor to detect outlier.
Validation is applied on Rawda, quantity used to check if a correction should be applied is

historical quantity datatype (DT5642).
If quantity is to be corrected this job is automatically updating current quantity datatype (DT5650),
historical quantity remains as before validation.
Product group instruction
HCE validation is applied at product group level. Scope of items to be considered are
defined using the concept of IBD. The IBD are defined in IBD Definition, IBD Char Selection and
IBD Rule Definition screens.
IBD to be used by HCE process is defined by static parameter IBD-ID.
Expansion factor
HCE considers a weight value for each HH. This value is retrieved from
FAMILYMONITORT42 for submission period, DT is defined by static parameter. Weight value
can be extracted from Sample DB by dedicated program and loaded in DV using Load Family
Monitor (EHDV21035P). Datatype code to store expansion factor in FAMILYMONITORT42 is
5909 (Expansion factor x HBE)
Two periods are considered in HCE process:

- Inspection period is the period(s) selected by the user during the job submission and the
number of periods. This is the Rawda period on which data are corrected.
- Benchmark period is the Rawda period used to represent the behaviour of the families
in the past weeks. Number of periods back to be considered for benchmark is defined by static
parameter PERBENCH.
E Instruction
Here are HCE main steps:

➢ Get Xf
➢ Determination of upper bounds for HH-Product Group totals
➢ Detection of heavy buyers by comparing individual (normalized) HH-PG totals with
boundaries.
➢ Once they have been identified, a “final” adjustment factor is computed to be applied to
each of the related data points (at transaction level) on inspection period:
➢ A non outlying pair (HH-PG) will have a unitary adjustment factor

Page 279 from 698
➢ If there are no sales in inspection period, then we can do nothing: adjustment factor
should therefore be set to 1 as well
➢ If there are sales in current week but if they are less than the gap between boundary
and aggregated HH-PG purchases (over 13 weeks), then it is clear that gap cannot
be narrowed completely. In no case we will delete transactions (in order to keep
unchanged distributions, penetrations, loyalties and suchlike), so solution is to adjust
those sales in order to end up with unitary purchases VALUE at HH-PG level for
current week.
➢ Finally, update adjustment factors in raw data for current period.
➢ Adjustment factor applied is stored in RAWDAILYT60 table in datatype dt5089,
➢ Flag for qty correction dt5939 is updated also
Final quantity is updated with following logic

DT5650 = Minimum(DT5089,DT5099) x DT5642
In case rounding option is selected, dt5650 and dt5089 are adjusted as follows :
Final corrected quantity is rounded to integer value (and dt5089 accordingly), for all
nan_key not present in NAN_VAR_WEIGHT_FROM_IMDB table
Final corrected quantity is not rounded, for all nan_key present in
NAN_VAR_WEIGHT_FROM_IMDB table
Note : For python version Hive data refresh is done at the beginning for these tables:
1.Prockeyst29
2.Rawdailyt60
For oracle data refresh is done at the end of this process in terminateJob for these tables:
1. Rawdailyt60
Recommendations:
It is recommended to submit this job in sequence, period one after each other in ascending order
as HCE run on previous period can impact current period HCE run.
Constraints:
It’s a single run process, so to be re-submitted the job should be submitted with “Force single run”
option on.
Prerequisites:
• Availability of the data in for the submission period T in Rawda

• Family characteristics used to compute CUP available for submission cycle/period
• HCE instructions should be set-up in GUI
• HBE should have been submitted already for the audit/period

Page 280 from 698
Table Description
PROCKEYST29 List of RAWDA processing keys (purchase + Tbsk)
IBD_CHARST105 Table containing the Item Break Down characteristics per cycle, IBD
FAMILYMONITORT42 Table containing list of families with family usability flag & Xf value by
cycle/period
Files:
Input Files:
No input file.
Output Files :
No output file
Parameters :
Static Parameters

Type
PARM IBD-ID 04 HBE group id to be used ( one IBD group can
be shared between HBE & HCE)
DTDT IDDT01 1111 DT for Xf value in FamilymonitorT42
PARM HCEINSPLEVEL 0.05 Statistical parameter
PARM RESET Y In case RESET=Y all HCE factors are set to 1
before HCE process

Page 281 from 698
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job Yes
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management:

IBD Definition, IBD Char Selection and IBD Rule Definition screens.

Page 282 from 698

code
it is required to
submit the job
in REWORK
mode
P9900196 No Records Selected for No Records S There are no Check if
processing Selected records in Rawda has
cycle/period
P9900178 Cycle already closed Cycle S Processing cycle is If production
already already closed cycle if locked,
closed it is required to
submit the job
in REWORK
mode
P0020518 Period not present Period not S Period doesn’t exist Enter valid
present in period(both
CYCLEPERIODT03 dt_from, dt_to)
tables in
CYCLEPERIO
DT03 period
P1025004 Characteristic requested HHSIZE not S ac_hhold field value Enter valid
not found in Family in T41 not found in ac_hhold filed
Usability table. FAMILYUSABILITY value in
T41 table FAMILYUSABI
LITYT41 table
P9900223 No Records Selected for No Records W Records are not Check if
processing Selected selected for Rawda has
processing been loaded
for submission
cycle/period
P9900210 Hhold in Rawdata without Hhold in S Requested char are Load HH char
CUP Rawdata wit missing for for HHOLD
submission period listed for
for displayed submission
HHOLD period

not closed closed

Page 283 from 698
Transactions adjusted Transactio
P1049030 Number of Records Update Cnt – Information message only. No action required
updated into RAWDA
RAWDAILYT60
P9900206 Number of NANS Number of Information message only. No action required
selected NANS selec
P9900207 Number of HHOLDS Number of Information message only. No action required
selected HHOLDS sel
P9900208 Number of DataPoints Number of Information message only. No action required
selected DataPoints
P9900209 Number of Outliers Number of Information message only. No action required
selected Outliers s

Page 284 from 698
EHDV21086P- Copy Median Price CLUSTER HH Geo /SH Chain
Phase:
This program copies the prices of retailer organized by “CLUSTER” into table P359_1T12.
Cluster price computation is computing a median price by shop Geo / shop Chain based on
prices coming from RMS. These prices are stored in table CLUSTER_PRICEST100.
So for example we have in this table a median price for Nan_Key 1 for Chain= Auchan &
Geo=Madrid Area
Countries that have implemented individual shop codes can then us job 56P (copy Cluster price)
to impute this price if missing for all cps purchase acts with shop code having Chain= Auchan &
Geo=Madrid Area
Such process can not be used for countries using generic shop code, in such case it is possible
to retrieve the chain from the shop but not the Geo. Proposed solution here is to retrieve the Geo
from 285anellist characteristic (assuming that 285anellist do shopping in are where they live).
The assumptions of this job are

• Table of the CLUSTER prices will contain historical data for 4 months
The job can manage imputation for several sources with specific selection of datatype for each
source.
Model :
Shop selection
Selection of shops to be included in this process using price_group char and char values
Cluster resolution
Assignment of purchase act to cells (GEO/CHAIN)
Geo is a family characteristic
chain is a shop characteristics.
Selection of all price=0 in P359_2T12
Update of price=0 using following logic
STEP 1: GEO/CHAIN CURRENT PERIOD
Look for cluster price for current period, for HH Geo, Sh Chain
If no price found go to next step
STEP 1bis: GEO/CHAIN PREVIOUS PERIODS

Page 285 from 698
Look for the latest price among last NUM_PER_CLS period (NUM_PER_CLS is a static
parameter) for HH Geo, Sh Chain
STEP 2: CHAIN CURRENT PERIOD
Look for cluster price for current period, for Sh Chain
STEP 2bis : CHAIN PREVIOUS PERIODS
Look for the latest price among last NUM_PER_CLS period (NUM_PER_CLS is a static
parameter) for Sh Chain
If no price found go to next purchase act.
Rem : All purchase acts for which price has been imputed are flagged :
14[SOURCE]: price copied from cluster Chain/Geo current period, price from source [SOURCE]
15[SOURCE]: price copied from cluster Chain/Geo previous per, price from source [SOURCE]
16[SOURCE]: price copied from cluster Chain/0 current period, price from source [SOURCE]
17[SOURCE]: price copied from cluster Chain/0 previous periods, price from source [SOURCE]
Recommendations:
This job can be set-up as a successor of NRSP price copy job (53P)
Constraints:
Prerequisites:

• File to be delivered by NRSP
• Median price by chain and geo computed (55P)
• Family characteristic for Geo available
• Shop characteristic for Chain Geo available
Table Description

Page 286 from 698
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters:
Static Parameters

Type
PG01 CHVLxx XXXXXXXXXX Identify the characteristic values for the PRICE
GROUP
CHID HH_GEO HH_GEO Identify the family characteristic ID for the
GEOGRAPHY
CHID CHAN CHAIN Identify the characteristic ID for the CHAIN
PARM NUM_PER_CLS 12 Number of periods back for steps 1bis and 2bis
PARM SOURCE1 01 Identify Source of prices #1
PARM S1_DT1_CODE 5662 Datatype code #1 for Source #1
PARM S1_DT1_VALUE 1 Datatype value #1 for Source #1
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 287 from 698
Rework Management:


Page 288 from 698

code
T40
Missing

P9900005 Required No. of Periods not enough Information message only. No action
Periods not Present required
P0020608 Static Value Static Value Information message only. No action
required
S0000006 Record updated Record updated Information message only. No action
successfully succe required
P0860001 # records imputed in Total No of Records Information message only. No action

all STEPS for all updated required
SOURCE(S)

Page 289 from 698
EHDV21088P – Cross Coding based on barcodes list
Phase :
This program is adding the NAN_KEY, from IMDB, to the original barcode delivered from
the household for a given list of barcodes.
When
MODE=FILE is selected, the list of Barcodes to be selected by the job are defined in an
input file,.For each barcode a date_from / date_to for reclassification is defined.
MODE=OGRDS is selected, the list of Barcodes to be selected by the job are selected
from the CROSSCODING_INSTRUCTIONST49 and CROSS_CODE_GROUPST26
tables where crosscoding instruction updated dates are between DATE01 and DATE02 .
If LAC_EXCLUSION=YES, then all LAC Nankeys are not taken
into account (filter to apply is AC_BCTYPE_MNEM <> ‘LAC’)
For the following steps, when retrieving items from ALLT11, if SHOP_GROUP is
present, select only purchases where shop is present in the shopgroup for purchase dates
between submitted period date range and the #Periods back.
If BASKET_CHECK=ON,select keys from ALLT11 based on the barcodes and

validity dates selected in the previous steps which are NOT present in
PROCKEYST29 ( these keys are not taken into account)
If BASKET_CHECK=OFF,select keys from ALLT11 based on the barcodes and
validity dates selected in the previous steps.
If SIMULATION=YES, Xcoding changes are NOT applied on ALLT11
If SIMULATION=NO, Xcoding changes are applied on ALLT11
The logics used to apply Xcoding are the same as the one in place in job 08P (Xcoding):
o Standard Xcoding ( same link for all shops )
o Xcoding by exception on shop (customized links by chain or shops)
o Xcoding by exception on Barcode Type
Recommendations :
This job is not supposed to be used in ongoing regular process flow. It should be used
instead to apply Xcoding on a given list of barcodes on a large period range (eg apply Xcoding on
10 barcodes on 2 years in ALLT11).
Constraints :
Prerequisites :
The OGRDS should be refreshed in advance (job Load OGRDS).

Services → X-Coding screens should be set-up.

Page 290 from 698
Table Description
During the load phase, all the single fields of the record are
separated
CROSS_CODE_GROUPST26 List of Xcodes group to be used for Xcoding (Xcode Group
screen)
NAN_ITEMST73 List of Nan_keys extracted from OGRDS
FILE_LOGT59 List of Files # loaded in DV (can be access in screen Services →
File log)
MODULEST31 List of Modules extracted from OGRDS
PRODUCT_GROUPST45 List of Product group
Files:
Input Files:
List of barcodes to be re-Xcoded (File id: FILEI1) :

36 8 DATE_FROM YYYYMMDD
44 8 DATE_TO YYYYMMDD
Output Files :
List of non Xcoded purchase acts (File id: FILEO1) :

FILE01 contains all purchases non-Xcoded including the list of nankeys equal to 0

1 4 AUDIT Xxxx
5 8 DAY Yyyymmdd
13 4 TIME Hhmm
27 10 SHOP 0000000000
37 4 SCAN PC 0000
41 35 BARCODE 0000000000…..
76 8 Error code
84 20 Error tag

Page 291 from 698
Keys to Crosscode: FILEO2 in CSV format
Position Name
1 Audit
2 Date
3 Time
4 Hhold
5 Shop
6 Barcode
7 Nankey
This file contains the crosscoded keys updated in ALLT11 when SIMULATION=NO else
when SIMULATION=YES, all candidates keys (Audit, Date, Time,Hhold,Shop,Barcode
and new candidate NANKEY).FILE02 contains only purchases with nankey greater than 0.
Summary by EAN Nankey: FILEO3 in CSV format
Position Name
1 Barcode
2 NanKey
3 # Records
4 Total Volume
5 First Date
6 Last Date
This file contains the number of distinct key records and the Total Volume by barcode,
nankey. The Total volume is the sum of the DATAVALUE when DATACODE= DT5650. First
and last dates refer to the min and max dates of purchases of the particular barcode/nankey.
Summary by ProductGroup Module: FILEO4 in CSV format
Position Name
1 Product Group (ID + Descr)
2 Module
3 # Records
4 Total Volume
5 First Date
6 Last Date

Page 292 from 698
This file contains contains the number of records and the Total Volume by Product Group,
Module The Total volume is the sum of the DATAVALUE when DATACODE= DT5650. First
and last dates refers to min and max dates of purchases of the particular Product Group.
List of barcodes impacted (File ID = FILEO5)

Barcode would generate file only with OGRDS mode with the layout compliant with 135P Data
Selection

1 35 Barcode Barcode
2 7 Price Price is left empty
Parameters :
Static Parameters
Parameter Type Value Mode Description

Type
LIST MODE Mandatory Mode FILE or OGRDS
DATE DATE01 Optional Update date from (Only valid when
MODE=OGRDS and it becomes Mandatory )
DATE DATE02 Optional Update date to (Only valid when
MODE=OGRDS and it becomes Mandatory )
LIST SIMULATION Mandatory Simulation mode ‘YES’ or ‘NO’
LIST BASKET_CHECK Mandatory ‘ON’ or ‘OFF’. When ON If keys present in
ALLT11 and not in PROCKEYST29, these keys
are not taken into account.
LIST LAC_EXCLUSION Optional ‘YES’ or ‘NO’
LAC Nankeys are to be excluded or NOT
(Only valid when MODE=OGRDS and it
becomes Mandatory )
NUMBER SHOP_GROUP Optional ShopGroup ID. This should exist for the
submitted Cycle / Audit
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection Yes

Page 293 from 698
Rework Management :

Xcoding instructions are the same as the one used for Xcoding (08P).

code
cycle/date_range
parameter are
defined for
submission
cycle
not present name
barcode type and
then an error
message is issued
P1008018 Error while updating File System S
Log Error
P9900047 Invalid Date Invalid Date S Date format in input Check input file
file is not valid
P9900242 Duplicate Barcode Duplicate S Check input file
Instruction Barcode In

Page 294 from 698
enough
P0020608 Static Value Static Information regarding Static Value defined
Value
P9900290 Total number of Barcode Information message only. No action required
distinct barcodes s NOT
NOT Xcoded Xcoded
distinct barcodes s
Xcoded Xcoded
transaction NOT Number
Xcoding
Purchase acts e acts
linked during cross linked
coding
distinct barcodes in s Input
input file File

Page 295 from 698
EHDV21089P – Intended User Alignment
Phase :
Intended user data
➢ Intended user purchase data are loaded in DV into a specific audit,

➢ Data should be loaded with standard loader job.
➢ Intended User data have 296lignm follow these rules:
➢ The AC_HHOLD for Intended User is containing an Individual Code (different from the
CPS Family Code)?
➢ AC_SHOP is same shop code as for main CPS panel data.
➢ quantity=1
➢ price=1.
➢ Intended user have two specific datatypes:
→ one DT for the Familycode (DT5086, family code for intended user)
→ one DT representing the percentage of the product assigned to each intended user,
w% - proportional distribution (DT5087, % units distribution for intended user)
After Standard Loader, following processes will be applied:
➢ Check Shop code (same shop code as for main audit, at least one shop char should be
present in T40 for the intended user cycle)
➢ Check panelist code (panelist code in intended user are individual codes different from
the main CPS panelist codes)
➢ Xcoding
➢ Data Selection,
➢ No validation applied,
➢ Load to RawdailyT60 table.
Then on Rawda, Intended User data have to be validated as follow by 89P job:
➢ Any reclassification applied on the basket (Private Brand, basket Selection) in main
audit should be replicated to Intended User data
➢ price for purchase act should be retrieved, same price as price imputed on main
audit,
➢ quantity should be imputed based on main audit qty & w% (percentage of the product
assigned to each intended user), rounding should be applied if option is selected
(dedicated static parameter)
The job is also getting the list of basket keys present in Intended User audit. Then it will check in
tbskt table and family log in main audit, tbskt records are copied in intended user audit (DT values
will be the same in both audits).
Before applying this, the job is deleting all tbskt records in intended user audit for submission
period.

Page 296 from 698
Recommendations:
➢ 89P should be submitted after all validation on main CPS household cycle are completed
Constraints:
Availability of shop code for Intended user cycle in T40
Availability of Intended User individual codes for Intended user cycle in T41
Tbskt13
Prockeyst29
Rawdailyt60

Page 297 from 698
Prerequisites:
Rawda should be loaded for Intended User cycle for period P

Rawda should be loaded for main CPS Household cycle for period P
Table Description
Files:
Input Files:
No input file.
Output Files:
No output file
Parameters:
Static Parameters

Type
PARM FAM_AUDIT 3033 audit code that should be used a reference for
family data
PARM QTY_ROUN Y Define if rounding should be applied when
DING assigning individual qty
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No

Page 298 from 698
DT Selection No
File Selection No
Rework Management:

Type

Page 299 from 698
EHDV21090P – Total Spent validation
Phase :
In most countries Nielsen panellists are reporting for each trip the total spent value.
This information is stored in DT5941 in TBSKT13 table.
Trip data validation process (24P) is proceeding with:

- Computation of DT5941 if not existing
- Update of DT5941 with DT5755 in case DT5941< DT5755 (DT5755 is for total value of
purchase transactions of the basket)
This 90P validation module aims to detect abnormal DT5941 value comparing to High limit
defined by Shop Type level. For example a panellist reports spending 900 euros in a corner shop,
because of a typing error, instead of 9 euros.
This program is computing then a corrected value based on DTFMCG (total value computed on
Xcoding purchase acts for FMCG categories)
Corrected value is compared to DT5755 (in case corrected value is lower than DT5755 we set
this corrected value to DT5755).
TBSKT13 is then updated with corrected value of DT5941 and a record in inserted in
Family_logT67 to keep trace of the change.
Note : For python version Hive data refresh is done at the beginning for these tables:
1.ShopusabilityT40
2.Tbskt13
For oracle data refresh is done at the end of this process in terminateJob for these tables:
1. Tbskt13
Validation model:
If DT5941 > MAX_SPENT_CHVLX

then
if DTFMCG=0
Set DT5941_corrected= DEFAULT_CHVLX (correction logic 1)
else
Set DT5941_corrected= MIN (DTFMCG * RATIO_CHVLX, DT5941) (correction
logic 2)
end if
if DT5941_corrected < DT5755
then
set DT5941_corrected=DT5755
end if
End if

Page 300 from 698
MAX_SPENT_CHVLX, RATIO_CHVLX and DEFAULT_CHVLX are static parameters defined at
Shop Types level.
Recommendations :
This job should be submitted after Trip Data validation (24P)
Constraints :
Prerequisites :
FMCG Datatype computated (job Shop Weight 60P)

Availability of data in Rawda for the period T
Table Description
CYCLEPERIODT03 Periods definition
Files:
Input Files:
No input file
Output Files :
No output file
Parameters :
Static Parameters

Type Value
PARM DTFMCG 5755 Datatype code for FMCG total value in TBSKT13
PG01 CV01 0000000001 Identify the characteristic values 1 for the PRICE
GROUP
CV01 VAL_MAX 999 Maximum value parameter for Total Spent for Shop
characteristic value 1
CV01 VAL_DEF 100 Default value parameter for Total Spent for Shop

Page 301 from 698
CV01 RATIO 1.5 Ratio parameter for Total Spent for Shop
PG01 CV02 0000000002 Identify the characteristic values 2 for the PRICE
GROUP
CV02 VAL_MAX 499 Maximum value parameter for Total Spent for Shop
CV02 VAL_DEF 75 Default value parameter for Total Spent for Shop
CV02 RATIO 1.3 Ratio parameter for Total Spent for Shop
PG01 CVXX XXXXXXXXXX Identify the characteristic values XX for the PRICE
GROUP
CVXX VAL_MAX XXX Maximum value parameter for Total Spent for Shop
characteristic value XX
CVXX VAL_DEF XXX Default value parameter for Total Spent for Shop
CVXX RATIO XXX Ratio parameter for Total Spent for Shop
PARM DTFMCG 5755 Define Datatype code of Total Spent FMCG
Kindly note: If CVxx is defined, its corresponding VAL_MAX, VAL_DEF and RATIO must be defined.

Page 302 from 698
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

Process XXXX 0000.000 normal Allowed
Load data if same basket not present in TBSKT13
Rework XXXX 0000.000 normal Allowed

Page 303 from 698
EHDV21091P – Assortment Validation
Phase :
The aim of this validation is to check the NAN / SHOP CHAIN consistency in RawdailyT60.
In this job we can limit scope of purchase transactions to be validated to:

- a limited scope of product by selection a IBD (Item Break Down)
- a limited scope of shops by selecting a Shop Chain value
Several shop chain/IBD can be defined
A NAN/SHOP CHAIN combination is considered as correct if any of following condition is

checked OK:
➢ NAN/SHOP CHAIN combination is present in SHOP_PRICEST99 table (Prices coming

from RMS)
➢ NAN/SHOP CHAIN combination is present in RAWDAILYT60 for past periods
➢ More than Min_pan distinct HHOLD have reported the NAN/SHOP CHAIN combination for
current RAWDAILYT60 period (list of NAN/SHOP CHAIN will be exported to output file)
➢ The number of distinct households for which the Assortment validation job has EITHER
already reclassified the NAN/SHOP CHAIN combination over last T67_PERIOD periods
OR would reclassify the NAN/SHOP CHAIN combination in the current week exceeds
Min_pan_2
If a Nan/Shop combination is found as not correct, the job will reclassify the purchase transaction
by replacing existing shop code by a generic shop code (defined as static parameter). Job is not
reclassifying the other transactions of same basket.
We will keep trace of reclassification in FamilylogT67
Validation model:
STEP 1: Get date from

Page 304 from 698
T99 Date_from based on T99_PERIOD static parameter and accessing Cycle_period table
T60 Date_from based on T60_PERIOD static parameter and accessing Cycle_period table
T60_validation Date_from based on #Periods in submission and accessing Cycle_period table
T67 period from based on T67_PERIOD static parameter and accessing Cycle_period table
T67 period as period just before submission period accessing Cycle_period table
Loop by CH1 VAL / IBD_ID
STEP 2: Get list of products

If IBD is not equal to -1: Access table IBD_RULEST106 for selected CYCLE / IBD-ID and resolve
instructions in table to obtain NAN list.
STEP 3: Get list of shops

Access table SHOPUSABILITYT40 by Cycle / Period/ Char_ID/ Char Values
Get list of SHOP / SHOP CHAIN to be considered for assortment validation.
STEP 4: Select assortment data from T99

Select distinct Nan_Key SHOP_CHAIN from T99 table
For selected Source
For selected list of Shops
For selected list of nan_key (if IBD instructed)
For date between [T99_Date_from] and submission Date_to
STEP 5 : Select assortment data from T60 past periods

For selected Audit
For date between [T60_Date_from] and [T60_validation Date_from-1 ]
STEP 6: Select data from T60 current period

For selected Audit
For date between [T60_validation Date_from] and [submission date_to]
For count(distinct HHOLD)> MIN_PAN
Write in output file Nan / Shop Chain combinations that are not present in RMS assortment, and
not in past assortment but that are reported by MIN_PAN distinct panelist in current period
STEP 7: Select data from T67 past periods + T60 current period
Select distinct Nan_Key SHOP_CHAIN from
(select HHOLD ,Nan_Key SHOP_CHAIN from T67 table

For selected Audit
For Period between [T67 period_from] and [T67 period_to]
Job is EHDV21091P

Page 305 from 698
union
Select HHOLD ,Nan_Key SHOP_CHAIN from T60 table

For selected Audit
For date between [T60_validation Date_from] and [submission date_to] )
For count(distinct HHOLD)> MIN_PAN_2
Write in separate output file Nan / Shop Chain combinations detected in STEP7 not detected in
previous steps
STEP 8 : Identify records to be reclassified

Select purchase keys from T60
For selected Audit
For date between [T60_validation Date_from] and [submission date_to]
Where Nan_key / Shop_Chain not selected on STEPS 4 / 5 / 6 / 7
In addition to this list of non-selection, if the static parameter PB_EXCLUDE is set to “YES”, then
the NAN key list got from the table TDV_PB_BASKET_DETECTION is excluded.To get the list of
NAN keys NAN_CHAR_VALUEST75 is used.
STEP 9 : Apply reclassification

For all purchase keys selected on STEP7: update shop code with DEF_SHOP shop code in T60
and T29.
STEP 10 : Udpate Family_log

Insert into T67, all reclassifications.
GO to next CH1 VAL / IBD_ID
Recommendations :
This job should be submitted before Trip Data Alignment (23P) & Trip Data validation (24P)
Constraints :
RMS shop chain & CPS shop chain should be consistent (same Char_id, same Char_values).
Prerequisites :
Rawda for current period should be loaded

RMS Shop price should be loaded

Page 306 from 698
Table Description
Files:
Input Files:
No input file
Output Files :
File 1 : Nan Chain combinations from step 6, not present in previous steps
This output file is containing list of Chain / Nan_key combination present in current T60 period for
more than Min_Pan 307anellist307 and not present in RMS assortment neither CPS history
assortment.

1 16 NAN_KEY Nan key value, left aligned
17 10 AC_SHOP Shop chain
Sorted by Nan/Key/Chain ascending
File 2 : Nan Chain combinations from step7, not present in previous steps

1 16 NAN_KEY Nan key value, left aligned
17 10 AC_SHOP Shop chain
Parameters :
Static Parameters

Type
CHID PG01 CHAIN Identify the characteristic ID for the Chain 1 in shop
usability table
PG01 CV01 0000000001 Identify characteristic value 1 in shop usability table

Page 307 from 698
PARM IBD_ID_01 1 IBD id defining the scope of products to be validated
for chain 1
PG01 CV02 0000000002 Identify characteristic value 2 in shop usability table
PARM IBD_ID_02 2 IBD id defining the scope of products to be validated
for chain 2
PG01 CVXX XXXXXXXX Identify characteristic value XX in shop usability table
PARM IBD_ID_XX XX IBD id defining the scope of products to be validated
for chain X
PARM DEF_SHOP 0000000111 Shop code used for reclassification
PARM MIN_PAN 5 Minimum number of panellists on current period to
consider a new NAN / SHOP as valid
PARM SOURCE 02 Source code in T99
PARM T60_PERIOD 12 Number of periods back in T60 to build CPS history
assortment
PARM T99_PERIOD 4 Number of periods back in T99 to build RMS
assortment
PARM MIN_PAN_2 5 Minimum number of panellists for step7 (family_log
past periods + T60 current period)
PARM T67_PERIOD 12 Number of periods back in T67 for step7
PARM PB_EXCLUDE Yes or No Private Brand instructed nans to exclude ‘YES’ or
‘NO’ –Default value is ‘NO’
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

IBD (Item Break Down) defining the scope of products to be validated is defined in IBD screens.

Page 308 from 698

code
P0020125 File Name is blank File Name is S Purchase Data Input
P0020126 File Name not present FileName S Purchase Data Input
parameter are
defined for
submission
cycle
Lock. RELEASE Lock.
Lock
P9900162 No instructions found for No S No Instruction Found Check IBD
IBD cycle Instruction for IBD for the definition
Found specific cycle
P9900287 No NANS selected S No NANS selected Check IBD
definition
P9900378 Incosistency in SHOP SHOP S
CHAIN/IBD Resolution CHAIN/IBD
inconsistenc
y

P9900309 Number #SHOP/
SHOP/SHOP SHOP
CHAIN Records CHAIN
Selected for shop
char
P9900310 Number SHOP #SHOP
CHAIN/NANS from CHAIN/
T99 for shop char NANS
T60 past period for NANS
shop char

Page 309 from 698
T60 current period NANS
> MIN_PAN,
distinct HHOLD for
shop char
past T67 and T60 NANS
current period >
MIN_PAN_2,
distinct HHOLD for
shop char
CHAIN/NANS CHAIN/
combinations valid NANS
on step7, not valid
on previous steps
CHAIN/NANS CHAIN/
combinations not NANS
valid for shop char
P9900315 Number of #
Purchase records Purchas
not valid for shop e not
char valid
P9900314 Number of basket #
not valid for shop Basket
char not valid

Page 310 from 698
EHDV21092P – DV KPI computation and extraction
Phase:
This program is computing general DV Key Process Indicator (KPI) at audit, period and House
Hold Group Level.
This process can be submitted in two modes:

• When MULTI_MODE=OFF the KPI values are computed without taking into the account
the House Hold Group Level. KPI values are stored in KPI_VALUEST107 table for the
specific audit/period having as value for HHGROUP=ALL
When MULTI_MODE=ON the KPI values are computed taking into the account the
SAMPLE Characterisic and the SAMPLE Characteristic Values of the House Hold Group
Levels defined as Static Parameters. KPI values are stored in KPI_VALUEST107 table
for the specific audit/period having as value for HHGROUP the Description defined as
Static Parameter depending on HHOLD SAMPLE Group Characteristic Value.
IF MULTI_MODE DE = ON THE
o Check new parameters consistency

o Check existence of chars in Sample DB
o Check existence of char values in Sample DB for the period
▪ Get the List of HH by HH Group using Sample DB link. Each group defined by user in
the parameters will be split in 2 parts depending on entry date of HH
o (last day of period – HH entry date) > DAYS_NEW ➔ On Going

o (last day of period – HH entry date) <= DAYS_NEW ➔ New
As a matter of example, if user define 3 groups as Mobile, Cipherlab and Swap. 6 groups
of KPI will be stored as follows:
Cipherlab New and Cipherlab On Going

Mobile New and Mobile On Going
Swap New and Swap On Going
▪ Compute KPI by HH Group

▪ Store them with appropriate description as HH_GROUP
Job 92P can compute KPI values for several periods in one run.
Once KPI_VALUEST107 is populated, KPI values can be accessed using dedicated screen of
GUI, this screen is also providing functionality to extract data to text file or xls file.
Note : For python version Hive data refresh is done at the beginning of this process for these two
tables:
• Rawdailyt60
• Tbskt13

Page 311 from 698
Here is the list of KPI computed:
KPI KPI description KPI short desc

code
0001 Total number of Panelists present in TBSK Nb of panellists
0002 Total number purchase acts from scanner (DT5948) Nb of purchase acts from scanner
0003 Total number purchase acts in Rawda (Xcoded purchase acts) Nb of purchase acts in rawda
0004 Total number of purchase acts codebook Nb of codebook purchase acts in rawda
0005 Total number of shopping trips Nb of shopping trips
0006 Total number of barcodes Nb of barcodes
0007 Total number of shops Nb of shops
0008 Average of the total number of shopping trips by 312anellist Nb of shopping trips by panelist
0009 Average of the total number of purchase acts by 312anellist in Nb of purchase acts by panelist
TBSK
0010 Average of the total spent declared by 312anellist Total spent (declared) by panelist
0011 Average of the total spent Xcoded by 312anellist Total spent (Xcoded) by panelist
0012 Average of the total number of distinct items by 312anellist in Nb of items by panelist
RAWDA
0013 Average of the total spent by shopping trip Total spent (declared) by shopping trips
0014 Average of the total spent Xcoded by shopping trip Total spent (Xcoded) by shopping trips
0015 Average of the total purchase acts per shopping trip in Nb of purchase acts by shopping trips
RAWDA
0016 Average of the total number of distinct items per shopping trip Nb of items by shopping trips
in RAWDA
0017 Total number of Panelists present in RAWDA Nb of panellists
0018 Average of the total number of purchase acts by panellist IN Nb of purchase acts by panellist
RAWDA
0019 Average of the total number of distinct items per shopping trip Nb of items by shopping trips
in TBSK
Recommendations:
Rework:
No rework option. Before computation of KPI for a given audit/period, content of

KPI_VALUEST107 is deleted.
IF static parameter is set MULTI_MODE=OFF then records for audit/period and
hhgroup=ALL are deleted.
IF static parameter is set MULTI_MODE=ON then records for audit/period and

hhgroup<>ALL are deleted.
Constraints:
Prerequisites:
- Trip data alignment and trip data validation should be completed

Page 312 from 698
Table Description
TBSKT13 Total Basket Rawda
KPI_INSTRUCTIONS List of KPIs with short and long description
KPI_VALUEST107 Values of KPIs by audit & period
Files:
Input Files:
No output file
Output Files:
Non fixed layout, fields separated comma (,)
Name Description
Audit Audit code
Period DV period code
KPI code KPI code
KPI desc KPI long description
KPI value Value of KPI
HHGROUP Households Group
Output file is sorted by AUDIT / PERIOD / KPI code ascending
Parameters:
Static Parameters

Type
PARM BCDTYP 01 Barcode type for Codebook
(optional if not set then BCDTYP=0)
PARM MULTI_MODE ON – Multiple Groups KPI to be computed
OFF – Standard computation
PARM GROUP_CHAR ScanTP Set the SAMPLE Characteristic to be used for
HH groups split
(mandatory if MULTI_MODE=ON)
PARM GROUP1_VALUE 0000000001 Set the SAMPLE Characteristic Value for
Group 1
PARM GROUP1_DESCR Cipherlab Set Description for group 1
Group 2

Page 313 from 698
PARM GROUP2_DESCR Mobile Set Description for group 2
Group 3
PARM GROUP3_DESCR Swap Set Description for group 3
PARM DAY_NEW 90 Number of Days after which HH is not
consider as New anymore
PARM GROUP4_DESCR Set Description for group 4
(considered only if both
GROUP4_DESCR and
GROUP4_VALUE are given)
PARM GROUP4_VALUE Set the SAMPLE Characteristic Value for
group 4
GROUP4_DESCR and
PARM GROUP5_DESCR Set Description for group 5
GROUP5_DESCR and
PARM GROUP5_VALUE Set the SAMPLE Characteristic Value for
group 5
GROUP5_DESCR and
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 314 from 698

code
-20999 Program Terminated – System Error S No Audit defined for Check
Problem in retrieving the submitted cycle dtparamters
Audit For the given Cycle-
screen
P0020125 File Name is blank File Name is S Output File is blank Add an output
blank file name
P0020126 File Name not present FileName S Output File name is Add an output
not present not present file name
P9900332 SAMPLEDB DBLINK SAMPLEDB S SAMPLEDB
does not exist DBLINK DBLINK does not
does exist
P9900370 GROUP_CHAR defined is GROUP_CH S GROUP_CHAR Check the
not present in AR defined defined is not GROUP_CHA
SAMPLEDB is not present in R static
present in SAMPLEDB parameter
SAMPLEDB

P9900316 KPI Value KPI Value Computed Value of a KPI
P1041009 Total Number of Record Cnt Total Number or records extracted in the output
records extracted file
P9900333 DBLINK name DBLINK: Provides the name of the DBLINK.

Page 315 from 698
EHDV21093P – Extract for panelist compliance
Phase:
This program extracts Key Process Indicator about data reported by Household.
Information is aimed to be used by Panel Management team to assess 316anellist compliance.
Information will be extracted out of DV as an external flat file.
o Select list of 316anellist to be managed : select all records for audit, period range
with minimum one record in TBSKT13
o Extract information from FamilyUsability
o Extract/compute information from TBSKT13
o Extract/compute information from RAWDAILYT60
o Extract/compute information from ALLT11
o Extract/compute information from RAWDAILYT60 by Shop type (if CH01 is
present)
o Extract/compute information from RAWDAILYT60 by IBD id IBD_ID_1 is present
o Write records in the output file according to the layout.
o Submit external unix script
Recommendations:
Rework:
No rework option
Constraints:
Prerequisites:
- IBD should be defined
Table Description
FAMILYUSABILITYT41 Panelist characteristics

Page 316 from 698
IBD_RULEST106 Detailed rules for IBD
ALLT11 All the data collected by the families are loaded inside TALL. During the
load phase, all the single fields of the record are separated in datatypes and
loaded in the table ALL as separate records
Files:
Input Files:
No Input file
Output Files:
Non fixed length layout, fields separated by comma (,)
Number of fields is not fixed.
First record of the file is a header record containing the name of each field.
Field # Name Description

1 PANELIST CODE NUMBER
2 PERIOD Yyyywww Period of submission
3 NB ADULT From Family usability
4 NB CHILDREN From Family usability
5 NO SHOPPING TRIPS Number of records in T13 by period/HH
6 TOTAL BASKET SPENT (DECLARED) sum of 5941 in T13 by period/HH
7 TOTAL BASKET SPENT (COMPUTED) sum of 5755 in T13 by period/HH
8 Nb of scan acts from scanner sum of 5948 in T13 by period/HH
9 Nb of scan acts cross-coded in Rawda Number of records in T60 by period/HH
10 Nb of items cross-coded in Rawda Number of distinct barcodes in T60 by period/HH
11 Nb of scan acts in ALLT11 Number of records in TALL by period/HH
12 Nb of barcodes in ALLT11 Number of distinct barcodes in TALL by period/HH
13 Nb of scan acts from codebook in Rawda Nb T60 records for codebook Barcode type by period/HH
Nb of distinct barcode in T60 records for codebook Barcode
14 Nb of distinct items from codebook in Rawda
type by period/HH
Nb of distinct shop types visited among shop types
15 Nb of different shop-type visited
instructed
16 Nb of scan acts for shop-type 1 Nb of scan acts in T60 for shop type value 1
17 Nb of scan acts for shop-type 2 Nb of scan acts in T60 for shop type value 2
XX Nb of scan acts for shop-type XX Nb of scan acts in T60 for shop type value XX
XX Nb of scan acts for IBD 1 Nb of scan acts in T60 for IBD value 1

Page 317 from 698
XX Nb of scan acts for IBD 2 Nb of scan acts in T60 for IBD value 2
XX Nb of scan acts for IBD XX Nb of scan acts in T60 for IBD value XX
File is sorted by Panelist code ascending.
Parameters:
Static Parameters

Type Value
CHID ADLT NBADLT Characteristic code in Familyusability for
number of adults in household
CHID CHLD NBCHLD Characteristic code in Familyusability for
number of children in household
PARM BCDTYP 01 Barcode type for Codebook
PARM IBD_ID_1 1 IBD id for the 1st group of items (optional)
PARM IBD_ID_2 5 IBD id for the 2nd group of items (optional)
PARM IBD_ID_XX 10 IBD id for the XXth group of items (optional)
CHID CH01 SHOPTY Characteristic code in Shopusability for shop
type (optional)
CH01 CHVL01 0000000002 Shop type value for 1st group of shop
(mandatory if CH01 present)
CH01 CHVL02 0000000010 Shop type value for 2nd group of shop
(optional)
CH01 CHVLXX 0000000322 Shop type value for XXth group of shop
(optional)
PARM SCRIPT_C Check Appendix for Name of shell script to be executed after job
FT Information completion
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 318 from 698
IBD screens

code
P0020505 Number of periods: Stat.Val not S NUM_PER static Define
present parameter has not NUM_PER
been defined static
parameter
P0020624 Error in retrieving Period System Error S
List
P1021015 No Shops Present in the Shop not in S No shops present in
Shops Usability table – T40 SHOPUSABILITYT4
PG01: ,CHVLxx: 0 for the
characteristic /
characteristics
values defined as
Static Parameters

date range
P0020608 SCRIPT_CFT: Static Value Static parameter SCRIPT_CFT
P0020608 ADLT: Static Value Static parameter ADLT:
P0020608 CHLD: Static Value Static parameter CHLD:
P0020608 BCDTYP: Static Value Static parameter BCDTYP:
P0020608 CH01: Static Value Static parameter CH01:
P0020608 IBD: Static Value Static parameter IBD:
P1041009 Total number of Record Cnt Number of records extracted in the Output File
records Extracted

code
following name:
ended JOB required

Page 319 from 698
displayed:
displayed:

Page 320 from 698
EHDV21094P- Basket Validation
Phase:
The automatic basket validation system is controlling the correct combination of a shop type and
the categories (or products) sold in that shop type.
Whether we find an inconsistency between the shop type and a category (or a product), we
automatically reclassify the purchase transaction in a new basket where the new shop / shop type
is consistent with the category under inspection.
Basket Validation instructions:
Instruction set 1:
Scope of shop type to be validated. Only purchase acts with shop code belonging to the list of
shop type instructed to be validated will be considered by this validation job.
Instruction set 2:
Basket validation is using the IBD (Item Break Down) concept.
IBD is a group of item defined by a rule using or several characteristics and logical operators.
IBD are created inside DV system using a dedicated GUI screen.
For each shop type that should be validated, the IBD defining the scope of product that are
“correct” should be defined. If the item of a purchase act is not belonging to the IBD defined for
the shop type of purchase shop code, then the purchase act is validated “Not OK” and is
reclassified.
Instruction set 3:
The reclassification is driven by IBD reclassification instructions. A set of reclassification IBD
should be defined, this IBD reclassification should not overlap (one particular item should belong
to one IBE reclassification only). For each IBD reclassification a reclassification shop type should
be defined.
Instruction set 4:
A shop code for reclassification should be instructed for each shop type.
Step 1: Get Shop_type / Shop

Get list of shop types to be validated. Using this list and shopusability information (current period
or if not found period 0) get the list of shop codes to be validated.
Step 2: Get IBD/Nan & IBD_RECLASS/Nan

Get list of all IBD to be validated (from table BASKET_IBDTXXX)
Resolve IBD using IBD rules to get all Nan_key that are to be validated by Basket Validation.

Page 321 from 698
Step 3: Get purchase acts to be validated
Select from T14/T12 all purchase transactions, keep for next steps only selected shop types &
scope of Nan_key.
Step 4: Prepare Validation

Link each purchase transaction to IBD and Shop type (one purchase transaction can be linked to
more than one IBD)
Step 5: Basket Validation

Get keys that are validated OK (IBD_ID || SHOP_TYPE present in BASKET_IBDT108)
Get keys that are not validated OK (IBD_ID || SHOP_TYPE not present in BASKET_IBDT108)
Step 6: Get reclassification details

For keys not validated OK, get IBD reclassification, from this get shop type for reclassification and
derive shop code.
The job is failing in case Nan_key belongs to more than one IBD reclassification or in case
Nan_key belongs to no IBD defined in IBD reclassification.
Step 7: Apply changes in purchase data

Check whether new key is present inside the T14 table. If present increase the time by 1.
Insert in P359_1T12 New key (new time, new shop) for non valid purchase act
Delete P359_1T12 Old key (old time, old shop) for non valid purchase act
Insert in KESYT14 New key (new time, new shop) for non valid purchase act. Old key are
remaining in T14 and will be reclassified cleaned by Rawda Loader.
Delete dup keys from KESYT14 for confirming no more dup key issue with 138P.
Step 8: Store changes in Family_logT67
Constraints:
Prerequisites:
➢ Load instructions for the basket validation.

➢ Load the shop characteristics into the shop usability table.
Table Description

Page 322 from 698
BASKETSHOPT28 This contains link to each shop type a shop code to be used when a new
basket has to be generated.
BASKET_IBDT108 Table containing instructions Shop Type values / IBD
BASKET_RECLASS_IBDT Table containing instructions IBD reclassification/Shop Type
09 reclassification
BASKETSELECTIONT71 This contains list of shop types to be inspected
Files:
Input Files:
No input file.
Output Files:
This output file contains the following datas
List of Nan_key present in purchase data and not instructed in

BASKET_IBDT108. These nan_key are not validated by 94P job.
 List of Nan_keys belongs to more than one IBD reclassification(When job
fails due to this reason)
 List of Nan_keys belongs to no IBD defined in IBD reclassification(When
job fails due to this reason)
This is a comma separated file.
Name Description
NAN_KEY Nan_key value
SG Super group
SG_DESCRIPTION Super group description
PG Product group
PG_ DESCRIPTION Product group description
MODULE Module
MODULE_DESCRIPTION Module description
Parameters:
Static Parameters

Type Value
CHID SHPTYP Shop Type value Used to identify the characteristic shop type in
the shop usability table.
Dynamic Parameters

Name (Yes/No)

Page 323 from 698
# Periods No
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management:
Type
Validation -> Basket Validation screens (shop type selection / IBD definition/IBD
reclassification/Basket Shop)

code
P1017001 No Transactions Available NoData for S No Purchase Correct the
for the Selected Timestamp Transactions input data and
Timestamp Available for the load the file
Selected again.
Timestamp.
P1017013 Product to be reclassified Product not S Product not in T08 A product
not found in T08 in T08 involved in the
purchase
transaction is
not defined in
Basket
definition table.
P1017003 Basket is not Correct Basket not S Basket not Correct Shop type and
Correct Product are not
consistent.
Basket is not
correct.
Arrays
P1017007 Error while trying to System Error S Validate Basket Some error
validate the basket Error occurred
during basket
validation.

Page 324 from 698
P1017017 Missing Shops Missing S Missing Shop The reported
Shop shop is not
present in shop
usability table.
P1017018 Error while fetching period System Error S Fetching Period Some error
occurred while
getting the
period.
P1017010 No basket selection is No basket S No basket selection No shop types

available selection are requested
for validation.
P1017009 No basket instructions are No Bskt S No Bskt instrns No Basket
available instrns definition
instructions are
available.
P1017011 No reclassification No S No reclassification No Shop type –
instructions present in reclassificati Shop
T28 on association is
defined.
P1017008 Shop Type requested not ShpTyp not S ShpTyp not in T08 A shop type is
available in the T08 in T08 requested for
validation. But,
there is no
definition for
the shop type
in Basket
definition table.
P1017012 Shop Code not available No ShpTyp- S No ShpTyp-Shop For a shop
in Shoptype – Shop Table Shop link link type defined in
basket
definition table,
shop type –
shop
association is
not defined.
P0020115 Data Value not found in DataValue S
P0020121 Error in getting the System Error S
DataValue from359 table
P0020103 Error while retrieveing the System Error S
Error Type
P0020146 Error while writing record System Error S
into Datalog Full Record
Array
P1017016 Error in Update Datalog System Error S Error in Update Data Some error
occurred while
trying to update
the Data Log.

Page 325 from 698
P1017006 No generic shop code for No generic S No generic shop A shop type is
shop type shop code code selected for
validation. But,
no generic
shop code
definition is
made for the
shop type.
P0020607 Error while getting Basket System Error S
Key Value from T29
P1017014 Error when trying to System Error S Update purchase Some error
update purchase data data Error occurred while
updating the
purchase data.
P1017015 Error when trying to System Error S Insert T359 Error Some error
update new record in occurred while
T359 trying to update
the new record
in T359.

P1038020 Total Number of Purchase act Number of purchase transactions selected in
purchase acts count T12 for selected cycle, timestamp, number of
periods, shop types
P9900319 Number of purchase #Keys to be Number of purchase transactions where
keys to be validated validate validation instruction where found (nan_key
present in one IBD)
P9900206 Number of NANS Number of Number of distinct Nan_key where validation
selected NANS instruction where found (nan_key present in
selected one IBD)
P9900320 Number of purchase #Keys not Number of purchase transactions where no
keys not validated validated validation instruction where found (nan_key not
because no IBD present in any IBD)
defined
P9900321 Number of purchase #Keys Number of purchase transactions where
keys validated OK validated OK IBD/shop type is ok
P9900322 Number of purchase #Keys not Number of purchase transactions where
keys not validated valid OK IBD/shop type is not ok
OK, shoptype/IBD not
correct
P0940001 Number of records Number of Number of records inserted in T12
inserted records in
P1022027 Number of Records Delete Cnt – Number of records deleted in T12
deleted from T359
P359_T12
P9900141 Number of records Number of Number of records in T14
inserted records in
P9900334 Number of Records Ins/Upd Number of records inserted or updated in
Ins/Upd into count T67 Family_logT67
FamilyLogT67

Page 326 from 698
EHDV21095P – Price & Total Value validation
Phase :
This validation has been designed to fix an issue faced during launch of scanning panel in
Finland.
Analyse of data coming from 327anellist327 showed that even so if 327anellist is instructed to
keyboard the unit price for standard barcode, they sometimes misunderstand the instructions and
keyboard the total value.
For example, a 327anellist is buying 3 bottles of soda and is paying 0.6 euro each. When
scanning this purchase act, 327anellist should scan the barcode, enter units=3 and price=0.6
euros. Some panellists sometimes enter unit=3 and price=1.8 euros.
Main purpose of this new validation job, is to detect if we have such case, by comparing price
entered by the panellist with reference price and if out of range, check if price/unit is close to
reference price. If this is the case, both DT5667 and DT5675 are replaced with original price /
units.
Scope of products : transactions relating to “regular” barcoded items (e.g. manufactured goods).
Excluded thus variable weight items (those with weight within barcode) as well as code book
items.Selection will be done using barcode type
No special attention required for multipacks – supposed to be tackled by the usual DV routines
All corrected are logged in FamilylogT67 table and price correction flag is updated with specific
value (18).
Validation model:
Step 1: get list of nan to be validated, initialise the reference price as standard price from previous
period
Select full list of Nan_key in KEYST14 for selected audit/period/timestamp and for barcode type
defined as static parameter
For all relevant nan_keys, set RefPrice(t):=StdPrice(t-1), here we select standard price in
CNANPRICET34 for Price Group 0 (and not standard price for specific price group / retailer) for
previous period
For next step split the data for week t into two parts
Part A: Transactions with units=1
Part B: Transactions with units > 1
Step 2: Process Part A

Lookup for RefPrice(t)
• For nan_keys with NO RefPrice(t) available: compute RefPrice(t):= MedPrice(t), the median
of all prices collected in week t within part A
• For nan_keys with a RefPrice(t) available: update RefPrice in the exact same way as is done
within the usual standard price computation

Page 327 from 698
o Compute the median price on the current week data: MedPrice(t) within part A
o Count the number of observations for the nan_key in current period: N(t)
o Compute “weight” for current period: w(t):=N(t) / (N(t) + K) where K is a static
parameter
o Update RefPrice(t) :=w(t)* MedPrice(t) + (1-w(t))* RefPrice(t)
Step 3 : Process Part B

Lookup for RefPrice(t) (Note that this is the updated version of RefPrice as per Step2)
• For nan_keys with NO RefPrice(t) available: leave unchanged
• For nan_keys with a RefPrice(t) available: compare collected price to RefPrice; if too far
away, divide price by units (since assumption to be checked is whether the price we got is
actually a value, the result of this division could be the true price …) and compare to
RefPrice(t)
o If price is too low then it cannot be a value, so we will consider only prices too high,
do nothing
o If Price <= Ratio * RefPrice(t), then leave unchanged. Ratio is a static parameter
o If Price > Ratio * RefPrice(t), then check further:
▪ set NewPrice:=Price/Units
• If NewPrice <= Ratio * RefPrice(t) then reset Price:=NewPrice
• If NewPrice > Ratio * RefPrice(t) then leave unchanged
Apply correction in P359_T12 table, both DT5667 and DT5675 are updated with new price.
Insert/update DT5904= 18 for updated records.
Recommendations :
Constraints :
Prerequisites :
DataSelection should be completed
Table Description
Files:
Input Files:

Page 328 from 698
No input file
Output Files :
File 1: List of the top TOPPAN 329anellist with highest number of corrections

1 10 HH
11 4 Number of
corrections
15 4 Number of
transactions
with units >1
Sorted by number of correction descending
Parameters :
Static Parameters

Type Value
DTDT BCT01 1 Barcode type # 1 to be included in validation
DTDT BCT02 1 Barcode type # 2 to be included in validation
DTDT BCTXX Barcode type # X to be included in validation
PARM K 5 Factor user in Std price formula
PARM RATIO 1.5 Define the ratio to detect out of range prices
PARM TOPPAN 50 Define the number of top 329anellist with highest
number of corrections to be written in output file
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :


Page 329 from 698

Page 330 from 698
EHDV21097P – Populate Shop_priceT99 from cps data (UK)
Phase :
This validation has been designed to provide a solution for Price Imputation in UK.
In UK RMS prices can not be used for cps price imputation process, and not all prices can be
collected. Only a part of the panel is collecting data, the main logic is to use prices collected from
cps panellists to impute on transactions where price is missing.
Aim of this job is to populate SHOP_PRICEST99 with data from TALL.
Only transactions with following conditions verified will be selected

• Price>0
• Nan_key>0 (Xcoded item)
• Submission audit
• Submission date range
• HH present in T43
• SH in T44
Median price at SHOP / NAN_KEY will be computed only for number of transactions > MIN_OBS
(static parameter).
Prices are loaded in table SHOP_PRICEST99, structure for this table is following :
Source Identifier for source of price, static parameter

Nan_Key Product identifier
Price Price value
P_Date_to Price validity date to
Shop code local Not used by this job
The “ Price validity date to “ inserted is the date_to of submission period.

The “ Price validity date from “ inserted is the date_from of submission period.
When rework mode is selected, program is deleting all the records already loaded for the number
of selected weeks ( # periods).
The job can populate prices in T99 for several sources. For each source, it is possible to define a
specific Min_obs value and some instructions on datatype + datatype value.
This functionality allows to compute price for purchase acts in promo
Prerequisites :
Standard loader, Xcoding, load of HH & SH char should be completed successfully.

Page 331 from 698
Table Description
SHOP_PRICEST99 Contains prices from RMS at NAN_KEY/ Shop code level
Files:
Input Files:
No input file
Output Files :
No output file
Parameters :
Static Parameters

Type
PARM MIN_OBS1 5 Minimum number of transactions for populating T99 for source #1
If no DT selection is instructed, all records from TALL will be selected.
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No

Page 332 from 698
F# Selection No
DTSelection No
File Selection No

Page 333 from 698
Rework Management :

Rework XXXX YYYY.PPP normal Delete data for the selected period, then load data

Page 334 from 698
EHDV21098P – Spain fresh food only basket validation
Phase :
This validation has been designed to provide a solution for a specific need for Spain.
Spain CPS panel is containing a sub-panel that is containing fresh food.
To produce CASE trip data and Shopper Track deliverables in DP, there is a need to exclude the
basket composed by fresh food products only. On the other hand, these particular basket should
not be deleted from TBSKT table in DV as they are required for other local analyses (like Shopper
Mission).
This job is identifying the baskets that are composed by fresh food product only and update a
specific datatype DT5126=1 for these basket. On DP side the specific parameter to take into
account DT5126 “Delete Basket flag” for the service that should use trip information. Those
specific basket will then be not loaded in DP/
o Get list of fresh food nan_key by resolving IBD on NAN_CHAR_VALUET75

o Get list of baskets keys for given audit/date range
o Identify basket with
▪ Minimum one purchase acts with nan_key >0
▪ All nan key of purchase acts present in list of fresh food items
o For all identified basket
▪ Update DT5126=1 in both T13 and T29
o Report total number of basket select and number of basket flagged as “fresh
food” only
* Data in oracle table is refreshed to parquet for Rawdailyt60,TBSKT13 and
Prockeyst29 tables.
*Error in Refresh parquet is logged using the error number - P9900605

* Data from TBSKT13 and Prockeyst29 tables.is refreshed back to oracle table.
*Error in Oracle Refresh is logged using the error number - P9900627
Prerequisites :

Page 335 from 698
Table Description

Page 336 from 698
Files:
Input Files:
No input file
Output Files :
No output file
Parameters :
Static Parameters

Type Value
PARM IBD_ID 1 IBD id for fresh food products
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

Rework XXXX YYYY.PPP normal Allowed, run like normal process
If the period is locked the job should be submitted in Rework mode.

Rework logic is similar to normal mode, only as first step job will Update DT5126=0 in both T13
and T29 for all basket in given audit/date_range.

Page 337 from 698
EHDV21099P–Variable weight barcodes management
Phase :
In most countries codebook products can be collected in two ways:

• Product without any barcode, like tomatoes bought in an open market, to report this Nielsen
338anellist338 are scanning codebook barcodes (and its characteristics)
• Product with specific variable weight barcode, like tomatoes bought in a supermarket, in that
case 338anellist is first scanning the barcode, then HHT is asking to scan a codebook
barcodes (and its characteristics). The original barcode is containing information about the
products + information about price. These barcodes can be easily identified as they start with
02 in most countries. From HHT Nielsen is receiving the codebook barcode + the original
barcode. With current design the original barcode is not loaded in DV and is not used within
NCPS, only codebook barcode is used in NCPS.
Some clients (eg Morrisson in UK) have interest in data related to this original barcode for
variable weight for some categories as it gives more information about the product (brand).
Solution implemented here is to load in DV both barcodes : codebook barcode in AC_BARCODE

field and original barcode in new datatype.
Aim of this job is to replace the codebook barcode with original barcode for some shops, some
group of product. The original barcode should be modified to skip the variable part (price) and
add EAN check digit.
To apply filter on shops and product job is using

• shop char + shop char values defining the scope of shops on which validation should be
applied
• IBD code describing scope of products for which validation should be applied (IBD can be
defined using any €imdb char, so it can be for example a list of legacy PC )
Select purchase acts in TALL using filter by:

• Submission Audit
• Submission period + submission number of periods back
• Scope of shops (eg Morrisons shops)
• Scope of barcodes
o Barcodes with original barcode not null
o Barcode with original barcode (without price, means replace with zero last X
digits and add new check digit), linked to a nan_key belonging to defined IBD
(Xcoding instruction are used here )
For all selected purchase acts:

• Replace AC_BARCODE with transformed barcode defined as:
Original barcode with 0 added in front to fit 13 digits,
replace the variable part (last DT_VAR_LENGHT digits) by 0

Page 338 from 698
replace last digits with new ean check digit
• DT containing original barcode, QTY, UNIT are remaining unchanged

• Update barcode type to barcode type value (if defined)
• Store change in Family_logT67 with old barcode + new barcode
Barcode transformation logic,
In UK when Morrisons use 02 barcodes, they use the 1st 6 digits of the barcode as the product
idendifier, the following six charaters is the price check digit and price field followed by the
barcode check digit :
025456/4/00065/9
When transforming Morrissons barcodes we need to keep the 1st 6 digits as they are, zero out the
proceeding 6 characters and then recalculate the barcode check digit:
025456/0/00000/4
In DV we will identify the variable part of the barcode with a static parameter named
DT_VAR_LENGHT
In that example DT_VAR_LENGHT parameter should be set to 7.
Prerequisites :
- Standard loader should be completed
- Shop chars should be loaded
Table Description
FAMILY_LOGT67 Log of changes applied by DV automatic validations processes (used here
to check if data loaded are already present in output tables)
CTIONST49
Files:
Input Files:
No input file
Output Files :
No output file

Page 339 from 698
Parameters :
Static Parameters

Type
CHID CH01 SHOPT Shop characteristic code
CH01 CHVL01 HYP Shop characteristic value 1 for shop char
code 1
CH01 CHVLXX SUP Shop characteristic value XX for shop char
code 1
PARM IBD_ID 1 IBD id to be used
PARM DT_VAR_LENGHT 7 Variable part of the original barcode (last X
digits)
PARM GR_ID_01 GB01 Group-ID in EIMDB to Xcode transformed
original barcode for shop char val 1
PARM GR_ID_02 GB02 Group-ID in EIMDB to Xcode transformed
original barcode for shop char val 2
PARM BCDTYPE 01 Barcode type for updated purchase act, if not
present barcode type remains unchanged
PARM PRICE_INC AVG/MAX To manage price in case duplicates records
to be merged
PRICE_INC parameter is a mandatory parameter used to drive cases of duplicated records:
if PRICE_INC=AVG, merge record will have a price computed as sum(pricexqty)/sum(qty)

if PRICE_INC=MAX, merge record will have a price computed as MAX(price)
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

Rework XXXX YYYY.PPP normal Not allowed, run like normal process

Page 340 from 698
EHDV21120P – Purchase File Extraction to CASE for LATAM
Phase:
This program extracts in an ASCII file, all purchase acts from RAWDAILYT60 for a given audit (or
multiple audits as instructed in the static parameter table) / period (or number of periods back).
The content of the output file as well as the final layout of the extracted ASCII file is driven by the
instructions in APP_FILES_LAYOUT112 table which are given by the user. Valid set of
instructions is considered to be the set that contains at least one instruction for RAWDAILYT60.
Since the content and layout of the extracted ASCII files could be different across countries the
instructions are instructed by audit.
flag(DT5090=1)
Detailed logic of extraction
The instruction table APP_FILES_LAYOUTT112 should be loaded with valid instructions that
specify for each of the entities to be extracted the following information:
➢ Source Table Name: Valid table values are

o RAWDAILYT60
o TBSKT13
o CYCLEPERIODT03
o SHOPUSABILYT40
o FAMILYUSABILITYT41
o NAN_CHAR_VALUEST75
o NAN_WEIGHT_VALUEST79
o V_NANSPGS_V
o NAN_MULTI (for MULTIPACK chars)
o FIXED (for fixed values, information not related to any DV source table)
➢ Source Field Name: Valid values are

o For RAWDAILYT60, TBSKT13 can be any DT field or row functions/logic,
DECODE, SUBSTR, TO_BASE, CASE etc
o For SHOPUSABILYT40, FAMILYUSABILITYT41 can be any Characteristic Code
Value
o For CYCLEPERIODT03 only NC_PERIOD
o For V_NANSPGS_V, NC_PRODUCT_GROUP or or row functions/logic,
DECODE, SUBSTR, TO_BASE, CASE etc
o For NAN_CHAR_VALUEST75 can be any Characteristic Code Value
o For NAN_MULTI the Characteristic Code Value for MULTIPACK
o For NAN_WEIGHT_VALUEST79 can be NC_NUMERIC_VALUE,
AC_UNIT_MNEM or row functions/logic, DECODE, SUBSTR, TO_BASE, CASE
etc
o For FIXED (for fixed values, when source is defined as FIXED)

Page 341 from 698
➢ Destination Field Name: Name of the field which will be present in the final SELECT
query of the extraction logic
➢ Field start position in ASCII file to be exported
➢ Field length in file to be exported
➢ If field is numeric and non integer, the length of the decimal part of the number
➢ The format of the field in the ASCII file. There are a number of different format
functions that can be used.
➢ The default value to be written in the ASCII file to be exported in case values are
not present in master tables.
In case that there is a need to instruct SPACES as FIXED value, the user should define
it as shown in Table(2). Any Gaps/overlaps between the instructions are not allowed.
Based on the instructions given for the purchase file extractor for a specific audit , the
program:
o Extracts information from FAMILYUSABILITYT41 for all the characteristics instructed
for all the distinct values of HHOLDS found in RAWDAILYT60 for the submitted audit
(s) period(s)
o Extracts information from SHOPUSABILITYT40 for all the characteristics instructed
for all the distinct values of SHOPS found in RAWDAILYT60 for the submitted audit
(s) period(s)
o Extracts information from NAN_CHAR_VALUEST75 for all the characteristics
instructed for all the distinct values of NANS found in RAWDAILYT60 for the
submitted audit (s) period(s)
o Extracts information from NAN_CHAR_VALUEST75 and CHAR_VALUEST24 for the
MULTIPACK characteristic instructed for all the distinct values of NANS found in
RAWDAILYT60 for the submitted audit (s) period(s)
o Extracts information from NAN_WEIGHT_VALUEST79 for all the characteristics
instructed for all the distinct values of NANS found in RAWDAILYT60 for the
submitted audit (s) period(s)
o Extracts PRODUCTGROUP information from V_NANSPGS_V or all the distinct
values of NANS found in RAWDAILYT60 for the submitted audit (s) period(s)
o Extracts TBSKT13 information for the submitted audit (s) period(s)
o Extracts RAWDAILYT60 information for all purchase transactions for the submitted
audit (s) period(s) combined with the information related to NAN, HHOLD and SHOP
information extracted in previous steps above
o Writes records in the output file according to the layout and the FORMAT
specification for each of the fields extracted.
Recommendations:
Rework:
No rework option
Constraints:
Prerequisites:
- Tbsk should be loaded
- Table APP_FILES_LAYOUTT112,should be loaded with valid instructions

Page 342 from 698
- System tables APP_FILEST111, APP_QUERY_TEMPLATEST113 should be loaded
(this is part of the SYSTEM SETUP)
Table Description
TBSKT13 Total Basket
CYCLEPERIODT03 Cycle Period information
FAMILYUSABILITYT41 Panelist characteristics
CHAR_VALUEST24 Characteristic Values from EIMDB
NAN_WEIGHT_VALUEST79 Item weight value id extracted from EIMDB
APP_FILES_LAYOUTT112 User instructions for File layout
APP_FILEST111 Definition of the output file type
APP_QUERY_TEMPLATEST113 These “Templates” define the table FROM and WHERE clauses that
will be used within the extraction jobs to be created
APP_SQL_LOGT114 All dynamically created strings created within the job are logged in
this table when the static parameter DEBUG is set to YES
ALL_TABLES Oracle Dictionary Table
Oracle Views Involved:
Table Description
V_NANSPGS_N Product Group for Nankey
Files:
Input Files:
No input file
Output Files:
File Name is created dynamically within the job:

Naming Convention
➢ For Homescan Panel: hh_snl<YYWW>.craw09.dat
➢ For Personal Consumption Panel: psp_snl<YYWW>.craw09.dat
where is the submission Period (e.g for 2009035 it will be 0935)
Fixed length layout, based on instructions in APP_FILES_LAYOUTT112 for the specific

File Type (purchase file) and Audit
Number of fields is fixed.
Table 2
AC_FLDNAME NC_START NC_LENGTH NC_DECIMAL AC_SOURCE AC_FIELD AC_TARGETFIELD

FIXED 1 10 0 FIXED SPACE FIXED

Page 343 from 698
HH_ID 11 8 0 RAWDAILYT60 AC_HHOLD AC_HHOLD
FIXED 19 3 0 FIXED 999 FIXED
Parameters:
Static Parameters
Parameter Type Value Valid Values Description

Type
PARM DEBUG YES When set to YES:
NO ➢ No working permanent tables are
created. If working permanent tables
have been created from previous job
submission for the same audit /period
they are used by the job.
➢ All the dynamically created strings
are logged in table APP_SQL_LOG
When set to NO (or not defined at all):

➢ All working permanent tables created
by previous job submission for the
same audit /period they are dropped
and created again.
➢ NO logging of the dynamically
created strings is done.
PARM PANEL HS When set to HS

PCP ➢ The prefix of the Output ASCII file is
set to: hh_snl
When set to PCP

➢ The prefix of the Output ASCII file is
set to: psp_snl
AUDT AUDITVL<XX> Any audit defined in Any valid (defined in auditdeft01 table) audit
auditdeft01 other the value
(e.g AUDITVL01 audit used for job
AUDITVL02 submission
etc)
PARM FILTER_DT5126 YES If value ‘YES’,logically deleted records are not

extracted.The default value for this is ‘YES’
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No

Page 344 from 698
DT Selection No
File Selection No

Page 345 from 698
Rework Management:

P9900381 Can NOT Create table

P9900382 File Definition does not exist
P9900383 No Instructions for File or Invalid Instruction set for PURCHASE DATA FILE
P9900384 Temporary Table does NOT exist
P9900385 Job should be submitted with debug=N first for temporary tables to be
available
P9900386 Wrong instruction definitions.There is gap/overlap between instructions
P1031026 # Distinct NANS for audit

P1038020 # Purchase acts selected for audit
P1041009 Total Number of records Extracted

Page 346 from 698
EHDV21121P – UPC File Extraction to CASE for LATAM
Phase:
This program extracts in an ASCII file, all NANS from RAWDAILYT60 for a given audit (or
multiple audits as instructed in the static parameter table) / period (or number of periods
back).The content of the output file as well as the final layout of the extracted ASCII file is driven
by the instructions IN APP_FILES_LAYOUT112 table which are given by the user. Valid set of
instructions is considered to be the set that contains at least one instruction for RAWDIALYT60.
flag(DT5090=1)
Same layout for all countries.

o RAWDAILYT60
o V_NANSPGS_V
o CYCLEPERIODT03

o For RAWDAILYT60: NC_NAN_KEY
o For V_NANSPGS_V, NC_PRODUCT_GROUP
o For CYCLEPERIODT03 only NC_PERIOD
➢ Target Field Name: Name of the field which will be present in the final SELECT query of
the extraction logic
it as shown in Table(2). Any Gaps/overlaps between the instructions are not allowed.
Based on the instructions given for the purchase file extractor for a specific audit, the
program:

Page 347 from 698
o Extracts all the distinct NAN values found in RAWDAILYT60 for the submitted audit
(s) period(s)
o Extracts PRODUCTGROUP information from V_NANSPGS_V or all the distinct NAN
values found in RAWDAILYT60 for the submitted audit (s) period(s)
Recommendations:
Rework:
No rework option
Constraints:
Prerequisites:
- System tables APP_FILEST111
Table Description
CYCLEPERIODT03 Cycle Period information
Oracle Views Involved:
Table Description
V_NANSPGS_N Product Group for Nankey
Files:
Input Files:
No input file
Output Files:
File Name as defined in the GROUPFILET63 table


Page 348 from 698
File is sorted by NANKEY ascending
AC_D
AC_FLD NC_ST NC_LEN NC_DEC AC_TARGE EF_V
NAME ART GTH IMAL AC_SOURCE AC_FIELD TFIELD AC_FORMAT ALUE
FIXED 1 10 0 FIXED SPACE FIXED String_Selection
String_Selection_ri 00000
HH_ID 11 8 0 RAWDAILYT60 AC_HHOLD AC_HHOLD ght 000
FIXED 19 3 0 FIXED 999 FIXED String_Selection 999
Table(2)
Parameters:
Static Parameters

Type

and created again.
AUDT AUDITVL<XX> Any audit defined in Any valid (defined in auditdeft01 table) audit
auditdeft01 other the value
(e.g AUDITVL01 audit used for job
AUDITVL02 submission
etc)

Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No

Page 349 from 698
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


available
P1038020 # Records selected for audit

P1041009 Total Number of records Extracted

Page 350 from 698
EHDV21122P – Trip File Extraction to CASE for LATAM
Phase:
This program extracts in an ASCII file, all trip data TBSKT13 for a given audit / period (or number
of periods back).The content of the output file as well as the final layout of the extracted ASCII file
is driven by the instructions IN APP_FILES_LAYOUT112 table which are given by the user. Valid
set of instructions is considered to be the set that contains at least one instruction for TBSKT13.
flag(DT5090=1)
Since the content and layout of the extracted ASCII files could be different across countries the
instructions are instructed by audit.

o TBSKT13
o CYCLEPERIODT03

o For TBSKT13: NC_NAN_KEY
o For CYCLEPERIODT03: NC_PERIOD
➢ Target Field Name: Name of the field which will be present in the final SELECT query of
the extraction logic
it as shown below. Any Gaps/overlaps between the instructions are not allowed.
AC_SOURCE AC_FIELD AC_TARGETFIELD AC_FORMAT AC_DEF_VALUE

FIXED SPACE FIXED String_Selection
RAWDAILYT60 AC_HHOLD AC_HHOLD String_Selection_right 00000000
FIXED 999 FIXED String_Selection 999

Page 351 from 698
Based on the instructions given for the purchase file extractor for a specific audit, the
program:
o Extracts all the distinct NAN values found in RAWDAILYT60 for the submitted audit
(s) period(s)
o Extracts PRODUCTGROUP information from V_NANSPGS_V or all the distinct NAN
values found in RAWDAILYT60 for the submitted audit (s) period(s)
Recommendations:
Rework:
No rework option
Constraints:
Prerequisites:
- TBSK should be loaded
- System tables APP_FILEST111
Table Description
CYCLEPERIODT03 Cycle Period Information
Files:
Input Files:
No input file
Output Files:
File Name as defined in the GROUPFILET63 table (should be defined as TRIP_ALL.001)


Page 352 from 698
File is sorted by panelistcode + purchase_date + purchase_place ascending
Parameters:
Static Parameters

Type

and created again.

Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 353 from 698
Rework Management:

Type

P9900384 Temporary Table does NOT exist
available
P1041011 Number of TBSK records Extracted

Page 354 from 698
EHDV21123P – Additional Trip Data Facts
Phase:
This package computes additional trip data facts for particular use in external tools (i.e. Sample
DB mdb or Case extractions).
There is a range of 100 columns in TBSK for the datatypes that are to be used: DT5800 to
DT5899. Only activated datatypes will be computed.
Computation process is modied such that only the validated records are comuted I,e the locked
flag(DT5090=1)
Logic of the job
If job is submitted in REWK mode then all DT58xx fields for the records within the submitted audit
and daterange are initialized to 0
The TRIP_DATAFACTST116 table controls the steps required to perform the job. This table has
the instructions that identify the formula, IBD if required, and Shop group if required. Each
instruction also identifies the column name to be populated in the Total Basket, TBSKTT13, table.
➢ Each instruction contains the following information:

o The name of the data type, a specific column in Total Basket table,
o The Compute flag: Activation Flag (Y=Instruction is Computed, N= Instruction is
Ignored)
➢ The Description of the instruction
➢ The IBD id that resolves to a selected list of Nan_Keys.
➢ The SHOP GROUP that resolves to a selected list of shops based on specific shop
characteristics.
➢ The Formula to be applied to the data.
Based on the submission period and cycle a set of instructions are taken from the
TRIP_DATAFACTST116 table where the Cycle corresponds to the submission cycle and the
Compute flags are ‘Y’.
REWORK mode is driven by DT_SCOPE parameter and it is mandatory for this mode.
O ALL will reset all DT in table and do the processing
o ACTIVE will reset DT only with Compute Flag = Y and then do the processing.
O DTxxxx will only reset referred DT and do the calculation
The following scenarios are possible:

➢ If the IBD is populated then a list of Nan_Keys are selected to be used to filter the
RAWDAILYT60 data, for the submission period, ensuring that only those Nan_Keys are
taken into consideration.
➢ If the SHOP GROUP is populated then a list of shops, with corresponding shop
characteristics, are selected to be used to filter the RAWDAILYT60 data, for the
submission period, ensuring that only those shops are taken into consideration.
➢ If the IBD is empty then all Nan_Keys will be selected from the RAWDAILYT60 data for
the submission period.
➢ If the SHOP GROUP is empty then all shops will be selected from the RAWDAILYT60
data for the submission period.

Page 355 from 698
➢ If the IBD and SHOP GROUP are populated then the RAWDAILYT60 data will be filtered
to only include those Nan_Keys and SHOPs that resolve to the specific IBD and SHOP
GROUP, for the submission period.
➢ If the IBD and SHOP GROUP are empty then there is no filtering and all the data is
included from RAWDAILYT60 data for the submission period.
➢ The formula is then applied or calculated using the set of RAWDAILYT60 data based on
the above filtering criteria. The specific Data Type is then updated with the result of the
calculation in the Total Basket table.
Resolution of scope of product

➢ Retrieve from IBD Group id table instructions related to the IBD selected in the respective
instruction.
➢ Resolve each unique IBD instruction to get a list of Nan_Keys to consider in each
respective calculation.
Resolution of scope of shop

➢ If not already created, the user can create shop groups and select shop characteristics
from the ShopUsabilityT40 table to populate TRIP_SHOPGROUPT117. Then filter further
by adding specific characteristic values to TRIP_SHOPGROUP_VALT119 to each
respective shop group.
➢ Retrieve Shop from ShopUsabilityT40 based on definition of shop group in
TRIP_SHOPGROUPT117 and TRIP_SHOPGROUP_VALT119 combination.
Recommendations:
Submit before the job locking production cycle [EHDV21026P]
Should be submitted after Trip Data Validation
Constraints:
Prerequisites:
- Rawda should be loaded.
Table Description
SHOPUSABILITYT40 Shop characteristics table
TRIP_DATAFACTST116 User defined Instructions
TRIP_SHOPGROUPT117 User defined shop groups with selected shop
characteristic
TRIP_SHOPGROUP_VALT119 User defined shop groups with selected shop
characteristic values
TRIP_FORMULAST118/ Predefined Formulae NOT modifiable by users
TRIP_FORMULAST118_REFACTOR
IBD_CHARST105 Table containing the Item Break Down characteristics
per cycle, IBD
IBD_DEFINITIONT104 Table containing the Item Break Down (IBD) definition
per cycle
IBD_RULEST106 Table containing the Item Break Down (IBD) rules per
cycle, IBD
NAN_CHAR_VALUEST75 NAN CHARACTERISTIC VALUES
CAL_FACTORT05 STATIC PARAMETER TABLE
APP_SQL_LOGT114 SQL Log used by the LATAM Extraction processes for
debugging purposes

Page 356 from 698
Files:
Input files:
No input file.
Output files:
No output file.
Parameters:
Static Parameters:
Parameter Type Value Name(example) Description
Type
PARM DEBUG NO/YES Execution mode for tracing problems
PARM BCDTYP01 1 BCDTYP01 – BCDTYP09 can be
defined.
PARM FILTER_DT5126 YES If value ‘YES’,logically deleted records
are not computed.The default value for
this is ‘YES’
PARM FILTER_DT5090 NO/YES If value ‘NO’, process can be submitted
for period which is still OPEN.
If value ‘YES’, process can be submitted
for period which has already been
CLOSED.
Default Value is NO
PARM DT_SCOPE ALL Values available are ALL,ACTIVE and
DT5800 TO 5896.This parameter is
mandatory in Rework mode
Dynamic Parameters:
Parameter Name Used? (Yes/No) Comments

Period Yes
# Period Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

Type
Process XXXX YYYY.P normal Allowed
PP
Process XXXX YYYY.P F# Not Allowed
PP
Process XXXX YYYY.P PC Not Allowed
PP
Process XXXX YYYY.P Normal Allowed

Page 357 from 698
PP
Process XXXX YYYY.P F# Not Allowed
PP
Process XXXX YYYY.P PC Not Allowed
PP

code
S00C0050 Column not in Column not in S Checks table structure of Open a Ticket
structure structure TBSKT13 to verify required
columns based on
instructions.
P0020308 Error while getting file System error S Generic error while Open a Ticket
lock. accessing a file
P9900245 Rawda tables could System error S RAWDAILYT60 could not be Open a Ticket
not be locked locked
P9900214 Can NOT RELEASE Can NOT S Open a Ticket
Lock. RELEASE
Lock.
P9900384 Temporary Table Temporary S Checks if temporary table
does NOT exist: Table does exists based on dynamic
NOT exist naming convension.
P9900385 Job should be S Job should be submitted
submitted with with debug=N first for
debug=N first for temporary tables to be
temporary tables to available
be available

code
P0000000 Submission period date Date Range Get Date range for submission period
range
P0020608 BCDTYP: List of The list of BCDTYPs defined in
BCDTYPEs CAL_FACTORT05
P9900005 Required number of periods Periods not Information message only. No action required.
not present enough
P0020608 DEBUG: Debug mode Shows the debug mode defined in
CAL_FACTORT05 (YES/ NO)
S0000006 # Records Updated in # Records Number of records updated in TBSKT13 for all
TBSKT13(Step1): Updated in instructions having IBDs and SHOP GROUPS
TBSKT13
TBSKT13(Step2): Updated in instructions having IBDs only
TBSKT13
TBSKT13(Step3): Updated in instructions having SHOP GROUPS only
TBSKT13
TBSKT13(Step4): Updated in instructions having neither IBDs or SHOP
TBSKT13 GROUPS

Page 358 from 698
P9900241 # TBSK Records which Update Cnt – Number of TBSK records which their DT58xx
DT58xx 359anellist359 to 0 : TBSK have been initialized to 0 when job is submitted in
REWK mode
List of Formulae common to all countries,

TRIP_FORMULAST118/TRIP_FORMULAST118_REFACTOR:
ID AC_TITLE AC_DESCRIPTION
1 TRANSACTIONS Number of transaction in basket
2 UNITS number of units
3 UNITS (excl VV) number of units considering variable weight as 1
4 VALUE total value
5 UNITS PROMO1 total units for items bought on promo1
12 UNIT ANY PROMO total units for items bought under any promo
13 VALUE PROMO1 total value for items bought on promo1
20 VALUE ANY PROMO total value for items bought under any promo
21 UNITS PROMO1 (excl VV) total units for items bought on promo1 variable weight as 1
UNIT ANY PROMO (excl total units for items bought under any promo weight as 1
28 VV)

Page 359 from 698
EHDV21131P – Formal Telecom Price Validation
Phase:
This package checks for missing prices (DT5675 = 0) and replaces them with the median price,
from table Telecom_PriceT120, which has been calculated using Median Telecom Price
Computation (EHDV21134P).
Logic of the job
For each expense type, for each analysis level, all the records with price (DT5675) = 0 are
identified. If a value exists in the median price table, Telecom_PriceT120, for the specific expense
type and analysis level then replace the value (DT5675) with price from Telecom_PriceT120. Also
set the appropriate value, 20, into the price update flag data type, DT5904.
Explained in another way:
For each cycle / period / expense type / nan

Select records having DT5675 = 0 or DT5667 = 0
For each analysis level
Check for existing replacement price in Telecom_PriceT120
If found then
Set price modification flag to 20
Set DT5667 to replacement price value from Telecom_PriceT120
Set DT5675 to replacement price value from Telecom_PriceT120
The analysis level setup can use family chars and/or nan chars to group as follow:
Family Char Nan Char Grouping at
Present Present Family Char value * Nan Char value
Present Missing Family Char value * Nan
Missing Present All Family * Nan Char value
Missing Missing All Family * Nan
Recommendations:
Constraints:
Prerequisites:
- Median Telecom Price Computation (EHDV21134P) processed.
Table Description
TELECOM_PRICET120 Computed median and quartiles prices from rawda
FAMILYUSABILITYT41 FAMILY CHARACTERISTIC VALUES
P359_1T12 Staging table which contains all the purchase records that
are eligible to go with Phase1 jobs

Page 360 from 698
Files:
Input files:
No input file.
Output files:
FILEO1 – containing values with no price after process.
5 7 PERIOD Submission period
12 16 ITEM Nan_Key
28 30 BARCODE Barcode
58 10 SHOP Shop
68 4 AC_TIME Expense Type
Parameters:
Static Parameters:
Parameter Type Value Description
Type
PARM AL1_HH_CHAR family char id for analysis level 1
PARM AL1_NAN_CHAR nan char id for analysis level 1
PARM AL2_HH_CHAR family char id for analysis level 2 (optional)
PARM AL2_NAN_CHAR nan char id for analysis level 2 (optional)
PARM DT_REPLACE datatype code of the replacement price to be used 2
PARM EXP_TYPE_n expense type to process (n could from 1 to 10)
PARM MINOBS minimum number of values required for computation
Dynamic Parameters:

Period Yes submission with period 0 is not allowed for this job
# Period Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
2 Should normally be the median telecom prices computed by EHDV134P – DT5853

Page 361 from 698
Rework Management:

Type
Process XXXX YYYY.PPP F# Not Allowed
Process XXXX YYYY.PPP PC Not Allowed
Rework XXXX YYYY.PPP Normal Not Allowed
Rework XXXX YYYY.PPP PC Not Allowed

code
P0020505 No HHOLD Analysis Static Value S Static Value not present
LVLs DEFINED as not present
STATIC parameter
P0020505 No NAN Analysis LVLs Static Value S Static Value not present
DEFINED as STATIC not present
parameter
P0020505 No Expense Types Static Value S Static Value not present
parameter
P0020505 No DT_REPLACE Static Value S Static Value not present
parameter
P0020505 Missing Levels between Static Value S Static Value not present
NAN LVLs and HH LVLs not present
P0020505 WRONG DEFINITION Static Value S
DT_REPLACE not present
PARAMETER
P1026005 Processing option not Rework not S
Correct – Rework not required
required

Page 362 from 698

code
P0000000 Submission period date range Date Range Get Date range for
submission period
P1029010 Period Selected Period Period used for Price
Selected Validation
P9900005 Required number of periods not present Periods not Information message only.
enough No action required.
P0020608 HHOLD Analysis LVL(1-3) : CITY Static Value List of HHOLD Analysis
Levels in CAL_FACTORT05
P0020608 NAN Analysis LVL(1-3) : 10022 Static Value List of NAN Analysis Levels
in CAL_FACTORT05
P0020608 Expense Type (1-10): 0001 Static Value List of Expense Types in
CAL_FACTORT05
P0020608 DT_REPLACE: 5853 Static Value Replacement datatype
P9900388 The # purchase acts for EXP_TYPE(1-10) Analysis # Values Expense type value (1-10)
LVL: (1-10) is:0 Computed
P9900388 The # HHOLDs for EXP_TYPE1-10) Analysis # Values Expense type value (1-10)
LVL: (1-10) is:0 Computed
P9900388 The # NANs for EXP_TYPE1-10) Analysis LVL: # Values Expense type value (1-10)
(1-10) is:0 Computed
P1043007 Total Number of records extracted into the o/p file Record Count

Page 363 from 698
EHDV21132P – Validate Telecom Prices against Catalogue prices
Phase:
This package validates prices against catalogue prices stored in IMDB characteristic.
Logic of the job
Loop for each EXP_TYPE defined in static parameters.
Step 1 : Get values under minimum catalogue price

Get list of nans for the cycle / period
Get catalogue price from IMDB char CAT_PRICE description for each nan
Get rawda rows for which prices are lower than CAT_PRICE
Step 2 : Update values and set modification flag

Update DT5667 with CAT_PRICE on selected rows
Update DT5675 with CAT_PRICE on selected rows
Update DT5904 with value = 19
Insert rows in family_log for the changes
Final Step :
Report number of purchase acts updated
Report number of families updated
Report number of nan updated
Report logs and messages
Recommendations:
Constraints:
Prerequisites:
- Availability of data in rawda for submission period
- IMDB load processed
- Family char load processed
Table Description
FAMILY_LOGT67 Table containing the list of changes applied by DV automatic
validation jobs + details on family collaboration (holidays.)
RAWDAILYT60 Purchase Rawda table containing purchase transactions at
barcode level

Page 364 from 698
Files:
Input files:
No input file.
Output files:
No input file.
Parameters:
Static Parameters:
Parameter Type Value Name(example) Description
Type
PARM EXP_TYPE_n 0001 expense type to process (n could be 1 to
10)
PARM CAT_PRICE 10022 Catalogue Price
Dynamic Parameters:

# Period Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

Type

code
S00C0050 Column not in Column not in S Checks that all datatypes Abort process
structure struct defined in table
DATACODET20 actually
exist as columns in
RawdailyT60
P0020308 Can NOT obtain System Error S Error while getting File Lock Abort process
Lock

Page 365 from 698
P9900245 Rawda tables could Can not lock S
not be locked Rawda t
P1026005 Processing option Rework not S
not Correct – Rework required
not required
P9900214 Can NOT RELEASE Can NOT S
Lock. RELEASE Lock
P9900155 No Records Selected No records found S List Expense Types in
for Processing for Cal_FactorT05
EXP_TYPE: 0001

code
submission period
P0020608 EXP_TYPE Vals: 0001 Static Value List of Expense Types in
CAL_FACTORT05
P0020608 CAT_PRICE : 10022 Static Value Catalogue price in
CAL_FACTORT05
P9900281 # Purchase Transactions Updated: 11088 Number of # of purchase transactions
purchase a updated
P1012008 # Families Updated: 2282 Distinct family # of families updated
Cnt
P9900206 # Nans Updated: 5296 Number of # nans updated
NANS selec

Page 366 from 698
EHDV21133P – Automatic Telecom Price Validation
Phase:
This package validates telecom prices according to country setup. It validates current prices
delivered by families using median and quartiles prices stored by package EHDV21134P into
TELCOM_PRICET120, a new table.
Logic of the job
Check for Process Conditions – Parameters3 ; if not OK then Exit

If rework mode set back prices to historical prices value for submission period
For each cycle / period / expense type

Get comparison prices values from check method
Get correction prices values from correction method
Select rows having check failure4
Apply correction method to failed rows
Set DT5904 = 21 on selected rows
Set DT5675 = correction price value on selected rows
Process consistency check for expense type = 0
Apply check formula and select faulting rows
Apply correction to selected rows
For each expense type based on the analysis group setup and check method definition the
appropriate statistic value is retrieved from the TELCOM_PRICET120 table for the cycle/period
value of the expense type and characteristic of the shop and item.
Recommendations:
Constraints:
Prerequisites:
- Successful completion of EHDV21134P process
3Value of parameters checks + coherence should be performed. If any check fail job is
stopped.
4 By default check is current price > comparison price

Page 367 from 698
Table Description
barcode level
CYCLEPERIODT03 Table containing all period definitions (date from, date to,
lock flag) by Cycle
Files:
Input files:
No input file.
Output files:
No input file.
Parameters:
Static Parameters:

Type
PARM EXP_TYPE_n expense type to process (n could from 1 to 10). There
must be at least 2 EXP_TYPEs.
PARM CHECK_METHOD_n check method to apply for expense type n
PARM CORR_METHOD_n correction method to apply for expense type n
PARM P_ n Optional: parameter for Inter Quartile computation
PARM ET_CMP_ n Optional: Expense type to compare (when needed)
PARM SINGLE_ n Mandatory: Defines if expense type needs to be alone
then Y or not then N when checking.
PARM REPLICATE_ n Optional: Defines if correction is replicated on expense
type 05
PARM CC_EXP_TYPE_ n Expense type to use for consistency check (n from 1 to
10). There must be at least 2 CC_EXP_TYPEs.
PARM CC_OPERATOR_ n Operator to apply on expense type for consistency check
n. There must be a CC_OPERATOR for each
CC_EXP_TYPE.
5When replicated the impact should be the same, so value will be incremented by the same
difference ET0 corrected = ET0 + (ETn corrected – Etn )

Page 368 from 698
Static Parameter Method Values:
Methods Values Description
Check Methods : Comparison is done with quartile3 for same analysis level group
GT_Q3 (DT5855). Method is using Analysis level setup
Comparison is done with interquartile (Q3-Q1) range using
additional P parameter. DT5853 + ((DT5855-DT5854)*P).
GT_IQ Method is using Analysis level setup
Comparison is done with other expense type using additional
parameter. ET_CMP_n. Method is not using Analysis level
GT_ET setup.
Corrections methods Use quartile3 as replacement value (DT5855). Analysis level
: Q3 check method required.
Use median as replacement value (DT5853) Analysis level check
MDN method required.
Use inter quartile range : DT5853 + ((DT5855-DT5854)*P)
IQ Analysis level check method required.
Use average rate between 2 expense type. Average computed
as preliminary step for each item having more than 3
observations and at total level. Average (EXP_TYPE_n /
ET_CMP). Method computes new values from the item value
when present else total value, with the following formula:
ET_CMP * average rate.
AVG_RATE
Consistency Check Sum related expense type to total values
Operators : +
- Subtract related expense type to total values
Dynamic Parameters:

# Period Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 369 from 698
Rework Management:

Type

code
S00C0050 Column not in structure Column not in S Checks that all datatypes Abort process
struct defined in table
DATACODET20 actually
exist as columns in
RawdailyT60
P0020505 No CORRECTION Stat.Val not S Static Value not present
METHOD DEFINED as present
STATIC parameter
P0020505 No HHOLD Analysis LVLs Static Value S Static Value not present
parameter
parameter
P0020505 No Expense Types Static Value S Static Value not present
parameter
P0020505 Wrong Correction Method is Stat.Val not S Static Value not present
defined: (1-10) present
P0020505 No CHECK METHOD Stat.Val not S Static Value not present
DEFINED as STATIC present
parameter
P0020505 wrong definition of NAN Stat.Val not S Static Value not present
LVLs as STATIC parameter present
P0020505 Wrong Check Method is Stat.Val not S Static Value not present
defined: (1-10) present
P0020505 Static Parameters are NOT Stat.Val not S Static Value not present
OK present
P0020505 ET_CMP_(1-10) HAS NOT Stat.Val not S Static Value not present
BEEN DEFINED present
P0020505 Missing Levels between Stat.Val not S Static Value not present
NAN LVLs and HH LVLs present
P0020505 No SINGLE option for Stat.Val not S Static Value not present
Expense Type DEFINED as present
STATIC parameter
P0020505 P_(1-10) has not been Stat.Val not S Static Value not present
defined’ present

Page 370 from 698
P0020505 Wrong definition of HHOLD Stat.Val not S Static Value not present
LVLs as STATIC parameter present
P1021004 No records selected for No record S No record selected for
validation check of selected f validation
EXP_TYPE (1-10) using ‘
(1-10)
correction for selected f validation
REPLICATION
correction of EXP_TYPE (1- selected f validation
10) using ‘ (1-10):
P1021004 No records selected from No record S
RAWDAILYT60 selected f
P9900220 Cycle/Period not found in Source Cycle S
CYCLEPERIODT03 table and sou
P9900225 Processing option and Processing S
period lock NOT OK option an
P1021004 No records selected for No record S
validation check of selected f
EXP_TYPE(1-10) using ‘ (1-
10):
P1009004 There are records for No keys S No keys available for
REPLICATION with no available processing
corresponding keys for
EXP_TYPE=0

code
submission period
P0020608 SINGLE option (1-10): N Static Value List of Single options in
CAL_FACTORT05
P0020608 Replicate (1-10) Static Value List of Replicate options in
CAL_FACTORT05
P0020608 Check Method (1-10): GT_Q3 Static Value List of Check methods in
CAL_FACTORT05
in CAL_FACTORT05
CAL_FACTORT05
P0020608 Consistency Check Expense Type (1-10): Static Value List of Consistency checks
Expense Types in
CAL_FACTORT05
P0020608 Consistency Check Operator (1-10): Static Value List of Consistency checks
Operators in
CAL_FACTORT05
P0020608 Static Parameters are set OK Static Value
P0020608 Correction Method(1-10): Static Value List of Correction Methods in
CAL_FACTORT05
P9900005 Required No. of Periods not Present Periods not

Page 371 from 698
enough
P9900224 Processing option and period lock OK Processing
option an
P9900388 The # purchase acts for EXP_TYPE: (1-10) is: (1- # Values
10): Computed
P9900388 The # NANs for EXP_TYPE: : (1-10) is: (1-10): # Values
Computed
P9900388 The # HHOLDs for EXP_TYPE: (1-10) is: (1-10): # Values
Computed
P9900388 The # purchase acts corrected after Consistency
Check has been performed
P9900388 The # HHOLDs impacted after Consistency Check
has been performed:
P9900388 The # NANs impacted after Consistency Check has
been performed:
S00C0015 No records selected from RAWDAILYT60 for No rows No rows selected for
EXP_TYPE_(1-10) : (1-10) selected for confirmation

Page 372 from 698
EHDV21134P – Median Telecom Price computation
Phase:
This package computes the median telecom prices according to country setup. It computes
median and quartiles prices from rawda into TELECOM_PRICET120, a new table.
Logic of the job
For each cycle / period / analysis level / expense type:

Select rawda information by grouping level
Compute median and store in TELECOM_PRICET120 for groups having MINOBS or
more rows
Compute quartile 1 and store in TELECOM_PRICET120 for groups having MINOBS or
more rows
Compute quartile 2 and store in TELECOM_PRICET120 for groups having MINOBS or
more rows
Recommendations:
Constraints:
Prerequisites:
- Availability of data in rawda for submission period
- IMDB load processed
- Family char load processed
Table Description
barcode level
Files:
Input files:
No input file.

Page 373 from 698
Output files:
No input file.
Parameters:
Static Parameters:
Type
PARM EXP_TYPE_n expense type to process (n could from 1 to 10)
PARM MINOBS minimum number of values required for computation
Dynamic Parameters:

# Period Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

Type
Rework is allowed: if invoked first step will be to remove previous value for the same submission
period.

Page 374 from 698
code
P0020505 No HHOLD Analysis LVLs Static Value S Static Value not present
parameter
parameter
P0020505 No Expense Types Periods not S Information message only.
DEFINED as STATIC enough No action required.
parameter
P0020505 No MINOBS DEFINED as Static Value S Static Value not present
STATIC parameter not present
P0020505 Missing Levels between Static Value S Static Value not present
NAN LVLs and HH LVLs not present
P9900114 REWORK required. Records Rework S Production cycle already
in TELECOM_PRICET120 requested completed-Rework mode
table for requested
:’||g_jobsub_info.CYCLE||’
‘||g_jobsub_info.period

code
submission period
P9900005 Required number of periods not present
in CAL_FACTORT05
CAL_FACTORT05
P0020608 MINOBS: (3) Static Value number of Minimum
Observations in
CAL_FACTORT05
P9900388 The # Values computed for EXP_TYPE1-10) # Values Expense type value = 1-10
Analysis LVL: (1-10) is:0 Computed

Page 375 from 698
EHDV21135P- Data Selection by exception (based on barcode list)
Phase:
This process is based on EHDV21068 and works similarly except that only specific Barcodes are
selected for processing. These barcodes are placed in an input file, which is used to drive the
data selection for the process.
Please note that EHDV21068 cannot be replaced by EHDV21135. EHDV21135 is intended to

allow users to insert back in rawda several items which have been cross-coded recently (using
88P).
This program selects the set of data to be processed by the criteria defined by the user and it
distributes the other data to the related tables.
For setting prices in the loading area, when loading the price from ALLT11 the following cases
are validated
★ Case When the Price field in the file is empty then let the current DATAVALUE
★ Case When the Price field in the file is filled but the price is already present in the
loading area for the corresponding barcode then let the current DATAVALUE
★ Case When the Price field in the file is filled and the price is missing in the loading area
then load into the temp table the price from the file to the corresponding barcode
Only valid purchase data are moved to P359_1T12 and KEYST14, this means:
• keys Xcoded
• keys not present in the new item table
• date of the purchase transaction consistent with the activity dates of the pc-item
inside the IMDB ( dt_val_from <= key date <= dt_val_to )
• panelist code present in Familyusability table
• shop code present in Shopusability table
• keys not present in validation area
While moving the DT5095 data it is inserted as -1 for APV rerun purpose.
If the key is already in validation area, we do not move any data of the same basket in T-359 until
the validation has been completed, the rawda loaded and the data already validated deleted from
T-359, T- ALL and KEY table
• Allt11
• Prockeyst29
• Rawdailyt60
• Tbskt13
• Familyusabilityt41
• Prockeyst29
• Rawdailyt60
• Tbskt13

Page 376 from 698
File number concept :
This is NOT allowed and is different from EHDV21068.
Standard processing mode :
We always load the full basket of the purchase transactions for each family in Table ALL, for
standard load process.
Rework is NOT allowed. This is different from EHDV21068.
Recommendations:
Constraints:
Prerequisites:
Data should be loaded in ALLT11 and Xcoded

HH + SHOP information should be loaded
Table Description
Files:
Input Files:

Page 377 from 698
Barcode data record layout :
36 7 PRICE XXXX.XX NUMBER
Output Files :
No output file.
Parameters :
Static Parameters
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No This job is creating a Timestamp
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

Barcode data record layout :

36 7 PRICE XXXX.XX NUMBER

Page 378 from 698

code
Info and DT
definition
screen
P0020125 File Name is blank File Name is S Barcode Data Input
blank File name is missing
P0020126 File Name not present FileName S Barcode Data Input
not present File name is not
present
Control Process Option
P9900114 Production cycle already Rework S Rework is not
completed-Rework mode requested allowed
requested
P1009002 No records selected from No records S
table ALL for the Audit + selected
Period
P0020578 Error while generating System Error S Open QLS
given Tag List
Process Audit Period

Records loaded in to
Keys table
loaded in to Keys
table
P1009038 Total Number of PCs PC Count Information message only. No action required
loaded in the Keys
table
selected

Page 379 from 698
Records selected
P9900305 # of Records already # Recs Warning message indicating that the selected
present in output Different F# records have already been processed
tables with different previously. No action required
File #:
P9900306 # of Records excluded # Recs Warning message indicating that the selected
from Keyst14 due to Excluded records have already been processed
basket already in previously. No action required
output tables:
created.
P9900264 Output Table #: Checked Information message only. No action required
Partition * Subpartition Partition
% checked

code
following name:
ended JOB required
displayed:
displayed:

Page 380 from 698
EHDV21136P- Family Usability Extraction from Sample DB
Phase:
This process is new and it is an extraction process that extracts the family usability flag from
Sample DB link.
An output file is created based on static parameters which are set up by the user.
Rework is NOT allowed.
Recommendations:
Constraints:
Prerequisites:
The SAMPLEDB Dblink must be available to extract the necessary data.
Table Description
collabcsamplet5a Usability Family flags
Files:
Input Files:
No input file.
Output Files :
FILEO1: Family usability:

1 10 FamilyID Family ID
11 1 UsabilityFlag Usability Flag

Page 381 from 698
Parameters:
Static Parameters

Type
PARM SDB_INDEX Mandatory – Sample DB index code
PARM SDB_CS Mandatory – Sample DB Common sample code.
PARM SDB_PERIOD Mandatory – Sample DB period code
PARM SCRIPT_CMD Check Appendix for Optional – Automation script file name.
Information
Dynamic Parameters

Name (Yes/No)
# Periods Yes Always 1 so would be ignored if different.
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management :

Process XXXX YYYY.PPP F# Not allowed, if submitted using this mode it will
automatically be switched to normal mode
Process XXXX YYYY.PPP PC Not allowed, if submitted using this mode it will
Rework XXXX YYYY.PPP normal Not allowed, if submitted using this mode it will
Rework XXXX YYYY.PPP F# Not allowed, if submitted using this mode it will
Rework XXXX YYYY.PPP PC Not allowed, if submitted using this mode it will
None.

Page 382 from 698

code
-20999 Program Terminated – System Error S No Audit defined for Check
Problem in retrieving the submitted cycle dtparamters
Audit For the given Cycle-
P9900332 SAMPLEDB DBLINK SAMPLEDB S SAMPLEDB
does not exist DBLINK DBLINK does not
does exist
P0020125 File Name is blank File Name is S
blank
P0020126 File Name not present File Name S
not present
P1027017 Error in retrieving the System Error S No records retrieved
Family Char from Sample DB
P1033015 Unhandled error occurred System Error S
while dropping temp
tables

P0000000 Date Range Date Range Information message only. No action required
Information
P1042001 Job cannot be Invalid Would always be 1 so would be ignored if
submitted with Instruction different. A warning is generated to inform the
#periods back > 1 user.
P0020608 SDB_INDEX: Static Value Provides the value of submitted SDB_INDEX:
P0020608 SDB_CS: Static Value Provides the value of submitted SDB_CS:
P0020608 SDB_PERIOD: Static Value Provides the value of submitted
SDB_PERIOD:
P0020608 SCRIPT_CMD: Static Value Provides the value of submitted
SCRIPT_CMD:
P9900333 DBLINK name DBLINK: Provides the name of the DBLINK.
P9900142 # Records Read: Number of Number of records read
records re
P9900369 # Records Exported: # Exported by Number of records exported
Chain

Page 383 from 698
EHDV21137P – Variable Weight Volume Validation / IMDB Max
Phase:
Main part of process of this new job is similar to existing Variable Weight Volume Validation (84P)
with few additional features that detect and correct erroneous values for Volume or Price due to
decimal position shifts.
These new features are as follows:

- Volume check for rescaling in min/max range when actual value outside of scope
- Volume correction impact on price when PRICE_SCALING set ON
- Price check for scaling ratio again std price when PRICE_SCALING set ON
- Volume correction if VOLUME_SCALING set ON..
Objective of this procedure is to validate units and price for variable weight items and apply
- For variable weight products situation is different as 384anellist384 are required to

provide a volume (in litre or kilogram), value can be decimal in following range of value
[0.001 , 99.999].
- High quantity are detected using a fixed maximum volume instructed in IMDB at
Nan_key level (maximum volume is stored in char value description)

Page 384 from 698
STEP 1: Select all Nan_key variable weight present in Rawda for submission period
List of variable weight nan is retrieved from table NAN_VAR_WEIGHT_FROM_IMDB based on

the NANS present in RAWDA for submission period and the PRODUCT SCOPE (IBD) defined in
the static parameter table.
STEP 2: Select minimum volume value by Nan_key from EIMDB

divided by 1000)
STEP 3: Select maximum volume value by Nan_key from EIMDB
Nan_key maximumum volume is retrieved from NAN_CHAR_VALUEST75 ( in IMDB char

divided by 1000)
std_volume ( t ) =
➔w=n/(n+k)
NUM_PER2 =4



NUM_PER1] is null

Page 385 from 698
STEP5: Rescaling Validation
If PRICE_SCALING is ON STEP 5a : get standard price (Per-1) by Nan_key, price group

If PRICE_SCALING is ON STEP 5b : check for price against standard price
If VOLUME_SCALING is ON STEP 5c : check for volume scaling against min / max

and with
Corrected_quantity=

std _ volume − ( * volume _ before _ validation 2 )
min_ volume 2
std _ volume
Corrected_quantity=
1−

Recommendations:
Rework:

Page 386 from 698
No rework option
Constraints:
Prerequisites:

- Maximum volume should be instructed in IMDB
-
- IMDB for minimum volume and maximum volume
Table Description

Files:
Input Files:
No output file
Output Files:
CSV delimited format with comma as delimiter.
Following layout :
Audit
Purchase Date
Purchase Time
Household ID
Shop ID
Barcode
Nankey
Initial price
Standard price
Price Group (used to get standard price)
Initial Volume
Price Adjustment factor (from PRICE_SCALING)
Price Correction Flag (Y or N)

Page 387 from 698
Corrected Volume
Corrected Price
Min Volume
Max Volume
Volume correction flag (Y or N)
Volume Adjustment Factor (from VOLUME_SCALING)
Item Description
Parameters:
Static Parameters

Type
PARM CHAR_MIN_VOL 1111 EIMDB char code for minimum volume (by
Nan_key)
PARM NUM_PERIOD1 1 Number of previous period to consider to
compute median volume (current period)
compute median volume (previous period)
PARM MAX_LCOR 0.05 Limit for ratio of variable weight purchase acts
corrected due to low volume (at Nan level)
PARM CHAR_MAX_VOL 2222 EIMDB char code for maximum volume (by
Nan_key), optional, if not present step 7 is not
applied
PARM PRICE_SCALING ON/OFF If not defined an error message is given
PARM VOLUME_SCALING ON/OFF If not defined an error message is given
PARM PRODUCT_SCOPE Set IBD defining Product Scope
PARM PRICE_RATIO Set ratio for PRICE comparison.
For example for 15% set is as 0.15
PARM PRICE_GROUP Price Group CharId to be used by the
process.
If Price Group is not defined an error
message is raised.
PARM RUN_MODE NORMAL/DETAILS/ If not defined NORMAL mode is assumed.
ANALYSIS
NORMAL – No output file is created
DETAILS – ALL records candidate for
Volume or Price Correction are exported.
ANALYSIS – Only records being corrected
are exported
Dynamic Parameters

Name (Yes/No)
# Periods No

Page 388 from 698
Single Run Job Yes
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


code
limit for given NAN
FAMILY_LOGT67
period.

w
rawdata
wi

Page 389 from 698
max_volume Records
wi
FAMILY_LOGT67:
DT5642 reset to
initial values:
from
FAMILY_LOGT67:
P9900281 # Purchase acts Number of records with Volume adjusted in
with Volume Rescaling step
adjusted(Rescaling)
:
P1039008 #Records exported #Records exported in <run_mode> OUTPUT File:
P1370001 Number of Records # Information message only. No action required
with VOLUME Records
changed updated
with PRICE Records
changed updated

Page 390 from 698
EHDV21137S – Variable Weight Volume Validation / Computed Max
Phase:
Main part of process of this new job is similar to existing Variable Weight Volume Validation (84P)
with few additional features that detect and correct erroneous values for Volume or Price due to
decimal position shifts.
These new features are as follows:

- Volume check for rescaling in min/max range when actual value outside of scope
- Volume correction impact on price when PRICE_SCALING set ON
- Price check for scaling ratio again std price when PRICE_SCALING set ON
- Volume correction if VOLUME_SCALING set ON..
Objective of this procedure is to validate units and price for variable weight items and apply
- For variable weight products situation is different as 391anellist391 are required to

provide a volume (in litre or kilogram), value can be decimal in following range of value
[0.001 , 99.999].
- High quantity are detected using a dynamic maximum volume limit computed by the
program on a logic closed to the one used in Purchase Inspection job (16P). This
validation is applied only if there are more than T observations at Nan_Key level

Page 391 from 698
STEP 1: Select all Nan_key variable weight present in Rawda for submission period
List of variable weight nan is retrieved from table NAN_VAR_WEIGHT_FROM_IMDB based on

the NANS present in RAWDA for submission period and the PRODUCT SCOPE (IBD) defined in
the static parameter table.
STEP 2: Select minimum volume value by Nan_key from EIMDB

divided by 1000)
STEP 3: Select maximum volume value by Nan_key from EIMDB
Nan_key maximumum volume is retrieved from NAN_CHAR_VALUEST75 ( in IMDB char

divided by 1000)
STEP 3: Compute maximum volume limit by Nan_key
Select Nan_key with at least T observations.

For each selected Nan_key,
select the max(T, CEIL(number of observations x RATIO)) which have the largest
volumes x
Example: T=5 and RATIO=0.1

if number of observation = 20
If number of observation = 150
Determine the critical limit as

max_volume = [MIN_VAL + (MEDIAN_VALUE – MIN_VAL) * Log (ALPHA) /
Log (0.5)]
Where MIN_VAL = Minimum(x) volume of L observations selected

MEDIAN_VALUE = Median(x) volume of L observations selected
std_volume ( t ) =

Page 392 from 698
➔w=n/(n+k)
NUM_PER2 =4



NUM_PER1] is null
STEP5: Rescaling Validation
If PRICE_SCALING is ON STEP 5a : get standard price (Per-1) by Nan_key, price group

If PRICE_SCALING is ON STEP 5b : check for price against standard price
If VOLUME_SCALING is ON STEP 5c : check for volume scaling against min / max

and with
Corrected_quantity=

Page 393 from 698
std _ volume − ( 2
* volume _ before _ validation 2 )
min_ volume
std _ volume
Corrected_quantity=
1−

STEP8: Validation of high volumes using fixed IMDB high volume limit from IMDB, for nan_key
without enough observations. This optional step is applied only if static parameter
CHAR_MAX_VOL is instructed
We select here all nan_key not corrected due to volume too low, and with max_volume null (due
to not enough information)
For each record with Volume > imdb_max_volume
If imdb_min_volume < Volume / 10 < imdb_max_volume

else
else
else
corrected volume= (imdb_min_volume + imdb_max_volume) /2

Update Price correction flag (DT5904) = 19
Recommendations:
Rework:
No rework option
Constraints:
Prerequisites:

- Maximum volume should be instructed in IMDB
-

Page 394 from 698
- IMDB for minimum volume and maximum volume
Table Description

Files:
Input Files:
No output file
Output Files:
CSV delimited format with comma as delimiter.
Following layout :
Audit
Purchase Date
Purchase Time
Household ID
Shop ID
Barcode
Nankey
Initial price
Standard price
Price Group (used to get standard price)
Initial Volume
Price Adjustment factor (from PRICE_SCALING)
Price Correction Flag (Y or N)
Corrected Volume
Corrected Price
Min Volume
Max Volume
Volume correction flag (Y or N)
Volume Adjustment Factor (from VOLUME_SCALING)
Item Description
Parameters:
Static Parameters

Page 395 from 698
Type
PARM CHAR_MIN_VOL 1111 EIMDB char code for minimum volume (by
Nan_key)
PARM T 7 Minimum number of observation to compute
max_volume
PARM RATIO 0.1 Define size of right tail to consider in
max_volume computation in case of high
number of observation
PARM ALPHA 0.5 Statistical parameter to compute max_volume
compute median volume (current period)
compute median volume (previous period)
PARM MAX_LCOR 0.05 Limit for ratio of variable weight purchase acts
corrected due to low volume (at Nan level)
PARM CHAR_MAX_VOL 2222 EIMDB char code for maximum volume (by
Nan_key), optional, if not present step 7 is not
applied
PARM PRICE_SCALING ON/OFF If not defined an error message is given
PARM VOLUME_SCALING ON/OFF If not defined an error message is given
PARM PRODUCT_SCOPE Set IBD defining Product Scope
PARM PRICE_RATIO Set ratio for PRICE comparison.
For example for 15% set is as 0.15
PARM PRICE_GROUP Price Group CharId to be used by the
process.
If Price Group is not defined an error
message is raised.
PARM RUN_MODE NORMAL/DETAILS/ If not defined NORMAL mode is assumed.
ANALYSIS
NORMAL – No output file is created
DETAILS – ALL records candidate for
Volume or Price Correction are exported.
ANALYSIS – Only records being corrected
are exported
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job Yes
Timestamp No
PC Selection No
F# Selection No
DT Selection No

Page 396 from 698
File Selection No
Rework Management:


code
P9900277 Not enough records for Not enough W
NAN, max volume records f
validation not applied
limit for given NAN
FAMILY_LOGT67
period.

w
rawdata
wi
max_volume Records
wi

Page 397 from 698
FAMILY_LOGT67:
DT5642 reset to
initial values:
from
FAMILY_LOGT67:
P9900281 # Purchase acts Number of records with Volume adjusted in
with Volume Rescaling step
adjusted(Rescaling)
:
P1039008 #Records exported #Records exported in <run_mode> OUTPUT File:
with VOLUME Records
changed updated
with PRICE Records
changed updated

Page 398 from 698
EHDV21138P- Private Brand Validation
Phase :
The “private brand” validation is checking the consistency between the “retailer” where the family
went shopping with the manufacturer of the products included in the shopping basket.
Whether we find an inconsistency and we identify a “private brand product” associated to the
wrong retailer, we apply an automatic correction and we re-classify the purchase transaction into
a new basket where we build the consistency between the “retailer” and the “manufacturer” of the
product (retailer itself).
This job combines the functionality of the existing EHDV21019P and EHDV21020P in a single
job.
o Get the Product Scope from TDV_PB_ITEM_CHARS
o PLC_FL_SENSITIVE = ‘Y’ ➔ Private Brand values will be taken directly from

CHAR_VALUEST24 for this char (PLC_NC_CHAR_ID) with specific condition
FL_SENSITIVE = ‘Y’
o PLC_FL_SENSITIVE = ‘N’ ➔ Private Brand values will be taken from

CHAR_VALUEST24 for this char (PLC_NC_CHARID) with specific condition
using additional static parameters (PL1C / CHVAL01 & CHVAL02)
o From this extract, filter the items related to private brand (having char values
sensitivity flag = Y), get private brand value at the same time in temporary table.
o Detect private brand for which no setup is done and inform user (Warning level)
o Perform validation (same as 20P).
o The validation of the Private Brands will be applied to a limited set of purchase
transactions by the following criteria:
o we will inspect only the shop types defined in PRIVATELABEL_BASKETT72

o we will inspect only the private brands defined in PRIVATELABEL_SHOPT22

Page 399 from 698
Shop types:
User selects the list of the shop types by an on line screen (Validation → Private
Brand → Shop Type)
Shop/Shop Type instruction are stored in shop usability table.
One or more shop types can be selected.
Private brands
 user can select the list of the private brands to be inspected by an on line
screen (as instructed in Validation → Private Brand → PB)
 The GUI will allow user to select private brand values according to
sensitivity flag values
 New GUI should allow user to select different PB chars for the setup
When we identify an inconsistency between a product and the characteristics of

the shops to which the product has been assigned, we reclassify the purchase
transaction by the following rules:
if
➢ a shopping basket doesn’t contain retailer products , we stop the
inspection
➢ a shopping basket contains a product ( or more than one ) belonging to the
same retailer and the chain code of the shop of the shopping basket is not
representing the retailer of the products , we change the shop code of the
whole basket selecting a new shop code having the same shop type and
the shop chain of the retailer owning the PB products.
The shop code used for reclassification is instructed in following online

screen: Validation → Private Brand → PB Shop.
➢ shopping basket contains more than one PB products belonging to

different retailers ( more than one ),
➢ we open a new shopping basket ,
one for each retailer ( the new shop has the same shop type and the shop
chain of the retailer) and we move to this basket only the purchase
transactions belonging to that retailer
The new shop code used is instructed in following online screen:
Validation → Private Brand → PB Shop.

Page 400 from 698
We leave all the original purchase transactions (manufacturer only) into
the original basket with the same original shop.
➢ the purchase transaction re-classified into new baskets are deleted

from the original baskets
➢ the total basket information are not modified and they will be adjusted
later in the rawda by a dedicated procedure ( basket adjustment ,
basket validation )
➢ All changes applied are logged in FAMILYLOGT67 table and can be checked
using FamilyLog online screen.
➢ Report changes in TEXT TAB delimeted external file.
Recommendations :
Constraints :
Prerequisites :

Table Description
TDV_PB_ITEM_CHARS List of Char IDs with their sensitivity to be used by the PB process
PRIVATE_LABELT21 Data as defined in screen :Validation Private Brand PB
PRIVATELABEL_SHOPT22 Data as defined in screen :Validation Private Brand Shop

Page 401 from 698
PRIVATELABEL_BASKETT72 Data as defined in screen :Validation Private Brand Shop type

It contains the characteristic identifier and the characteristic value by
SHOPUSABILITYT40
Files:
Input Files:
No input file.
Output Files:
2 Files
1. Shop Reclassification Report.
This report is a TAB delimited report that contains all the changes (shop
reclassification) performed by the process.
Description
- Retailer
- Retailer Name
- Brand
- Nankey
- Barcode
- Date
- Time
- Household
- Original Shop
- Original Shop Name
- New Shop
- New Shop Name
2. Corrected Items by Brand Report
This report is SEMICOLON delimited and contains all the corrected items by brand.
Description
- Chain
- Chain Description
- Brand
- Brand Description
- Total Number of Records
- Number of Records Corrected
- % of Records Corrected
3. Corrected Items by Shop Report
This report is SEMICOLON delimited and contains all the corrected items by shop.

Page 402 from 698
Description
- Shop
- Shop Description
- Total Number of Records
- Number of Records Corrected
- % of Records Corrected
4. New Brand Report
This report is SEMICOLON delimited and contains all the new Peivate Label products
that need to be included in the rules.
Description
- Retailer
- Private Label NOT instructed
- Private Label Description
- % of such Records
Parameters :
Static Parameters

Type
CHID PL1C 24059 Char ID used for PB detection
PL1C CHVAL01 10727455 Char Value 01 for PB detection
PL1C CHVAL02 Char Value 02 for PB detection
PARM PCT_BR_CORR Minimum % of corrections per Brand
(FILEO2)
PARM BR_NB_TRANS Minimum Nb of transactions per Brand
(FILEO2)
PARM PCT_SHOP_CORR Minimum % of corrections per Shop (FILEO3)
PARM SHOP_NB_TRANS Minimum Nb of transactions per
Shop(FILEO3)
PARM PCT_BR_NOT_INSTR Minimum % of trans. Per Brand not instructed
(FILEO4)
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No

Page 403 from 698
Rework Management:


code
P0020125 File Name is blank File Name is S
blank
P0020126 File Name not present FileName not S
present
P0020608 Static Value Static Value S PL1C Static Define PL1C
Parameter is Static Parameter
missing
P0020608 Static Value Static Value S CHVAL01 for PB When PL1C static
Detection is missing parameter is
defined, CHVAL01
Static Parameter
is mandatory
P1021004 No Products have been S No products have
selected for validation been selected based
on Product Scope
defined
P1021004 No Shops have been S No shops have been
selected for validation selected based on
Shop Scope defined
P1021004 No KEYS have been S No KEYS have been
selected for validation selected based on
Product Scope and
Shop Scope defined
P1020012 PB CharValues not in W PB Char values from
GUI PB setup: candidate KEYS not
in GUI PB setup

Page 404 from 698

P9900405 Basket KEYS inserted: Information message only. No action required
P1047009 Basket KEYS deleted: Information message only. No action required
P1380001 # Purchase records Information message only. No action required
inserted
P1380002 # Purchase records Information message only. No action required
deleted
P1380003 Number of Records Information message only. No action required
Updated

Page 405 from 698
EHDV21139P- Beverages Pack Detection
Phase:
This process is used to detect multi pack purchases from Units and Prices declared by Families
for a list of specific barcodes.
A referential file of barcodes and their potential Multi Packs is used giving Standard and Promo
prices as well as number of bottles in pack.
Process detects any potential errors done by Families.

Such errors include, reporting in purchase records
o The number of packs and pack price
o The bottle or total purchases for price
o The number of bottles for quantity.
Comparing loaded records with referential for same barcode, process will detect the Multi Pack
purchases and assign barcode, new price and new units according to following matrix
Comparing loaded records with referential for same barcode, process will detect the Multi Pack
purchases and assign barcode, new price and new units according to following matrix.
When Pack Detected

Given Price Given Units New Barcode New Units New Price
Bottle Price # Bottles Multi Code # Packs Pack Price
Bottle Price # Packs Cannot be
detected would be
same as above
Pack Price # Bottles Multi Code # Packs
Pack Price # Packs Multi Code
Total Sales # Bottles Multi Code # Packs Pack Price
Total Sales # Packs Multi Code Pack Price
Process Details:
Phase 0: Load input file and compute additional information for process : smallest multi-size by
barcode, prices per bottle, price allowed range.
Phase 0b: Normalizes the existing allt11 data if NORMALISATION_STEP parameter is set to
ON, else the existing allt11 data will be processed directly. The normalization procedure is as
follows.
• Step 1: Finds price and Units for each trip data in ALLT11 and generates the
processable data using input referential data.
• Step 2: For each trip data available, the following matrix will be generated.

Page 406 from 698
• Step 3: For each record of the above generated matrix, the following calculations
are made:
▪ Proposed_price = (price - (BIR_NC_RETURN_VALUE*B)) *C
▪ Distance = abs(Proposed_price –
REF_PRICE)/BTL_NC_NUMBOTTLES
▪ Balanced_distance = Distance when B=0 Distance*(1+RVP) WHEN B>0
• Step 4: Orders the records by Balanced_distance, BTL_NC_NUMBOTTLES,
priority and retains the records ranking first.
• Step 5: If Balanced_distance <= MAX_DIST (static parameter’s value), then the
Proposed_price will be used for further steps, else already existing price will be
used.
• Step 6: The records that are normalized will be logged into
TDV_BEVERAGE_TRACKING_LOG with BTL_AC_STAGE = ‘NORM’.
Phase 1: Detect purchase having correct price and units given in Packs.
• Step 1: associate purchases with referential items having same barcode. Applied
only on purchases having Given Units (DT5650) less than smallest multi-Size.
• Step 2: Check Given Price (DT5675) against referential standard price range and
promo referential price range
• Step 3: if one of the price checks is OK, assign multi barcode to new
barcode1.Set barcode type to parameter value
Phase 2: Detect purchase having correct price in Packs but units given in Bottles.
• Step 1: associate remaining purchases with referential items having same

barcode and Given Units (DT5650) is multiple of #bottles per Packs.
• Step 2: Compute potential # Packs as Given Units (DT5650) / #bottles per
Packs.
• Step 3: Check Given Price (DT5675) against referential standard price range and
• Step 4: if one of the price checks is OK, assign multi barcode to new barcode2.
Assign # packs to new Units. Set barcode type to parameter value

Page 407 from 698
Phase 3: Detect purchase having both price and units given in Bottles.
• Step 1: associate remaining purchases with referential items having same

barcode and Given Units (DT5650) is multiple of #bottles per Packs.
• Step 2: Compute potential # Packs as Given Units (DT5650) / #bottles per
Packs.
• Step 3: Check Given Price (DT5675) against referential standard price per bottle
range and promo referential price per bottle range
• Step 4: if one of the price checks is OK, assign multi barcode to new barcode 3.
▪ Assign # packs to new Units.
▪ New Price=Given Price (DT5675) * Given Units(DT5650) / # Packs.
▪ Set barcode type to parameter value.
Phase 4: Detect purchase having price given as Total Sales but units given in Bottles.
• Step 1: associate purchases with referential items having same barcode and
Given Units(DT5650) is multiple of #bottles per Packs.
• Step 2: Compute potential # Packs as Given Units(DT5650) / #bottles per Packs.
Compute Given price per Pack as Given Price (DT5675) / # Packs
• Step 3: Check Given Price per pack against referential standard price range and
• Step 4: if one of the price checks is OK, assign multi barcode to new barcode4.
▪ Assign # packs to new Units.
▪ New Price=Given Price(DT5675) / # Packs
Phase 5: Detect purchase having price given as Total Sales and units given in Packs.
• Step 1: associate purchases with referential items having same barcode.

• Step 2: Compute Given price per Pack as Given Price (DT5675) / Given
Units(DT5650)
• Step 3: Check Given Price per pack against referential standard price range and
• Step 4: if one of the price checks is OK, assign multi barcode to new barcode 5.
▪ New Price = Given Price(DT5675) * Given Units(DT5650)
Phase 6: Final update.
• Update purchase acts records (all DT) with new barcode for corresponding keys.
• Update price (DT5675) with new price for corresponding keys.
• Update units (DT5650) with new units for corresponding keys.
• Report counts of records updated in total.
• Report counts of keys updated for barcode.
• Report counts of keys updated for price.
• Report counts of keys updated for units.
• For the records that are not corrected but normalized, Update allt11 price data
with the normalized price.

Page 408 from 698
Prerequisites:
• Records needs to be already loaded using Standard Loader for the specific
Cycle/Period.
Table Description
separated
Hive Tables involved:
TABLE DESCRIPTION
TDV_BEVERAGE_ITEM_REFERENTIAL Table used to store the data from input referential
file based on PERIOD_RETENTION parameter.
TDV_BEVERAGE_TRACKING_LOG The candidate data of multipack detection with

BTL_AC_STAGE = ‘KASTEN’ and the records
which are normalized WITH BTL_AC_STAGE =
‘NORM’ are loaded into this table for tracking
purposes.
Files:
Input Files:
Barcode Referential File

Position Length Description
1 4 Production Cycle
5 7 Period Code
12 35 Single Barcode Code
47 2 Number of Bottles
49 35 Multi Barcode
84 10 Standard Price
94 10 Promo Price
104 10 Return Value
114 30 Nankey Single
144 30 Nankey Multi
174 35 New Multi Barcode
Output Files:
No output Files. The candidates of multipack detection and the normalised data are
loaded into tdv_beverage_tracking_log.

Page 409 from 698
Parameters:
Static Parameters
PARAMETER TYPE VALUE  NAME DESCRIPTION 

TYPE  (EXAMPLE) 
PARM  PRICE_DEVIATION  0.3  To set price range allowed 0.3 for 30%.
Mandatory Static Parameter. 
PARM  BCD_TYPE  999  Barcode type value to set for Multi-Packs (Value
for 5098 Data Type). 
Mandatory Static Parameter. 
PARM  RVP  0.5  Return value parameter for the normalisation
procedure. 
Mandatory Static Parameter when
NORMALISATION_STEP is set to ON. 
PARM  MAX_DIST  0.5  Maximum distance for the record normalisation. 
Mandatory Static Parameter when
NORMALISATION_STEP is set to ON. 
Backend Parameters
These parameters are stored in TDV_JOB_GLOBAL_PARAMS
PARAMETER DESCRIPTION DEFAULT

VALUE
NORMALISATION_STEP Option describing whether normalization has to be OFF
applied on allt11 dataset for the given date range.
REFERENTIAL_LOAD Option describing whether the referential data has ON
to be taken from the input file (ON) or the existing
referential data in
TDV_BEVERAGE_ITEM_REFERENTIAL (OFF)
needs to be used.
PERIOD_RETENTION Describes the maximum number of period data 206
that TDV_BEVERAGE_ITEM_REFERENTIAL
table should hold.
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No

Page 410 from 698
Rework Management:


code
P0020505 Static Value not Static Value S Static parameter is Define a value
present not present missing from for missing
CAL_FACTOR static
table parameter
P9900155 No records loaded from S Either ASCII file is Check ASCII
Referential ASCII File empty or ASCII file’s file
layout is wrong
P9900155 No purchase acts have S No purchase found
been identified as to be processed for
Beverage Multipack Beverage Pack
Candidates Detection

code
P9900123 Number of records imported in Information message only. No action
Barcode Referential File required
P9900142 # Candidate records in Information message only. No action
Stage1: required
Stage2: required
Stage3: required
Stage4: required

Page 411 from 698
Stage5: required
P1390001 # Keys updated for barcode: Information message only. No action
required
P1390002 # Keys updated for price: Information message only. No action
required
P1390003 # Keys updated for units: Information message only. No action
required
P1390004 Total number records Information message only. No action
updated: required
P1046005 #Records Deleted from Information message only. No action
ALLT11: required
P1390005 # Prices Normalized Price Information message only. No action
Modified  required
P1001038 #Records Inserted in ALLT11: Information message only. No action
required

Page 412 from 698
EHDV21140P- Table ALL Cleaning
Phase :
Job EHDV21140P detects the keys present in both output and loading area tables and removes
them from ALLT11 table.
Job can be submitted for a specific period periods back or for a selection of FNUMs.
• Allt11
• Prockeyst29
• Allt11
Recommendations :
Constraints :
Prerequisites : .
Table Description
Files:
Input Files:
No input file.
Output Files :
No output file.

Page 413 from 698
Parameters :
Static Parameters
No static parameters should be defined for the job.

Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection Yes
DTSelection No
File Selection No
Rework Management:

Process XXXX YYYY.PPP F# allowed
Rework XXXX YYYY.PPP F# Not allowed, run F# option x process

code
P1009004 No Keys have been System Error S No records have
detected for cleaning been selected based
on the submission
parameters.

P1046005 #Records deleted Information message only. No action required
from ALLT11

Page 414 from 698
EHDV21141P- COMPUTATION OF EAN WEIGHT, FR HARD DISCOUNTERS
Phase:
This process splits items for RMS data for Aldi/ Lidl using CPS panel.
Shop selection
The scope of shops to be considered in this process must be selected in SHOPUSABILITYT40 table
using shop characteristic and char values instructed as static parameters.
We would like to have 2 set of parameters to process separately the chain ALDI and LIDL.
Period definition
The number of periods backward to be considered must be defined as a static parameter.

Page 415 from 698
Weight computation
The datatype used to compute the weight of each EAN in the total item must be DT5650.
The weight is computed using RAWDA purchase information of submission period + the number of
periods backward defined in static parameter.
The logic to compute the weights is the following:
• selection of the chain/item_code key which presents “split”
• for each chain/item_code key,
o for each EAN (or split), the weight is updated with computed value :
Weight Si (t ) =
 DT 5650 [t − NUM _ PER; t ]  100
Si
Sn
 DT 5650 [t − NUM _ PER; t ]

s1
Si
Remark : the result must be an integer
n is the number of EAN clustered on the same chain/item_code
If one or several EANs are not present in RAWDA, the updated weight must be equal to 0.
It is mandatory to have :
Weight
1
Si (t ) = 100
Note :For python version, Hive data refresh is done at the beginning of this process for the
• Rawdailyt60

Page 416 from 698
Recommendations:
Constraints:
Prerequisites:
Table Description
Files:
Input File:
Referential file:
The referential file contains the list of EANs clustered on the same item code for the chains ALDI
and LIDL.
Referential file will contain only item codes which are linked to several EANs (i.e there is a split).
File Extension: “.csv”
Field Separator: Semicolon ( will be used as field separator
Line Terminator: CR/LF
File Columns:
CHAIN;ITEM_CODE;WEIGHT;EAN;NAN
Output File:
The updated referential file is similar to the intput file. The updated weight of each EAN must
overwrite the old one in the field WEIGHT.
File Extension: “.csv”
Field Separator: Semicolon ( will be used as field separator
Line Terminator: CR/LF
File Columns:
CHAIN;ITEM_CODE;WEIGHT;EAN;NAN

Page 417 from 698
Parameters :
Static Parameters:
Paramete Type Name(example) Description

r Type Value
PARM NUM_PER The number of periods backward to be
considered when selecting from RAWDA
PG01 CVxx
PG01 CMxx
Dynamic Parameters:

Name (Yes/No)
Period Yes
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

code
P0020125 File Name is blank File Name is S Input File is missing
blank
P0020126 File Name not present FileName S Input File is missing
not present
P0020125 File Name is blank File Name is S OUTPUT File is
blank missing

Page 418 from 698
P0020126 File Name not present FileName S OUTPUT File is
not present missing
parameter are
defined for
submission
cycle
P10210105 No Shops Present in the No Shops have
Shops Usability table for been selected
Charid based on the
shopusability
parameters
P9900196 No Records Selected for S No records
Processing have been
selected from
RAWDA
oracle syntax

P0000000 Submission Date Range Get Date range for submission period
period date range
P0020608 Static Value Static Value
P9900363 Counter # weight change # Records with weight change by Chain:
P9900364 Counter # NEW NO weight # Records with NO weight=0 by Chain:
P9900365 Counter # NEW weight=0 # Records with NEW weight=0 by Chain:
P9900366 Counter # NEW weight=1 # Records with NEW weight=1 by Chain:
P9900367 Counter # NEW weight<1 # Records with NEW weight<1 by Chain:
P9900359 Counter # Records by # Records by Chain:
Chain:
P9900360 Counter # ITEM_CODES # ITEM_CODES by Chain:
by Chain:
P9900368 Counter #EAN NOT IN T60 # EAN by chain NOT present in RAWDA:
P9900369 Counter # Exported by # Records by Chain exported:
Chain

Page 419 from 698
EHDV21142P – GENERIC SHOP EDIT
Phase:
This job replaces the generic shop code by the most appropriate real shop code using
Geographical information from both Households and Shops
It detects purchases and trips made in generic shops (using a specific Shop Group in user setup).
Whenever a link to a real shop code is found using Geographical areas for both Shops and
Households and the Retailer Shop Groups defined by user, then the purchases and trips are
assigned to another Shop Codes
Recommendations:
Constraints:
Prerequisites:
o Data Selection must have been run for the period and cycle
Table Description
FAMILY_LOGT67 Log of changes applied by DV automatic validations
processes (used here to check if data loaded are already
present in output tables)
Files:
Input Files:
No input file
Output Files:
No output file

Page 420 from 698
Parameters :
Static Parameters
Parameter Type Type Value Mode Description

GENERIC_SHOPS NUMERIC Mandatory Shop Group ID
RETAILER CHAR Mandatory Retailer Chain Char
HH_AREA_L1 CHAR Mandatory Family Char Level 1
SH_AREA_L1 CHAR Mandatory Shop Char Level 1
HH_AREA_L2 CHAR Optional Family Char Level 2
SH_AREA_L2 CHAR Optional Shop Char Level 2
APPEND CHAR Optional Allow job to run multiple times in a period
using ‘Y’
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management:


Page 421 from 698
Process Logic
- Check parameters consistency (Static and dynamic)

- If APPEND is Y ,job can be rerun in normal mode too.
- IF in REWK reset KEYS from previous submission
- Resolve Shop Group for GENERIC_SHOPS
- Get Retailer chars values for generic shops
- Select basket keys for generic shops ( shop code in GENERIC_SHOPS codes)
- for each level of Regional split
o Get Household Regional split chars values for remaining keys (not yet assigned)
o Get Shop Codes for same Regional Split as HH and same Retailer char as
Generic Shop
o When found, update shop code with new shop code
- Log counters :
o Total # of keys found on generic shops
o Total # of updates
o # Keys not updated
o # update by Retailer Value
o # update by Regional Split level
Message Description Type Explanation Action

code
P9900195 Submit in REWORK S There are already Submit job in
Mode. Keys present in records for this REWORK mode
FamilyLog T67 table for job/cycle/period in
job / timestamp FAMILY_LOGT67
table.
P0020505 Mandatory Static S GENERIC_SHOPS Define
Parameter NOT defined – Mandatory Static GENERIC_SHOP
GENERIC_SHOPS Parameter has not S static parameter
been defined
P0020505 Mandatory Static S RETAILER Define RETAILER
Parameter NOT defined – Mandatory Static static parameter
RETAILER Parameter has not
been defined
P0020505 Mandatory Static S Mandatory Static Define
Parameter NOT defined – Parameter NOT HH_AREA_L1
HH_AREA_L1 defined – static parameter
HH_AREA_L1
P0020505 Mandatory Static S Mandatory Static Define
Parameter NOT defined – Parameter NOT SH_AREA_L1
SH_AREA_L1 defined – static parameter
SH_AREA_L1
P0020505 One of the LEVEL<i> S One of the Check parameter
Static parameters is LEVEL<i> Static HH_AREA_L<i>
missing parameters is and
missing SH_AREA_L<i>
consistency
P9900323 No Generic SHOPS S No Generic SHOPS
selected selected
P1021004 No KEYS selected for S No KEYS selected
validation for validation

Page 422 from 698
P1021004 No records selected from S No records selected
FAMILYUSABILITYT41 from
FAMILYUSABILITY
T41
P0010035 Oracle Error S Oracle Error Check for oracle
syntax

period date range
P0020608 Static Value Static Value GENERIC_SHOPS:
P0020608 Static Value Static Value RETAILER:
P0020608 Static Value Static Value HH_AREA_L<i>
P0020608 Static Value Static Value SH_AREA_L<i>
P9900181 Counter # Generic SHOPS # Generic SHOPS selected
selected
P9900307 Counter Total # of keys Total # of keys found on generic shops:
found on generic
shops:
P1049046 Counter # Update by # Update by Regional Split level:
Regional Split
level:
P1049046 Counter # Update by # Update by Retailer Value:
Retailer Value:
P1044027 Counter Total # of updates: Total number of rows for which shops updated
in T14 table
P9900406 Counter # KEYS not Total number of shops for which no new shop
Updated: is available to update in T14 table
P1002006 Counter # Records Total number of records inserted into T67
inserted in table.
FAMILY_LOG:
P1002008 Counter # Records deleted # Records deleted from FAMILY LOG:
from FAMILY
LOG:

Page 423 from 698
EHDV21143P – Country Rawda Copy
Phase:
A job EHDV21143P is introduced to allow user copying purchases (I.e. Rawda) and baskets details in another
country
Job will allow getting data from another country. No filtering options could be activated. Alternate country code
will be taken from setup data which cannot be modified by user.
Job could be used in different ways using allowed copy and run mode according to below table :
14 Copy modes are allowed :

- FULL : for a complete copy of source cycle and country
- DATE: for a partial copy of source cycle and country on a date range.

Page 424 from 698
Run Mode
PROCESS REWORK
Data is first cleared for all dates.
No dates used to copy.
No dates used to copy.
FULL
Complete copy from Source Country
Complete copy from Source Country
and Cycle
and Cycle
Copy Mode
DateFrom and DateTo computed from

Period and PerNum dynamic
DateFrom and DateTo computed
parameters.
from Period and PerNum dynamic
parameters.
DATE Data is first removed for this date
range.
Data is copied from Source Country
and Cycle for this date range.
Data is copied from Source Country
and Cycle for this date range.
Item Handling:
- As target it to ReXcode the purchaches the destination Nankey field will be populated with 0.
- User can choose which code is to load into the destination barcode field :
o Replicate barcode field value from source country
o Copy it from nankey field from source country.
- When BCD_SOURCE = ‘NANKEY’, Then
1. load is done only for Xcoded purchases on RawdailyT60 & Prockeyst29
2. Each data type is Merged in RawdailyT60 & Prockeyst29 for the purchase key using SUM,AVG and MAX functions.

Page 425 from 698
Rework:
PROCESS or REWORK mode allowed.
Constraints:
Submission with period 0 is not allowed for this job
Prerequisites:
Whole production cycle must have been performed in source country/cycle to ensure copying final data.
Table Description
RAWDAILYT60 RawDaily
PROCKEYST29 Purchase and Basket key
Files:
Input Files:
No input file
Output Files:
No output file
Parameters:

Page 426 from 698
Static Parameters

Type
PARM SOURCE _AUDIT Mandatory Source Audit to be copied
PARM COPY_MODE Mandatory Copy Mode :
- FULL : complete copy
DATE : partial copy on date range
PARM DEBUG Optional Default as – No
PARM BCD_SOURCE Mandatory Source for Barcode Field
- BARCODE
- NANKEY
Dynamic Parameters

Name (Yes/No)
# Periods Yes Default as 1
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 427 from 698
P9900397 Source Country is not available in SETUP_PARAM table.
P9900399 Source Country Schema is missing in the Database.
P0000400 Source Audit **** is missing in the Source Country Schema: ****
P1038014 Key Already Present-Rework Option Required – Process Mode – Not Allowed
P0020516 Period Selection required for Job

Page 428 from 698
EHDV21144P – Cycle Rawda Copy
Phase:
A job EHDV21144P is introduced to allow user copying purchases (I.e. Rawda) and baskets details in another
cycle.
Job will offer some basic filtering options as follow :

- Based on list of shops (i.e. : a shop group)
- Based on products criteria (i.e. : ibd)
- Disabling Constraints and drop indexes are not applied in INCR mode.
Note: all conditions will be linked with AND logical operator.
Filter applied differently on each table to be copied
Table
Applies On RAWDA TBSKT PROCKEY
Shop Group Yes Yes Yes

Filter
Yes on Purchase Keys

IBD Yes No
No on Basket Keys

Page 429 from 698
Note that : Only IBD is mandatory when load mode is INCR. (Remove Shop group)
Rework:
PROCESS or REWORK mode allowed.
Constraints:
Submission with period 0 is not allowed for this job
Prerequisites:
->Whole production cycle must have been performed in source country/cycle to ensure copying final data.
-> HH load and Shops load have to be performed in target cycle
-> Shop Group & IBD have to be created in cycle source

Page 430 from 698
Table Description
RAWDAILYT60 RawDaily
PROCKEYST29 Purchase and Basket key
TDV_IBD_RESOLUTION IBD Resolution
TDV_SHOP_GROUP_CONTENTS Shop group content details
Files:
Input Files:
No input file
Output Files:
No output file
Parameters:
Static Parameters

Type
PARM SOURCE _AUDIT Mandatory Source Audit to be copied
PARM SHOP_GROUP Optional Shop Group ID – Must exist in Source
Cycle/Audit
PARM SHOP_FILTER Optional Shop filtering mode (IN or NOT IN)
Become mandatory when SHOP_GROUP is
filled ; default value = IN assumed when
missing
PARM IBD Optional IBD ID – Must exist in Source Cycle/Audit
PARM IBD_FILTER Optional IBD filtering mode (IN or NOT IN)
Become mandatory when IBD is filled ;
default value = IN assumed when missing

Page 431 from 698
PARM DEBUG Optional Default as – No
PARM LOAD_MODE Mandatory Mode INCR is added along with APPEND,
REPLACE, UPDATE
Dynamic Parameters

Name (Yes/No)
# Periods Yes Default as 1
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 432 from 698
P9900211 Source Audit: **** is not available.

P0020505 Static Value not present Type Name: PARM,Type Value: SOURCE_AUDIT
P1038014 Key Already Present-Rework Option Required – Process Mode – Not Allowed
P0020516 Period Selection required for Job
P9900404 IBD – Not associated with the Source Cycle/Audit.
P9900403 Shop Group ID – Not associated with the Source Cycle / Audit.
P0010003 Data has not been COMMITTED. ROLLBACK performed – Kindly go to the
option Ultilities -> Cleaning -> Database and then Select RAWDA and
TOTALBASKET for the 433lignment period and index.
P1440001 Source and destination cycle should be different

Page 433 from 698
EHDV21147P – Cycle Price Copy
Phase:
This new Cycle Price Copy job loads the standard price from source Cycle to destination Cycle for the submitted period range.
Recommendations:
Constraints:
Prerequisites:
Table Description
CNANPRICET34 Table containing standard price and MEDIAN by cycle, price
Group
Files:
Input Files:
No Input File

Page 434 from 698
Output Files :
No output file
Parameters :
Static Parameters

Type
PARM SOURCE_CYCLE Mandatory Source Cycle from where the Standard Price
to be copied
Dynamic Parameters

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Rework Management:


Page 435 from 698
Page 436 from 698
Process Logic
- Start and End Period using Period and PerNum is computed.

- If Run_mode = REWORK
o Delete data in CNANPRICET34 for the submitted cycle and period range
o Report count of deleted keys
- Insert query in CNANPRCET34 for the submitted cycle and period range using :
o Source cycle
- Report count of inserted keys
- Statistics for the table CNANPRCET34 will be gathered.

code
P0010035 Program Terminated – S Source Cycle is not Define the Source
Source Cycle available in static Cycle.
‘||g_src_cycle||’ not parameter.
available
P0010035 Data already Exists – S Data already exists Submit the job in
Normal mode is not for the submitted Rework Mode.
possible, Please use cycle and period
Rework mode range.
P9900262 Statistics computed for I Statistics collected No action
table CNANPRICET34 for the table required.
CNANPRICET34.

Page 437 from 698

P0020608 ‘Period from : ‘ Date Range Get Date range for submission period
|| g_periodfrom
|| ‘ Period to : ‘
|| g_periodto
P0020608 Static Value Static Value SOURCE_CYCLE:
P0020608 Counter No. Of records No. Of records Inserted into table
Inserted into table
CNANPRICET34
–
‘||m_affected_rec
P0020608 Counter No. Of records No. Of records Deleted from table
Deleted from table CNANPRICET34
CNANPRICET34

Page 438 from 698
PRODUCTION FLOW
Phase:
PHASE 8 – Production Flow
Process Description
The process is about automating production to link Groups together and to have automatic groups launch. Objective is to speed up activities and
reduce manual actions from users. The country user can monitor the whole production for a period from first to last step of same production
period.
Phases involved in the Production Flow are as Follows:

SETUP
- User to setup the regular flow(s)6 according to country needs by
o Listing the steps and the related details
▪ Period (relative)7 and number of periods
▪ Filenames (generic)
• Along with generic names ,following keywords can be used
• @CYCLE@ - Will replace this keyword with stepcycle/alternate cycle based on step definition
• @NUMPER@ - Will replace this keyword with step number of periods
• @PERIOD-5@ to @PERIOD+5@ - Will replace this keyword as shown below with step period
6 There can be more than one in a country, for ex: Weekly process and 4Weekly process.
7 0=Current Period , -1=Previous Period, …

Page 439 from 698
▪ Run mode
▪ Schedule
▪ Trigger events
o Sequencing the steps
o Listing the links between the steps (predecessors)
o Alternate cycle and repetition count
OPEN
- Open a flow for a specific period in a cycle by
o Reviewing the flow for the specific period when needed
▪ Set Period (absolute) and number of periods
▪ Set alternate cycle if predefined
▪ Set specific filenames
▪ Change links and steps sequence
o Launching the flow
MONITOR
- Monitor the flow by
o Following the progress
o Releasing step pending for User action
o Skipping useless step
o Stopping the flow
o Restarting flow
Run :
- an instance of a production flow for a specific selection period and cycle
Steps :
- Correspond to a job. Same job can be used multiple times when needed.
- When same set of jobs is needed more than once, corresponding steps will be repeated in flow with different details.
- Each step can have up to 2 predecessors: it will be candidate to start only when the 2 predecessors are completed successfully or
with warning or Skipped.
Wait Event:
- User : user action is needed and user will release step when done
- File : file arrival is needed, start will be automatic or step can be released by user
- Schedule : specific date and time to be reached, user can also release step

Page 440 from 698
Start conditions:
Events
File Schedule User
Predecessor None
Arrived Not Arrived Reached Not Reached Released Not Released
None Start Start Start Start
Not Completed
Completed Start Start Start Start
To Skip
Skipped Start Start Start Start
Scheduler will be looking for candidates every 5 minutes to start the job.
A starting candidate step is a step in PENDING or TO_SKIP or RELEASED status with all predecessors Completed, Warning or Skipped plus its
events raised (FILE or SCHEDULE).
Run Status:
- OPEN(01): flow is open but not launched yet
- IN PROGRESS(99): Flow has been launched and first step can start
- COMPLETED(00): All steps are completed successfully
- STOPPED(32): User has stopped the run.
- FAILED(16): Step(s) has failed, no further step can run.
Steps Status:
- RUNNING (99): Step has started
- QUEUED(77): Step has been queued for a run but flow is not launch yet.
- PENDING(97): Step waiting for start when conditions will be ready (Predecessors or Wait Event)
- RELEASED(95) : User has requested to release step.
- TO_SKIP(96) : User has requested to skip step.
- SKIPPED(98): User has skipped the step
- STOPPED(32): User has stopped the flow
- COMPLETED(00): Step has completed successfully
- WARNING(08): Step has completed with warning message
- FAILED(16): Step has triggered an error

Page 441 from 698
Relative Periods : the following elements can be selected at setup time
- Period 0: can be used for general job
- Current + 1 : one period after the one on which flow is open
- Current : the one on which flow is open
- Previous – 1 : the one before the one on which flow is open
- Previous – 2 : 2 periods before the open one
Scheduler will be looking for candidates in both the submitted cycle and alternate cycle every 5 minutes to start the job.
Process Mode:
The process mode chosen for the flow can be NORMAL or REWORK mode. When a Change is done to the process mode for a step in a flow,
then the mode will be reflected when a new run is launched.
Oracle Tables involved
Table Description
TDV_PRODUCTION_FLOW Flow details will be available.
TDV_PRODUCTION_FLOW_SETUP Table contains all the details of the step(Jobs used) for the
particular flow.
TDV_PRODUCTION_FLOW_FILE Table contains the information of the Files related to the steps
for the particular flow
TDV_PRODUCTION_RUN Table contains the details of the corresponding Run launched.
TDV_PRODUCTION_RUN_STEP Step Details for the respective RUN will be stored in the table.
TDV_PRODUCTION_RUN_FILE Input/Output File Details for the run launched will be available
in the table.
TDV_PRODUCTION_FLOW_PARAMS Specific Static parameters of each Step ID and Flow tag
combination are stored here.

Page 442 from 698
Files:
Input Files:
The Files related to run the respective jobs submitted in the flow are required.
Output Files:
The Files related to run the respective jobs submitted in the flow are required.
Same sequence number used more Check the flow and correct the sequence.
than once in the flow
Predecessor1 and Predecessor2 cannot Change the Predecessor of the step so that they are
be equal not equal.
Predecessor must be lesser than Predecessor chosen for the step should not have
existing step sequence sequence greater than the existing step.
Period not available for the Cycle Change the Period for the Cycle which is available
and open the flow
No File exists, File must be chosen for Input or Output File required for the step.
the step

Page 443 from 698
EHDV21148P – EXTERNAL PRICE LOADER
Phase:
This new external price loader job loads ASCII price files coming from external sources (clients) at barcode level.
The whole process is driven by the description of the external price file layout, as this has been setup from GUI at two levels and stored in the
following tables:
Level 1 Table: TDV_

Column Name Seq Pk Type Nullable Comment
EPF_FILE_ID 1 1 NUMBER No File ID (generated from Sequence)
EXTERNAL_PRICES_FILE
EPF_AC_CYCLE 2 CHAR(4) No Cycle
EPF_FILE_DESCRIPTION 3 VARCHAR2(50) No File Description
EPF_COLUMN_COUNT 4 NUMBER No Column count
EPF_HEADER_LINES 5 NUMBER No Header Lines Count (to skip)
EPF_DECIMAL_SEPARATOR 6 CHAR(1) No Decimal Separator character
EPF_PRICEGROUP_CHAR 7 CHAR(6) No Price Group Characteristic
EPF_FIELD_DELIMITER 8 CHAR(1) No Field Delimiter
EPF_FIELD_ENCLOSED 9 CHAR(1) No Field Enclosing flag

Page 444 from 698
Level 2 Table: TDV_ EXTERNAL_PRICES_COLUMNS
Column Name Seq Pk Type Nullable Comment
File ID (fk-
EPC_EPF_FILE_ID 1 1 NUMBER No
TDV_EXTERNAL_PRICES_FILE)
EPC_COLUMN_SEQ 2 2 NUMBER No Column Sequence
EPC_COLUMN_TYPE 3 CHAR(1) No Column Type (I B L P)
VARCHAR2(2
EPC_XCODE_GROUP 4 Yes Xcoding Group ( B or L Type)
0)
VARCHAR2(1
EPC_PRICEGROUP_VALUE 5 Yes Price Group (P type)
0)
EPC_LIST_SEPARATOR 6 CHAR(1) Yes List Separator (L Type)
EPC_LIST_QUOTED 7 CHAR(1) Yes List is Enclosed (L Type)
VARCHAR2(2
EPC_EMPTY_VALUE 8 Yes Empty Value (to ignore)
0)
Recommendations:
Constraints:
Prerequisites:
o OGRDS or IMDB must have been loaded previously.

o Standard Price must have been computed and loaded already for same period on other prices
Table Description
TDV_EXTERNAL_PRICES_FILE External Price File description

Page 445 from 698
TDV_EXTERNAL_PRICES_COLUMNS External Price File Columns description
CNANPRICET34 Table containing standard price and MEDIAN by cycle, price
Group
Files:
Input Files:
ASCII external price files
Output Files :
No output file
Parameters :
Static Parameters

Type
PARM LAYOUT_ID Mandatory External Price file layout ID
PARM MEDIAN_PRGRP0 Mandatory Compute Median at Price Group 0(ON/OFF)
Dynamic Parameters

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No

Page 446 from 698
Rework Management:

Process Logic
- Check parameters
o Presence: LAYOUT_ID and MEDIAN_PRGRP0 are mandatory
o Consistency : LAYOUT_ID must exist, corresponding file setup must have at least one price column and one barcode or
barcode list column
- Load File in temporary table
o Create External table script to read file using
▪ Column count
▪ Field Separator
▪ Field Enclosing
▪ Skipping header rows
o For each Price column
▪ For each Barcode column Insert into temporary table
• Filtering empty values
• Formatting number using Decimal Separator
• Adding xCoding Group and Price Group
▪ For each Barcode List column Insert into temporary table
• Splitting column values in multiple barcode using list separator and list quoted
• Filtering empty values
• Formatting number using Decimal Separator
• Adding xCoding Group and Price Group
- Perform xCoding
o using appropriate xCoding groups when not STANDARD

Page 447 from 698
o using selected xCoding groups when STANDARD
- Load data from temporary table to Standard Price (DT5653) table for xCoded rows
- If MEDIAN _PRGRP0 = ON
o Select nankeys not having Median price (DT5094) for period and price group 0
o Compute Median price at price group 0 for selected items only
▪ Simple Median computation considering the input file only
▪ No MIN_OBS parameter
▪ One period only (submitted one)
o Store Median at price group 0
- Report figures :
o – Total Count of Record read from file
o – Total Count of Record loaded in temp table (NAN*PRICE)
o – Total Count of Record xCoded
o – Total Count of Record not xCoded
o – Total Count of Price loaded
o – Total Count of Median price computed

code
P9900414 File Layout is not defined S File Layout is not Check if FILE
for Cycle defined for Cycle LAYOUT is
defined for job
submission cycle
P0020505 Mandatory Static S LAYOUT_ID Define
Parameter NOT defined – Mandatory Static LAYOUT_ID
LAYOUT_ID Parameter has not static parameter
been defined
P0020505 Mandatory Static S MEDIAN_PRGRP0 Define
Parameter NOT defined – Mandatory Static MEDIAN_PRGRP
MEDIAN_PRGRP0 Parameter has not 0 static parameter
been defined

Page 448 from 698
P9900412 Price Column missing S No PRICE column Check COLUMN
from file layout definition has been defined for 449lignment449s
the specified for the the
LAYOUT_ID specified
LAYOUT_ID
P9900413 Barcode/Barcode List S No Check COLUMN
Column missing from file BARCODE/BARCO 449lignment449s
layout definition DE LIST column has for the the
been defined for the specified
specified LAYOUT_ID
LAYOUT_ID
P0020125 File Name is blank S Check File name
P0020126 File Name not present S Check File name
P9900419 Price value contains non Price value contains Check PRICE
numeric non numeric columns for non
characters.Column: characters.Column: NUMERIC
characters
P9900407 No records loaded in S No records loaded in
temp table temp table
P9900232 Input file format not S File format is not OK Check File layout
correct.Bad file
generated:
P0010035 Oracle Error S Oracle Error Check for oracle
syntax

period date range
P0020608 Static Value Static Value LAYOUT_ID:
P0020608 Static Value Static Value MEDIAN_PRGRP0:
P9900123 Counter # Records read # Records read from input file:
from input file:

Page 449 from 698
P9900415 Counter # Records Loaded # Records Loaded in temp table
in temp table
P9900408 Counter # Records Xcoded # Records Xcoded
P9900409 Counter # Records NOT # Records NOT Xcoded
Xcoded
P9900411 Counter # Records Median # Records Median Price Computed
Price Computed
P9900410 Counter # Records Price # Records Price Loaded
Loaded
P9900416 Counter # Records Price # Records Price Updated
Updated

Page 450 from 698
EHDV21149P – FRESH FOOD RECOVERY TOOL
Phase:
PHASE 0 – Input Area
This tool maps Fresh Food codes to Nielsen Code Book references so that purchases can be cross coded and used for production as if panellist
had done properly the job. New feature will be implemented based on Spain expectations but has been made as generic as can be so that it can
be used in any other country.
Recommendations:
Constraints:
Prerequisites:
Standard Loader (01P) must have been run before.
Table Description
separated in datatypes and loaded in the table ALL as
separate records
Files:
Input Files:
Mapping Reference

Page 451 from 698
Format : Delimited with ‘;’
Layout :
Field Type Description

Fresh Food prefix Numeric Prefix to match with collected barcode
Code Book code AlphaNum Nielsen reference to map (including characteristics)
RMS Price Numeric 2 decimals with a ‘.’ – can be left empty if not used
Validity date Date YYYYMMDD – for reference – not used
Volume Method AlphaNum To select the Volume calculation Method
Output Files :
No output file
Parameters :
Static Parameters
Parameter Type Mode Comment

BCD_TYPE CHAR Optional Barcode type to assign when needed
VOL_METHOD LIST Mandatory To select the DEFAULT volume
calculation method if for a record in the
Input file the Volume Method is NULL:
• COPY
• GET_VOL_PRICE_8_5_2
• COPY_VOLUME_9_5_3
KEEP_CODE LIST Mandatory ON/OFF is original barcode has to be

kept in DT5129
SHOP_GROUP NUMERIC Optional Shop Group to be used when needed

Page 452 from 698
Dynamic Parameters
Parameter Used Comment

Period Yes Submission with period 0 is not allowed for this job
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
File Selection No
Rework Management:


Page 453 from 698
Process Logic
- Process is going thru the following steps :
Initialization Step :
- Load mapping reference file in external table
Process Step :
- Compute Start and End Date using Period and #Periods
- Process Data
o Map allt11 and reference file on prefix field within SHOP_GROUP
o Insert allt11 records with Code Book code for new volume, based on the Volume Method for File
Prefix defined in the Input File. If the Volume Method value of the File Prefix in the Input File is
NULL then the value of VOL_METHOD static parameter is used to compute volume.
o Insert allt11 records with Code Book code for price = Reference Price
o If KEEP_CODE = ON insert allt11 records with Code Book code for DT5129 = ac_barcode
o If BCD_TYPE <> 0 insert allt11 records with Code Book code for DT5098 = BCD_TYPE
o Insert all11 records with Code Book code for other datatypes
o Delete allt11 records with Fresh Food mapped to prefix
Final Step :
- Report count of inserted records
o Total
o Volume computed
o Price from Reference
o Fresh Food code
o Barcode Type
o Others
- Report count of deleted records on Fresh Food code
- Report count of distinct Fresh Food codes
- Report count of distinct Code Book codes
Records management: it can happen that 2 different Fresh Food codes map to same Code Book code. In such case, there would be need to
merge records.

Page 454 from 698
Datatypes management: each datatype will be considered in a specific way.
Following table Description Process Merge action

summarize the 2
things for each
datatype: Datatype
DT5675 Price Taken from reference file Same Value
DT5650 Units/Volume can be copied or computed Summed Up
using appropriate method :
- COPY – copied from initial
record
- GET_VOL_PRICE_8_5_2 –
compute vol using price from
barcode position 8 on 5 digits
with 2 decimals1 as (DT5650 *
(Code Price(8,5,2)/ Reference
Price)
- COPY_VOLUME_9_5_3 –
read volume from barcode
using position 8 on 5 digits
with 3 decimals
DT5098 Barcode Type can be assigned to a specific Same Value

value (optional)
DT5129 Fresh Food Original can be assigned from initial Max
Barcode record barcode field(optional)

Page 455 from 698

code
P0020125 Mapping Reference Input S File not available. Check the path and place the file correctly in the
File path.
P0020126 Mapping Reference Input S File name not found Check the file name and correct the same.
File
P0010035 Rework mode is not S Rework mode is not allowed. Submit the job in process mode.
allowed, Please use
Normal mode
P9900136 Static Parameter value for S Valid value for BCD_TYPE Valid value for BCD_TYPE should be a positive
BCD_TYPE is NOT a should be a positive integer. integer
NUMBER.
P0010035 Mandatory Static Param: S Mandatory VOL_METHOD Define a value for Mandatory VOL_METHOD
VOL_METHOD is not static parameter has not been static parameter
defined defined
P9900175 Ignoring the barcode xxx W Total wil be 0 and the job will No Action required.
which leads to Volume be completed with warnings
Zero value
P1042002 No VARIABLE WEIGHT W No VARIABLE WEIGHT
CODES were found for CODES were found for
correction correction
P9900085 Invalid VOLUME S A value which is not one of Check the input File for a Volume Method value
METHOD value found in the expected Volume Methods other than:
Input File value has been found in in • COPY
Volume Method field of the • COPY_VOLUME_9_5_3
Input File • COPY_VOL_PRICE_8_5_2
P9900232 Input file format NOT S Input file format NOT correct Check the Input File format
correct

Page 456 from 698
P9900045 Prefixes with different S There are Barcode Prefixes in Check the Input File for different Barcode
LENGTHS within same the Input File having different Prefixes lengths for the same Volume Method
VOLUMETRIC METHOD: lengths for the same Volume
Method
P9900045 Same Prefix is used more S Same Prefix is used more Check the Input File for Barcode Prefixes that
than once in shortest than once in shortest length occur as part of other Barcode Prefixes in the
length Input File

P1039009 Number of Distinct Shops Shops # Shops selected
P1001038 Volume Computed Volume Computed # Records inserted for Volume computed (DT5650)
P1001038 Price from Reference Price from Reference # Records inserted for Price from Reference File
(DT5675)
P1001038 Fresh Food Code Fresh Food Code # Records inserted for Fresh Food Barcode (DT5129).
When KEEP_CODE is ON
P1001038 Barcode Type Barcode Type # Records inserted for Barcode Type (DT5098). When
BCD_TYPE<>0
P1001038 Others Others # Records of Other Datatypes
P1490001 Total Total # Total Records inserted into ALLT11
P1001038 Report count of distinct Code Report count of # Distinct Code Book codes
Book codes distinct Code Book
codes
P1001038 Report count of distinct Fresh Report count of # Distinct Fresh Food codes updated.
Food codes distinct Fresh Food
codes
P1490002 Report count of deleted Report count of # Records Deleted from ALLT11:
records from ALLT11 deleted records from
ALLT11
P1046006 Report count of deleted Report count of # Records Deleted from ALLT11 having CODEBOOK as
records from ALLT11 deleted records from BARCODE:
ALLT11

Page 457 from 698
EHDV21150P- US Data Loader
Phase:
US Data Loader is used to load data in a batch mode directly to the output tables: RAWDAILYT60, TBSKT13, PROCKEYST29 as well as the shop
master table SHOPT44 and the family master table FAMILYT43 from backdata input ASCII files, prepared by the legacy system. This process is
used by US to initialise their DV environment with already validated data.
Note : For python version Hive data refresh is done at the beginning and oracle data refresh is done at the end of this process in terminateJob for
these three tables:
1.Prockeyst29
2.Rawdailyt60
3.Tbskt13
Recommendations:
Constraints:
Input files should come in pairs, one file for the purchase records (rawdaily records) and one for total basket records (tbsk records). In case the job
is submitted for N periods > 1 then these ASCII files should contain all the records for all the periods submitted.
Prerequisites:
Table Description

Page 458 from 698
Files:
Input Files:
FileName Comment
FILEI1 DEAL CODE MAPPING REFERENCE
FILEI2 NCP TRIP DATA FILE
FILEI3 NCP PURCHASE DATA FILE
FILEI4 NCP MAGNET PURCHASE DATA FILE
Deal Code Mapping Reference:

Format: Text file
Layout: Field Type Length Description

US Deal Code AlphaNum 2 Deal Code to match
with NCP Purchase
records
DT5662 Num 1 DT5662 flag – can
be 0 or 1
be 0 or 1
be 0 or 1
be 0 or 1

Page 459 from 698
NCP Trip Data File:
Format: Text file

PanId NUM 8 Panelist Identification
Period NUM 4 Process Period
TripDate NUM 6 Trip Date (Date Shopped)
NHS_Outlet NUM 4 NHS Outlet
MarketCode NUM 3 Major Market Code
ACNStore NUM 5 CAN Primary Store
NumberOfItems NUM 3 Number of Items (Unique
UPC’s)
TotalOutlet NUM 5 Total Outlet $/Total UPC $
(Before Coupon)
Quantity NUM 5 Quantity (After Editing)
TripHour NUM 2 Trip Hour
Blank NUM 1 Blank
PrimShopper NUM 3 Primary Shopper (Sex, Age)
SecShopper NUM 3 Secondary Shopper (Sex, Age)
StoreType NUM 1 Store Type
PanelCode NUM 1 Panel Code
TotalDollars NUM 5 Total Dollars Spent/Ring
Dollars (Cents)
OnDealTransactions NUM 2 Transactions (UPC’s) On-Deal
MfrCouponTotal NUM 4 Mfr. Coupon Total (Cents)
StoreCouponTotal ALPHA 4 Store Coupon Total (Cents)
StoreMfrCouponTotal ALPHANUM 4 Store/Mfr. Coupon Total
(Cents)
PaymntMethod NUM 2 Freq. Shopper/Method of

Page 460 from 698
Paymnt*
GrocerID NUM 7 Grocer ID Number
NCP Purchase Data File:

Format: Text file

TripDate NUM 6 Trip Date (Date
Shopped)
Module NUM 4 Product Module
PrimShopper NUM 3 Primary Shopper
SecShopper NUM 3 Secondary Shopper
UPC NUM 12 UPC Number
Quantity NUM 2 Quantity (Possibly
Modified)
Total_Price NUM 5 Total Unit Price
DealCode ALPHANUM 2 Deal Code
Total_Coupon NUM 4 Total Coupon Value
CausalAd NUM 1 Causal Ad
CausalDisplayInfo NUM 2 Causal Display Info
PriceIndicator NUM 1 Price Indicator
BrandCode NUM 6 Brand Code
Size ALPHA 2 Size
SizeDescription ALPHANUM 6 Size Description
SizeNumber NUM 7 Size Number
Multi NUM 3 Multi
Flavor NUM 6 Flavor Code

Page 461 from 698
OrgCode ALPHA 3 Org. Code
PaymntMethod NUM 2 Freq.
Shopper/Method of
Paymnt*
Panel Code ALPHA 1 Panel Code
CpnDealValue NUM 4 MFG Cpn Deal Value
NCP Magnet Purchase Data File:
Format: Text file

Shopped)
PrimShopper NUM 3 Primary Shopper
SecShopper NUM 3 Secondary Shopper
Modified)
Size ALPHA 2 Size
Multi NUM 3 Multi

Page 462 from 698
PaymntMethod NUM 2 Freq.
Shopper/Method of
Paymnt*
CpnDealValue NUM 4 MFG Cpn Deal Value
NCP File Rules

The job applies field transformation, records filtering and collapsing , deal code mapping as per below tables :
NCP Trip Data File Rules
Trip Data file DV Field Mapping Transformation Filtering Collapsing

Field
PanId ac_hhold Left pad with 0 up to 10 None None
chars
TripDate dt_pur_date Reformat to yyyymmdd None None
NHS_Outlet ac_shop Left pad with 0 up to 10 None None
chars
TotalOutlet DT5128 (Basket Total FMCG) Divide by 100 (Dec=2) None None
DT5895 (Total FMCG)
Quantity DT5666 (Basket Total Units) None None
DT5896 (Units FMCG)
TotalDollars DT5941 (Basket Total Spent) Divide by 100 (Dec=2) None None

Page 463 from 698
All other fields are ignored, as they are useless for the process.
NCP Purchase Data File Rules
Purchase Data file DV Mapping Transformation Filtering Collapsing

Field
PanId ac_hhold Left pad with 0 up None Merging Key
to 10 chars
TripDate dt_pur_date Reformat to None Merging Key
yyyymmdd
NHS_Outlet ac_shop Left pad with 0 up None Merging Key
to 10 chars
UPC ac_barcode UPC <> 000000000000 UPC <> Merging Key
999999
Quantity DT5650 (Purchase Units) Summed Up
DT5642 (Purchase Units Historical DT)
Total_Price DT5675 (Price) See formula
Total_Coupon DT5667 (Price Historical DT) below
DealCode DT5662 to DT5665 (Promotions) See mapping
below
All other fields are ignored as they are useless for the process.
Values will be merged for same key: ac_hhold, dt_pur_date, ac_shop, barcode
Price formula: ( sum(Total_Price – Total_Coupon) / sum(Quantity) ) / 100
When formula leads to 0 , price will be loaded as 0
Deal Code mapping: For each record with the 4 DV promotion flags. For each promotion flag keep maximum value

Page 464 from 698
NCP Magnet Purchase Data File Rules

Field
to 10 chars
yyyymmdd
to 10 chars
999999
Total_Coupon DT5667 (Price Historical DT) below
below
Total_Coupon DT5130 (Total coupon Value) Summed Up

Page 465 from 698
Output Files :
FileName Comment
FILEO1 NCP TRIP DATA REJECTED FILE
FILEO2 NCP PURCHASE DATA REJECTED FILE
FILEO3 NCP MAGNET PURCHASE DATA REJECTED FILE
Two output ASCII files will be created by the job, one for RAWDATA records and one for TBSK records.
The layout of these files will be the same as the layout of the Input ASCII files.
When job is submitted with

● LOAD_MODE=NORMAL the rejected files will contain any records from the NCP ASCII files having date values out of the period date
range.
● LOAD_MODE=APPEND the rejected files will contain any records from the NCP ASCII files having same keys already in the output tables
that exist in the NCP ASCII files.
Parameters :
Static Parameters:

Name (Yes/No)
AC-AUDIT Yes

Page 466 from 698
AC-JOB Yes
AC-TYPE Yes PARM
AC-TYPE VALUE Yes LOAD_MODE
AC-NAME Yes Valid values are
NORMAL
APPEND
REPLACE
Default value if parameter is not defined is NORMAL
PROCESS SUBMISSION MODES
LOAD_MODE PROCESS REWORK

NORMAL Initial Partitions checks Not Allowed
Load in Append mode
REPLACE Not Allowed Initial Deletion of existing data in Period Range
Load in Append Mode
APPEND Not Allowed Check for existing data in database with same keys
If none found, Load in Append Mode
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management :

Page 467 from 698

code
table
DATACODET20
P0020125 File Name is blank File Name is S File is missing Check if any of
blank the files is
missing
P0020126 File Name not present FileName S File is missing Check if any of
not present the files is
missing
Lock. RELEASE Lock.
Lock
P9900232 Input file format not Input file S Dealcode Reference Check input file

Page 468 from 698
correct format no Mapping file format layout
not correct.Bad file
generated
P9900232 Input file format not Input file S NCP Purchase Data Check input file
correct format no file format not layout
correct.Bad file
generated
P9900232 Input file format not Input file S NCP Trip Data file Check input file
correct format no format not layout
correct.Bad file
generated
P9900014 Some or all input files are Empty File S Some or all input Check Input
empty files are empty files. Some of
the input files
are empty
P0020722 Deal Code Mapping Input File S Deal Code Mapping
Reference file does not Not Present Reference file does
exist/or format is not not exist/or format
correct is not correct
P0020722 NCP Purchase Data file Input File S NCP Purchase Data
does not exist/or format Not Present file does not exist/or
is not correct format is not correct
P0020722 NCP Magnet Purchase Input File S NCP Magnet

Data file does not exist/or Not Present Purchase Data file
format is not correct does not exist/or
P0020722 NCP Trip Data file does Input File S NCP Trip Data file
not exist/or format is not Not Present does not exist/or

Page 469 from 698
correct format is not correct
P9900437 Invalid option. Can not Invalid Job S Invalid option. Can Check process
submit job in REWK mode submission not submit job in submission
and option REWK mode and rules
LOAD_MODE=NORMAL LOAD_MODE=NOR
MAL
submit job in PROC mode submission not submit job in submission
and option PROC mode and rules
LOAD_MODE=APPEND or LOAD_MODE=APPE
REPLACE ND or REPLACE
P9900195 Data already present in Data already S Data already
output tables. Load IN present in present in output
REWK mode and output tables tables. Load IN
– Rwk option
LOAD_MODE=REPLACE required for REWK mode and
data LOAD_MODE=REPLA
processing CE
P0020111 # Records having same Attempt to S # Records having
Key that already exists in insert same Key that
TBSK: duplicate key already exists in
values
TBSK:
RAWDA: duplicate key already exists in
values
RAWDA:
oracle syntax

Page 470 from 698

date range
P0020608 Static Value Static Value LOAD_MODE: Information message only. No
action required
P9900123 Total Number of Record Count # DealCode Records read from input file:
Records Present in Information message only. No action required
the input file
P9900123 Total Number of Record Count # Purchase Records read from input file:
the input file
P9900123 Total Number of Record Count # Magnet Purchase Records read from input
Records Present in file:
the input file Information message only. No action required
P9900123 Total Number of Record Count # Trip Records read from input file:
the input file
Records inserted message only. No action required
P1030010 Number of records Record Count # Records skipped due to dates out of range
skipped in the Purchase Data file: Information
message only. No action required
skipped in the Magnet Purchase Data file: Information
skipped in the Trip Data file: Information message

Page 471 from 698
P1037008 Total Number of Record Count # New Shops inserted: Information message
Records inserted only. No action required
P1034006 Total Number of Record Count # New Hholds inserted: Information message
TBSKT13 table
deleted from RAWDA
RAWDAILYT60
deleted from PRKEYS
PROCKEYST29 table
% checked
created.
P9900432 Number of records Record Count # Trip records that already exist in TBSK and
rejected and exported rejected, exported in outputfile:
in the output file Information message only. No action required
P9900432 Number of records Record Count # Trip records out of date range records
P9900432 Number of records Record Count # Purchase records that already exist in
rejected and exported RAWDA and rejected, exported in outputfile:

Page 472 from 698
P9900432 Number of records Record Count # Purchase records out of date range records
P9900390 Statistics has been Statistics has Statistics has been calculated for table:
calculated for table: been Information message only. No action required
calculated for
table

Page 473 from 698
EHDV21152P- US Data Edit
Phase:
US Data Edit is used to load edit already existing data in a batch mode directly to the output tables: RAWDAILYT60, TBSKT13, PROCKEYST29
as well as the shop master table SHOPT44 from backdata input ASCII files, prepared by the legacy system.

* Data in oracle table is refreshed to parquet for TDV_XCD_SHRINK_LINKS,RAWDAILYT60,PROCKEYST29,TBSKT13 tables.
* Error in Refresh parquet is logged using the error number - P9900605

* Data from RAWDAILYT60,PROCKEYST29,TBSKT13 table is refreshed back to oracle table.
* Error in Oracle Refresh is logged using the error number - P9900627
Recommendations:
Constraints:
Prerequisites:
• US Data Loader (EHDV21150P)
• US xCoding Shrink (EHDV21154P)

Page 474 from 698
Table Description
TDV_XCD_SHRINK_LINKS Useful xCoing Links
Files:
Input Files:
FileName Comment

Format: Text file

with NCP Purchase

Page 475 from 698
records
be 0 or 1
be 0 or 1
be 0 or 1
be 0 or 1
NCP Trip Data File:

Format: Text file

PanId_After NUM 8 Panelist Identification
Period_After NUM 4 Process Period
TripDate_After NUM 6 Trip Date (Date
Shopped)
NHS_Outlet_After NUM 4 NHS Outlet
MarketCode_After NUM 3 Major Market Code
ACNStore_After NUM 5 CAN Primary Store
NumberOfItems_After NUM 3 Number of Items
(Unique UPC’s)
TotalOutlet_After NUM 5 Total Outlet $/Total
UPC $ (Before
Coupon)
Quantity_After NUM 5 Quantity (After Editing)
TripHour_After NUM 2 Trip Hour
Blank_After NUM 1 Blank
PrimShopper_After NUM 3 Primary Shopper (Sex,
Age)
SecShopper_After NUM 3 Secondary Shopper
(Sex, Age)
StoreType_After NUM 1 Store Type
PanelCode_After NUM 1 Panel Code

Page 476 from 698
TotalDollars_After NUM 5 Total Dollars
Spent/Ring Dollars
(Cents)
OnDealTransactions_After NUM 2 Transactions (UPC’s)
On-Deal
MfrCouponTotal_After NUM 4 Mfr. Coupon Total
(Cents)
StoreCouponTotal_After ALPHA 4 Store Coupon Total
(Cents)
StoreMfrCouponTotal_After ALPHANUM 4 Store/Mfr. Coupon
Total (Cents)
Space ALPHA 1 Blank
PanId_before NUM 8 Panelist Identification
Period_before NUM 4 Process Period
TripDate_before NUM 6 Trip Date (Date
Shopped)
NHS_Outlet_before NUM 4 NHS Outlet
MarketCode_before NUM 3 Major Market Code
ACNStore_before NUM 5 CAN Primary Store
NumberOfItems_before NUM 3 Number of Items
(Unique UPC’s)
TotalOutlet_before NUM 5 Total Outlet $/Total
UPC $ (Before
Coupon)
Quantity_before NUM 5 Quantity (After Editing)
TripHour_before NUM 2 Trip Hour
Blank_before NUM 1 Blank
PrimShopper_before NUM 3 Primary Shopper (Sex,
Age)
SecShopper_before NUM 3 Secondary Shopper
(Sex, Age)
StoreType_before NUM 1 Store Type
PanelCode_before NUM 1 Panel Code
TotalDollars_before NUM 5 Total Dollars
Spent/Ring Dollars
(Cents)

Page 477 from 698
OnDealTransactions_before NUM 2 Transactions (UPC’s)
On-Deal
MfrCouponTotal_before NUM 4 Mfr. Coupon Total
(Cents)
StoreCouponTotal_before ALPHA 4 Store Coupon Total
(Cents)
StoreMfrCouponTotal_before ALPHANUM 4 Store/Mfr. Coupon
Total (Cents)
Space ALPHA 1 Blank

Format: Text file
Field Type Length Description

TripDate_After NUM 6 Trip Date (Date Shopped)
Module_After NUM 4 Product Module
PrimShopper_After NUM 3 Primary Shopper
SecShopper_After NUM 3 Secondary Shopper
UPC_After NUM 12 UPC Number
Quantity_After NUM 2 Quantity (Possibly Modified)
Total_Price_After NUM 5 Total Unit Price
DealCode_After ALPHANUM 2 Deal Code
Total_Coupon_After NUM 4 Total Coupon Value
CausalAd_After NUM 1 Causal Ad
CausalDisplayInfo_After NUM 2 Causal Display Info
PriceIndicator_After NUM 1 Price Indicator
BrandCode_After NUM 6 Brand Code
Size_After ALPHA 2 Size
SizeDescription_After ALPHANUM 6 Size Description
SizeNumber_After NUM 7 Size Number

Page 478 from 698
Multi_After NUM 3 Multi
Flavor_After NUM 6 Flavor Code
OrgCode_After ALPHA 3 Org. Code
PaymntMethod_After NUM 2 Freq. Shopper/Method of
Paymnt*
Panel Code_After ALPHA 1 Panel Code
CpnDealValue_After NUM 4 MFG Cpn Deal Value
PanId_Before NUM 8 Panelist Identification
Period_Before NUM 4 Process Period
TripDate_Before NUM 6 Trip Date (Date Shopped)
Module_Before NUM 4 Product Module
PrimShopper_Before NUM 3 Primary Shopper
SecShopper_Before NUM 3 Secondary Shopper
ACNStore_Before NUM 5 CAN Primary Store
UPC_Before NUM 12 UPC Number
Quantity_Before NUM 2 Quantity (Possibly Modified)
Total_Price_Before NUM 5 Total Unit Price
DealCode_Before ALPHANUM 2 Deal Code
Total_Coupon_Before NUM 4 Total Coupon Value
CausalAd_Before NUM 1 Causal Ad
CausalDisplayInfo_Before NUM 2 Causal Display Info
PriceIndicator_Before NUM 1 Price Indicator
BrandCode_Before NUM 6 Brand Code
Size_Before ALPHA 2 Size
SizeDescription_Before ALPHANUM 6 Size Description
SizeNumber_Before NUM 7 Size Number
Multi_Before NUM 3 Multi
Flavor_Before NUM 6 Flavor Code
StoreType_Before NUM 1 Store Type
OrgCode_Before ALPHA 3 Org. Code
PaymntMethod_Before NUM 2 Freq. Shopper/Method of
Paymnt*
MarketCode_Before NUM 3 Major Market Code

Page 479 from 698
Panel Code_Before ALPHA 1 Panel Code
CpnDealValue_Before NUM 4 MFG Cpn Deal Value

Page 480 from 698
NCP File Rules
Trip Data file DV Field Mapping Transformation Filtering Collapsing

Field
PanId ac_hhold Left pad with 0 up to 10 chars None None
TripDate dt_pur_date Reformat to yyyymmdd None None
NHS_Outlet ac_shop Left pad with 0 up to 10 chars None None
TotalOutlet DT5128 (Basket Total FMCG) Divide by 100 (Dec=2) None None
DT5895 (Total FMCG)
DT5896 (Units FMCG)
TotalDollars DT5941 (Basket Total Spent) Divide by 100 (Dec=2) None None

Field
PanId ac_hhold Left pad with 0 up to 10 chars None Merging Key
TripDate dt_pur_date Reformat to yyyymmdd None Merging Key
NHS_Outlet ac_shop Left pad with 0 up to 10 chars None Merging Key
UPC ac_barcode UPC <> 000000000000 UPC <> 999999 Merging Key
Total_Price DT5675 (Price) See formula below
Total_Coupon DT5667 (Price Historical DT)
DealCode DT5662 to DT5665 (Promotions) See mapping below

Page 481 from 698
Output Files :
FileName Comment
• If in ASCII files there are records with KEYS (using the BEFORE fields) that have NO corresponding records in the Output tables, the
rejected files will contain these missing KEY records from the NCP ASCII files.
• Otherwise the rejected files will contain any records from the NCP ASCII files having date values out of the period date range.
Parameters :
Static Parameters:
NONE

Page 482 from 698
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management :


Page 483 from 698

code
table
DATACODET20
blank the files is
missing
missing
Lock. RELEASE Lock.
Lock
generated
correct.Bad file
generated

Page 484 from 698
correct.Bad file
generated
the input files
are empty

oracle syntax
P9900438 NO matching KEYS found Keys (using W NO matching KEYS
in RAWDA BEFORE
fields) in
Purchase
input file has
no
correspondin
g keys in
RAWDA

Page 485 from 698
in TBSK BEFORE
fields) in Trip
input file has
no
correspondin
g keys in
TBSK

date range
the input file
the input file
the input file

Page 486 from 698
TBSKT13 table
deleted from RAWDA
RAWDAILYT60
deleted from PRKEYS
PROCKEYST29 table
P9900432 Number of records Record Count # Trip records with missing KEYS in TBSK and
P9900432 Number of records Record Count # Purchase records with missing KEYS in

Page 487 from 698
EHDV21153P- Duplicate Transmission Detection
Phase:
PHASE 2 – Automatic Validation
Group:
2153 – Duplicate Transmission Detection
As part of Mobile Scan Initiative, there will be a short intermediate period with swapping families having both scanner and new app on
smartphone enabled. A family may report the same trips with both devices to ensure getting their incentives.
There is need to implement a new validation to detect and remove same baskets that would be transmitted twice by the same family to avoid
any wrong reporting to clients.
Following criteria have been agreed to consider duplicate transmissions:

- Same Household, Shop, Date and Total Spent with a registration ID different (DT58972)
- Trip having highest number of items is the reference
- Other Trips (Duplicated) will be checked for Rate3 of common items with Reference against
o Rate > α➔ Basket and Items are removed
o Rate <= α ➔ Basket and Items are retained
Since some countries may have intermediate situation with Legacy loading for Scanners and NCPS suite for Mobile (DSR to DV), detection will be
applied at final stage on validated data (Rawda and Total Basket)
Key terms and definitions

1. Swapping family = family having already a scanner and starting to download and use Nielsen Mobile App
2. Registration ID is member level information that will be newly stored with Mobile data into one of Trip Data facts. There will be some impact
on the management of these facts to be also considered as part of this evolution.
3. Rate will be computed against duplicate basket item count – ex : with a reference basket having 10 items ; a duplicate having 8 Items including
6 in common with reference will have a rate of 6/8=75%

Page 488 from 698
Prerequisites:
Rawda Loader or Back Data Load completed for all sources of data.
Table Description
Files:
Input Files: NONE
Output Files :
FileName Comment
FILEO1 Removed Keys report

Page 489 from 698
Layout : Text file reporting removed keys as follow :
Household – Shop – Date – Time – Barcode
Barcode would be set to 0 when Key is a basket one.
Parameters :
Static Parameters:
Parameter Type Mode Comment

ALPHA NUM Mandatory Alpha rate
expressed in
percentage
ALPHA value will be set as 75 for 75% but not 0.75
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
File Selection No

Page 490 from 698

code
P0010035 Mandatory Static S ALPHA values ALPHA value in
Param: ALPHA is cannot be empty Static parameter
not defined. for the job
should be
defined
P9900171 Rework mode is not S Rework mode Submit the job in
allowed, Please use cannot be used. Normal Mode
Normal mode.
P0020608 Count of Reference I Defines the Defines the
Keys. number of number of
distinct duplicate distinct duplicate
key combination. key combination.
P0020608 Total Count of I Defines the total Defines the total
Duplicate Basket number of number of
keys duplicate key duplicate key
combination. combination
P0020608 Count 1 Duplicate I Defines the Defines the
Keys number of number of
keys which has keys which has
one duplicate. one duplicate.
P0020608 Count 2 Duplicate I Defines the Defines the
Keys number of number of
keys which has keys which has
two duplicates. two duplicates.
P0020608 Count 3 or more I Defines the Defines the
Duplicate Keys number of number of
keys which has 3 keys which has
or more 3 or more
duplicates. duplicates.

Page 491 from 698
P9900435 Removed baskets I Defines the Defines the
number of number of
baskets baskets
removed. removed.
P9900142 Retained baskets I Defines the Defines the
number of number of
baskets retained baskets retained
due to condition due to condition
not getting not getting
matched. matched.
P9900142 Count of No Items I Defines the keys Defines the keys
Keys which are not which are not
available in T60 available in T60
and available and available
only in T13. only in T13.
P9900142 Number of I Defines the Defines the
Rawdailyt60 number of keys number of keys
Records deleted which are which are
removed in T60. removed in T60.
P9900142 Number of I Defines the Defines the
Prockeyst29 number of keys number of keys
Records deleted which are which are
removed in removed in
PROCKEYST29. PROCKEYST29.
P9900142 Number of allt11 I Defines the Defines the
Record deleted number of keys number of keys
which are which are
ALLT11. ALLT11.
P9900142 Number of Tbskt13 I Defines the Defines the
Records deleted number of keys number of keys
which are which are
TBSKT13. TBSKT13.

Page 492 from 698
EHDV21154P- US xCoding Shrink
Phase:
The US xCoding Shrink process objective to populate the TDV_XCD_SHRINK_LINKS table by :
• Populating analysis table TDV_XCD_DUPCHECK_LINKS table
• Filtering LAC links (Excluding LAC links)
• Keeping unique links
• Reporting multi links or potential coding errors
• Keeping most recent or most frequent link when multiple exists
• Removing or ignoring overlapping links
Recommendations:
Constraints:
Prerequisites:
EHDV21070P – OGRDS Load must have been run previously

Page 493 from 698
Table Description
TDV_XCD_DUP_LINKS xCoding analysis Links
Files:
Output File:
FileName Comment
FILEO1 xCoding Errors Report File
Text file reporting all potential coding errors with the Rejection Reason.
Key is written as Barcode – Date From – Nankey – Date To
followed by rejection reason which can be (MULTI-LINK – MULTI-DATE_TO – RANGE_OVERLAP).
Parameters :
Static Parameters:

Type
PARM ALIGN_UPC Mandatory YES or NO
Dynamic Parameters:

Name (Yes/No)
# Periods Yes Value will be ignored

Page 494 from 698
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

NONE

Page 495 from 698

code
blank the files is
missing
missing
oracle syntax

range
P9900415 # Records Loaded in # Records Loaded
TDV_XCD_DUP_LINK
S
P9900439 # Unique Links Loaded # Unique Links
Loaded
P9900440 # Overlap Date Range # Overlap Date
Links Loaded Range
P9900441 # Multi Links Loaded # Multi Links Loaded
P9900442 # Overlapping records Record Count # Records exported
removed
P9900432 # Barcodes exported in Record Count # Records exported
the output file

Page 496 from 698
EHDV21155P- US xCoding
Phase:
The US xCoding process objective is to use new TDV_XCD_SHRINK_LINKS table to update the rawda and prockeys data with appropriate nankey
or zero. Mapping will be done using barcode and purchase data compared to links effective dates
Recommendations:
Constraints:
Prerequisites:
EHDV21154P – xCoding Shrink must have been run previously
Table Description
PROCKEYST29 List of Rawda & TBSK Processing keys
Files:
Output File:
FileName Comment
FILEO1 Non xCoding Report File

Page 497 from 698
Text file reporting all barcodes cross coded to zero
Record is written as Barcode – count of purchase acts (sorted descending)
Parameters :
Static Parameters:
NONE
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 498 from 698
NONE

Page 499 from 698

code
blank the files is
missing
missing
oracle syntax

date range
P9900408 # PROCKEY Records # Records
Xcoded Xcoded
P9900408 # RAWDA Records # Records
Xcoded Xcoded
P9900409 # PROCKEY Records # Records NANs are set to zero
NOT Xcoded NOT Xcoded
P9900409 # RAWDA Records # Records NANs are set to zero
NOT Xcoded NOT Xcoded
P9900432 # Barcodes exported Record Count # Records exported
in the output file

Page 500 from 698
EHDV21156P- US Reclassification
Phase:
The US Reclassification process objective is to use new TDV_XCD_SHRINK_LINKS table to

apply any change in the last OGRDS refresh onto the already loaded data.
✓ 2 period ranges will be considered for this :
• Date Range for changes to apply – considering update date of links (The date range of changed links is computed based on job
submission period and Number of Periods back)
• Date range of Purchases to be impacted by links changes (This is computed using the PERIOD_BACK dynamic parameter)
As a matter of example, user can decide to apply every week the cross coding changes updated in that week onto the last 3 years of data prior
that week
Recommendations:
Constraints:
Prerequisites:
EHDV21154P – xCoding Shrink must have been run previously
Table Description

Page 501 from 698
PROCKEYST29 List of Rawda & TBSK Processing keys
Files:
Input Files:
Input file layout :

Output File:
FileName Comment
FILEO1 List of barcodes with changed NANKEYS
Text file with a header record: barcode – “nankey before” – “nankey after” – # Records
One record by combination with the count of purchases records reclassified for it.
Parameters :
Static Parameters:

Type
PARM PERIOD_BACK Mandatory PERIOD_BACK <> 0 ➔Start date = start day of submitted period –
PERIOD_BACK -1
▪ PERIOD_BACK = 0 ➔ Start date fixed to 20100101
▪ End date = end day of submitted period
PARM MODE Optional FILE or OGRDS

Page 502 from 698
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

NONE

Page 503 from 698

code
blank the files is
missing
missing
P0020505 Mandatory Static Param: Stat.Val not S Missing Mandatory Define a value
PERIOD_BACK is not present Static Parameter for the static
defined parameter
PERIOD_BAC
K
P9900136 Value given to Static Item not S A non numeric value Valid values
Parameter number was defined as a are only
PERIOD_BACK is NOT a value for the positive
NUMBER PERIOD_BACK INTEGERS
oracle syntax

date range
P9900408 # Records Xcoded in # Records
RAWDA Xcoded
P9900408 # Records Xcoded in # Records
PROCKEYS Xcoded
P9900408 # Records with # Records NAN values changed from a NANKEY to 0
NANKEY reset to 0 Xcoded
P9900408 # Records set to a # Records NAN values changed from 0 to a NANKEY
NANKEY from 0 Xcoded

Page 504 from 698
P9900408 # Records set to # Records NAN values changed from one NANKEY to
another NANKEY Xcoded another NANKEY
P9900408 # Different Number of different Barcodes impacted from
BARCODES impacted the Reclassification process
P9900408 # Different NANKEYS Number of “NANKEYS Before” impacted from
BEFORE impacted the Reclassification process
P9900408 # Different NANKEYS Number of “NANKEYS After” impacted from
AFTER impacted the Reclassification process
P9900432 The List of Changed The List of Changed NANKEYS exported to
NANKEYS exported the OUTPUT File
to the OUTPUT File

Page 505 from 698
EHDV21157P- DV File Merger
Phase:
New DV module objective is to merge a set of files together.
✓ Up to 10 files can be instructed for the input

✓ 1 Output file will be created by concatenating the input files together
✓ Specific method for RMS Price Files :
o First file is considered as master file is copied to output file as it is

o With other files header records are filtered / skipped so that output file has only
o 1 header record
o No Transformation is applied onto records contents
Recommendations:
Files:
Output File:
FileName Comment
FILEO1 Final Merged output file
Merged output file with records combined together from all input files .
Parameters :

Page 506 from 698
Static Parameters:

Type
LIST MERGE_METHOD Mandatory Merge method to apply
NUMERIC FILE_COUNT Mandatory Mandatory Number of active files to merge
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection Yes Input files

code
P9900435 InputFiles count Err FILE_COUNT S Available files and Check if any of
and input files static parameter the files is
count mismatch mismatch missing
P0020126 File Name not present FileName not S File is missing Check for
present master file or
secondary
input file 1
P9900014 Empty File Empty File S Master Input File is Master input
empty file has no
contents
P9900014 Empty File Empty File S Secondary input file 1 Secondary

Page 507 from 698
is empty input file 1 has
no contents
P9900395 External job failed External job S STANDARD_ERROR Check the
failed message gives status external job
of the external job logs with help
fo support
team.
P9900434 Static Param Error File_count must S Static parameter Define a
be between 2 value must be number
and 10 between 2 and 10 between 2 and
10
P9900246 Submission for period Submission for S Job cannot be Submit a valid
period 0 is not submitted with Period period
allowed 0

date range
P9900109 FileId – FileName FileId – Display the name of the input file and its type
FileName
P0020608 Static Value Static Value Static parameter value
P9900394 External Job name External Job Name of the external merging job name
name
P9900393 External Job status External Job Status of the submitted external job
status

Page 508 from 698
EHDV21158P- Brand Exception
Phase:
PHASE 1 – VALIDATION AREA
This job checks and detects any purchases wrongly declared in some shops by 509ehaviour509.
For example, purchase of Mars Pet Foods in LIDL in which those products are not referenced. This cannot be setup or detected by current
Private Brand module since Mars brand can be present but not for that category.
Today’s solution is to manually detect those wrong combinations and prepare surgery files.
More specifically it handles the same process as manual one in an automated way.
A new set of screens is provided so that user can drive the validation and adapt appropriate setup.
This new job takes care of :

- Detecting exceptions
- Correct them to assigned shop with similar rules as Private Brand :

o Whole basket is wrong ➔ whole basket move to new shop
o Some items wrong in basket ➔ create new basket with wrong items
- Log the corrections in the system for reference
Recommendations:
Constraints:
Prerequisites:
● Data Selection (EHDV21068P)

Page 509 from 698
Table Description
TDV_BRAND_EXCEPTION This table contains the Brand Exception instructions, by defining the IBD and SHOPGROUP for validating
purchase data for BRAND EXCEPTIONS. Also the reclassification shop to be used for these BRAND
EXCEPTIONS is defined.
Files:
Input Files:
NONE
Output Files:
FILEO1 in normal txt format (comma ‘,’ separated file).
Report of changes
Position Description
1 Audit
2 Date
3 Time
4 Hhold
5 Old Shop
6 Old Shop Name
7 New Shop
8 New Shop Name
9 Brand
10 Barcode
11 Nankey

Page 510 from 698
Output Files (cont’d):
FILEO2 in .csv format (comma ‘,’ separated file).
Corrected Items by Shop
Position Description
1 Shop
2 Shop Description
3 Total number of records
4 Number of records corrected
5 Percentage(%) of records corrected
Parameters:
Static Parameters:

Type
NUMERIC PCT_SHOP_CORR Optional Minimum % of corrections per Shop (FILEO2)
NUMERIC SHOP_NB_TRANS Optional Minimum Number of transactions per Shop(FILEO2)
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 511 from 698
Rework Management:


code
oracle syntax
P9900223 No KEYS have been No Records W No eligible records
selected for validation Selected for validation have
been selected,
based on the Brand
Exception
Instructions

date range
P1044027 # Purchase Records Update # Purchase Records Updated in KEYST14
Updated in KEYST14 Count- Information message only. No action required
KEYST14
P0020612 Total Number of UPD Counter # Purchase Records Updated in P359_1T12
Records updated Information message only. No action required
P1580001 # Purchase Records UPD Counter # Purchase Records Updated in P359_1T12
Updated in Information message only. No action required

Page 512 from 698
P359_1T12
P1002007 # Records Updated in Update count # Records Updated in FAMILY_LOG.
FAMILY_LOG – T67 Information message only. No action required
P9900405 # Purchase Records Insert Cnt- # Purchase Records Inserted in KEYST14
Inserted in KEYST14 KEYST14 Information message only. No action required
P0020612 # Purchase Records UPD Counter # Purchase Records Inserted in P359_1T12
Inserted in Information message only. No action required
P359_1T12
P1580002 # Records Inserted in Insert count – # Records Inserted in FAMILY_LOG
FAMILY_LOG T67 Information message only. No action required
P9900405 # Total Records Insert Cnt- # Total Records Inserted/Updated in
Inserted/Updated in KEYST14 KEYST14
KEYST14 Information message only. No action required
P0020612 # Total Records UPD Counter # Total Records Inserted/Updated in
Inserted/Updated in P359_1T12
P359_1T12 Information message only. No action required
P1002006 # Total Records Insert count – # Total Records Inserted/Updated in
Inserted/Updated in T67 FAMILY_LOG
FAMILY_LOG Information message only. No action required
P1047009 # Basket KEYS Delete Cnt- # Basket KEYS deleted
deleted KEYST14 Information message only. No action required
P1047009 # Purchases deleted Delete Cnt- # Purchases deleted
KEYST14 Information message only. No action required

Page 513 from 698
EHDV21159P- US Threshold Computation
Phase:
PHASE 5 – DATA MANIPULATION AREA
This job computes the “Total Threshold” (DT5942) according to the following US specific rules:
o Exclude non coded items (nankey = 0 after cross coding)
o Exclude some specific magnets items based on short list (Excluded Magnet reference File)
o Sum Up Item Total Value ( Unit Price * Quantity) + Coupon
If Trip Key does not exist yet it will be created as follow:

o DT5942 = new value from formula
o DT5126 = 1 (deleted flag)
o DT5666 = sum(quantity) – all items
o DT5941 = sum (Unit Price * Quantity) – all items
o DT5128 = sum ( ( Unit Price * Quantity) – Coupon) – all items
Note : For python version, Hive data refresh is done at the beginning of this process for the submission audit, period (back-periods included) for
the following tables:
• Prockeyst29
• Rawdailyt60
• Tbskt13
oracle data refresh is done at the end of this process for the submission audit, period (back-periods included) for following tables:
• Prockeyst29
• Tbskt13

Page 514 from 698
Recommendations:
Constraints:
Prerequisites:
● US Data Loader (EHDV21150P)
● US xCoding Shrink (EHDV21154P)
● US xCoding (EHDV21155P)
Table Description
Files:
Input Files:
FileName Comment
FILEI1 EXCLUDED MAGNETS REFERENCE
Excuded Magnets Reference:
Format: Text file delimited by ‘;’
Layout: Field Type Description

UPC Code AlphaNum UPC Code
NANKEY Num Nankey Code
Output Files:
NONE

Page 515 from 698
Parameters:
Static Parameters:
NONE
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management :


Page 516 from 698

code
blank the files is
missing
missing
P9900232 Input file format not Input file S “Excluded Magnet Check input file
correct format no Reference” file layout
format not
correct.Bad file
generated
P0020722 “Excluded Magnet Input File S “Excluded Magnet
Reference” file does not Not Present Reference” file does

oracle syntax

Page 517 from 698

date range
P9900123 Total Number of Record Count # Records read from input file “Excluded
Records Present in Magnet Reference” :
P9900241 Total Number of Record Count # TBSK Records updated: Information
Records updated message only. No action required

Page 518 from 698
EHDV21160P- Uncoded Extraction
Phase:
As part of ZTV CPS project, there is need to automate the extraction of the barcodes that are PGI’s.
Process Steps :
- Read the input file to get the list of multilink barcodes

- For each set Groupid in TDV_NPP_EXTR_SETUP
o Resolve shop group
o From the submission period to all back periods (depending on number of periods
NB_PERIOD_LOAD_AREA), get uncoded data from ALLT11 table where
NAN_KEY== 0 and BARCODE <> 0 and BARCODE length <= 13 and exclude all
multilink barcodes
o if PGI_NAN_KEY present in static parameter then
▪ From the submission period to all back periods (depending on number of periods
NB_PERIOD_LOAD_AREA), get uncoded data from ALLT11 table where
NAN_KEY == PGI_NAN_KEY and BARCODE <> 0 and BARCODE length<= 13 and exclude all multilink barcodes
▪ For all back periods (depending on number of periods), get uncoded data from
RAWDAILYT60 table where NAN_KEY == PGI_NAN_KEY and
BARCODE<>0 and BARCODE length <= 13 and exclude all multilink
barcodes
o Sum transaction count by barcode and apply filter based on NES_NC_TRANSC if
present
o Keep top barcodes based on NES_NC_FIRST if present
o Store data in temporary table
- If NES_RD_FILE set to ‘Y (YES)’ Store in the file FILEO1 all records from the temporary table
sorted by transactions count respecting these rules :
o if barcode length is 13 and start with 00000 ➔ keep only 8 last digits
o if barcode length 13 and start with 0 ➔ keep only last 12 digits
o else keep barcode as it is
- If NES_NPP_FILE set to ‘Y (YES)’, Extract and store in the file FILEO2 (without transactions
count) first MAX_ITEM records from the temporary table sorted by transactions count

Page 519 from 698
respecting these rules :
o if barcode length is 13 and start with 00000 ➔ keep only 8 last digits
o if barcode length 13 and start with 0 ➔ keep only last 12 digits
o else keep barcode as it is
- If SCRIPT_CFT present in static parameter then execute this script
- If NES_SI01_FILE set to ‘Y (YES)’. Store in the file FILEO3 all records

Header is always:
<hd><mstp>101</mstp><ss>1</ss></hd>, then for each element
o <pgc> is filled with the crosscode id taken from TDV_NPP_EXTR_SETUP (NES_AC_CROSSGROUPID)
o <xcd> is filled with barcode with these conditions:
▪ if barcode length is 14 or 13, then let as it is
▪ if barcode is lesser than length 13, add left padding with ‘0’ to make length 13
o <itmd> is always “CPS Code Date(date format YYYYMMDD)”
The item description is included a date in it so that same item can be send multiple times.
O <noss> is filled with number of transactions
o <prc> is the median price
o <iccd> is the current country
Note that NES_SI01_FILE is in XML format
Summary:
- Report number of uncoded records found by Retailer (Shop Group)
- Report number of PGI’s records found in ALLT11 by Retailer (Shop Group)
- Report number of PGI’s records found in RAWDAILYT60 by Retailer (Shop Group)
- Report number of uncoded records extracted in file by Retailer (Shop Group)
- Report number of uncoded records extracted in file in ALLT11 by Retailer (Shop Group)
- Report number of PGI’s records extracted in file in RAWDAILYT60 by Retailer (Shop Group)
- Report the total number of lines extracted in file
Recommendations:
Constraints:
Prerequisites:
EHDV21008P – xCoding

Page 520 from 698
Table Description
TDV_NPP_EXTR_SETUP Useful xCoing Links
TDV_NES_SETGROUP_SETUP Set group id and its description
separated
Files:
Input File:File id FILEI1
List of barcodes with more than one record in Xcoding Instruction table(08P Table)

1 GROUP ID Xxxx
2 BARCODE 0000000000…..
Output File:
Extracted data in CSV format (File id: FILEO1) and FILEO2
FILEO1:
Column Name
1 Retailer
2 BARCODE
3 Transaction

Page 521 from 698
FILEO2:
Column Name
1 Retailer
2 BARCODE
FILEO3:
FileO3 is xml format file which contains <xitm>, <pgc>,<xcd>,<itmd>,<noss>, <prc>, <iccd>
As an example:
<hd><mstp>101</mstp><ss>1</ss></hd>
<xitm>
<pgc>FICPSLIDL</pgc>
<xcd>0000012345678</xcd>
<itmd>CPS Code</itmd>
<noss>7</noss>
<prc>139</prc>
<iccd>FI</iccd>
</xitm>
<xitm>
<pgc>FICPSLIDL</pgc>
<xcd>0000012345678</xcd>
<itmd>CPS Code</itmd>
<noss>9</noss>
<prc>134</prc>
<iccd>FI</iccd>
</xitm>
Parameters :
Static Parameters:

Type
NUMERIC MAX_ITEM Mandatory The MAX number of items to be extracted in the

Page 522 from 698
Output file
NUMERIC PGI_NAN_KEY Optional The PGI Nan Key to be searched in ALLT11 and
RAWDAILYT60
LIST SCRIPT_CFT Optional External Script to execute
NUMERIC NB_PERIOD_LOAD_AREA Optional Number of back periods, used for Loading
Area.Default value is 1.Zero is not allowed as
input.
NUMERIC SET_GROUPID Optional Set groupid to choose ,default is zero
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 523 from 698
NONE

code
blank the files is
missing
missing
oracle syntax
P1043001 No Exraction Set Up has No records W No Exraction Set Up Setup
been defined for cycle: present has been defined for instructions in
cycle in TDV_NPP_EX
TDV_NPP_EXTR_S TR_SETUP
ETUP table table
P9900136 Static Parameter value for Item not S Static parameter Check the
PGI_NAN_KEY is NOT a number value given is not value given to
NUMBER. NUMERIC static
parameter
PGI_NAN_KE
Y to be
NUMERIC
P9900136 Static Parameter value for Item not S Static parameter Check the
MAX_ITEM is NOT a number value given is not value given to
NUMBER NUMERIC static
parameter
MAX_ITEM to
be NUMERIC

Page 524 from 698

date range
P9900199 # Uncoded records Number of The Total Number of( Barcode <> 0 and
found by Retailer in Transactions NAN_KEY=0 OR Barcode <> 0 and
ShopGroup: NAN_KEY=PGI_NAN_KEY) by Retailer
selected from both ALLT11 and
RAWDAILYT60
P9900199 # PGI’s records found Number of The Number of Barcode <> 0 and
in ALLT11 by Retailer: Transactions NAN_KEY=PGI_NAN_KEY by Retailer from
ALLT11
P9900199 # PGI’s records found Number of The Number of Barcode <> 0 and
in RAWDAILYT60 by Transactions NAN_KEY=PGI_NAN_KEY by Retailer from
Retailer: RAWDAILYT60
P9900369 # Lines exported in # Records Total Number of records exported based on
the output file: exported the MAX_ITEM static parameter value
P9900199 # Uncoded (ALLT11) Number of The Number of Barcode <> 0 and
records extracted in Transactions NAN_KEY=0 by Retailer (exported in the
file by Retailer: output file)
P9900199 # PGI’s(T11 and Number of The Number of Barcode <> 0 and
T60)records extracted Transactions NAN_KEY=PGI_NAN_KEY by Retailer
in file by Retailer: (exported in the output file)
P9900394 Automation Job External JOB The script executed is the one defined in
submitted with the name SCRIPT_CFT in static parameters (if present)
following name:
P9900393 Automation Job External JOB
ended SUCCEED
SUCCESSFULLY
P9900392 & Automation Job External JOB
P9900395 FAILED FAILED

Page 525 from 698
EHDV21161P- PGI Xcoding by Default
Phase:
PGI Xcoding by Default simplifies the process of assigning PGI code to the uncoded products.
This functionality detects all uncoded barcodes and sets the default PGI.
Finally, the barcodes for which the PGI’s assignment is done are stored in a file.
Process Step :
- Read the input file to get the list of multilink barcodes

All the following steps use the ALLT11 table and apply the filter BARCODE <> 0 and
BARCODE length <= 13 , exclude all multilink barcodes
- Get distinct purchase keys from the table where NAN_KEY == 0
- Get distinct purchase keys already coded with PGI (where NAN_KEY == PGI_NAN_KEY)
- The items where NAN_KEY == 0 and NC_DATAVALUE=BCTYPE_EXCEPTION when
AC_DATACODE=5098 (means barcode type) are assigned with PGI_EXCEPTION
- Store in a file the distinct barcodes where PGI_EXCEPTION is applied.
- Store in the same file the distinct barcodes where NAN_KEY == 0 on which PGI_NAN_KEY
will be applied. This file contains 3 columns, Processing Group Code, barcodes and Nan Keys
(PGI_EXCEPTION or PGI_NAN_KEY). The Processing Group Code is obtained using
SHOPCOLLID_LINKT19 table with AC_SHOP. If there is no match in SHOPCOLLID_LINKT19,
then Processing Group Code will be DEFAULT_CCGROUP.
- Update the NAN_KEY with the PGI_NAN_KEY parameter value where NAN_KEY == 0
Summary:
- Report number of uncoded transactions from the 2nd step
- Report number of existing PGI transactions from the 3rd step
- Report number of barcodes coded with PGI_EXCEPTION from 4th step
- Report number of barcodes coded with PGI from 6th step
-Report total count of updated records (count from step 4 + count from step 6)
Recommendations:

Page 526 from 698
Constraints:
Prerequisites: Cross Coding must have been already done
Table Description
Files:
Input File:File id FILEI1
List of barcodes with more than one record in Xcoding Instruction table(08P Table)

1 GROUP ID Xxxx
2 BARCODE 0000000000…..
Output File:
Extracted data in CSV format (File id: FILEO1)

FILEO1:
Column Name
1 Processing Group Code
2 BARCODE
3 Nankey

Page 527 from 698
Parameters :
Static Parameters:

Type
NUMERIC PGI_NAN_KEY Mandatory The PGI Nan Key to be searched in ALLT11
and RAWDAILYT60
NUMERIC PGI_EXCEPTION Optional PGI Nan Key for special case
NUMERIC BCTYPE_EXCEPTION Optional Mandatory when PGI_EXCEPTION is defined
CHAR DEFAULT_CCGROUP Mandatory Default Processing Group
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 528 from 698
NONE

code
P0020125 File Name is blank File Name is S File is missing Check if any of the files
blank is missing
P0020126 File Name not present FileName S File is missing Check if any of the files
not present is missing
P0010035 Oracle Error Oracle Error S Oracle Error Check for oracle syntax
P1043001 No default PGI xCoding No records W
applied. present

date range
P9900213 Count of incorrect Number of
barcodes ignored: barcodes
P1006006 # Uncoded Update Cnt-
Transactions: ALL table
P1006006 # Existing PGI Update Cnt-
Transactions: ALL table
P1006006 # Barcodes coded Update Cnt-
with PGI: ALL table
P1006006 Total Count of Update Cnt-
Updated Records: ALL table
P9900369 # Barcodes exported # Lines
in the output file: Exported

Page 529 from 698
EHDV21162P- PRICE COMPUTATION BY CHARACTERISTICS
Phase:
PHASE 4 – SERVICE AREA
As part of ZTV CPS project, the computation of standard price needs to be improved especially
because of the low penetration.
Currently, when the penetration is low for specific products the price computed with the standard price computation is far from the reality as its
based on few items.
The solution proposed is to compute prices differently. The idea is to compute the price of a set of homogeneous items with the same
characteristics and affect this price to the items with no prices.
Process Principles:
✓ For every Set of Module/Characteristics, get all characteristic values, NAN_KEYs and Weights
✓ Get the purchases of related NAN_KEYs and compute LEVELs
✓ Compute the median price by LEVEL and assign to the NAN_KEYs
✓ Store price using filters
• Rawdailyt60
• Cnanpricet34
• Cnanpricet34
Process Step:
Resolve SHOP_GROUP_ID (if defined)

➢ Step 1 : Getting characteristics
• For each module from the TDV_PCC_SETUP and for the corresponding shop

Page 530 from 698
○ Get all NAN_KEYS from NAN_ITEMST73 and all characteristic values codes from the table NAN_CHAR_VALUEST75 for the list of
characteristic types ( CHARTYPE1, CHARTYPE2, CHARTYPE3, CHARTYPE4 ) and NC_NUMERIC_VALUE from the table
NAN_WEIGHT_VALUEST79 using PCC_WEIGHTCODE
➢ Step 2 : Get RAWDATA purchases

○ Get all NAN_KEY and DT5667 from the RAWDAILYT60 except the items where DT5904 is 3, 11 and 20
➢ Step 3 : Median Price Computation
○ Count all purchases by LEVEL4 and by module, if count is greater than NBOBS then
the median price MP1 is computed with CONV PRICE
○ For each LEVEL4 found, if the same LEVEL4 is matched then assign the price with NC_NUMERIC_VALUE*MP1
○ For each LEVEL3 found, if the same LEVEL3 is matched and if the price is not present then assign the price with
NC_NUMERIC_VALUE*MP2
➢ Step 4 : Store the price to the CNANPRICET34
Finally all new prices computed are assigned to the price (when missing) of CNANPRICET34 to the corresponding NAN_KEY according to the
filters :
● If the price in CNANPRICET34 is missing for the period P-1 and the price is 0 (can be
get with DATACODE 5667 historical price) in P359_1T12 then store the price for the
period P-1
● If the price in CNANPRICET34 is missing for the period P then store the price for the
period P

Page 531 from 698
Note : A new key EXCLUDE_WEIGHT_CODES is added in the setup_param table. By default this field will be kept empty and for GB the
values are NDS,OZ – to be set on request
This following figure shows how the storage happens for different contexts :
FPV context on P-1 :

Price Value in
Price Value in P359
all = 0 any <> 0
Std Price at Yes No Computation No Computation

Nan P-1 in T34 No Store computation No Storage
P359
APV context on P :td Price at
Na
Yes No Computation
Std Price at
Nan P in T34 Store computation
No
n P in T34
➢ Step 5 : Output file
○ List of NAN_KEY, BARCODE and module description of all items purchased in
P359_1T12 for the period P and where the price is not assigned in CNANPRICET34
Summary :
➢ Report the number of records found by module
➢ Report the number of items without price by module (with description)
➢ Report number of items for which the price is assigned at LEVEL4 and compute the

Page 532 from 698
percentage (number of current items / total number of items without price * 100) by module
percentage (number of items / total number of items without price * 100) by module
Recommendations:
Constraints:
Prerequisites: Availability of the data in validation area and in RAWDAILYT60 for the previous periods
Table Description
NAN_ITEMS List of NAN_KEY loaded from EIMDB
NAN_CHAR_VALUEST75 Item characteristics values extracted from EIMDB
NAN_WEIGHT_VALUEST79 Item weight value id extracted from EIMDB
TDV_PCC_SETUP Price Computation setup table
TRIP_SHOPGROUPT117 User defined shop groups with selected shop characteristic
TRIP_SHOPGROUP_VALT119 User defined shop groups with selected shop characteristic
values
TRIP_FORMULAST118 Predefined Formulae NOT modifiable by users

Page 533 from 698
Files:
Input File:
NONE
Output File:
Extracted data in CSV format (File id: FILEO1)
Column Name
1 NAN KEY
2 BARCODE
3 MODULE DESCRIPTION
Parameters :
Static Parameters:

Type
NUMERIC NBOBS Mandatory Minimum number of transactions
NUMERIC SHOP_GROUP_ID Optional Shop group on which user works
Dynamic Parameters:

Name (Yes/No)

Page 534 from 698
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection Yes
Rework Management:

NONE

code
P9900136 Item is not a number Item is not a S Static parameter Give number
number value given is value for Static
NOT a number parameter

Page 535 from 698
P0020505 Mandatory Static Mandatory S Mandatory Static Set the
Parameter has not been Static Parameter has mandatory static
set Parameter not been set parameter and
NOT defined then submit the
job again
P9900155 No purchase records No purchase S No records Check the
selected records selected from TDV_PCC_SETU
selected RAWDAILYT60 P for the
based on submitted cycle
TDV_PCC_SETU
P
P9900223 There is no standard price W All the NANS Check the
computed for this selected from TDV_PCC_SETU
Module/536lignment536sti RAWDAILYT60 P for the
cs set based on submitted cycle
TDV_PCC_SETU
P have already
Standard Price
computated in
CNANPRICET34
table

date range
P1044021 # NanKeys found Instruction
without Price:
P1044021 # NanKeys which Instruction
Price assigned at
LEVEL
P1044021 % NanKeys which Instruction
Price assigned at
LEVEL

Page 536 from 698
P1044021 Output File has NOT Instruction
been generated.
P1044021 Output File has been Instruction
generated. # NanKeys
which Price has NOT
been computed:
P1049038 # Records Inserted in Insert Cnt-
CNANPRICET34 for CNANPRICE
submitted period
P1049038 # Records Inserted in Insert Cnt-
CNANPRICET34 for CNANPRICE
submitted period -1
P9900175 # Nans with NULL Missing Value
Weight Value:

Page 537 from 698
EHDV21163P- PGI DATA SELECTION
Phase:
PHASE 1 – VALIDATION AREA
This process detects purchases related to PGI and MultiEANlink Nans which are present in RAWDAILYT60. Having detected these NANs, these
are moved back to the validation area and deleted from the output tables.
Process Step:
Detects all PGI or Mul_EANlink NANKEYs changed in the EHDV21061P and EHDV21078P jobs
using the table TDV_REX_LOG according to the period and cycle.
• Get the list of DATACODES used by the standard loader for loaded having as destination the RAWDAILYT60. If in this list DT5675 and
DT5650 are missing, then check if DT5674 and DT5650 are present. In case that these Datatypes are also missing, this means that the
Standard Loader job was not correctly finished. In this case, stop this current job. If DT5675 and DT5650 are present or DT5674 and
DT5650 are present go further.
• For all NANKEYS found, extract from RAWDAILYT60 all records with AC_AUDIT, DT_PUR_DATE, AC_TIME, AC_HHOLD, AC_SHOP,
AC_BARCODE and all DATACODES with values( DT5642,DT5667,DT5904,DT5095 and other datacodes from the list got from the previous
step ). GET the NEWNAN for all these NANs selected from TDV_REX_LOG by joining on the purchase key.
• For each record found in the RAWDAILYT60, create in the validation area P359_1T12 as many records as the number of DATACODES. All
the datacodes identified in step 1 are loaded back.
• When doing this transformation, apply the following rule:
If PRICE_ZRO is ‘Yes’ then DT5675=0

If PRICE_ZRO is ‘No’ then the DT5675 value is taken from DT5667
(DT5667 is not loaded back into P359_1T12)
o The DT5650 is taken the value of the DT5642 (DT5642 is not loaded back into P359_1T12)
o While moving the DT5095 data it is inserted as -1 for APV rerun purpose.(Same for keysT12 insert)
o For each record found in the RAWDAILYT60, a new key is created in KEYST14 and the datacode associated with filenumber
(DT5095). Also TM_TIMESTAMP should be the job submission timestamp.

Page 538 from 698
o All records identified in the RAWDAILYT60 are deleted from this table.
o All records identified in the PROCKEYST29 with the same keys are deleted from this table.
o All changed records (for the purchase keys identified in previous steps) and are present in the FAMILY_LOGT67are also
deleted,this deletion is done using an intermediate table GT_TDV_REX_LOG
Recommendations:
Constraints:
Prerequisites:
Nan reclassification by X-Coding job and Re-Xcoding by exception job must have been completed before job submission.
Table Description
DATATYPEST07 The table contains the list of the datatypes and the position in
the record
PROCKEYST29 List of Rawda Processing keys (used here to check if data
loaded are already present in output tables)
MODULES_TRANST32 Description of modules
NAN_ITEMST73 List of NAN_KEY loaded from OGRDS
P359_1T12 DV Validation area data
Files:
Input File:

Page 539 from 698
NONE
Output File:
NONE
Parameters :
Static Parameters:

Type
LIST PRICE_ZRO Mandatory ‘YES’ or ‘NO’
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 540 from 698
NONE

code
job again
P1024002 Datacodes 5675 and S
5650 not found, Job is
stopped. Standard Loader
job may not be terminated
correctly

date range
P0020608 Static PRICE_ZRO Static Value
Parameter value
P9900141 # Records Inserted in Count of records inserted in P3591_T12 table
P3591_T12 table
P9900405 # Records Inserted in Count of records inserted in KEYST14 table
KEYST14 table
P1049031 # Records Deleted Count of records deleted from RAWDAILYT60
from RAWDAILYT60 table
table

Page 541 from 698
P1049037 # Records Deleted Count of records deleted from
from PROCKEYST29 PROCKEYST29 table
table
from FAMILY_LOGT67 table
FAMILY_LOGT67
table

Page 542 from 698
EHDV21164P- LOYALTY CARD USAGE RATE COMPUTATION
Phase:
Compute data of loyalty card percentage use and store into TDV_LOYALTY_VAL.
Process Step:
From TRIP_SHOPGROUPT117 when AC_LOYALTY_CARD is set to ‘Y’, get all shop groups and
do shop resolution to get all shop lists
- For each shop group, count all purchases in TBSKT13 where loyalty cards are used. This can be
checked using the DT5949 with the DT_VALUE_FID. Count all total purchases for this shop
group, finally calculate the percentage of use of the loyalty card.
- The percentage computed is inserted into the TDV_LOYALTY_VAL , with Shop Descrip_on and
period.
• Tbskt13
Recommendations:
Constraints:
Prerequisites:
Data are already loaded in the output tables.
Table Description
TBSKT13 Total Basket Keys

Page 543 from 698
TDV_LOYALTY_VAL Table where Loyalty Card Usage Rate Computations are stored
TRIP_SHOPGROUP_VALT119 User defined shop groups with selected shop characteristic
values
Files:
Input File:
NONE
Output File:
NONE
Parameters :
Static Parameters:

Type
NUMERIC DT_VALUE_FID Mandatory DT5949 value to be checked
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No

Page 544 from 698
F# Selection No
DT Selection No
File Selection No
Rework Management:

NONE

code
job again

Page 545 from 698

date range
P0020608 Static Static Value
DT_VALUE_FID
Parameter value
P1044021 ShopGroup Resolved ShopGroup Information of the ShopGroup that has been
Resolved resolved
from TDV_LOYALTY_VAL table
TDV_LOYALTY_VAL
P9900141 # Records Inserted in Count of records inserted in
TDV_LOYALTY_VAL TDV_LOYALTY_VAL table
P9900184 No Basket Keys No Keys found for processing
selected for
processing from
TBSKT13
P1042001 NO ShopGroup found All ShopGroup definitions in
for processing.All SGs TRIP_SHOPGROUPT117 are flagged as
are flagged LOYALTY LOYALTY_CARD=N for the job submission
CARD=N cycle
P9900184 No purchases For the specificShopGroup no Keys have
selected for been selected from TBSKT13
ShopGroup

Page 546 from 698
Page 547 from 698
EHDV21165P- NEW PRODUCT ALERT
Phase:
Extract all information related to the barcode list given in the input file.
Process Step:
- Read all barcodes from the input file.
- Select all records from ALLT11 and RAWDAILYT60 , from submitted period to back periods (depending on #Periods), select the Household
and Nankeys, then join with PRODUCT_GROUPS_TRANST46 to find the description of the product.
- Store Barcodes, User Description (from the input file), Product Description and Household in the output file.
Recommendations:
Constraints:
Prerequisites:
Table Description
separated
MODULEST31 List of Modules extracted from EIMD (used there only in case of
Rework PC for Homepannel only)

Page 548 from 698
Files:
This file should be in CSV format (separated by 😉. Both Barcodes and User Description are mandatory
FILE02:

Page 549 from 698
Parameters :
Static Parameters: NIL
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection Yes
Rework Management:

NONE

Page 550 from 698

code
P0020126 Mandatory input file is not Input file S Input file is not Give input file as
available present per the layout
defined.
P9900014 Mandatory input file is Input file S Input file is Give input file
EMPTY EMPTY with barcodes.
P9900232 Input file format not Input file S Input file format is Give input file as
correct bad file genearted wrong per the layout
defined.
P9900381 External table issue Input File S Issue with input Check with
file support team

date range
P9900123 # Records read from Count of records available in the input file
Input file
P9900369 # Records exported to Count of records written in output file.
output file

Page 551 from 698
EHDV21166P- IBD RESOLUTION PERMANENT STORAGE
Phase:
PHASE 4 – SERVICE AREA
This process gives the capability of resolving all the IBDs defined for a specific cycle in one shot and store the outcome of these IBDs’ resolutions
to the permanent table TDV_IBD_RESOLUTION. The content of this table is visible through a dedicated screen in GUI.
Process Step:
• Check if any IBD_RULEST106.DT_CHANGED > IBD_DEFINITIONT104. DT_LAST_RESOLUTION

o If YES Proceed the process
If NO display a message that the resolution are already aligned and exit with warning
• Check if IBD_DEFINITIONT104.DT_LAST_RESOLUTION > IMDB_HISTORICAL_EVENT_LOG.DT_PUBLICATION

o If NO and MODE = INCR, display a message to use FULL mode only and exit with failure
• If MODE = FULL , FOR EACH IBD in the cycle, call EHPK_IBD.IBD_Resolution with IP_NC_FORCE set to ‘Y’
• If MODE = INCR, FOR EACH IBD in the cycle:
o Check if IBD_RULEST106.DT_CHANGED for this IBD > IBD_DEFINITIONT104.DT_LAST_RESOLUTION

o Check if IBD_RULEST106.DT_CHANGED for this IBD > IMDB_HISTORICAL_EVENT_LOG.DT_PUBLICATION
• IF one condidion is satisfied,

o call EHPK_IBD.IBD_Resolution with parameter IP_NC_FORCE set to ‘Y’

Page 552 from 698
• Report number of resolved / non resolved count
Recommendations:
Constraints:
Prerequisites:
After OGRDS load completion, 166P job has to be re-run again. So that the jobs using the result of IBD resolution will get the last refreshed
resolution data.
Table Description
IBD_CHARST105 Table containing the Item Break Down characteristics per cycle,
IBD
NAN_CHAR_VALUEST75 Item characteristics values extracted from EIMDB
TDV_IBD_RESOLUTION Table containing resolved NAN KEYS per cycle, IBD
IMDB_HISTORICAL_EVENT_LOGT10 List of all publications (Full and Inc) published by EIMDB
Files:
Input File:
NONE
Output File:
NONE

Page 553 from 698
Parameters :
Static Parameters:

Type
LIST MODE Mandatory INCR or FULL
Dynamic Parameters:

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:


Page 554 from 698
NONE

code
P9900447 Use FULL mode instead and Use FULL S ALL IBDS are Use FULL mode to
submit again. mode already resolved enforce Resolution
P0020505 Mandatory Static Parameter Mandatory S Mandatory Static Set the mandatory
has not been set Static Parameter has not static parameter and
Parameter been set then submit the job
NOT defined again

range
P0020608 Static MODE Parameter Static Value
value
P9900446 ALL IBD resolutions are IBD alligned
555ehavio.
P9900448 # IBDs resolved: # IBDs resolved Count of IBDs resolved based on process logic
P9900448 # IBDs NOT resolved: # IBDs NOT Count of IBDs NOT resolved based on process logic
resolved
P9900449 IBD resolved : IBD resolved IBD that has been resolved based on process logic
P9900449 IBD NOT resolved : IBD NOT IBD that has NOT been resolved based on process
resolved logic

Page 555 from 698
EHDV21167P- CA Data Loader
Phase:
US Data Loader is used to load data in a batch mode directly to the output tables: RAWDAILYT60, TBSKT13, PROCKEYST29 as well as the shop
master table SHOPT44 and the family master table FAMILYT43 from backdata input ASCII files, prepared by the legacy system. This process is
used by CA to initialise their DV environment with already validated data.
Recommendations:
Constraints:
Prerequisites:
Table Description

Page 556 from 698
Files:
Input Files:
FileName Comment

Format: Text file

with NCP Purchase
records
be 0 or 1
be 0 or 1
be 0 or 1
be 0 or 1
NCP Trip Data File:

Format: Text file

Page 557 from 698
Shopped)
NumberOfItems NUM 3 Number of Items
(Unique UPC’s)
TotalOutlet NUM 5 Total Outlet $/Total
UPC $ (Before
Coupon)
Quantity NUM 5 Quantity (After
Editing)
TripHour NUM 2 Trip Hour
Blank NUM 1 Blank
PrimShopperSex NUM 1 Primary Shopper
Sex
PrimShopperAge NUM 2 Primary Shopper
Age
SecShopperSex NUM 1 Secondary Shopper
Sex
SecShopperAge NUM 2 Secondary Shopper
Age
PanelCode NUM 1 Panel Code
TotalDollars NUM 5 Total Dollars
Spent/Ring Dollars
(Cents)
OnDealTransactions NUM 2 Transactions
(UPC’s) On-Deal
MfrCouponTotal NUM 4 Mfr. Coupon Total
(Cents)
StoreCouponTotal ALPHA 4 Store Coupon Total

Page 558 from 698
(Cents)
StoreMfrCouponTotal ALPHANUM 4 Store/Mfr. Coupon
Total (Cents)
NotUsed NUM 1 Not Used

Format: Text file

Shopped)
PrimShopperSex NUM 1 Primary Shopper Sex
PrimShopperAge NUM 2 Primary Shopper
Age
SecShopperSex NUM 1 Secondary Shopper
Sex
SecShopperAge NUM 2 Secondary Shopper
Age
Modified)
Size ALPHA 2 Size

Page 559 from 698
Multi NUM 3 Multi
OriginalQuantity NUM 2 Original Quantity
NCP File Rules

Trip Data file Field DV Field Mapping Transformation Filtering Collapsing

PanId ac_hhold Left pad with 0 up to None None
10 chars
TripDate dt_pur_date Reformat to None None
yyyymmdd
NHS_Outlet ac_shop Left pad with 0 up to None None
10 chars
TotalOutlet DT5128 (Basket Total FMCG) Divide by 100 None None
DT5895 (Total FMCG) (Dec=2)
DT5896 (Units FMCG)
TotalDollars DT5941 (Basket Total Spent) Divide by 100 None None
(Dec=2)
PrimShopperSex DT5944 None None
PrimShopperAge DT5945 None None
SecShopperSex DT5946 None None
SecShopperAge DT5947 None None
OnDealTransactions DT5756 None None
MfrCouponTotal DT5800 Divide by 100 None None

Page 560 from 698
(Dec=2)
StoreCouponTotal DT5801 Divide by 100 None None
(Dec=2)
StoreMfrCouponTotal DT5802 Divide by 100 None None
(Dec=2)

Field
to 10 chars
yyyymmdd
to 10 chars
999999
OriginalQuantity DT5642 (Original Units)

DT5667 (Price Historical DT) below
below
Total_Coupon DT5130 Summed Up

Page 561 from 698
Output Files :
FileName Comment
When job is submitted with

● LOAD_MODE=NORMAL the rejected files will contain any records from the NCP ASCII files having date values out of the period date
range.
● LOAD_MODE=APPEND the rejected files will contain any records from the NCP ASCII files having same keys already in the output tables
that exist in the NCP ASCII files.
Note : For python version Hive data refresh is done at the beginning and oracle data refresh is done at the end of this process in terminateJob
for these three tables:
• Prockeyst29
• Rawdailyt60
• Tbskt13
Parameters :
Static Parameters:

Name (Yes/No)
AC-AUDIT Yes
AC-JOB Yes
AC-TYPE Yes PARM
AC-TYPE VALUE Yes LOAD_MODE

Page 562 from 698
AC-NAME Yes Valid values are
NORMAL
APPEND
REPLACE
Default value if parameter is not defined is NORMAL
PROCESS SUBMISSION MODES
LOAD_MODE PROCESS REWORK

NORMAL Initial Partitions checks Not Allowed
Load in Append mode
REPLACE Not Allowed Initial Deletion of existing
data in Period Range
Load in Append Mode
APPEND Not Allowed Check for existing data in
database with same keys
If none found, Load in
Append Mode
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 563 from 698
Rework Management :


Page 564 from 698

code
table
DATACODET20
blank the files is
missing
missing
Lock. RELEASE Lock.
Lock
generated
correct.Bad file
generated
correct.Bad file
generated

Page 565 from 698
P9900014 Some or all input files are Empty File S Some or all input Check Input files.
empty files are empty Some of the input
files are empty
P0020722 NCP Magnet Purchase Input File S NCP Magnet

Data file does not exist/or Not Present Purchase Data file
format is not correct does not exist/or
submit job in REWK mode submission not submit job in submission rules
and option REWK mode and
LOAD_MODE=NORMAL LOAD_MODE=NOR
MAL
submit job in PROC mode submission not submit job in submission rules
and option PROC mode and
LOAD_MODE=APPEND or LOAD_MODE=APPE
REPLACE ND or REPLACE

Page 566 from 698
P9900195 Data already present in Data already S Data already present
output tables. Load IN present in in output tables.
REWK mode and output tables Load IN REWK mode
– Rwk option
LOAD_MODE=REPLACE required for and
data LOAD_MODE=REPLA
processing CE
TBSK: duplicate key already exists in
values
TBSK:
RAWDA: duplicate key already exists in
values
RAWDA:
P0010035 Oracle Error Oracle Error S Oracle Error Check for oracle
syntax

Page 567 from 698

date range
P0020608 Static Value Static Value LOAD_MODE: Information message only. No
action required
the input file
the input file
P9900123 Total Number of Record Count # Magnet Purchase Records read from input
Records Present in file:
the input file
skipped in the Magnet Purchase Data file: Information

Page 568 from 698
P1034006 Total Number of Record Count # New Hholds inserted: Information message
TBSKT13 table
deleted from RAWDA
RAWDAILYT60
deleted from PRKEYS
PROCKEYST29 table
% checked
created.
P9900432 Number of records Record Count # Trip records that already exist in TBSK and
P9900432 Number of records Record Count # Purchase records that already exist in

Page 569 from 698
P9900390 Statistics has been Statistics has Statistics has been calculated for table:
calculated for table: been Information message only. No action required
calculated for
table

Page 570 from 698
EHDV21168P- CA Data Edit
Phase:
CA Data Edit is used to load edit already existing data in a batch mode directly to the output tables: RAWDAILYT60, TBSKT13, PROCKEYST29
as well as the shop master table SHOPT44 from backdata input ASCII files, prepared by the legacy system.
• Prockeyst29
• Rawdailyt60
• Tbskt13
• TDV_XCD_SHRINK_LINKS
• Prockeyst29
• Rawdailyt60
• Tbskt13
Recommendations:
Constraints:
Prerequisites:
• CA Data Loader (EHDV21167P)

Page 571 from 698
• US xCoding Shrink (EHDV21154P)
Table Description
Files:
Input Files:
FileName Comment

Format: Text file

Page 572 from 698
with NCP Purchase
records
be 0 or 1
be 0 or 1
be 0 or 1
be 0 or 1
NCP Trip Data File:

Format: Text file

TripDate_After NUM 6 Trip Date (Date
Shopped)
NumberOfItems_After NUM 3 Number of Items
(Unique UPC’s)
TotalOutlet_After NUM 5 Total Outlet $/Total
UPC $ (Before
Coupon)
Quantity_After NUM 5 Quantity (After Editing)
TripHour_After NUM 2 Trip Hour
Blank_After NUM 1 Blank
PrimShopperSex_After NUM 1 Primary Shopper Sex
PrimShopperAge_After NUM 2 Primary Shopper Age
SecShopperSex_After NUM 1 Secondary Shopper

Page 573 from 698
Sex
SecShopperAge_After NUM 2 Secondary Shopper
Age
PanelCode_After NUM 1 Panel Code
TotalDollars_After NUM 5 Total Dollars
Spent/Ring Dollars
(Cents)
OnDealTransactions_After NUM 2 Transactions (UPC’s)
On-Deal
MfrCouponTotal_After NUM 4 Mfr. Coupon Total
(Cents)
StoreCouponTotal_After ALPHA 4 Store Coupon Total
(Cents)
StoreMfrCouponTotal_After ALPHANUM 4 Store/Mfr. Coupon
Total (Cents)
NotUsed ALPHA 1 Not Used
PanId_before NUM 8 Panelist Identification
Period_before NUM 4 Process Period
TripDate_before NUM 6 Trip Date (Date
Shopped)
NHS_Outlet_before NUM 4 NHS Outlet
MarketCode_before NUM 3 Major Market Code
ACNStore_before NUM 5 CAN Primary Store
NumberOfItems_before NUM 3 Number of Items
(Unique UPC’s)
TotalOutlet_before NUM 5 Total Outlet $/Total
UPC $ (Before
Coupon)
Quantity_before NUM 5 Quantity (After Editing)
TripHour_before NUM 2 Trip Hour
Blank_before NUM 1 Blank
PrimShopperSex_before NUM 1 Primary Shopper Sex
PrimShopperAge_before NUM 2 Primary Shopper Age
SecShopperSex_before NUM 1 Secondary Shopper
Sex

Page 574 from 698
SecShopperAge_before NUM 1 Secondary Shopper
Age
StoreType_before NUM 1 Store Type
PanelCode_before NUM 1 Panel Code
TotalDollars_before NUM 5 Total Dollars
Spent/Ring Dollars
(Cents)
OnDealTransactions_before NUM 2 Transactions (UPC’s)
On-Deal
MfrCouponTotal_before NUM 4 Mfr. Coupon Total
(Cents)
StoreCouponTotal_before ALPHA 4 Store Coupon Total
(Cents)
StoreMfrCouponTotal_before ALPHANUM 4 Store/Mfr. Coupon
Total (Cents)
NotUsed ALPHA 1 Not Used

Format: Text file
Field Type Length Description

TripDate_After NUM 6 Trip Date (Date Shopped)
Module_After NUM 4 Product Module
PrimShopperSex_After NUM 1 Primary Shopper Sex
PrimShopperAge_After NUM 2 Primary Shopper Age
SecShopperSex_After NUM 1 Secondary Shopper Sex
SecShopperAge_After NUM 2 Secondary Shopper Age
UPC_After NUM 12 UPC Number
Quantity_After NUM 2 Quantity (Possibly Modified)
Total_Price_After NUM 5 Total Unit Price
DealCode_After ALPHANUM 2 Deal Code

Page 575 from 698
Total_Coupon_After NUM 4 Total Coupon Value
CausalAd_After NUM 1 Causal Ad
CausalDisplayInfo_After NUM 2 Causal Display Info
PriceIndicator_After NUM 1 Price Indicator
BrandCode_After NUM 6 Brand Code
Size_After ALPHA 2 Size
SizeDescription_After ALPHANUM 6 Size Description
SizeNumber_After NUM 7 Size Number
Multi_After NUM 3 Multi
Flavor_After NUM 6 Flavor Code
OrgCode_After ALPHA 3 Org. Code
OriginalQuantity_After NUM 2 Original Quantity
Panel Code_After ALPHA 1 Panel Code
PanId_Before NUM 8 Panelist Identification
Period_Before NUM 4 Process Period
TripDate_Before NUM 6 Trip Date (Date Shopped)
Module_Before NUM 4 Product Module
PrimShopper_Before NUM 3 Primary Shopper
SecShopper_Before NUM 3 Secondary Shopper
ACNStore_Before NUM 5 CAN Primary Store
UPC_Before NUM 12 UPC Number
Quantity_Before NUM 2 Quantity (Possibly Modified)
Total_Price_Before NUM 5 Total Unit Price
DealCode_Before ALPHANUM 2 Deal Code
Total_Coupon_Before NUM 4 Total Coupon Value
CausalAd_Before NUM 1 Causal Ad
CausalDisplayInfo_Before NUM 2 Causal Display Info
PriceIndicator_Before NUM 1 Price Indicator
BrandCode_Before NUM 6 Brand Code
Size_Before ALPHA 2 Size
SizeDescription_Before ALPHANUM 6 Size Description

Page 576 from 698
SizeNumber_Before NUM 7 Size Number
Multi_Before NUM 3 Multi
Flavor_Before NUM 6 Flavor Code
StoreType_Before NUM 1 Store Type
OrgCode_Before ALPHA 3 Org. Code
OriginalQuantity_Before NUM 2 Original Quantity
MarketCode_Before NUM 3 Major Market Code
Panel Code_Before ALPHA 1 Panel Code
CpnDealValue_Before NUM 4 MFG Cpn Deal Value

Page 577 from 698
NCP File Rules
Trip Data file Field DV Field Mapping Transformation Filtering Collapsing

PanId ac_hhold Left pad with 0 up None None
to 10 chars
TripDate dt_pur_date Reformat to None None
yyyymmdd
NHS_Outlet ac_shop Left pad with 0 up None None
to 10 chars
TotalOutlet DT5128 (Basket Total FMCG) Divide by 100 None None
DT5895 (Total FMCG) (Dec=2)
DT5896 (Units FMCG)
TotalDollars DT5941 (Basket Total Spent) Divide by 100 None None
(Dec=2)
PrimShopperSex DT5944 None None
PrimShopperAge DT5945 None None
SecShopperSex DT5946 None None
SecShopperAge DT5947 None None
OnDealTransactions DT5756 None None
MfrCouponTotal DT5800 Divide by 100 None None
(Dec=2)
StoreCouponTotal DT5801 Divide by 100 None None
(Dec=2)
StoreMfrCouponTotal DT5802 Divide by 100 None None
(Dec=2)

Page 578 from 698

Field
to 10 chars
yyyymmdd
to 10 chars
999999
OriginalQuantity DT5642 (Original Units)

DT5667 (Price Historical DT) below
below
Total_Coupon DT5130 Summed Up

Page 579 from 698
Output Files :
FileName Comment
• If in ASCII files there are records with KEYS (using the BEFORE fields) that have NO corresponding records in the Output tables, the
rejected files will contain these missing KEY records from the NCP ASCII files.
• Otherwise the rejected files will contain any records from the NCP ASCII files having date values out of the period date range.
Parameters :
Static Parameters:
NONE
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 580 from 698
Rework Management :


code
table
DATACODET20

Page 581 from 698
blank the files is
missing
missing
Lock. RELEASE Lock.
Lock
generated
correct.Bad file
generated
correct.Bad file
generated
the input files
are empty

Page 582 from 698

oracle syntax
in RAWDA BEFORE
fields) in
Purchase
input file has
no
correspondin
g keys in
RAWDA
in TBSK BEFORE
fields) in Trip
input file has
no
correspondin
g keys in
TBSK

date range

Page 583 from 698
the input file
the input file
the input file
TBSKT13 table
deleted from RAWDA
RAWDAILYT60

Page 584 from 698
deleted from PRKEYS
PROCKEYST29 table
P9900432 Number of records Record Count # Trip records with missing KEYS in TBSK and
P9900432 Number of records Record Count # Purchase records with missing KEYS in

Page 585 from 698
EHDV21169P- UNCODED TRACKING
Phase:
As part of ZTV Project, there is a need to watch the rate of uncoded items by shop by period and for along period.
A new job “Uncoded Tracking” will handle the same process as manual one in an automated way.
The module will take care of:

- Compute the rate of uncoded items by shop group
- Store in a new table
- Report the number of items computed
Process Step:
Process is going through the following steps:
Initialization Step:
Process Steps:
• From TRIP_SHOPGROUPT117 when AC_UNCODED_RATE is set to ‘Y’, get all candidate shop groups
• Resolve shop resoluon to get all shop lists
• Get dates ranges based on submied period. #numper is not taken into account

Page 586 from 698
• By shop group (for that submied period).
o Count all uncoded items from ALLT11 or RAWDAILYT60. Uncoded items are NANKEY=0 (and Barcode <> ‘0’ for ALLT11). It
depends also from the PGI_UNCODED. When PGI_UNCODED is YES, all PGI is counted as uncoded so all NANKEYS with
FL_PGI=Y at module level are counted as uncoded (uncoded items are specified in the below array).
o Count all items in ALLT11 and count all items in RAWDAILYT60
o The percentage computed (see the descripon below) is inserted into the TDV_UNCODED_RATE , with Shop Descripon and
period.
UNCODED
ALLT11 RAWDAILYT60
PGI_UNCODED=YES FL_PGI=Y OR NAN=0 FL_PGI=Y OR NAN=0
PGI_UNCODED=NO NAN=0 NAN=0
In all situaons described above the filter Barcode <> ‘0’ should be added when extracng from ALLT11
Final Step:
● Report the number of Shop Groups for which the rate is computed
● Report job status
○ Exit job with Warning if no uncoded items found for any shop group
○ Exit job with Success if no issue found
Note : For python version Hive data refresh is done at the beginning of this process for these two tables:
• Rawdailyt60
• Allt11.
Recommendations:
Constraints:
Prerequisites:
Crosscoding must have been finished

Page 587 from 698
Table Description
separated in datatypes and loaded in the table ALL as separate
records
TDV_UNCODED_RATE Table where Uncoded Rate Computations are stored
NAN_ITEMST73 List of NAN_KEY loaded from EIMD
MODULEST31 List of Modules extracted from EIMD
Files:
Input File:
NONE
Output File:
NONE
Parameters :
Static Parameters:

Type
LIST PGI_UNCODED Optional ‘YES’ or ‘NO’. ‘YES’ means PGI Nankeys are
considered as uncoded. By default ‘NO’ is
considered

Page 588 from 698
Dynamic Parameters:

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

NONE

code
P9900246 Submission for period Submission S Job cannot be Submit a valid
for period 0 submitted with period
is not Period 0
allowed


Page 589 from 698
date range
P0020608 Static Static Value
PGI_UNCODED
Parameter value
P1690001 # Shop groups # Shop Count of records shop groups inserted in
inserted groups TDV_UNCODED_RATE
inserted
P1690002 No uncoded items # Shop No uncoded items found for any shop group
found for any shop groups
group inserted

Page 590 from 698
EHDV21170P- Price Management for ReXCoding and Recovering
Phase:
In the context of ReXcoding and Recovering, candidate items are selected and for those items, price
need to be revalidated in the context of FPV or in APV. So, for this purpose we need to find a standard
price. Different methods will be implemented to compute the prices of these items
The solution is to create a new job to implement the new method of computation of the price.
✓ A new job Price for ReXcoding and Recovering (EHDV21170P) is created with a new set of static parameters
✓ Select candidate nankeys from KEYST14 where file number is -1 and for selected timestamp
o if for nankeys the price is present in CNANPRICET34 for the submitted period and previous period, nothing to do
o else if the price (datatype DT5675) present for the nankeys, computation of median price should happen based on the price group
o else if the price is missing then compute median price from T99 or T100
o else price is got from TDV_NAN_PRICE_ACTIVITY
o All computed price should be stored in the CNANPRICET34 to the correct period (in the period P and in the period P-1)
✓ Report number of nankeys for which the computation is happened
Process steps:
− Check parameters consistency (static and dynamic)
Process Step:
1. Get list of shops, charval and charid from SHOPUSABILITYT40, for the charid equals to PRGRP, CHAN, GEO values and store into a
temporary table GT_SHOP_CHARVALS.
2. Get all purchases with shop, nankeys and price from KEYST14 and P359_1T12.
2.1. Shop, nankeys are from KEYST14
2.2. Price from P359_1T12
2.3. Filter to apply 1 : filenumber = -1, selected timestamp, audit and date range selected. When evaluating Date range take into
account NUM_PER.
2.4. Filter to apply 2 : Nankeys should not have price in CNANPRICET34 for the submitted period P and previous period P-1.

Page 591 from 698
2.5. Joining with GT_SHOP_CHARVALS and using pivot clause, Store shop, nankeys, price and charvals of PRGRP, CHAN,
GEO (all charvals for PRGRP is stored in the column AC_PRICE_GROUP_VAL, all charvals for CHAN is stored in the column
AC_CHAIN_VAL, all charvals for GEO is stored in the column AC_GEO_VAL ,used only when GEO and CHAN are defined) into a temporary
table GT_REX_RECOV_NAN.
3. Using GT_REX_RECOV_NAN, Compute median price by nankey and by charval of price group
PRGRP (column AC_PRICE_GROUP_VAL), having more than MINOBS observations and store into GT_REX_RECOV_PRICE
4. If PRGRP0= Y, Using GT_REX_RECOV_NAN, Compute median price by nankey, having more than MINOBS observations and store
into GT_REX_RECOV_PRICE .
5. For the nankeys where the price computation is not happened in GT_REX_RECOV_NAN, compute the median price from
SHOP_PRICEST99 using nankey and SOURCE. Select price on most recent dates.This check happens only when SOURCE is defined
6. For the nankeys where the price computation is not happened in GT_REX_RECOV_NAN, compute the median price from
CLUSTER_PRICEST100 using nankey, AC_CHAIN_VAL and AC_GEO_VAL and SOURCE. Select price on most recent 10 periods.
This check happens only when SOURCE ,GEO and CHAN are defined.If CLUSTER_PRICEST100 is empty warning is thrown and this step is
skipped
7. If there are any remaining nankeys for which price is not assigned in GT_REX_RECOV_NAN, get price from
TDV_NAN_PRICE_ACTIVITY using NPA_NC_NANKEY, for the country used.
8. Using GT_REX_RECOV_PRICE, assign the price into CNANPRICET34 to the corresponding NAN_KEY for the period P-1 and P with
price group value
Final Step:
9. Report number nankeys assigned with a price in each step
Recommendations:
Constraints:
Prerequisites:
Data in validaon data is available with ReXcoding and Recovering items (using Data Selecon by Barcode or PGI Data Selecon)
Run mode : PROC or REWORK

Input Files : None
Output File: None

Page 592 from 698
Static Parameters:

Type
CHAR PRGRP Mandatory Shop Characteristic to be used for Price
Group
CHAR CHAN Optional Identify the characteristic ID for the CHAIN
CHAR GEO Optional Identify the characteristic ID for the
GEOGRAPHY
NUMBER MINOBS Optional Minimum number of purchase act by
NAN_KEY/ Price Group, by default 1
LIST PRGRP0 Optional Option to compute standard price at Product
Group & Price Group=0. ‘Y’ OR ‘N’.
NUMBER NUM_PER Optional Number of periods backward used in
validation area to compute the price, by
default 1
CHAR SOURCE Optional Source used for SHOP_PRICEST99 and
CLUSTER_PRICEST100
Dynamic Parameters:

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp Yes
PC Selection No
F# Selection No
DT Selection No

Page 593 from 698
File Selection No

code
P9900246 Submission for period Submission S Job cannot be Submit a valid period
for period 0 submitted with
is not Period 0
allowed
P0020593 PRGRP is not defined Static S PRGRP static Kindly define PRGRP
parameter parameter is not
missing defined
P1700008 Cluster price table is Cluster price W Cluster price table User can check the static
empty table is is empty while parameter setup fo the
empty GEO is defined job run

Page 594 from 698

P0000000 Submission period Date Range Get Date range for submission
date range period
P1700001 Computation count Number of Items computed for PRGRP #Items computed for PRGRP
having More than MINOBS having More than MINOBS
P1700002 Computation count Number of Items computed when #Items computed when PRGRP0=
PRGRP0= Y Y
P1700003 Computation count Number of Items computed from #Items computed from
SHOP_PRICEST99 SHOP_PRICEST99
CLUSTER_PRICEST100 CLUSTER_PRICEST100
TDV_NAN_PRICE_ACTIVITY TDV_NAN_PRICE_ACTIVITY
P1700006 Number of Final Data Nankeys assigned with a price in final #Nankeys assigned with a price in
step final step

Page 595 from 698
EHDV21171P- New Standard Price and MAD Computation
Phase:
To solve some price validation issues, in particular with Code Books showing more heterogeneity, DsCI has defined a new methodology to
compute Standard Price and MAD.
Major differences with current method:
- Ignore outlier values in computation process
- Consider validated values for the benchmark / historical period
- Apply clustering method when prices are not homogenous for same level
Process steps:
- Check Parameters (static and dynamic)
- Compute P1 and if activated P2 / P3 date ranges from Periods parameter
- Resolve IBD when activated
Process Step:
- Get data for process
o Using IBD when activated
o Using largest period date range
o Link with Price Group thru shop characteristics
o Link with Product Group thru nankeys
o With historical price (DT5667) for submission period
o With validated price (DT6575) for past periods
o Without records having DT5904 = 3 or DT5904 = 11 or DT5904 = 20 when FPV_3_4_EXCLUDE=YES
- Collect level data for Period 1
o Level 1 – Nankey/Price Group using NOBS_P1_L1
o Level 2 – Nankey using NOBS_P1_L2
o Level 3 – Product Group/Price Group using NOBS_P1_L3
o Level 4 – Product Group using NOBS_P1_L4
- If P2 requested Collect level data for Period 2
o Level 1 – Nankey/Price Group using NOBS_P2_L1 and not yet collected for Period 1

Page 596 from 698
o Level 2 – Nankey using NOBS_P2_L2 and not yet collected for Period 1
o Level 3 – Product Group/Price Group using NOBS_P2_L3 and not yet collected for Period 1
o Level 4 – Product Group using NOBS_P2_L4 and not yet collected for Period 1

o Level 1 – Nankey/Price Group using NOBS_P3_L1 and not yet collected for Period 1 nor Period 2
o Level 2 – Nankey using NOBS_P3_L2 and not yet collected for Period 1 nor Period 2
o Level 3 – Product Group/Price Group using NOBS_P3_L3 and not yet collected for Period 1 nor Period 2
o Level 4 – Product Group using NOBS_P3_L4 and not yet collected for Period 1 nor Period 2
- Compute percentile 25 and 75, lower bound and upper bound by nankey, product group, price group
- Detect and remove outliers from collected data
- Determine cluster count by nankey, product group, price group
- Copy data with cluster count = 1 to final table
- Process data with cluster count = 2
o Apply clustering method by nankey, product group , price group
o Compute cluster score by nankey, product group , price group
o Copy data from cluster with minimal score to final table
- Compute median, standard price and MAD by nankey, product group, price group from final
table
o Median price = median (nc_price)
o MAD = median ( abs(nc_price – median price))
o Std price = w*median price + (1- w )*most recent std price from CNANPRICET34
Where w is computed as n/(n+k) with n= number of observations for the level
being processed
0 When most recent std price is empty default value 1 is assigned to w
- Load standard price, mad into CNANPRICET34

- When Price/MAD already there in CNANPRICET34 table then replace value
- When not present insert a new record in CNANPRICET34 for Price/MAD
No deletion to apply at all whatever is mode and scope
Final Step:
- Report count of records selected for each level and period
- Report count of standard prices computed for each level and period
Recommendations:

Page 597 from 698
Constraints:
Prerequisites:
Availability of the data in rawda for the submission period
Price Group information available in Shop Usability table

Input Files : None
Output File: None
Static Parameters:

Type
NUMBER P1 Mandatory Period length1 to consider at first step
NUMBER NOBS_P1_L1 Mandatory Nb Obs minimum for Period 1 and Level 1(Nankey / Price Group)
NUMBER NOBS_P1_L2 Mandatory Nb Obs minimum for Period 1 and Level 2(Nankey)
NUMBER NOBS_P1_L3 Mandatory Nb Obs minimum for Period 1 and Level 3(Product Group / Price Group)
NUMBER NOBS_P1_L4 Mandatory Nb Obs minimum for Period 1 and Level 4(Product Group)
NUMBER P2 Optional Period length2 to consider at second step
NUMBER NOBS_P2_L1 Optional Nb Obs minimum for Period 2 and Level 1(Nankey / Price Group)
NUMBER NOBS_P2_L2 Optional Nb Obs minimum for Period 2 and Level 2(Nankey)
NUMBER NOBS_P2_L3 Optional Nb Obs minimum for Period 2 and Level 3(Product Group / Price Group)
NUMBER NOBS_P2_L4 Optional Nb Obs minimum for Period 2 and Level 4(Product Group)
NUMBER P3 Optional Period length3 to consider at Third step
NUMBER NOBS_P3_L1 Optional Nb Obs minimum for Period 3 and Level 1(Nankey / Price Group)
NUMBER NOBS_P3_L2 Optional Nb Obs minimum for Period 3 and Level 2(Nankey)
NUMBER NOBS_P3_L3 Optional Nb Obs minimum for Period 3 and Level 3(Product Group / Price Group)
NUMBER NOBS_P3_L4 Optional Nb Obs minimum for Period 3 and Level 4(Product Group)
NUMBER PRICE_GRP Mandatory Char ID for Price Group determination
NUMBER K Mandatory K factor to use in standard price formula
NUMBER SCOPE Optional IBD number when need to reduce scope
LIST FPV_3_4_EXCLUDE Optional Can be Yes or NO ,it excludes records having DT5904 = 3 or DT5904 = 11
when selected YES

Page 598 from 698
SHOP_SC NUMBER Optional Shop Group ID when need to reduce Shop Scope
OPE
MIN_ITEM NUMBER Mandatory Minimum number of items for MAD computation,default value is zero
• The MIN_ITEM is used to compute MAD and DT5665 in level 3 and level 4.
• At Level 1 and 2 MAD will be unchanged as median ( abs(nc_price – median price)).
• At Level 3 and 4
• MAD = avg( (abs(nc_price – item median price)/ item median).
• Number of items is the count of distinct nc_item to be stored in DT5665 .
• For calculating the above values ‘number of items’ must be >= MIN_ITEM parameter value.
2 Represents the number of periods back, with a value of 4, 5 periods would be inspected when adding the submitted period.
Dynamic Parameters:

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

Page 599 from 698

code
P9900246 Submission for period Submission S Job cannot be Submit a valid
for period 0 submitted with period
is not Period 0
allowed

P0000000 Submission period Date Range Get Date range for submission
date range period
P1024026 T60 Bskt Process Cnt Number of RAWDAILYT60 baskets to T60 records processed
be processed
P9900209 Number of Outliers Number of Price Outliers detected Outliers detection
P9900450 Cluster Stats Cluster Statistics have been calculated Clusters calculation
P9900451 Number of Final Data Number of Final Data selected Final data processed

Page 600 from 698
EHDV21172P- Converted Unit Outlier Threshold
Phase:
The new algorithm (CUOE) inspects the household purchase units using multiple conversion
factors to assess more accurately the total converted volume. Final units will then be edited, if
needed. It takes different unit conversion “rates” created in OGRDS available in DV to define
consecutive thresholds by module and adjustment ratios by purchase act.
Input to the Process
The CUOE is applied at module level. Scope of items to be considered remains the same as
HBE approach and defined using the concept of IBD. The definitions of the different algorith
inputs are as follow:
1. Inspection week as the week on which quantity will be inspected from rawdata and
adjusted if needed
2. Benchmarking weeks as the number of weeks (13 by default) used to fit the
distribution 601ehaviour of HHs (12 from history + inspection week) from rawdata –
units to benchmark are a mix of input units and final (corrected) units, when available
in history (corrected by this algorithm (after a transition period, only final units are used
to benchmark)
3. CPS rawdata (records level) for inspection and bench-marking containing
● Date of purchasing
● HH code
● Shop
● Bar-code / Nankey
● Original quantity (HH entry)
● [Original quantity before Multi-pack validation module]
● Final quantity after Multi-pack validation module
● [Original quantity before HBE validation module]
● Final quantity after HBE validation module
● Final quantity
4. Household characteristics (for CUP calculation)

● Number of adults
● Number of children
5. Product characteristics
● Module / Product group
● Variable weight item (to be excluded)
● IBD rules regarding CUP application for standardization purpose
6. Unit conversion information
● Extracted from system:
● NC_NAN_KEY
Page 601 from 698
● exactly 3 unit converting variables – no repetition allowed – from the
following:
● NO_PACKS_IN_MULTIPACK [MANDATORY – 2nd column]:
corresponding to the conversion in Number of packs in multipack per
NANKEY
● TOTAL_VOLUME [MANDATORY – 3rd or 4th column]: corresponding
to the conversion in VOLUME or MASS/WEIGHT per NANKEY
● TOTAL_ITEMS_IN_PACKAGE [OPTIONAL]: corresponding to the
conversion in Number of items per Pack per NANKEY
● NO_SERVINGS_DOSES [OPTIONAL]: corresponding to the
conversion in Number of servings or doses per NANKEY
Consumption unit computation
• Link Raw data with product characteristics and computed CUP and IBD rules
• The Outlier detection algorithm consists of multiple runs on converted units, using the
product characteristics described above, by that order.
On first run (Multipack conversion units)
• replace NA and 0 by 1 to initiate the algorithm
• Define algorithm_input_units = benchmark unit.
NOTE: in particular, for the inspection week, benchmark unit = input unit.
• Steps to take on each run:

o Define weighted unit as: Weighted_UNITS = algorithm_input_units * conversion
unit.
o Normalising Multipack weighted unit per CUP according to defined IBD rules:
Nor_Weighted_UNITS
o Compute Criticalvalue = log(2*Cutoff_level)/log(5)*(-1)
o Compute 70th and 90th percentiles and threshold (by module) of normalised
converted units distribution:
2. PCTL70 = 70th percentile of Nor_Weighted_UNITS

24. PCTL90 = 90th percentile of Nor_Weighted_UNITS
2. THRESHOLD = PCTL70 + Criticalvalue * (PCTL90 – PCTL70)
• Rawdailyt60
• Familyusabilityt41
Run mode: PROC
Input File: None
Output File: None
Prerequisites:
• Availability of the data in RAWDAILYT60 for the submission period.
• Household information available in FAMILYUSABILITYT41 table.
• OGRDS characteristics correctly filled for the following weight codes:

Page 602 from 698
NC_WEIGHT_CODE AC_LANGUAGE_MNEM AC_DESCR
1 CL MASS – WEIGHT GROSS
2 CL NO. OF PACKS IN MULTIPACK
4 CL TOTAL VOLUME
7 CL TOTAL ITEMS IN PACKAGE
8 CL NUMBER OF SERVINGS/DOSES
Weights should be selected by users according to the following table:
Group Weight Unit Description Mandatory Run Conversion

NO. OF PACKS IN
1 2 MULTIPACK Y 1 -
1 MASS – WEIGHT GROSS KG
2 Y 2 or 3
4 TOTAL VOLUME LIT
TOTAL ITEMS IN
3 7 PACKAGE N 2 or 3 or 0 -
NUMBER OF SERVNG
4 8 DOSES N 2 or 3 or 0 -
o 3 groups need to be selected

o Groups 1 and 2 are mandatory , 3rd can be chosen between groups 3 and 4
o Sequence for group 1 is fixed to 1 , others can be set according to country needs
Alternate char can be defined for one or more group
Table Description
TDV_CUOE_SETUP Setup table for CUOE process main elements
TDV_CUOE_THRESHOLD Table that stores Threshold Values by Module
IBD
NAN_CHAR_VALUEST75 Item characteristics values id extracted from OGRDS
NAN_WEIGHT_VALUEST79 Item weight value id extracted from OGRDS
FAMILYUSABILITYT41 Table contains the characteristic identifier and the characteristic
value by Family by cycle and period (YYYYPPP)
Static Parameters:

Page 603 from 698
SCOPE NUMBER Mandatory IBD for product scope to process
CUT_OFF NUMBER Mandatory Cut off level to apply
This is the proportion of cutoff expected based on
exponential distribution hypothesis (ex: 0.05%)
CUP_ZERO LIST Mandatory Method for handling of CUP missing or zero value.
Same value and 604ehaviour as HBE using

ALLOWED FORBIDDEN or MISSING
ADULT_FACTOR NUMBER Optional Adult Factor Value if needed to overwrite setup
CHILD_FACTOR NUMBER Optional Child Factor Value if needed to overwrite setup
Dynamic Parameters:

Name (Yes/No)
# Periods Yes Used to define the Benchmark period which would not include submitted
period itself
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

code
P9900246 Submission for period S Job cannot be submitted Submit a valid period
with Period 0
P1021004 No Setup found in S No setup found in Check
TDV_CUOE_SETUP TDV_CUOE_SETUP for TDV_CUOE_SETUP for
table for submitted cycle submitted cycle submitted cycle
P1027011 Missing Family Char S Characteristic is not Check
present in FAMILYUSABILITYT41
FAMILYUSABILITYT41 for the specific Char
P1021004 No Purchase acts have S No records found in Check RAWDAILYT60 for
been selected for RAWDAILYT60 for inspection date range
Threshold Computation inspection date range
P9900428 Families having Cup S Families having Cup Check
value 0 is not allowed value 0 are present FAMILYUSABILITYT41
Message code Description

P1040011 # Purchase acts selected for Threshold Computation
P1720001 # Number of records inserted in TDV_CUOE_THRESHOLD
P1720002 # Number of records inserted in TDV_CUOE_THRESHOLD for Run
Page 604 from 698
Page 605 from 698
EHDV21173P- Converted Unit Outlier Edit
Phase:
The process follows the same steps with172p.

Having calculated the threshold value per Module then:
• An adjustment ratio is calculated when unit values exceed the THRESHOLD (otherwise,
use value 1) and apply it to the input unit
o ADJ_Ratio = min(THRESHOLD / Nor_Weighted_UNITS, 1)
o Output Unit = algorithm_input_units * ADJ_Ratio
• Algorithm_input_units = Output Unit (for next run)
• After the last (third) run, FINAL_UNITS are calculated using the following rules:
a. PROPOSED_UNITS = max(round(Output Unit, 0), 1)
b. Five (user) parameters need to be set:
• inputLower: minimum input quantity allowing correction adjustment
• pckInPackLower: minimum packs in multipack allowing correction
adjustment
• gapLower: minimum gap between PROPOSED_UNITS and original
• input quantities allowing correction adjustment
• inputLowerPGI: minimum input quantity allowing correction adjustment
• (for PGI items)
• gapLowerPGI: minimum gap between PROPOSED_UNITS and original
input quantities allowing correction adjustment (for PGI items)
2. Calculate:
• gapCorr = abs(Input Units – PROPOSED_UNITS)
• In case of PGI:
If ( (inputUnits < inputLowerPGI OR gapCorr < gapLowerPGI )

then FINAL_UNITS = inputUnits (no correction applied)
• In case of NON-PGI:
If
((inputUnits < inputLower
AND max(NO_PACKS_IN_MULTIPACK, 1) <
pckInPackLower))
OR gapCorr < gapLower)
then
FINAL_UNITS = inputUnits (no correction applied)
Else
FINAL_UNITS = PROPOSED_UNITS (apply correction) on
DT5650 in RAWDA
• Log DT5650 changes in FAMILY_LOGT67 table
✓ In rework mode, there would be the possibility to re-use computed thresholds on a limited
scope of items (in case Recovering / reXcoding) or to completely process again the period using
a RERUN mode parameter (ON / OFF)

Page 606 from 698
Input File : None
Output File: None
Prerequisites:
• Availability of the data in RAWDAILYT60 for the submission period.
• Household information available in FAMILYUSABILITYT41 table.
• OGRDS characteristics correctly filled for the following weight codes:
Table Description
TDV_CUOE_SETUP Setup table for CUOE process main elements
TDV_CUOE_THRESHOLD Table that stores Threshold Values by Module
IBD
NAN_CHAR_VALUEST75 Item characteristics values id extracted from OGRDS
NAN_WEIGHT_VALUEST79 Item weight value id extracted from OGRDS
FAMILYUSABILITYT41 Table contains the characteristic identifier and the characteristic
value by Family by cycle and period (YYYYPPP)
Static Parameters:

SCOPE NUMBER Mandatory IBD for product scope to process
SCOPE can be defined differently for corrections than

threshold computation
RERUN LIST Mandatory Rerun mode ON or OFF
Can be set ON to apply on recovered /

reXcoded items – would then apply on
records with FileNumber = -1 only
RERUN=ON can be used only in REWORK mode,
SCOPE becomes optional in that case
MIN_PACK NUMBER Mandatory Minimum Packs in Pack value for correction

MIN_INPUT NUMBER Mandatory Minimum Input value for correction
MIN_GAP NUMBER Mandatory Minimum Gap value for correction
MIN_INPUT_PGI NUMBER Mandatory Minimum Input value for correction if item is
MIN_GAP_PGI NUMBER Mandatory Minimum Gap value for correction if item is PGI
Page 607 from 698
CUP_ZERO LIST Mandatory Method for handling of CUP missing or zero value.
Same value and 608ehaviour as HBE using

ALLOWED FORBIDDEN or MISSING
ADULT_FACTOR NUMBER Optional Adult Factor Value if needed to overwrite setup
CHILD_FACTOR NUMBER Optional Child Factor Value if needed to overwrite setup
Dynamic Parameters:

Name (Yes/No)
# Periods Yes This is ignored in Normal mode, used only in Rework mode
Single Run Job No

Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

code
with Period 0
P1021004 No Setup found in S No setup found in Check
TDV_CUOE_SETUP TDV_CUOE_SETUP for TDV_CUOE_SETUP for
table for submitted cycle submitted cycle submitted cycle
P1027011 Missing Family Char S Characteristic is not Check
present in FAMILYUSABILITYT41
FAMILYUSABILITYT41 for the specific Char
P1021004 No Purchase acts have S No records found in Check RAWDAILYT60 for
been selected for RAWDAILYT60 for inspection date range
Threshold Computation inspection date range
P9900428 Families having Cup S Families having Cup Check
value 0 is not allowed value 0 are present FAMILYUSABILITYT41
P9900437 Invalid Job submission S RERUN=ON can only Check the submission
option with RERUN be submitted in mode and the RERUN
parameter REWORK mode not in value
PROC mode

Page 608 from 698

P1040011 # Purchase acts selected for Threshold Computation
P9900141 Total Number of records inserted in TDV_CUOE_THRESHOLD
P1730001 Total Number of records Updated in RAWDAILYT60
P1730002 # Records Updated in RAWDAILYT60 having MODULE
P0180002 # Records Reset in RAWDAILYT60
P1030011 # Records Deleted from FAMILY_LOGT67
P1380001 # Records inserted in FAMILY_LOGT67

Page 609 from 698
EHDV21174P- Load Consolidated Items Weight
Phase:
This job loads the Average Weight and the Price Point by period/shop/nankey into the
table TDV_ITEM_WEIGHT.
Data is loaded per source as this is defined by the SOURCE static parameters.
In order to avoid the size of the table to become very large, the user can use the static parameter
NBPER_TO_RETAIN which defines the number of periods to retain in the table.
Process Flow:
1. The maximum locked period is retrieved from CYCLEPERIODT03 for the submission
cycle.
2. Having retrieved the MAX PERIOD (point 1 above) and based on the Number of Periods
to retain (NBPER_TO_RETAIN) the first period to retain is computed (p_from retrieved)
3. Delete from TDV_ITEM_WEIGHT all records for submission cycle and Source with
datefrom < p_from
4. Read the input file
5. In NORMAL mode, first check if the period in the file is already in the
TDV_ITEM_WEIGHT if not insert all items from the file into TDV_ITEM_WEIGHT.
6. In REWORK mode, delete data from the TDV_ITEM_WEIGHT for the periods found in
the input file and for the selected SOURCE and insert items from the input file into the
TDV_ITEM_WEIGHT
Run mode: PROC or REWORK
Input File:
CSV (comma ‘,’ separated file and ‘.’ Point used for decimal)
Position Name
1 Period
2 ShopID
3 Nankey
4 Price point
5 Average Weight
Example of the input file:

Page 610 from 698
Output File: None
Prerequisites: None
Table Description
TDV_ITEM_WEIGHT Average Weight by Item
Static Parameters:

SOURCE NUMBER Mandatory To identify the Source
NBPER_TO_RETAIN NUMBER Mandatory Number of periods to retain in the
TDV_ITEM_WEIGHT table ,it can be
defined value greater than zero
Dynamic Parameters:

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

code
with Period 0
P0020505 Missing Static Parameter S Missing Static Check if Static Parameter
Parameter is defined

Page 611 from 698
P0020125 Input File is blank S Check if Input File exists Check if Input File exists
in the directory or the in the directory or the File
File Name is correct Name is correct
P0020126 Input File Not Found S Check if Input File exists Check if Input File exists
in the directory or the in the directory or the File
File Name is correct Name is correct
P9900096 More than one period is S More than one period is Check the Input File for
present in Input file OR present in Input file OR the existence of different
Period in Input file is Period in Input file is periods
different from the job different from the job
submitted Period submitted Period
P9900232 Input file format not S Input file format not Check the format of the
correct. Bad file correct. Bad file Input File
generated generated
P9900014 Consolidated Item S Consolidated Item Check the format of the
Weights File is empty Weights File is empty Input File
P9900381 External table creation S External table creation Check the format of the
issue issue Input File
P1038014 Period / Source already S Period / Source already Submit the job in
present in present in REWORK mode
TDV_ITEM_WEIGHT. TDV_ITEM_WEIGHT.
Rework Option Required Rework Option
Required

P9900123 # Records read from input file
P1380002 # Records deleted from TDV_ITEM_WEIGHT
P9900141 # Records inserted into TDV_ITEM_WEIGHT

Page 612 from 698
EHDV21175P- Copy Consolidated Items Weight
Phase:
This job computes the Item weight based on certain criteria and copies the computed value to the DT5131 in
RAWDAILYT60.
Process Flow:
1. Records that satisfy the job submission criteria Cycle/Period/NumPeriodsBack are selected from
RAWDAILYT60. The period value that corresponds to the purchase date is retrieved from the
CYCLEPERIODT03 and their Price Point (PP) is calculated using the following Formula:
PP_COMPUTED = ROUND(DT5675 / PPC_ COEFFICIENT) * PPC_ COEFFICIENT
where PPC_COEFFICIENT is the Price Coefficient based in which Price range the item Price
(DT5675) resides.
In rework mode for records where DT5098 = BCDTYPE parameter value

• DT5675 is updated as (DT5675 * DT5650) / DT5131
• DT5650 is updated as DT5131
2. Once PP is calculated, in order to retrieve/calculate the corresponding conv_weight for each

selected RAWDAILYT60 record, an iterative process takes place based on different criteria. Once
these criteria are satisfied the conv_weight value is retrieved or calculated based on the Item
Weight from TDV_ITEM_WEIGHT. Once this is done the DT5131 in RAWDAILYT60 is updated.
Join with Shop is done only when the static parameter SPECIFIC_SHOP is activated (is set to
Y), so why in the following steps Shop is in BOLD. When SPECIFIC_SHOP is not activated (is
set to N), then the MEDIAN is used across the shops. The flow of the process is based on the
following steps.
2.12. For Period, Shop, Nankey and PricePoint, if conv_weight found update
DT5131 in RAWDAILYT60.
2.2. For the items where conv_weight is not found in the above step, join is done using
Shop, Nankey, PricePoint and the conv_weight is retrieved for the nearest period, if
found DT5131 in RAWDAILYT60 (If for the nearest period we have multiple records
then median is taken)
Period, Nankey , PricePoint and compute median weight across the shops, and update
DT5131 in RAWDAILYT60.
Period, Shop, nankey and the conv_weight is retrieved for the nearest price point, and
update DT5131 in RAWDAILYT60 (If for the nearest period we have multiple records
then median is taken)
Nankey, PricePoint and get all conv_weight for the nearest period from different shops,
and compute the median weight across the shops, and update DT5131 in
RAWDAILYT60.

Page 613 from 698
2.6. For the items where conv_weight is not found in the above step, join is done using Shop
and Nankey and get all conv_weight for the nearest PricePoint and among them get the
conv_weight for the nearest period and update DT5131 in RAWDAILYT60.
2.7. For the items where conv_weight is not found in the above step, join is done using Period
and Nankey and get all conv_weight for the nearest PricePoint from different shops, and
compute the median weight across the shops, update DT5131 in RAWDAILYT60.
Nankey and get all conv_weight for the nearest PricePoint and among the list found get
all conv_weight for the nearest period from different shops, and compute the median
weight across the shops, and update DT5131 in RAWDAILYT60.
2.9. For remaining records update DT5131=1 as conv_weight in RAWDAILYT60.
3. New final step is added ,where for records DT5098 = BCDTYPE parameter value
▪ New Price is stored in DT5675 as (DT5650*DT5675)/DT5131

▪ DT5131 and DT5650 values are swapped
▪ Total number of records swapped are displayed in message log with error number
P1750001
Run mode: PROC or REWORK
Input File: None
Output File: None
Prerequisites: None
Table Description
TDV_ITEM_WEIGHT Average Weight by Item
TDV_PRICEPOINT_CALC Price Point Reference Table
● TDV_PRICEPOINT_CALC

Page 614 from 698
● TDV_PRICEPOINT_CALC – US specific (tech data)
Static Parameters:

SOURCE NUMBER Mandatory To identify the Source
SPECIFIC_SHOP LIST Mandatory Shop is to taken into consideration
Valid Values are:
‘Y’ for YES
‘N’ for NO
BCDTYPE NUMBER Mandatory Barcode Type value to identify Var
Weight Transactions
Dynamic Parameters:

Name (Yes/No)
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

code
with Period 0
P0020505 Missing Static Parameter S Missing Static Check if Static Parameter
Parameter is defined

Page 615 from 698
P9900196 No Records Selected for S No Records Selected for Check the job submission
Processing Processing from cycle/period values
RAWDAILYT60

P1038020 # Records selected for processing
P1049030 # Records Updated in Stepx
P1750001 # Swap record count
EHDV21176P- Price Load using OGRDS Char

Phase:
Price Load using OGRDS char computes standard price of specific items and loads into CNANPRICET34
automatically using the defined static parameters.
Process Step:
• Based on the IBD defined nankeys are selected

• From NAN_CHAR_VALUEST75 table NC_CHARVAL_CODE is obtained for the defined price
and quantity static parameters.
• Standard price for each nankey is computed using the formula std_price = price * quantity.
• The price and quantity are from NC_CHARVAL_NUMERIC_VAL column in CHAR_VALUEST24.
• Then the standard priceis loaded for the nankey in current and previous period.
Recommendations:
Prerequisites: Price and quantity coming from OGRDS should be loaded into
NC_CHARVAL_NUMERIC_VAL column in CHAR_VALUEST24
Table Description
Parameters :

Page 616 from 698
Static Parameters:

IBD SQL Mandatory IBD to process
CHARTYPE_PRICE SQL Mandatory Char type code used for Price
CHARTYPE_QTY SQL Mandatory Char type code used for Quantity
Dynamic Parameters:

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

NONE

code
P0020505 Mandatory Static Parameters S Parameter not Give the
parameters XXXX is not defined expected static
defined parameter some
value

Page 617 from 698
P0000000 Submission period date Date Get Date range for submission period
range Range
P1049038 NUMBER OF ITEMS Count of records inserted/merged into

INSERTED/MERGED CNANPRICET34
INTO CNANPRICET34:
EHDV21177P- Assortment Items Loader

Phase:
This process loads the items coming from non-cooperators based on the crosscoding instructions which are
loaded by OGRDS.
In order to massively identify these items, the Xcoding Group screen has been enhanced by a new field the
“Assortment New Shop” which in case a Xcode Group is related to items coming from non-cooperators, this
is filled by the associated shop code:
Process Step:
• For all the Xcode Groups that an associated “Assortment New Shop” has been defined (where field is
NOT NULL) all items greater than 0 (NAN_KEYS) and which are valid in the job submitted period, are
retrieved from the CROSSCODING_INSTRUCTIONST49 related to the group
• For the non already existing SHOP_PRICEST99 records (based on AC_SOURCE, AC_SHOPID,
NC_NAN_KEY, DT_FROM, DT_TO) these NAN_KEYS are inserted in SHOP_PRICEST99 using the
following values:
Field Value
AC_SOURCE SOURCE from the static parameter
PERIOD Job submitted Period
AC_SHOPID Assortment New Shop value from
CROSS_CODE_GROUPST26
NC_NAN_KEY NC_NAN_KEY from
CROSSCODING_INSTRUCTIONST49
DT_FROM Job submitted Period – FromDate

Page 618 from 698
DT_TO Job submitted Period – ToDate
AC_DTYPE 5651
NC_PRICE 1
Recommendations:
Prerequisites:
OGRDS Loader has been completed successfully.
Run mode : PROC
Input File : None
Output File: None
Table Description
CROSS_CODE_GROUPST26 List of Xcode Group ID 619lignme to be used for Xcoding
SHOP_PRICEST99 Table containing RMS prices by period, shop, NAN_KEY
Parameters :
Static Parameters:

SOURCE CHAR Mandatory Source used for inserting records in
SHOP_PRICEST99
Dynamic Parameters:

Name (Yes/No)
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No

code
P0020505 Mandatory Static Parameters S Parameter not Give the
parameters XXXX is not defined expected static
defined parameter some
value

Page 619 from 698

P9900155 No candidate records No candidate records selected for processing
selected for processing
P0940001 # Records inserted in # Records inserted in SHOP_PRICEST99

SHOP_PRICEST99 table table
P9931000 No Records inserted in No Records inserted in SHOP_PRICEST99
SHOP_PRICEST99 table table

Page 620 from 698
EHDV21178P – Additional ASIN Xcoding
Phase:
This program is used to apply the specific coding for ASIN codes using loaded decodering information into
TDV_OGRDS_DESCR_DECODER table.
•
Applies in loading AREA only
•
Applies in a limited shop list using Shop Group concept
•
Applies only on remaining uncoded purchases after usual xCoding is applied
Specific method used for ASIN codes:
▪ Check for any ASIN record related to collected barcode in

TDV_OGRDS_DESCR_DECODER table
▪ When found get nankey related to ASIN code in the cross coding instructions
▪ Assign nankey to purchase related to barcode
Example :
Collected Barcode 8004260922029 on 12/05/2020 in Italy
BAU coding No link found for Amazon Group nor any other for this barcode
ASIN coding One key found with barcode in decodering information :
Coding Key 994259482

Decodering 8004260922029|8004260922029||||||
Description Regina Toilet Roll 500 Ties per Roll – 100% FSC Certified Paper – 42
Rolls|Confezione da 42 Maxi Rotoli a 2 veli
Coding Link xCoding Group IT534 – NEXT(IT)

ASIN Barcode B07JYZ68YT
Coding Key 994259482
Nankey 58221857
Item Description SOFIDEL,REGINA (SOFIDEL),REGINA,ROTOLONI,TOILET
TISSUE,DRY,,ROLL,PLASTIC,PACKET,500’S,X1,
Action Nankey 58221857 is assigned to purchase

Page 621 from 698
Prerequisites :
EHDV21070P OGRDS Loader
Table/View Description
TDV_OGRDS_DESCR_DECODER Table that stores the OGRDS decodering details
separated in datatypes and loaded in the table ALL as separate
records
TDV_SHOP_GROUP_CONTENTS Shop group content details
CROSSCODING_INSTRUCTIONST49 Crosscoding links information extracted from EIMDB
Files:
Input Files:NONE
Output File:NONE
Parameters :
Static Parameters
Parameter Type Type Value Mode Description

PARM SHOP_GROUP Mandatory Shop Group for shop SCOPE filtering
Dynamic Parameters

# Periods Yes
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No

Page 622 from 698
Run Management :
Proc.TYPE CYCLE PERIOD Option Comments

Process XXXX YYYY.PPP normal Process generates factor set and inserts correction factors
- Resolve shop group

- Select data and apply coding
o Select data from ALLT11 for appropriate period range using submission period
and Numper dynamic parameter
▪
Filtering on selected shop group shops
▪
Filtering on nankey = 0
o Assign nankey according to preparation step outcome

▪
Purchase date within effective from and to dates
▪
Purchase barcode = predicted barcode
Report total count of selected records

- Report total count of assigned records
- Report total count of un-assigned records
-

code
P9900246 Job cannot be Period 0 S Job cannot be Please submit job in a valid
submitted with submission submitted with period
Period 0 Period 0
P1009004 No Records No Records S No Records No records have been
Selected for Selected for Selected for selected from ALLT11 for the
processing processing processing specific SHOP_GROUP
scope defined

code

Page 623 from 698
P9900371 # Selected # Selected Number of records selected from ALLT11 for the
records records SHOP_GROUP scope defined
P1006006 # Assigned # Assigned Number of records that have Xcoded, nankey has
records records been assigned
P1043001 # Unassigned # Unassigned Number of records that have not been Xcoded,
Records Records nankey has not been assigned

Page 624 from 698
EHDV21179P – PCP Correction Factor computation
Phase:
This program is used to calculate the computation factors which in turn will be used to determine the
distribution factors for each trip.The program uses the predefined PCP setup done via GUI and generates a
new set if required.
Prerequisites :
89P Intended User Alignment must have been processed earlier on same set of data
TDV_PCP_FACTOR_SET Table to store the set data of each PCP run
TDV_PCP_CORRECTION_FACTORS Table to store the correction factors of each PCP run
TDV_PCP_CATEGORY_EXCLUSION Table to store the excluded Stratum charvalue and IBD
TDV_PCP_CATEGORY_SETUP Table to store the PCP setup defined via GUI
VSMP_COLL_CHAR_VALUES View of SMP table – TSMP_COLL_CHAR_VALUES
VSMP_INDXWEEKSEXPFT39 View of SMP table – INDXWEEKSEXPFT39
VSMP_COLLABEXPFT57 View of SMP table - COLLABEXPFT57
VSMP_INDEXT30 View of SMP table - INDEXT30
Files:
Input Files:NONE
Output File:NONE
Parameters :
Static Parameters

Type
PARM ACTIVATION_DATE Optional Activation Date (New Set). If missing , would be set by
default with the last day of period being processed +
one day
PARM SAMPLE_INDEX Mandatory Member Index code
PARM SAMPLE_CS Mandatory Member Common Sample Code,Supposed to be 52W
CS from DsCI recommendation
PARM STRATUM_CHARID Mandatory Stratum Sample Char ID
PARM CATEGORY_ID Optional IBD ID for a single category run

Page 625 from 698
Dynamic Parameters

# Periods Yes Compute back data period range
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Run Management :

Process XXXX YYYY.PPP normal Process generates factor set and inserts correction factors
Under Validation Menu->PCP Setup->PCP1 screen user can configure categories and review
correction factors.
• IBD resolution
o Process will resolve IBD from specific PCP setup and if IBD’s are mutually exclusive ,it will
fail with error.
• Getting Expansion factor and Stratum Char values
o Process will use Index code ,Stratum Char ID and period range to fetch the respective
stratum char values for members
o Also XF for each member will be fetched using Index code , Common Sample and period
range
• Rawda data
o Using nankeys resolved from IBD ,member charvalues and period range respective raw
data are fetched from T60 table
o If there are any Category/Stratum combinations to be excluded in
TDV_PCP_CATEGORY_EXCLUSION ,their data will be removed for further processing.
• Factor set management
o Delete correction factors and factor set with date_from greater than or equal to activation
date

Page 626 from 698
o Close any existing factor set with activation date between date_from and date_to
o Open a new set with activation date
• Finally the correction factor is computed by Sum(ibd,stratum) / Sum(ibd) and stored in
TDV_PCP_CORRECTION_FACTORS.

code
P9900246 Job cannot be Period 0 S Job cannot be Please submit job in a
submitted with submission submitted with Period valid period
Period 0 0
P9900223 No Records No Records S No Records Selected Recheck the static
Selected for Selected from T60 using given parameters and date
processing from static parameters range used for
T60 submission
P9900378 Defined IBDs are IBD issue S IBDs are not mutually There are common
not mutually exclusive nankeys between 2 or
exclusive more IBD’s
P9900378 Defined IBD(s) are Empty IBD S No nans selected for Please resolve the IBD
empty given IBD again or recheck the
IBD rule setup.
P9900223 No chars found for Empty Chars S No chars found in SMP Please check the Date
period range for period range range or SMP char
used

code
P0720022 No of records inserted in xxxx Records It gives data value on number of records
inserted inserted in table xxxx
count
S0000041 Total Number of records excluded for Records It gives data value on number of records
processing deleted excluded as it is present in
count TDV_PCP_CATEGORY_EXCLUSION
P0720022 Set activation details Set It gives information on factor set
activated activated for which IBD
details
S0000041 Total Number of records deleted for Records Records deleted in
processing deleted TDV_PCP_CORRECTION_FACTORS
count for IBD :xxx
P9900141 No of records inserted in Records It gives data value on number of records
TDV_PCP_CORRECTION_FACTORS inserted inserted in table
count TDV_PCP_CORRECTION_FACTORS
P9900206 Total Number of NANs selected Records Total Number of NANs selected
inserted
count

Page 627 from 698
EHDV21180P – PCP Distribution Factor computation
Phase:
This program is used to calculate the distribution factor of each trip using the computed correction factor and
update the same in the T60 table.
Prerequisites:
89P Intended User Alignment must have been processed earlier on the same set of data.
179P – PCP Correction Factor computation must have been processed
TDV_PCP_FACTOR_SET Table to store the set data of each PCP run
TDV_PCP_CORRECTION_FACTORS Table to store the correction factors of each PCP run
TDV_PCP_CATEGORY_EXCLUSION Table to store the excluded Stratum charvalue and IBD
TDV_PCP_CATEGORY_SETUP Table to store the PCP setup defined via GUI
VSMP_COLL_CHAR_VALUES View of SMP table – TSMP_COLL_CHAR_VALUES
VSMP_INDXWEEKSEXPFT39 View of SMP table – INDXWEEKSEXPFT39
VSMP_INDEXT30 View of SMP table - INDEXT30
Files:
Input Files: NONE
Output File:NONE
Parameters:
Static Parameters

Type
PARM ACTIVATION_DATE Optional Activation Date (Set to use). If missing , would be set by
default with the last day of period being processed + one
day
PARM SAMPLE_INDEX Mandatory Member Index code
PARM SAMPLE_CS Mandatory Member Common Sample Code,Supposed to be 52W
CS from DsCI recommendation
PARM STRATUM_CHARID Mandatory Stratum Sample Char ID
PARM CATEGORY_ID Optional IBD ID for a single category run

Page 628 from 698
Dynamic Parameters:

# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DTSelection No
File Selection No
Run Management:

Process XXXX YYYY.PPP normal Process generates and updates distribution factors
Under Validation Menu->PCP Setup screen -> user can configure categories and review correction
factors.
• IBD resolution
o Process will resolve IBD from specific PCP setup and if IBD’s are mutually exclusive, it will
fail with error.
• Getting Stratum Char values
o Process will use Index code ,Stratum Char ID and period range to fetch the respective
stratum char values for members
• Getting correction factor
o Selecting appropriate set from TDV_PCP_FACTOR_SET using Activation Date parameter
• Rawda data
o Distribution factor is reset in T60 for the submitted period range and IBD’s.
o Using nankeys resolved from IBD ,member charvalues and period range respective raw
data are fetched from T60 table
o Correction factor is assigned to each purchase-using stratum and category, when it is not
found default value zero is assigned.
• Computing Distribution factors
o Distribution factor is computed as Correction Factor / Sum(Trip Correction Factors) and it is
updated into DT5953 column of T60 for each purchase record

Page 629 from 698

P9900246 Job cannot be Period 0 S Job cannot be Please submit
submitted with submission submitted with job in a valid
Period 0 Period 0 period
P9900223 No Records No Records S No Records Recheck the
Selected for Selected Selected from static
processing from T60 using given parameters and
T60 static date range
parameters used for
submission
P9900378 Defined IBDs IBD issue S IBDs are not There are
are not mutually mutually common
exclusive exclusive nankeys
between 2 or
more IBD’s
P9900378 Defined IBD(s) Empty IBD S No nans Please resolve
are empty selected for the IBD again
given IBD or recheck the
IBD rule setup.
P9900223 No processing No activation S No processing Kindly give a
date available available date available proper
in in activation date
TDV_PCP_CO TDV_PCP_CO
RRECTION_FA RRECTION_FA
CTORS for CTORS for
selected selected
activation date activation date
P9900223 No chars found Empty Chars S No chars found Please check
for period range in SMP for the Date range
period range or SMP char
used

code
P0720022 No of records inserted in xxxx Records inserted It gives data value on number of
count records inserted in table xxxx
P9900141 No of records updated in T60 Records inserted It gives data value on number of
/updated count records updated in T60
P9900206 Total Number of NANs Records inserted Total Number of NANs selected
selected count
P9900110 Processing date used Processing date uise Processing date used to select
din process correction factor
P0720021 Total Number of records Records inserted Total Number of records selected
selected for processing count from T60

Page 630 from 698
EHDV21181P – Automatic Price Validation Standard
Phase:
PHASE 2 –AUTOMATIC DATA VALIDATION
This program validates the prices delivered by the families in either of two methods which are standard and
temporary.
• Validation is applied on Rawda, comparing historical price datatype (DT5667) with some price limits
computed based on Standard price and MAD (median price deviation) of the current period. If price
is out of limits, this job is automatically updating current price datatype (DT5675), historical price
remains the same.
• If PROMOTION_STEP parameter is set to ON, then PRICE value of records with any of the
promotion datatypes defined by the static parameter PROMO_DT is rescaled by the
PROMO_RESCALE factor before comparing with the usual boundaries.
• All changes are logged in FAMILYLOGT67 table and can be displayed using FamilyLog screen.
• Rawda date range to be validated is defined by submission period and number of periods back.
• Validation is using the price group concept, shop characteristic to be used for price group is defined
as static parameter PRICE_GROUP.
• AF, BF and CF, METHOD, MADADJ are global parameters and their values have been setup in
TDV_JOB_GLOBAL_PARAMS.
If the RERUN Mode is ‘Y’ then

• Keys are selected for Price Validation from RAWDAILYT60. When the option is ‘Y’,select only
keys from RAWDAILYT60 where file number is -1 in PROCKEYST29. The filenumber is column
DT5095 in PROCKEYST29. Basket key is obtained using same set of AC_AUDIT,
DT_PUR_DATE, AC_TIME, AC_HHOLD, AC_SHOP and AC_BARCODE=’0’
inPROCKEYST29.
Scope of purchase record to be validated by APV:

• CASE 1 – When SHOP_SCOPE is instructed, only purchase acts belonging to that
particular shop scope will be validated by APV.
• CASE 2– When PRODUCT_SCOPE is instructed, only purchase acts belonging to that
particular product scope will be validated by APV.
• CASE 3– When both SHOP_SCOPE and PRODUCT_SCOPE are instructed, only purchase
acts belonging to both the scopes will be validated by APV.
• CASE 4– When both SHOP_SCOPE and PRODUCT_SCOPE are not instructed, then all
the purchase acts will be validated by APV.
MAD is validated as follows for TEMPORARY method,

• MAD adjustment factor (MADADJ) and minimum value of MAD (MINMAD) are defined as
static parameters.
• A new parameter ‘USE_DEF_MAD’ has been added in SETUP_PARAM table for temporary
method. The accepted values for this are ‘Y’ or ’N’ or any cycle value (Example:0004).
• Default value is given as ‘N’.
• If a country set up this parameter as either ‘Y’ or any cycle value, the process will take the
MINMAD static parameter value as MAD for those items where standard price exists but no
MAD exists at Product Group level and Price group 0 level in T34.

Page 631 from 698
Check for mandatory Parameters
The following parameters are mandatory if the method is either standard or temporary,
• If PROMOTION_STEP = ON, then at least one of PROMO_DT5662, PROMO_DT5663,
PROMO_DT5664, PROMO_DT5665 should be ON.
• PROMOTION_RESCALE and PRICE_GROUP are mandatory.
The following parameters are mandatory if the method is standard,
• LOW_PRICE_ADJ and MIN_TOL are mandatory.
The following parameters are mandatory if the method is temporary,
• LOWP_CHK_VAL1, LOWP_STD_VAL1,LOWP_CHK_VAL2, LOWP_STD_VAL2 either all
should be filled or none of them should be filled.
• MINMAD is mandatory.
Selecting appropriate Standard Price for data to process,
• Should obtain Standard Price at same Price Group at nankey level
• If missing then get Standard Price at Price Group 0 at nankey level
Selecting appropriate MAD for data to process
• Should obtain MAD at same Price Group at nankey level
• If missing then get MAD at Price Group 0 at nankey level
• If missing then get CVPC at same Price Group at Product Group level and set MAD = CVPC * Std
Price for same Price Group at nankey level
• If missing then get CVPC at Price Group 0 at Product Group level and set MAD = CVPC * Std Price
for Price Group 0 at nankey level
Validating MAD values
The following validation is followed if method is standard,
• If ORIGINAL_PRICE < LOW_PRICE_ADJ then MAD = 0
The following validation is followed if method is temporary,
• If LOWP_CHK_VAL1 and LOWP_CHK_VAL2 is not null,
If ORIGINAL_PRICE < LOWP_CHK_VAL1 then MAD = LOWP_STD_VAL1
If ORIGINAL_PRICE >= LOWP_CHK_VAL1 and ORIGINAL_PRICE < LOWP_CHK_VAL2
then MAD = LOWP_STD_VAL2
If the above two conditions are not satisfied then, MAD = max (MAD, MINMAD) *MADADJ
• If LOWP_CHK_VAL1 and LOWP_CHK_VAL2 is null then,
MAD = max (MAD, MINMAD) *MADADJ
Compute Standard Deviation
The following steps are followed if method is standard,
• MINMAD is computed with the formula,
MINMAD= (Std Price * MIN_TOL)/(AF*MADADJ)
• STD_DEVIATION is computed with the formula,
STD_DEVIATION=max (MAD, MINMAD) *MADADJ
The following steps are followed if method is temporary,
• Use MINMAD static parameter value.
• STD_DEVIATION = MAD (which was obtained in the previous step)
The following steps are followed for both standard and temporary method
Compute Standardize Price(Zp)
• Zp is computed with the formula,
Zp = (ORIGINAL PRICE – Std Price) / STD DEVIATION

Page 632 from 698
Correct standardized price(Zc)
Zc is computed with the below conditions,
• If Zp >= CF then Zc -> 0
• If BF <= Zp and Zp < CF then Zc -> AF * (CF – Zp) / (CF – BF)
• If AF <= Zp and Zp < BF then Zc -> AF
• If –AF < Zp and Zp < AF then Zc -> Zp
• If –BF < Zp and Zp <= -AF then Zc -> -AF
• If -CF < Zp and Zp <= -BF then Zc -> -AF * (CF + Zp) / (CF – BF)
• If Zp <= -CF then Zc -> 0
Convert back corrected price to unstandardized format with the formula show below,
• 𝑉𝐴𝐿𝐼𝐷𝐴𝑇𝐸𝐷𝑃𝑅𝐼𝐶𝐸 = 𝑍𝐶∗𝑆𝑇𝐷𝐷𝐸𝑉𝐼𝐴𝑇𝐼𝑂𝑁+𝑆𝑇𝐷𝑃𝑅𝐼𝐶𝐸
If promotion step =ON and promo flag = 1 repeat same steps with ORIGINAL PRICE * PROMO_RESCALE
• Zp = (ORIGINAL PRICE * PROMO_RESCALE –Std Price) / STD DEVIATION
Correct standardized price with the following conditions,
• If Zp >= CF then Zc -> 0
• If BF <= Zp and Zp < CF then Zc -> AF * (CF – Zp) / (CF – BF)
• If AF <= Zp and Zp < BF then Zc -> AF
• If –AF < Zp and Zp < AF then Zc -> Zp
• If –BF < Zp and Zp <= -AF then Zc -> -AF
• If -CF < Zp and Zp <= -BF then Zc -> -AF * (CF + Zp) / (CF – BF)
• If Zp <= -CF then Zc -> 0
Convert back corrected price to unstandardized format as shown below,
• 𝑉𝐴𝐿𝐼𝐷𝐴𝑇𝐸𝐷𝑃𝑅𝐼𝐶𝐸𝑃𝑅𝑂𝑀𝑂=𝑍𝐶∗𝑆𝑇𝐷𝐷𝐸𝑉𝐼𝐴𝑇𝐼𝑂𝑁+𝑆𝑇𝐷𝑃𝑅𝐼𝐶𝐸
Check for correction
• MIN_CORRECTION is computed with the formula,
MIN_CORRECTION = (ORIGINAL_PRICE+ 1)/100
• CORRECTION is computed with the formula,
CORRECTION = abs (VALIDATED Price –ORIGINAL PRICE)
The following steps are followed for both standard and temporary method
• The following conditions are checked before applying corrections,
If VALIDATED Price <> ORIGINAL PRICE then
if(Promotion_STEP=OFF or promo_flag= 0 or (Promotion step = ON and promo flag = 1 and
VALIDATED Price Promo<> (ORIGINAL PRICE * PROMO_RESCALE) then we should apply
correction.
Applying Correction
• Should check if CORRECTION > MIN_CORRECTION then
Set DT5675 = VALIDATED PRICE and track change in Family log T67 Table
The following steps are followed if method is temporary,
• Set DT5675 = VALIDATED PRICE and track change in Family log T67 Table
Report counters the following message logs,
• Total Number of Purchase Acts Validated (Tracking)
• Number of Distinct Families Validated (Tracking)
• Number of Items without MAD(Tracking)
• Number of Purchase Acts without MAD and not inspected (Tracking)
• # Records for which price is corrected automatically (Tracking)
• Number of Warnings (Tracking)

Page 633 from 698
Sample Calculation (Temporary)
The following is an example of a purchase act validated by APV Temporary method,

The parameters used are as follows,
AF - 3.48
BF - 4.56
CF - 5.73
MADADJ - 1.4826
MINMAD - 0.1
PRICE_GROUP - PGROUP
PROMOTION_STEP - ON
RERUN - N
LOWP_CHK_VAL1: 0.1
LOWP_CHK_VAL2: 0.2
LOWP_STD_VAL1: 0.01
LOWP_STD_VAL2: 0.02
PROMO_DT01: 5662
PROMO_RESCALE: 0.5
DEFAULT MAD is used
Purchase Act for which validation is applied is as follows,

audit dt_pur_date time hhold shop barcode nankey
4034 20201003 1412 0000017953 0000000034 30340111018711D 9685700
dt5667 promo_flag product_grp price_grp Std_price MAD

25.20645 1 8780 0000000002 8.15128 2.053

Here LOWP_CHK_VAL1 and LOWP_CHK_VAL2 is not null,
• If ORIGINAL_PRICE < LOWP_CHK_VAL1 --> FALSE
• If ORIGINAL_PRICE >= LOWP_CHK_VAL1 and ORIGINAL_PRICE < LOWP_CHK_VAL2 --
>FALSE
• The above two conditions are not satisfied so, MAD = max (MAD, MINMAD) *MADADJ
MAD = max (2.053,0.1) * 1.4826
MAD = 3.0437777999999995
• Using MINMAD static parameter value, MINMAD = 0.1
• STD_DEVIATION = MAD (which was obtained in the previous step)
STD_DEVIATION = 3.0437777999999995
• Zp = (ORIGINAL PRICE – Std Price) / STD DEVIATION
Zp = (25.20645 - 8.15128)/3.0437778
Zp = 5.6032900956173615

Page 634 from 698
• If Zp >= CF then ZC -> 0 -->FALSE
• If BF <= Zp and Zp < CF -->TRUE
then Zc -> AF * (CF – Zp) / (CF – BF)
Zc = 3.48*(5.73- 5.6032900956173615)/ (5.73 - 4.56)
Zc = 0.3768807412406698
• If AF <= Zp and Zp < BF then Zc -> AF -->FALSE
• If –AF < Zp and Zp < AF then Zc -> Zp -->FALSE
• If –BF < Zp and Zp <= -AF then Zc -> -AF -->FALSE
• If -CF < Zp and Zp <= -BF then Zc -> -AF * (CF + Zp) / (CF – BF) -->FALSE
• If Zp <= -CF then Zc -> 0 -->FALSE
Convert back corrected price to unstandardized format
validated_price = 0.3768807412406698 * 3.0437777999999995 + 8.15128
validated_price = 9.29842
Since promotion step =ON and promo flag = 1 repeat same steps with ORIGINAL PRICE *
PROMO_RESCALE factor
• Zp = (ORIGINAL PRICE * PROMO_RESCALE –Std Price) / STD DEVIATION
Zp = (25.20645 * 0.5 - 8.15128)/3.0437778
Zp = 1.4626379757418564
• Correct standardized price with the following conditions,
If Zp >= CF then Zc -> 0 -->FALSE
If BF <= Zp and Zp < CF then Zc -> AF * (CF – Zp) / (CF – BF) -->FALSE
If AF <= Zp and Zp < BF then Zc -> AF -->FALSE
If –AF < Zp and Zp < AF then Zc -> Zp -->TRUE
Zc = 1.4626379757418564
If –BF < Zp and Zp <= -AF then Zc -> -AF -->FALSE
If -CF < Zp and Zp <= -BF then Zc -> -AF * (CF + Zp) / (CF – BF) -->FALSE
If Zp <= -CF then Zc -> 0 -->FALSE
Convert back corrected price to unstandardized format as shown below,
• 𝑉𝐴𝐿𝐼𝐷𝐴𝑇𝐸𝐷𝑃𝑅𝐼𝐶𝐸𝑃𝑅𝑂𝑀𝑂=𝑍𝐶∗𝑆𝑇𝐷𝐷𝐸𝑉𝐼𝐴𝑇𝐼𝑂𝑁+𝑆𝑇𝐷𝑃𝑅𝐼𝐶𝐸
• Validated_price_promo = 1.4626379757418564 * 3.0437777999999995+ 8.15128
Validated_price_promo = 12.60323
If VALIDATED Price <> ORIGINAL PRICE then -->TRUE
9.29842<> 25.20645
• if(Promotion_STEP=OFF or promo_flag= 0 -->FALSE
or (Promotion step = ON and promo flag = 1
and VALIDATED Price Promo<> (ORIGINAL PRICE * PROMO_RESCALE) -->FALSE
12.60323 <> (25.20645 * 0.5)
12. 60323 <> 12. 60323 -->FALSE
since the above two conditions are not satisfied so there is no need to apply correction for this
record.

Page 635 from 698
Sample Calculation (Standard)
The following is an example of a purchase act validated by APV Standard method,

The parameters used are as follows,
AF - 3.48
BF - 4.56
CF - 5.73
MADADJ - 1.4826
METHOD - STANDARD
PROMOTION_STEP - OFF
RERUN - N
LOW_PRICE_ADJ – 0.5
MIN_TOL –0.8
Purchase Act for which validation is applied is as follows,
audit dt_pur_date time hhold shop barcode nankey
6120 20220709 1637 108276 2301 9310151811105 43043566
dt5667 promo_flag product_grp price_grp Std_price MAD

0.6875 0 1838 0000000004 6.875 0

The following validation is followed if method is standard,
• If ORIGINAL_PRICE < LOW_PRICE_ADJ then MAD = 0
0.6875 < 0.5 -->FALSE
• MINMAD is computed with the formula,
MINMAD= (Std Price * MIN_TOL)/(AF*MADADJ)
MINMAD = (6.875 * 0.8)/ (3.48 * 1.4826)
MINMAD = 1.0660055106670328
• STD_DEVIATION is computed with the formula,
STD_DEVIATION=max (MAD, MINMAD) *MADADJ
STD_DEVIATION=max (0, 1.0660055106670328) *1.4826
STD_DEVIATION = 1.5804597701149428
• Zp = (ORIGINAL PRICE – Std Price) / STD DEVIATION
Zp = (0.6875 - 6.875)/1.5804597701149428
Zp = -3.9149999999999996
• If Zp >= CF then ZC -> 0 -->FALSE
• If BF <= Zp and Zp < CF then Zc -> AF * (CF – Zp) / (CF – BF) -->FALSE
If AF <= Zp and Zp < BF then Zc -> AF -->FALSE
• If –AF < Zp and Zp < AF then Zc -> Zp -->FALSE
• If –BF < Zp and Zp <= -AF then Zc -> -AF -->TRUE
-4.56 <-3.9149999999999996 and -3.9149999999999996 <= -3.48
Zc = -3.48
• If -CF < Zp and Zp <= -BF then Zc -> -AF * (CF + Zp) / (CF – BF) -->FALSE
• If Zp <= -CF then Zc -> 0 -->FALSE
Convert back corrected price to unstandardized format

Page 636 from 698
validated_price = -3.48 * 1.5804597701149428 + 6.875
validated_price = 1.375
• MIN_CORRECTION is computed with the formula,
MIN_CORRECTION = (ORIGINAL_PRICE+ 1)/100
MIN_CORRECTION = (0.6875+1)/100
MIN_CORRECTION = 0.016875
• CORRECTION is computed with the formula,
CORRECTION = abs (VALIDATED Price –ORIGINAL PRICE)
CORRECTION = abs (1.375 - 0.6875)
CORRECTION = 0.6875
If VALIDATED Price <> ORIGINAL PRICE then -->TRUE
1.375 <> 0.6875
• if(Promotion_STEP=OFF or promo_flag= 0 -->TRUE
or (Promotion step = ON and promo flag = 1
and VALIDATED Price Promo<> (ORIGINAL PRICE * PROMO_RESCALE) -->FALSE
since one condition is satisfied, need to apply correction for this record.
Applying Correction
• Should check if CORRECTION > MIN_CORRECTION then -->TRUE
0.6875 < 0.016875
Setting DT5675 = VALIDATED PRICE and track change in Family log T67 Table
DT5675 = 1.375
Corrections are applied in DT5675.
The REWORK flag is handled in a way that even when the production cycle is locked, the job will not get
failed when the user submits the job in REWORK mode
Recommendations:
Constraints:
If the production cycle has been locked, it is required to submit this job in Rework mode.
Prerequisites:
➢Availability of the data in for the submission period T in Rawda
➢Availability of statistical parameters MAD (median price deviation) for the period T
➢Availability of standard price in T

Table Description
Hive Tables Involved:

Table Description
group, NAN_KEY

Page 637 from 698
Files:
Input Files:
No Input files
Output Files:
File containing all purchase acts with price=0 (File id: FILEO1)
1 500 STR VARCHAR2
Parameters:
Static Parameters
Parameter Type Mandatory Comment STD TMP
PRICE_GROUP CHAR Y Shop Char to use for Price ✓ ✓
Group
PRODUCT_SCOPE SQL N IBD for nankeys to validate ✓ ✓
SHOP_SCOPE SQL N Shop Group for shop to ✓ ✓
validate
MIN_TOL NUMERIC N Minimum Tolerance ✓
LOW_PRICE_ADJ NUMERIC N Low Price Adjustment ✓
RERUN LIST N Rerun mode selection (Y or ✓ ✓
N)
PROMOTION_STEP LIST Y Promotion Step handling ✓ ✓
(ON or OFF)
PROMO_DT5662 LIST N UseDT5662 for Promo Flag ✓ ✓
(Y or N)
(Y or N)
(Y or N)
(Y or N)
PROMO_RESCALE NUMERIC N Rescaling factor for ✓ ✓
Promotions
MINMAD NUMERIC N Minimum Mad value ✓
LOWP_CHK_VAL1 NUMERIC N Low Price Check Value 1 ✓
LOWP_CHK_VAL2 NUMERIC N Low Price Check Value 2 ✓
LOWP_STD_VAL1 NUMERIC N Low Std Value 1 ✓
LOWP_STD_VAL2 NUMERIC N Low Std Value 2 ✓

Page 638 from 698
Global Parameters
AF, BF and CF, METHOD, MADADJ are global parameters and their values have been setup in
TDV_JOB_GLOBAL_PARAMS.
Parameter Type Value Comment
METHOD CHAR STANDARD Can be TEMPORARY to maintain current process
AF NUMERIC 3.48
BF NUMERIC 4.56
CF NUMERIC 5.73
MADADJ NUMERIC 1.4826
Dynamic Parameters:

# Periods Yes Yes
Single Run Job No
Timestamp No
PC Selection No Obsolete feature
F# Selection No Obsolete feature
File Selection Yes
Rework Management:


code
P0020505 Static Value not present Stat.Val not S Static value is not Enter valid Static
present present in value into
CAL_FACTORT05 CAL_FACTORT05
table table
P9900225 Processing option and Processing option S Processing option and If production cycle is
Period Lock: NOT OK an Period Lock: NOT OK locked, it is required
to submit the job in
REWORK mode
P9900196 No Records Selected for No Records S Records are not Check if Rawda has
processing Selected selected from been loaded for
RAWDAILYT60 or submission
PROCKEYST29 table cycle/period
for processing

Page 639 from 698
P1021006 Error in Identifying System Error S Unable to identify the Open a QLS
Standard Price Standard Price
P9900015 Divide By Zero Error Divide By Zero S Divide By Zero Error Open a QLS
Error
P9900550 Low Price Calculation can System S Not all the static Review the static
notperformed.MissingStatic parameters for the parameters for Low
parameters. LOW price validation price validation
have been defined.
P0020516 Job cannot be submitted System S Job cannot be Submit the job with a
for period 0 submitted for period 0 valid period
P1021015 Shop not Found in T40 Shop not in T40 S No Shops Present in Check content of
the SHOPUSABILITYT40
SHOPUSABILITYT40 table
table
P9900235 There are Shops that they There are Shops W Shops are Found in Check content of
are not in that RAWDAILYT60but not SHOPUSABILITYT40
ShopUsabilityT40 they are not in in available in table
ShopUsabilityT40 SHOPUSABILITYT40
table
P0020724 IBD empty, please check IBD empty, S IBD empty, please Enter a valid IBD and
your setup please check your check your setup submit the job
setup
P0020725 Shop Group empty, please Shop Group S Shop Group empty, Enter a valid Shop
check your setup empty, please please check your Group and submit the
check your setup setup job

code
P9900005 Required No. of Periods not Periods not Information message only. No action required
Present enough
P9900224 Processing option and Period Processing Information message only. No action required
Lock: OK option and
Period Lock:
OK
P1021001 Stdprice not found for PC-Item Stdprice not Information message only. No action required
found
P1021002 MAD not found for PC-Item MAD not found Information message only. No action required
P1810006 Total Number of Purchase Validated Information message only. No action required
Acts Validated Record Cnt
P1810001 Number of Distinct Families Distinct Family Information message only. No action required
Validated Cnt
P1810003 Number of Items without MAD No MAD –Item Information message only. No action required
count
P1810004 Number of Purchase acts No MAD-Pur. Information message only. No action required
without MAD and not Act cnt
inspected
P1810002 # of Records for which price is Corrected Information message only. No action required
corrected Automatically Record Cnt
P1810005 Number of Warnings Warnings Count Information message only. No action required

Page 640 from 698
EHDV21182P – Price and MAD Computation Standard
Phase:
PHASE 4 –SERVICES
A new method has been defined in DsCI for Automatic Price Validation. DsCI has defined a new
methodology to compute Standard Price, MAD, CV_PC by either temporary or standard.
Temporary method is a replica of EHDV21171P- New Standard Price and MAD Computation job.
Main differences between standard method with temporary method:
- Add calculation at module level (with and without price group – 𝑆𝑇𝐷_𝑃𝑅𝐼𝐶𝐸 and 𝐶𝑉_𝑃𝐶)
- Decision to use 2 clusters: coefficient_of_variance <= CLUSTER_THRESHOLD (static parameter) [instead
of variance < 0.5, in legacy] and num_obs (after removing outliers) < MIN_CLUSTER_OBS (static
parameter).
- STD Price “decisions” (outlier removal and cluster definition) with Original Price | Std price calculation with
validated price - MAD calculated with all observations and original price or validated price (parameterized -
by IBD)
Process details:
• Check Parameters (static and dynamic)
METHOD= STANDARD METHOD=TEMPORARY

PRICE_GROUP is mandatory
PRODUCT_SCOPE is optional when filled should be valid
SHOP_SCOPE is optional when filled should be valid
FPV_3_4_EXCLUSION is mandatory
MAD_DATATYPE is mandatory
if P2 is filled ➔ NOBS_P2_NAN and NOBS_P2_PG_MOD should be filled as well
if P3 is filled ➔ NOBS_P3_NAN and NOBS_P3_PG_MOD should be filled as well
• MAD_DATATYPE (dt5667 or dt5675) is mandatory for Standard method.

• No check to perform on existing data and processing mode.
• Compute P1 and if activated P2 / P3 date ranges from Periods parameter.
▪ P1, P2, P3 are back period length. By using this back period, we get data
from rawdailyt60 table for those back periods and processing.
▪ P1 is mandatory static parameter NOBS_P1_NAN, NOBS_P1_PG_MOD
should be filled.
▪ P2 is not mandatory static parameter. If filled NOBS_P2_NAN,
NOBS_P2_PG_MOD should be filled.
▪ P3 is not mandatory static parameter. If filled NOBS_P3_NAN,
NOBS_P3_PG_MOD should be filled.
• Resolve IBD using PRODUCT_SCOPE parameter if filled.
• Resolve Shop Group using SHOP_SCOPE parameter if filled.
• FPV_3_4_EXCLUSION is mandatory (ON/OFF).
• More than MIN_ITEM count values only enter for MAD calculation.
• K factor is used in standard price calculation.

Page 641 from 698
Process Step
Get data for process

• Using IBD when activated:
Get distinct nc_nan_key from tdv_ibd_resolution table.
• Using SHOP_SCOPE when activated:
Get distinct sgr_ac_shop from tdv_shop_group_contents table.
• Using the largest period date range:
Use larger date range P3 or P2 or P1 according to static parameter
selection.
• Link with Price Group through shop characteristics:
Using price_group parameter and get data from shopusability table.
• Link with Module and Product Group through nankeys:
Get nc_module and product group information from modulest31 table with
nan_keys from nan_itemst73 table.
• Select appropriate price values and keep in separate column:
FIELD METHOD= STANDARD METHOD=TEMPORARY
PRICE FOR = historical price (DT5667) for = historical price (DT5667) for
SELECTION all periods submission period
= validated price (DT5675) for
past periods
= historical price (DT5667) for = historical price (DT5667) for

PRICE FOR submission period submission period
COMPUTATION = validated price (DT5675) for = validated price (DT5675) for
past periods past periods
= historical price (DT5667) for

submission period = historical price (DT5667) for
PRICE FOR MAD = validated price submission period
(MAD_DATATYPE) for past = validated price (DT5675) for
periods. past periods
MAD_DATATYPE comes from

Static parameter which is either
DT5667 or DT5675.
Note: price_for_selection, price_for_computation, Price_for_mad fields are collected there

and would be used differently based on Method and Step.
In Standard method price_for_selection and price_for_computation field used for

standard price calculation and price_for_mad used for MAD calculation.

Page 642 from 698
Step_1:
• Remove records having DT5904 = 3 or DT5904 = 11 or DT5904 = 20 from
rawdailyt60 table when FPV_3_4_EXCLUDE = ON.
• Store NOBS as count of records
Note: FPV_3_4_EXCLUDE comes from static parameter
Step_2:
Need to collect 6 or 4 levels of data for the submitted period.
- Collect level data for Period 1

• Level 1 – Nankey/Price Group using NOBS_P1_NAN
• Level 2 – Nankey using NOBS_P1_NAN
• Level 3 – Module/Price Group using NOBS_P1_PGMOD
• Level 4 – Module using NOBS_P1_PGMOD
• Level 5 – Product Group/Price Group using NOBS_P1_PGMOD
• Level 6 – Product Group using NOBS_P1_PGMOD
Note: Level 3 and Level 4 are called when user select method as standard method.
Example data for cluster 1 standard method calculation:

Period Level Nan_key Price_grp Product_group Module
P3 L1 43965701 0000000008 0 0
Price_for_selection Price_for_computation Price_for_mad

31.4 31.4 31.4
25.12 25.12 25.12
31.4 31.4 31.4
15.7 15.7 15.7
43.96 43.96 43.96
Note: Here only 5 observations are collected for process after outlier one record removed
and 4 records entering into standard price calculation and calculate population standard
deviation and mean and finding coefficient of variance and applied condition check for
cluster count. It comes under cluster count =1.

Page 643 from 698
- Standard Price Computation
• Price Selection using PRICE_FOR_SELECTION field.
• If Temporary method:
▪ Compute percentile 25 and 75, lower bound and upper bound by nankey,
product group, price group.
• If Standard method:
▪ Compute percentile 25 and 75, lower bound and upper bound by nankey,
module, product group, price group.
• Formulas:
1. Lower bound = (percentile25-(1.5*(percentile75- percentile25)))
2. Upper bound = (percentile75+(1.5*(percentile75- percentile25)))
3. status = case when (price_for_selection < (percentile25-(1.5*(percentile75-
percentile25)))) then 'OUTLIER' when (price_for_selection >
(Percentile 75+ (1.5 *(percentile 75- percentile 25)))) then 'OUTLIER'
else 'VALID' end.
• Detect and remove outliers from collected data use valid data for further calculation.
Note: Using price for selection field find 25 and 75 percentiles and find lower and
upper bound by given formula and detect the price_for_selection is valid or
outlier by using status formula. Removes the outlier from dataset and takes
valid records for further calculations.
Sample calculation using nan_key = 43965701 and price_grp = 0000000008 data:

Percentile 25: 25.12 (Percentile 25 calculation done by using
price_for_selection)
Percentile 75: 31.4 (Percentile 75 calculation done by using
price_for_selection)
Lower bound = (25.12 -(1.5*(31.4 - 25.12)))
Lower bound = 15.7
Upper bound = (31.4 +(1.5*(31.4 - 25.12)))
Upper bound = 40.82
Status = (31.4 < (25.12 - (1.5*(31.4 - 25.12)))) then ‘OUTLIER’ when (31.4 >
(31.4+(1.5*(31.4-25.12)))) then 'OUTLIER' else 'VALID' end.
Status = valid (Here used price value as 31.4)
Status = (43.96 < (25.12 - (1.5*(31.4 - 25.12)))) then ‘OUTLIER’ when (43.96
> (31.4+(1.5*(31.4-25.12)))) then 'OUTLIER' else 'VALID' end.
Status = outliers (Here used price value as 43.96)
- Determining cluster:
• If Standard method: (nankey, module, product group, price_group)
▪ Compute Coeff of Variance = Std_deviation(population std) / Mean
▪ If Coeff of Variance <= CLUST_THRESH (Global parameter) or NOBS <
MIN_CLUSTER_OBS (Global parameter) then cluster count=1 else cluster
count = 2.
Note: To find price belonging to cluster 1 or 2 first find population standard
deviation and mean by nankey, module, product group, price_group and find
count(price_for_selection) by nankey, module, product group, price_group. And
compare coeff of variance and NOBS with static parameter and conclude price
belongs to 1 or 2 cluster.
• If Temporary method: (nankey, product group, price_group)
▪ Compute Variance
▪ If Variance < 0.5 then cluster count=1 else cluster count = 2
Note: For temporary find sample variance by nankey, product group,
price_group.

Page 644 from 698
• If cluster count = 1 copy data into final table.
• If cluster count = 2 apply clustering method.
o Apply K-mean clustering method with cluster 2 and random state as none
for price_for_selection field.
o Cluster(count) <= MIN_OBS_BY_CLUSTER (Global parameter) ➔ ignore it
Note: cluster value count for 0 and 1 should not be less than
MIN_OBS_BY_CLUSTER. If less than MIN_OBS_BY_CLUSTER
ignore
o Compute cluster score by nankey, module (Standard only), product group,
price group, cluster.
Note: 1. Find sample variance, count by nankey, module (Standard only),
product group, price group, cluster. Scoreclass = sample
variance/count.
2. Scoreclass value minimum take into for further calculations.
o Copy data from cluster with minimal score into final table.
• Standard price formula:
▪ Median price = median (PRICE_FOR_COMPUTATION)
▪ Std price = w*median price + (1- w) * most recent std price from
CNANPRICET34)
Where w is computed as n/(n+k) with n= number of observations for the level being
processed
When the most recent std price is empty default value 1 is assigned to w.
Note: 1. Load Std price for nankey and product group levels into
CNANPRICET34 by inserting new and updating existing records.
2. If standard method: Load Std price for module level into
TDV_MODULE_PRICES by inserting new and updating existing
records.
Standard price calculation using nan_key = 43965701 and price_grp = 0000000008

data:
Coeff of Variance = 41.287075/25.905
Coeff of Variance = 10.32176875
Condition check for cluster:
1.593787879 <= 0.3 (Global parameter) or 4 < 10 (Global parameter) then

cluster count=1 else cluster count = 2.
cluster count = 1
Cluster count = 1:
Median price = 28.26

w = 5/(5+0) = 1
recent std price from CNANPRICET34 = 32.04876
standard price = 1 * 28.26 + ((1-1) * 32.04876)
standard Price = 28.26

Page 645 from 698
Cluster count = 2:
Nan_Key Price_grp Product_grp Price_for_selection Cluster

36598872 0 0 3.75 0
36598872 0 0 3.75 0
36598872 0 0 2.0 1
36598872 0 0 2.1 1
36598872 0 0 2.0 1
36598872 0 0 3.75 0
36598872 0 0 3.80 0
Note: 1. After applied k-mean clustering price_for_selection values divided
into two cluster 0 and 1.
2. Cluster value count for 0 and 1 should not be less than
MIN_OBS_BY_CLUSTER. If less than MIN_OBS_BY_CLUSTER
ignore. (Here cluster 0 and 1 contain 4 and 3 observations so, no
need to eliminate)
3. For cluster 0 variance = 0.00625 and cluster 1 variance = 0.00333.
4. Scoreclass for 0 cluster = 0.001563
Scoreclass for 1 cluster = 0.00111
5. Here score class is minimum for cluster 1. So, cluster 1 datas goes
for further calculations.
- MAD Computation step at Nankey level using PRICE_FOR_MAD field
Use all observations from collected data (Take data for calculation before
outlier detection).
Use only non-outlier(Valid) observations from the final table.
• Compute median_price by nankey, price group (Use price for mad field)
• Compute MAD = median (abs (PRICE_FOR_MAD – median_price)) by
nankey, price group.
• Load MAD and NOBS into CNANPRICET34. If data already present for submitted
period and cycle update the data and not present insert the data.
Sample calculation using nan_key = 43965701 and price_grp = 0000000008 data:
Median price = 28.26
Compute MAD = 3.14
Note: Here first find median for price_for_mad by nan_key and price_group then
subtract price_for_mad values with calculated median value. In some cases,
negative value will come. So, take absolute then take median for absolute
column.
- CV_PC Computation step at Module level using PRICE_FOR_MAD field
This module level is only applicable for standard methods.

• Use all observations from collected data (Take data for calculation before
outlier detection)
• Compute number of items price(count) by module, price group, nan_items.
• Check if number of items(count) >= MIN_ITEMS (static parameter) then go for
further calculation else eliminate that data.

Page 646 from 698
• Compute median_price by nan_items(nankey), module, price_group (Using
price_for_mad).
• Compute MAD by nan_items(nankey), module, price_group (using price_for_mad)
• Compute MAD = median (abs (PRICE_FOR_MAD – median_price))
• Compute CV_PC by module, price group
CV_PC = average (MAD / Median price)
Note: 1. For standard method takes all nan_items count for particular module,
price_group.
2. Load CV_PC and NOBS into TDV_MODULE_PRICES by inserting
new and updating existing records.
3. Sample calculation did for product group below that is similar for this
as well.
- CV_PC Computation step at Product Group level using PRICE_FOR_MAD field
Example data for product group:

Nc_module Product_grp Nan_key Price_grp Nan_items Price for mad
0 8475 0 0 81444562 0.98
0 8475 0 0 81444562 2.6
0 8475 0 0 81444562 4.8
0 8475 0 0 43182387 10
0 8475 0 0 43182387 10
0 8475 0 0 82943347 4.5
0 8475 0 0 82943347 4.5
0 8475 0 0 82943347 7.5
Use all observations from collected data (Take data for calculation before
outlier detection)
Use only non-outlier observations from final table
• Compute number of items price(count) by product group, price group, nan_items.
• Check if number of items (distinct count) >= MIN_ITEMS (static parameter) then
go for further calculation else eliminate that data.
3 >= 2(static parameter)
Note: Here are 3 distinct nan items present in the 8 records. So, that is greater than
static parameter min_item it enters into calculation part.
•
• Compute median_price by nan_items(nankey), product group, price_group (Using
price_for_mad).
• Compute MAD by nan_items(nankey), product group, price_group (using
price_for_mad)
• For standard:
Compute MAD = median (abs (PRICE_FOR_MAD – median_price))
Compute MAD absolute = (abs (PRICE_FOR_MAD – median_price))
Compute MAD = median (Compute MAD absolute) by
nan_items,product_grp, price_grp
CV_PC = average (MAD / Median price) by product_group, price_grp

Page 647 from 698
• For Temporary:
Compute MAD = abs (PRICE_FOR_MAD – median_price)
CV_PC = average (MAD / Median price)
Note: 1. For Temporary method NOBS count find by taking distinct
nan_items(nan_key) count for particular product group, price_group
whereas standard method takes all nan_items count for particular
product group, price_group.
2. Load CV_PC/MAD and NOBS into CNANPRICET34 by inserting new
and updating existing records.
Sample calculations:
Nan_items Price Medianprice PRICE_FOR_MAD abs MAD MAD/Median

for – median_price price
mad
81444562 0.98 2.6 -1.62 1.62 1.62 0.623076

81444562 2.6 2.6 0 0 1.62 0.623076
81444562 4.8 2.6 2.2 2.2 1.62 0.623076
43182387 10 10 0 0 0 0
43182387 10 10 0 0 0 0
82943347 4.5 4.5 0 0 0 0
82943347 4.5 4.5 0 0 0 0
82943347 7.5 4.5 3 3 0 0
CV_PC = average(MAD/Median price column)

CV_PC = 0.207692307666666
Final Step:
- Report count of records selected for each level and period
- Report count of standard prices computed for each level and period
- Report count of MAD computed for each level and period

Table Description
MODULEST31 List of Modules extracted from EIMDB.
group, NAN_KEY.
shop by cycle and period (0 or YYYYPPP).
RAWDAILYT60 To store Standard price, CV_PC computed at Module level.
Prerequisites:
Availability of the data in rawda for the submission period.
Price Group information is available in Shop Usability table.
Run mode: PROC
Input Files: None
Output File: None

Page 648 from 698
Static Parameters:
Parameter Type value Mode Description

type
List FPV_3_4_EXCLUDE Mandatory Can be ON or OFF, it excludes records having
DT5904 = 3 or DT5904 = 11 when selected ON.
Numeric K Mandatory K factor to use in standard price formula.
Numeric MIN_ITEM Mandatory Minimum number of items for MAD computation,
default value is zero.
Numeric P1 Mandatory Period length to consider at first step.
Numeric P2 Optional Period length to consider at second step.
Numeric P3 Optional Period length to consider at Third step.
Char PRICE_GRP Mandatory Char ID for Price Group determination
SQL PRODUCT_SCOPE Optional IBD number when need to reduce scope
SQL SHOP_SCOPE Optional Shop Group ID when need to reduce Shop
Scope
Numeric NOBS_P1_NAN Mandatory Min observation for P1 Nankey level
Numeric NOBS_P1_PG_MOD Mandatory Min observation for P1 Module or Product Group
level.
Numeric NOBS_P2_NAN Optional Min observation for P2 Nankey level.
Numeric NOBS_P2_PG_MOD Optional Min observation for P2 Module or Product Group
level.
Numeric NOBS_P3_NAN Optional Min observation for P3 Nankey level
Numeric NOBS_P3_PG_MOD Optional Min observation for P3 Module or Product Group
level.
List MAD_DATATYPE Mandatory Data code (dt5667 or dt5675) for MAD
Computation only for standard method.
Dynamic parameters:
Parameter Name Used? (Yes/ No) Comments

# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Global parameters:
Job Cycle Name Value

EHDV21182P 0000 CLUSTER_THRESHOLD 0.3
EHDV21182P 0000 MIN_CLUSTER_OBS 10
EHDV21182P 0000 METHOD STANDARD/TEMPORARY
EHDV21182P 0000 MIN_OBS_BY_CLUSTER 2

P9900246 Submission for Submission for period S Jobs cannot be Submit a
period 0 is not allowed submitted with Period 0 valid period

Page 649 from 698
Message code Description Tag Commands

P0000000 Submission period date Date Range Get Date range for
range submission period.
P1820001 # Records selected for Selected Count T60 records selected.
Process
P1820002 # Records processed Processed Count T60 records processed
P1820003 # Prices Outliers Outliers Detected Outliers' detection
detected
P9900450 Cluster Stats Cluster Statistics have Clusters calculation
been calculated
P1820004 # Record inserted in Price Price Inserted Final data processed
Table

Page 650 from 698
EHDV21183P - Total Value Normalization
Phase:
Phase: 1 – Formal Data Validation
• The proposed methodology will correct prices based on the likelihood (instead of a rule-based
approach) that the HH provided the total value and not the unitary price.
• Validation is applied on validation area (P359_1T12), By filtering the data with anyone of IBD and
Shop scope. Performing the calculation with the formula. If price is out of limits, this job is
automatically updating current price datatype (DT5675), historical price remains the same.
• All changes are logged in FAMILYLOGT67 table and can be displayed using FamilyLog screen.
• The validation area date range to be validated is defined by submission period and number of
periods back.
• IBD_VALUE, IBD_PRICE and SHOP_SCOPE are global parameters and their values have been
set up in TDV_JOB_GLOBAL_PARAMS.
Validation model:
Step 1:
• Get list of nankey to be validated, initialize the reference price as standard price from previous
period .
• Select full list of Nan_key in P359_1T12 for selected audit/period/timestamp and for all relevant
nan_keys, set RefPrice(t):=StdPrice(t-1), here we select standard price in CNANPRICET34 for Price
Group 0 (and not standard price for specific price group / retailer) for previous period.
• For next step split the data for week t into two parts
o Part A: Data filtered by IBD_VALUE and
o Part B: Data filtered by IBD_PRICE
Step 2:
Process Part A [IBD_VALUE]

• For nan_keys with NO RefPrice(t) available with filtered data with IBD_ VALUE.
• Computing missing reference price DT5653 as median of dt6575 for the current period from
P359_1T12.
• Compute new price when units = 1 as set new price = DT5675
• when units > 1
o Check price 1 = ((DT5675 * DT5650) - DT5653) / DT5653
o Check price 2 = (DT5675 - DT5653) / DT5653
o If TESTING_WEIGHT × ABS (Check price 2) ≤ ABS (Check price 1)
▪ set new price = DT5675
▪ Else set new price = DT5675 * DT5650
• Updated data are applied in P359_T12 table, DT5675 is updated with a new price. Insert/update
DT5904= 18 for updated records.
Step 3:
Process Part B [IBD_PRICE]

• For nan_keys with NO RefPrice(t) available with filtered data with IBD_PRICE

Page 651 from 698
• Computing missing reference price DT5653 as median of dt6575 for the current period from
P359_1T12.
• Compute new price when units = 1 as set new price = DT5675
• when units > 1
o Check price 1 = ((DT5675 / DT5650) - DT5653)/ DT5653
o Check price 2 = (DT5675 - DT5653) / (DT5653)
o If TESTING_WEIGHT × ABS (Check price 2) ≤ ABS (Check price 1)
▪ set new price = DT5675
▪ Else set new price = DT5675 / DT5650
• Updated data are applied in P359_T12 table, DT5675 is updated with a new price. Insert/update
DT5904= 18 for updated records.
Final Step:
Report counters
• Report counters for total corrections
• Report counter for selection by IBD
• Report counter for corrections by IBD
Table Description
Files:
Input /Output Files:
No Input /Output files
Parameters:
Static Parameters:
Parameter Type Mandatory Comment

IBD_VALUE SQL N IBD for nankeys collected in value
IBD_PRICE SQL N IBD for nankeys collected in price

SHOP_SCOPE SQL N Shop group for limited scope
Global Parameters:
Parameter Type Value Comment

TESTING_WEIGHT NUMERIC 0.95 Testing weight values
to confirm correction
need
Dynamic Parameters:

Page 652 from 698
Period Yes Submission with period 0 is not
allowed for this job
# Periods Yes
Single Run Job No
Timestamp No
PC Selection No Obsolete feature
F# Selection No Obsolete feature
File Selection No

code
P0020505 Static Value Static Value S Static value is not Enter valid Static
not present not present present in value into
CAL_FACTORT05 table CAL_FACTORT05
table
P9900223 No Records No Records S Records are not selected Check if
Selected for Selected from CNANPRICET34 CNANPRICET34 has
processing table for processing been loaded for
submission
cycle/period
P0020516 Job cannot System S Job cannot be submitted Submit the job with a
be submitted for period 0 valid period
for period 0

P0020608 Static Value Static Value Information message only. No action
required
P9900005 Required No. of Periods not Information message only. No action
Periods not enough required
Present
P1830002 Purchase acts Corrected Record Information message only. No action
selected by IBD Count required
Price and Value
P1830003 Total Records Corrected Record Information message only. No action

corrected Count required
P9900446 ALL IBD IBD aligned Information message only. No action

resolutions are required
behavior.
Sample Calculation:
when units = 1
set new price = DT5675
when units > 1

Page 653 from 698
DT5675 ===> 2.186
DT5650 ===> 1.148
DT5653 ===> 1.99 (Median)
Check price 1 = ((DT5675 / DT5650) - DT5653)/ DT5653
(DT5675 / DT5650) = (2.186 / 1.148) = 1.9041
(DT5675 / DT5650) - DT5653) = (1.9041 -1.99) = -0.08581
(DT5675 / DT5650) - DT5653)/ DT5653 = -0.08581 /1.99 = -0.043125
Check price 1 = -0.043125
--------------------------------------------------------------------------------------------------------------------
Check price 2 = (DT5675 - DT5653) / (DT5653) = (2.186 - 1.99/ 1.99)
Check price 2 =0.09849
--------------------------------------------------------------------------------------------------------------------
TESTING_WEIGHT =0.95
Check price 1 = -0.043125
Check price 2 =0.09849
If condition: TESTING_WEIGHT × ABS (Check price 2) ≤ ABS (Check price 1)

0.95 * 0.09849 <= 0.043125 0.09356 <= 0.043125 [Condition Fails]
else:
set new price = DT5675 / DT5650
= 2.186/1.148 =1.90418118466899
set new price = 1.90418118466899
1.90418118466899 is the value updated in the t12 data value
EHDV21301P – EHDV21309P – FLAT FILE IMDB LOADER

Phase:

Page 654 from 698
This program is updating IMDB product referential tables in NCPS DV. Those tables are used by many other
DV jobs to retrieve Xcoding instructions, product characteristics, to check Nan_key validity…
Each IMDB load is identified by a unique identifier named “Publication number”.
IMDB flat file loader is submitted as a single GROUP and all the different jobs in the GROUP (EHDV21301P
– EHDV21309P) are run in parallel.
Table Description DV jobs used for loading

IMDB_HISTORICAL_EVENT_LOGT10 List of all publications
(Full and Inc) published EHDV21301P
by EIMDB
LAST_PUB_EVENTT30 List of published copy
EHDV21301P
loaded in NCPS DV
IMDB_LANGUAGEST16 List of languages in use
EHDV21301P
in EIMDB
CROSS_CODE_GROUPST26 List of Xcode Group ID
EHDV21302P
for the country
CROSSCODING_INSTRUCTIONST49 Crosscoding links
information extracted
from EIMDB
OR
RIPPLE Xcoding
Instructions. EHDV21302P
In case duplicate keys are

found the output file
‘DUP_KEYS_302P.TXT’ is
created containing all the
external codes which
generate duplicate keys.
X_CODE_GROUPST26A List of Xcode groups for
the country with flag to EHDV21302P
use or not for Xcoding
PRODUCT_GROUPST45 List of Product group EHDV21303P
PRODUCT_GROUPS_TRANST46 Description of product
EHDV21303P
groups
SUPER_GROUPST47 List of super groups EHDV21303P
SUPER_GROUPS_TRANST48 Description of super
EHDV21303P
group
UNITST85 List of units EHDV21303P
UNIT_CONVERSIONST86 EHDV21303P
WEIGHT_CODEST68 List of Weight codes EHDV21303P

Page 655 from 698
WEIGHT_CODES_TRANST69 Description of Weight
EHDV21303P
codes
MODULEST31 List of Modules (link to
Product group, EHDV21303P
SuperGroup
MODULES_TRANST32 Description of modules EHDV21303P
MOD_CTYPE_ASGNST77 List of characteristic
EHDV21303P
assigned to Modules
MOD_WEIGHT_ASGNST78 List of weight codes
EHDV21303P
assigned to Modules
WEIGHT_UNIT_ASGNST80 EHDV21303P
CVALS_IN_USET27 List of characteristic
EHDV21303P
values in use
CHAR_TYPEST17 List of product
characteristic codes from EHDV21304P
EIMDB
CHAR_TYPES_TRANST18 Description of product
characteristic codes from EHDV21304P
EIMDB
CHAR_VALUEST24 List of product
characteristic values EHDV21304P
from EIMDB
CHAR_VALUES_TRANST25 Description of product
characteristic values EHDV21304P
from EIMDB
NAN_VAR_WEIGHT_FROM_IMDB Table containing list of
EHDV21305P
variable weight Nan_Keys
NAN_ITEMST73 List of Nan_keys
extracted from EIMD, link EHDV21305P
to Module
NAN_ITEMS_TRANST74 Description of Nan_keys EHDV21306P
NAN_WEIGHT_VALUEST79 Weight of Nan_keys EHDV21307P
NAN_CHAR_VALUEST75 Nan Characteristic values EHDV21308P
NAN_ITEM_LINKST76 Table containing all
linked Nan_key (eg EHDV21309P
banded packs items)
Recommendations:
It is recommended to update IMDB before any production.
Constraints:
All IMDB tables are unique across audits (one product referential for all audits).
Prerequisites:
The following tables should be initialized at country setup with the following values:
Table: SETUP_PARAM with the following:
DEFAULT_SCHEMA dv_<CC>
COUNTRY <CC>

Page 656 from 698
where <CC> is the country code
Table: T_IMDB_PARAMS with the following:

CHARACTERSET <Oracle Character set for country’s language>
Files:
Input Files:
The input files used for IMDB load should follow the naming convention (as shown in the table below) where
<CC> is the country code in capital letters. For example the input file for Finland (FI) to load the
CHAR_TYPEST17 table should be IMDB_CHAR_TYPES.FI.dat.
Fields in the ASCII files should be tab delimited and records are delimited by CR/LF.
These files should be placed in the country’s default directory.
Table Input File Name

CROSS_CODE_GROUPST26 IMDB_CROSS_CODE_GROUPS.<CC>.dat
CROSSCODING_INSTRUCTIONST49 If FILE_FORMAT=STANDARD then
IMDB_XCD_INSTRUCTIONS.<CC>.dat
Elsif FILE_FORMAT=RIPPLE then
RIPP_XCD_INSTRUCTIONS.<CC>.dat
CHAR_TYPEST17 IMDB_CHAR_TYPES.<CC>.dat
CHAR_TYPES_TRANST18 IMDB_CHAR_TYPES_TRANS.<CC>.dat
CHAR_VALUEST24 IMDB_CHAR_VALUES.<CC>.dat
CHAR_VALUES_TRANST25 IMDB_CHAR_VALUES_TRANS.<CC>.dat
CVALS_IN_USET27 IMDB_CVALS_IN_USE.<CC>.dat
IMDB_HISTORICAL_EVENT_LOGT10 IMDB_HIST_EVENT_LOG.<CC>.dat
IMDB_LANGUAGEST16 IMDB_LANGUAGES.<CC>.dat
MODULEST31 IMDB_MODULES.<CC>.dat
MODULES_TRANST32 IMDB_MODULES_TRANS.<CC>.dat
MOD_CTYPE_ASGNST77 IMDB_MOD_CTYPE_ASGNS.<CC>.dat
MOD_WEIGHT_ASGNST78 IMDB_MOD_WEIGHT_ASGNS.<CC>.dat
NAN_CHAR_VALUEST75 IMDB_NAN_CVALS_CTY.<CC>.dat
NAN_ITEMST73 IMDB_NAN_ITEMS_CTY.<CC>.dat
NAN_ITEMS_TRANST74 IMDB_NAN_ITEMS_TR_CTY.<CC>.dat
NAN_ITEM_LINKST76 IMDB_NAN_LINKS_CTY.<CC>.dat
NAN_WEIGHT_VALUEST79 IMDB_NAN_WTVALS_CTY.<CC>.dat
PRODUCT_GROUPST45 IMDB_PRODUCT_GROUPS.<CC>.dat
PRODUCT_GROUPS_TRANST46 IMDB_PGROUPS_TRANS.<CC>.dat
SUPER_GROUPST47 IMDB_SUPER_GROUPS.<CC>.dat
SUPER_GROUPS_TRANST48 IMDB_SGROUPS_TRANS.<CC>.dat
UNITST85 IMDB_UNITS.<CC>.dat
UNIT_CONVERSIONST86 IMDB_UNIT_CONVERSIONS.<CC>.dat
WEIGHT_CODEST68 IMDB_WEIGHT_CODES.<CC>.dat

Page 657 from 698
WEIGHT_CODES_TRANST69 IMDB_WTCODES_TRANS.<CC>.dat
WEIGHT_UNIT_ASGNST80 IMDB_WEIGHT_UNIT_ASGNS.<CC>.dat
LAST_PUB_EVENTT30 IMDB_LAST_PUB_EVENT.<CC>.dat
NAN_VAR_WEIGHT_FROM_IMDB NAN_VAR_WEIGHT_FROM_IMDB.<CC>.dat
Output Files:
No output file
Parameters:
Static Parameters:

Type Value
PARM GRID0 ZR0 To load or not NAN 0
(This is applicable for EHDV21302P as loading
option)
PARM PARM INCR_MODE Y/N. If Y is selected then no table
TRUNCATION is applied for
o CROSS_CODE_GROUPST26
o CROSSCODING_INSTRUCTIONST49
IF parameter is not defined then the default is

N.
(This is applicable for EHDV21302P as loading

option)
PARM PARM FILE_FORMAT Valid values:
• STANDARD
(file is aligned with IMDB Xcoding
format)
• RIPPLE
(file is aligned with RIPPLE format)
(This is applicable for EHDV21302P as
loading option)
PARM PARM NANKEY_CHAR If FILE_FORMAT is RIPPLE, NANKEY_CHAR
must be present.
(This is applicable for EHDV21302P as

loading option)
Dynamic Parameters

Name (Yes/No)

Page 658 from 698
# Periods No
Single Run Job No
Timestamp No
PC Selection No
F# Selection No
DT Selection No
File Selection No
Rework Management:

code
P0020593 No job parameter found in System Error S
T57
P9900171 Invalid Publication Number Publication S
Present
P9900417 NOT VALID FILE_FORMAT Invalid S
PARAMETER FILE_FORMAT
P0020504 NANKEY_CHAR NOT Invalid S
DEFINED FOR NANKEY_CHAR
FILE_FORMAT RIPPLE

code
P9900173 Country Selection Country Information message only. No action required
Selection
P9900169 Publication Number Publication Information message only. No action required
Number
S0000004 CleanUp Permanent Information message only. No action required
EIMDB Tables
S0000004 Record(s) Deleted Information message only. No action required
Successfully from
<table>
S0000004 Disable Constraints Information message only. No action required
S0000004 Enable Constraints Information message only. No action required
S0000004 Record(s) Inserted Record Information message only. No action required
Successfully in added
<table> success
Surgery Tracking

Page 659 from 698
DP Restatement mode has been implemented to increase the performance. As part of it, there is need to
track the changes applied in the NCPS Data Validation production cycle on 2 dimensions:
- Time (Periods)
- Products (Nankeys)
This will be used in DP process to determine the depth of period to be reprocessed at Datamart
level considering the products involved for Datamart.
As part of DP restatement mode, The changes are to be tracked in DV system at two levels (Period and
Nankey).
There are two kinds of tracking process to achieve the same.
Massive Tracking :
This is performed when the change is applied on the Complete RAWDA on Periods.
Process:
When the FL_TRACKING flag is set to ‘M’ Then,
a)The impacted periods based on the submitted period and NUMPER period are retrieved.
b)Insert a record into new table TDV_ITEM_EVENT_TRACKING with Nankey = 0 for each period
submitted by the run.
Jobs affected : EHDV21000P, EHDV21026P, EHDV21048P, EHDV21075P, EHDV21076P, EHDV21084P,
EHDV21085P, EHDV21091P, EHDV21137P, EHDV21137S, EHDV21143P, EHDV21144P
Item Tracking : This process is performed when the change is limited to few purchases on periods.
Process :
Jobs impacted are as follows:
EHDV21042P
Insert into TDV_ITEM_EVENT_TRACKING table for each period,nankey impacted in RAWDAILYT60.
EHDV21061P – This job require changes at both tracking level, Massive and Item.
When Mode = ‘PUBL’ Then
The impacted periods based on the submitted period and NUMPER period are retrieved.
Insert a record into new table TDV_ITEM_EVENT_TRACKING with Nankey = 0 for each period submitted by
the run.
When Mode = ‘DATE’ Then
Insert into table TDV_ITEM_EVENT_TRACKING using replaced Nankey.
Insert into table TDV_ITEM_EVENT_TRACKING using replacing Nankey.

Page 660 from 698
EHDV21078P
a) Insert into table TDV_ITEM_EVENT_TRACKING using replaced Nankey.
b) Insert into table TDV_ITEM_EVENT_TRACKING using replacing Nankey.
DB Cleaning : New procedure to be implemented to handle the cleaning process in EHDV21048P.
Delete from table TDV_ITEM_EVENT_TRACKING based on the from and to period for the submitted Audit.

Page 661 from 698
Page 662 from 698
Input File Layout Specifications
File Name COLUMN TYPE FORMAT DESCRIPTION NOTES

LATEST
IMDB_HIST_EVENT_LOG.<CC>.dat PUBLICATION
NUMBER
YYYYMMDDHH24MISS The date of
HLG_PUBLICATION_DATE DATE
publication
Publication
HLG_PUBLICATION_NUM NUMBER(38)
Number
Type of Use
HLG_EVENT_TYPE VARCHAR2(1 BYTE) Publication ‘F’
(‘F’ for FULL) only
‘C’ = this is
HLG_CURRENT_FLAG VARCHAR2(1 BYTE) the CURRENT
publication
CHARACTERISTIC
IMDB_CHAR_TYPES.<CC>.dat
TYPE
Characteristic
F_CHR_CODE NUMBER
Code
F_CHR_MNEMONIC VARCHAR2(5 BYTE) Short Mnemonic
YYYYMMDDHH24MISS Last changed
F_CHR_LAST_CHANGED_DATE DATE
date
Type of
characteristic
F_CHR_CATEGORY VARCHAR2(1 BYTE)
‘P’ = PHYSICAL
‘M’ = MARKETING
Can the
characteristic
also have a
numeric value?
F_CHR_NUMERIC_FLAG VARCHAR2(1 BYTE) ‘N’ = non-
numeric
characteristic
‘Y’ = numeric
characteristic
Type of value
F_CHR_VALUES_DISTRIBUTION VARCHAR2(20 BYTE)
distribution

Page 663 from 698
(BINARY,
DISCRETE,
CONTINUOUS)
For internal
Set to
F_CHR_INTERNAL_USE_FLAG VARCHAR2(1 BYTE) use only. ‘Y’ =
‘N’
YES, ‘N’ = NO
F_CHR_COUNTRY VARCHAR2(2 BYTE) Country Code
Characteristic
F_CHR_CODE NUMBER
Code
F_CHR_MNEMONIC VARCHAR2(5 BYTE) Short Mnemonic
F_CHR_LAST_CHANGED_DATE DATE
date
Type of
characteristic
F_CHR_CATEGORY VARCHAR2(1 BYTE)
‘P’ = PHYSICAL
‘M’ = MARKETING
Can the
characteristic
also have a
numeric value?
F_CHR_NUMERIC_FLAG VARCHAR2(1 BYTE) ‘N’ = non-
numeric
characteristic
‘Y’ = numeric
characteristic
Type of value
distribution
F_CHR_VALUES_DISTRIBUTION VARCHAR2(20 BYTE) (BINARY,
DISCRETE,
CONTINUOUS)
For internal
Set to
F_CHR_INTERNAL_USE_FLAG VARCHAR2(1 BYTE) use only. ‘Y’ =
‘N’
YES, ‘N’ = NO
F_CHR_COUNTRY VARCHAR2(2 BYTE) Country Code
CHARACTERISTIC
IMDB_CHAR_TYPES_TRANS.<CC>.dat TYPE
TRANSLATION
Characteristic
F_CHT_CTYPE_CODE NUMBER
Code
F_CHT_LANGUAGE_MNEMONIC VARCHAR2(3 BYTE) Language code

Page 664 from 698
F_CHT_LAST_CHANGED_DATE DATE
date
VARCHAR2(100 Long
F_CHT_DESC
BYTE) description
Short
F_CHT_SHORT_DESC VARCHAR2(30 BYTE)
Description
CHARACTERISTIC
IMDB_CHAR_VALUES.<CC>.dat
VALUE
Characteristic
F_CVA_CODE NUMBER
Value Code
Characteristic
F_CVA_CTYPE_CODE NUMBER
Code
Numeric value
F_CVA_NUMERIC_VALUE NUMBER (where
appropriate)
The unit for
F_CVA_UNIT VARCHAR2(3 BYTE) the numeric
value
F_CVA_LAST_CHANGED_DATE DATE
date
Status of the
vale
F_CVA_DICTIONARY_PROPOSED_
VARCHAR2(1 BYTE) ‘A’ =
FLAG
AUTHORISED
‘P’ = PROPOSED
For internal Set to
F_CVA_INTERNAL_USE_FLAG VARCHAR2(1 BYTE) use only. ‘Y’ = ‘N’
YES, ‘N’ = NO
Exclude this
value from item
F_CVA_EXCLUDE_FROM_HEADING VARCHAR2(1 BYTE) descriptions
‘Y’ = YES, ‘N’
= NO
CHARACTERISTIC
IMDB_CHAR_VALUES_TRANS.<CC>.dat VALUE
TRANSLATION
Characteristic
F_CVT_CVAL_CODE NUMBER
Value Code
F_CVT_CTYPE_CODE NUMBER Characteristic

Page 665 from 698
Code
F_CVT_LANGUAGE_MNEMONIC VARCHAR2(3 BYTE) Language code
F_CVT_LAST_CHANGED_DATE DATE
date
VARCHAR2(100 Long
F_CVT_DESC
BYTE) description
Short
F_CVT_SHORT_DESC VARCHAR2(30 BYTE)
Description
CHARACTRISTIC
IMDB_CVALS_IN_USE.<CC>.dat VALUES IN USE
BY MODULE
F_CVU_MOD_CODE NUMBER Module Code
Characteristic
F_CVU_CTYPE_CODE NUMBER
Value Code
Characteristic
F_CVU_CVAL_CODE NUMBER
Code
Is the Value
Authorised for
the Module
F_CVU_MODULE_PROPOSED_FLAG VARCHAR2(1 BYTE)
‘A’ =
AUTHORISED
‘P’ = PROPOSED
F_CVU_LAST_CHANGED_DATE DATE
date
IMDB_LANGUAGES.<CC>.dat LANGUAGES
F_LNG_MNEMONIC VARCHAR2(3 BYTE) Language code
YYYMMDDHH24MISS Date last
F_LNG_LAST_CHANGED_DATE DATE
changed
Language
F_LNG_DESC VARCHAR2(50 BYTE)
description
Is this the
central
F_LNG_CENTRAL_FLAG VARCHAR2(1 BYTE) language
‘Y’ = YES, ‘N’
= NO

Page 666 from 698
LAST
IMDB_LAST_PUB_EVENT.<CC>.dat PUBLICATION
EVENT
YYYMMDDHH24MISS Publication
F_LPE_PUBLICATION_DATE DATE
Date
Publication
F_LPE_PUBLICATION_NUM NUMBER(38)
Number
Incremental Set to
F_LPE_INCREMENTAL NUMBER(38)
number Zero
IMDB_MODULES.<CC>.dat MODULES
F_MOD_CODE NUMBER Module Code
Product Group
F_MOD_PG_CODE NUMBER
Code
F_MOD_SG_CODE NUMBER Supergroup Code
F_MOD_LAST_CHANGED_DATE DATE
date
Is this a
Banded Pack
F_MOD_BAND_PACK_FLAG VARCHAR2(1 BYTE) Module?
‘Y’ = YES, ‘N’
= NO
Is this a
Product Group
F_MOD_PGI_FLAG VARCHAR2(1 BYTE) Item Module?
‘Y’ = YES, ‘N’
= NO
Are multi-EAN
links allowed
F_MOD_MULTI_EAN_LINK_FLAG VARCHAR2(1 BYTE)
‘Y’ = YES, ‘N’
= NO
Internal use
only?
F_MOD_INTERNAL_USE_FLAG VARCHAR2(1 BYTE)
‘Y’ = YES, ‘N’
= NO
Holds
unidentified
F_MOD_UNIDENTIFIED_ITEMS_F
VARCHAR2(1 BYTE) items?
LAG
‘Y’ = YES, ‘N’
= NO
F_MOD_DEFAULT_PGI_FLAG VARCHAR2(1 BYTE) Default PGI?

Page 667 from 698
‘Y’ = YES, ‘N’
= NO
MODULE
IMDB_MODULES_TRANS.<CC>.dat
TRANSLATIONS
F_MOT_MOD_CODE NUMBER, Module Code
F_MOT_LANGUAGE_MNEMONIC VARCHAR2(3 BYTE) Language code
F_MOT_LAST_CHANGED_DATE DATE
date
VARCHAR2(100 Long
F_MOT_DESC
BYTE) description
Short
F_MOT_SHORT_DESC VARCHAR2(30 BYTE)
Description
CHARACTERISTICS
IMDB_MOD_CTYPE_ASGNS.<CC>.dat ASSIGNED TO
MODULES
F_MCA_MOD_CODE NUMBER Module Code
Characteristic
F_MCA_CTYPE_CODE NUMBER
Type Code
Sequence number
in Common Item
F_MCA_CIG_SEQ_NUMBER NUMBER
Group
generation
YYYYMMDDHH24MISS Last Changed
F_MCA_LAST_CHANGED_DATE DATE
Date
Is the
characteristic
mandatory for
F_MCA_MANDATORY_FLAG VARCHAR2(1 BYTE)
the Module
‘Y’ = YES, ‘N’
= NO
Is the
characteristic
locked for the
F_MCA_LOCKED_FLAG VARCHAR2(1 BYTE)
Module
‘Y’ = YES, ‘N’
= NO
F_MCA_FIELD_COLLECTION_FLA Indicates if
VARCHAR2(1 BYTE)
G the

Page 668 from 698
characteristic
is for field
collection for
the Module
‘I’ = Internal,
‘F’ = Field
WEIGHTS
IMDB_MOD_WEIGHT_ASGNS.<CC>.dat ASSIGNED TO
MODULES
F_MWA_MOD_CODE NUMBER Module Code
F_MWA_WEIGHT_CODE NUMBER Weight Code
F_MWA_LAST_CHANGED_DATE DATE
Date
Is the weight
mandatory for
F_MWA_MANDATORY_FLAG VARCHAR2(1 BYTE) the Module
‘Y’ = YES, ‘N’
= NO
Country
specific NAN
IMDB_NAN_CVALS_CTY.<CC>.dat
Characteristic
Value links
F_NCV_NAN_KEY NUMBER NAN key code
Characteristic
F_NCV_CTYPE_CODE NUMBER
Type Code
Characteristic
F_NCV_CVAL_CODE NUMBER
Value Code
F_NCV_LAST_CHANGED_DATE DATE
date
YYYYMMDDHH24MISS Effective From
F_NCV_EFFECTIVE_FROM DATE
Date
YYYYMMDDHH24MISS Effective To
F_NCV_EFFECTIVE_TO DATE
Date
Publication
F_NCV_PUBLICATION_NUM NUMBER
Number
F_NCV_COUNTRY VARCHAR2(2 BYTE) Country Code
Country Owner
F_NCV_CHAR_COUNTRY VARCHAR2(2 BYTE) for the
Characteristic

Page 669 from 698
Type
Country
IMDB_NAN_ITEMS_CTY.<CC>.dat Specific Nan
Items
F_NAN_KEY NUMBER NAN Key Code
F_NAN_CODE NUMBER NAN Code
F_NAN_MOD_CODE NUMBER Module Code
F_NAN_CREATED_DATE DATE YYYYMMDDHH24MISS Created Date
F_NAN_PROP_LAST_CHANGED_DA YYYYMMDDHH24MISS Property Last
DATE
TE Changed Date
F_NAN_LAST_CHANGED_DATE DATE
date
Nan State.
‘C’ = COMPLETE,
F_NAN_STATE VARCHAR2(1 BYTE) ‘P’ = Physical
Complete, ‘N’ =
None Complete
Banded pack
F_NAN_BANDED_PACK_FLAG VARCHAR2(1 BYTE)
flag
To be deleted
F_NAN_TO_BE_DELETED_FLAG VARCHAR2(1 BYTE)
flag
F_NAN_ON_HOLD_FLAG VARCHAR2(1 BYTE) On hold flag
Pending
F_NAN_PENDING_AUTH_FLAG VARCHAR2(1 BYTE) 670uthorisation
flag
F_NAN_LOCKED_FLAG VARCHAR2(1 BYTE) Locked Flag
Publication
F_NAN_PUBLICATION_NUM NUMBER
number
F_NAN_COUNTRY VARCHAR2(2 BYTE) Country Code
Country
IMDB_NAN_ITEMS_TR_CTY.<CC>.dat Specific NAN
descriptions
F_NAT_NAN_KEY NUMBER NAN Key Code
F_NAT_LANGUAGE_MNEMONIC VARCHAR2(3 BYTE) Language Code
VARCHAR2(1000 First
F_NAT_DESC1
BYTE) description
VARCHAR2(1000 Second
F_NAT_DESC2
BYTE) Description

Page 670 from 698
F_NAT_LAST_CHANGED_DATE DATE
date
Publication
F_NAT_PUBLICATION_NUM NUMBER
Number
F_NAT_COUNTRY VARCHAR2(2 BYTE) Country Code
Country
Specific NAN
IMDB_NAN_LINKS_CTY.<CC>.dat ITEM LINKS
(e.g. for
Banded Packs)
F_NIL_PARENT_NAN_KEY NUMBER NAN Key Code
Child NAN Key
F_NIL_CHILD_NAN_KEY NUMBER
Code
Value
F_NIL_BAND_PACK_VALUE_PCT NUMBER(8,5)
Percentage
Link Type.
‘BPACK_CONSTIT’
= Banded Pack
F_NIL_LINK_TYPE VARCHAR2(20 BYTE) constituent
‘APPORTIONED’ =
Apportioned
Item
F_NIL_LAST_CHANGED_DATE DATE
date
Publication
F_NIL_PUBLICATION_NUM NUMBER
Number
F_NIL_COUNTRY VARCHAR2(2 BYTE) Country Code
Volume
F_NIL_BAND_PACK_VOL_PCT NUMBER
Percentage
Country
IMDB_NAN_WTVALS_CTY.<CC>.dat Specific NAN
WEIGHT LINKS
F_NWV_NAN_KEY NUMBER NAN key code
F_NWV_WEIGHT_CODE NUMBER Weight Code
F_NWV_UNIT_MNEMONIC VARCHAR2(3 BYTE) Unit
F_NWV_NUMERIC_VALUE NUMBER(11,5) Weight value
F_NWV_LAST_CHANGED_DATE DATE
date
F_NWV_EFFECTIVE_FROM DATE YYYYMMDDHH24MISS Effective from

Page 671 from 698
F_NWV_EFFECTIVE_TO DATE YYYYMMDDHH24MISS Effective to
Publication
F_NWV_PUBLICATION_NUM NUMBER
Number
F_NWV_COUNTRY VARCHAR2(2 BYTE) Country Code
IMDB_PRODUCT_GROUPS.<CC>.dat PRODUCT GROUP

Product Group
F_PGR_CODE NUMBER
code
F_PGR_SG_CODE NUMBER Supergroup code
F_PGR_LAST_CHANGED_DATE DATE
Date
Banded Pack
Flag.
F_PGR_BAND_PACK_FLAG VARCHAR2(1 BYTE)
‘Y’ = YES, ‘N’
= NO
PRODUCT GROUP
IMDB_PGROUPS_TRANS.<CC>.dat
TRANSLATIONS
Product Group
F_PGT_PG_CODE NUMBER
code
F_PGT_LANGUAGE_MNEMONIC VARCHAR2(3 BYTE) Language code
F_PGT_LAST_CHANGED_DATE DATE
date
VARCHAR2(100 Long
F_PGT_DESC
BYTE) description
Short
F_PGT_SHORT_DESC VARCHAR2(30 BYTE)
description
IMDB_SUPER_GROUPS.<CC>.dat SUPERGROUP
F_SGR_CODE NUMBER Supergroup code
F_SGR_LAST_CHANGED_DATE DATE
Date
Banded Pack
Flag.
F_SGR_BAND_PACK_FLAG VARCHAR2(1 BYTE)
‘Y’ = YES, ‘N’
= NO
SUPERGROUP
IMDB_SGROUPS_TRANS.<CC>.dat
TRANSLATIONS
F_SGT_SG_CODE NUMBER Supergroup code

Page 672 from 698
F_SGT_LANGUAGE_MNEMONIC VARCHAR2(3 BYTE) Language code
F_SGT_LAST_CHANGED_DATE DATE
date
VARCHAR2(100 Long
F_SGT_DESC
BYTE) description
Short
F_SGT_SHORT_DESC VARCHAR2(30 BYTE)
description
IMDB_UNITS.<CC>.dat WEIGHT UNITS

Unit code (e.g.
F_UNI_MNEMONIC VARCHAR2(3 BYTE) ‘KG’ for
kilograms)
Unit
F_UNI_DESC VARCHAR2(25 BYTE)
Description
F_UNI_LAST_CHANGED_DATE DATE
date
UNIT
IMDB_UNIT_CONVERSIONS.<CC>.dat
CONVERSIONS
F_UNC_BASE_MNEMONIC VARCHAR2(3 BYTE) Base unit
F_UNC_DERIVED_MNEMONIC VARCHAR2(3 BYTE) Target unit
YYYYMMDDHH24MISS Date last
F_UNC_LAST_CHANGED_DATE DATE
changed
Factor to
convert from
the Base to the
Target unit.
Conversion is
done by
F_UNC_CONVERSION_FACTOR NUMBER(15,7)
multiplication,
so if the Base
is KG and the
Target is G
then the factor
is 1000.
IMDB_WEIGHT_CODES.<CC>.dat WEIGHT CODES

F_WTC_CODE NUMBER Weight Code
F_WTC_LAST_CHANGED_DATE DATE
date

Page 673 from 698
UNITS USED WITH
IMDB_WEIGHT_UNIT_ASGNS.<CC>.dat
EACH WEIGHT
F_WUA_WEIGHT_CODE NUMBER 674lignm Code
F_WUA_UNIT_MNEMONIC VARCHAR2(3 BYTE) Unit Code
F_WUA_LAST_CHANGED_DATE DATE
date
WEIGHT CODE
IMDB_WTCODES_TRANS.<CC>.dat TRANSLATIONS
(DESCRIPTIONS)
F_WTT_WEIGHT_CODE NUMBER Weight Code
F_WTT_LANGUAGE_MNEMONIC VARCHAR2(3 BYTE) Language code
F_WTT_LAST_CHANGED_DATE DATE
date
Weight
F_WTT_DESC VARCHAR2(60 BYTE)
Description
IMDB_XCD_INSTRUCTIONS.<CC>.dat CROSS-CODINGS
XCODE
F_XCI_XCODE_KEY NUMBER
identifier
Cross-code
F_XCI_XCODE_GROUP_CODE VARCHAR2(20 BYTE)
Group code
External Item
F_XCI_EXTERNAL_CODE VARCHAR2(30 BYTE)
ID
F_XCI_EXTERNAL_CODE_SUFFIX VARCHAR2(10 BYTE) Suffix
Code type, e.g.
F_XCI_CODE_TYPE_MNEMONIC VARCHAR2(8 BYTE)
‘EAN’, ‘LAC’
F_XCI_NAN_KEY NUMBER NAN key code
F_XCI_NAN_CODE NUMBER NAN code
F_XCI_MOD_CODE NUMBER Module Code
Product Group
F_XCI_PG_CODE NUMBER
Code
YYYYMMDDHH24MISS Effective from
F_XCI_EFFECTIVE_FROM DATE
date
YYYYMMDDHH24MISS Effective to
F_XCI_EFFECTIVE_TO DATE
date
YYYYMMDDHH24MISS
F_XCI_CREATION_DATE DATE Creation date
F_XCI_LAST_CHANGED_DATE DATE
date

Page 674 from 698
Current cross-
F_XCI_CURRENT_REC VARCHAR2(1 BYTE) code flag
‘C’ = CURRENT
Publication
F_XCI_PUBLICATION_NUM NUMBER
Number
F_XCI_COUNTRY VARCHAR2(2 BYTE) Country Code
IMDB_CROSS_CODE_GROUPS.<CC>.dat CROSS CODE

GROUP
F_XCG_CODE VARCHAR2(20 BYTE) Group Code
Group
F_XCG_DESC VARCHAR2(50 BYTE)
Description
F_XCG_COUNTRY_MNEMONIC VARCHAR2(3 BYTE) Country Code
F_XCG_LAST_CHANGED_DATE DATE
date
NAN_VAR_WEIGHT_FROM_IMDB.<CC>.d
at
F_NAN_KEY NUMBER

Page 675 from 698
Document Revision History:
Version Date Description of modification Author
1.0 28-Sept-2006 Initial version N Cousin

V Ganeshan
1.1 19-Dec-2006 43P : New static parameter N Cousin
41P : New static parameter and logic description
70P : New static parameter for 70P to load PGI
07P : New output + New static parameter MIS_GR_ID
76P : exclusion of PGI
54P : add of missing static parameter PRG0
Add of DT table in Appendix
Add File Id of jobs Input/Output files
New job added : 24P – Trip Data Validation
New jobs added : 65L/65U/65M/65P - Housekeeping
1.2 12-Feb-2007 15P : new static parameter, step 4 N Cousin

54P : new static parameter
39P : new static parameter
Add of list of values for qty & price correction flags
in appendix
1.3 9-Marc-2007 75P : implementation of rounding logic N Cousin

70P : new static parameter to manage load of variable
weight item table
74P : change in price validity date range (now driven
by submission period)
New job added : 35P – Family Monitor Loaded

Page 676 from 698
1.4 10-Apr-2007 15P : implementation of step 1 bis and 2 bis, control of N Cousin
Ratio = (step3 + Step 4) / total number of records
41P : addition of filtering by file of
HHOLD/SHOP/NAN/BARCODE
1.5 26-Jun-2007 76P : new static parameter to manage scope of N Cousin

Price Group / Barcode Type
New job added : 23P – Trip Data 677lignment
New job added : 17P – Basket Validation
New job added : 60P – Shop weight Computation
New job added : 28P – Weekly family usability
computation
01P : new output file (list of deleted keys in ALLT11)
42P : new static parameter to manage option to not
load purchase act in case TBSKT record is not existing
1.6 28-Aug-2007 02P : add of new output file to report 677anellist with more N Cousin
than 7 days of holidays or holiday start date lower than
date from of submission period
New job added : 82P – Extract from ALLT11
1.7 07-Nov-2007 39P : correction of output file layout N Cousin

43P : addition of filtering by file of
HHOLD/SHOP/NAN/BARCODE
New job added : 84P – Validation volume of variable
weight items
1.8 12-Dec-2007 75P: management of 2 Group Id N Cousin

New job added : 85P- Heavy Contributor Edit (projected
Data)

Page 677 from 698
1.9 17-Mar-2008 New job added : 86P – Copy cluster price N Cousin
HH GEO/SH CHAIN
1.10 4-July-2008 New job added : 88P – Xcoding on barcode list N Cousin
New job added : 89P – Intended User Alignment
New job added : 90P – Total Spent Validation
61P: change in counters organisation
68P : change control of basket key already present in
output table
23P & 24P : recommendation about purchase keys &
Tbskt keys alignment for Tbskt fact computation (DP09)
1.11 29-Sept-2008 New job added : 91P – Assortment Validation N Cousin

Add description for 83P – Extract Family Monitor
60P: change to use IBD
1.12 27-nov-2008 New job added : 92P – DV KPI computation and N Cousin
extraction
New job added : 93P – Extract for 678anellist compliance
84P: change to include additional high volume validation
in case of few observations
1.13 13-Fev-2009 75P : implementation of IBD N Cousin

85P : implementation of IBD
28P : new output files
83P : new static parameters, multiperiod logic
25P : user manual description added

Page 678 from 698
1.14 18-May-2009 37P : change to populate shop / Xcoding group link N Cousin
based on shop char
94P : new version of basket validation based on IBD
34P/36P/37 : option to load Sample DB data for HH and
SH by dblink
92P : KPI figures stored in tables and can be accessed
by screen
1.15 28-Sept-2009 91P : new step added (check on familylog) N Cousin

94P : some additional details on 94P (output file,
counters descriptions)
73P : period 0 active
01P : check output table becomes optional
89P : management of tbskt
24P : change description of v004
34P/37P : new option ACTIVE
1.16 20-Dec-2009 35P : update description of logic for cycle & period N Cousin
95P : new job : price/total value validation
96P : new job: extraction for HD items split
90P : update description of model for DT5941 validation
1.17 13-Feb-2010 01P : new static parameter PRICE_INC N Cousin

97P : new job to populate T99 from CPS data (UK),
with multiple sources
55P : management of several sources
86P : management of several sources
50P : new option to load price from IMDB
91P : management of multi chain
98P : new job :fresh food only basket validation
56P : new static parameter SOURCE

Page 679 from 698
1.18 20-Aug-2010 99P : new job for fresh food barcode transformation N Cousin
70P : change to manage PGI by IBD
93P : new fields in output file (records in TALL)
74P : management of duplicated shops
1.19 06-Dec-2010 120P: new job Purchase File Extraction to CASE for
LATAM
121P: new job UPC File Extraction to CASE for LATAM
122P: new job Trip File Extraction to CASE for LATAM
24P : changes related to DT5950, DT5676, DT5951
Add new DTs for Latam in DT list
1.20 21-Mar-2011 23P: change related to management of empty basket

Records.
123P: new job Additional Trip Data Facts.
Add 100 new DTs
1.21 27-May-2011 61P: Removed incorrect information

78P: Removed incorrect information
92P: Added new KPI computations.
123P: Added Formula Table, TRIP_FORMULAST118,
information.
1.22 10-Oct-2011 131P: Formal Telecom Price Validation

132P: Validate Telecom Prices against Catalogue prices
133P: Automatic Telecom Price Validation
134P: Median Telecom Price computation
301P – 309P – Flat File IMDB Loader
1.23 02-Dec-2011 20P: Addition of Priv.Lbl Description to messagelog.

New Static Parameter: Language.
1.24 12-Jan-2012 135P: Data Selection by exception
(based on barcode list)

Page 680 from 698
1.25 06-Mar-2012 133P: Add method parameters values list
1.26 15-May-2012 136P: Family Usability Extraction from Sample DB
1.27 08-Jun-2012 DT5126 filter for 043P,120P,121P,122P,123P Grifith Mathew
DT5090 filter(validated purchase act extraction)
1.28 19-Jun-2012 00P: New Backdata Loader Document
1.29 15-Oct-2012 138P:New job for the Private brand validation and New job for the Private brand validation
Grifith
andMathew
extraction
1.30 18-Jan-2013 CCM 3794346 – 123P Sophocles Savvides
CCM 4003336 – 123P
CCM 4009116 - 91P
CCM 4119274 - 00P
CCM 4134573 - 00P
CCM 3837801 - 50P
1.31 21-Feb-2013 JJ6797 – 76P APV Sophocles Savvides
JJ6798 – 137P Volume Validation for Variable Weight
(NEW)
1.32 05-Apr-2013 JJ8553 – 139P Beverages Pack Detection Sophocles Savvides
1.33 05-Jun-2013 CCM 4449903 Merge Duplicate Records Sophocles Savvides
1.34 29-Jul-2013 CCM 4773137 Incremental option for Xcoding tables Sophocles Savvides
1.35 04-09-2013 JJ8550 – 140P Table ALL Cleaning Sophocles Savvides
1.36 06-12-2013 141P – COMPUTATION OF EAN WEIGHT, Sophocles Savvides
FR HARD DISCOUNTERS
1.37 12-03-2014 Java Replacement Sophocles Savvides
1.38 19-06-2014 CCM 5599345 – EIMDB loader. Sophocles Savvides
Allow Reloading same PUB number in Full mode
1.39 17-09-2014 Describe separately: Sophocles Savvides
137P-Variable Weight Volume Validation / IMDB Max
137S-Variable Weight Volume Validation / Computed
Max

Page 681 from 698
1.40 28-11-2014 Rawda Copy Mercy Joseph
1.41 28-11-2014 EHDV21073P – Item Activity Sophocles Savvides
1.42 03-12-2014 EHDV21138P – Private Brand Validation Sophocles Savvides
Version=2
New parameter addition – BCD_SOURCE
Static Parameter extra information added and
pre-requisite for shop_group and IBD.
1.45 22-Jan-2015 Rawda Copy Gomathinayagam.B
Validations Handled when BCD_SOURCE is NANKEY
1.46 25-Feb-2015 Added the Script details for the Jobs in Appendix for DV Gomathinayagam.B
automation simplification
1.47 13-Mar-15 Production Flow – Process Flow update Gomathinayagam.B
1.48 13-Mar-15 JJ 13786 – EHDV21142P Generic Shop Edit Sophocles Savvides
JJ 15734 – EHDV21148P External Price Loader
1.49 10-Apr-15 Added the Changes related to DP Restatement - Gomathinayagam.B
Surgery Tracking
1.50 14-Apr-15 RIPPLE Xcoding Instructions Load Sophocles Savvides
1.51 30-June-15 INC00535543 – EHDV21076P Use NUMPER dynamic Sophocles Savvides
parameter instead of static parameter
RITM0387615 – EHDV21075P prevent from 0 division

Page 682 from 698
1.52 12-Sept-15 DV8.0 Sophocles Savvides
MND007 – EHDV21002P DV Split Log Table
PAC 2015-002 – EHDV21145P DV RMS Average Feed
for Non Cooperator
DMND 0002066 – EHDV21139P DV Beverages Pack
Detection Version 2
1.53 04-Nov-15 137P&S – Changes made to accommodate AU special

Needs
302P - Export Output file with external codes which
Create duplicate keys
145P – Enclose all fields in the export file in double
quotes
1.54 15-Febr-16 146P – NCE Extraction for SirVal
INC013013027 – EHDV21002P Holiday handling
1.55 26-Oct-16 147P – Cycle price copy Gomathinayagam.B
1.56 26-Oct-16 149P – Fresh Food recovery tool Gomathinayagam.B
1.57 02-May-17 DMND 21084 – EHDV21150P – US Data Loader Sophocles Savvides
1.58 27-June-17 DMND 21116 – EHDV21152P - US Data Edit Sophocles Savvides
DMND 21081 – EHDV21001P – Multiple Audit management
1.59 14-Sept-17 DMND 21077 – EHDV21154P – US xCoding Shrink Sophocles Savvides

- EHDV21155P – US xCoding
- EHDV21152P – US Data Edit (using new US xCoding logic)
1.60 07-Nov-17 153P – Duplicate Transmission Detection Vemal – On Behalf of

Gomathinayagam B
1.61 08-Mar-18 149P – DMND 21267 – DV Fresh Food Improvement Sophocles Savvides
156P – DMND 21392 – DV US Item Reclassification
150P – DMND 21398 – DV USA Data Feed V2 (Handling of Magnet Purchase Data)
1.62 12-Mar-18 157P – DMND 21397 – DV File Merger Hemanth Kumar J

Page 683 from 698
1.63 05-April-18 DMND 21466 – EHDV21150P – US Data Feed V3 (DT5130 – Total Coupon value) Sophocles Savvides
– EHDV21159P – US Threshold Computation
1.64 02-May-18 DMND 21406 – EHDV21158P – Brand Exception Sophocles Savvides
1.65 08-May-18 INC03825548 – 92P Manual updations Annamalai Srinivasan
1.66 21-May-18 TDV_PRODUCTION_FLOW_PARAMS addition in production flow Hemanth Kumar J
1.67 22-May-18 DMND 21083 – EHDV2092P – New KPI for Mobile Tracking Sophocles Savvides
1.68 01-Jun-18 INC03818775 – EHDV21144P – Cycle Rawda Copy is failing : Data duplication issue Annamalai Srinivasan
1.69 28-Jun-18 DMND 21529 ZTV CPS – Price Phase 1 (EHDV21039P & EHDV21040P) Sophocles Savvides
DMND 21532 ZTV CPS – Uncoded Extraction (EHDV21160P)
DMND 21533 ZTV CPS – PGI Xcoding by Default (EHDV21161P)
1.70 30-July-18 DMND 21532 ZTV CPS –Uncoded Extraction Phase 2 & Cross Coding job Hemanth Kumar J
(EHDV21008P)
DMND 21533 ZTV CPS – PGI Xcoding by Default Phase 2 (EHDV21161P)
1.71 30-July-18 DMND 21621 Annamalai Srinivasan
ZTV CPS – DV ReXcoding Jobs Improvement(EHDV21061P & EHDV21078P)
1.72 27-Aug-2018 DMND-21531-Data selection by barcode 135p Hemanth Kumar J
1.73 25-Sep-2018 DMND 21621-ZTV CPS – DV ReXcoding Jobs Improvement –V3.1 Hemanth Kumar J
DMND 21717-ZTV CPS – DV Assortment Validation
1.74 25-Sep-2018 DMND 21530 ZTV CPS – Price Phase 3 & Phase 4 Annamalai Srinivasan
1.75 12-Oct-2018 DMND 21532 ZTV CPS –Uncoded Extraction Version 3- NB_PERIOD_LOAD_AREA Hemanth Kumar J
1.76 20-Oct-2018 DMND 21530 ZTV CPS – Price Phase 2 (EHDV21162P) Sophocles Savvides
DMND 21554 ZTV CPS – DV New Outlier Validation Module (EHDV21166P)
1.77 11-Nov-2018 DMND 21788 LAC Exclusion Hemanth Kumar J
1.78 19-Nov-2018 DMND 21531 – PGI Data Selection (EHDV21163P) Sophocles Savvides
DMND 21535 – Loyalty Card Usage Rate Computation(EHDV21164P)
CA Data Loader (EHDV21167P) and CA data edit (EHDV21168P)
1.79 20-Nov-2018 DMND 21775 ZTV CPS – DV Basket validation report improvement Aswathi Sivaraman

Page 684 from 698
1.80 20-Nov-2018 DMND 21793 ZTV CPS – DV improvement job 37P Annamalai Srinivasan
DMND 21776 ZTV CPS – DV new product alert (EHDV21165P)
1.81 08-Jan-2019 DMND 21532 ZTV CPS –Uncoded Extraction Version 4 Hemanth Kumar J
DMND 21621 ZTV CPS –Rexcoding Improvement TDV REX LOG
1.82 10-Jan-2019 DMND 21722 – PRIVATE BRAND IMPROVEMENT (EHDV21138P) Sophocles Savvides
INC04123654 Improvement EHDV21138P to avoid duplicate basket keys
INC03992648 – EHDV21158P empty baskets fix
INC04126317 – EHDV21075P Improvement in HBE job for the missing CUP values
1.83 02-Feb-2019 DMND 21531- APV Rerun 135P & 163P changes Hemanth Kumar J
INC04473015 – 78P nankey not set to zero for purchase barcodes
1.84 11-Feb-2019 DMND 21944 – Recover Simulation Module (EHDV21088P) Sophocles Savvides
DMND 21531 – Validation of ReXcoding & Recovering – APV Rerun Mode
(EHDV21076P modification)
1.85 11-Feb-2019 DMND 21408 – Uncoded Tracking (EHDV21169P) Aswathi Sivaraman
DMND 21403 – DV Report Sign-off ,Message alignment changes in
02P,15P,75P,137P,137S,18P,138P&139P
1.86 13-Mar-2019 DMND 21944 – ZTV CPS – Recover Simulation Module – improvements Hemanth Kumar J
DMND 21776 – Product alert phase 2
INC04538712- COPY CYCLE CONTROL –Destination cycle different from source
cycle
INC04899663 – Prockeys rework mode and high period count
1.87 02-Apr-2019 DMND 22063 – Implement a new interface to send SI01 information from CPS to Hemanth Kumar J
OGRDS(160P)
DMND 21829 – Mercury Project – DV surgery automation and tracking – Cleaning of
final tables before reloading(48P)
1.87 03-Apr-2019 DMND 21944 – Recover Simulation Module (EHDV21088P) Sophocles Savvides
DMND 21557 – APV Low Price Validation (EHDV21076P)
1.88 09-Apr-2019 DMND 22021 – ZTV CPS – Improve Copy Audit tool mode(EHDV21144P) Annamalai Srinivasan
DMND 22145 – Mercury project – Item Activity enhancement (EHDV21073P)
1.89 10-May-2019 DMND 21410 – ZTV – Improvement Aswathi and Sophocles

DMND 21557 – New price computation

Page 685 from 698
1.90 31-May-2019 DMND 21531- DV Validation of ReXcoding & Recovering Price Management Aswathi
1.91 31-Jul-2019 DMND 21531- DV Validation of ReXcoding & Recovering Price Management Aswathi
Price computation changes in T100 and T99 .Change in 22p,26p,76p and 88p
1.92 31-Jul-2019 DM22194 – Fix start / end date of item activity Hemanth Kumar J
DM 22174 – First set of jobs (00P, 07P, 08P, 15P, 41P, 42P, 43P) Annamalai Srinivasan
1.93 27-Aug-2019 INC05134170 -144P fix for stats gather Hemanth Kumar J
OGRDS – step2 – other jobs ( 61P, 78P, 20P, 88P, 135P) Annamalai Srinivasan
1.94 25-Sep-2019 OGRDS – step2 – other jobs ( 61P, 78P, 20P, 88P, 135P) Hemanth Kumar J
Tracking message updation – 01p,149p,158p,94p Annamalai Srinivasan
1.95 23-Oct-2019 54P/55P & 86P : Parameter alignment & tracking changes Hemanth Kumar J
Annamalai Srinivasan
1.96 07-Jan-2020 DMND 22291 – ZTV CPS – Price Load using OGRDS char Rohith Shenoy
DMND 22716 – ZTV – 78P Rexcoding by Shop Group Hemanth Kumar J
1.97 06-Feb-2020 DMND 22681 – Data Validation (DV) US reXcoding on barcode list (156P) Hemanth Kumar J
DMND 22439 – New FMCG Units datatype computation
DMND 22744 – New Std Price and MAD Computation – FPV
INC05543291 – 171P – wrong failure message preventing from using NORMAL
mode
TASK3085470 and INC05570771 – Backdata Loader Improve Counters
1.98 02-Mar-2020 170P changes related to INC05717970 Hemanth Kumar J
FMCG Phase 2 US and CA Impacts
1.99 30-Mar-2020 DMND 22699 – DV OGRDS Alignment (EHDV21070P) Sophocles Savvides
88P job for the OGRDS mode by using dt_update but not dt_created for the selection Annamalai Srinivasan
of Xcoding instructions
EHDV21171P – Period length explanation update
DMND 22717 – Assortment Items Loader (EHDV21177P)
INC05776978 – SOURCE3 catalog alignment for 86P

Page 686 from 698
2.00 28-Apr-2020 DMND 22783 – Amazon Project – Handling ASIN code in DV xCoding Sophocles Savvides
INC05768057 – APV Validation failure for single record – SG Hemanth Kumar J
INC05753525 – DV 01P data consistency fix Annamalai Srinivasan
INC05888511 – DT5953 Change in T07 Rohith Shenoy
INC05757238 – 142P Append parameter
DMND 22784 – DV Beverages Validation improvement QH1
INC05842306 – 170P optional parameter handling with proper error note
2.01 21-May-2020 OGRDS Incremental Mode Removal Sophocles Savvides
2.02 22-May-2020 158P New Output file – DMND 22290 – ZTV – DV Brand Exception new report Rohith Shenoy
2.03 19-June-2020 070P- DMND 22793 – DV Amazon ASIN code Handling Sophocles Savvides
070P – DMND 22861- OGRDS Headings alignment
DV data Types free for Finland Receipt Capture
2.04 20-July-2020 158P New Output file 2 – DMND 22290 – ZTV – DV Brand Exception new report – Hemanth Kumar J
QH3
2.05 20-July-2020 DMND22721 – DV Basket change tracking Rohith Shenoy
2.06 12-Oct-2020 INC05378010 – user manual change for 66P reference to T43 removed Rohith Shenoy
DMND 23987 – Shop resolution changes Hemanth Kumar J
2.07 09-Nov-2020 DMND 23860 – PCP Latam – Distribution Factor calculation Hemanth Kumar J
2.08 16-Nov-2020 DMND 22769 – ZTV – Manual Panel Late Transmission Hemanth Kumar J
2.09 16-Jan-2021 DMND 24054 – DV HBE Mercury Enhancement Hemanth Kumar J
2.10 18-Jan-2021 RITM5049655 – TASK5053178 – FPV exclusion value=20 Rohith Shenoy

Page 687 from 698
2.11 18-Jan-2021 DMND 24106 – ZTV – Prod Flow Tracking for CUOE modules Annamalai Srinivasan
DMND 24127 – Mercury – Surgery Enhancement for Hhold and Shops checks de-
activation
DMND 24063 – Hermes – Ticket Capture JobID handling – Phase 2
2.12 18-Jan-2021 INC10037488 – PRIVATE_LABEL_CHARS references removed and replaced with Rohith Shenoy
TDV_PB_ITEM_CHARS – 138P
2.13 22-Jun-2021 DMND 24308 – DV HBE Automation – Production flow changes alignment Hemanth Kumar J
2.14 04-Aug-2021 July & Aug content alignment which includes 79P doc addition, 75P, 169, 171P, 41P, Vemal M
48P, 92P, 93P, 26P Hemanth Kumar J
Rohith Shenoy
Dhanush
Maheswaran S
Maheswaran P
Reshma Koshy
Ismail Babu
2.15 01-Sep-2021 DV2109 alignment for 175P tech data alignment and 162P new key addition in Rajesh Gupta
setup_param Dhanush
2.16 29-Sep-2021 DV2110 alignment for 01P, 142P, 152P, 140P, 150P, 168P, 144P, 153P and 73P Hemanth Kumar J
Dhanush
Maheswaran S
Ismail Babu
2.17 26-Oct-2021 175P Process description updates Hemanth Kumar J

Page 688 from 698
2.18 24-Nov-2021 178P Addition Hemanth Kumar J
INC10634081 – EHDV21023P and EHDV21024P Updates Sophocles Savvides
DMND 24728 – Refactoring Jobs – EHDV21167P Maheswaran S
DMND 24932 – Sirval File Automatic Pick-up Rohith Shenoy
2.19 08-Dec-2021 26P enhancement to push parquet files to DP via AZcopy Reshma Koshy
Vemal Manoharan
2.20 19-Dec-2021 DMND 24728 – Refactoring Jobs – 135P, 00P and 68P Reshma Koshy
Ismail Babu
2.20 16-Feb- 2022 DMND 24728 : Refactoring Jobs - EHDV21001P. Dhanush
2.21 DMND 25172 : Refactoring Jobs - EHDV21098P. Mahes Periyasamy
2.22 28-Feb-2022 92P,169P,150P,152P,73P and 22P aligned with refactoring changes Dhanush,Reshma,Mahes,
Rohith
2.23 12-Apr-2022 EHDV21062P and EHDV21063P addition Sophocles Savvides
Refactored jobs aligned with latest changes. Maheswaran, Ismail ,Reshma
,Rahul
2.24 11-May-2022 EHDV21089P Reshma
DMND 25086 - OmniShopper - Receipt ID handling in NCPS. Sangeetha
2.25 26-May-2022 EHDV21023P and EHDV21123P refactoring changes Reshma
Venkatesh
2.26 22-June -2022 EHDV21085P and EHDV21090P refactoring changes Ismail
Rahul
2.27 26-July -2022 INC10921123 and INC10921130 - 154P and 156P User manual update Hemanth
2.28 09-Sept-2022 DMND 25145 - EHDV21139P -Beverage Multipack Detection and EHDV21048P Cleaning Celcia
DMND 25146 - EHDV21183P-NCPS DV Normalization (replacing job 95P) - Phase 1 Sangeetha
DMND 25148 - EHDV21181P-new method to validate prices in NCPS DV – APV Rahul
DMND 25147 - EHDV21182P- Price and MAD Computation Standard Venkatesh
2.29 09-Nov-2022 RITM5499571 - EHDV21139P – User manual update Sangeetha
RITM5504337 – TASK5576040 - EHDV21182sP – Phase change updation Venkatesh
DBUREF–3489 – 73p – Hive to oracle source change manual update Reshma

Page 689 from 698
2.30 23-Nov-2022 EHDV21092P - Hive to Oracle Source Change Celcia
EHDV21182P - Hive to Oracle Source Change Hema Sai Teja
EHDV21139P - Hive to Oracle Source Change Ezhilarasan
EHDV21183P - hive to Oracle Source Change Ismail
INC11097839 - 181P Ticket - Rework Failing in Flag Lock
Sangeetha

Page 690 from 698
Appendix
List of output tables datatypes
DT CODE DESCRIPTION DESTINATION LENGTH DECIMAL
4111 Bonus pack adjustment factor RAWDAILYT60 16 5

5017 Any Promo TBSKT13 10 0
5071 Day of the week TBSKT13 10 0
5082 currency type TBSKT13 10 0
5083 shop code - source of price RAWDAILYT60 10 0
5084 Receipt ID TBSKT13 10 0
5086 Family code for intended user RAWDAILYT60 10 0
5087 % units distribution for intended user RAWDAILYT60 16 5
5089 HCE adjustment factor RAWDAILYT60 16 5
5090 Lock Flag PROCKEYST29 10 0
5095 File Number PROCKEYST29 10 0
5096 HH usability flag PROCKEYST29 10 0
5098 Bar Code Type RAWDAILYT60 10 0
5099 HBE adjustment factor RAWDAILYT60 16 5
5122 Promotion 5 RAWDAILYT60 10 0
5126 Flag for Basket deletion PROCKEYST29 10 0
5127 Basket classification TBSKT13 10 0
5128 Total value spent by set of products TBSKT13 16 5
5129 Fresh Food Original barcode RAWDAILYT60 15 0
5130 QTY/price in Receipt Capture RAWDAILYT60 15 5
5131 Consolidated Item Weight RAWDAILYT60 13 3
5132 Where consumed RAWDAILYT60 15 5
5642 Purchase Units historical RAWDAILYT60 16 5
5650 Purchase Units RAWDAILYT60 16 5

Page 691 from 698
5666 # Units Purchased TBSKT13 16 5
5667 Price historical RAWDAILYT60 16 5
5675 Price RAWDAILYT60 16 5
5676 Number of Different Items TBSKT13 10 0
5693 internet order TBSKT13 10 0
5694 Home delivery code TBSKT13 10 0
5695 Points Voucher Used TBSKT13 10 0
5696 RECORD TYPE - UK RAWDAILYT60 10 0
5698 COUNTRY ORIGIN OF PRODUCT - UK RAWDAILYT60 10 0
5748 Total Value Spent on Promo- by hh TBSKT13 16 5
5749 Outlet Value on Promo TBSKT13 16 5
5755 Total Outlet value TBSKT13 16 5
5756 # Transactions on Deal TBSKT13 10 0
5757 Activity Deal 1 TBSKT13 10 0
5904 Price correction flag (1) RAWDAILYT60 10 0
5939 Units correction flag (2) RAWDAILYT60 10 0
5941 Total Shopping Expenses by hh TBSKT13 16 5
5942 Total cupon value by hh TBSKT13 16 5
5943 Payment Type TBSKT13 10 0
5944 Primary Shopper Sex TBSKT13 14 0
5945 Primary Shopper Year Born TBSKT13 10 0
5946 Secondary Shopper Sex TBSKT13 10 0
5947 Secondary Shopper Year Born TBSKT13 10 0
5948 Number of Detail Records from HHT TBSKT13 10 0
5949 Fidelity Card TBSKT13 10 0
5950 Total number of items collected computed TBSKT13 10 0
5951 Number of units purchased on deal/promo TBSKT13 10 0
5952 Latam personal or family consumption RAWDAILYT60 10 0
5953 Latam distribution factor RAWDAILYT60 16 5
5954 Latam special question 1 RAWDAILYT60 10 0
5955 Latam special question 2 RAWDAILYT60 10 0

Page 692 from 698
5956 Latam Telecom Spend type RAWDAILYT60 10 0
5957 Latam Telecom Who paid RAWDAILYT60 10 0
5958 Latam Telecom Reload type RAWDAILYT60 10 0
5800 Additional Trip Data fact #01 TBSKT13 16 5

Page 693 from 698

Page 694 from 698
5888 Initial Purchase Date (Shifted) TBSKT13 16 5
5889 Initial Collection Period TBSKT13 16 5
5892 Historical Total Shop Expenses TBSKT13 16 5
5893 Receipt Capture Trip TBSKT13 16 5
5897 Registration ID(Mobile) TBSKT13 16 5
5898 Total Gross TBSKT13 16 5
5899 Total Discount TBSKT13 16 5

Page 695 from 698
(1) : possible values for DT 5904 are
1 : price copied by FPV (15P), step 1

4 : price copied by copy NRSP price (53P), case of missing price
5 : price copied by copy NRSP price (53P), case of price out of MIN/MAX limit
6 : price copied by Copy Median Price CLUSTER (56P), price of cluster Chain/Geo
7 : price copied by Copy Median Price CLUSTER (56P), price of cluster Chain/0
8 : price copied by Copy Opus Price (63P)
9 : price copied by Supplier price process (51P)
10 : price corrected by On Line correction screen
12 : price copied by FPV (15P), step 1 bis (std price NAN/price group on past periods)
13 : price copied by FPV (15P), step 2 bis (std price NAN/price group 0 on past periods)
14[SOURCE] : price copied by Copy Median Price CLUSTER HH GEO, price of cluster Chain/Geo current period, price from source [SOURCE]
15[SOURCE] : price copied by Copy Median Price CLUSTER HH GEO, price of cluster Chain/Geo previous periods, price from source [SOURCE]
16[SOURCE]: price copied by Copy Median Price CLUSTER HH GEO, price of cluster Chain/0 current period, price from source [SOURCE]
17[SOURCE]: price copied by Copy Median Price CLUSTER HH GEO, price of cluster Chain/0 previous periods, price from source [SOURCE]
18: price corrected by price/total validation validation (95P)
19: price corrected by Volume Validation for Variable Weight (137P)
20: price copied by FPV (15P), step 5
21: price corrected by TELECOM APV (133P)
22: price corrected by TELECOM APV (133P)
(2) : possible values for DT 5939 are
0: No correction
1 : quantity corrected by multipack validation (18P)
2 : quantity corrected by HBE (75P)
4 : quantity corrected by On line correction screen
6 : record inserted by Drug Estimation
7: quantity corrected by CUOE (173P)

Page 696 from 698
10 : quantity corrected by Variable Weight Volume validation (84P) or (137P)
20 : quantity corrected by HCE (85P)
List of Common Scripts used in Jobs
Job Name Script Used Description

/CPS/ncps/@int_env@/ncpsdv/script/no_action_2.cmd Dummy Script to get the automation completed even if no
EHPK_EHDV21007P transfer is required
/CPS/ncps/d/@int_env@/ncpsdv/script/rms_price_file_merger.cmd Script to perform the merging of input files
EHPK_EHDV21157P

/CPS/ncps/@int_env@/ncpsdv/script/script_73P.cmd script for Extract Active Item job (73P)
/CPS/ncps/@int_env@/ncpsdv/script/script_79P_IU.cmd Script sending LFC extraction file Fidelity Cards

EHPK_EHDV21079P

Page 697 from 698
/CPS/ncps/@int_env@/ncpsdv/script/script_79P_SMP.cmd Script sending LFC extraction file Fidelity Cards

transfer is required
/CPS/ncps/@int_env@/ncpsdv/script/script_83P.cmd script for Family Monitor Job (83P)
EHPK_EHDV21083P

/CPS/ncps/@int_env@/ncpsdv/script/script_93P.cmd Script to transfer FMCG file to logistic db server China
EHPK_EHDV21093P


Page 698 from 698

NCPS DV Reference Manual

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

NCPS DV Reference Manual

Uploaded by

Copyright:

Available Formats

Consumer Insights

Reference Manual for Backend Jobs

Document Revision History:

© 2022 The Nielsen Company NCPS DV Reference Manual

EHDV21000P- Backdata Loader ................................................................................................................................................................................................. 5

© 2022 The Nielsen Company NCPS DV Reference Manual

© 2022 The Nielsen Company NCPS DV Reference Manual

© 2022 The Nielsen Company NCPS DV Reference Manual

The IMDB table (nan_itemst73)

© 2022 The Nielsen Company NCPS DV Reference Manual

© 2022 The Nielsen Company NCPS DV Reference Manual

Rawda file structure:

Position Length Name Description

TBSK file structure:

Position Length Name Description

© 2022 The Nielsen Company NCPS DV Reference Manual

Parameter Type Mode Description

• When set to ON, data type values coming from

• Valid values are

Default value if parameter is not defined is OFF

Valid values are

Default value if parameter is not defined is Y

Default value if parameter is not defined is OFF

This parameter is used only when the job’s process mode

By setting the parameter to ON only the specific modified

© 2022 The Nielsen Company NCPS DV Reference Manual

Default value if parameter is not defined is OFF

This parameter is used only when the job’s process mode

By setting the parameter to ON only the specific modified

When set to ON, household values in the ASCII file are

Valid values are

Default value if parameter is not defined is ON

Valid values are

Default value if parameter is not defined is ON

Valid values are

Default value if parameter is not defined is ON

© 2022 The Nielsen Company NCPS DV Reference Manual

Parameter Used? Comments

Proc.TYPE CYCLE PERIOD option

Other Screen Parameters:

© 2022 The Nielsen Company NCPS DV Reference Manual

Message Description Tag Type Explanation Action

© 2022 The Nielsen Company NCPS DV Reference Manual

Message code Description Tag Comments

© 2022 The Nielsen Company NCPS DV Reference Manual

© 2022 The Nielsen Company NCPS DV Reference Manual

PHASE 0 – INPUT AREA

Parquet Refresh [Before the job]

Oracle Refresh [End of the job]

Oracle Tables Involved :

© 2022 The Nielsen Company NCPS DV Reference Manual

Purchase file (File id : FILEI1)

Purchase data record layout :

Basket data record layout :

Positio Length Name Description

© 2022 The Nielsen Company NCPS DV Reference Manual

© 2022 The Nielsen Company NCPS DV Reference Manual

Output Files (File id : FILEO1) :

Positio Length Name Description

TICKET file (File id : FILEI3)

Parameter Type Value Name (example) Description

© 2022 The Nielsen Company NCPS DV Reference Manual

Target is to check the presence of Gross and

If PRICE_INC=AVG, merge record will have a price computed as sum(pricexqty)/sum(qty)