IMPORT SETS- Import sets allow administrators to import data from various data sources, and then map

that data
into ServiceNow tables.

Import Set table(Staging Table) -A table that acts as a staging location for records imported from a data
source before transforming those records. Fields on these tables are generated automatically based on imported data
and should not be modified manually.

Data source-A record that defines where to get the data to import. A data source may point to a file, a JDBC-compatible
database, or an LDAP organizational unit.

Transformation--The conversion of data from an import set table to another table according to the rules defined in a
transform map.

Transform map-A set of field maps that define the relationships between fields in an import set and fields on a table,
such as Incident.

During transformation, data is copied from the Import Set table to the destination table based on the transform map.

A single import set field may be mapped to multiple fields on other tables.

Foreign record insert A foreign record insert occurs when an import makes a change to a table that is not the target
table for that import.
This happens when updating a reference field on a table. For example, when updating a value for the caller on an
incident, the import is actually updating the sys_user table.

Coalesce-Lets you update existing target table records when transforming import data.

You can specify whether a Target field should be used to coalesce on when import set records are transformed. If the
field map Coalesce check box is selected, when the import set row is transformed, the instance checks for an existing
record in the target table that has the same value in the Target field as the import set row Source field.

If an existing record with a matching value in the target table is found, that record is updated. If no matching record is
found, then a new record is created in the target table.

External data sources

Data Source Limitations

JDBC JDBC imports require a valid transform map. Some network configurations may require a MID Server.

LDAP LDAP imports require a valid transform map.

OIDC OIDC imports require a valid transform map.

REST (IntegrationHub) REST imports require a valid transform map.

Custom (Load by Script) Custom imports require a valid transform map

Import log table- import_log_list

Data Source List- sys_data_source_list

Enforce Mandatory Fields

ServiceNow does not force import sets to provide values for all mandatory fields on a table. The Enforce mandatory
fields option on a Transform Map determines if mandatory fields are required to have values when importing data.
The Import Sets table acts as a staging area for records imported from a data source.
Note: You should not import data in extremely large chunks. Creating an extremely large import set can
cause delays and system outages.
A transform map determines the relationships between fields displaying in an Import Set table and
fields in an existing ServiceNow table, such as the Incidents or Users table.
Importing sets skip records when the data in the instance matches the data being imported.
Note: Import Sets run as user System and therefore cannot add data to encrypted fields.

Terminology

Key terms

Term Definition

Import Set table A table that acts as a staging location for records imported from a data source before transfor
those records. Fields on these tables are generated automatically based on imported data and
not be modified manually.
Data source A record that defines where to get the data to import. A data source may point to a file, a JDB
compatible database, or an LDAP organizational unit.
Transformation The conversion of data from an import set table to another table according to the rules defined
transform map.
Transform map A set of field maps that define the relationships between fields in an import set and fields on
such as Incident.
During transformation, data is copied from the Import Set table to the destination table based
transform map.
A single import set field may be mapped to multiple fields on other tables.

Robust import set

transformer
Foreign record insert A foreign record insert occurs when an import makes a change to a table
that is not the target table for that import.
This happens when updating a reference field on a table. For example,
when updating a value for the caller on an incident, the import is actually
updating the sys_user table.

Coalesce Lets you update existing target table records when transforming import data.
You can specify whether a Target field should be used to coalesce on when
import set records are transformed. If the field map Coalesce check box is selected,
when the import set row is transformed, the instance checks for an existing record
in the target table that has the same value in the Target field as the import set row
Source field.
If an existing record with a matching value in the target table is found, that record is
updated. If no matching record is found, then a new record is created in the target table.

Supported import formats

You can import data from several different file formats or external data sources.
File formats
Format Limitations
CSV CSV files must comply with the proposed CSV standard in RFC4180. In particular,
this means that double quotes may not appear inside fields. The first row of data in
an imported CSV file becomes the header row and defines the columns for that import set.
Excel Excel files must have the XLS or XLSX extension.
XML XML files must have a consistent XPath for each data row.
JSON  For JSON arrays, the path for each row must specify the array root element twice, such
as /incidents/incidents.
 JSON files should follow RFC-4627. For example, a single comma should separate a
 value from the following name. Names within an object should be unique.
 Predicates such as @element, [index], or text(), as well as Axis such as children,
 siblings, or ancestors are not supported.
Custom (Parse by You can write custom Javascript to parse input lines into the list of maps.
Script)

ServiceNow can import data from the following external data sources.

External data sources

Data Source Limitations
JDBC JDBC imports require a valid transform map. Some network configurations may require a
Server.

LDAP LDAP imports require a valid transform map.

OIDC OIDC imports require a valid transform map.

REST (IntegrationHub) REST imports require a valid transform map.

Custom imports require a valid transform map.

Custom (Load by
Script)

Transform maps versus robust import set transformers

Both transform maps and robust import set transformers convert data. Transform maps convert data from
the staging table to the destination file. Robust import set transformers convert data from a source table to a
 message listener. If a data source has both a transform map and a robust import set transformer, the system
uses the robust import set transformer instead of the transform map.

Import set scheduled cleanup

The Import Set Deleter scheduled job cleans up old import set data every day at midnight.
By default, the job deletes all import sets that are older than 7 days, as well as the associated import set table
records. If this scheduled job is not active on your instance, activate it to mitigate the growing import set
tables you may have, especially when using recurring imports.
Note: If scheduled cleanup is not currently active on your system, you must ensure that your import set
tables are not too big (> 5 million records total across all tables) before enabling this cleanup. If the tables are
too big, manually delete the records first before proceeding. Please contact ServiceNow Technical Support to
coordinate the deletion of this data.

Import set table deleter

You can manually delete any or all of the following import set tables:
 Computer
 User
 Notification
 Location
You can also delete the related transform maps, or you can delete data only and preserve the table
structure.
 Create a transform map
Every import operation to a production table requires at least one transform map associated with an
import set.

About this task

The transform map specifies the data relationships between the import set and the target table. For
every transformation, you must either create a new transform map or select an existing one.
Procedure
1. Navigate to System Import Sets > Create Transform Map.
2. Complete the form.

Table transform map fields

Field Description
Name User-friendly label to identify the transform map.

Source table Import table containing the raw import set data. An import table is any table that extends the
Import Set Row [sys_import_set_row] table. You can select only tables within the
currently selected application scope.

Active Check box to make the transform map available for use.

Run business rules Check box to run business rules, workflows, approval engines, auditing, and field normalization
while the transformation inserts or updates data into the target table. Clearing this check box
runs GlideRecord.setWorkflow() with a value of false.
Enforce mandatory Selects whether to enforce mandatory fields on the target table:
fields o No: Do not enforce mandatory fields.
o Only Mapped Fields: Enforce mandatory mapped fields only.
o All Fields: Enforce all mandatory fields.

Copy empty fields Check box to copy empty field values and override existing target field values.

Create new record Check box to create a new record when all coalesce fields are empty, instead of ignoring the
on empty coalesce record or overwriting an existing record.
fields If Coalesce empty fields is selected for any field in the record, the record is coalesced.

Application Application scope: global or per application.

Created Transform map creation date (populated automatically).

Target table The table in which you want the transformed data to be inserted. You can select only tables
within the currently selected application scope or the global scope, or tables that grant write
access to other applications.

Order Order in which to apply transform maps if more than one map fits the conditions. The system
runs transform maps from lowest to highest Order.

Run script Check box to define a script that transforms field values in the source table to the target table.

Script Transform map script that transforms field values in the source table to the target table. The
system runs the transform map script in addition to any field maps.

Transform Event Scripts

Transformation events occur during the process of transforming an import set table onto a target
table. Transformation Event Scripts modify the transformation behavior at different points in the
transformation process.

To create a Transform Event Script, switch to the Transform Scripts related list in a Transform Map
and click the New button.
The When option in the Transform Script trigger specifies when in the transformation process the
script should run.

The When field choices are:

 onStart: executes at the start of an import before any rows are read
 onAfter: executes at the end of a row transformation and after the source row has been
transformed into the target row and saved
 onBefore: executes at the start of a row transformation and before the row is transformed into
the target row
 onChoiceCreate: executes at the start of a choice value creation before the new choice value is
created
 onComplete: executes at the end of an import after all rows are read and transformed
 onForeignInsert: executes at the start of the creation of a related, referenced record before the
record is created
 onReject: executes during foreign record or choice creation if the foreign record or choice is
rejected. The entire transformation row is not saved.
Transformation script variables :

 source : Contains the import source record currently being transformed. Specify a specific field from the
source record as an object property.
 target : Contains the import target record currently being inserted. Specify a specific field from the target
record as an object property.
 map : Contains the transformation map record currently being used for the transformation process. Specify
a specific field from the transform map record with one of these properties.
 log : Log information about the current import process. Each log level has its own method.
 action : Specify the transformation action occurring on the target record. This value can be either "insert" or
"update".
 ignore : When set to true, skips or aborts the current import action. In onStart scripts, this variable aborts
the entire transformation process. In onBefore scripts, this variable only skips the current row being
transformed.
 error : When set to true, aborts the current import action and logs an error message in the Import Set Log.
 error_message :When an error occurs, adds the specified error message to SOAP response.
 status_message : Adds the specified status message to SOAP response.

For example, when coalescing, the requirement might be to update only records that match and not
insert any new records. To meet the update records only requirement requires an onBefore Transform
Script. The onBefore script executes after ServiceNow has determined whether there is a matching
record in the target table and before the insert happens.

The action string variable is automatically created. It has two possible values: insert and update. The action
variable is set after coalescing determines if there is a match (update) or no match (insert).
The ignore boolean variable is automatically created. When true, the ignore variable stops the
transformation process for the source data row.
transform events and their contextual variables

Event name Event Parameters

onStart When: The onStart event script is processed at the start of an import run,
before any data rows are read.

Import Set JS objects available to be referenced and their context in

the onStart event
Import Set Type Context in the onStart import set
JS object event
source GlideRecord The first row of the source table,
there is no data yet since the row
has not been read.
import_set GlideRecord The import set that is currently
being transformed.
map GlideTransformMap Read-only information about the
current transform map record.
log Function The log object for the current
import run. For example,
log.info(...), log.warn(...),
log.error(...).
ignore Boolean When set to true, the entire
transformation will be stopped
and no further processing will
occur.
error Boolean When set to true, has the same
effect as the ignore flag of
stopping the entire
transformation, with an error
message.
Example:
var info = "Starting Import Set Transformation Mapping operation using Data source, "
+ map.table_source + " and Transform Map " + map.name + " on import set " + import_set.number;
log.info( info );

onComplete When: The onComplete event script is processed at the end of an import run, after all data rows are read
and transformed.

Import Set JS objects available to be referenced and their context in

the onComplete event
Import Set JS Type Context in the onComplete
object import set event
source GlideRecord The last row of the source import
set table.
target GlideRecord The last row of target table.
import_set GlideRecord The import set that is currently
being transformed.
 map GlideTransformMap Read-only information about the
current transform map record.
log Function The log object for the current
import run. For example,
log.info(...), log.warn(...),
log.error(...).
error Boolean When set to true, will mark the
current import set status to error
after it completes.
Example:
// Create a myimport_completed event that can be reacted by an email notification or script action
// (there is already an import.finished event that the system will create at the end of an import)
var e = new GlideEvent("myimport_completed", import_set.sys_id, map.sys_id, "");
e.insert();

onBefore When: The onBefore event script is processed at the start of a row transformation, before the source row
is transformed into the target row.

Import Set JS objects available to be referenced and their context in

the onBefore event
Import Set JS Type Context in the onBefore
object import set event
source GlideRecord The row of the source table
that is currently being
processed.
target GlideRecord The row of the target table
that is currently being
processed.
import_set GlideRecord The import set that is
currently being transformed.
map GlideTransformMap Read-only information about
the current transform map
record.
log Function The log object for the current
import run. For example,
log.info(...), log.warn(...),
log.error(...).
action String Action returns a value of
either "insert" or "update"
indicating whether the
current target row is about to
be created or updated.
ignore Boolean When set to true, the current
row transformation will be
skipped and the remaining
 rows will be processed.
If you have defined an onAfter
script, the onAfter script runs
for the current row even if the
ignore flag is set to true.

status_message String Defines a custom message to

be sent in the
<status_message> XML
response.
error Boolean When set to true, will halt the
entire transformation for the
current import set, with an
error message.
error_message String Defines a custom message to
be sent in the
<error_message> XML
response.
Example:
var name = source.u_name.toString();
var info = "Before the row is transformed, " + name;
log.info( info );

// Make sure a company name has been provided

var company = source.u_company.toString();
if(company.length == 0 ){
ignore = true;
info ="No company name, row ignored! " + name;
log.info( info );
}
onAfter When: The onAfter event script is processed at the end of a row transformation, after the source row
has been transformed into the target row and saved.

Import Set JS objects available to be referenced and their context in

the onAfter event
Import Set JS Type Context in the onAfter import
object set event
source GlideRecord The row of the source table
that is currently being
processed.
target GlideRecord The row of the target table
that is currently being
processed.
import_set GlideRecord The import set that is
currently being transformed.
map GlideTransformMap Read-only information about
the current transform map
record.
log Function The log object for the current
import run. For example,
log.info(...), log.warn(...),
log.error(...).
action String Action returns a value of
either "insert" or "update"
indicating whether the
current target row was
created or updated.
status_message String Defines a custom message to
be sent in the
<status_message> XML
response.
error Boolean When set to true, will halt the
entire transformation for the
current import set, with an
error message.
error_message String Defines a custom message to
be sent in the
<error_message> XML
response.
 Example:
if(source.new=="true"){
gs.include('Cart');var bundle =new GlideRecord('sc_cat_item');
bundle.addQuery('name','CONTAINS','comp');
bundle.query();
if(bundle.next()){
var new_comp_cart = new Cart();
var cart = new Cart();
var item = cart.addItem(bundle);
var rc = cart.placeOrder();
log.info(rc.number);}}

When: The onForeignInsert event script is processed at the start of the creation of a related,
referenced record, before the record is created.
onForeignIns
ert
Import Set JS objects available to be referenced and their context in
the onForeignInsert event
Import Type Context in the onForeignInsert
Set JS import set event
object
source GlideRecord The row of the source table that is
currently being processed.
target GlideRecord The row of the target table that is
currently being processed.
map GlideTransformMap Read-only information about the
current transform map record.
log Function The log object for the current import
run. For example, log.info(...),
log.warn(...), log.error(...).
action String Action returns a value of either
"insert" or "update" indicating
whether the current target row is
about to be created or updated.
name String Evaluates to the field name of the
target record for which a foreign
record that is about to be created.
value String Evaluates to the display value from
the source record for which a foreign
record is about to be created.
action String Action returns a value of either
"insert" or "update" indicating
 whether the current target row is
about to be created or updated.
ignore Boolean When set to true, ignores the
creation of a new foreign or related
record.
error Boolean When set to true, rejects the entire
transformation row, no data is saved
for this row.
Example:
//Create an event.

var e = new GlideEvent("myimport_ForeignInsert", action, name, "");

e.insert();

onChoiceCreate When: The onChoiceCreate event script is processed at the start of a choice value creation,
before the new choice value is created.

Import Set JS objects available to be referenced and their context in

the onChoiceCreate event
Import Type Context in the onChoiceCreate
Set JS import set event
object
source GlideRecord The row of the source table that is
currently being processed.
target GlideRecord The row of the target table that is
currently being processed.
map GlideTransformMap Read-only information about the
current transform map record.
log Function The log object for the current import
run. For example, log.info(...),
log.warn(...), log.error(...).
action String Action returns a value of either
"insert" or "update" indicating
whether the current target row is
about to be created or updated.
name String Evaluates to the field name of the
target record for which a choice
value is about to be created.
value String Evaluates to the display value from
the source record for which a choice
is about to be created.
ignore Boolean When set to true, ignores the
creation of a choice value.
error Boolean When set to true, rejects the entire
 transformation row, no data is saved
for this row.
Example:
//Create an event
var e = new GlideEvent("myimport_ChoiceCreate", action, value, "");
e.insert();

onReject When: The onReject event script is processed during the occurrence of a foreign record or choice
creation, and the foreign record or choice is rejected ,the entire transformation row is not saved.

Import Set JS objects available to be referenced and their context in

the onError event
Import Type Context in the onError import set
Set JS event
object
source GlideRecord The row of the source table that is
currently being processed.
target GlideRecord The row of the target table that is
currently being processed.
map GlideTransformMap Read-only information about the
current transform map record.
action String Action returns a value of either
"insert" or "update" indicating
whether the current target row is
about to be created or updated.
log Function The log object for the current import
run. For example, log.info(...),
log.warn(...), log.error(...).

Example:
//Create an event
var e = new GlideEvent("myimport_onReject", action, "", "");
e.insert();

Map with transformation event scripts

Transformation events occur during the process of transforming an import set table onto a table.
These events modify the transformation behavior from any type of mapping specification. Transformation
Event scripts modify the processing of the events at various stages of the transformation.
For example, the processing of a mapping operation defined using the Mapping Assist Utility can be
manipulated using the event scripts. There are a number of import set JavaScript objects that are accessible
during these events. These objects represent tables or portions of tables. It is important to note that what
these objects refer to varies depending on the context of the event in which they are referenced.

The following is a table of all of the transform events and their contextual variables:
Transformation script variables

Multiple variables can be used to define explicit mapping relationships in a transform map script.

Variable name: source

Type: GlideRecord object

Description: Contains the import source record currently being transformed. Specify a specific

field from the source record as an object property.

Example:

var x = source.incident_state;

Variable name: target

Type: GlideRecord object

Description: Contains the import target record currently being inserted. Specify a specific field

from the target record as an object property.

Example:

target.incident_state = "active";

Variable name: map

Type: GlideRecord object

Description: Contains the transformation map record currently being used for the transformation

process. Specify a specific field from the transform map record with one of these properties.

 name
 sys_id
 source_table
 target_table
 order
 Example:

var x = map.order;

Variable name: log

Type: Function

Description: Log information about the current import process. Each log level has its own

method.

Example:

log.info("This is an information message");

log.warn("This is a warning message");

log.error("This is an error message");

Variable name: action

Type: Function

Description: Specify the transformation action occurring on the target record. This value can be

either "insert" or "update".

Example:

if(action =="insert"){

ignore = true;

Variable name: ignore

 Type: Boolean

Description: When set to true, skips or aborts the current import action. In onStart scripts, this

variable aborts the entire transformation process. In onBefore scripts, this variable only skips the

current row being transformed.

Example:

if(source.u_user_name.nil()){

ignore = true;

Variable name: error

Type: Boolean

Description: When set to true, aborts the current import action and logs an error message in the

Import Set Log.

Example:

if(source.name=="no_tranform"){

error = true;

Variable name: error_message

Type: String (output message)

Description: When an error occurs, adds the specified error message to SOAP response.

Example:

if(source.name=="no_tranform"){

error = true;

error_message = "Source is not intended for transformation";

Variable name: status_message

 Type: String (output message)

Description: Adds the specified status message to SOAP response.

Example:

if(action =="insert"){

status_message = "Inserting record";

Script:
There are two types of Transform Map scripts:
 Explicit Transform Map scripts, which explicitly define mapping relationships

 Transformation Event scripts, which modify the processing of events at different stages of a transformation

Explicit
An explicit Transform Map script explicitly defines mapping relationships in the Transform Map record itself. It runs in
the context of "after" the source field values have been copied over to the target record, and before they are written to
the database.

Event Scripts
 The onStart event script is processed at the start of an import run, before any data rows are read.
 The onComplete event script is processed at the end of an import run, after all data rows are read and
transformed.
 The onBefore event script is processed at the start of a row transformation, before the source row is
transformed into the target row.
 The onAfter event script is processed at the end of a row transformation, after the source row has been
transformed into the target row and saved.
 The onForeignInsert event script is processed at the start of the creation of a related, referenced record,
before the record is created.
 The onChoiceCreate event script is processed at the start of a choice value creation, before the new choice
value is created.
 The onReject event script is processed during the occurrence of a foreign record or choice creation, and the
foreign record or choice is rejected, the entire transformation row is not saved.

JSON vs XML
Both JSON and XML can be used to receive data from a web server.

The following JSON and XML examples both define an employee’s object, with an array of 3
employees:

JSON Example
{"employees":[
{ "firstName":"John", "lastName":"Doe" },
{ "firstName":"Anna", "lastName":"Smith" },
{ "firstName":"Peter", "lastName":"Jones" }
]}

XML Example
<employees>
<employee>
<firstName>John</firstName> <lastName>Doe</lastName>
 </employee>
<employee>
<firstName>Anna</firstName> <lastName>Smith</lastName>
</employee>
<employee>
<firstName>Peter</firstName> <lastName>Jones</lastName>
</employee>
</employees>

JSON is Like XML Because

 Both JSON and XML are "self describing" (human readable)
 Both JSON and XML are hierarchical (values within values)
 Both JSON and XML can be parsed and used by lots of programming languages
 Both JSON and XML can be fetched with an XMLHttpRequest

JSON is Unlike XML Because

 JSON doesn't use end tag.
 JSON is shorter.
 JSON is quicker to read and write.
 JSON can use arrays.

The biggest difference is:

XML has to be parsed with an XML parser. JSON can be parsed by a standard JavaScript
function.

Why JSON is Better Than XML

XML is much more difficult to parse than JSON.
JSON is parsed into a ready-to-use JavaScript object.

For AJAX applications, JSON is faster and easier than XML:

Using XML

 Fetch an XML document

 Use the XML DOM to loop through the document
 Extract values and store in variables

Using JSON

 Fetch a JSON string

 JSON.Parse the JSON string