2006/02/23

Oracle 10g Data Pump Say good bye to exp and imp (or not)!
Simon Pane

Agenda

Data Pump Overview Using Data Pump Demonstration Data Pump Test Cases

Data Pump Overview

What is Data Pump?

A replacement of the traditional export/import utilities?

The evolution of the traditional export/import utilities?

A completely new 10g utility serving a similar yet slightly different purpose?

Other Options for Moving Data

Traditional Export and Import

Pros
Easy

to use most DBAs have years of experience using these utilities various options available; can specify what to include independent

Versatile Platform Serial

output slow / resumable

Cons
Comparatively Can

be network intensive filtering options (for example, can exclude just VIEWS) remapping options (i.e. from one tablespace to another)
5

Non-interruptible Limited Limited

Other Options for Moving Data

Transportable Tablespaces
Pros
Undoubtedly Can

the fastest way to move data support if the platform byte-order is the same

use the traditional exp/imp or Data Pump to move meta-data

Cross-platform

Cons
Tablespaces
Not

must be made read-only

selective (must move the entire tablespace) is not possible (tablespace is read only when copied) physical reorganization is performed

Flashback No

Datafile
Must

sizes remain constant

use RMAN to convert the datafile if migrating to a platform with a different byte-order (check V$TRANSPORTABLE_PLATFORM)
6

Other Options Used Less Frequently

Extraction to a flat file and loading using SQL Loader Direct copy using database links (SQL Plus COPY command) Oracle Streams 3rd Party data ETL or reorg tools

Top 10 Reasons to Love DataPump

10.
9. 8. 7. 6. 5.

Similar look and feel to the old exp/imp

Can filter on the full range of object types Can re-map datafiles and or tablespaces on import

Estimates the export file size (space needed)

Parallelizable Significantly faster than the traditional exp/imp

4.
3. 2. 1.

PL/SQL Interface programmable

A file is not actually required - can import through a network link Track in v$session_longops Resumable (interruptible and restartable)
8

Top 10 Reasons Not to Love Data Pump

10.

Still generates redo (unlike direct path inserts)

9.
8. 7.

Aggregation of exported data is not possible (sort only)

Performance on the server Harder to tell what its doing at any given time

6.
5.

No equivalent to the STATISTICS option

Cannot be used with sequential media such as tapes and pipes (not read/written serially)

4. 3. 2. 1.

Only accesses files on the server, never the client

Oracle directories are required in the DB to access the files Does not support COMMIT on imp or CONSISTENT on exp

If constraints are violated on import, the load is discontinued

9

Operation Fundamentals

Export/Import
These utilities would basically connect to the Oracle database

via Oracle NET and run queries or DDL/DML

Processing of returned results and I/O operations were done on

the client

Data Pump
The executables call PL/SQL APIs Therefore processing is done on the database server This can be an advantage or a disadvantage depending on the

situation
Self-Tuning: no longer need to use BUFFER or RECORDSET

10

Export Operation

exp.exe Network

Oracle Database

Export File(s)

11

Data Pump Export Operation

expdp.exe Network

Oracle Database

Export File(s)

12

Key Differences

Dump and log files are on the server, not the client Must have a DIRECTORY created in the Oracle database for I/O
Permissions for the userid connecting to the instance, not the

schemas being exported or imported

Canceling the client process does not stop the job

Doesnt automatically overwrite dump file if it already exists returns an error instead
Parameters (command line) are reported in the log file

Exported objects order based on table size (descending) instead of alphabetically

13

Multiple Interfaces
1.

Command line utilities expdb and impdb

Similar to the familiar exp and imp in usage Use HELP=Y for a list of commands Oracle documentation provides a comparison table to exp/imp

2. 3.

Enterprise Manager PL/SQL

Can be used independently but is difficult

All of these call the DBMS_DATAPUMP API

Uses Oracle Advanced Queuing
Uses DBMS_METADATA
14

Unload Mechanisms

Data Pump automatically chooses to unload data either using:

Direct path External Tables (new driver called ORACLE_DATAPUMP)

Same External Tables mechanism that was introduced in Oracle9i

When will it use External tables:

When parallelism can be used When the table contains a complex data type or structure that

prevents direct path unloads

A

lot of tables fall under this situation see Oracle documentation for a complete list

It doesnt really matter to us which method is used

15

Multiple Processes

Master Control Process

Spawns worker processes Populates the master control table and log file The master control table can be queried to track the jobs

process
At the end of an export, the master control table is written to the

dump file and dropped from the database

Worker Processes
Performs the loading/unloading Number of processes depends on the degree of parallelism

(the PARALLEL option)

16

Detaching and Re-Attaching

Issuing Ctrl-C from the data pump import will detach

Import is running on the server so it will continue Brings you into interactive-command mode

To re-attach, run impdp with the ATTACH= option

Example: impdp userid=system/oracle attach=JOB_01

Brings you back into interactive-command mode

17

New Views

DBA_DATAPUMP_JOBS and USER_DATABASE_JOBS

Identify all jobs regardless of their state Identify any master tables not associated with an active job

DBA_DATAPUMP_SESSIONS
Identify user sessions that are attached to a job

Data pump sessions populate v$session_longops

Documentation says that it is 100% accurate for imports but

testing proves otherwise!!!

18

Security Considerations

Still uses the EXP_FULL_DATABASE and IMP_FULL_DATABASE A privileged user will have these two roles A privileged user can:
Export/import objects owned by other schemas Export non-schema objects (metadata) Attach to, monitor, and control jobs initiated by others Perform schema, datafile, and tablespace remapping

Similar to the traditional export/import

Supports label security

If exporting user has the EXEMPT ACCESS POLICY role
19

Object Statistics

From Oracle documentation regarding data pump exports:

A parameter comparable to STATISTICS is not needed.

Statistics are always saved for tables.

From Oracle documentation regarding data pump imports:

A parameter comparable to STATISTICS is not needed. If the

source table has statistics, they are imported.

20

Other Random Points

Can still use a parameter file and the PARFILE command line option Fully supports Automatic Storage Management (ASM) Can still flashback to a specified time or SCN Can still extract (or backup) DDL (meta data)
Using the SQLFILE option instead of the traditional

INDEXFILE or SHOW options

Full support of LOBS

21

Using Data Pump

22

Oracle Directory Objects

Must first create an Oracle directory object and give the user who will be performing the Data Pump activities permission to use it (or rely on defaults):
SQL> create or replace directory dpump_demo as 'C:\temp'; Directory created. SQL> grant read,write on directory dpump_demo to simon; Grant succeeded. SQL>

23

Key Data Pump Export Parameters

CONTENT={ALL | DATA_ONLY | METADATA_ONLY} DIRECTORY=directory_object (default=DATA_PUMP_DIR) DUMPFILE=[directory_object:]file_name [,] ESTIMATE={BLOCKS | STATISTICS} ESTIMATE_ONLY={Y | N} EXCLUDE=object_type[:name_clause] [,]

FILESIZE=integer[B | K | M |G] (default=unlimited)

FLASHBACK_SCN=scn_value FLASHBACK_TIME=TO_TIMESTAMP(time_value) FULL={Y | N} INCLUDE=object_type[:name_clause] [,] JOBNAME=jobname_string
24

Key Data Pump Export Parameters

LOGFILE=[directory_object:]file_name NOLOGFILE={Y | N} PARALLEL=integer (default=1) QUERY=[schema.][table_name:]query_clause SCHEMAS=schema_name [,] TABLES=[schema_name.]table_name[:partition_name] [,]

TABLESPACES=tablespace_name [,]

25

Data Pump Export Parameter Samples

Multiple dump files using a substitution variable (%U):

DUMPFILE=DP_DIR1:SCOTT_20060223_%U.dmp

Excluding indexes that start with EMP:

EXCLUDE=INDEX:LIKE EMP%

Excluding the SCOTT schema from a FULL export:

EXCLUDE=SCHEMA:=SCOTT

Mimicking the traditional CONSITENT parameter:

FLASHBACK_TIME=TO_TIMESTAMP

Exporting only TABLES, FUNCTIONS and VIEWS:

INCLUDE=TABLE,FUNCTION,VIEW

Using a query clause

QUERY=emp:WHERE salary > 100000
26

Key Data Pump Import Parameters

CONTENT={ALL | DATA_ONLY | METADATA_ONLY} DIRECTORY=directory_object (default=DATA_PUMP_DIR) DUMPFILE=[directory_object:]file_name [,] EXCLUDE=object_type[:name_clause] [,] FULL={Y | N} INCLUDE=object_type[:name_clause] [,]

JOBNAME=jobname_string
LOGFILE=[directory_object:]file_name NOLOGFILE={Y | N} PARALLEL=integer (default=1) QUERY=[schema.][table_name:]query_clause

27

Key Data Pump Import Parameters

REMAP_DATAFILE=source_datafile:target_datafile REMAP_SCHEMA=source_schema:target_schema REMAP_TABLESPACE=source_tablespace:target_tablespace REUSE_DATAFILES={Y | N} SCHEMAS=schema_name [,] SKIP_UNUSABLE_INDEXES={Y | N}

SQLFILE=[directory_object:]file_name
TABLE_EXISTS_ACTION={SKIP|APPEND|TRUNCATE|REPLACE} TABLES=[schema_name.]table_name[:partition_name] [,] TABLESPACES=tablespace_name [,]

28

Interactive Mode Commands

ADD_FILE=[directory_object:]file_name [,] CONTINUE_CLIENT EXIT_CLIENT FILESIZE=number KILL_JOB PARALLEL=integer

START_JOB
STATUS STOP_JOB

29

Demonstration

30

Exporting and Importing Sample Schemas

expdp system/oracle@ORA1020 dumpfile=scott.dmp schemas=scott impdp system/oracle@ORA1020 dumpfile=scott.dmp schemas=SCOTT remap_schema=SCOTT:LARRY expdb system/oracle@ORA1020 dumpfile=larry.dmp schemas=larry SELECT * FROM DBA_DATAPUMP_JOBS;
31

Using Interactive Mode

Ctrl-C to detach from the current export

Export> status
Export> stop_job

expdp system/oracle@ORA1020 attach=SYS_EXPORT_SCHEMA_01 Export> start_job

Export> exit_client
32

Data Pump Test Cases

33

Test Scenario #1

Generated sample SIEBEL data

Brushed off the dust on some old SIEBEL data population

scripts (circa 07/2000)

Designed for SIEBEL 6 on Oracle 8.1.6 Actual data is not important

Schema objects
Tables: 218 (many empty tables)
Indexes: 1180 (SIEBEL is a heavily indexed application)

Schema size (from DBA_SEGMENTS)

Tables: 1255MB Indexes: 148MB

34

Export Performance Test Criteria

All work performed on laptop C681 SGA remains constant

SGA_TARGET=0
SGA_MAX_SIZE=256MB BUFFER CACHE=152MB SHARED_POOL=60MB

Mostly default parameters Not monitoring CPU utilization Performed 4 runs

Disregarded results from 1st run and averaged the other 3

35

Export Scripts

EXP
sqlplus exp.exe sqlplus

-s system/oracle @timestamp.sql > exp.log

userid=system/oracle@ORA1020 file=SIEBEL.dmp log=SIEBEL.log owner='SIEBEL' -s system/oracle @timestamp.sql >> exp.log

EXPDP
erase

SIEBEL.dmp -s system/oracle @timestamp.sql > expdp.log

sqlplus

expdp.exe

userid=system/oracle@ORA1020 dumpfile=SIEBEL.dmp logfile=SIEBEL.log schemas='SIEBEL' directory=test_dir -s system/oracle @timestamp.sql >> expdp.log

36

sqlplus

Export Performance Test Results

exp 6:02 N/A 965 MB expdp 3:18 1.201 GB 621MB

Average Export Time Estimated File Size Actual File Size

37

Import Scripts

IMP
sqlplus imp.exe

-s system/oracle @timestamp.sql > exp.log

userid=system/oracle@ORA1020 file=SIEBEL.dmp log=SIEBEL.log fromuser='SIEBEL' touser='SCOTT' commit=y -s system/oracle @timestamp.sql >> exp.log

sqlplus

IMPDP
sqlplus

-s system/oracle @timestamp.sql > expdp.log

impdp.exe

userid=system/oracle@ORA1020 dumpfile=SIEBEL.dmp logfile=SIEBEL.log schemas='SIEBEL' directory=test_dir remap_schema=SIEBEL:SCOTT remap_tablespace=TOOLS:SCOTT_DATA -s system/oracle @timestamp.sql >> expdp.log
38

sqlplus

Import Performance Test Results

imp Average Import Time Average Import Time (no indexes) Average Import Time (no rows) impdb 27:07 25:19 27:27

Database was in ARCHIVELOG mode

Destination tablespace and archived log destination were both on ASM drives
Machine performance was degraded much more by impdb import No import tuning performed (only COMMIT=Y)
39

Test Scenario #2

Data taken from an actual CGI internal application Schema objects

Tables: 22
Indexes: 26

Schema size (from DBA_SEGMENTS)

Tables: 300MB Indexes: 101MB

40

Export Scripts

EXP
sqlplus exp.exe sqlplus

-s system/oracle @timestamp.sql > exp.log

userid=system/oracle@ORA1020 file=SCOTT.dmp log=SCOTT.log owner='SCOTT' -s system/oracle @timestamp.sql >> exp.log

EXPDP
erase

SIEBEL.dmp -s system/oracle @timestamp.sql > expdp.log

sqlplus

expdp.exe

userid=system/oracle@ORA1020 dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT' directory=test_dir -s system/oracle @timestamp.sql >> expdp.log

41

sqlplus

Export Performance Test Results

exp 1:24 N/A 261MB expdp 1:32 290MB 233MB

Average Export Time Estimated File Size Actual File Size

42

Import Scripts

IMP
sqlplus imp.exe

-s system/oracle @timestamp.sql > exp.log

userid=system/oracle@ORA1020 file=SCOTT.dmp log=SCOTT.log fromuser='SCOTT' touser='LARRY' commit=y

sqlplus

-s system/oracle @timestamp.sql >> exp.log

IMPDP
sqlplus

-s system/oracle @timestamp.sql > expdp.log

impdp.exe

userid=system/oracle@ORA1020 dumpfile=SCOTT.dmp logfile=SCOTT.log schemas='SCOTT' directory=test_dir remap_schema=SCOTT:LARRY remap_tablespace=SCOTT_DATA:LARRY_DATA

sqlplus

-s system/oracle @timestamp.sql >> expdp.log

43

Import Performance Test Results

imp 5:48 impdb 2:26

Average Import Time

Database was in NOARCHIVELOG mode

Destination tablespace and archived log destination were both on ASM drives No import tuning performed (only COMMIT=Y)

44

Conclusions

45

Conclusions

Data Pump is an exciting new Oracle 10g tool that provides many benefits over the traditional export and import utilities Whether to use Data Pump, Transportable Tablespaces, or even the traditional exp/imp will depend on the situation Since the command line interface is easy to use and so similar to the traditional exp/imp, DBAs and developers should spend the time to learn how to use it Final thought: since Data Pump dump files and traditional export dump files are not compatible/interchangeable, should a new file extension be used??? (.dmp vs .dpd)
46

The End

Comments, Questions ???

47