Best Practices at AT&T

Oracle Real Application Clusters

(Oracle RAC) & Oracle E-Business
Suite
Bryon Rickey
Ravi Kayarthodi
September 20th, 2010

2010 AT&T Intellectual Property. All rights reserved. AT&T, the AT&T logo and all other AT&T marks contained herein are
trademarks of AT&T Intellectual Property and/or AT&T affiliated companies. All other marks contained herein are the property
of their respective owners.

Agenda
Welcome and Introductions
Supply Chain Management (SCM) environment
Overview/highlights
Architecture
Partnership with Oracle to transform our Business
Key project highlights
Future initiatives
Solution Support Center (SSC) and on-site
support
Q&A

About AT&T
AT&T is one of the worlds largest
Telecommunications companies
AT&T Global Headquarters is based in Dallas, TX
AT&T has approximately 290,000 employees
AT&T has more than 90 million wireless customers
AT&T has more than 16 million high speed
internet subscribers
AT&T has more than 2.5 million uverse
subscribers
Exclusive U.S. provider of iPhone 3G / 3GS / 4

AT&T SCM: Architecture highlights

Oracle E-Business Suite Centric
Scale-up with cluster database on very large
infrastructure
High availability with Oracle Data Guard, Oracle
GoldenGate, and site level redundancy
Central inventory/financial management; utilizes
product hub, planning/forecast with ASCP/Demantra
Order orchestration and brokering with in-house
developed Oracle based application system
Internet deployment for interaction with external
suppliers
Management reporting/profitability analysis;
procurement and spend analytics
Industry leading warehouse/distribution
management systems

AT&T SCM Platform: Financial Integration

Both Network and Retail SCM - financial
counterparts are integrated in the same platform:
Shared AP, AR, PA, FA, and GL
Supply Chain transactions leverage same
referential data and accounting rules
Financial integration of supply chain transactions
are seamless and are recorded real-time in the
general ledger
No need for reconciliation between sub ledgers
Ability to forecast financial implications based on
SCM forecasts
Online financial budgeting validation on
purchasing and requisitions
Analytical GL Cube and Inventory Cube

Results
AT&T has saved more than $500 million and has
improved its purchasing power by combining
accounts with its vendors.
Estimated to save $1B in inventory costs, and
$300M per year in reduced operating costs.
Warehouses have been able to keep inventories
low without missing SLA, resulting reduced overall
inventory costs.
Ability to identify sold merchandise, resulting in
fewer fraudulent returns

Industry Recognition
2010 IDGs CIO Award
Recognizes

organizations around the world that

exemplify the highest level of operational and
strategic excellence in information technology

Systems Architecture

AT&T SCM: E-Business Suite Centric

AT&T SCM: Key Systems Architecture

ED
I

I
ED

eAuction

ODP
OM

ASCP

Inventory Planning

RNF

Purchasing

PO

GL

AP
PA

FA

Financials

Oracle E-Business Suite 11i

Receipts
Shipments

Warehouse Mgmt
Distribution Center

Receipts
Moves

Audits

Repairs

Inventory Tracking
Corporate Asset Tracking

Oracle 11i E-Business Environment

Forward Orders
Store
Store
Replenishment
Replenishment

National
National
Retailers
Retailers

Dealers/
Dealers/
iStore
iStore

Planning
Planning

Purchasing
Purchasing

DSL
DSLand
and
uVerse
HSIA
uVerse HSIA

Order
Order
Management
Management

Direct
DirectFulfilment
Fulfilment

Inventory
Inventory

Upfront
Upfront

Receivables
Receivables

EPL
EPL

Financials
Financials

Oracle E-Business Suite 11i Apps

Store
Store

Reverse

XBM/Insurance
XBM/Insurance

Dealer/NR
Dealer/NR

Service
ServiceDepots
Depots

RU30
RU30

Sales
Sales

Opus
Opus

Returns
Returns

Company Owned Retail

416 CPU Cores & 1.7TB of Memory in

DB Cluster
14TB of storage for transaction DB
3TB of storage for Archived DB

AT&T SCM Platform: Oracle10g RAC

Infrastructure: VCS+ CRS

AT&T SCM Platform: Transaction Volume

Approximately 3.35 million logons to the
production areas per day
2.06 billion SQL executions per day
Approximately 136.5 Billion logical reads and 41TB
of physical reads performed per day
Approximately 7 TB of writes performed per day
Approximately 1.05 billion user calls and 52
million transactions completed per day
200,000 concurrent request per day

AT&T SCM Platform: Operational architecture

Decoupled month-end book closing preparation
Statistics gathered on a clone
Separate reporting environment refreshed
nightly; Inventory and GL Cubes for reporting
Revenue share and allocation on a separate
system
Direct supplier interaction with Internet-deployed
application modules
Utilizes market making mechanisms (sourcing)

AT&T SCM: Oracle 11i Dev/Test Configurations

Bothell, WA

HDS USP-V

Data Guard/
Custom
Shipping

NetApp
GOLD

L-TRA

Data
Guard

Shadow
Image

HDS USP-V

PROD

Alpharetta, GA

Data
Guard

R-TRA

Shadow Image

Data Guard/
Custom Shipping
NetApp
GOLD

MET2

NetApp

Master
Flex Clone

SnapMirror

FTP
Shadow
Image

Flex
Clone

DEV
DEV
QA
DEV DEV

MET1
FTP

QA

RPT
QA
DEV
QA

Flex Clone

Flex Clone

DEV
DEV
DEV

Storage Efficiency with NetApp de-duplication Tools

Performance test environment on tier-1 storage

NetApp
Master

AT&T SCM: Oracle Automatic Storage

Management
Local and Remote DR databases and also
Reporting databases have been migrated to ASM
storage to meet storage and capacity needs
Avoided SFRAC costs
Performance
SCMP DR/BC database converted to RAC database
Saved SFRAC license cost on E25K frames
Comply with AT&T standard for cluster database
(ASM is required)
Expected to save cost in the future when we
upgrade other Corporate Systems databases

AT&T SCM: Archive and Purge

Archive Stats
16 custom and core modules
including (GL, INV, OM, AR,
WSH, PO, FA)
Executed weekly, monthly,
quarterly, and yearly
History instance
Informatica Data Archive
application used
Schedule is monitored real-time
to ensure adherence
Purge Stats
70 jobs via Oracle concurrent
programs
Executed daily and weekly
Schedule is monitored real-time
to ensure adherence

AT&T Supply Chain

Geographic Distribution DB Flow to Test and Dev

Primary Site
XX

XX
XX

XX

Test Site 1
Development
Site 2

XX
XX

Test Site 2

XX
XX

XX
XX

DR Site
XX
Prod
Dev/Test

XX
XX

XX

XX

XX

XX
XX

XX

XX

Development
Site 1
XX

XX

XX

XX
XX

XX

XX

AT&T SCM: Spend Analytics

Oracle Business Intelligence
Enterprise Edition (Oracle OBIEE)
Powered
Sourcing from Oracle E-Business
Suite and Legacy Applications

Top 5 Categories

Spend

Top 5 Suppliers

Line Of Business

Contract Leakage

Off Contract Spend Payables Leakage

AT&T SCM: Database

Maintenance/Performance Management
Monthly month-end close of books stats
validation database statistics preparation on
production clone
Table Re-organization routine and ad-hoc
Periodic index rebuild
Daily performance call
Daily/weekly/monthly/yearly purge/archive
schedules vigilance in adhering to them
Extensive performance monitoring on all levels
of the system

Partnering with Oracle

Advanced Customer Service
Oracle Solution Service Center
(SSC)

SSC: Project Spotlight: Patch Recommendations

Project Activities
o

Identification of available latest

required patches for AT&T
environment
Participation in meetings to review
recommendations

Description
An initiative to incorporate E-Business
Suite recommended patches, such as,
Rollup (RUP) in AT&Ts release
schedules for 2010

Value Addition

SSC and AT&T worked together to create a new process to efficiently identify required
patches with the goal to minimize the impact of rolling them out in our production
environment and to current/future projects:
Quarterly document for AT&T environments with the latest recommended patches.

Participation on AT&T IT meetings to clarify recommendations and questions

Provide detailed patch analysis on a ad-hoc basis

Successful roll out of AP.O patchset along with other Financials patches in February
2010: stabilized production environment and brought environment to the latest code
base.

Successful completion of the ASCP Rup 37, Collections Rollup (RUP) 32 and Application
Technology Group (ATG) RUP-7 for the April 2010 production rollout.

2
3

SSC: Project Spotlight: Performance Improvement

Project Activities
o
o
o

Understand AT&T batch window

requirements and transaction volumes
Identification and root cause analysis
of problem areas
Implementation of resolutions

Description
An initiative to address various
performance issues related to MonthEnd-Close processes and other batch
processes after Oracle 10g database
upgrade.

Value Addition

Active participation on Conference bridge calls and OWC sessions.

Engage Oracle development groups (Financials and Apps Performance) to provide SQL
tuning improvements.

Improve month end close activities like AR close, depreciation and asset retirement,
optimizer, Invoice Validation processes.

Maximize performance during month end, preventing delay; AT&T was able to successfully
meet the SLAs for the Month End Close Processes month after month.

Provide recommendations for major upgrades (DB, Applications and Tools)

SSC: Critical SR Spotlight: Other Key areas of focus

Project Activities

o
o

Description

Workflow background process

WF_DEFERRED queue growing exponentially; AT&T engages

SSC to evaluate and address the issue.
upgrade.

RAC CRS Configuration

Collect diagnostic and configuration data

Troubleshoot frequent RAC node reboot/performance issue
post upgrade
Discover unforeseen problems
Validate AT&T configuration with RAC Best Practices

Device Life Cycle Database Upgrade

Provided upgrade expertise and support (pre and post

deployment for 11g R2 upgrade.

Value Addition

Active participation on conference bridge calls and OWC sessions

Review current setups and schedules of workflow Background Process

Engage on Severity-1 SR and work collaboratively with Oracle development to provide workaround (eg.
skip script)

Work with at&t and Oracle development organization to provide scripts to clear workflow DEFERRED
queue

Monitor the implementation of recommended suggestions

Engage Oracle Development and RAC COE team

Engage on Architectural design discussions

Stand-by support for DB/RAC upgrade deployment activities

Ensure successful implementation of RAC upgrade Rollout

SSC: AT&T Targeted Services: Operations

The Services Yield to the Customer targeted for the next 6-month planning period
is comprised of key Support activities, aligned with key customer
objectives/activities towards achievement of AT&Ts Account Goals
Supply Chain Transformation (Streamline Operations for newly deployed
modules)

Work with ACS sales to determine appropriate support for upcoming SC rollout.
Determine/secure funding for new modules (Demantra and Transportation Management
Suite)

Develop, Enhance and Secure development sponsors

Participate in scheduled project Deep Dive calls with PMO and project team

Site visit to Memphis to understand warehouse and logistic operations understand

business functions.

Supply Chain Business develop key relationships with Retail and Network business
users

11i Financials and Supply Chain Production Supportability (System)

Next generation iPhone launch and 2010 Holiday Retail Season readiness

Top Ten Prioritization

SSC: AT&T Targeted Services: Transportation Manager

OneATT- New Horizon to transform Network and Asset Management: Logistics TMS

Key Secondary Activities

Participate in scheduled project Deep Dive calls with PMO and project
team

Work with PMO/SSC to determine appropriate support model for New Horizons

Work closely with OCS /PMO on project milestones develop relationships

Continue to maintain Dev OTM Network

Supply Chain Business develop key relationships

Site visit to Warehouses

SSC walk-thrus

Key Deliverables

Enhance Supply Chain SSC presence and understand usage

and architecture for OTM

Site visit to understand implementation, document in

configuration guide.

Performance testing as requested.

Amend SSC contract to include OTM products

SSC: Oracle E-Business Suite Release 12/Database 11g R2

Upgrade
Upgrade existing Oracle 11i.10 SCM and Planning environments to E-

Business Suite R12. This will enhances supportability for our complex
environment and provide additional needed functionalities

Three stage upgrade (2 3 year journey)

Supply Chain planning to release 12

RDBMS for 10g to 11G

Supply chain platform to release 12

Objective:

System Availability, Supportability, and Software Currency

Key Secondary Activities

SSC Top Ten Prioritization: Service Request review meetings with Supply Chain,
Financials, and DBA support teams

Patching/Release Management strategy for Release 12 interoperability;

Leverage Tools and Templates

Daily SSC Performance Calls

Key Deliverables

Proactive Diagnostics (Configuration/Performance)

Contracted deliverables

Semi-Annual Account Review

Solution Support Center

Service Highlights
Focused Engineering Team of Oracle Subject Matter Experts
Fast track solutions based on intimate Knowledge of customer
Environment and business needs
Proactive Software Advice and Knowledge Transfers
Performance and Patch Assessments
Single point of contact for all technical issues - Technical Lead
Single point of contact for all escalations SDM
24/7 coverage

Value
Mitigate Upgrade Risks substantially
All project milestones are met
Ensured optimal usage of Oracle features and functionality
Reduced overall Maintenance/upgrade/ training costs

Solution Support Center

Key Delivered Features
Focused Engineering Team
Single Point Accountability
SDM
Exclusive 1 800 number
24x7x365
Review of Critical Issues as
Needed
Patch Analysis

Stand-by Support for

Critical Go-lives
Proactive Alerts / Patch
Notifications
Project Planning /
Migration Assistance
Periodic Review Sessions
Applied Research

Questions

For More Information

On this presentation, e-mail Bryon Rickey (BR3681@att.com) or

Ravi Kayarthodi (RK8409@att.com).

For general information on Advanced Customer Services, go to

oracle.com/acs
or e-mail

acsdirect_us@oracle.com