Exadata Migration
How to modernize your database platform to realize consolidation savings
Chris Bradham
About Applied Engineering Management
Applied Engineering Management (AEM) Corporation Founded in 1986 as a
100% woman-owned business
More than 25 years of profitable growth
Headquartered in Chantilly, VA with offices located in major metropolitan
areas including; Los Angeles, San Antonio, and Jacksonville
Diversified client base including Fortune 500 and major government
agencies
Industry recognized awards and certifications for performance, capability
and delivery
Chris Bradham
Oracle DBA experience 1997 to present (Oracle 7 to 11.2)
Replication, Materialized Views, Data Guard, GoldenGate, Exadata, RAC
Part-time Instructor George Mason University (OCA/OCP)
Oracle Certified Exadata Implementation Specialist, Oracle Certified
Professional (11g), Oracle Performance Tuning Certified(11g), ITIL
Foundation, Security+
Oracle Application Server, SAN Administration, Linux Administration,
VMWare, IBM / Sun Blade Administration
chris.bradham@aemcorp.com
Whats being covered?
Technology Refresh
Legacy Environment / Options
Exadata Components
Operational Considerations
Migration Considerations
Results of Migration
Lessons Learned
References
Q & A
Disclaimer
The US Navy, Commander Navy Installations Command (CNIC), nor
any related government organization in no way endorse Oracle
Corporation or the Oracle Exadata Database Machine. The opinions
expressed during the presentation are solely those of the presenter
and in no way reflects those of the government or any government
interest.
Background Information
Global multi-service DoD Web-based Housing application
Over 300 schemas
750 Gb of data
4,300 Active Users
4.2 million annual log ins
4,500 Reports Generated Per Day
AEM Corporation responsible for Hosting / Operations & Maintenance / Technology Refresh
Legacy Configuration
Physical Application Servers
4 Gbps Fibre Channel
Clarion Disk Array (10k / 15k RPM)
5 Node RAC Servers
Oracle 10gR2 RAC install
Pre-Tech Refresh Issues
Legacy hardware over six years old
Patches (5 nodes, slower machines)
Deployments, data updates time consuming
Large or complex reports often hang
Node evictions due to network / disk speed issues
Oracle 10.2.0.4 Support ended 6/31/11
Data Warehouse delay due to performance requirements
(Oracle Streams attempt)
Alternative 1 : Based on Legacy Solution
Virtualized application servers
Network bonding
8 Gbps Fibre Channel
EMC Disk Array
5 Node RAC Servers
Oracle 11gR2 RAC install
Alternative 2 : Based on Exadata Solution
Virtualized application servers
Network bonding
40 Gbps backbone
Oracle Storage Servers
2 node Quarter Rack
Oracle 11gR2 RAC preconfigured
Surprise, we chose Exadata!
Exadata Selection Points
Licensing fees made Exadata the low cost solution
Total database hardware solution
o2 nodes vs. 5 nodes
oNumber of Vendors
Patch Strategy
Sizable and expandable
Oracle vested to help DoD succeed
Storage Indexes / Smart Scan / Smart Flash Cache
Throughput Gb/Second
80.0
75
70.0
60.0
2 Gbps Fibre Channel x2
50.0
4 Gbps Fibre Channel x2
40.0
8 Gbps Fibre Channel x2
37
30.0
Exadata 1/4 - Disk
25
20.0
12.5
10.0
0.4
0.0
0.8
1.6
5.4
Exadata 1/2 - Disk
16
Exadata Full - Disk
Exadata 1/4 - Disk & Flash
Exadata 1/2 - Disk & Flash
Exadata Full - Disk & Flash
Exadata Planning/Considerations
Pre-configuration customer worksheet (DBM Configuration spreadsheet)
Initial Setup Activities
Network Configuration / Software Installation
Procurement considerations (2 Quarter Racks)
Space Choices
High Capacity (3 Tb / 7.2 K RPM)
High Performance ( 600 Gb / 15k RPM)
Space Allocation (ASM Disk Group layout)
78% DATA, 20% RECO, 2% DBFS_DG
o25 Tb
<- Production (Triple Mirror)
o8 Tb
<- Test (Mirror)
Tech Refresh Challenges
100% hardware replacement and Data Center move
Narrow window for outage to avoid negative impact on end users
Performance of system, database growth, and network bandwidth
Exadata unproven in DoD space at the time (Security)
Upgrading Database versions (data/code/reports)
Lots of change, what if issues surface???
Summary of Choices (X2)
X2-8 Full
X2-2 Full
X2-2 Half
X2-2 Quarter
$$$
$$$
$$
Database Grid Cores
160
96
48
24
Database Grid Memory (GB)
4096
768 (max 1152)
384 (max 576)
192 (max 288)
InfiniBand switches
Ethernet switch
KVM
No
Yes
Yes
Yes
Exadata Storage Servers
14
14
Storage Grid CPU Cores
168
168
84
36
5.3 TB
5.3 TB
2.6 TB
1.1 TB
High Perf
100 TB
100 TB
50 TB
21.6 TB
High Cap
504 TB
504 TB
252 TB
108 TB
High Perf
45 TB
45 TB
22.5 TB
9.5 TB
High Cap
224 TB
224 TB
112 TB
48 TB
High Perf
30 TB
30 TB
15 TB
6.5 TB
High Cap
150 TB
150 TB
75 TB
32 TB
Initial / Licensing Cost
Database Servers
Raw Flash Capacity
Raw Storage
Capacity
Usable mirrored
capacity
Usable Triple
mirrored capacity
Summary of Choices (X3)
X3-8 Full
X3-2 Full
X3-2 Half
X3-2 Quarter
X3-2 Eighth
$$$
$$$
$$
Database Grid Cores
160
128
64
32
16
Database Grid Memory (GB)
4096
1024 (max
2048)
InfiniBand switches
Ethernet switch
KVM
Yes
Yes
Yes
Yes
Yes
Exadata Storage Servers
14
14
Storage Grid CPU Cores
168
168
84
36
36
22.4 TB
22.4 TB
11.2 TB
4.8 TB
2.4TB
Initial / Licensing Cost
Database Servers
Raw Flash Capacity
512 (max 1024) 256 (max 512) 256 (max 512)
Raw Storage
Capacity
High Perf
100 TB
100 TB
50 TB
21.6 TB
10.8TB
High Cap
504 TB
504 TB
252 TB
108 TB
54 TB
Usable mirrored
capacity
High Perf
45 TB
45 TB
22.5 TB
9.5 TB
4.5 TB
High Cap
224 TB
224 TB
112 TB
48 TB
23 TB
Smart Flash Cache Considerations
Helps with
Write-Through cache voids caching data that will not be reused
Holds hot data, much faster than disk (small, random I/O)
Data not duplicated from cache in other Storage Servers
Reduce latency of log write by simultaneous write to flash / disk
(faster writes) with minimal space (512 Mb)
Write-Back cache 11.2.0.3.9
Dont touch except for
Alter table <table_name> flash_cache keep;
Create Flash Disks out of the Flash Cache
Reassign portion for TEMP tablespace on index builds
Database Node Considerations
Database Consolidation
SGA Settings
AMM Bad! ASMM Good! (set minimum values)
Huge Pages (reduce overhead)
Wheres the shared storage space?
DBFS is the answer (fix_control=8,ac_timeout=60 and SGA=2Gb)
Is everything setup correctly?
Exachk is the answer
Indexes / Hints / Compression
Large Segments <- 8 Mb Initial / Next Size with Autoallocate
TEMP <- BIGFILE, Autoextend 1 Gb, Uniform 1 Mb
Exadata Patch Management
Multiple Patches
Infiniband (once per year)
DB Nodes / Storage Server (quarterly)
Bundle Patch (BP) DB Software (quarterly)
Additional components (Ethernet switch, KVM, PDU)
Bug Fixes included so important to apply
One-off patches / opatch before and after patch
Rolling option time a consideration
Security
DoD 8570 Requirements
Security Technical Implementation Guide (STIG)
oOracle installation not customizable
oDBFS and idle_time dont play well together
oAutomatic Service Request (ASR) / Configuration Manager Limitation
oGrid Control / Third Party Certificates (September release)
oBanners / SQLNET.ORA settings impact on tools
Dont assume security settings will not have impact. Must TEST!!!
Migration Strategies
10.2.0.4 to 11.2.0.x Options Considered
DBFS with external tables (5 to 7 GB/sec file system I/O throughput)
GoldenGate with datapump (near-zero downtime)
Datapump
Factors
Maintenance window
Risk of data loss
Familiarity with technology
Whatever the choice, perform multiple trial runs for optimal settings.
2011 Technical Refresh (Data Center move)
On 9/9/11 at 7pm application
servers at legacy site were Turned
Off:
Transferred all data to Data Center
Reloaded data for all installations
Network outage occurred during
data transfer (2 hours)
On 9/10/11at 7am New System
Testing was Initiated
Users were on the system by 3pm
In Future-> X3 or Storage Expansion
Migration Timeline
1/4 Rack
Grid
Delivered
Migrate Control
(Test) Options Setup
1/11
2/11
Chris Initial DB
First Setup /
Load
Day
3/11
STIG
Test
Text
Setup
DBFS
Migration
Test
4/11
Apply BP
Test
5/11
Initial
DB
Oracle Setup / STIG Prod/
CAB
Apply BP
Load
6/11
Migration
Selection
7/11
1/4 Rack
Delivered
(Prod)
8/11
Cutover
to
Exadata
9/11
Migration
Grid
Test
Control
Setup
Post Tech Refresh Performance (in hours)
Event
Legacy
Exadata
BOR1 Process
4 hours
1 hour
BOR2 Process
4 minutes
23 seconds
Datamart IU
3 hours
35 minutes
MV refresh
37 minutes
13 minutes
Exadata Lessons Learned
Ensure hosting center can accommodate Exadatas dimensions (door height)
Staff requirements (more communication necessary)
Smart Scan <- direct path reads, table access full, fast full index scans,
parallel with parallel_degree_policy not auto
Chained rows / small tables void smart scans
Ideally 2 Exadata Database machines
EHCC 10x space and performance (DML)
Expect CPU utilization to decrease
In-Memory
X2-8
Massive Memory
2010
All I/Os to Memory
X3
2012
Exadata Lessons Learned (cont.)
Grid Control for monitoring / managing components
Platinum Support, major assistance
Exachk and opatch before / after patching
Standard tuning principles apply (OLTP)
Bring over OLTP database AS-IS
Time, Experience keys to stability
References
http://www.oracle.com/technetwork/database/options/olap/olap-exadata-x2-2performance-1429042.pdf
http://www.scribd.com/doc/57026846/Exadata-Smart-Flash-Cache-366203
http://www.oracle.com/technetwork/database/exadata/exadata-technical-whitepaper134575.pdf
http://www.oracle.com/technetwork/server-storage/engineeredsystems/exadata/exadata-storage-technical-overview-128045.pdf
http://www.oracle.com/technetwork/database/features/availability/exadata-consolidation522500.pdf
http://www.ioug.org/Portals/0/Documents/Anuj%20Mohan.pdf
http://nyoug.org/Presentations/2011/March/Niemiec_Keynote_Exadata.pdf
http://www.oracle.com/technetwork/articles/oem/exadata-commands-part1-402441.html
http://www.centroid.com/webinars/ExadataChangestheGame.pdf
http://www.centroid.com/knowledgebase/blog/using-dbfs-and-external-tables
References (cont.)
Database Machine and Exadata Storage Server (888828.1)
Oracle Exadata Database Machine exachk (1070954.1)
Oracle Exadata Best Practices (757552.1)
Best Practices for OLTP on the Sun Oracle Database Machine (1269706.1)
Best Practices for Data Warehousing on Database Machine (1297112.1)
Oracle Sun Database Machine Application Best Practices for Data Warehousing
(1094934.1)
Oracle Sun Database Machine Diagnosability and Troubleshooting Best Practices
(1274324.1)
Expert Oracle Exadata (Osborne, Johnson, Poder)
Questions???
Backup Slides
Exadata Rack Options
2 six Core Processors / 96 Gb RAM per DB node
2 six Core Processors / 24 Gb RAM per Storage Server
Dual ported 40 Gb/sec InfiniBand
Quarter Rack
2 DB nodes
2 Infiniband switches
Half Rack
4 DB nodes
3 Infiniband switches
Full Rack
8 DB nodes
3 Infiniband switches
Infiniband 10x faster than Fiber Channel
Cache Hierarchy (Full Rack X2-2)
Database DRAM
768 GB Raw Capacity
100 GB / Second
Flash Cache
5 TB Raw Capacity
50 GB / Second
Disk
100 300 TB Raw Capacity
21 GB / Second
Quarter Rack Selected
2 Xeon-based Dual-processor Database Servers (Sun Fire
X4170 M2)
24 cores (12 per server)
192 GB memory expandable to 288 GB (96 GB per
server expandable to 144 GB)
10 GigE connectivity to Data Center
4 x 10GbE ports (2 per server)
1.1 TB High Speed Flash
3 Exadata Storage Servers X2-2
All with High Performance 600GB disks
OR
All with High Capacity 3 TB disks
2 Sun Datacenter InfiniBand Switch
36-port Managed QDR (40Gb/s) switch
1 Admin Cisco Ethernet switch
Can Upgrade to a Half
Keyboard, Video, Mouse (KVM) hardware
or just add storage
Redundant Power Distributions Units (PDUs)
Rack
Storage Components and Expansion (X2)
-Local Storage (750 GB)
-Storage Servers (12 Disks per server)
Full
Half
Quarter
InfiniBand switches
Ethernet switch
Yes
Yes
Yes
Exadata Storage Servers
18
Storage Grid CPU Cores
216
108
48
6.75 TB
3.4 TB
1.5 TB
High Perf
128 TB
64 TB
28 TB
High Cap
648 TB
324 TB
144 TB
Usable mirrored
capacity
High Perf
58 TB
29 TB
13 TB
High Cap
288 TB
144 TB
64 TB
Usable Triple
mirrored capacity
High Perf
38 TB
19 TB
8.5TB
High Cap
192 TB
96 TB
43 TB
KVM
Raw Flash Capacity
Raw Storage
Capacity
