Oracle RAC Assessment Report

System Health Score is 90 out of 100 (detail)

 
Benefit / Impact:

Stability, Availability, Standardization



Risk:

Potential cluster instability due to clusterware version mismatch on cluster nodes.
It is possible that if the clusterware versions do not match that some incompatibility
could exist which would make diagnosing problems difficult or bugs fixed in the
later clusterware version still being present on some nodes but not on others.



Action / Repair:

Unless in the process of a rolling upgrade of the clusterware it is assumed
that the clusterware versions will match across the cluster.  If they do not then it is
assumed that some mistake has been made and overlooked.  The purpose of
this check is to bring this situation to the attention of the customer for action and remedy.
 

 Make sure machine clocks are synchronized on all nodes to the same NTP source.

NOTE: raccheck expects the NTP time source to be the same across the cluster based on the NTP server IP address.  In cases where the customer is using a fault tolerant configuration for NTP servers and the customer is certain that the configuration is correct and the same time source is being utilized then a finding for this check can be ignored.

Implement NTP (Network Time Protocol) on all nodes.
Prevents evictions and helps to facilitate problem diagnosis.

Also use  the -x option (ie. ntpd -x, xntp -x) if available to prevent time from moving backwards in large amounts. This slewing will help reduce time changes into multiple small changes, such that they will not impact the CRS. Enterprise Linux: see /etc/sysconfig/ntpd; Solaris: set "slewalways yes" and "disable pll" in /etc/inet/ntp.conf. 
Like:- 
       # Drop root to id 'ntp:ntp' by default.
       OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
       # Set to 'yes' to sync hw clock after successful ntpdate
       SYNC_HWCLOCK=no
       # Additional options for ntpdate
       NTPDATE_OPTIONS=""

The Time servers operate in a pyramid structure where the top of the NTP stack is usually an external time source (such as GPS Clock).  This then trickles down through the Network Switch stack to the connected server.  
This NTP stack acts as the NTP Server and ensuring that all the RAC Nodes are acting as clients to this server in a slewing method will keep time changes to a minute amount.

Changes in global time to account for atomic accuracy's over Earth rotational wobble , will thus be accounted for with minimal effect.   This is sometimes referred to as the " Leap Second " " epoch ", (between  UTC  12/31/2008 23:59.59 and 01/01/2009 00:00.00  has the one second inserted).

RFC "NTP Slewing for RAC" has been created successfully. CCB ID 462 
 

• HP Leap Second Epoch to the referenced time.
• Solaris changes for Leap Second
• Note: 759143.1 - NTP leap second event causing CRS node reboot
• Linux Leap Second
• Earth Rotation and reference System Service (IERS).
• Note: 551704.1 - Linux OS Service 'ntpd'
• Network Timp Protocol

 Make sure machine clocks are synchronized on all nodes to the same NTP source.
Implement NTP (Network Time Protocol) on all nodes.
Prevents evictions and helps to facilitate problem diagnosis.

Also use  the -x option (ie. ntpd -x, xntp -x) if available to prevent time from moving backwards in large amounts. This slewing will help reduce time changes into multiple small changes, such that they will not impact the CRS. Enterprise Linux: see /etc/sysconfig/ntpd; Solaris: set "slewalways yes" and "disable pll" in /etc/inet/ntp.conf. 
Like:- 
       # Drop root to id 'ntp:ntp' by default.
       OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
       # Set to 'yes' to sync hw clock after successful ntpdate
       SYNC_HWCLOCK=no
       # Additional options for ntpdate
       NTPDATE_OPTIONS=""

The Time servers operate in a pyramid structure where the top of the NTP stack is usually an external time source (such as GPS Clock).  This then trickles down through the Network Switch stack to the connected server.  
This NTP stack acts as the NTP Server and ensuring that all the RAC Nodes are acting as clients to this server in a slewing method will keep time changes to a minute amount.

Changes in global time to account for atomic accuracy's over Earth rotational wobble , will thus be accounted for with minimal effect.   This is sometimes referred to as the " Leap Second " " epoch ", (between  UTC  12/31/2008 23:59.59 and 01/01/2009 00:00.00  has the one second inserted).

More information can be found in Note 759143.1
"NTP leap second event causing Oracle Clusterware node reboot"
Linked to this Success Factor.

RFC "NTP Slewing for RAC" has been created successfully. CCB ID 462 
 

 
Benefit / Impact:

Clusterware deployment requirement



Risk:

Potential cluster instability



Action / Repair:

Oracle Clusterware requires the same time zone setting on all cluster nodes. During installation, the installation process picks up the time zone setting of the Grid installation owner on the node where OUI runs, and uses that on all nodes as the default TZ setting for all processes managed by Oracle Clusterware. This default is used for databases, Oracle ASM, and any other managed processes.

If for whatever reason the time zones have gotten out of sych then the configuration should be corrected.  Consult with Oracle Support about the proper method for correcting the time zones.
 

 Operating system versions should match on each node of the cluster
 

 
Benefit / Impact:

Stability, Availability, Standardization



Risk:

Potential cluster instability due to kernel version mismatch on cluster nodes.
It is possible that if the kernel versions do not match that some incompatibility
could exist which would make diagnosing problems difficult or bugs fixed in the l
ater kernel still being present on some nodes but not on others.



Action / Repair:

Unless in the process of a rolling upgrade of cluster node kernels it is assumed
that the kernel versions will match across the cluster.  If they do not then it is
assumed that some mistake has been made and overlooked.  The purpose of
this check is to bring this situation to the attention of the customer for action and remedy.
 

 
Benefit / Impact:

Stability, Availability, Standardization



Risk:

Potential cluster instability due to clusterware version mismatch on cluster nodes.
It is possible that if the clusterware versions do not match that some incompatibility
could exist which would make diagnosing problems difficult or bugs fixed in the
later clusterware version still being present on some nodes but not on others.



Action / Repair:

Unless in the process of a rolling upgrade of the clusterware it is assumed
that the clusterware versions will match across the cluster.  If they do not then it is
assumed that some mistake has been made and overlooked.  The purpose of
this check is to bring this situation to the attention of the customer for action and remedy.
 

 
Benefit / Impact:

Stability, Availability, Standardization



Risk:

Potential database or application instability due to version mismatch for database homes.
It is possible that if the versions of related RDBMS homes on all the cluster nodes do not
match that some incompatibility could exist which would make diagnosing problems difficult
or bugs fixed in the later RDBMS version still being present on some nodes but not on others.



Action / Repair:

It is assumed that the RDBMS versions of related database homes will match across the cluster. 
If the versions of related RDBMS homes do not match then it is assumed that some mistake has
been made and overlooked.  The purpose of this check is to bring this situation to the attention
of the customer for action and remedy.
 

 
Benefit / Impact:

Clusterware deployment requirement



Risk:

Potential cluster instability



Action / Repair:

Oracle Clusterware requires the same time zone setting on all cluster nodes. During installation, the installation process picks up the time zone setting of the Grid installation owner on the node where OUI runs, and uses that on all nodes as the default TZ setting for all processes managed by Oracle Clusterware. This default is used for databases, Oracle ASM, and any other managed processes.

If for whatever reason the time zones have gotten out of sych then the configuration should be corrected.  Consult with Oracle Support about the proper method for correcting the time zones.
 

 
Benefit / Impact:

Stability, Availability, Standardization



Risk:

Potential application instability due to incorrectly named network interfaces used for node VIP.



Action / Repair:

The Oracle clusterware expects and it is required that the network interfaces used for
the public interface used for the node VIP be named the same on all nodes of the cluster.
 

 
Benefit / Impact:

Availability, stability




Risk:

Potential OCR logical corruptions and permission problems accessing OCR keys when multiple O/S users share the same UID which are difficult to diagnose.

Action / Repair:

For GI/CRS, ASM and RDBMS software owners ensure one unique user ID with a single name is in use across the cluster.
 

 
Benefit / Impact:

Availability, stability



Risk:

Potential OCR logical corruptions and permission problems accessing OCR keys when multiple O/S users share the same UID which are difficult to diagnose.



Action / Repair:

For GI/CRS, ASM and RDBMS software owners ensure one unique user ID with a single name is in use across the cluster.
 

 If Checkpoints are not being completed the database may hang or experience performance degradation.  Under this circumstance the alert.log will contain "checkpoint not complete" messages and it is recommended that the online redo logs be recreated with a larger size.
 

• Note: 265831.1 - Automatic Checkpoint Tuning in Oracle 10g
• Note: 147468.1 - Checkpoint Tuning and Troubleshooting Guide

 
Benefit / Impact:

Setting this kernel parameter to 1 causes the kernel to panic if a kernel oops or bug is encountered.


Risk:

Kernel panic or oops represents potential problems with the server. If it's set to 0, the kernel will try to keep running (with whatever awful consequences). If it's set to 1, the kernel will enter the panic state and halt.

Action / Repair:

As root user, execute the following command from the shell prompt

echo 1 > /proc/sys/kernel/panic_on_oops

To set this kernel parameter permanently, edit /etc/sysctl.conf and make the following entry

kernel.panic_on_oops = 1
 

 
Benefit / Impact:

Documented value is the /etc/security/limits.conf file as documented in Oracle 12c Database software Installation Guide, section 10 Configuring Kernel Parameters and Resource Limits.  

If the /etc/security/limits.conf file is not configured as described in the documentation then log in to the system as the database software owner (eg., oracle) and check the soft stack configuration as described below.

Risk:

The soft stack shell limit for the Oracle DB software install owner as defined in /etc/security/limits.conf should be >= 10240.  As long as the soft stack limit is 10240 or above then the configuration should be ok.



Action / Repair:

To change the DB software install owner soft stack shell limit, execute the following command at the shell prompt

$ ulimit -Ss 10240

To set this limit permanently for the DB software install owner (ex, oracle), edit /etc/security/limits.conf as root user and add the following entry

oracle soft stack 10240
 

 
Benefit / Impact:

When the size of SGA is increased with the proper minimum value set for shared pool and buffer cache, the number of resize operations should be reduced.

Risk:

An undersized SGA can lead to frequent resizing operations of the SGA which could potentially cause slow performance or database hang situations.

Action / Repair:

Perform the following 3 actions:

1.  Increase the SGA_TARGET or MEMORY_TARGET initialization parameters. If frequent resizing is occurring V$SGA_TARGET_ADVICE can be used to ensure that SGA_TARGET is optimally set.  Refer to 鈥淭uning SGA_TARGET Using V$SGA_TARGET_ADVICE (Doc ID 1323708.1)鈥? for more assistance.

2.  Set the minimum value for shared pool or buffer cache to the high watermark size of past resize  assuming that is the peak level .

Ex:    
SQL> SELECT MAX(final_size) FROM V$SGA_RESIZE_OPS WHERE component='shared pool';
SQL> ALTER SYSTEM SET SHARED_POOL_SIZE= scope=both;

3.  Increase the time interval between resizes. To do this, the following parameter can be set dynamically:
  
SQL> alter system set "_memory_broker_stat_interval"=999;
  
This will increase the time between resize to at least 999 seconds and thereby reducing the number of resize operations. The parameter "_memory_broker_stat_interval" is expressed in seconds with a default setting of 30 seconds. If you are running on RAC you will need to do this on all nodes.
 

• Note: 742599.1 - High 'cursor: pin S wait on X' and/or 'library cache lock' Waits. Cause: Shared Pool/Buffer Cache Resize Activity
• Note: 1344228.1 - Severe Performance Problems Due to Sudden drop/resize/shrink of buffer cache
• Note: 1323708.1 - Tuning SGA_TARGET Using V$SGA_TARGET_ADVICE

 
Benefit / Impact:

When the session_cached_cursors database parameter is set to a high value then the number of soft parses will be reduced.  This will reduce the contention in the library cache for sharing the cursors during repeated executions of the SQL statements.

Risk:

The amount of soft parses could increase if the value for the session_cached_cursors parameter is not increased. This may lead to slow performance due to excessive parse calls.

Action / Repair:

The default value for session_cached_cursors is 50.
The parameter value can be modified based on the USAGE from the report.
If the current value is set to less than or equal to 300 and the USAGE is greater than or equal to 100% then the value can be doubled.

Set the parameter value in the parameter file (init.ora or spfile) and restart the database.

SQL> alter system set session_cached_cursors= scope=spfile sid='*';

 

• Note: 62143.1 - Troubleshooting: Tuning the Shared Pool and Tuning Library Cache Latch Contention
• Note: 30804.1 - Init.ora Parameter "SESSION_CACHED_CURSORS" Reference Note

 
Benefit / Impact:

Changes to schema objects could lead to bad execution plan for the SQL statements involving the objects that changed. Investigating the changes may help in identifying the root cause behind the sudden change in the execution plan for the SQL statements or slow application performance.

Risk:

The generation of bad execution plan due to the changes to schema objects might lead to application performance issues.

Action / Repair:

Action/Repair:

When performance issues are noticed on SQL statements, check with the DBA or Application Developers about the changes to the schema objects and take corrective action to revert it back if possible.
 

• Note: 1662591.1 - How To Track The Changes That May Affect Database Performance

 Candidate disk devices were found having incorrect permissions, adding these disks to a diskgroup can lead to unexpected results including database instance crashes.  The proper permissions for ASM devices is 0660 (brw-rw-----).  Refer to the Oracle Grid Infrastructure Installation Guide  for additional details. 
 

 
Benefit / Impact:

If any disk on a diskgroup irrespective of redundancy,have less than 50Mb of free space rebalance operation fails to complete with ORA-15041 

Risk:

As rebalance does not complete successfully ,allocation unit distribution  will not be in symmetrical nature. Results in future allocation unit failure . This causes imbalance within the diskgroup.

Action / Repair:

release space by removing some redundant files from this diskgroup.  
 

• Note: 552082.1 - Script to report the list of files stored in ASM and CURRENTLY NOT OPENED

 
Benefit / Impact:

One or more optimizer related parameters are set to a non-default value.



Risk:

The settings for these parameters should be recommended by the application vendor or by Oracle Support. If any optimizer related parameters are set, the application should be certified and tested to work well with the parameter settings. Ensure that the performance is as expected.

Non default settings of optimizer related parameters may result in performance issues. The settings of such optimizer parameters can have an impact on the entire application performance.

Please Note:  If the database instance is running Siebel/Peoplesoft Applications then you may not need to change any settings, please check your product documentation.



Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.
 

 
Benefit / Impact:

One or more optimizer related parameters are set to a non-default value.



Risk:

The settings for these parameters should be recommended by the application vendor or by Oracle Support. If any optimizer related parameters are set, the application should be certified and tested to work well with the parameter settings. Ensure that the performance is as expected.

Non default settings of optimizer related parameters may result in performance issues. The settings of such optimizer parameters can have an impact on the entire application performance.

Please Note:  If the database instance is running Siebel/Peoplesoft Applications then you may not need to change any settings, please check your product documentation.



Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.
 

 
Benefit / Impact:

An attempt to start a database using one or more obsolete initialization parameters will succeed, but a warning is returned and recorded in the alert log.



Risk:

In most cases setting an obsolete initialization parameter will have minimal to no impact on the database or application. However, there are some cases where this can lead to unexpected behavior or performance issues. Obsolete parameters could also affect the upgrade.



Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.

Obsolete Parameters:

dblink_encrypt_login
hash_join_enabled
log_parallelism
max_rollback_segments
mts_circuits
mts_dispatchers
mts_listener_address
mts_max_dispatchers
mts_max_servers
mts_multiple_listeners
mts_servers
mts_service
mts_sessions
optimizer_max_permutations
oracle_trace_collection_name
oracle_trace_collection_path
oracle_trace_collection_size
oracle_trace_enable
oracle_trace_facility_name
oracle_trace_facility_path
partition_view_enabled
plsql_native_c_compiler
plsql_native_linker
plsql_native_make_file_name
plsql_native_make_utility
row_locking
serializable
transaction_auditing
undo_suppress_errors
enqueue_resources
ddl_wait_for_locks
logmnr_max_persistent_sessions
plsql_compiler_flags
drs_start
gc_files_to_locks
max_commit_propagation_delay
plsql_native_library_dir
plsql_native_library_subdir_count
sql_version
 

• Note: 957543.1 - Health Check Alert: Consider removing obsolete initialization parameters from the PFILE/SPFILE

 The hard nproc shell limit for root should be >= 137524.
 

 
Benefit / Impact:

New optimizer features and fixes are enabled when the OPTIMIZER_FEATURES_ENABLE parameter is set to the current database version.  Setting the OPTIMIZER_FEATURES_ENABLE parameter to a value other than the current database version should only be used as a short term measure at the suggestion of oracle support.



Risk:

By reducing the OPTIMIZER_FEATURES_ENABLE level, new optimizer features and fixes are disabled.  This can potentially have a serious negative impact on performance because it eliminates the possibility of choosing better plans which are only available with the new features and fixes.



Action / Repair:

If Oracle Support has not recommended that you modify this parameter, then you should set the OPTIMIZER_FEATURES_ENABLE parameter to the current database version to ensure that new optimizer features are enabled.  This parameter can be set with the following command:

SQL> alter system set optimizer_features_enable='X.X.X.X' scope=spfile;

Refer to the following MOS document for more details.
 

• Note: 1362332.1 - Use Caution if Changing the OPTIMIZER_FEATURES_ENABLE Parameter After an Upgrade

 
Benefit / Impact:

When Oracle calculates the value of the DB_FILE_MULTIBLOCK_READ_COUNT parameter, the I/O used for table scans will be optimized.  Oracle calculates the value of DB_FILE_MULTIBLOCK_READ_COUNT by using: 
* operating system optimal I/O size
* the size of the buffer cache
* the maximum number of sessions



Risk:

The optimizer will favor large I/Os if you explicitly set the DB_FILE_MULTIBLOCK_READ_COUNT parameter to a large value.  This can affect the performance of your queries.



Action / Repair:

Unset the  DB_FILE_MULTIBLOCK_READ_COUNT parameter in the spfile or init.ora file.  This parameter can also be unset with the following command, but the database needs to be restarted in order for the change to take effect:

SQL> alter system reset db_file_multiblock_read_count scope=spfile sid='*';

Refer to the following MOS document for more details.
 

• Note: 1398860.1 - How is Parameter DB_FILE_MULTIBLOCK_READ_COUNT Calculated?

 
Benefit / Impact:

Setting NLS_SORT to BINARY will result in the ability for your SQL to use standard indexes and queries will be more efficient.



Risk:

Setting NLS_SORT to a value other than BINARY will make regular indexes unusable.  This will negatively affect the performance of your SQL unless you have specific indexes to support the new sorting.



Action / Repair:

Set NLS_SORT to BINARY.   Change the value in the initialization parameter file and then restart the instance. If you do need to have a different sorting in place, then make sure you have the necessary indexes created using a different sorting (NLSSORT function).

Refer to the following MOS document for more details.
 

• Note: 227335.1 - Linguistic Sorting - Frequently Asked Questions

 (1) Online patches are recommended when a downtime cannot be scheduled and the patch needs to be applied urgently
(2) Online patches consume additional memory and if kept permanently the memory consumption increases as the number of processes in the system increases.
(3) It is strongly recommended to rollback all Online patches and replace them with regular (offline) patches on next instance shutdown or the earliest maintenance window.
 

• Note: 761111.1 - RDBMS Online Patching Aka Hot Patching
• Online patching

 For tables with SecureFiles LOB storage, the associated tablespace extent management local allocation type should be AUTOALLOCATE (or SYSTEM managed) to allow Oracle to automatically determine extent size based on data profile
 

• Note: 847951.1 - How to Change Extent Management of a Tablespace from Uniform Size to Autoallocate

 
Benefit / Impact:

Enhanced database stability when using Oracle ASM for database storage



Risk:

In some cases ASM default memory allocation for some customers has proven to be too small leading to ORA-4031 errors which can in turn lead to database instability.  


Action / Repair:

It is a best practice and recommended to use Automatic Memory Management (AMM) for the ASM instance in version 11.2 and above regardless of platform.  AMM is configured in part through the use of the memory_target initialization parameter.

As a result, the default setting for memory_target in 11.2.0.4 and 12.1 versions has been increased to 1024M from the previous default of 256M.  Oracle also advises increasing memory_target to 1024M as a minimum in version 11.2 as a proactive measure to match the new defaults.  No patch is required in order to increase the value of memory_target.
 

 (1) Online patches are recommended when a downtime cannot be scheduled and the patch needs to be applied urgently.
(2) Online patches consume additional memory and if kept permanently the memory consumption increases as the number of processes in the system increases.
(3) It is strongly recommended to rollback all Online patches and replace them with regular (offline) patches on next instance shutdown or the earliest maintenance window.
 

• Note: 761111.1 - RDBMS Online Patching Aka Hot Patching
• Online patching

 
Benefit / Impact:

Improper setting of COREDUMPSIZE could prevent or prolong troubleshooting.



Risk:

The system may not have an adequate COREDUMPSISZE. It is recommended to set the OS core file size to 2GB or greater to aid in troubleshooting issues that generate core dumps (i.e. ORA-7445 errors).



Action / Repair:

A value of 0 means that no core files will be generated. A value between 0 and 2GB could mean that required data for troubleshooting may not be collected.
 

• Note: 567524.1 - Setting Shell Limits For The User "oracle" In Case Of 10gR2 And 11gR1 RDBMS Software

 
Benefit / Impact:

One or more optimizer related parameters are set to a non-default value.



Risk:

The settings for these parameters should be recommended by the application vendor or by Oracle Support. If any optimizer related parameters are set, the application should be certified and tested to work well with the parameter settings. Ensure that the performance is as expected.

Non default settings of optimizer related parameters may result in performance issues. The settings of such optimizer parameters can have an impact on the entire application performance.

Please Note:  If the database instance is running Siebel/Peoplesoft Applications then you may not need to change any settings, please check your product documentation.



Action / Repair:

To modify the parameter value, use the alter session or alter system commands. Refer to the Initialization Parameter chapter of the Oracle Database Reference Documentation for more details on setting these parameters.
 

 
Benefit / Impact:

After cloning a Grid Home the Berkeley Database configuration file ($GI_HOME/crf/admin/crf.ora) in the new home should not be pointing to the previous GI home where it is cloned from. During previous patch set updates Berkeley Database configuration files were found still pointing to the 'before (previously cloned from) home'. It was due an invalid cloning procedure the Berkeley Database location of the 'new home' was not updated during the out of place bundle patching procedure.



Risk:

Berkeley Database configurations still pointing to the old GI home, will fail GI Upgrades to 11.2.0.3. Error messages in $GRID_HOME/log/crflogd/crflogdOUT.log logfile


Action / Repair:

Detect:
cat $GI_HOME/crf/admin/crf`hostname -s`.ora | grep CRFHOME | grep $GI_HOME | wc -l 

cat $GI_HOME/crf/admin/crf`hostname -s`.ora | grep BDBLOC | egrep "default|$GI_HOME | wc -l
 
For each of the above commands, when no '1' is returned, the CRFHOME or BDBLOC as mentioned the crf.ora file has the correct reference to the GI_HOME in it.

To solve this, manually edit $GI_HOME/crf/admin/crf.ora in the cloned Grid Infrastructure Home and change the values for BDBLOC and CRFHOME and make sure none of them point to the previous GI_HOME but to their current home. The same change needs to be done on all nodes in the cluster. It is recommended to set BDBLOC to "default". This needs to be done prior the upgrade.
 

• Note: 1485970.1 - Upgrade 11.2.0.1/11.2.0.2 to 11.2.0.3 Failure Of Rootupgrade Script on previously cloned GI

 Benefit / Impact:

Starting with the 2.6 kernel, for example Red Hat Enterprise Linux 4 or 5, the I/O scheduler can be
changed at boot time which controls the way the kernel commits reads and writes to disks. For more
information on various I/O scheduler, see Choosing an I/O Scheduler for Red Hat Enterprise Linux 4
and the 2.6 Kernel 1


The Completely Fair Queuing (CFQ) scheduler is the default algorithm in Red Hat Enterprise Linux
4 which is suitable for a wide variety of applications and provides a good compromise between
throughput and latency. In comparison to the CFQ algorithm, the Deadline scheduler caps maximum
latency per request and maintains a good disk throughput which is best for disk-intensive database
applications. Hence, the Deadline scheduler is recommended for database systems.

Action / Repair:

Red Hat Enterprise Linux 5 and above allows users to change I/O schedulers dynamically,
one way this can be done is by executing the command echo sched_name > /sys/block/
/queue/scheduler.

 

• Setting the Disk I/O Scheduler on Linux
• Note: 1352304.1 - Linux: Should 'CFQ Scheduler' or 'DEADLINE Scheduler' Be Used In Oracle RAC
• recommended I/O scheduler for a database workload in RHEL

 
Benefit / Impact:

When a Recovery Manager catalog is not used, the initialization parameter "control_file_record_keep_time" controls the period of time for which circular reuse records are maintained within the database control file. RMAN repository records are kept in circular reuse records.  The optimal setting is the maximum number of days in the past that is required to restore and recover a specific database without the use of a RMAN recovery catalog.  Setting this parameter within a recommended range (1 to 9 days) has been shown to address most recovery scenarios by ensuring archive logs and backup records are not prematurely aged out making database recovery much more challenging.    

The impact of verifying that the initialization parameter control_file_record_keep_time value is in the recommended range is minimal. Increasing this value will increase the size of the controlfile and possible query time for backup meta data and archive data.



Risk:

If the control_file_record_keep_time is set to 0, no RMAN repository records are retained in the controlfile, which causes a much more challenging database recovery operation if RMAN recovery catalog is not available.

If the control_file_record_keep_time is set too high, problems can arise with space management within the control file, expansion of the control file, and control file contention issues.



Action / Repair:

To verify that the FRA space management function is not blocked, as the owner userid of the oracle home with the environment properly set for the target database, execute the following command set:

CF_RECORD_KEEP_TIME="";
CF_RECORD_KEEP_TIME=$(echo -e "set heading off feedback off\n select value from V\$PARAMETER where name = 'control_file_record_keep_time';" | $ORACLE_HOME/bin/sqlplus -s "/ as sysdba");
if [[ $CF_RECORD_KEEP_TIME -ge "1" && $CF_RECORD_KEEP_TIME -le "9" ]]
then echo -e "\nPASS:  control_file_record_keep_time is within recommended range [1-9]:" $CF_RECORD_KEEP_TIME;
elif [ $CF_RECORD_KEEP_TIME -eq "0" ]
then echo -e "\nFAIL:  control_file_record_keep_time is set to zero:" $CF_RECORD_KEEP_TIME;
else echo -e "\nWARNING:  control_file_record_keep_time is not within recommended range [1-9]:" $CF_RECORD_KEEP_TIME;
fi;

The expected output should be:

PASS:  control_file_record_keep_time is within recommended range [1-9]: 7

If the output is not as expected, investigate and correct the condition(s).

NOTE: The use of an RMAN recovery catalog is recommended as the best way to avoid the loss of RMAN metadata because of overwritten control file records.
 

• Oracle庐 Database Backup and Recovery User's Guide 11g Release 2 (11.2)
• Initialization parameter CONTROL_FILE_RECORD_KEEP_TIME

 
Benefit / Impact:

The control file is a binary file that records the physical structure of the database and contains important meta data required to recover the database. The database cannot startup or stay up unless all control files are valid. When a Recovery Manager catalog is not used, the control file is needed for database recovery because it contains all backup and recovery meta data.

The impact of verifying and setting "CONTROLFILE AUTOBACKUP" to "ON" is minimal. 



Risk:

When a Recovery Manager catalog is not used, loss of the controlfile results in loss of all backup and recovery meta data, which causes a much more challenging database recovery operation


Action / Repair:

To verify that RMAN "CONTROLFILE AUTOBACKUP" is set to "ON", as the owner userid of the oracle home with the environment properly set for the target database, execute the following command set:

RMAN_AUTOBACKUP_STATUS="";
RMAN_AUTOBACKUP_STATUS=$(echo -e "set heading off feedback off\n select value from V\$RMAN_CONFIGURATION where name = 'CONTROLFILE AUTOBACKUP';" | $ORACLE_HOME/bin/sqlplus -s "/ as sysdba");
if [ -n "$RMAN_AUTOBACKUP_STATUS" ] && [ "$RMAN_AUTOBACKUP_STATUS" = "ON" ]
then echo -e "\nPASS:  RMAN "CONTROLFILE AUTOBACKUP" is set to \"ON\":" $RMAN_AUTOBACKUP_STATUS;
else
echo -e "\nFAIL:  RMAN "CONTROLFILE AUTOBACKUP" should be set to \"ON\":" $RMAN_AUTOBACKUP_STATUS;
fi;

The expected output should be:

PASS:  RMAN CONTROLFILE AUTOBACKUP is set to "ON": ON

If the output is not as expected, investigate and correct the condition(s).

For additional information, review information on CONFIGURE syntax in Oracle脗庐 Database Backup and Recovery Reference 11g Release 2 (11.2).

RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON;

NOTE: Oracle MAA also recommends periodically backing up the controlfile to trace as additional backup.

SQL> ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
 

 ASM disk groups are registered in clusterware by automatic,after you mount them. please make sure required ASM disk groups are mounted.
 

 Install the operating system package cvuqdisk. Without cvuqdisk, Cluster Verification Utility cannot discover shared disks, and you receive the error message "Package cvuqdisk not installed" when you run Cluster Verification Utility. Use the cvuqdisk rpm for your hardware (for example, x86_64, or i386).
 

 Any Kind of OLR corruption should be remedied before attempting upgrade otherwise 11.2 GI rootupgrade.sh fails with "Invalid  OLR during upgrade"
 

• Note: 1544774.1 - 11.2 GI rootupgrade.sh fails with "Invalid OLR during upgrade.."
• Note: 1193643.1 - How to backup or restore OLR in 11.2 Grid Infrastructure

 This is required to make the shell limits work properly and applies to 10g,11g and 12c.  

Add the following line to the /etc/pam.d/login file, if it does not already exist:

session    required     pam_limits.so

 

• Setting Shell Limits for the oracle User

 
Benefit / Impact:

Maintaining vm.min_free_kbytes >= 524288 (512MB) && <= 1048576 (1GB)  helps a Linux system to reclaim memory faster and avoid LowMem pressure issues which can lead to node eviction or other outage or performance issues.

The impact of verifying vm.min_free_kbytes >= 524288 && <= 1048576 is minimal. The impact of adjusting the parameter should include editing the /etc/sysctl.conf file and rebooting the system. It is possible, but not recommended, especially for a system already under LowMem pressure, to modify the setting interactively. However, a reboot should still be performed to make sure the interactive setting is retained through a reboot.



Risk:

Exposure to unexpected node eviction and reboot.



Action / Repair:

To verify that vm.min_free_kbytes is properly set to  >= 524288 && <= 1048576 execute the following command

/sbin/sysctl -n vm.min_free_kbytes

cat /proc/sys/vm/min_free_kbytes

If the output is not as expected, investigate and correct the condition. For example if the value is incorrect in /etc/sysctl.conf but current memory matches the incorrect value, simply edit the /etc/sysctl.conf file to include the line "vm.min_free_kbytes  >= 524288 && <= 1048576" and reboot the node. 
 

• Note: 1367153.1 - Top 5 Issues That Cause Node Reboots or Evictions or Unexpected Recycle of CRS

 
Benefit / Impact:

Linux transparent huge pages are enabled by default in OEL 6 and SuSE 11 which might cause soft lockup of CPUs and make system unresponsive which can cause node eviction.

if AnonHugePages are found and value is greater than 0 in /proc/meminfo, means transparent huge pages are enabled. 

Risk:

Because Transparent HugePages are known to cause unexpected node reboots and performance problems with RAC, Oracle strongly advises to disable the use of Transparent HugePages. In addition, Transparent Hugepages may cause problems even in a single-instance database environment with unexpected performance problems or delays. As such, Oracle recommends disabling Transparent HugePages on all Database servers running Oracle

Action / Repair:

To turn this feature off,put this in /etc/rc.local:-

echo never > /sys/kernel/mm/transparent_hugepage/enabled 
echo never > /sys/kernel/mm/transparent_hugepage/defrag 
 

• Note: 1557478.1 - RAC - Disable Transparent HugePages on SLES11, RHEL6 and OEL6
• Disabling transparent hugepages (THP) on Red Hat Enterprise Linux 6 is not taking effect

 
Benefit / Impact:

When you perform a DML or DDL operation using the NOLOGGING or UNRECOVERABLE clause, database backups made prior to the unrecoverable operation are invalidated and new backups are required. You can specify the SQL ALTER DATABASE or SQL ALTER TABLESPACE statement with the FORCE LOGGING clause to override the NOLOGGING setting; however, this statement will not repair a database that is already invalid.



Risk:

Changes under NOLOGGING will not be available after executing database recovery from a backup made prior to the unrecoverable change.


Action / Repair:

To verify that the data files are recoverable, execute the following Sqlplus command as the userid that owns the oracle home for the database:
select file#, unrecoverable_time, unrecoverable_change# from v$datafile where unrecoverable_time is not null;
If there are any unrecoverable actions, the output will be similar to:
     FILE# UNRECOVER UNRECOVERABLE_CHANGE#
---------- --------- ---------------------
        11 14-JAN-13               8530544
If nologging changes have occurred and the data must be recoverable then a backup of those datafiles that have nologging operations within should be done immediately. Please consult the following sections of the Backup and Recovery User guide for specific steps to resolve files that have unrecoverable changes

The standard best practice is to enable FORCE LOGGING at the database level (ALTER DATABASE FORCE LOGGING;) to ensure that all transactions are recoverable. However, placing the a database in force logging mode for ETL operations can lead to unnecessary database overhead. MAA best practices call for isolating data that does not need to be recoverable. Such data would include:

Data resulting from temporary loads
Data resulting from transient transformations
Any non critical data

To reduce unnecessary redo generation, do the following:

Specifiy FORCE LOGGING for all tablespaces that you explicitly wish to protect (ALTERTABLESPACE FORCE LOGGING;)
Specify NO FORCE LOGGING for those tablespaces that do not need protection (ALTERTABLESPACE NO FORCE LOGGING;).
Disable force logging at the database level (ALTER DATABASE NO FORCE LOGGING;) otherwise the database level settings will override the tablespace settings.

Once the above is complete, redo logging will function as follows:

Explicit no logging operations on objects in the no logging tablespace will not generate the normal redo (a small amount of redo is always generated for no logging operations to signal that a no logging operation was performed).

All other operations on objects in the no logging tablespace will generate the normal redo.
Operations performed on objects in the force logging tablespaces always generate normal redo.

Note:-Please seek oracle support assistance to mitigate this problem. Upon their guidance, the following commands could help validate, identify corrupted blocks.

              oracle> dbv file= userid=sys/******
              RMAN> validate check logical database;
              SQL> select COUNT(*) from v$database_block_corruption;

 

• Backing Up Database Files with RMAN
• Recovering NOLOGGING Tables and Indexes

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Starting with Oracle 11gR2, our recommendation is to use Oracle ASM to store OCR and Voting Disks. With appropriate redundancy level (HIGH or NORMAL) of the ASM Disk Group being used, Oracle can create required number of Voting Disks as part of installation
 

• Adding, Replacing, Repairing, and Removing Oracle Cluster Registry Locations
• Adding, Deleting, or Migrating Voting Disks

 This audit check captures information related to Oracle Parallel Query (PQ), DOP, PQ/PX Statistics, Database Resource Plans, Consumers Groups etc. This is primarily for Oracle Support Team consumption. However, customers may also review this to identify/troubleshoot related problems.
For every database, there will be a zip file of format  in raccheck output directory. 
 

 /etc/init.d/halt file is called when system is rebooted or halt. this file must have instructions to synchronize system time to hardware clock.

it should have commands like 

[ -x /sbin/hwclock ] && action $"Syncing hardware clock to system time" /sbin/hwclock $CLOCKFLAGS
 

 TFA Collector (aka TFA) is a diagnostic collection utility to simplify  diagnostic data collection on Oracle Clusterware/Grid Infrastructure and RAC  systems.  TFA is similar to the diagcollection utility packaged with Oracle  Clusterware in the fact that it collects and packages diagnostic data however  TFA is MUCH more powerful than diagcollection with its ability to centralize  and automate the collection of diagnostic information. This helps speed up  the data collection and upload process with Oracle Support, minimizing delays  in data requests and analysis.
TFA provides the following key benefits:- 
  - Encapsulates diagnostic data collection for all CRS/GI and RAC components  on all cluster nodes into a single command executed from a single node 
  - Ability to "trim" diagnostic files during data collection to reduce data  upload size 
  - Ability to isolate diagnostic data collection to a given time period 
  - Ability to centralize collected diagnostic output to a single server in  the cluster 
  - Ability to isolate diagnostic collection to a particular product  component, e.g. ASM, RDBMS, Clusterware 
  - Optional Real Time Scan Alert Logs for conditions indicating a problem (DB 
  - Alert Logs, ASM Alert Logs, Clusterware Alert Logs, etc) 
  - Optional Automatic Data Collection based off of Real Time Scan findings 
  - Optional On Demand Scan (user initialted) of all log and trace files for  conditions indicating a problem 
  - Optional Automatic Data Collection based off of On Demand Scan findings 
 

• Note: 1513912.1 - TFA Collector- The Preferred Tool for Automatic or ADHOC Diagnostic Gathering Across All Cluster Nodes
• Trace File Analyzer Collector (TFA)

 Resources were found to be in an UNKNOWN state on the system.  Having  resources in this state often results in issues when upgrading.  It is  recommended to correct resources in an UNKNOWN state prior to upgrading.   

 

• Note: 845709.1 - Resource in UNKNOWN state and srvctl Cannot Start/Stop the Resource

 ORA-15196 errors means ASM encountered an invalid metadata block. Please see the trace file for more information next to ORA-15196 error in ASM alert log.  If this is an old error, you can ignore this finding otherwise open service request with Oracle support to find the cause and fix it


 

 The GROUP_NUMBER and DISK_NUMBER columns in GV$ASM_DISK will only be valid if the disk is part of a disk group which is currently mounted by the instance. Otherwise, GROUP_NUMBER will be 0, and DISK_NUMBER will be a unique value with respect to the other disks that also have a group number of 0. Run following query to find out the disks which are not part of any disk group.

select name,path,HEADER_STATUS,GROUP_NUMBER  from gv$asm_disk where group_number=0;
 

 
Benefit / Impact:

Various Oracle processes use the loopback address for checking the status of a localhost.

Risk:

Incorrect loopback address configuration in /etc/hosts may result in failure of those processes.

Action / Repair:

IP address 127.0.0.1 should only map to localhost and/or localhost.localdomain, not anything else.  

Correct configuration for the loopback address in /etc/hosts is as follows:

127.0.0.1 localhost.localdomain localhost
or
127.0.0.1 localhost  localhost.localdomain

The following configuration would be considered incorrect:

127.0.0.1  localhost  myhost.com  localhost.localdomain
or
127.0.0.1  myhost.com  localhost  localhost.localdomain

Where myhost.com is the actual hostname or network identity of the host and which should map to the public IP address

In other words, the actual short hostname and/or the fully qualified hostname should not be configured at all for the loopback address.
 
See the below referenced documents for more details on this subject.
 

• Note: 315125.1 - What to Check If The Cluster Synchronization Service Daemon (OCSSD) Does not Start
• Note: 315720.1 - TNS Listener Fails To Start With Errors: TNS-12537...Error: 29: Illegal seek
• Note: 1054902.1 - How to Validate Network and Name Resolution Setup for the Clusterware and RAC

 When the latency hits 500ms, a Warning message is written to the lgwr trace file(s). For example:

Warning: log write elapsed time 564ms, size 2KB

Even though this threshold is very high and latencies below this range could impact the application performance, it is still worth to capture and report it to customers for necessary action.The performance impact of LGWR latencies include commit delays,Broadcast-on-Commit delays etc.
 

• Note: 601316.1 - LGWR Is Generating Trace file with "Warning: Log Write Time 540ms, Size 5444kb" In 10.2.0.4 Database

 all public and private interconnect network cards should be able to arping to all remote nodes in cluster.

For example using public network card, arping remote node using following command and output should be "Received 1 response(s)"

/sbin/arping -b -f -c 1 -w 1 -I eth1 nodename2.

Here eth1 is public network interface and nodename2 is second node in cluster.

 

• Broadcast Requirements for Networks Used by Oracle Grid Infrastructure

 Oracle 11g and higher Active Data Guard is the real-time data protection and availability solution that eliminates single point of failure by maintaining one or more synchronized physical replicas of the production database. If an unplanned outage of any kind impacts the production database, applications and users can quickly failover to a synchronized standby, minimizing downtime and preventing data loss. An Active Data Guard standby can be used to offload read-only applications, ad-hoc queries, and backups from the primary database or be dual-purposed as a test system at the same time it provides disaster protection. An Active Data Guard standby can also be used to minimize downtime for planned maintenance when upgrading to new Oracle Database patch sets and releases and for select migrations.  
 
For zero data loss protection and fastest recovery time, deploy a local Data Guard standby database with Data Guard Fast-Start Failover and integrated client failover. For protection against outages impacting both the primary and the local standby or the entire data center, or a broad geography, deploy a second Data Guard standby database at a remote location.

Key HA Benefits:

With Oracle 11g release 2 and higher Active Data Guard and real time apply, data block corruptions can be repaired automatically and downtime can be reduced from hours and days of application impact to zero downtime with zero data loss.

With MAA best practices, Data Guard Fast-Start Failover (typically a local standby) and integrated client failover, downtime from database, cluster and site failures can be reduced from hours to days and seconds and minutes.

With remote standby database (Disaster Recovery Site), you have protection from complete site failures.

In all cases, the Active Data Guard instances can be active and used for other activities.

Data Guard can reduce risks and downtime for planned maintenance activities by using Database rolling upgrade with transient logical standby, standby-first patch apply and database migrations.

Active Data Guard provides optimal data protection by using physical replication and comprehensive Oracle validation to maintain an exact byte-for-byte copy of the primary database that can be open read-only to offload reporting, ad-hoc queries and backups. For other advanced replication requirements where read-write access to a replica database is required while it is being synchronized with the primary database see Oracle GoldenGate logical replication.Oracle GoldenGate can be used to support heterogeneous database platforms and database releases, an effective read-write full or subset logical replica and to reduce or eliminate downtime for application, database or system changes. Oracle GoldenGate flexible logical replication solution鈥檚 main trade-off is the additional administration for application developer and database administrators.
 

• Oracle Data Guard: Disaster Recovery for Oracle Exadata Database Machine
• Note: 1265700.1 - Oracle Patch Assurance - Data Guard Standby-First Patch Apply
• Note: 1302539.1 - Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration
• HABP: Chapter 9: Configuring Oracle Data Guard
• Goldengate resources

 In order to reduce contention to the data dictionary, rollback data, and reduce the amount of generated redo, locally managed tablespaces should be used rather than dictionary managed tablespaces.Please refer to the below referenced notes for more information about benefits of locally managed tablespace and how to migrate a tablesapce from dictionary managed to locally managed.
 

• Use Locally Managed Tablespaces
• Note: 105120.1 - Advantages of Using Locally Managed vs Dictionary Managed Tablespaces

 Starting with Oracle 9i Auto Segment Space Management (ASSM) can be used by specifying the SEGMENT SPACE MANAGEMENT clause, set to AUTO in the CREATE TABLESPACE statement. Implementing the ASSM feature allows Oracle to use bitmaps to manage the free space within segments. The bitmap describes the status of each data block within a segment, with respect to the amount of space in the block available for inserting rows. The current status of the space available in a data block is reflected in the bitmap allowing for Oracle to manage free space automatically with ASSM. ASSM tablespaces automate freelist management and remove the requirement/ability to specify PCTUSED, FREELISTS, and FREELIST GROUPS storage parameters for individual tables and indexes created in these tablespaces. 
 

• Use Automatic Segment Space Management

 Its recommended to set default temporary tablespace at database level to achieve optimal performance for queries which requires sorting the data.
 

• Use Temporary Tablespaces and Specify a Default Temporary Tablespace

 Running the database in ARCHIVELOG mode and using database FORCE LOGGING mode are prerequisites for database recovery operations. The ARCHIVELOG mode enables online database backup and is necessary to recover the database to a point in time later than what has been restored. Features such as Oracle Data Guard and Flashback Database require that the production database run in ARCHIVELOG mode.
 

• Set the Database ARCHIVELOG Mode and FORCE LOGGING Mode

 
Benefit / Impact:

With AUDIT_TRAIL set for database (AUDIT_TRAIL=db), and the AUD$ and FGA_LOG$ tables located in a dictionary segment space managed SYSTEM tablespace, "gc" wait events are sometimes observed during heavy periods of database logon activity. Testing has shown that under such conditions, placing the AUD$ and FGA_LOG$ tables in the SYSAUX tablespace, which uses automatic segment space management, reduces the space related wait events.

The impact of verifying that the AUD$ and FGA_LOG$ tables are in the SYSAUX table space is low. Moving them if they are not located in the SYSAUX does not require an outage, but should be done during a scheduled maintenance period or slow audit record generation window.



Risk:

If AUD$ and FGA_LOG$ tables are not verifed to use automatic segment space management, there is a risk of a performance slowdown during periods of high database login activity.


Action / Repair:

To verify the segment space management policy currently in use by the AUD$ and FGA_LOG$ tables, use the following Sqlplus command:

select t.table_name,ts.segment_space_management from dba_tables t, dba_tablespaces ts where ts.tablespace_name = t.tablespace_name and t.table_name in ('AUD$','FGA_LOG$');

The output should be:
TABLE_NAME                     SEGMEN
------------------------------ ------
FGA_LOG$                       AUTO
AUD$                           AUTO 
If one or both of the AUD$ or FGA_LOG$ tables return "MANUAL", use the DBMS_AUDIT_MGMT package to move them to the SYSAUX tablespace:

BEGIN
DBMS_AUDIT_MGMT.set_audit_trail_location(audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_AUD_STD,--this moves table AUD$ 
audit_trail_location_value => 'SYSAUX');  
END;  
/

BEGIN
DBMS_AUDIT_MGMT.set_audit_trail_location(audit_trail_type => DBMS_AUDIT_MGMT.AUDIT_TRAIL_FGA_STD,--this moves table FGA_LOG$ 
audit_trail_location_value => 'SYSAUX');
END;
/   

The output should be similar to:
PL/SQL procedure successfully completed. 

If the output is not as above, investigate and correct the condition.
NOTE: This "DBMS_AUDIT_MGMT.set_audit_trail" command should be executed as part of the dbca template post processing scripts, but for existing databases, the command can be executed, but since it moves the AUD$ & FGA_LOG$ tables using "alter table ... move" command, it should be executed at a "quiet" time
 

• Use Automatic Segment Space Management

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Read errors can be the result of a loss of access to the entire disk or media corruptions on an otherwise a healthy disk. ASM tries to recover from read errors on corrupted sectors on a disk. When a read error by the database or ASM triggers the ASM instance to attempt bad block remapping, ASM reads a good copy of the extent and copies it to the disk that had the read error.

If the write to the same location succeeds, then the underlying allocation unit (sector) is deemed healthy. This might be because the underlying disk did its own bad block reallocation.

If the write fails, ASM attempts to write the extent to a new allocation unit on the same disk. If this write succeeds, the original allocation unit is marked as unusable. If the write fails, the disk is taken offline.

One unique benefit on ASM-based mirroring is that the database instance is aware of the mirroring. For many types of logical corruptions such as a bad checksum or incorrect System Change Number (SCN), the database instance proceeds through the mirror side looking for valid content and proceeds without errors. If the process in the database that encountered the read is in a position to obtain the appropriate locks to ensure data consistency, it writes the correct data to all mirror sides.

When encountering a write error, a database instance sends the ASM instance a disk offline message.

If database can successfully complete a write to at least one extent copy and receive acknowledgment of the offline disk from ASM, the write is considered successful.

If the write to all mirror side fails, database takes the appropriate actions in response to a write error such as taking the tablespace offline.

When the ASM instance receives a write error message from an database instance or when an ASM instance encounters a write error itself, ASM instance attempts to take the disk offline. ASM consults the Partner Status Table (PST) to see whether any of the disk's partners are offline. If too many partners are already offline, ASM forces the dismounting of the disk group. Otherwise, ASM takes the disk offline.

The ASMCMD remap command was introduced to address situations where a range of bad sectors exists on a disk and must be corrected before ASM or database I/O
 

 The V$DATABASE_BLOCK_CORRUPTION view displays blocks marked corrupt by Oracle Database components such as RMAN commands, ANALYZE, dbv, SQL queries, and so on. Any process that encounters a corrupt block records the block corruption in this view.  Repair techniques include block media recovery, restoring data files, recovering with incremental backups, and block newing. Block media recovery can repair physical corruptions, but not logical corruptions. It is also recommended to use RMAN 鈥淐HECK LOGICAL鈥? option to check for data block corruptions periodically. Please consult the Oracle庐 Database Backup and Recovery User's Guide for repair instructions
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 Please review MOS Note 169706.1 - Oracle Database on Unix AIX,HP-UX,Linux,Mac OS X,Solaris,Tru64 Unix Operating Systems Installation and Configuration Requirements Quick Reference (8.0.5 to 11.2)
 

 For Oracle Database 11g Release 2, the REMOTE_LISTENER parameter should be set to the SCAN. This allows the instances to register with the SCAN Listeners to provide information on what services are being provided by the instance, the current load, and a recommendation on how many incoming connections should be directed to the
instance.
 

• SINGLE CLIENT ACCESS NAME (SCAN)

 If value of remote_listener parameter is set to non-pingable tnsnames,instances will not be cross registered and will not balance the load across cluster.In case of node or instance failure, connections may not failover to serviving node. For more information about remote_listener,load balancing and failover.

 

• Note: 453293.1 - 10g & 11g :Configuration of TAF(Transparent Application Failover) and Load Balancing

 
Benefit / Impact:

Database instance SCAN listener registration, connection failover and load balancing implications



Risk:

Potential problems with database instances registering with SCAN listeners, connection load balancing and fail over issues

Action / Repair:

No server-side TNSNAMES.ora should have an alias resolving using the SCAN EZ-syntax,eg., 
 
myscan.oracle.com:1521= (DESCRIPTION= (ADDRESS_LIST=(ADDRESS=鈥?

If any such syntax exists, for whatever reason such as third party application configuration, it should be removed. 
 

 EZCONNECT eliminates the need for service name lookups in tnsnames.ora files when connecting to an Oracle database across a TCP/IP network. In fact, no naming or directory system is required when using this method.It extends the functionality of the host naming method by enabling clients to connect to a database with an optional port and service name in addition to the host name of the database.
 

• Note: 1070607.1 - 11.2 Scan and Node TNS Listener Setup Examples

 The LOCAL_LISTENER parameter should be set to the node VIP. If you need fully qualified domain names, ensure that LOCAL_LISTENER is set to the fully qualified domain name (node-vip.mycompany.com). By default a local listener is created during cluster configuration that runs out of the grid infrastructure home and listens on the specified port(default is 1521) of the node VIP.
 

• SINGLE CLIENT ACCESS NAME (SCAN)

 
Benefit / Impact: 

Experience and testing has shown that certain database initialization parameters should be set at specific values. These are the best practice values set at deployment time. By setting these database initialization parameters as recommended, known problems may be avoided and performance maximized.
The parameters are common to all database instances. The impact of setting these parameters is minimal.
The performance related settings provide guidance to maintain highest stability without sacrificing performance. Changing the default performance settings can be done after careful performance evaluation and clear understanding of the performance impact.

Risk: 

If

Risk: 

If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization parameter is not set as recommended, and the actual set value.


Action / Repair: 

PARALLEL_EXECUTION_MESSAGE_SIZE = 16384 Improves Parallel Query performance
 

• Parallel execution message size from Oracle Docs
• Note: 810394.1 - RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent)

 Ways to troubleshoot database hang and deadlocks:- 

1. V$Wait_Chains - The DB (the dia0 BG process) samples local hanganalyze every 3 seconds and global hanganalyze every 10 seconds and stores it in memory.  V$Wait_Chains is an interface to seeing this "hanganalyze cache".  That means at any moment you can query v$wait_chains and see what hanganalyze knows about the current wait chains at any given time.  In 11.2 with a live hang this is the first thing you can use to know who the blocker and final blockers are.For more info see following: NOTE:1428210.1 - Troubleshooting Database Contention With V$Wait_Chains

2. Procwatcher - In v11, this script samples v$wait_chains every 90 seconds and collects interesting info about the processes involved in wait chains (short stacks, current wait, current SQL, recent ASH data, locks held, locks waiting for, latches held, etc...).  This script works in RAC and non-RAC and is a proactive way to trap hang data even if you can't predict when the problem will happen.  Some very large customers are proactively either planning to, or using this script on hundreds of systems to catch session contention.  For more info see followings: NOTE:459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes AND NOTE:1352623.1 - How To Troubleshoot Database Contention With Procwatcher.

3. Hanganalyze Levels - Hanganalyze format and output is completely different starting in version 11.  In general we recommend getting hanganalyze dumps at level 3. Make sure you always get a global hanganalyze in RAC.

4. Systemstate Levels - With a large SGA and a large number of processes, systemstate dumps at level 266 or 267 can dump a HUGE amount of data and take even hours to dump on large systems.  That situation should be avoided.  One lightweight alternative is a systemstate dump at level 258.  This is basically a level 2 systemstate plus short stacks and is much cheaper than level 266 or 267 and level 258 still has the most important info that support engineers typically look at like process info, latch info, wait events, short stacks, and more at a fraction of the cost.

Note that bugs 11800959 and 11827088 have significant impact on systemstate dumps.  If not on 11.2.0.3+ or a version that has both fixes applied, systemstate dumps at levels 10, 11, 266, and 267 can be VERY expensive in RAC.  In versions < 11.2.0.3 without these fixes applied, systemstate dumps at level 258 would typically be advised. 

NOTE:1353073.1 - Exadata Diagnostic Collection Guide while for Exadata many of the concepts for hang detection and analysis are the same for normal RAC systems.

5.Hang Management and LMHB provide good proactive hang related data.  For Hang Management see following note: 1270563.1 Hang Manager 11.2.0.2
 

• Note: 459694.1 - Procwatcher: Script to Monitor and Examine Oracle DB and Clusterware Processes
• Note: 1352623.1 - How To Troubleshoot Database Contention With Procwatcher
• Note: 1428210.1 - Troubleshooting Database Contention With V$Wait_Chains
• Note: 1353073.1 - Exadata Diagnostic Collection Guide

 
Benefit / Impact: 
  
Experience and testing has shown that certain database initialization parameters should be set at specific values. These are the best practice  values set at deployment time. By setting these database initialization  parameters as recommended, known problems may be avoided and performance  maximized. 
The parameters are common to all database instances. The impact of setting  these parameters is minimal.  The performance related settings provide guidance to maintain highest stability without sacrificing performance. Changing the default performance  settings can be done after careful performance evaluation and clear understanding of the performance impact. 
  


Risk: 
  
If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization  parameter is not set as recommended, and the actual set value. 
  

Action / Repair: 
  
"RECYCLEBIN = ON" provides higher availability by enabling the Flashback Drop  feature. "ON" is the default value and should not be changed. 

 

 Result: A - SCN Headroom is good.

Apply the latest recommended CPU or PSU patches based on your maintenance schedule.

For further information review the referenced MOS document.
 

• Note: 1393363.1 - Installing, Executing and Interpreting output from the "scnhealthcheck.sql" script

 
Benefit / Impact:

To optimize run time performance for write/redo generation intensive workloads.  Increasing fast_start_mttr_target from the default will reduce checkpoint writes from DBWR processes, making more room for LGWR IO.



Risk:

Performance implications if set too aggressively (lower setting = more aggressive), but a trade-off between performance and availability.  This trade-off and the type of workload needs to be evaluated and a decision made whether the default is needed to meet RTO objectives.  fast_start_mttr_target should be set to the desired RTO (Recovery Time Objective) while still maintaing performance SLAs. So this needs to be evaluated on a case by case basis.


Action / Repair:

Consider increasing fast_start_mttr_target to 300 (five minutes) from the default. The trade-off is that instance recovery will run longer, so if instance recovery is more important than performance, then keep fast_start_mttr_target at the default.

Keep in mind that an application with inadequately sized redo logs will likley not see an affect from this change due to frequent log switches so follow best practices for sizing redo logs.

Considerations for a direct writes in a data warehouse type of application: Even though direct operations aren't using the buffer cache, fast_start_mttr_target is very effective at controlling crash recovery time because it ensures adequate checkpointing for the few buffers that are resident (ex: undo segment headers).
 

 Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area.  Application monitoring is required for early detection.  Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database).

Key HA Benefits:

With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency.

Fast repair for logical failures caused by malicious or accidental DML or DDL operations.

Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, or database.
 
Questions:

Can your application or monitoring infrastructure detect logical inconsistencies?

Is your operations team prepared to use various flashback technologies to repair quickly and efficiently?

Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?
 

 
Benefit / Impact:

"MAXBYTES" is the SQL attribute that expresses the "MAXSIZE" value that is used in the DDL command to set "AUTOEXTEND" to "ON". By default, for a bigfile tablespace, the value is "3.5184E+13", or "35184372064256". The benefit of having "MAXBYTES" set at a non-default value for "BIGFILE" tablespaces is that a runaway operation or heavy simultaneous use (e.g., temp tablespace) cannot take up all the space in a diskgroup.

The impact of verifying that "MAXBYTES" is set to a non-default value is minimal. The impact of setting the "MAXSIZE" attribute to a non-default value "varies depending upon if it is done during database creation, file addition to a tablespace, or added to an existing file.



Risk:

The risk of running out of space in a diskgroup varies by application and cannot be quantified here. A diskgroup running out of space may impact the entire database as well as ASM operations (e.g., rebalance operations).


Action / Repair:

To obtain a list of file numbers and bigfile tablespaces that have the "MAXBYTES" attribute at the default value, enter the following sqlplus command logged into the database as sysdba:
select file_id, a.tablespace_name, autoextensible, maxbytes
from (select file_id, tablespace_name, autoextensible, maxbytes from dba_data_files where autoextensible='YES' and maxbytes = 35184372064256) a, (select tablespace_name from dba_tablespaces where bigfile='YES') b
where a.tablespace_name = b.tablespace_name
union
select file_id,a.tablespace_name, autoextensible, maxbytes
from (select file_id, tablespace_name, autoextensible, maxbytes from dba_temp_files where autoextensible='YES' and maxbytes = 35184372064256) a, (select tablespace_name from dba_tablespaces where bigfile='YES') b
where a.tablespace_name = b.tablespace_name;

The output should be:no rows returned 

If you see output similar to:

   FILE_ID TABLESPACE_NAME                AUT   MAXBYTES
---------- ------------------------------ --- ----------
         1 TEMP                           YES 3.5184E+13
         3 UNDOTBS1                       YES 3.5184E+13
         4 UNDOTBS2                       YES 3.5184E+13

Investigate and correct the condition.
 

 Oracle clusterware is required for complete client failover integration.  Please consult the following whitepaper for further information
 

• Client Failover Best Practices for Highly Available Oracle Databases: Oracle Database 11g Release 2

 Oracle Flashback Technology enables fast logical failure repair. Oracle recommends that you use automatic undo management with sufficient space to attain your desired undo retention guarantee, enable Oracle Flashback Database, and allocate sufficient space and I/O bandwidth in the fast recovery area.  Application monitoring is required for early detection.  Effective and fast repair comes from leveraging and rehearsing the most common application specific logical failures and using the different flashback features effectively (e.g flashback query, flashback version query, flashback transaction query, flashback transaction, flashback drop, flashback table, and flashback database).

Key HA Benefits:

With application monitoring and rehearsed repair actions with flashback technologies, application downtime can reduce from hours and days to the time to detect the logical inconsistency.

Fast repair for logical failures caused by malicious or accidental DML or DDL operations.

Effect fast point-in-time repair at the appropriate level of granularity: transaction, table, or database.
 
Questions:

Can your application or monitoring infrastructure detect logical inconsistencies?

Is your operations team prepared to use various flashback technologies to repair quickly and efficiently?

Is security practices enforced to prevent unauthorized privileges that can result logical inconsistencies?
 

• HABP: Chapter 5.1.4: Enable Flashback Database

 
Benefit / Impact:

Intially db_block_checking is set to off due to potential performance impact. Performance testing is particularly important given that overhead is incurred on every block change. Block checking typically causes 1% to 10% overhead, but for update and insert intensive applications (such as Redo Apply at a standby database) the overhead can be much higher. OLTP compressed tables also require additional checks that can result in higher overhead depending on the frequency of updates to those tables. Workload specific testing is required to assess whether the performance overhead is acceptable.


Risk:

If 

Risk:

If the database initialization parameters are not set as recommended, a variety of issues may be encountered, depending upon which initialization parameter is not set as recommended, and the actual set value.


Action / Repair:

Based on performance testing results set the primary or standby database to either medium or full depending on the impact. If performance concerns prevent setting DB_BLOCK_CHECKING to either FULL or MEDIUM at a primary database, then it becomes even more important to enable this at the standby database. This protects the standby database from logical corruption that would be undetected at the primary database.
For higher data corruption detection and prevention, enable this setting but performance impacts vary per workload.Evaluate performance impact.

 

• Note: 1302539.1 - Best Practices for Corruption Detection, Prevention, and Automatic Repair - in a Data Guard Configuration
• Protect Against Data Corruption

• Note: 68304.1 - WHAT IS UMASK
• Configuring the oracle User's Environment

 
Benefit / Impact:

The audit file destination directories for an ASM instance can grow to contain a very large number of files if they are not regularly maintained. Use the Linux cron(8) utility and the find(1) command to manage the number of files in the audit file destination directories.

The impact of using cron(8) and find(1) to manage the number of files in the audit file destination directories is minimal.



Risk:

Having a very large number of files can cause the file system to run out of free disk space or inodes, or can cause Oracle to run very slowly due to file system directory scaling limits, which can have the appearance that the ASM instance is hanging on startup.



Action / Repair:

Refer to the referenced MOS Note
 

• Note: 1298957.1 - Manage Audit File Directory Growth with cron

 The hard stack shell limit for the Oracle Grid Infrastructure software install owner should be >= 10240.

What's being checked here is the /etc/security/limits.conf file as documented in 11gR2 Grid Infrastructure Installation Guide, section 2.15.3 Setting Resource Limits for the Oracle Software Installation Users.  

If the /etc/security/limits.conf file is not configured as described in the documentation then to check the hard stack configuration while logged into the software owner account (eg. grid).

$ ulimit -Hs
10240

As long as the hard stack limit is 10240 or above then the configuration should be ok.

 

• Setting Resource Limits for the Oracle Software Installation Users

 ASM_POWER_LIMIT specifies the maximum power on an Automatic Storage Management instance for disk rebalancing. The higher the limit, the faster rebalancing will complete. Lower values will take longer, but consume fewer processing and I/O resources.

Syntax to specify power limit while adding or droping disk is :- alter diskgroup  add disk '/dev/raw/raw37' rebalance power 10;
 

 Make sure machine clocks are synchronized on all nodes to the same NTP source.
Implement NTP (Network Time Protocol) on all nodes.
Prevents evictions and helps to facilitate problem diagnosis.

Also use  the -x option (ie. ntpd -x, xntp -x) if available to prevent time from moving backwards in large amounts. This slewing will help reduce time changes into multiple small changes, such that they will not impact the CRS. Enterprise Linux: see /etc/sysconfig/ntpd; Solaris: set "slewalways yes" and "disable pll" in /etc/inet/ntp.conf. 
Like:- 
       # Drop root to id 'ntp:ntp' by default.
       OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"
       # Set to 'yes' to sync hw clock after successful ntpdate
       SYNC_HWCLOCK=no
       # Additional options for ntpdate
       NTPDATE_OPTIONS=""

The Time servers operate in a pyramid structure where the top of the NTP stack is usually an external time source (such as GPS Clock).  This then trickles down through the Network Switch stack to the connected server.  
This NTP stack acts as the NTP Server and ensuring that all the RAC Nodes are acting as clients to this server in a slewing method will keep time changes to a minute amount.

Changes in global time to account for atomic accuracy's over Earth rotational wobble , will thus be accounted for with minimal effect.   This is sometimes referred to as the " Leap Second " " epoch ", (between  UTC  12/31/2008 23:59.59 and 01/01/2009 00:00.00  has the one second inserted).

More information can be found in Note 759143.1
"NTP leap second event causing Oracle Clusterware node reboot"
Linked to this Success Factor.

RFC "NTP Slewing for RAC" has been created successfully. CCB ID 462 
 

• Note: 759143.1 - NTP leap second event causing CRS node reboot
• Linux Leap Second
• Note: 551704.1 - Linux OS Service 'ntpd'

 A performance improvement can be seen with MTU frame size of approximately 9000.  Check with your SA and network admin first and if possible, configure jumbo frames for the interconnect.  Depending upon your network gear the supported frame sizes may vary between NICs and switches.  The highest setting supported by BOTH devices should be considered.  Please see below referenced notes for more detail specific to platform.    

To validate whether jumbo frames are configured correctly end to end (ie., NICs and switches), run the following commands as root.  Invoking ping using a specific interface requires root.

export CRS_HOME= To your GI or clusterware home like export CRS_HOME=/u01/app/12.1.0/grid

/bin/ping -s 8192 -c 2 -M do -I `$CRS_HOME/bin/oifcfg getif |grep cluster_interconnect|tail -1|awk '{print $1}'` hostname 

Substitute your frame size as required for 8192 in the above command.  The actual frame size varies from one networking vendor to another.

If you get errors similar to the  following then jumbo frames are not configured properly for your frame size.

From 192.168.122.186 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 192.168.122.186 icmp_seq=1 Frag needed and DF set (mtu = 1500)

--- hostname@company.com ping statistics ---
0 packets transmitted, 0 received, +2 errors


if jumbo frames are configured properly for your frame size you should obtain output similar to the following:

8192 bytes from hostname (10.208.111.43): icmp_seq=1 ttl=64 time=0.683 ms
8192 bytes from hostname(10.208.111.43): icmp_seq=2 ttl=64 time=0.243 ms

--- hostname ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.243/0.463/0.683/0.220 ms
 

• CISCO GIANT/JUMBO sized frame support
• JUMBO Frame Clean Network Gear
• INTEL PC1000 Network info
• Note: 563566.1 - GC Lost Blocks Diagnostics
• Jumbo Frames for RAC Interconnect
• Note: 341788.1 - Recommendation for the Real Application Cluster Interconnect and Jumbo Frames
• Netgear - What is the Jumbo Frame Supported by Switches and Adapters?

 Operating System Watcher  (OSW) is a collection of UNIX shell scripts intended to collect and archive operating system and network metrics to aid diagnosing performance issues. OSW is designed to run continuously and to write the metrics to ASCII files which are saved to an archive directory. The amount of archived data saved and frequency of collection are based on user parameters set when starting OSW.
 

• Note: 301137.1 - OS Watcher User Guide

 Reboottime (default 3 seconds) is the amount of time allowed for a node to complete a reboot after the CSS daemon has been evicted.
 

• Note: 294430.1 - CSS Timeout Computation in Oracle Clusterware
• 12.1 CRSCTL Utility Reference
• CRSCTL Utility Reference

 The maximum amount of time allowed for a voting file I/O to complete; if this time is exceeded the voting disk will be marked as offline.  Note that this is also the amount of time that will be required for initial cluster formation, i.e. when no nodes have previously been up and in a cluster.
 

• Note: 294430.1 - CSS Timeout Computation in Oracle Clusterware

 Due to Bug 9837321 or if for any other reason the ownership of certain clusterware related log files is changed incorrectly it could result in important diagnostics not being available when needed by Support.  These logs are rotated periodically to keep them from growing unmanageably large and if the ownership of the files is incorrect when it is time to rotate the logs that operation could fail and while that doesn't effect the operation of the clusterware itself it would effect the logging and therefore problem diagnostics.  So it would be wise to verify that the ownership of the following files is root:root:

$ls -l $GRID_HOME/log/`hostname`/crsd/*
$ls -l $GRID_HOME/log/`hostname`/ohasd/*
$ls -l $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
$ls -l $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

If any of those files' ownership is NOT root:root then you should change the ownership of the files individually or as follows (as root):

# chown root:root $GRID_HOME/log/`hostname`/crsd/*
# chown root:root $GRID_HOME/log/`hostname`/ohasd/*
# chown root:root $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
# chown root:root $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*
 

• Oracle Bug # 9837321 - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT
• Note: 9837321.8 - Bug 9837321 - Ownership of crsd traces gets changed from root by patching script

 Due to Bug 9837321 or if for any other reason the ownership of certain clusterware related log files is changed incorrectly it could result in important diagnostics not being available when needed by Support.  These logs are rotated periodically to keep them from growing unmanageably large and if the ownership of the files is incorrect when it is time to rotate the logs that operation could fail and while that doesn't effect the operation of the clusterware itself it would effect the logging and therefore problem diagnostics.  So it would be wise to verify that the ownership of the following files is root:root:

$ls -l $GRID_HOME/log/`hostname`/crsd/*
$ls -l $GRID_HOME/log/`hostname`/ohasd/*
$ls -l $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
$ls -l $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

If any of those files' ownership is NOT root:root then you should change the ownership of the files individually or as follows (as root):

# chown root:root $GRID_HOME/log/`hostname`/crsd/*
# chown root:root $GRID_HOME/log/`hostname`/ohasd/*
# chown root:root $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
# chown root:root $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*
 

• Oracle Bug # 9837321 - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT

 Due to Bug 9837321 or if for any other reason the ownership of certain clusterware related log files is changed incorrectly it could result in important diagnostics not being available when needed by Support.  These logs are rotated periodically to keep them from growing unmanageably large and if the ownership of the files is incorrect when it is time to rotate the logs that operation could fail and while that doesn't effect the operation of the clusterware itself it would effect the logging and therefore problem diagnostics.  So it would be wise to verify that the ownership of the following files is root:root:

$ls -l $GRID_HOME/log/`hostname`/crsd/*
$ls -l $GRID_HOME/log/`hostname`/ohasd/*
$ls -l $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
$ls -l $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*

If any of those files' ownership is NOT root:root then you should change the ownership of the files individually or as follows (as root):

# chown root:root $GRID_HOME/log/`hostname`/crsd/*
# chown root:root $GRID_HOME/log/`hostname`/ohasd/*
# chown root:root $GRID_HOME/log/`hostname`/agent/crsd/orarootagent_root/*
# chown root:root $GRID_HOME/log/`hostname`/agent/ohasd/orarootagent_root/*
 

• Oracle Bug # 9837321 - CSS Timeout Computation in Oracle Clusterware - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT
• Note: 9837321.8 - Bug 9837321 - Ownership of crsd traces gets changed from root by patching script

 CRSD trace file should owned by "root:root", but due to Bug 9837321application of patch may have resulted in changing the trace file ownership for patching and not changing it back.
 

• Oracle Bug # 9837321 - Bug 9837321 - Ownership of crsd traces gets changed from root by patching script - OWNERSHIP OF CRSD TRACES GOT CHANGE FROM ROOT TO ORACLE BY PATCHING SCRIPT