In this Document
Goal
Solution
1. Stop all CRS managed resources
2. Stop all CRS services
3. Reformat ALL partitions for OCR and voting disk files
4. Set TEMP and TMP variables
5. Locate the crssetup.config.bat script
6. Execute racgons from $CRS_HOMEbin
7. Run oifcfg to configure public and cluster_interconnect
8. Run Virtual IP Configuration Assistant (VIPCA)
9. Run Cluster Verification Uitility (cluvfy)
10. Create an export copy of the OCR file
11. Check for Oracle Process Manager
12. Reboot all nodes in the cluster, if possible.
13. Run netca to create listener
14. Recreate all necessary CRS managed resources
15. Start all CRS managed resources
16. Run crs_stat -t
References

Applies to:

Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.1.0.7
Microsoft Windows Server 2003 (64-bit AMD64 and Intel EM64T)
Microsoft Windows (32-bit)
Oracle Server Enterprise Edition - Version: 10.1.0.2 to 11.1.0.7

Goal

Step by step approach to reinitialize the OCR and Vote Disks without performing a full reinstallation of the Oracle Clusterware binaries for RAC on Windows platforms. These steps may be necessary in a few different scenarios:

1. To troubleshoot an installation where the Cluster Configuration Assistant (which calls crssetup.config.bat ) has errored out

OR

2. For a cluster that was running fine, but the OCR and/or Vote disks have become damaged or unusable

OR

3. The OS user account that did the original CRS installation needs to be changed (such as to switch from using a Domain Administrator account with local administrator privileges to using a local administrator account), or has been deleted, or no longer has appropriate administrator privileges.

By following this note as the correct user, the OCR will be recreated under the new, desired account privileges.

Please note that some of the listed steps (such as stopping the clusterware services) may not be relevant to installation troubleshooting.

Solution

Note: If you have an OracleCRSToken service configured in this environment, it is advised to remove that service before implementing these steps, then recreate that service once these steps have been completed. Knowledge Document Note 338000.1 explains OracleCRSToken services including how and why to create them and how to delete them.

1. Stop all CRS managed resources

Ensure all CRS resources are stopped using the 'crs_stat -t' utility to check the status and the "srvctl" utility to stop as required.

In particular before step 2 below make sure that you stop nodeapps.

srvctl stop nodeapps -n

Note: Failure to stop the nodeapps with the above command will result in an error since just stopping the services (as per step 2) does not stop the ons.exe and that will prevent the crssetup.config.bat script from running.

2. Stop all CRS services

Stop all services running from the CRS home and set them to MANUAL from the Windows Services panel:

Stop OracleCRService and set to "Manual"on ALL nodes Stop OracleEVMService and set to "Manual"on ALL nodes Stop OracleCSService and set to "Manual" on ALL nodes Stop OracleClusterVolumeService and set to "Manual" on ALL nodes Stop Oracle Object Service and set to "Manual" on ALL nodes

3. Reformat ALL partitions for OCR and voting disk files

All partitions that you will reuse for OCR and voting disk information should be reformatted in preparation for their recreation.

Example where OCR and Voting disks are stored on RAW partitions:

a. invoke the guioracleobjmanager.exe utility from $CRS_HOMEbin and note the assigned symbolic link names for ALL of the configured OCR and Voting disk partitions, example:

b.

- Go to a command prompt
- navigate to $CRS_HOMEbin
- run logpartformat /q

E:oracleproduct10.2.0crsBIN>logpartformat /q .ocrcfg E:oracleproduct10.2.0crsBIN>logpartformat /q .votedsk1

Example where OCR and Voting disks are stored on OCFS partition(s):

a. to locate your ocr partition(s) look to the following key by invoking 'regedit'

HKEY_LOCAL_MACHINESoftwareOracleocrocrconfig_loc

b. to locate your voting disk partition(s), use the following command:

$CRS_HOMEbin> crsctl query css votedisk

c. create backups of the existing ocr and voting disk files AND create new 'placeholder' files for the replacement ocr and voting disk files

Example:

Rename O:cdataclusternamevotedsk.ocr to O:cdataclusternamevotedsk.ocr.old Rename O:cdataclusternameocrcfg.ocr to O:cdataclusternameocrcfg.ocr.old notepad O:cdataclusternamevotedsk.ocr notepad O:cdataclusternameocrcfg.ocr

d.

- Go to a command prompt
- navigate to $CRS_HOMEbin
- run logpartformat /q

E:oracleproduct10.2.0crsBIN>logpartformat /q F: E:oracleproduct10.2.0crsBIN>logpartformat /q G:

4. Set TEMP and TMP variables

Open a Windows command prompt and set TEMP and TMP to use c:temp

set temp=c:temp
set tmp=c:temp

Note:

As a guide for Steps 5-8 utilize the commands listed in:

$CRS_HOMEcfgtoollogsconfigtoolallcommands
OR
$CRS_HOMEcfgtoollogsconfigtoolallcommands.bak

Whichever is correctly populated for your environment.

5. Locate the crssetup.config.bat script

From node1 (or whichever node was used to apply the latest patchset to this cluster), locate the crssetup.config.bat script, check that the contents of this .bat file are correct for your environment, and execute this file.

Example: assuming your CRS_HOME is e:oracleproduct10.2.0crs

cd e:oracleproduct10.2.0crsinstall

OR

if you have already applied a patchset:

Example:
e:oracleproduct10.2.0crsinstallprepatch10204install

Again, execute whichever crssetup.config.bat script is correctly populated for your environment.

E:oracleproduct10.2.0crsinstallprepatch10204install>crssetup.config.bat Step 1: checking status of CRS cluster Step 2: creating directories (E:oracleproduct10.2.0crs) Step 3: configuring OCR repository Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node : node 1: racnode1 racnode1-p racnode1 node 2: racnode2 racnode2-p racnode2 Creating OCR keys for user 'administrator', privgrp ''.. Operation successful. Now formatting voting device: .votedsk1 Format of 1 voting devices complete. Step 4: configuring safe mode for CRS stack Step 5: Starting up CRS stack on all nodes Successfully completed configuration of 2 nodes

Optionally - perform a check to confirm the CRS version as per the OCR file as follows:
E:oracleproduct10.2.0crsBIN>crsctl query crs softwareversion Oracle Clusterware version on node [racnode1] is [10.2.0.4.0] E:oracleproduct10.2.0crsBIN>crsctl query crs activeversion Oracle Clusterware active version on the cluster is [10.2.0.4.0]

It is important that this step be executed while logged on as the OS user account that you have selected to be the owner of the CRS installation. In most cases, this will be the same account that did the original installation.

If you are attempting to change the OCR to run under a different account, you must ensure that you are logged on to the OS under that new account prior to running crssetup.config.bat. This step will create the OCR and set permissions on certain resources for the account that runs the crssetup.config.bat command. If that OS user account is deleted in the future, or if that account's access permissions change, then the steps in this note will need to be re-run under a valid user account. Therefore, using temporary accounts for the installation of CRS should be avoided where possible.
Upon completion of this step, Oracle Clusterware services should have started automatically.

6. Execute racgons from $CRS_HOMEbin

This step configures / ensures that all nodes are configured with the same port for the ONS

racgons.exe add_config : :
Example:
E:oracleproduct10.2.0crsBIN>racgons.exe add_config racnode1:6200 racnode2:6200

7. Run oifcfg to configure public and cluster_interconnect

Example: as oracle user:
oifcfg setif -global /:public oifcfg setif -global /:cluster_interconnect

Example:
E:oracleproduct10.2.0crsBIN>oifcfg.exe setif -global "Public"/10.143.137.0:public "Private"/10.10.10.0:cluster_interconnect

8. Run Virtual IP Configuration Assistant (VIPCA)

VIPCA must be run to recreate the nodeapp resources:

a. to use VIPCA in GUI mode simply type: 'vipca' at the $CRS_HOMEbin prompt
OR
b. to run VIPCA in silent mode, use the command as listed in the configtoolallcommands file
Example:
E:oracleproduct10.2.0crsBIN>E:oracleproduct10.2.0crs/bin/vipca.bat -silent -nodelist "racnode1,racnode2" -nodevips "racnode1/racnode1-vip/255.255.255.0/Public,racnode2/racnode2-vip/255.255.255.0/Public" Creating VIP application resource on (2) nodes... Creating GSD application resource on (2) nodes... Creating ONS application resource on (2) nodes... Starting VIP application resource on (2) nodes... Starting GSD application resource on (2) nodes... Starting ONS application resource on (2) nodes... Done.
Once this step is complete, the output of 'crs_stat -t' should show vip, ons, and gsd resources for all nodes.
E:oracleproduct10.2.0crsBIN>crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora.racnode1.gsd application ONLINE ONLINE racnode1 ora.racnode1.ons application ONLINE ONLINE racnode1 ora.racnode1.vip application ONLINE ONLINE racnode1 ora.racnode2.gsd application ONLINE ONLINE racnode2 ora.racnode2.ons application ONLINE ONLINE racnode2 ora.racnode2.vip application ONLINE ONLINE racnode2

9. Run Cluster Verification Uitility (cluvfy)

Run Cluster Verification Uitility (cluvfy) to confirm post crs installation

cluvfy.bat stage -post crsinst -n "," Example: E:oracleproduct10.2.0crsBIN>E:oracleproduct10.2.0crs/bin/cluvfy.bat stage -post crsinst -n "racnode1,racnode2" Performing post-checks for cluster services setup Checking node reachability... Node reachability check passed from node "racnode1". Checking user equivalence... User equivalence check passed for user "Administrator". Checking Cluster manager integrity... Checking CSS daemon... Daemon status check passed for "CSS daemon". Cluster manager integrity check passed. Checking cluster integrity... Cluster integrity check passed Checking OCR integrity... Checking the absence of a non-clustered configuration... All nodes free of non-clustered, local-only configurations. Uniqueness check for OCR device passed. Checking the version of OCR... OCR of correct Version "2" exists. Checking data integrity of OCR... Data integrity check for OCR passed. OCR integrity check passed. Checking CRS integrity... Checking daemon liveness... Liveness check passed for "CRS daemon". Checking daemon liveness... Liveness check passed for "CSS daemon". Checking daemon liveness... Liveness check passed for "EVM daemon". Checking CRS health... CRS health check passed. CRS integrity check passed. Checking node application existence... Checking existence of VIP node application (required) Check passed. Checking existence of ONS node application (optional) Check passed. Checking existence of GSD node application (optional) Check passed. Post-check for cluster services setup was successful.

10. Create an export copy of the OCR file

ocrconfig -export c:tmpocrdump_base.out

11. Check for Oracle Process Manager

Ensure the "Oracle Process Manager" service is set to "Automatic" on ALL nodes

Oracle Support highly recommends configuring Oracle Process Manager (OPMD) to run in all Windows RAC environments. Reference Note 358156.1 for full details.

12. Reboot all nodes in the cluster, if possible.

A reboot of all nodes will ensure that the clusterware services (including Oracle Process Manager service) are working as expected.

13. Run netca to create listener

a. rename $ORACLE_HOME/network/admin/listener.ora
(ASM HOME and/or RDBMS ORACLE_HOME) to any other name on all nodes
AND

b. From the command prompt, invoke netca from the ASM and/or RDBMS home (on just one node)
E:oracleproduct10.2.0db_1binnetca

Note:
Be sure to select 'Cluster Configuration' and to create a listener with the default name LISTENER

It is necessary to recreate the listener using NETCA at this point, as that is the only way to furnish the reinitialized OCR with .lsnr resource information.

Once these steps are completed, 'crs_stat -t' should show a '.lsnr' resource for all nodes
E:oracleproduct10.2.0crsBIN>crs_stat -tName Type Target State Host ------------------------------------------------------------ ora....E1.lsnr application ONLINE ONLINE racnode1 ora.racnode1.gsd application ONLINE ONLINE racnode1 ora.racnode1.ons application ONLINE ONLINE racnode1 ora.racnode1.vip application ONLINE ONLINE racnode1 ora....E2.lsnr application ONLINE ONLINE racnode2 ora.racnode2.gsd application ONLINE ONLINE racnode2 ora.racnode2.ons application ONLINE ONLINE racnode2 ora.racnode2.vip application ONLINE ONLINE racnode2

14. Recreate all necessary CRS managed resources

Run the 'srvctl add' command to add asm, database, instance and services resources back into the OCR

For full syntax usage, please execute:
srvctl add asm -h srvctl add database -h srvctl add instance -h srvctl add service -h (if needed)

For example: as oracle user:
srvctl add asm -n -i -o srvctl add database -d -o srvctl add instance -d -i -n srvctl modify instance -d -i -s *** srvctl modify instance -d racdb -i racdb3 -s asm3 *** <== repeat as needed to make the database instance resources dependent on their corresponding ASM instance resources in the OCR

Please note, assuming that you are using ASM the asm_inst_name can only be +ASM1, +ASM2 and so on.
Failure in providing the correct name could cause an OCR corruption.

15. Start all CRS managed resources

Once all necessary resources have been registered, start them via srvctl commands

For example: as oracle user:
srvctl start asm -n srvctl start instance -d -i

16. Run crs_stat -t

Check 'crs_stat -t' output, it should show all resources with TARGET and STATUS of ONLINE for all resources on ALL nodes[@more@]

RAC on Windows: How To Reinitialize the OCR and Vote Disk (without a full reinstall of Oracle Cluste