crs hidden files are deleted

Jan 6 posted in Oracle (RSS), 543 views.

Problem Description:
Inspection of a customer's rac database and found that there are a large number of Baocuo alert log as follows:
Tue Dec 29 10:33:17 2009
Errors in file / oracle/app/admin/BILL/udump/bill2_ora_12230.trc:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 42, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol3'
ORA-29701: unable to connect to Cluster Manager
ORA-00206: error in writing (block 42, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol2'
ORA-29701: unable to connect to Cluster Manager
ORA-00206: error in writing (block 42, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol1'
ORA-29701: unable to connect to Cluster Manager
rn

See its trace file content:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = / oracle/app/product/RAC10g
System name: HP-UX
Node name: M-BILL2
Release: B.11.31
Version: U
Machine: ia64
Instance name: BILL2
Redo thread mounted by this instance: 2
Oracle process number: 46
Unix process pid: 12230, image: oracle @ M-BILL2 (TNS V1-V3)
rn
*** ACTION NAME: (0000001 FINISHED70) 2009-12-29 10:33:16.050
*** MODULE NAME: (rman @ M-BILL2 (TNS V1-V3)) 2009-12-29 10:33:16.050
*** SERVICE NAME: (SYS $ USERS) 2009-12-29 10:33:16.050
*** SESSION ID: (704.12066) 2009-12-29 10:33:16.050
clsc_connect: (60000000000fef00) no listener at (ADDRESS = (PROTOCOL = ipc) (KEY = OCSSD_LL_m-bill2_crs_bill))
2009-12-29 10:33:16.051: [CSSCLNT] clsssInitNative: connect failed, rc 9
kgxgncin: CLSS init failed with status 3
kjfmsgr: unable to connect to NM for reg in shared group
ORA-00206: error in writing (block 1633, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol3'
ORA-29701: unable to connect to Cluster Manager
ORA-00206: error in writing (block 1633, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol2'
ORA-29701: unable to connect to Cluster Manager
ORA-00206: error in writing (block 1633, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol1'
ORA-29701: unable to connect to Cluster Manager
rn

rn
Analysis of reasons:
From the trace of:
*** MODULE NAME: (rman @ M-BILL2 (TNS V1-V3)) 2009-12-29 10:33:16.050
Judgments should be doing rman backup error.
rn
Examination found that in the directory
/ usr / netvault / scripts
Indeed, under the backup archive logs on a regular basis scripts.
And in the implementation of the script will report the error:
RMAN> connect target /
rn
RMAN-06900: WARNING: unable to generate V $ RMAN_STATUS or V $ RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V $ RMAN_STATUS and V $ RMAN_OUTPUT rows
ORACLE error from target database:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 1634, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol3'
ORA-29701: unable to connect to Cluster Manager
ORA-00206: error in writing (block 1634, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol2'
ORA-29701: unable to connect to Cluster Manager
ORA-00206: error in writing (block 1634, # blocks 1) of control file
ORA-00202: control file: '/ dev/datavg1/rcontrol1'
ORA-29701: unable to connect to Cluster Manager
rn

rn
The main features of the error are:
ORA-29701: unable to connect to Cluster Manager
Refer to metalink document:
Unable To Connect To Cluster Manager Ora-29701 [ID 391790.1].
Found that the node on the / tmp / .oracle less than the other nodes in the following documents:
srwxrwxrwx 1 oracle10 dba 0 Dec 8 12:43 sAm-bill1_crs_bill_evm
srwxrwxrwx 1 root root 0 Dec 8 12:43 sCRSD_UI_SOCKET
srwxrwxrwx 1 oracle10 dba 0 Dec 8 12:43 sCm-bill1_crs_bill_evm
srwxrwxrwx 1 oracle10 dba 0 Dec 8 12:43 sOCSSD_LL_m-bill1_crs_bill
srwxrwxrwx 1 oracle10 dba 0 Dec 8 12:43 sOracle_CSS_LclLstnr_crs_bill_1
srwxrwxrwx 1 oracle10 dba 0 Dec 8 12:43 sSYSTEM.evm.acceptor.auth
srwxrwxrwx 1 root root 0 Dec 8 12:43 sm-bill1DBG_CRSD
srwxrwxrwx 1 oracle10 dba 0 Dec 8 12:43 sm-bill1DBG_CSSD
srwxrwxrwx 1 oracle10 dba 0 Dec 8 12:43 sm-bill1DBG_EVMD
srwxrwxrwx 1 root root 0 Dec 8 12:43 sora_crsqs
srwxrwxrwx 1 root root 0 Dec 8 12:43 sprocr_local_conn_0_PROC
NOTE: These files are usually placed in / var / tmp / .oracle directory, but in some of the operating system on the / tmp / .oracle next.
These documents are special socket file for local client for example, through the IPC protocol to connect to the TNS listener, CSS, CRS, and EVM daemons, and even the database or ASM instance.
This is consistent with the trace file information:
clsc_connect: (60000000000fef00) no listener at (ADDRESS = (PROTOCOL = ipc) (KEY = OCSSD_LL_m-bill2_crs_bill))
rn

Solution:
Can only restart the CRS to rebuild these files.
In order to root user login:
# $ ORA_CRS_HOME / bin / crsctl stop crs
# $ ORA_CRS_HOME / bin / crsctl start crs
rn
The cessation of the process may encounter the following error:
Stopping resources. This could take several minutes.
Error while stopping resources. Possible cause: CRSD is down.
This is because the result can not be accessed CRSD, but because crs background processes still exist, you must reboot the system.
If it is kill ocssd.bin process, the same will cause the node to restart.
rn

Summary:
If you need to organize on a regular basis to delete / tmp or / var / tmp directory, need to take note of the directory under the hidden directory. Oracle.
rn

Follow-up experiments:
LINUX in a normal in rac environment, delete a node in / var / tmp / .oracle directory, in which the node check crs an error message:
$ crs_stat-t
CRS-0184: Cannot communicate with the CRS daemon.
$ crsctl check crs
Failure 1 contacting CSS daemon
Cannot communicate with CRS
Cannot communicate with EVM
rn
However, in another node examination, crs all the services are normal, and in an instance of the node failure is still working properly.
Check crs logs found fault nodes continue to report similar to the following error:
2009-12-24 01:50:13.575: [CSSCLNT] [1] clsssInitNative: connect failed, rc 9
2009-12-24 01:50:13.576: [RACG] [1] [29076] [1] [ora.m-bill2.gsd]: clsrccssgetctx: clsssinit () failed. Rc = 3
2009-12-24 01:50:13.578: [RACG] [1] [29076] [1] [ora.m-bill2.gsd]: clsrcgetprsrctx: prsr_init_ext returned rc = 3
2009-12-24 01:50:14.568: [RACG] [1] [29076] [1] [ora.m-bill2.gsd]: PRKH-1010: Unable to communicate with CRS services.
[Communications Error (Native: prsr_initCLSS: [3])]
Failed to get list of active nodes from clusterware
......
2009-12-24 01:50:14.600: [COMMCRS] [1] clsc_connect: (6000000000040400) no listener at (ADDRESS = (PROTOCOL = ipc) (KEY = SYSTEM.evm.accepto
r.auth))
Rman implementation of an error message:
RMAN> connect target /
rn
RMAN-06900: WARNING: unable to generate V $ RMAN_STATUS or V $ RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V $ RMAN_STATUS and V $ RMAN_OUTPUT rows
ORACLE error from target database:
ORA-00221: error on write to control file
ORA-00206: error in writing (block 814, # blocks 1) of control file
ORA-00202: control file: '+ RECOVERYDEST/ora10g/controlfile/current.256.688420755'
ORA-29701: unable to connect to Cluster Manager
ORA-00206: error in writing (block 814, # blocks 1) of control file
ORA-00202: control file: '+ DG1/ora10g/controlfile/current.256.688420747'
ORA-29701: unable to connect to Cluster Manager
rn
connected to target database: ORA10G (DBID = 4007292866)
using target database control file instead of recovery catalog