Friday, July 24, 2015

OEM 11g Database Control - Troubleshooting

I was about to continue my Oracle Backup & Recovery Series further and my next post in the series was to show how to multiplex the redo logs and Setting up FRA location appropriately. I thought of doing this activity with OEM Dbconsole instead of using SQLPLUS, and that's when I saw my OEM Dbconsole is not properly configured.

After login into the Dbconsole my cluster database status is shown as not up, But the cluster tab all my instances are showing as up and running - Sure there is some discrepancy. Agent upload was failing, tried upload, secure everything, But finally messed up everything and decided to recreate the OEM repo.










Tried to upload agent and only to fail, So i decided to drop my OEM repository and recreate it.

1) Dropped the repository



ajithpathiyil1:/home/oracle[ajith1]$ emca -deconfig dbcontrol db -repos drop

STARTED EMCA at Jul 24, 2015 9:38:10 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: ajith1
Listener port number: 1521
Password for SYS user: 
Password for SYSMAN user:

----------------------------------------------------------------------
WARNING : While repository is dropped the database will be put in quiesce mode.
----------------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Jul 24, 2015 9:39:49 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/ajithpathiyil/emca_2015_07_24_21_38_10.log.
Jul 24, 2015 9:39:49 PM oracle.sysman.emcp.util.DBControlUtil stopOMS
INFO: Stopping Database Control (this may take a while) ...
Jul 24, 2015 9:40:09 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Dropping the EM repository (this may take a while) ...
Jul 24, 2015 9:42:17 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully dropped
Enterprise Manager configuration completed successfully
FINISHED EMCA at Jul 24, 2015 9:42:19 PM
ajithpathiyil1:/home/oracle[ajith1]$


2) Recreated the repository, But it errored with ORA-12541



ajithpathiyil1:/home/oracle[ajith1]$ emca -config dbcontrol db -repos create -cluster

STARTED EMCA at Jul 24, 2015 9:42:39 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database unique name: ajithpathiyil
Service name: ajithpathiyil.lab.com
Listener port number: 1521
Listener ORACLE_HOME [ /u01/grid/oracle/product/11.2.0.4/grid_1 ]:
Password for SYS user: 
Jul 24, 2015 9:45:39 PM oracle.sysman.emcp.util.GeneralUtil initSQLEngineRemotely
WARNING: Error during db connection : ORA-12541: TNS:no listener

Password for DBSNMP user:
Password for SYSMAN user: 
Cluster name: ajith-cluster
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
ASM ORACLE_HOME [ /u01/grid/oracle/product/11.2.0.4/grid_1 ]: /u01/grid/oracle/product/11.2.0.4/grid_1
ASM port [ 1521 ]:
ASM username [ ASMSNMP ]:
ASM user password:
Jul 24, 2015 9:52:25 PM oracle.sysman.emcp.util.GeneralUtil initSQLEngineRemotely
WARNING: Error during db connection : ORA-12541: TNS:no listener

-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /u01/app/oracle/product/11.2.0.4/db_1

Database instance hostname ................ Listener ORACLE_HOME ................ /u01/grid/oracle/product/11.2.0.4/grid_1
Listener port number ................ 1521
Cluster name ................ ajith-cluster
Database unique name ................ ajithpathiyil
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............
ASM ORACLE_HOME ................ /u01/grid/oracle/product/11.2.0.4/grid_1
ASM port ................ 1521
ASM user role ................ SYSDBA
ASM username ................ ASMSNMP

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Jul 24, 2015 9:52:48 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/ajithpathiyil/emca_2015_07_24_21_44_58.log.
Jul 24, 2015 9:52:59 PM oracle.sysman.emcp.util.GeneralUtil initSQLEngineRemotely
WARNING: Error during db connection : ORA-12541: TNS:no listener

Jul 24, 2015 9:53:14 PM oracle.sysman.emcp.ParamsManager checkListenerStatusForDBControl
WARNING: ORA-12541: TNS:no listener

Jul 24, 2015 9:53:14 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Scan Listener is not up or database service is not registered with it. Start the Scan Listener and register database service and run EM Configuration Assistant again .
Refer to the log file at /u01/app/oracle/cfgtoollogs/emca/ajithpathiyil/emca_2015_07_24_21_44_58.log for more details.
Could not complete the configuration. Refer to the log file at /u01/app/oracle/cfgtoollogs/emca/ajithpathiyil/emca_2015_07_24_21_44_58.log for more details.

ajithpathiyil1:/home/oracle[ajith1]$



3) Verified the scan listener status as mentioned in the above error. Scan listener was down.



ajithpathiyil1:/home/oracle[+ASM1]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is not running
ajithpathiyil1:/home/oracle[+ASM1]$


ajithpathiyil1:/home/oracle[+ASM1]$ lsnrctl status LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 25-JUL-2015 00:16:32

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
TNS-12541: TNS:no listener
 TNS-12560: TNS:protocol adapter error
  TNS-00511: No listener
   Linux Error: 2: No such file or directory
ajithpathiyil1:/home/oracle[+ASM1]$

 

 4)  Started scan listener



ajithpathiyil1:/home/oracle[+ASM1]$ lsnrctl start LISTENER_SCAN1

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 25-JUL-2015 00:16:41

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Starting /u01/grid/oracle/product/11.2.0.4/grid_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 11.2.0.4.0 - Production
System parameter file is /u01/grid/oracle/product/11.2.0.4/grid_1/network/admin/listener.ora
Log messages written to /u01/app/oracle/diag/tnslsnr/ajithpathiyil1/listener_scan1/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                25-JUL-2015 00:16:41
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/grid/oracle/product/11.2.0.4/grid_1/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/ajithpathiyil1/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
The listener supports no services
The command completed successfully
ajithpathiyil1:/home/oracle[+ASM1]$



ajithpathiyil1:/home/oracle[+ASM1]$ oenv
SIDs here are: +ASM1 ajith1 ajiebs ajithpathiyil
ORACLE_SID = [+ASM1] ? ajith1
The Oracle base remains unchanged with value /u01/app/oracle
ajithpathiyil1:/home/oracle[ajith1]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.4.0 - Production on 25-JUL-2015 00:16:55

Copyright (c) 1991, 2013, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.4.0 - Production
Start Date                25-JUL-2015 00:09:56
Uptime                    0 days 0 hr. 6 min. 59 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Log File         /u01/app/oracle/diag/tnslsnr/ajithpathiyil1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ajithpathiyil1)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=ajithpathiyil1)(PORT=8080))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "ajithXDB.lab.com" has 3 instance(s).
  Instance "ajith1", status READY, has 1 handler(s) for this service...
  Instance "ajith2", status READY, has 1 handler(s) for this service...
  Instance "ajith3", status READY, has 1 handler(s) for this service...
Service "ajithpathiyil.lab.com" has 3 instance(s).
  Instance "ajith1", status READY, has 2 handler(s) for this service...
  Instance "ajith2", status READY, has 1 handler(s) for this service...
  Instance "ajith3", status READY, has 1 handler(s) for this service...
The command completed successfully
ajithpathiyil1:/u01/grid/oracle/product/11.2.0.4/grid_1/network/admin[ajith1]$
 

5)  Again, recreated the OEM repository, and this time it was successful.



ajithpathiyil1:/home/oracle[ajith1]$ emca -config dbcontrol db -repos create -cluster

STARTED EMCA at Jul 25, 2015 12:19:11 AM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database unique name: ajithpathiyil
Service name: ajithpathiyil.lab.com
Listener port number: 1521
Listener ORACLE_HOME [ /u01/grid/oracle/product/11.2.0.4/grid_1 ]: /u01/app/oracle/product/11.2.0.4/db_1
Password for SYS user:
Password for DBSNMP user:
Password for SYSMAN user:
Cluster name: ajith-cluster
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
ASM ORACLE_HOME [ /u01/grid/oracle/product/11.2.0.4/grid_1 ]:
ASM port [ 1521 ]:
ASM username [ ASMSNMP ]:
ASM user password:
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /u01/app/oracle/product/11.2.0.4/db_1

Database instance hostname ................ Listener ORACLE_HOME ................ /u01/app/oracle/product/11.2.0.4/db_1
Listener port number ................ 1521
Cluster name ................ ajith-cluster
Database unique name ................ ajithpathiyil
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............
ASM ORACLE_HOME ................ /u01/grid/oracle/product/11.2.0.4/grid_1
ASM port ................ 1521
ASM user role ................ SYSDBA
ASM username ................ ASMSNMP

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Jul 25, 2015 12:20:04 AM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /u01/app/oracle/cfgtoollogs/emca/ajithpathiyil/emca_2015_07_25_00_19_11.log.
Jul 25, 2015 12:20:07 AM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...
Jul 25, 2015 12:24:52 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Repository successfully created
Jul 25, 2015 12:25:02 AM oracle.sysman.emcp.EMReposConfig uploadConfigDataToRepository
INFO: Uploading configuration data to EM repository (this may take a while) ...
Jul 25, 2015 12:25:48 AM oracle.sysman.emcp.EMReposConfig invoke
INFO: Uploaded configuration data successfully
Jul 25, 2015 12:25:49 AM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /u01/app/oracle/product/11.2.0.4/db_1/oc4j/j2ee/OC4J_DBConsole_ajithpathiyil1_ajithpathiyil to remote nodes ...
Jul 25, 2015 12:25:51 AM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /u01/app/oracle/product/11.2.0.4/db_1/oc4j/j2ee/OC4J_DBConsole_ajithpathiyil2_ajithpathiyil to remote nodes ...
Jul 25, 2015 12:25:52 AM oracle.sysman.emcp.EMDBCConfig instantiateOC4JConfigFiles
INFO: Propagating /u01/app/oracle/product/11.2.0.4/db_1/oc4j/j2ee/OC4J_DBConsole_ajithpathiyil3_ajithpathiyil to remote nodes ...
Jul 25, 2015 12:25:56 AM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /u01/app/oracle/product/11.2.0.4/db_1/ajithpathiyil1_ajithpathiyil to remote nodes ...
Jul 25, 2015 12:25:58 AM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /u01/app/oracle/product/11.2.0.4/db_1/ajithpathiyil2_ajithpathiyil to remote nodes ...
Jul 25, 2015 12:26:02 AM oracle.sysman.emcp.EMAgentConfig deployStateDirs
INFO: Propagating /u01/app/oracle/product/11.2.0.4/db_1/ajithpathiyil3_ajithpathiyil to remote nodes ...
Jul 25, 2015 12:26:04 AM oracle.sysman.emcp.util.ClusterUtil getVIP
SEVERE: Error retrieving VIP for null
Jul 25, 2015 12:26:04 AM oracle.sysman.emcp.EMAgentConfig addScanLsnrTarget
SEVERE: No targets.xml file at /u01/app/oracle/product/11.2.0.4/db_1/null_ajithpathiyil/sysman/emd/targets.xml
Jul 25, 2015 12:26:04 AM oracle.sysman.emcp.util.DBControlUtil secureDBConsole
INFO: Securing Database Control (this may take a while) ...
Jul 25, 2015 12:26:54 AM oracle.sysman.emcp.util.DBControlUtil startOMS
INFO: Starting Database Control (this may take a while) ...
Jul 25, 2015 12:28:17 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: Database Control started successfully
Jul 25, 2015 12:28:17 AM oracle.sysman.emcp.EMDBPostConfig performConfiguration
INFO: >>>>>>>>>>> The Database Control URL is https://ajithpathiyil1:5500/em <<<<<<<<<<<
Jul 25, 2015 12:28:22 AM oracle.sysman.emcp.EMDBPostConfig showClusterDBCAgentMessage
INFO:
****************  Current Configuration  ****************
 INSTANCE            NODE           DBCONTROL_UPLOAD_HOST
----------        ----------        ---------------------

ajithpathiyil     ajithpathiyil1     ajithpathiyil1
ajithpathiyil     ajithpathiyil2     ajithpathiyil1
ajithpathiyil     ajithpathiyil3     ajithpathiyil1


Jul 25, 2015 12:28:22 AM oracle.sysman.emcp.EMDBPostConfig invoke
WARNING:
************************  WARNING  ************************

Management Repository has been placed in secure mode wherein Enterprise Manager data will be encrypted.  The encryption key has been placed in the file: /u01/app/oracle/product/11.2.0.4/db_1/ajithpathiyil1_ajithpathiyil/sysman/config/emkey.ora. Ensure this file is backed up as the encrypted data will become unusable if this file is lost.

***********************************************************
Enterprise Manager configuration completed successfully
FINISHED EMCA at Jul 25, 2015 12:28:22 AM
ajithpathiyil1:/home/oracle[ajith1]$



6) As shown in the EMCA output, The Database Control URL is https://ajithpathiyil1:5500/em

Finally , OEM screen was up and the status of database and cluster was not having any sync issues.




HAPPY LEARNING!

No comments:

Post a Comment

Thanks for you valuable comments !