Total Pageviews

Tuesday, 24 February 2015

Database & cluster related log files

Database Alert log, OUI log, ASM Alert log, CRS Log, OS Watcher log and OS Messages Location Listed at one Place

When working as Remote DBA log files plays very important role in day to day monitoring and trouble shooting. Log file checking starts from Software installation and continue through out life of database. Now a days, Database is not just a single s/w to work upon, Real Application Cluster and Automatic Storage Management has became an integral part of it.

Sometimes remote DBA also needs to look into OS level logs as well. Here, I am trying to give a central place for all database related log files like Database Alert log, OUI log, ASM Alert log, CRS Log, OS Watcher log and OS Messages Location.

Let's start from Oracle Database Software Installation. Oracle Universal Installer (OUI) is commonly used tool for s/w installation.

1.  Oracle Universal Installer (OUI) log Files (Oracle 9.2.0.X.X, 10.1.0.X.X, 10.2.0.X.X and 11.2.0.X.X ): The log files are in /logs directory.

In Unix the location of central inventory (oraInventory) can be found from the oraInst.loc file, which exists in the /var/opt/oracle or /etc/ (By default).

[oracle@database etc]$ pwd
/etc
[oracle@etc]$ cat oraInst.loc
#Oracle Installer Location File Location
#Thu Oct 21 16:12:34 PDT 2010
inst_group=oracle
inventory_loc=/etc/oraInventory

Here, the inventory_loc is '/etc/oraInventory'

The log files are below:

/logs/InstallActions.log
/logs/oraInstall.out
/logs/oraInstall.err
/logs/silentInstall.log   (only for Silent installations)

For 10.2.0.X.X to 11.2.0.X.X only

/cfgtoollogs/oui/InstallActions.log
/cfgtoollogs/oui/oraInstall.out
/cfgtoollogs/oui/oraInstall.err 
/cfgtoollogs/oui/silentInstall.log   (only for Silent installations)

2. Database alert log: Database Alter log are files, where every database related activities are recorded like instance start, stop, redo log switching, ORA error etc.

A DBA need to check this file for two reasons.

    a. To check health of Database.
    b. Check details of any error in Database.


In Oracle 11g Default path for Alter log files is

$ORACLE_BASE/diag/rdbms/{DBNAME}/{sid}/trace/alert_{sid}.log


Ex: /u01/app/oracle/diag/rdbms/db01/db001/trace/alert_db001.log

In Oracle 10g Environment, use below command to find path for alter log files.

SQL> show parameter background_dump_dest

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
background_dump_dest                 string      /etc/oracle/diag/rdbms/testdb/testdb/trace

3. Automatic Storage Management alert log files: ASM is used for providing redundancy and balancing data and load across different disks. Any issue related to ASM below is the path for alter log files (Oracle 11g).

$ORACLE_BASE/diag/asm/+asm/+ASM{instance number}/trace/ alert_+ASM {instance number}.log
Ex: /u01/app/oracle/diag/asm/+asm/+ASM2/trace/alert_+ASM2.log

4. Clusterware CRS alert log: Real application Cluster is used for high availability and work load distribution among diff-2 servers. To check health of cluster and trouble shooting below is path for alter log file.

The following sections describe the locations of the clusterware log files in Database Software Oracle 10g Release 1.

Cluster Ready Services Log Files

Cluster Ready Services (CRS) has daemon processes that generate log information. Log files for the CRS daemon (crsd) can be found in the following directories:

/crs/init
/crs/.log

Oracle Cluster Registry Log Files
The Oracle Cluster Registry (OCR) records log information in the following location:

/srvm/log/

Cluster Synchronization Services (CSS) Log Files
You can find CSS information that the OCSSD generates in log files in the following locations:

/css/log/ocssd.log
/css/init/.log

Event Manager Log Files
Event Manager (EVM) information generated by evmd is recorded in log files in the following locations:

/evm/log/evmdaemon.log
/evm/init/.log

Oracle High Availability Log Files
===========================The Oracle RAC high availability trace files are located in:

$ORACLE_BASE//admin/hdump

Where $ORACLE_BASE is configured and $ORACLE_HOME/racg/log when $ORACLE_BASE is not available.

In Oracle 11g Release 1 onwards logs can be found at this location.

$GRID_HOME/log/{node name}/alert{node name}.log
Ex: /u01/app/11.2.0/grid/log/db02/alertdb02.log

5. Diskmon log files : Disk mon log are basically to check any issue with the disk in the system.

$GRID_HOME/log/{node name}/diskmon/diskmon.lo*
Ex: /u01/app/11.2.0/grid/log/dmorldb02/diskmon/diskmon.log

6. OS Watcher output files : OS watcher logs are helpful for remote DBA when Database Server is hang or reboot automatically. Below is path for Linux/Unix env.

/opt/oracle.oswatcher/osw/archive/

To get OS watcher data of specific date :
cd /opt/oracle.oswatcher/osw/archive
find . -name '*12.01.13*' -print -exec zip /tmp/osw_`hostname`.zip {} \; 
where 12- year 01- Month 13-day

7. Os message logfile: OS message log file are logs related to OS related messages. Path for Linux/Unix syatem is below.

/var/log/messages

Ref:
http://www.dbas-oracle.com/2013/03/Database-Aler-log-OUI-log-ASM-Alert-log-CRS-Log-OS-Watcher-log-OS-Messges-Location-at-one-Place.html

Health Check of asm

Creating a HTML file to view details of ASM
=================================
Create .sh file and run it through cron or run manually.

healthcheck.sh
==========
spool asm_health_check.html
SET MARKUP HTML ON
set echo on
set pagesize 1000
set linesize 250
set feedback off
col bytes format 999,999,999,999
col space format 999,999,999,999
col gn format 999
col name format a20
col au format 99999999
col state format a12
col type format a12
col total_mb format 999,999,999
col free_mb format 999,999,999
col od format 999
col compatibility format a12
col dn format 999
col mount_status format a12
col header_status format a12
col mode_status format a12
col mode format a12
col failgroup format a20
col label format a12
col path format a45
col path1 format a40
col path2 format a40
col path3 format a40
col bytes_read format 999,999,999,999,999
col bytes_written format 999,999,999,999,999
col cold_bytes_read format 999,999,999,999,999
col cold_bytes_written format 999,999,999,999,999
alter session set nls_date_format='DD-MON-YYYY HH24:MI:SS' ;
select to_char(sysdate, 'DD-MON-YYYY HH24:MI:SS' ) current_time from dual;
select group_number gn, name, allocation_unit_size au, state, type, total_mb, free_mb, offline_disks od, compatibility
from v$asm_diskgroup;
select group_number gn,disk_number dn, mount_status, header_status,mode_status,state, total_mb, free_mb,name, failgroup, label, path,create_date, mount_date
from v$asm_disk order by group_number, disk_number;
break on g_n skip 1
break on failgroup skip 1
compute sum of t_mb f_mb on failgroup
compute count of failgroup on failgroup
select g.group_number g_n,g.disk_number d_n,g.name , g.path , g.total_mb t_mb,g.free_mb f_mb,g.failgroup from v$asm_disk g order by g_n, failgroup, d_n;

select 'THIS ASM REPORT WAS GENERATED AT: ==)> ' , sysdate " " from dual;

select 'HOSTNAME ASSOCIATED WITH THIS ASM INSTANCE: ==)> ' , MACHINE " " from v$session where program like '%SMON%';
select * from v$asm_diskgroup;
SELECT * FROM V$ASM_DISK ORDER BY GROUP_NUMBER,DISK_NUMBER;
SELECT * FROM V$ASM_CLIENT;
select * from V$ASM_ATTRIBUTE;
select * from v$asm_operation;
select * from v$version;
show parameter
show sga

spool off
exit;

========================================================================check the status of individual nodes and all the necessary applications,VIP, GSD, Listener and the ONS daemons are alive.
========================================================================
[root@obia bin]# ./srvctl status nodeapps -n obia
VIP obia-vip.oracle.com is enabled
VIP obia-vip.oracle.com is running on node: obia
Network is enabled
Network is running on node: obia
ONS is enabled
ONS daemon is running on node: obia

[root@obia bin]# ./srvctl status nodeapps -n obie
VIP obie-vip.oracle.com is enabled
VIP obie-vip.oracle.com is running on node: obie
Network is enabled
Network is running on node: obie
ONS is enabled
ONS daemon is running on node: obie

SQL> select instance_name, host_name, archiver, thread#, status from gv$instance;
INSTANCE_NAME    HOST_NAME                                                        ARCHIVE    THREAD# STATUS
---------------- ---------------------------------------------------------------- ------- ---------- ------------
PROD2            obie.oracle.com                                                  STOPPED          2 OPEN
PROD1            obia.oracle.com                                                  STOPPED          1 OPEN

Getting the status of all the groups, type, membership (if any)...
==================================================================

SQL> select group#, type, member, is_recovery_dest_file from v$logfile order by group#;

    GROUP# TYPE    MEMBER                                                                                                                          
         1 ONLINE  +DATA/PROD/ONLINELOG/group_1.259.861032749                                                                                                                        YES
         1 ONLINE  +DATA/PROD/ONLINELOG/group_1.262.861032747                                                                                                                        NO
         2 ONLINE  +DATA/PROD/ONLINELOG/group_2.257.861032753                                                                                                                        YES
         2 ONLINE  +DATA/PROD/ONLINELOG/group_2.260.861032751                                                                                                                        NO
         3 ONLINE  +DATA/PROD/ONLINELOG/group_3.269.861032927                                                                                                                        NO
         3 ONLINE  +DATA/PROD/ONLINELOG/group_3.270.861032929                                                                                                                        YES
         4 ONLINE  +DATA/PROD/ONLINELOG/group_4.271.861032931                                                                                                                        NO
         4 ONLINE  +DATA/PROD/ONLINELOG/group_4.272.861032931                                                                                                                        YES

8 rows selected.

Querying the v$asm_diskgroup view...
====================================
SQL> select group_number, name,allocation_unit_size alloc_unit_size,state,type,total_mb,usable_file_mb from v$asm_diskgroup;

GROUP_NUMBER NAME                           ALLOC_UNIT_SIZE STATE       TYPE     TOTAL_MB USABLE_FILE_MB
------------ ------------------------------ --------------- ----------- ------ ---------- --------------
           2 ASMDATA                                1048576 MOUNTED     EXTERN       5023           4726
           1 ARCHIVE                                1048576 MOUNTED     EXTERN       3998           3895
           3 DATA                                   1048576 CONNECTED   EXTERN     185759         183440
All datafiles information
==================

SQL> select name from v$datafile union select name from v$controlfile union select name from v$tempfile union select member from v$logfile;


To check if CRS is up
================

$./crsctl check crs

To check the status of all services on the cluster
==================================-
$./crs_stat -t -v




Friday, 20 February 2015

Location of log files in 11i and R12

Database Tier Logs
===============
Alert Log File location:
$ORACLE_HOME/admin/$CONTEXT_NAME/bdump/alert_$SID.log
sql>show parameter dump;

Trace file location:
==============
$ORACLE_HOME/admin/SID_Hostname/udump

Application Tier Logs
===============
Start/Stop script log files location:
$COMMON_TOP/admin/log/CONTEXT_NAME/ 

OPMN log file location
=================
$ORACLE_HOME/opmn/logs/ipm.log

Apache, Jserv, JVM log files locations:
============================
$IAS_ORACLE_HOME/Apache/Apache/logs/ssl_engine_log
$IAS_ORACLE_HOME/Apache/Apache/logs/ssl_request_log
$IAS_ORACLE_HOME/Apache/Apache/logs/access_log
$IAS_ORACLE_HOME/Apache/Apache/logs/error_log
$IAS_ORACLE_HOME/Apache/JServ/logs

Concurrent log file location:
====================
$APPL_TOP/admin/PROD/log or $APPLLOG/$APPLCSF

Patch log file location:
================
$APPL_TOP/admin/PROD/log

Worker Log file location:
==================
$APPL_TOP/admin/PROD/log

AutoConfig log files location:
=====================
Application Tier:
$APPL_TOP/admin/SID_Hostname/log//DDMMTime/adconfig.log

Database Tier:
$ORACLE_HOME/appsutil/log/SID_Hostname/DDMMTime/adconfig.log

Error log file location:
===============
Application Tier:
$APPL_TOP/admin/PROD/log

Database Tier :
$ORACLE_HOME/appsutil/log/SID_Hostname


In Oracle Applications R12, the log files are located in $LOG_HOME (which translates to $INST_TOP/logs)
==============================================================================
Below list of log file locations could be helpful for you:

Concurrent Reqeust related logs
=======================
$LOG_HOME/appl/conc - > location for concurrent requests log and out files
$LOG_HOME/appl/admin - > location for mid tier startup scripts log files

Apache Logs (10.1.3 Oracle Home which is equivalent to iAS Oracle Home - Apache, OC4J and OPMN)
===========================================================================
$LOG_HOME/ora/10.1.3/Apache - > Location for Apache Error and Access log files
$LOG_HOME/ora/10.1.3/j2ee - > location for j2ee related log files
$LOG_HOME/ora/10.1.3/opmn - > location for opmn related log files

Forms & Reports related logs (10.1.2 Oracle home which is equivalent to 806 Oracle Home)
==================================================================
$LOG_HOME/ora/10.1.2/forms
$LOG_HOME/ora/10.1.2/reports

Startup/Shutdown Log files location:
==========================
$INST_TOP/apps/$CONTEXT_NAME/logs/appl/admin/log

Patch log files location:
=================
$APPL_TOP/admin/$SID/log/ 

Clone and AutoConfig log files location in Oracle E-Business Suite Release 12 
========================================================
Logs for the adpreclone.pl are located: 
On the database tier: 
RDBMS $ORACLE_HOME/appsutil/log/< context >/StageDBTier_< timestamp >.log 

On the application tier: 
$INST_TOP/admin/log/StageAppsTier_< timestamp >.log 

Where the logs for the admkappsutil.pl are located? 
On the application tier: 
$INST_TOP/admin/log/MakeAppsUtil_< timestamp >.log 

To check url from backend
=====================
SELECT home_url FROM icx_parameters;