Audit file pileup in $GRID_HOME/rdbms/audit

May 22, 2018

One of the items highlighted by the dashboard is where  filesystem utilization is greater than 90% on any server where an Oracle instance is found to be running (There are various other servers maintained within the same OEM estate, so this filter was implemented to focus on the relevant hosts for the DBA team)

 

Here is how the interactive report for this particular item looked.

When building the initial revisions of the various items, I felt it important to get some context for greater insights and built in the historical indicators to compare the current value to the values of 7 and 30 days prior.

 

What the above image shows is that for some_host_01, there has been a gradual increase in the utilization for the /opt/apps filesystem over the past 30 days, which happens to be the mount point used for the Oracle software binaries.

 

That called for a logon to the host in question where, after some digging around, the following was discovered.

 

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms> du -sh *

99M     admin

12G     audit

12K     demo

12K     doc

168K    install

7.2M    jlib

38M     lib

1.1M    log

70M     mesg

144K    public

464K    utl

45M     xml

 

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> ls -1f | wc -l

2957049

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> df -h .

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/ora_vg-ora_lv

                       50G   44G  3.6G  93% /opt/apps

 

 

Well that wasn't expected! This specific directory filling up with audit files and consuming wads of space is a known issue documented in various MOS notes [Manage Audit File Directory Growth with cron (Doc ID 1298957.1) being the most prominent], to the extent that scripts have been deployed and implemented via cron jobs on all our systems to manage it.

 

What we missed here was that after a GI upgrade from 11.2.0.4 to 12.2.0.1, the shell script was not updated to reflect the new $GRID_HOME. Admittedly it was a bad idea on our part to hardcode the $GRID_HOME path in the first place and that's another positive that's come out of this, in that we've revised the script to be more flexible going forward. By the way, I'm including the line below as we've found it to be one of the more efficient methods to remove a large number of files on a Linux system.

 

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> find . -name "*.aud" -type f -delete

 

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> df -h .

Filesystem            Size  Used Avail Use% Mounted on

/dev/mapper/ora_vg-ora_lv

                       50G   32G   15G  68% /opt/apps

 

In closing, it's yet another lesson to not take those "simple" administrative tasks for granted and a great win for the APEX Oracle Database Dashboard!

 

Ian

 

P.S. For those of you interested, here is the SQL that queries the OEM repository to produce the report as above. As with anything you find on the internet, please take care when executing this within your own environment

 

 

Please reload

Recent Posts

Please reload

Archive

Please reload

Tags

Please reload