Recent Posts

Archive

Tags

Audit file pileup in $GRID_HOME/rdbms/audit

One of the items highlighted by the dashboard is where filesystem utilization is greater than 90% on any server where an Oracle instance is found to be running (There are various other servers maintained within the same OEM estate, so this filter was implemented to focus on the relevant hosts for the DBA team)

Here is how the interactive report for this particular item looked.

When building the initial revisions of the various items, I felt it important to get some context for greater insights and built in the historical indicators to compare the current value to the values of 7 and 30 days prior.

What the above image shows is that for some_host_01, there has been a gradual increase in the utilization for the /opt/apps filesystem over the past 30 days, which happens to be the mount point used for the Oracle software binaries.

That called for a logon to the host in question where, after some digging around, the following was discovered.

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms> du -sh *

99M admin

12G audit

12K demo

12K doc

168K install

7.2M jlib

38M lib

1.1M log

70M mesg

144K public

464K utl

45M xml

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> ls -1f | wc -l

2957049

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> df -h .

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/ora_vg-ora_lv

50G 44G 3.6G 93% /opt/apps

Well that wasn't expected! This specific directory filling up with audit files and consuming wads of space is a known issue documented in various MOS notes [Manage Audit File Directory Growth with cron (Doc ID 1298957.1) being the most prominent], to the extent that scripts have been deployed and implemented via cron jobs on all our systems to manage it.

What we missed here was that after a GI upgrade from 11.2.0.4 to 12.2.0.1, the shell script was not updated to reflect the new $GRID_HOME. Admittedly it was a bad idea on our part to hardcode the $GRID_HOME path in the first place and that's another positive that's come out of this, in that we've revised the script to be more flexible going forward. By the way, I'm including the line below as we've found it to be one of the more efficient methods to remove a large number of files on a Linux system.

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> find . -name "*.aud" -type f -delete

grid@some_host_01:/opt/apps/oracle/grid/12.2.0/rdbms/audit> df -h .

Filesystem Size Used Avail Use% Mounted on

/dev/mapper/ora_vg-ora_lv

50G 32G 15G 68% /opt/apps

In closing, it's yet another lesson to not take those "simple" administrative tasks for granted and a great win for the APEX Oracle Database Dashboard!

Ian

P.S. For those of you interested, here is the SQL that queries the OEM repository to produce the report as above. As with anything you find on the internet, please take care when executing this within your own environment