Issue 1

Got error when trying to login into sql prompt as SYS

ORA-09925: Unable to create audit trail file

Linux-x86_64 Error: 30: Read-only file system

Additional information: 9925

ORA-09925: Unable to create audit trail file

Linux-x86_64 Error: 30: Read-only file system

Additional information: 9925

Issue 2 (related to above issue#1)

I then restarted the VM but got the issue the following error and the VM wouldn't boot

/dev/rdsk/string: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY

  (i.e., without -a or -p options)

 *** An error occurred during the file system check.

 *** Dropping you to a shell; the system will reboot

 *** when you leave the shell.

Give root password for maintenance

 

(or type CTRL-D to continue)

Cause of Issue

1. Above Issue 1 was related to issue 2.

2. It appears that I had one of the Fusion Middleware server had gone down abrubptly because of low disk space, and subsequent to that I couldn't shut down the database. Because of this issue Issue#1 started appearing and I had to shutdown the VM without shutting down the database.

3. Issue#2 started appearing when I restarted the VM without shutting down the database. Naturally the abrupt shutdown had impact on the virtual harddisk sectors.

4. Technical explanation for Issue#2 - During a boot, the /etc/rcS script runs the fsck(1M) command to check the integrity of file systems marked "fsck" in /etc/vfstab. If fsck(1M) cannot repair a file system automatically, it interrupts the boot procedure and produces this message. When fsck(1M) gets into this state, it cannot repair a file system without losing one or more files, so it wants to defer this responsibility to you, the administrator. Data corruption has probably already occurred.

Resolution

1. The solution to Issue 1 has been pretty well described under MOS Doc ID 1227964.1. 

2. The solution provided in above MOS note is to delete audit files from directory associated with parameter audit_file_dest. This full path of audit_file_dest parameter can be found from spfile<SID>.ora located under $ORACLE_HOME/dbs directory.

3. In my case the VM Harddisk was created with dynamic allocation, and hence it was dependent on host machine harddisk space. To fix Issue#1,  I simply made space in the host machine harddisk (on which VM harddisk was created) by moving large files. And hence I didn't had to delete my db audit files from directory associated with parameter audit_file_dest.

4. The next step was to start the VM. When I started VM I got the above mentioned Issue#2.

5. I have taken the following steps to fix issue#2

5.1. Run fsck in single user mode. The produre to run fsck in single user mode is as follows:

1. At grub boot screen (after restart)
2. Select the kernel
3. Press the e key to edit the entry
4. Select second line (the line starting with the word kernel)
5. Press the e key to edit kernel entry so that you can append single user mode
6. Add word 'Single' to the end of the (kernel) line
7. Press ENTER key
8. Now press the b key to boot the Linux kernel into single user mode
9. When prompted give root password and you be allowed to login into single user mode.

5.2. First run fsck -n on the file system, to see how many and what type of problems exist. 

5.3. The next step is to fix your VM harddisk for bad sectors (preferably for all user created partitions). Use following command to do it:

fsck -yvf /dev/sda1

where

-y — cause the fs-specific fsck to always attempt to fix any detected filesystem corruption automatically. 

-f — force a check even if reported in a clean state 

-v — Produce verbose output, including all file system-specific commands that are executed.

5.4. Reboot the VM. And this time the VM will start without any issues.

5.5. Login as root > then open a new terminal > start the database from sql prompt > the database will start without any issues (and you will not see error reported above under issue#1 now)

Environment

• Oracle Fusion Middleware 11g
• Oracle Linux 5.7