How to recover deleted open files

In Linux, a file is deleted completed when:

1. No more hard link reference to the file
2. Processes opening the file is terminated

From why du and df show different filesystem usage (http://linuxscripter.blogspot.com/2013/11/why-du-and-df-show-different-filesystem.html), we know that if we delete an open file, Linux won't release its space until the opening processes are stopped.

So if we delete an open file by mistake, is there a way to recover it?
YES, we can check which process is opening the file, and recover file content by checking the process file descriptor.

Again, let's assume our Apache error_log is deleted, we can check which process is opening this file:

[root@centos ~]# lsof | sed -n '1p;/error_log.*deleted/p'
COMMAND    PID      USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
httpd     3155      root    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3157    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3158    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3159    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3160    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3161    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3162    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3163    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3164    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)

From the output we know that, process 3155 still opens this file. Go to /proc/3155/fd/ to confirm this, 2w means error_log is opened for write, and we need to check softlink "2" in /proc/3155/fd.

[root@centos ~]# cd /proc/3155/fd/
[root@centos fd]# ls -l 2
l-wx------ 1 root root 64 Nov  5 09:46 2 -> /var/log/httpd/error_log (deleted)
[root@centos fd]# tail 2
[Tue Nov 05 09:46:35 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Nov 05 09:46:35 2013] [notice] Digest: generating secret for digest authentication ...
[Tue Nov 05 09:46:35 2013] [notice] Digest: done
[Tue Nov 05 09:46:35 2013] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_wsgi/3.2 Python/2.6.6 configured -- resuming normal operations

To recover the content of error_log, we can just copy 2 to a temporary location, stop Apache and rename copy of 2 to /var/log/httpd/error_log

[root@centos fd]# cp 2 /tmp/error_log
[root@centos ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@centos ~]# mv /tmp/error_log /var/log/httpd/error_log
[root@centos ~]# service httpd start
Starting httpd:                                            [  OK  ]

Oracle: How to roll forward standby database using archivelogs

At times oracle standby database may get out of sync with primary database. After fixing the issue causing standby out of sync, we need to roll forward standby database so that it can sync up with primary again.

To roll forward standby database, we can use archivelogs.

1. If the archivelogs are deleted from primary already.

We can restore the archivelogs from backups. Suppose our catdb stores RMAN catalog.

we can use this code to restore the archivelogs.

connect target /

connect catalog catuser/catpass@catdb

RUN {
allocate channel c1 type 'SBT_TAPE' 
parms 'ENV=(NB_ORA_CLIENT=prod1)' maxopenfiles 10;

allocate channel c2 type 'SBT_TAPE'
parms 'ENV=(NB_ORA_CLIENT=prod2)' maxopenfiles 10;
restore archivelog from time 'sysdate - 1' until time 'sysdate';

release channel c1;
release channel c2;

}

Depending on how long the standby has been out of sync, we can change the 'from time' accordingly. 

2. If the archivelogs are still on primary database.

We can simply copy them to the standby database,
after copying archivelogs to the standby, we need to catalog the files copied over.

RMAN> catalog start with '/tmp/logs_from_primary';

Otherwise oracle cannot recognize these files.

Once we have the archivelogs, either through method 1 or method 2,
we can roll forward the standby database.

SQL> recover managed standby database disconnect;