How to recover deleted open files

In Linux, a file is deleted completed when:

1. No more hard link reference to the file
2. Processes opening the file is terminated

From why du and df show different filesystem usage (http://linuxscripter.blogspot.com/2013/11/why-du-and-df-show-different-filesystem.html), we know that if we delete an open file, Linux won't release its space until the opening processes are stopped.

So if we delete an open file by mistake, is there a way to recover it?
YES, we can check which process is opening the file, and recover file content by checking the process file descriptor.

Again, let's assume our Apache error_log is deleted, we can check which process is opening this file:

[root@centos ~]# lsof | sed -n '1p;/error_log.*deleted/p'
COMMAND    PID      USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
httpd     3155      root    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3157    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3158    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3159    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3160    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3161    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3162    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3163    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3164    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)

From the output we know that, process 3155 still opens this file. Go to /proc/3155/fd/ to confirm this, 2w means error_log is opened for write, and we need to check softlink "2" in /proc/3155/fd.

[root@centos ~]# cd /proc/3155/fd/
[root@centos fd]# ls -l 2
l-wx------ 1 root root 64 Nov  5 09:46 2 -> /var/log/httpd/error_log (deleted)
[root@centos fd]# tail 2
[Tue Nov 05 09:46:35 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Nov 05 09:46:35 2013] [notice] Digest: generating secret for digest authentication ...
[Tue Nov 05 09:46:35 2013] [notice] Digest: done
[Tue Nov 05 09:46:35 2013] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_wsgi/3.2 Python/2.6.6 configured -- resuming normal operations

To recover the content of error_log, we can just copy 2 to a temporary location, stop Apache and rename copy of 2 to /var/log/httpd/error_log

[root@centos fd]# cp 2 /tmp/error_log
[root@centos ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@centos ~]# mv /tmp/error_log /var/log/httpd/error_log
[root@centos ~]# service httpd start
Starting httpd:                                            [  OK  ]