How to recover deleted open files

In Linux, a file is deleted completed when:

1. No more hard link reference to the file
2. Processes opening the file is terminated

From why du and df show different filesystem usage (http://linuxscripter.blogspot.com/2013/11/why-du-and-df-show-different-filesystem.html), we know that if we delete an open file, Linux won't release its space until the opening processes are stopped.

So if we delete an open file by mistake, is there a way to recover it?
YES, we can check which process is opening the file, and recover file content by checking the process file descriptor.

Again, let's assume our Apache error_log is deleted, we can check which process is opening this file:

[root@centos ~]# lsof | sed -n '1p;/error_log.*deleted/p'
COMMAND    PID      USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
httpd     3155      root    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3157    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3158    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3159    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3160    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3161    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3162    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3163    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)
httpd     3164    apache    2w      REG      253,0      370      15396 /var/log/httpd/error_log (deleted)

From the output we know that, process 3155 still opens this file. Go to /proc/3155/fd/ to confirm this, 2w means error_log is opened for write, and we need to check softlink "2" in /proc/3155/fd.

[root@centos ~]# cd /proc/3155/fd/
[root@centos fd]# ls -l 2
l-wx------ 1 root root 64 Nov  5 09:46 2 -> /var/log/httpd/error_log (deleted)
[root@centos fd]# tail 2
[Tue Nov 05 09:46:35 2013] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Nov 05 09:46:35 2013] [notice] Digest: generating secret for digest authentication ...
[Tue Nov 05 09:46:35 2013] [notice] Digest: done
[Tue Nov 05 09:46:35 2013] [notice] Apache/2.2.15 (Unix) DAV/2 PHP/5.3.3 mod_wsgi/3.2 Python/2.6.6 configured -- resuming normal operations

To recover the content of error_log, we can just copy 2 to a temporary location, stop Apache and rename copy of 2 to /var/log/httpd/error_log

[root@centos fd]# cp 2 /tmp/error_log
[root@centos ~]# service httpd stop
Stopping httpd:                                            [  OK  ]
[root@centos ~]# mv /tmp/error_log /var/log/httpd/error_log
[root@centos ~]# service httpd start
Starting httpd:                                            [  OK  ]

why du and df show different filesystem usage

Today I saw a forum post asking on his system, why df shows over 200G space used, while du only shows 50G ussed.
Most probably this is caused by open files being deleted.
When a file is opened by a process, deleting the file won't release the space occupied by it. we also need to terminate the process, otherwise df and du will show different filesystem usage.

Let's assume du and df show huge difference for /var, we can check which open files are deleted and the processes opening them.

[root@linux ~]# lsof | sed -n '1p;/var.*deleted/p'
COMMAND    PID     USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
httpd     1779     root   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1810   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1811   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1812   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1813   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1814   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1815   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1816   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
httpd     1817   apache   10w      REG      253,0    22034       8117 /var/log/httpd/access_log (deleted)
From the output we can see that access_log is deleted, but Apache was not restarted, httpd process still has this file open.
to release the space, we can restart httpd process:
[root@linux ~]# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
Now check again
[root@linux ~]# lsof | sed -n '1p;/var.*deleted/p'
COMMAND    PID     USER   FD      TYPE     DEVICE SIZE/OFF       NODE NAME
And du df report same file system usage.

Before we restart httpd process, Linux won't release the space used by access_log, if access_log is deleted by mistake, is there a way to recover it?
Yes, I will demo how to recover deleted open files in http://linuxscripter.blogspot.com/2013/11/how-to-recover-deleted-open-files.html

How to get the process listening on certain port

At times we are asked: "what program is listening on port XX?"

In Redhat, we can easily get this using lsof. let's say if we want to know which program is listening on port 80.
[root@localhost ~]# lsof -i :80
COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
httpd   2047   root    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2049 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2050 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2051 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2052 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2053 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2054 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2055 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)
httpd   2056 apache    4u  IPv6  12636      0t0  TCP *:http (LISTEN)

From the output, we know that httpd is listening on port 80.

Another way to get the process listening on certain port is using netstat.
[root@localhost ~]# netstat -anp | grep 80
tcp        0      0 :::80     :::*            LISTEN      2009/httpd
 
[root@localhost ~]# ps -ef | awk '$2 == 2009'
root      2009     1  0 23:21 ?        00:00:00 /usr/sbin/httpd
 
[root@localhost ~]#
In AIX, most of the time it has no lsof installed, and the netstat is also different from Redhat, but we still can get our question answered.
$ netstat -Aan | grep '.22 ' | grep LISTEN
f1000e0001382bb8 tcp4   0  0  *.22          *.*        LISTEN

$ rmsock f1000e0001382bb8 tcpcb
The socket 0xf1000e0001382808 is being held by proccess 3670142 (sshd).
 
$ ps -ef | grep 3670142
root 3670142 3014676   0   Sep 06      -  0:00 /usr/sbin/sshd

Please note rmsock will not remove the socket, you can confirm this by checking the content of /var/adm/ras/rmsock.log.

reference: http://unix.ittoolbox.com/groups/technical-functional/ibm-aix-l/determine-which-process-is-listening-on-a-port-without-using-lsof-on-aix-1468555