Recovering Deleted or Altered Files
By Boris Loza, PhD, CISSP
Introduction
The art of recovering deleted or altered files requires much patience and some technical knowledge. In this article we will show you how one can do this if, of course, you do not have a backup of the file in question!
Recovering Text Files
If you suspect that an intruder has modified or deleted your log file, you may still try to recover what has been deleted from this file. When a file is deleted under the UNIX system, its inode number is set to 0 and disk blocks belonging to a file are marked “free” and returned to a pool of blocks that can be used by the system in case it needs it. If the blocks of the deleted or altered file were not used by the system for another file, this information is still present on the hard disk somewhere.
In this example, we are going to try to recover our log file. Let’s assume that we log all network connections with inetd –t. We suspect that intruder modified our /var/adm/messages file to hide her successful connection from June 4. Current output from the /var/adm/messages (Solaris OS):
$ tail messages
Jun 3 10:49:42 birch inetd[132]: exec[25727] from 10.56.49.194 2199
Jun 3 11:29:25 birch inetd[132]: exec[28958] from 10.56.49.194 2254
Jun 3 11:35:14 birch inetd[132]: telnet[29398] from 10.56.49.194 2255
Jun 3 14:04:21 birch inetd[132]: telnet[9711] from 10.56.53.55 57779
Jun 3 14:21:30 birch inetd[132]: ftp[10914] from 10.56.49.194 2430
Jun 3 14:51:04 birch inetd[132]: telnet[17225] from 10.56.49.194 2486
Jun 3 14:56:35 birch inetd[132]: telnet[17622] from 10.56.49.194 2487
Jun 5 09:55:00 birch inetd[132]: exec[14029] from 10.56.49.194 3248
Jun 5 11:13:33 birch inetd[132]: exec[19439] from 10.56.49.194 3281
Jun 6 14:17:14 birch inetd[132]: telnet[10520] from 10.56.49.194 3747
We can see that entries from Jun 4 are missing. That looks suspicious.
Because, as we already know, everything in UNIX is a file, we can use the grep utility against the device file in which the /var/adm/messages is located for the string “Jun 4”.
Let’s find on which disk device our /var/adm/messages file resides (this output is produced on Solaris 2.6):
$ df -k
Filesystem Kbytes used avail capacity Mounted on
/dev/dsk/c0t0d0s0 771110 509858 207275 72% /
/dev/dsk/c0t0d0s3 7361601 1993933 5294052 28% /usr
/proc 0 0 0 0% /proc
fd 0 0 0 0% /dev/fd
/dev/dsk/c0t0d0s4 290065 93826 167233 36% /var
swap 319808 848 318960 1% /tmp
We know (from the full path of the file: /var/adm/messages) that our file belongs to the /var file system. From the df –k output we found that /var file system was created on /dev/dsk/c0t0d0s4 partition of the c0t0d0 disk.
We will use grep against the /dev/dsk/c0tod0s4 partition:
$ su -
Password:
Sun Microsystems Inc. SunOS 5.6 Generic August 1997
# grep ‘Jun 4’ /dev/dsk/c0t0d0s4 > /tmp/123
# more /tmp/123
Jun 4 09:16:54 sundvl25 inetd[132]: telnet[2872] from 10.32.112.159 1137
Jun 4 09:23:06 sundvl25 inetd[132]: telnet[3306] from 10.32.112.159 1140
Jun 4 10:07:44 sundvl25 inetd[132]: ftp[6484] from 10.56.49.183 1072
Jun 4 10:08:17 sundvl25 inetd[132]: ftp[6519] from 10.56.49.183 1073
Jun 4 09:16:54 sundvl25 inetd[132]: telnet[2872] from 10.32.112.159 1137
Jun 4 09:23:06 sundvl25 inetd[132]: telnet[3306] from 10.32.112.159 1140
Jun 4 10:07:44 sundvl25 inetd[132]: ftp[6484] from 10.56.49.183 1072
Jun 4 10:08:17 sundvl25 inetd[132]: ftp[6519] from 10.56.49.183 1073
We redirected the output from the grep to the /tmp/123 file (choose any meaningless name in case the intruder is on the system) which is on a different partition of the hard disk. This will ensure that the data we are trying to recover is not overwritten.
So, we can see what entries had been deleted from /var/adm/messages file (based on the format of the messages file) by our intruder.
You can also restore any deleted text file if you know what to look for (what string of text to include into grep’s search string), if it was not overwritten by the OS of course!
This process is time consuming. Be patient. On a busy system it could take a lot of time to grep through the disk’s blocks.
Recovering Binary Files
You may easily recover a binary file if it is still running as a process on your system. There may be situations in which a hacker runs the application and deletes an executable file. The process will still run because a link to the process exists in the /proc/[PID]/object/a.out directory, but you may not find the file by its name running the find(1) command. You may identify the process number belonging to the deleted file you want to restore using ps command or lsof utility.
For example, let’s assume that we are going to restore a file that belongs to the process ID 22889 from the suspicious srg application that we found running on our system (Solaris OS):
# ps –ef | more
UID PID PPID C STIME TTY TIME CMD
root 0 0 0 May 10 ? 0:00 sched
root 1 0 0 May 10 ? 2:21 /etc/init -
……………..
root 22889 16318 0 10:09:25 pts/1 0:00 ./srg
root 16318 25824 0 08:56:27 pts/1 0:00 csh
If you need to recover an executable file that is still running on your system, simply run the copy command:
# cp /proc/2289/object/a.out /tmp/srg
This is because, as we already know, everything in UNIX is presented as a file, even a process. The /proc is a pseudo filesystem that abstracts the kernel’s process architecture and /proc/2289/object/a.out is the process’s executable binary.
Under Linux OS (kernel version 2.2.x and higher), you may try to recover a binary file that is not running on the system as a process. The following technique works well if the file size is not large then 12 blocks. This is because Linux inode table stores up to the first 12 block numbers of each file directly inside the inode table itself. To recover a deleted file, use the debugfs command to generate a list of all files that have been recently deleted. The following example generates a list of all deleted files for hda3 partition:
# echo lsdel | debugfs /dev/hda3
debugfs 1.27 (8-Mar-2002)
debugfs:
Inode Owner Mode Size Blocks Time deleted72473 500 20600 93 1/ 1 Tue Mar 29 10:38:44 2005
1 deleted inodes found
You can see that no filenames are given. If you have a list of hundreds of deleted files you would have to know the file size and when it was deleted in order to choose a right one!
To restore the file use debugfs to dump data from the inode to another file:
debugfs /dev/hdc6
debugfs 1.27 (8-Mar-2002)
debugfs: dump <72473> /tmp/my.file
debugfs: q
#
Check the /tmp directory. Your file should be restored!
You may also want take a look at Dan Farmer and Wietse Venema’s The Coroner’s Toolkit (TCT) that includes unrm and lazarus applications to automate this process.
Conclusion
Do not be frustrated if you can not recover a deleted or altered file. Sometimes chances to recover a file are very slim. Be patient and take everything with a sense of humour!
