Dean Frickey has written an interesting article titled “A More Useful 404.” In it he describes how he uses a perl script to provide relevant information to the user while also providing immediate feedback to the developer so that, when possible, the problem can be fixed. I have done something similar on my sites for quite a while, but I use PHP instead of Perl.
One difference between my approach and that of Dean Frickey regards sending an email. I don’t like to send an email for each 404 page because I have noticed that most such pages are actually the result of attempted hacks (e.g., bots trying to find known exploits in popular third party scripts). So, instead of sending an email I write the relevant information (page trying to be accessed, the referring page used to reach that page and the timestamp) to a special 404 table I created in my database. To prevent table bloating, I do a table check to see if the target URL and the referring URL already exist and only write to the table if they don’t.
Of course, using a database to track 404 pages has its benefits, but you need to have some mechanism in place to ensure you regularly monitor and act upon legitimate 404 page problems. Since I already have a daily administrative script which I run each night and which sends me an email summarizing key webmaster information (number of new registrations, newsletter subscriptions, etc.), I just added a summary of all the 404 pages for that day as well.
Finally, I also have a generic contact script that I include on most of my websites in the contact page. Since some users who encounter a 404 page may actually want to be helpful, I also include that script to provide them a quick way to send me an email with any clarifying information they might deem appropriate.