Discussion Locked

The nitemare of unused files on a website.

0 helpful
28 replies
Closed to new repliesThis discussion is closed. It stays readable, but it takes no new replies.

I recently took over the management of a website that has nearly 3,000 html pages and around 500 images. The problem is that this site has been created over several years and several other web designers. Each designer apparently left the previous designers files on the site whenever they made a design change. The last designer had been adding pages to his design version for several years. It would appear that they all used MS FrontPage.

Now I'm faced with a site that has about 5,000 files and no good way to determine what files are OK to delete from the webserver without messing up what is already there. FrontPage has several reports in it, like the "Unlinked Files" report. But I discovered that it considers any file that is not listed inside an tag is considered "unlinked". So an image file that is on a page using the tag is considered unlinked. Rendering that report useless to me.

I've looked at several different utilities that I found searching the web and there are very few that will give you a list of files that "for sure" are not being referenced by any of the currently active web pages. The one that had much promise descibed it's process as parsing all the files that make up your site by parsing the site from the home page and it could even handle file references that are located inside .js files that are called from an html page. Then you give it your FTP login info and the starting website folder and it will gather a list of all files on the webserver. In the final step it compares the two lists and whatever files are not in both lists are a pretty safe bet are not being used by the current website.

However, after downloading this fairly old application and install it I was not able to make it work. :(

I can't believe someone else hasn't had this kind of problem and hopefully can share with me their solution. I can't delete a file and then browse through 2,000 web pages to see if it was being used by the website. It would take me way too much time.

I can't use FrontPage to publish the site elsewhere because FP will copy all files whether they are being used by the site or not.

I look forward to reading the responses to this request.

Thanks in advance to all.

28 replies

by tony hopkinson

Well first you don't need to use FP to copy your site.
Presumably you are in control of the web server. Just create a copy site with a different name and copy the files in and test. Course that's a shed load of testing.

I'm sure there's a logging option in IIS which will give you a list of all pages requested through it. Site Server seems to ring a bell. It's a bit of messing about but you should be able to get a list of all the HTML files used since you started logging. Even better these will be the most popular ones. Once you've got that move the unpopular ones out of there and log page not founds (404) for the one or twenty that you shouldn't have. Cant't think of another way round it without developing an IIS Filter dll.

When you've set this in motion, you can hunt down those who did this to you and give them a good kicking.

PS Copy your site anyway. Develop on the copy and move the new stuff over when it's tested, that way you or your successor won't have to go thru all this again.

by lynbor

Sorry, thought I mentioned this in my first post, but I'm working with a Linux web server, so no IIS. But I do have access to the server logs and you are correct in that I could filter out all pages that have been viewed. However, that isn't really going to save me a whole lot of time over just doing the job one page at a time nor is it going to be accurate enough. Mainly because there are "doorway" pages that are on the site for search engine ranking. It's likely that several have never been accessed and as such would not show in the access log making that report just as unreliable as using FrontPage reporting features.

I just can't believe that there isn't some functional little program out there that can spider a site pulling out all files referenced by the html and .js pages, then comparing that list to a complete filename dump of the site to determine what files are not being used by any web pages. If I don't find something I guess it would be worth the effort to dust off the old programming books and make such a utility myself. :(

Thanks for the suggestions.

by tony hopkinson

in Technical Q&A and ask about logging which pages are used in apache. I'd be shocked if there isn't a way of doing this. I'm not sure a site spider will do you any good, depends on how your site is linked together. But capturing which pages have been requested over a decent period of time will cut down the work load enormously. Setting up a development site under linux is easy as well. In fact you could run apache twice and keep it completely separate.

by apotheon

Oh, that makes things MUCH easier for something like this. It sounds to me like you need a Perl script. Assuming you have root access on the server, Perl should make this a trivial operation -- and if you ask around in the right places, you'll probably find that someone else has already written a script to do exactly what you want, or nearly enough to make it a simple thing to modify it for your purposes.

If you're interested in learning Perl, I recommend http://www.perlmonks.org very, very highly. I've been far too busy doing web development with PHP lately to put in much time there, but there are a great many very good Perl programmers there and there's a wealth of information. You may even find some sample code there that'll do what you need.

by deepsand

Assuming that you're not blocking spiders from indexing portions of your site, look to the web log for recent records of your site being crawled.

They'll be easy to find, as well behaved spiders first look for the robots.txt file.

If you are blocking portions of your site, look for ill behaved spider activity.

That should give you a working list of the files currently active.

by lynbor

Exactly my thoughts, however in searching the various script depositories I have found nothing pre-made that addresses this problem.

I don't wish to learn perl, if given a choice I think I'd prefer to learn PHP, but really I don't have time to learn either.

If someone here doesn't have a good working solution I'm probably going to end up hiring someone to make a nifty script to do this and I'll hire the script out to other designers until it has paid for itself. After that I'll put it up on one of those script depositories as a free utility. :)

by apotheon

For $100/hr I'll do it for you -- or for $50/hr if you don't mind me licensing it to you under a FLOSS license rather than "selling" it to you. I can't guarantee that I'd get it done quickly, with the workload I'm laboring under right now, so if rapid development is a concern I'm probably not the guy to do it right now.

If you want someone to do it quickly, I can point you at some people I know who are very good at Perl scripting.

by Member 31336961

Since you're on a linux box, lynx may be installed. It can be made to crawl your site and will give you a list of all files travelled with images referenced

lynx -traversal -crawl -image_links -realm http://yoursite.com

start this in a directory of it's own because it will also output a file lnk00000.dat for each page found.

Look in traverse.dat for all your pages. Look in reject.dat for external links and images.

You may have to start this at each of your feeder pages to make sure you find everything.

by lynbor

Thanks Kevin, I'll give Lynx a look. Just to be clear, my web server is running RH Linux in a remote server farm location and I'm running Windows on my work PC, but I also have a Linux machine for playing with here that I can do your suggestion on.

Thanks again for your advice.

by apotheon

On RedHat, I believe the Links browser is installed by default. You may want to try that out, since I think it's there instead of Lynx. It's essentially the same thing, just with a slightly different (homonymous) name.

by tony

I've used Visio's site mapping feature to map out the flow of sites a lot like that. It's not going to tell you which files aren't being used, but it should find everything that is being referenced, giving you something to go off of.

Chances are with all those files that very few are actually important and the rest are older copies of the same thing, it'll be easier to focus on the smaller relevant pile than the larger crap pile.

Tony

by chaleepas

Download the whole site using the FrontPage, switch to the view that shows a tree of how each page is connected. If the pages aren't connected, then they aren't being used. Does that help a little?

by lynbor

That works for the HTM pages, but not for the image files. Plus, with somewhere near 2,000 web pages I would have to display each page, one at a time and write down the linked pages if there weren't already on my written list. As you can see, this method would take a very long time. And that is what I'm trying to avoid. Hours or days of hunting down dead files.

by mikiel

I think you're screwed.

While eliminating files that have been accessed will cut down on those you need to search through - not being accessed in 6 months or a year doesn't mean it's not a legitimate page.

A brute force program that looks to see if any other page/file references file X sounds great. However, if the previous owners just kept old files, then old file A will be linked to from old file B and thus your brute force program will not recognize that old file A is no longer being used by the real site.

A program that spiders a site starting at, e.g., the homepage would be good. (I don't use FrontPage so I'm not sure if it has this feature or not.) Of course such a program starting at the homepage may or may not pick up images depending on what the program considers to be "linked". Also such a program might miss the "doorway" search engine ranking pages you mentioned ('cause no internal pages go to the doorway pages), but hopefully these are not too many and are known by you. If you go this route, be careful that some of the orphaned files aren't active application pages used internally by some other group at your company that the previous webmaster was nice enough to post to the site's directory without linking to them.

If you can remove all the webpages that aren't being used, then perhaps the remaining set of webpages will be much smaller and some sort of text search (for .gif and .jpg) of those webpages will tell you all the images that are still being used.

Good luck!

by chaleepas

One of the things i stumbled onto accidentally with Frontpage is that it appears to have the ability when importing a website from an http address is only following items that are linked together and bringing in those files only.

At least that is how it appeared to work when I tried it.

You might want to try opening a fresh frontpage and importing the website from it's http: address and see what results you get.

by apotheon

Don't use Frontpage to work with your files unless you want them all altered so that they likely won't work properly with non-MS software.

by charliespencer

I wouldn't save the FP results, but you should be able to compare the files it imports with the files on the site. If FP didn't pull it, it may indicate the file can be deleted. It's a starting point.

by too old for it

I am setting up a new webserver and made sure that the Front Page extensions were turned off.

Just me I guess ...

by lynbor

Be careful about assuming what FrontPage does or does not do. The fact is that when you import or PUBLISH a website, FrontPage copies every single file from every single folder. Even more, if you have an empty folder on the destination site FrontPage will delete that folder from the destination site if you don't have the same empty folder on the source site. This is very evil behavior in my opinion.

But the bottom line is that the import or publish process is no help in filtering out file that are not longer referenced by the website.

by apotheon

"[i]This is very evil behavior in my opinion.[/i]"
I'd have a hard time disagreeing with that one.

by tony hopkinson

Emphasis on appears, Front Page is a horrible product. I would n't recommend to my worst enemy's favourite dog after it bit me.
I suffered it for about a week and then binned it in favour of two far more useful tools. A book about HTML and notepad.exe. That was in 98, I've seen, read and heard nothing from anyone that would even start to change my opinion

by apotheon

Pulling copies of the files doesn't mean you have to keep what you downloaded, after all.

by lynbor

I've tried this and unfortunately FP pulls every file, whether the site is linked to it or not, so this method actually gains me nothing in trying to determine which pages are no longer being used by the website. Thanks for the warm thoughts though.

by tenet2jjc

I realize that this is an old thread, but the company, Concept Draw has just released a program that may serve for this and future needs.

The program is "ConceptDraw WebWave," and it can map the components of a site, including images. The program will create an actual diagram of the site and the site's components.

Best of all, a free trial is available.

http://www.conceptdraw.com/en/products/webwave/main.php

Good luck.

by tenet2jjc

I realize that this is an old thread, but the company, Concept Draw has just released a program that may serve for this and future needs.

The program is "ConceptDraw WebWave," and it can map the components of a site, including images. The program will create an actual diagram of the site and the site's components.

Best of all, a free trial is available.

http://www.conceptdraw.com/en/products/webwave/main.php

Good luck.

by terrypeck

You already have a method for HTML pages courtesy of cpacshen. I don't imagine this is perfect but it worked for me. Follow the steps below exactly and you'll identify unlinked images and have them taken out of your site, safely stored away in a folder just in case.

FIRST FIND THE UNLINKED IMAGE FILES:
1. Back up the entire offline website.
2. Open Frontpage.
3. Make a new folder called Unused_Images.
4. Choose View/Reports/Unlinked Files.
5. View by Type.
6. Highlight all bmp files.
7. View by Links To.
8. You will have several blocks of highlighted bmp files.
9. Choose the highlighted block of files with 0 (zero) Links To.
10. Cut and Paste to the Unused_Images folder.
11. Repeat steps 2 to 10 for all image extensions (gif, png, jpg, etc.)
12. Exit Frontpage.
13. In Windows Explorer, locate the Unused_Images folder and move it to a location outside the offline website.

NEXT EXAMINE ALL JS FILES FOR IMAGE LINKS (unfortunately, they do not count as links in the Reports).
14. Open Frontpage.
15. Choose View/Reports/Files/All Files.
16. View by type.
17. Highlight all the js files.
18. Right click and open.
19. Choose Edit/Find and search for all image extensions (bmp, gif etc.) in Open pages(s) only - Find in source code.
20. Check out the references found one by one by double-clicking on them and looking at the pages. (This might not be as onerous as it sounds - there will likely be only a few references to image files in js files.)
21. Cross check any js file references found with images in your Unused_Images folder.
22. Import any required images from the Unused_Images folder offsite to your website image folder.

THIS WOULD BE ENOUGH IF ALL JS SCRIPTS WERE EXTERNAL. IF NOT, KEEP GOING...
23. Choose Edit/Find and search for all image extensions (bmp, gif etc.) in Source but with a single quote after each - like this: bmp', jpg' etc. This will catch most, but possibly not all image references within internal javascript routines (some might be double-quoted).
24. Cross check any js file references found with images in your Unused_Images folder.
25. Import any required images from the Unused_Images folder offsite to your website image folder.

I've bound to have missed something. If so, please let me know. But it cleaned up my site.

This discussion is closed to new replies.