Windows

Five tools for getting rid of duplicate files

If your system has gotten gunked up with duplicate files, clean up the mess with one of the handy tools on this list.

Duplicate files and folders can clutter up a user's computer, and they can wreak havoc on collaborative work. Although manually tracking down dupes is hardly practical, a number of specialized tools can handle the task.

Note: This list is also available as a photo gallery.

1: WinMerge

WinMerge (Figure A) is an open source package for Windows that can find duplicate files and perform merge and comparison of various text file types. The upcoming version 3 will work on both Windows and Linux platforms. The tool can perform some advanced folder comparison, which is a great feature for developers and system administrators. And because you can run it directly from the executable file, it's useful on a desktop support tech's USB drive.

Figure A

WinMerge

2: DoubleKiller

DoubleKiller (Figure B) offers powerful comparison features, including hash comparison and filtering of which files to include in the comparison. One neat feature is the ability to compare a small number of "fresh" files against a collection of "known" files, which could be a considerable help in a team or collaborative environment. It's available as both a free version and a paid version. The paid version is substantially more useful than the free version.

Figure B

DoubleKiller

3: Easy Duplicate Finder

Easy Duplicate Finder (Figure C) lives up to its name, being easy to use indeed. But it also includes some advanced duplicate detection options, like MP3 tag and JPEG EXIF comparison. And it can hook into Outlook to detect duplicate emails, which can be a useful function for some. Easy Duplicate Finder is free with a restricted license (which can be upgraded), but it will look to add browser toolbars when you install it.

Figure C

Easy Duplicate Finder

4: dupeGuru Music Edition

Despite its name, dupeGuru Music Edition (Figure D) is not just for comparing music files. It adds the music functionality on top of a robust duplicate finding application. Using a smart filename comparison algorithm, dupeGuru Music Edition finds duplicates even when the files have different names. dupeGuru runs on Windows, OS X, and Linux. It also follows a "fairware" pricing model: You can see how many hours the developers put into the application and how many have been paid for by donations and then make a donation yourself.

Figure D

dupeGuru Music Edition

5: Duplicate Images Finder

Duplicate Images Finder (Figure E) is a special-purpose application that compares image files and detects duplicates based on image similarity. Now be prepared --the matching can take quite some time. But the comparisons are much more difficult than those performed by the other apps on this list. This is an excellent tool for someone who takes a ton of pictures and wants to easily find the best ones to keep or print. Duplicate Images Finder is open source.

Figure E

Duplicate Images Finder

About

Justin James is the Lead Architect for Conigent.

31 comments
Ozaruu
Ozaruu

Checking through the reviews on CNet, none of the apps listed here, either in the article itself or the other comments (indeed none of the duplicate finders I can find on CNet) appears to be *totally* free and harmless - they either install unwanted software, or expect payment for an 'enhanced' version with features that you'd really need. For now I'll be sticking with my original Dupeless from PC Magazine around 1998 - it was totally free and unrestricted, still works on Windows 7, and does byte-by-byte across all (or selected) drives/folders, etc. However, I can't find a legal download link for this version. There is an updated version for under US$8 at http://www.pcmag.com/article2/0,2817,2422145,00.asp which could be good. Another genuine freebie that has been recommended (and features a 'percentage' check, useful e.g. for photos where the only difference is a few bytes indicating the date downloaded) is DupFinder, here: http://www.12oclocker.com/programs.htm

markbun
markbun

I use "DuplicateFilesDeleter" a great fix for finding and removing duplicate files.

pczyky
pczyky

Doublekiller looked good to me, but it looks to be last updated in 2007. And duplicate image finder was last updated in 2009. I'm reluctant to try apps that were released way before the operating system I am using. Hopefully some more recent products exist and can be swapped into this list. (The other three apps look to be actively supported.)

tiwad
tiwad

Hi all There is also a Panaustik (en.panaustik.com). It is abble, not only to find exact duplicates, but also identify pictures showing the same subject (even with different view point or zoom factor). Y'll be surprise to see how much time you shot the same subject! A must have!

GRAYGOAST52
GRAYGOAST52

I DOWNLOADES IT AND PAID FOR IT IT WORKS VERY WELL FOR ME, I ADD EVERY FOLDER TO CHECK DOR DUPILATE FILES I ALSO RUN IT ON AUTOMATIC AND IT DOES FINF WHAT DUBBLED OM NY COMPUTER I LIKE IT

kamchatsk
kamchatsk

One of the best, cleanest, free apps I have used, with byte-for-byte comparison.

AllanBraun
AllanBraun

I use Directory Report It can do a quick search based on same name, size Then you can manually binarily compare the results

BuckG
BuckG

I was surprised you didn't include Check Identical Files in your list; it's been very useful to me inasmuch as it can find identical files of any type, regardless of file name or location (it can also limit comparisons to specific directories, as granular and exclusive - or inclusive - as you like). It's one of the best software acquisitions I've ever made.

yass974
yass974

Nice list... But what about macosx ? Any tools ?

walter.kirsch
walter.kirsch

Can I use the lnk facility to eliminate duplicates. I've never used lnk before, so any recommendations? OK, Hows about junction points or hard links, or whatever they're called?

collins5ed
collins5ed

Anything for dedupping video files and sippets of videos? Had a program to do that but it got lost when my PC crashed a few years ago.

ait10101
ait10101

Anthonym: Try Similarity. Gives the information you are asking for.

Joe_Wulf
Joe_Wulf

Consider this superior filemanager (at: www.ztree.com). Has its history in the stellar XtreeGold from the past. One of its many functions is finding duplicate files, based on the criteria you select, across ALL disk drives.

anthonym
anthonym

what do you recommend to sort out your music - especially - quality versus types etc...

Tink!
Tink!

Can totally use a music file duplicate finder!

CdnITGuy
CdnITGuy

I'm currently working on de-duplicating our file server and found a great tool Directory Compare 2.3 by Glenn Alcott. Will give you a side by side comparison of two folders and offsets non-similar files so its a great visual queue to see what is in fact a duplicate. I highly recommend this app having worked through over 85,000 duplicate files.

Chilidog67
Chilidog67

Been using Auslogics Duplicate File Finder for years and I love it. I'll definitely give some of these a look too.

allegory
allegory

Thank's a mil, this is very useful.

dogknees
dogknees

Do these compare the actual content or just the name, date, etc? And do they find identical files with different names?

Neon Samurai
Neon Samurai

I've been on the look-out for a duplicate image finder. Offhand, will this find images of different sizes? For example; the same image at two different resolutions?

allancass
allancass

@Ozaruu I totally agree that the list above is very poor and that the tools are not  free and install some crap. The tool that I recommended below is FREE in the modes "100% identical files" and "Similar File Names" and it does not install any extra crap on your system. I am using it for quite some time and I am really happy with it. Here is a comparison table: http://www.mindgems.com/products/Fast-Duplicate-File-Finder/Fast-Duplicate-File-Finder-Features.htm

You are not right about photos. Those duplicate finders can not find exact duplicate photos in different formats, just because the complete file data is different. That is why the company that provides the above tool has a dedicated duplicate image finder (which is not free): http://www.mindgems.com/products/VS-Duplicate-Image-Finder/VSDIF-About.htm

but it is well worth the 25 bucks as it supports more than 100 image formats and more than 300 raw formats. I have tested quite some tools an I can assure you that the ones I recommend are of the highest quality. The photo finder tool has a free demo so give it a try.

dymsharon
dymsharon

I use NoClone duplicate file finder to delete duplicate photos, mp3 files, and outlook emails. It works perfect! Strongly recommended!

NickNielsen
NickNielsen

One of them, at least: [i]dupeGuru Music Edition finds duplicates even when the files have different names. [/i] I don't know if any of the others will or not, although Justin does say that Doublekiller will compare based on file hash.

Justin James
Justin James

Some start with the name alone. Most do a byte sum or similar hash, then compare those, so things like date, name, etc. are not part of the equation (or are optionally part of the equation). J.Ja

Justin James
Justin James

Yes, Duplicate Image Finder does a fuzzy match on the appearance of the pictures. It will even find pictures in the same series if they are of *almost* the same thing (like if you have frames from a film). J.Ja

Neon Samurai
Neon Samurai

Cheers, I'll be downloading this little utility tonight

gjm123
gjm123

I will be downloading that. OK, I've said it is useful, but maybe it is more my appreciation of pattern recognition than anything else.