best way to consolidate multiple back up and duplicates in a single place.

Dump everything on your NAS and run a dedupe app. Many free ones, just hope it. Apparently CCleaner can do this too https://www.laptopmag.com/articles/remo ... windows-10

Get something which will recurse folders.

I also suggest, aside from your NAS which I will assume has some kind or fault tolerance, raid1, raid5, raid6....get an external drive, once you dedupe, do a full back up to it, then a weekly/monthly resync, then throw it in a safe. Also get a safe, it's less than $100.

Think fire, or ransomware.

I'll just add on to @vodka 's point that if you use a dedupe app, make sure you select one that does CRC-based deduplicating - where it creates the unique checksum of the file. Some de-dupe program in "fast" mode, the really "dumb" (lame) ones, or geared towards more basic users don't do this CRC-based check which is the way of getting a true unique signature for the file's contents - they might only check exact name (which doesn't help multiple names for the same file, leaving extra duplicates) and not "contents", or size (and possibly extension with size). You'd be surprised, as you increase the number of files to check, how many actually share the same size (including pictures take one after the other with only subtle differences) and you don't want to either skip legitimate duplicates or toast perceived duplicates that really aren't. Some attempt to speed things up by doing a CRC sampling at various parts of the file which is better than the other "dumb" (name, size) methods but not true CRC.

https://www.duplicatefilter.com/ supports CRC checking and I use it currently. I used to like "NoClone" a lot but it looks like it's dead/not supported. The better ones let you select a default directory to remove any duplicates as the priority "delete from here first" by "smart" marking them and ensuring to keep at least one of the original.

vodka wrote: ↑ Dump everything on your NAS and run a dedupe app. Many free ones, just hope it. Apparently CCleaner can do this too https://www.laptopmag.com/articles/remo ... windows-10

Get something which will recurse folders.

I also suggest, aside from your NAS which I will assume has some kind or fault tolerance, raid1, raid5, raid6....get an external drive, once you dedupe, do a full back up to it, then a weekly/monthly resync, then throw it in a safe. Also get a safe, it's less than $100.

Think fire, or ransomware.

When you say safe are you talking physical safe or is it a software ?

fishing4one wrote: ↑ When you say safe are you talking physical safe or is it a software ?

Physical. You should have one for the most important stuff, passports, etc. I have 2 HDs, I rotate them 1/month, one of them is always in the safe.

Thank Vodka and Casualcauli,
Was reading for the last few days.

Ended making 3 copies of all my data in 3 external drive.
will run CCleaner, Duplicatefilter, and AllDup on them. will then throw them in to a 4th Drive together and run one of the pogram.

The final copy should be as clean as possible without loosing any data.

By the way the most important thing I learned is the word Deduplicate. That is what send me down the right search path.

I'm in the exact same boat. Got 5 different external HD and 2 laptops with lots of duplicates. I just bought a QNAP and put everything on it. Will have a copy before cleaning and after cleaning.
I plan to set up q sync and q filing afterwards to limit the duplicates we will see if that actually works.
At one point in time I actually had an external drive fail and had to pay $1,000 to recover the drive but not everything was recovered. And a lot of the pictures/videos on that drive are corrupted (bad colours or pixels) luckily they were duplicated on the drive and some of the duplicates were recovered. This included my kids baby pics so had to pay! Learned my lesson and have multiple copies now