Computers & Electronics

best way to consolidate multiple back up and duplicates in a single place.

  • Last Updated:
  • Jan 21st, 2021 8:38 am
[OP]
Newbie
Dec 1, 2012
69 posts
3 upvotes
MILTON

best way to consolidate multiple back up and duplicates in a single place.

Hi,

I am trying to back up my data into 1 or 2 place. ( just got the synology DS 420+ and 2 X 14 TB on recent sale )

I have been backing up all the data for years and each time into different place. from Floppy to CD to DVD to external drives.
Now I have a lot of external drive with different back ups which would also be a lots of duplicates.

What I want to know are idea of how to consolidate all the data into 1 place and remove the duplicate.

I don't think I can go through all the individual file to see if they are duplicate to remove them. I'm sure there are a lot of better way to do this.

Thank you all in advance.
7 replies
Deal Addict
Sep 12, 2007
2831 posts
974 upvotes
Dump everything on your NAS and run a dedupe app. Many free ones, just hope it. Apparently CCleaner can do this too https://www.laptopmag.com/articles/remo ... windows-10

Get something which will recurse folders.

I also suggest, aside from your NAS which I will assume has some kind or fault tolerance, raid1, raid5, raid6....get an external drive, once you dedupe, do a full back up to it, then a weekly/monthly resync, then throw it in a safe. Also get a safe, it's less than $100.

Think fire, or ransomware.
Jr. Member
User avatar
Sep 30, 2020
143 posts
145 upvotes
Toronto
I'll just add on to @vodka 's point that if you use a dedupe app, make sure you select one that does CRC-based deduplicating - where it creates the unique checksum of the file. Some de-dupe program in "fast" mode, the really "dumb" (lame) ones, or geared towards more basic users don't do this CRC-based check which is the way of getting a true unique signature for the file's contents - they might only check exact name (which doesn't help multiple names for the same file, leaving extra duplicates) and not "contents", or size (and possibly extension with size). You'd be surprised, as you increase the number of files to check, how many actually share the same size (including pictures take one after the other with only subtle differences) and you don't want to either skip legitimate duplicates or toast perceived duplicates that really aren't. Some attempt to speed things up by doing a CRC sampling at various parts of the file which is better than the other "dumb" (name, size) methods but not true CRC.

https://www.duplicatefilter.com/ supports CRC checking and I use it currently. I used to like "NoClone" a lot but it looks like it's dead/not supported. The better ones let you select a default directory to remove any duplicates as the priority "delete from here first" by "smart" marking them and ensuring to keep at least one of the original.
[OP]
Newbie
Dec 1, 2012
69 posts
3 upvotes
MILTON
vodka wrote: Dump everything on your NAS and run a dedupe app. Many free ones, just hope it. Apparently CCleaner can do this too https://www.laptopmag.com/articles/remo ... windows-10

Get something which will recurse folders.

I also suggest, aside from your NAS which I will assume has some kind or fault tolerance, raid1, raid5, raid6....get an external drive, once you dedupe, do a full back up to it, then a weekly/monthly resync, then throw it in a safe. Also get a safe, it's less than $100.

Think fire, or ransomware.
When you say safe are you talking physical safe or is it a software ?
Deal Addict
Sep 12, 2007
2831 posts
974 upvotes
fishing4one wrote: When you say safe are you talking physical safe or is it a software ?
Physical. You should have one for the most important stuff, passports, etc. I have 2 HDs, I rotate them 1/month, one of them is always in the safe.
[OP]
Newbie
Dec 1, 2012
69 posts
3 upvotes
MILTON
Thank Vodka and Casualcauli,
Was reading for the last few days.

Ended making 3 copies of all my data in 3 external drive.
will run CCleaner, Duplicatefilter, and AllDup on them. will then throw them in to a 4th Drive together and run one of the pogram.

The final copy should be as clean as possible without loosing any data.
[OP]
Newbie
Dec 1, 2012
69 posts
3 upvotes
MILTON
By the way the most important thing I learned is the word Deduplicate. That is what send me down the right search path.
Member
Nov 26, 2003
380 posts
148 upvotes
Ottawa
I'm in the exact same boat. Got 5 different external HD and 2 laptops with lots of duplicates. I just bought a QNAP and put everything on it. Will have a copy before cleaning and after cleaning.
I plan to set up q sync and q filing afterwards to limit the duplicates we will see if that actually works.
At one point in time I actually had an external drive fail and had to pay $1,000 to recover the drive but not everything was recovered. And a lot of the pictures/videos on that drive are corrupted (bad colours or pixels) luckily they were duplicated on the drive and some of the duplicates were recovered. This included my kids baby pics so had to pay! Learned my lesson and have multiple copies now

Top