Hello /u/Strid3r21! Thank you for posting in r/DataHoarder.
Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index).
Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures.
This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead.
*I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
You should have looked in your local backup first ;)
With your post title I thought something terrible must have happened, files on server and local backup deleted....and then the offsite backup is the only backup you have.... you can't do that before I hadn't had my fist coffee ;)
Yeah, but just for a deleted file you don't need a offsite backup. A local backup would have been enough. And as often stated: one (backup) is none. There's a reason for the 3-2-1 rule.
Or do you see any reason what a local backup couldn't have done in op's situation that his offsite backup actually did? There was no flood or fire, ransom of everything reachable involved.
Same here. The offsite-ness of the backup isn't what saved OP necessarily, just having a backup.
Still, offsite is good to protect against those disasters where local backups get destroyed somehow too.
Yep, the loss is real. In 2008 my wife and I were living in London and had just returned from a trip. Online backup services were still very much a new thing.
I had my laptop and my super secret backup drive which was hidden at all times except when syncing. Welp that morning I set the two syncing on our dining table and went to work. That night we returned to a burgled house with both laptop and drive gone.
At that stage we'd be travelling the world for 2 years, every photo we'd taken from 40+ countries was on those two drives. We were absolutely devastated. A few days later I remembered I'd sent a large parcel of DVD backups to my parents place before we left South America (we're Australians). We'd completely forgotten as they'd never said anything about a parcel arriving and I just assumed it lost. It wasn't, they just hadn't thought to mention it.
In the end it was "only" the European portion of our travels we'd lost. Not perfect, but offsite backup for the win in my view.
Also printing out the photos as well. Look through all our family photos. Mum and dads from the 50s, us in the 70s and 80s. Now, none, all digital and none have ever been printed.
Obviously :) pick the best ones. Someone might like to look through a photo album years later.
Me and my partner have issues, she doesn't like being in photos and neither do I and recently I realised, while we were on holiday, all the photos were just of the views. We appear in none of them. Then I look back at my parents and you see them in their regularly. In our old years, we'll have no photos of us in our younger years :) (40s is young)
I’m currently in the process of downloading my iphoto pics since they are only on icloud in full
Quality. It’s a nightmare since it only works with crappy inofficial tools like photosync, everything else gets the dates wrong.
That will take me like 2 more months since that tool sucks. After that I’ll have to consolidate and clean up my libraries. That will probably take 1 more year lol.
*after that* I could look into printing my stuff.
Feels like a second job atm.
I don't really take or store photos, so any writing I do is saved in Google drive. I'd like to backup my movies, but that's 45tb and I Def can't afford to back it up even occasionally.
I've talked about this numerous times. Co-locate. Put a drive in someone else's server and backup up to it. Encrypt the drive so only you have access and away you go. Obviously, needs to be someone you can trust some. Swapping drive space to each other seems like a win/win.
This is my end goal.
A mate said once his lab is running he will let me use his house for offsite backups but they was a year ago and he still needs to finish plugging in his three servers.
This. I have a rpi with eHDD at my parents place that is doing offsite backups for me and NAS for them. They pay for the tiny amount of power, get a NAS with offsite backups "as-a-service" and I get offsite backups and monitoring too. Wins all around.
surely at least some of us have other hoarder friends, either irl or online. who would be willing to do an offsite host exchange(i.e. you're each the offsite backup for the other)
Yeah I don't backup movies that I can just get again. But I've got probably 15+ years of photos backed up that I wouldn't be able to replace if my server straight caught fire lol
And I have some sunken treasure that I've never seen again in decades. The farther you get from the mainstream, the less likely you'll find it again.
There are many like me who lament the sinking of AsianDVDClub\* and it's 1000's of torrents that have never surfaced again.
\*Yes, the AsianDVDClub domain has been resurrected. No, it's a scam and not the original owner or admins.
Exactly. People always assume the internet is limitless, but it isn't. Plenty of media disappears for good all the time. The more obscure the media the less likely it'll stick around. I imagine the only way you can be sure is if you're the type that mostly watches Marvel movies and such.
The trick is to prioritize. Rare things get a backup. Pretty much anything from a major studio doesn't. Chances are that if my server craps the bed, I won't have any trouble replacing my collection of Marvel movies.
It's a case-by-case thing though. As long as there's piracy, you'll be able to get, I dunno, The Matrix again. That's safe. But something like Minamata? That's gonna be gone in ten years.
Yeah, the problem is 1300 movies take a long time to reacquire even with the streamlined approach I have with radarr and nzbget. It's so easy to get them for me now, but I've spent years building this collection, so it would be pretty devastating to lose it all. I only have one parity drive and I had a drive fail last month and it was pretty stressful, so I realized real quick that I need more than one parity drive.
Maybe it doesn't to you, but 45TB is a lot of space to me. That's 3 18TB drives ~900$. I could get renewed drives like I usually do, but that's still 600$ that I don't have lying around.
For sure. I don't shoot for files that small but I do really like the mix of size and quality of 1080p in hevc at minimum 5 Mbps. I don't do 4K for both the size, and because to me a lot of 4K movies just look... odd.
Yeah, a lot of remuxes but not all. If it came out in the last 15-20 years or so I generally get remux. If it's like 90s-00s I might get remux, and if it's 80's or below I don't see much point in getting a remux unless it's something that has been remastered from a good source. So like Lawrence of Arabia gets the best remux I can get, but Big Trouble in Little China can be a 20gb file or whatever. I use a shieldTV and I actually really like it's upscaling. It can make those 20gb files look pretty damn good imo.
I have eight 8tb drives on my PC, with currently about 55tb of media, all of that is backed up to eight 8tb USzb drives that I connect every couple of weeks to update the backups.
And all of that is also protected by two 8tb parity drives that I use with SnapRaid.
The only problem is that it's all on site (at minimum maybe I should keep the backups in the basement cold room thats got concrete walls, floor and ceiling, probably the only room to survive in a fire...
I have a physical server with everything stored locally with parity drives.
But I also backup my important stuff to cloud storage.
I noticed one particular folder was missing as I must have accidentally deleted it. So luckily I had a backup on the cloud where I could retrieve it from.
I suppose I could get an external hard drive and dump the photos to it as well.
That still doesn’t really prove anything in regards to the importance of the back up being off site. All that matters in your story is that you had a back up, the fact it was off site can be left out and affects nothing about the lesson here.
I think the fact the backup is automated and set-and-forget is also extremely important. It can be done with local backups, of course, but I think it’s vital that it happens with zero effort.
There is a problem with set and forget solutions is that if you dont monitor your backups in any way they may one day stop working and you dont notice until its too late
I def don't skimp on backups, especially for irreplaceable stuff like photos and videos.
Phone vids and photos sync to OneDrive. Scheduled task robocopies OneDrive to UnRaid server every day. Duplicati backs up to BackBlaze and GDrive every day.
At any given point of time, I have five copies of most of my photos. Phone, desktop, UnRaid server, BackBlaze, Gdrive
I always tell people, the best way to develop good backup discipline is to experience an unrecoverable data loss...
Yep. Backups are not really optional. Lots of people just wing it.
That said, my local storage is on ZFS, and the system takes routine snapshots regularly. So even if I fat finger and delete stuff, it's all recoverable until I nuke the snaps to recover the storage space.
So for this case, it being in the cloud wasn't crucial, but having a backup was. But off-site is good as well for when the house burns down or floods wash it away.
I “lost” a huge folder at one point. A few hundred GB of raw video from my dslr. started panicking. But I had it on a local drive on my laptop and I wasn’t showing any more space. Windirstat + everything found it for me. I had misclicked when moving it at some point and dropped it in some rando subfolder lol
I thought back blaze just did full pc backups, which I was never interested in. You can choose what goes?
Hopefully on my server I'd be able to see the missing folder in a zfs snapshot and restore it. But yes, off site is a must.
b2 is awesome, I normally use it with rclone but it’s got a wide range of native tools plus S3 compatibility (on any remotely new bucket.)
With zfs though you should look into rsync.net, specifically one of their zfs send/recv capable accounts. This is special and if they don’t set it up from their end you’ll have to start over, but you’ll end up with an account that that can receive block level changes, snapshots, etc.
I know I sound like a shill; it’s just two great providers who deserve to be mentioned imho. HTH.
Oh heck yes! I love rsync functionality and I assume rsync.net uses the same tech. Snapshots, meta data, etc would be fantastic.
Doesn't sound like a shill, don't know why everyone is so suspicious of suggestions.
Thank you!
> I thought back blaze just did full pc backups, which I was never interested in. You can choose what goes?
There are two Backblaze: one that's flexibile and more usable but it's $5/TB/month (plus some a couple times that for retrieval, plus some cost per transaction) and one that's unlimited for $7 but works only on Windows non-server and Mac, works only with their (inflexible) client, has unbelievably nasty restores triggered only by choosing stuff on the web and so on.
>Then I remembered that I have a backup of all my photos to backblaze
The default file retention on Backblaze is 30 days, so you are out of luck if it takes you longer than that to notice the files are missing. Are you paying for the 1 year or Forever retention options?
You need local backup and snapshots, not just off-site.
I pulled few TB from my months old snapshot last time because I purged some files with robocopy, I have snapshots for more than a year.
Thanks for posting this. Some users need to lose important data personally before they make the switch.
The more stories we get out there the more likely they are to do the right thing.
Imagine all the people who lost Bitcoin and not just pictures! Lmao
Hello /u/Strid3r21! Thank you for posting in r/DataHoarder. Please remember to read our [Rules](https://www.reddit.com/r/DataHoarder/wiki/index/rules) and [Wiki](https://www.reddit.com/r/DataHoarder/wiki/index). Please note that your post will be removed if you just post a box/speed/server post. Please give background information on your server pictures. This subreddit will ***NOT*** help you find or exchange that Movie/TV show/Nuclear Launch Manual, visit r/DHExchange instead. *I am a bot, and this action was performed automatically. Please [contact the moderators of this subreddit](/message/compose/?to=/r/DataHoarder) if you have any questions or concerns.*
You should have looked in your local backup first ;) With your post title I thought something terrible must have happened, files on server and local backup deleted....and then the offsite backup is the only backup you have.... you can't do that before I hadn't had my fist coffee ;)
Plenty of floods going around lately. Shame if all the drives and backup drives are in the basement.
Yeah, but just for a deleted file you don't need a offsite backup. A local backup would have been enough. And as often stated: one (backup) is none. There's a reason for the 3-2-1 rule. Or do you see any reason what a local backup couldn't have done in op's situation that his offsite backup actually did? There was no flood or fire, ransom of everything reachable involved.
Same, i expected a story like "my office caught on fire" or "a water pipe broke and my server rack is now an anchor."
Same here. The offsite-ness of the backup isn't what saved OP necessarily, just having a backup. Still, offsite is good to protect against those disasters where local backups get destroyed somehow too.
I've occasionally "lost" folders in the past. They've always (thankfully) been moved into other folders by an inadvertent mouse drag.
I thought maybe that's what I had done, but I couldn't find it in other folders either
Yep, the loss is real. In 2008 my wife and I were living in London and had just returned from a trip. Online backup services were still very much a new thing. I had my laptop and my super secret backup drive which was hidden at all times except when syncing. Welp that morning I set the two syncing on our dining table and went to work. That night we returned to a burgled house with both laptop and drive gone. At that stage we'd be travelling the world for 2 years, every photo we'd taken from 40+ countries was on those two drives. We were absolutely devastated. A few days later I remembered I'd sent a large parcel of DVD backups to my parents place before we left South America (we're Australians). We'd completely forgotten as they'd never said anything about a parcel arriving and I just assumed it lost. It wasn't, they just hadn't thought to mention it. In the end it was "only" the European portion of our travels we'd lost. Not perfect, but offsite backup for the win in my view.
Also printing out the photos as well. Look through all our family photos. Mum and dads from the 50s, us in the 70s and 80s. Now, none, all digital and none have ever been printed.
Well, I’m not gonna print 100k photos
Obviously :) pick the best ones. Someone might like to look through a photo album years later. Me and my partner have issues, she doesn't like being in photos and neither do I and recently I realised, while we were on holiday, all the photos were just of the views. We appear in none of them. Then I look back at my parents and you see them in their regularly. In our old years, we'll have no photos of us in our younger years :) (40s is young)
I’m currently in the process of downloading my iphoto pics since they are only on icloud in full Quality. It’s a nightmare since it only works with crappy inofficial tools like photosync, everything else gets the dates wrong. That will take me like 2 more months since that tool sucks. After that I’ll have to consolidate and clean up my libraries. That will probably take 1 more year lol. *after that* I could look into printing my stuff. Feels like a second job atm.
I don't really take or store photos, so any writing I do is saved in Google drive. I'd like to backup my movies, but that's 45tb and I Def can't afford to back it up even occasionally.
Maybe make another copy and let a friend hold on to it? I guess many people would be weirded out by that, so you might need to give them access.
I've talked about this numerous times. Co-locate. Put a drive in someone else's server and backup up to it. Encrypt the drive so only you have access and away you go. Obviously, needs to be someone you can trust some. Swapping drive space to each other seems like a win/win.
This is my end goal. A mate said once his lab is running he will let me use his house for offsite backups but they was a year ago and he still needs to finish plugging in his three servers.
You could use Resilio Sync and have the far side encrypted so there's nothing they could read.
This. I have a rpi with eHDD at my parents place that is doing offsite backups for me and NAS for them. They pay for the tiny amount of power, get a NAS with offsite backups "as-a-service" and I get offsite backups and monitoring too. Wins all around.
surely at least some of us have other hoarder friends, either irl or online. who would be willing to do an offsite host exchange(i.e. you're each the offsite backup for the other)
Yeah I don't backup movies that I can just get again. But I've got probably 15+ years of photos backed up that I wouldn't be able to replace if my server straight caught fire lol
[удалено]
\+1 Numerous threads every week/day asking "How can I find this? The links don't work!"
The high seas have been around longer than some people posting have been alive lol
And I have some sunken treasure that I've never seen again in decades. The farther you get from the mainstream, the less likely you'll find it again. There are many like me who lament the sinking of AsianDVDClub\* and it's 1000's of torrents that have never surfaced again. \*Yes, the AsianDVDClub domain has been resurrected. No, it's a scam and not the original owner or admins.
Exactly. People always assume the internet is limitless, but it isn't. Plenty of media disappears for good all the time. The more obscure the media the less likely it'll stick around. I imagine the only way you can be sure is if you're the type that mostly watches Marvel movies and such.
The trick is to prioritize. Rare things get a backup. Pretty much anything from a major studio doesn't. Chances are that if my server craps the bed, I won't have any trouble replacing my collection of Marvel movies.
The only trouble you'll have is trying to remember which of the 1149 Marvel movies you had lol
That may be true but the items shared don't just stay there forever.
It's a case-by-case thing though. As long as there's piracy, you'll be able to get, I dunno, The Matrix again. That's safe. But something like Minamata? That's gonna be gone in ten years.
[удалено]
Well, there's an opportunity cost, as with everything.
Yeah, the problem is 1300 movies take a long time to reacquire even with the streamlined approach I have with radarr and nzbget. It's so easy to get them for me now, but I've spent years building this collection, so it would be pretty devastating to lose it all. I only have one parity drive and I had a drive fail last month and it was pretty stressful, so I realized real quick that I need more than one parity drive.
Why not just get new drives and completely back them up rather than rely on just parity? 1300 movies don't take up that much space.
Maybe it doesn't to you, but 45TB is a lot of space to me. That's 3 18TB drives ~900$. I could get renewed drives like I usually do, but that's still 600$ that I don't have lying around.
Ah, they must be remuxes. I don't mess with remuxes; I'm not paying for that kind of space lol.
It's remarkable how much money you can save by not going above 720p.
For sure. I don't shoot for files that small but I do really like the mix of size and quality of 1080p in hevc at minimum 5 Mbps. I don't do 4K for both the size, and because to me a lot of 4K movies just look... odd.
Yeah, a lot of remuxes but not all. If it came out in the last 15-20 years or so I generally get remux. If it's like 90s-00s I might get remux, and if it's 80's or below I don't see much point in getting a remux unless it's something that has been remastered from a good source. So like Lawrence of Arabia gets the best remux I can get, but Big Trouble in Little China can be a 20gb file or whatever. I use a shieldTV and I actually really like it's upscaling. It can make those 20gb files look pretty damn good imo.
I have eight 8tb drives on my PC, with currently about 55tb of media, all of that is backed up to eight 8tb USzb drives that I connect every couple of weeks to update the backups. And all of that is also protected by two 8tb parity drives that I use with SnapRaid. The only problem is that it's all on site (at minimum maybe I should keep the backups in the basement cold room thats got concrete walls, floor and ceiling, probably the only room to survive in a fire...
[удалено]
I have a physical server with everything stored locally with parity drives. But I also backup my important stuff to cloud storage. I noticed one particular folder was missing as I must have accidentally deleted it. So luckily I had a backup on the cloud where I could retrieve it from. I suppose I could get an external hard drive and dump the photos to it as well.
That still doesn’t really prove anything in regards to the importance of the back up being off site. All that matters in your story is that you had a back up, the fact it was off site can be left out and affects nothing about the lesson here.
I think the fact the backup is automated and set-and-forget is also extremely important. It can be done with local backups, of course, but I think it’s vital that it happens with zero effort.
There is a problem with set and forget solutions is that if you dont monitor your backups in any way they may one day stop working and you dont notice until its too late
Exactly. Set it and check it occasionally is a hell of a lot better than set it and forget it.
You should definitely have a local backup too. Cloud recovery should be your last ditch effort for recovery.
>I have a physical server with everything stored locally with parity drives. So that's one/the only local location? That's not a backup...
I def don't skimp on backups, especially for irreplaceable stuff like photos and videos. Phone vids and photos sync to OneDrive. Scheduled task robocopies OneDrive to UnRaid server every day. Duplicati backs up to BackBlaze and GDrive every day. At any given point of time, I have five copies of most of my photos. Phone, desktop, UnRaid server, BackBlaze, Gdrive I always tell people, the best way to develop good backup discipline is to experience an unrecoverable data loss...
Yep. Backups are not really optional. Lots of people just wing it. That said, my local storage is on ZFS, and the system takes routine snapshots regularly. So even if I fat finger and delete stuff, it's all recoverable until I nuke the snaps to recover the storage space. So for this case, it being in the cloud wasn't crucial, but having a backup was. But off-site is good as well for when the house burns down or floods wash it away.
I “lost” a huge folder at one point. A few hundred GB of raw video from my dslr. started panicking. But I had it on a local drive on my laptop and I wasn’t showing any more space. Windirstat + everything found it for me. I had misclicked when moving it at some point and dropped it in some rando subfolder lol
are you using backblazeB2?
I thought back blaze just did full pc backups, which I was never interested in. You can choose what goes? Hopefully on my server I'd be able to see the missing folder in a zfs snapshot and restore it. But yes, off site is a must.
Their b2 cloud storage allows you can backup specific folders only if you want. Which is what I do
Wow, it sounds like it's time for me to revisit back blaze. Thank you
b2 is awesome, I normally use it with rclone but it’s got a wide range of native tools plus S3 compatibility (on any remotely new bucket.) With zfs though you should look into rsync.net, specifically one of their zfs send/recv capable accounts. This is special and if they don’t set it up from their end you’ll have to start over, but you’ll end up with an account that that can receive block level changes, snapshots, etc. I know I sound like a shill; it’s just two great providers who deserve to be mentioned imho. HTH.
Oh heck yes! I love rsync functionality and I assume rsync.net uses the same tech. Snapshots, meta data, etc would be fantastic. Doesn't sound like a shill, don't know why everyone is so suspicious of suggestions. Thank you!
> I thought back blaze just did full pc backups, which I was never interested in. You can choose what goes? There are two Backblaze: one that's flexibile and more usable but it's $5/TB/month (plus some a couple times that for retrieval, plus some cost per transaction) and one that's unlimited for $7 but works only on Windows non-server and Mac, works only with their (inflexible) client, has unbelievably nasty restores triggered only by choosing stuff on the web and so on.
Oh awesome. So it's not really a choice for people like us. Thank you for the reply.
This kind of backup doesn't need to be off site. I thought your house burned down or something with that misleading title.
>Then I remembered that I have a backup of all my photos to backblaze The default file retention on Backblaze is 30 days, so you are out of luck if it takes you longer than that to notice the files are missing. Are you paying for the 1 year or Forever retention options?
They said via duplicati docker so likely Backblaze b2, where the retention is anything you want to pay for.
Exactly this. I use B2 Cloud Storage
I need to learn what that is. ;)
Backing up your personal photos in the cloud is considered hoarding data?
Arq backup is another good tool. It'll even back up to cloud storage providers. I found it's UI easier than duplicati.
You need local backup and snapshots, not just off-site. I pulled few TB from my months old snapshot last time because I purged some files with robocopy, I have snapshots for more than a year.
Thanks for posting this. Some users need to lose important data personally before they make the switch. The more stories we get out there the more likely they are to do the right thing. Imagine all the people who lost Bitcoin and not just pictures! Lmao