SoylentNews is people
SoylentNews
This is probably one of those topics that gets regurgitated periodically, but it's always good to get some fresh answers.
The small consultancy business I work for wants to set up a new file server with remote backup. In the past we have used a Windows XP file server and plugged in a couple of external USB drives when space runs out. Backups were performed nightly to a USB drive and taken offsite to a trusted employees home.
They are looking to Linux for a new file server (I think more because they found out how much a new Windows file server would be).
I'm not a server guy but I have set up a simple Debian-based web server at work for a specific intranet application, but when I was asked about ideas for the new system the best I could come up with was maybe ssh+rsync (which I have only recently started using myself so I'm no expert by any means). Using Amazon's cloud service has been suggested, as well as the remote being a dedicated machine at a trusted employee's home (probably with a new dedicated line in) or with our local ISP (if they can offer such a service). A new dedicated line out of the office has also been suggested, I think mainly because daily file changes can potentially be quite large (3D CAD models etc). A possible advantage of the remote being nearby is that the initial backup could be using a portable hard drive instead of having to uploading terabytes of data (I guess there is always courier services though).
Anyway, just thought I'd chuck it out there. A lot of you guys probably already set up and/or look after remote backup systems. Even if anyone just has some ideas regarding potential traps/pitfalls would be handy. The company is fairly small (about 20-odd employees) so I don't think they need anything overly elaborate, but all feedback is appreciated.
(Score: 2) by AudioGuy on Friday July 11 2014, @05:11AM
I did look it up, (it has some problems with sql data, etc.) and was not unaware of the de-duplication algorithms existence.
I picked a poor example on how incrementals can be fooled. The real point was meant to be simply that large amounts of data can change in ways where you would not expect this to be the case. I think the color change mentioned below would have been a better choice.
It is possible my experience is slightly skewed by many of the businesses I deal with being involved in the arts.
However, the original poster did specifically mention 'because daily file changes can potentially be quite large (3D CAD models etc)'. To me that means 'many gigabytes of data every day' - NEW data. What would be more useful is if he were to mention what the typical amounts actually were.
If the 3D Cad he is talking about is the kind used for say, video/movie production, just a simple, slight color change will rewrite the whole file, pretty much every byte of the rgb data, and that file could easily be 20-100 GB. He hasn't said, so I don't know.
But even other companies surprise me - they have huge print files, they are generating simple video, editing it, color correcting, etc.
It adds up, and while deduplication sure looks like a useful tool I have my doubts it is -enough- to compensate for the woefully inadequate internet speeds many of us have to deal with. Maybe in Finland it is enough. :-)
I don't understand the comment about compression (sorry, replying to two different posters at once, probably I shouldn't), I only mentioned that I do not compress the files on disk on the local copy. This just makes it simpler and faster for others to find files in the archive. If I were transferring general files over the net I would certainly want it. It doesn't help a whole lot on the case of already compressed files like jpegs and much video. I said nothing about stressing the processor.
Most of the small businesses I work with have several terabytes of data to back up, so that initial backup would take quite some time. You can dismiss that, but I can't. :-)