I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100GB != 10TB).
I've got about 10TB that I need to move from one raid to another, gigabit net, XFS file systems. My biggest concern is having the transfer die midway and not being able to resume easily. Speed would be nice, but ensuring transfer is much more important.
Normally I'd just tar & netcat, but the raid I'm moving from has been super flaky as of late and I need to be able to recover and resume if it drops mid process. Should I be looking at rsync?
EDIT: looking into this a bit more, i think rsync might be too slow, i'd like to avoid this taking 30+ days. So now i'm looking for suggestions on how to monitor / resume the transfer with netcat.
-
yep, rsync
outside oddball, the async features DRBD came out with recently.
GruffTech : +1 for Rsync. King of system-to-system transfers on linux.Marco Ramos : +1 rsync ftw...Chris S : +1, rsync (and it's Windows counterpart DeltaCopy) are the end all of efficient server to server generic file duplication.James Sneeringer : +1 for rsync, but I'd add that it may be faster to do it "locally" over an NFS mount instead of incurring the rsync or SSH protocol overhead.From cagenut -
You could try setting up an FTP server on the server with the data to be copied and use an FTP client with "resume" on the receiving end. I use Filezilla server and client and I use the "resume" feature of the client quite often and it has always worked without a hitch.
From Puddingfox -
Never underestimate the bandwidth of a station wagon full of tapes. 10TB would be feasible with relatively cheap consumer grade NAS equipment if you can divide it into (say 2TB) chunks. If this is a one-off then a semi-manual process might be workable, and a 2TB NAS is only a few hundred dollars.
If you need an ongoing process then you could set up RSYNC after you've done the initial transfer.
Ladadadada : The original question mentioned that this is already over a local network.ConcernedOfTunbridgeWells : Which appears to be too slow and/or unreliable for the OP's purposes.
0 comments:
Post a Comment