I just bought a 8TB external drive on which I want to backup the
contents of my 1, 2 and 4TB hard disks. However copying all that stuff
turns out to be extremely slow, and it gets even slower the further the
transfer progresses.

At first I thought it was because I still have USB 2.0 ports, so I tried
it on a laptop with two USB 3.0 ports. This was even worse (although the
laptop has Windows 10, maybe that has something to do with it) and the
backup would have taken forever if I hadn't stopped it.

As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...

Also tried using the rsync command on another folder, but that was
equally slow.

Is there any way to *really* speed up the transfer of large chunks of data?

Appreciate any info.

NP2 wrote:
> I just bought a 8TB external drive on which I want to backup the
> contents of my 1, 2 and 4TB hard disks. However copying all that stuff
> turns out to be extremely slow, and it gets even slower the further the
> transfer progresses.
>
> At first I thought it was because I still have USB 2.0 ports, so I tried
> it on a laptop with two USB 3.0 ports. This was even worse (although the
> laptop has Windows 10, maybe that has something to do with it) and the
> backup would have taken forever if I hadn't stopped it.
>
> As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...
>
> Also tried using the rsync command on another folder, but that was
> equally slow.
>
> Is there any way to *really* speed up the transfer of large chunks of data?
>
> Appreciate any info.

When I was reading 'stuff' about btrfs to learn something about it,
comments compared how fast btrfs did its snapshot thing compared to rsync.

But I haven't learned enough yet to be making suggestions.

--
Mike Easter

On Sat, 25 Dec 2021 20:33:19 +0100
NP2 <NP2@somewhere.invalid> wrote:

> I just bought a 8TB external drive on which I want to backup the
> contents of my 1, 2 and 4TB hard disks. However copying all that stuff
> turns out to be extremely slow, and it gets even slower the further the
> transfer progresses.
>
> At first I thought it was because I still have USB 2.0 ports, so I tried
> it on a laptop with two USB 3.0 ports. This was even worse (although the
> laptop has Windows 10, maybe that has something to do with it) and the
> backup would have taken forever if I hadn't stopped it.
>
> As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...
>
> Also tried using the rsync command on another folder, but that was
> equally slow.
>
> Is there any way to *really* speed up the transfer of large chunks of data?
>
> Appreciate any info.

If I am copying drives or partions, I use dd. I mainly have 2TB hard drives and use a line like this:

dd if=/dev/sda1 of=/dev/sdb1 bs=16384 status=progress

The status=progress gives the copy measure every 30 seconds.

If I am doing an incremental backup to another machine on the LAN, I use this:

rsync -rlptD --exclude-from=/home/me/Scripts/exclude.txt --chown=me:me --progress /home/me/ /spare/backup/me/

This slow initially but speeds up once a complete copy has been backed up. The benefit of rsync.

It is probably not much help. Best of luck.

Alan

--
Mint 20.2, kernel 5.4.0-88-generic, Cinnamon 5.0.5
running on an AMD Phenom II X4 Black edition processor with 16GB of DRAM.

On 12/25/2021 2:33 PM, NP2 wrote:
> I just bought a 8TB external drive on which I want to backup the contents of my 1, 2 and 4TB hard disks. However copying all that stuff turns out to be extremely slow, and it gets even slower the further the transfer progresses.
>
> At first I thought it was because I still have USB 2.0 ports, so I tried it on a laptop with two USB 3.0 ports. This was even worse (although the laptop has Windows 10, maybe that has something to do with it) and the backup would have taken forever if I hadn't stopped it.
>
> As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...
>
> Also tried using the rsync command on another folder, but that was equally slow.
>
> Is there any way to *really* speed up the transfer of large chunks of data?
>
> Appreciate any info.

Partclone.

https://linuxconfig.org/how-to-use-partclone-to-create-a-smart-partition-backup

Drag and drop the partitions in Gparted ?

GParted doesn't have to use "dd" for everything it does.
There are partclone and ntfsclone and other clone utilities for the job.
And they can do "intelligent copy" and copy only occupied parts of the
disk, in disk order. This reduces head movement and speeds up the transfer.

+------+----------+
| MBR | 1TB |
+------+----------+

+------+------------+
| MBR | 2TB |
+------+------------+

+------+--------------+
| MBR | 4TB |
+------+--------------+

+------+----------+------------+--------------+---------+ Transfer is fast, however
| MBR | 1TB | 2TB | 4TB | Unalloc | representation is not efficient
+------+----------+------------+--------------+---------+ and will have to be "merged" at
some point.

Try dragging and dropping in Gparted and see how it goes.

You may need to list the BLKIDs later and see whether the
results are workable or not (check for duplicated values).

Backups can be stored as "images" instead of doing clones
like I show above (the Linuxconfig article shows commands for it).
For example, Macrium Reflect Free can make smart backups of EXT4.
I did that a few minutes ago with a 1TB drive having SWAP and three EXT4 on it.

Live is full of tradeoffs.

Paul

On 12/25/21 4:32 PM, this is what pinnerite wrote:
> On Sat, 25 Dec 2021 20:33:19 +0100
> NP2 <NP2@somewhere.invalid> wrote:
>
>> I just bought a 8TB external drive on which I want to backup the
>> contents of my 1, 2 and 4TB hard disks. However copying all that stuff
>> turns out to be extremely slow, and it gets even slower the further the
>> transfer progresses.
>>
>> At first I thought it was because I still have USB 2.0 ports, so I tried
>> it on a laptop with two USB 3.0 ports. This was even worse (although the
>> laptop has Windows 10, maybe that has something to do with it) and the
>> backup would have taken forever if I hadn't stopped it.
>>
>> As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...
>>
>> Also tried using the rsync command on another folder, but that was
>> equally slow.
>>
>> Is there any way to *really* speed up the transfer of large chunks of data?
>>
>> Appreciate any info.
>
> If I am copying drives or partions, I use dd. I mainly have 2TB hard drives and use a line like this:
> dd if=/dev/sda1 of=/dev/sdb1 bs=16384 status=progress
> The status=progress gives the copy measure every 30 seconds.
> If I am doing an incremental backup to another machine on the LAN, I use this:
> rsync -rlptD --exclude-from=/home/me/Scripts/exclude.txt --chown=me:me --progress /home/me/ /spare/backup/me/
> This slow initially but speeds up once a complete copy has been backed up. The benefit of rsync.
> It is probably not much help. Best of luck.
> Alan
>
I too like rsync for linux. Robocopy for Windows. Both have the same feature that pinnerite mention, slow for the initial but fast after
that since you are only updating the changes.

I see pinnerite's command and mine is different.
rsync -av --delete-before --exclude='lost+found' "${SRCE}" "${DEST}"
Not sure if I have simplified mine or not but mine simply updates the destination to mirror the source. It does what I want currently, I'll
have to check the switches pinnerite has and see if I'm missing something good.

--
Linux Mint Cinnamon 20.2 64bit, Dell Inspiron 5570 laptop
Quad Core i7-8550U, 16G Memory, 512G SSD, 750G & 1TB HDDs

On 2021-12-25, Mike Easter <MikeE@ster.invalid> wrote:
> NP2 wrote:
>> I just bought a 8TB external drive on which I want to backup the
>> contents of my 1, 2 and 4TB hard disks. However copying all that stuff
>> turns out to be extremely slow, and it gets even slower the further the
>> transfer progresses.
>>
>> At first I thought it was because I still have USB 2.0 ports, so I tried
>> it on a laptop with two USB 3.0 ports. This was even worse (although the
>> laptop has Windows 10, maybe that has something to do with it) and the
>> backup would have taken forever if I hadn't stopped it.
>>
>> As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...
>>
>> Also tried using the rsync command on another folder, but that was
>> equally slow.
>>
>> Is there any way to *really* speed up the transfer of large chunks of data?
>>
>> Appreciate any info.
>
> When I was reading 'stuff' about btrfs to learn something about it,
> comments compared how fast btrfs did its snapshot thing compared to rsync.

btrfs snapshots, like zfs ones are fast. As in a couple of seconds vs couple
of minutes.

Timeshift snapshots under btrfs are so fast you wonder if it has worked.

On 2021-12-25, NP2 <NP2@somewhere.invalid> wrote:

Some thoughts

> I just bought a 8TB external drive on which I want to backup the
> contents of my 1, 2 and 4TB hard disks. However copying all that stuff
> turns out to be extremely slow, and it gets even slower the further the
> transfer progresses.

Trust you are aware of CMS and SMR hard drives. Drives can be made cheaper
if the read/write strips/area overlapped. This makes no difference to the
reading but really slows down the writing as the disk fills up as the data
which is in the way needs to be shifted and then rewritten back after the
new data is written. These are the so called Shingled drives (SMR)

https://blocksandfiles.com/2020/04/15/seagate-2-4-and-8tb-barracuda-and-desktop-hdd-smr/
>
> At first I thought it was because I still have USB 2.0 ports, so I tried
> it on a laptop with two USB 3.0 ports. This was even worse (although the
> laptop has Windows 10, maybe that has something to do with it) and the
> backup would have taken forever if I hadn't stopped it.
>
> As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...

You should be getting in the order of 100-120 GB/hr (Which 8+ hours per TB)

I have found that small files take longer, as in say 1000 1MB files will
take longer than a single 1GB file.

>
> Also tried using the rsync command on another folder, but that was
> equally slow.
>
> Is there any way to *really* speed up the transfer of large chunks of data?
>
> Appreciate any info.

If you just want to do the transfer once, then it probably is worth the
onerhead to place the drive in the PC and connect via the SATA interface.

Let it run over night also "speeds" things up.

On 26-12-2021 04:38, Gordon wrote:
> On 2021-12-25, NP2 <NP2@somewhere.invalid> wrote:
>
> Some thoughts
>
>> I just bought a 8TB external drive on which I want to backup the
>> contents of my 1, 2 and 4TB hard disks. However copying all that stuff
>> turns out to be extremely slow, and it gets even slower the further the
>> transfer progresses.
>
> Trust you are aware of CMS and SMR hard drives. Drives can be made cheaper
> if the read/write strips/area overlapped. This makes no difference to the
> reading but really slows down the writing as the disk fills up as the data
> which is in the way needs to be shifted and then rewritten back after the
> new data is written. These are the so called Shingled drives (SMR)
>
> https://blocksandfiles.com/2020/04/15/seagate-2-4-and-8tb-barracuda-and-desktop-hdd-smr/
>>
>> At first I thought it was because I still have USB 2.0 ports, so I tried
>> it on a laptop with two USB 3.0 ports. This was even worse (although the
>> laptop has Windows 10, maybe that has something to do with it) and the
>> backup would have taken forever if I hadn't stopped it.
>>
>> As I write this, I am copying a 65GB folder: 3 hours 40 minutes left...
>
> You should be getting in the order of 100-120 GB/hr (Which 8+ hours per TB)
>
> I have found that small files take longer, as in say 1000 1MB files will
> take longer than a single 1GB file.
>
>>
>> Also tried using the rsync command on another folder, but that was
>> equally slow.
>>
>> Is there any way to *really* speed up the transfer of large chunks of data?
>>
>> Appreciate any info.
>
> If you just want to do the transfer once, then it probably is worth the
> onerhead to place the drive in the PC and connect via the SATA interface.
>
> Let it run over night also "speeds" things up.

You might want to look at
https://groups.google.com/g/alt.os.linux/c/WYFTO6pL0us
which discusses backups with rsync in depth.

NP2 <NP2@somewhere.invalid> writes:
> Also tried using the rsync command on another folder, but that was
> equally slow.

The thing about all of these incremental backups like 'rsync' is the *very
first time* it is forced to do a COMPLETE backup AND THAT IS SLOW. Even
hours long. When I first looked at trying it, I kept aborting rsync because
it was taking too long. I thought it would take hours every day.

But ... Grit your teeth and take it. It's only once after all.

From there on in, you are just looking at file changes. I do several 4TB
backups of my archives on a daily basis, they take less than 10 minutes on
average, and that's with fairly slow external USB drives.

My backups are probably slower than some people's because I tend to script
the rsync'ing of specific directories rather than an overall filesystem.
Just my personal idiosyncrasy for more specific logging.
> Is there any way to *really* speed up the transfer of large chunks of data?
Fast devices are the only solution to that problem. The only other
alternative is to reduce the amount of data you transfer.

When it comes to backups, it really doesn't matter how long they take, so
long as you get one backup completed before the next one is scheduled. Of
course it's pointless backing up something if you've already got a backup
of it on your backup medium. Hence the widespread use of rsync and similar
tools.

Automate, set and forget. But verify it by looking at your logs.

Regards,

Jack
--
"My mother says I don't know what good, clean fun is.
She's right. I don't know what good it is."

- Laugh-In, 1968

On 2021-12-27, Jack Strangio <jackstrangio@yahoo.com> wrote:
> NP2 <NP2@somewhere.invalid> writes:
>> Also tried using the rsync command on another folder, but that was
>> equally slow.
>
> The thing about all of these incremental backups like 'rsync' is the *very
> first time* it is forced to do a COMPLETE backup AND THAT IS SLOW. Even
> hours long. When I first looked at trying it, I kept aborting rsync because
> it was taking too long. I thought it would take hours every day.
>
> But ... Grit your teeth and take it. It's only once after all.
>
> From there on in, you are just looking at file changes. I do several 4TB
> backups of my archives on a daily basis, they take less than 10 minutes on
> average, and that's with fairly slow external USB drives.
>
> My backups are probably slower than some people's because I tend to script
> the rsync'ing of specific directories rather than an overall filesystem.
> Just my personal idiosyncrasy for more specific logging.
>
>> Is there any way to *really* speed up the transfer of large chunks of data?
>
> Fast devices are the only solution to that problem. The only other
> alternative is to reduce the amount of data you transfer.
>
> When it comes to backups, it really doesn't matter how long they take, so
> long as you get one backup completed before the next one is scheduled. Of
> course it's pointless backing up something if you've already got a backup
> of it on your backup medium. Hence the widespread use of rsync and similar
> tools.
>
> Automate, set and forget. But verify it by looking at your logs.
>
Better yet include some random restores.

On 27-12-2021 08:56, Gordon wrote:
> On 2021-12-27, Jack Strangio <jackstrangio@yahoo.com> wrote:
>> NP2 <NP2@somewhere.invalid> writes:
>>> Also tried using the rsync command on another folder, but that was
>>> equally slow.
>>
>> The thing about all of these incremental backups like 'rsync' is the *very
>> first time* it is forced to do a COMPLETE backup AND THAT IS SLOW. Even
>> hours long. When I first looked at trying it, I kept aborting rsync because
>> it was taking too long. I thought it would take hours every day.
>>
>> But ... Grit your teeth and take it. It's only once after all.
>>
>> From there on in, you are just looking at file changes. I do several 4TB
>> backups of my archives on a daily basis, they take less than 10 minutes on
>> average, and that's with fairly slow external USB drives.
>>
>> My backups are probably slower than some people's because I tend to script
>> the rsync'ing of specific directories rather than an overall filesystem.
>> Just my personal idiosyncrasy for more specific logging.
>>
>>> Is there any way to *really* speed up the transfer of large chunks of data?
>>
>> Fast devices are the only solution to that problem. The only other
>> alternative is to reduce the amount of data you transfer.
>>
>> When it comes to backups, it really doesn't matter how long they take, so
>> long as you get one backup completed before the next one is scheduled. Of
>> course it's pointless backing up something if you've already got a backup
>> of it on your backup medium. Hence the widespread use of rsync and similar
>> tools.
>>
>> Automate, set and forget. But verify it by looking at your logs.
>>
> Better yet include some random restores.
>

....possibly mail a summary to yourself.

Thanks all for the input, I've learned a few things.

I tried cloning with gparted, but frankly thas was just as slow. Four
hours after starting cloning my 2Tb drive, it was at about 20%...

Seems like I'll have to learn to be a bit more patient. I do not like to
leave my computer on overnight, so I guess I'll just do a few folders at
a time every day. I'll get there eventually...

NP2 <NP2@somewhere.invalid> writes:
>
> I do not like to leave my computer on overnight

I remember the first time I did that, I was quite twitchy about doing it. I
got up several times during the night to check that everything was OK.
Then I went to the other extreme, and I went on a European holiday for six
weeks or so without a problem, followed by an attempt at getting a huge uptime
that was thwarted at the six-month mark by a Solaris update. These days I'm
more relaxed either way about uptimes, but leaving the system on overnight
is when most of my cronjobs get done.

[ has a look at current uptime]

12:44:01 up 19 days, 23:36, 5 users, load average: 2.17, 2.12, 2.07

Yep that seems about normal, I usually end up doing a shutdown every month
or so for some unrememberable reason or other.

Regards,

Jack
--
Why do meteorites always land in craters?