Hello, I was never very good at linux scripting, and now I've forgotten
much what I did know. I hope somebody here can help me incorporate gzip
with the dd copy lines I use for backing up laptop to an external drive.
The two different thumb drives I use to boot laptop and copy SDD are
1) Ubuntu Installer and 2) Knoppix. Both of those come with gzip.
I've never actually ever used gzip.

Here's the lines I use to copy the SDD contents to an external drive:

## first do fdisk -l to sort out the devices
sudo mkdir -p /mnt/sdb1 (creating a mount point)
sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
sudo mkdir -p /mnt/sdb1/images (if u don't already have that directory)
sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096
conv=notrunc,noerror

Here's the lines to restore the SDD if I ever need to:

sudo mkdir -p /mnt/sdb1 (creating a mount point)
sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
sudo dd if=/mnt/sdb1/images/asyymmdd.img of=/dev/nvme0n1 bs=4096
conv=notrunc,noerrer

Thanks in advance.
Bill S.

Am 02.11.2023 um 00:52:24 Uhr schrieb bilsch01:

> 1) Ubuntu Installer and 2) Knoppix. Both of those come with gzip.
> I've never actually ever used gzip.

dd doesn't support compression, it simply copies byte for byte of input
to the output.
You can later compress that with gzip <filename>.

On 2023-11-02, bilsch01 wrote:
> Hello, I was never very good at linux scripting, and now I've forgotten
> much what I did know. I hope somebody here can help me incorporate gzip
> with the dd copy lines I use for backing up laptop to an external drive.
> The two different thumb drives I use to boot laptop and copy SDD are
> 1) Ubuntu Installer and 2) Knoppix. Both of those come with gzip.
> I've never actually ever used gzip.
>
> Here's the lines I use to copy the SDD contents to an external drive:
>
> ## first do fdisk -l to sort out the devices
> sudo mkdir -p /mnt/sdb1 (creating a mount point)
> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
> sudo mkdir -p /mnt/sdb1/images (if u don't already have that directory)
> sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096
> conv=notrunc,noerror

You'd need something to the effect of:
dd if=/the/device (NO if=) [...] | gzip > /mnt/sdb1/images/xx.img.gz

restore would be:

gunzip -c xx.img.gz | dd (NO if=) of=/dev/whatever [...]

But that's just from a quick spin on the corresponding manpages for the
commands themselves. I'd test it on something small (e.g. a USB stick
image) before doing it on the main drive.

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On 11/2/2023 3:52 AM, bilsch01 wrote:
> Hello, I was never very good at linux scripting, and now I've forgotten much what I did know. I hope somebody here can help me incorporate gzip with the dd copy lines I use for backing up laptop to an external drive. The two different thumb drives I use to boot laptop and copy SDD are
> 1) Ubuntu Installer and 2) Knoppix.  Both of those come with gzip.
> I've never actually ever used gzip.
>
> Here's the lines I use to copy the SDD contents to an external drive:
>
> ## first do fdisk -l to sort out the devices
> sudo mkdir -p /mnt/sdb1  (creating a mount point)
> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
> sudo mkdir -p /mnt/sdb1/images (if u don't already have that directory)
> sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096 conv=notrunc,noerror
>
> Here's the lines to restore the SDD if I ever need to:
>
> sudo mkdir -p /mnt/sdb1  (creating a mount point)
> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
> sudo dd if=/mnt/sdb1/images/asyymmdd.img of=/dev/nvme0n1 bs=4096 conv=notrunc,noerrer
>
> Thanks in advance.
> Bill S.
>

sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096 conv=notrunc,noerror

becomes

sudo dd if=/dev/nvme0n1 bs=4096 conv=notrunc,noerror | gzip -3 > /mnt/sdb1/images/asyymmdd.img.gz

The dd program, when you remove the "of" specification, uses STDOUT.

GZIP accepts piped input. There are options of --fast and --slow which
correspond to -1 and -9. The value -3 for compression is the default,
and is shown as an example of a passed parameter. You might want to
test the output, and see if the size changes or not

... | gzip -1 | wc -c # count the bytes of output

... | gzip -9 | wc -c # count the bytes of output

With gzip, you have to be careful of the syntax,
so the original file is not deleted. That's what I spent
most of my time playing with, is checking for deletion.

*******

Later:

gzip -c -d /mnt/sdb1/images/asyymmdd.img.gz | sudo dd of=/dev/nvme0n1 bs=4096 conv=notrunc,noerrer

The -c sends the output to STDOUT.
The -d says "switch to decompression mode".
The input file is a compressed one.
Sending to STDOUT, also stops deletion of the input file.

The dd command, when there is no "if" specification, it inputs from STDIN.

The output file, the target, is the nvme.

*******

An alternate compressor you can try, is "pigz".

sudo apt install pigz
man pigz

Syntax is the same as gzip, and it accepts a -p 4
parameter, which is the number of processor
cores to use for compression.

If you use exactly the same compression syntax as the gzip -3
example, it uses all the cores on your CPU.

pigz -3 # On a 4C 8T processor, the default is -p 8

pigz output is ".gz" as before. gzip can be used for decompression.
pigz can be used for decompression. But during decompression, it
only uses one core. One claim I've seen, is decompression is typically
faster, so the lack of multithreading is not the end of the world.
It is compression that needs the horsepower.

You can use "top" or "gnome-system-monitor" to monitor the amount
of CPU usage during compression.

Anyway, I'm sure you have a whole collection of compressor commands.
There are compressors to produce smaller output, but they don't
run as fast.

Paul

On 2023-11-02, Dan Purgert wrote:
> On 2023-11-02, bilsch01 wrote:
>> Hello, I was never very good at linux scripting, and now I've forgotten
>> much what I did know. I hope somebody here can help me incorporate gzip
>> with the dd copy lines I use for backing up laptop to an external drive.
>> The two different thumb drives I use to boot laptop and copy SDD are
>> 1) Ubuntu Installer and 2) Knoppix. Both of those come with gzip.
>> I've never actually ever used gzip.
>>
>> Here's the lines I use to copy the SDD contents to an external drive:
>>
>> ## first do fdisk -l to sort out the devices
>> sudo mkdir -p /mnt/sdb1 (creating a mount point)
>> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
>> sudo mkdir -p /mnt/sdb1/images (if u don't already have that directory)
>> sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096
>> conv=notrunc,noerror
>
> You'd need something to the effect of:
> dd if=/the/device (NO if=) [...] | gzip > /mnt/sdb1/images/xx.img.gz
^^^^^ Ugh, that should be "NO of="

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On 11/2/2023 6:39 AM, Paul wrote:
> On 11/2/2023 3:52 AM, bilsch01 wrote:
>> Hello, I was never very good at linux scripting, and now I've forgotten much what I did know. I hope somebody here can help me incorporate gzip with the dd copy lines I use for backing up laptop to an external drive. The two different thumb drives I use to boot laptop and copy SDD are
>> 1) Ubuntu Installer and 2) Knoppix.  Both of those come with gzip.
>> I've never actually ever used gzip.
>>
>> Here's the lines I use to copy the SDD contents to an external drive:
>>
>> ## first do fdisk -l to sort out the devices
>> sudo mkdir -p /mnt/sdb1  (creating a mount point)
>> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
>> sudo mkdir -p /mnt/sdb1/images (if u don't already have that directory)
>> sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096 conv=notrunc,noerror
>>
>> Here's the lines to restore the SDD if I ever need to:
>>
>> sudo mkdir -p /mnt/sdb1  (creating a mount point)
>> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
>> sudo dd if=/mnt/sdb1/images/asyymmdd.img of=/dev/nvme0n1 bs=4096 conv=notrunc,noerrer
>>
>> Thanks in advance.
>> Bill S.
>>

If you really do have an NVMe, you should read this page, starting from scroll-bar-50%.

https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing

That will give you some idea, the naming convention is a little more complicated
than expected. Some references to the drive will be /dev/nvme0n1 namespace,
while some references are /dev/nvme device. This is most important,
when studying what the hell all the erase options mean :-/ You may not need
to erase your NVMe all that often.

For normal usage of the device, carry on with what you're doing.

*******

When making a backup of the device, there may be better ways to do it.

For the recipe below, I tried to "make" an output, exactly the same as your current method.

It is possible, for example, that only certain areas of the thing need to be backed up.

https://clonezilla.org/show-live-doc-content.php?topic=clonezilla-live/doc/01_Save_disk_image

Say, for example, we purchased a HDD exactly the same size as the NVMe. It
turns out, that the "faked" size of the devices, can be the same, for commonality.
I checked a 4TB SSD, a 4TB WD drive, a 4TB Seagate drive, all were 4,000,787,030,016 bytes.
And that is a fake size based on CHS principles. When CHS is involved, the number is
evenly divisible by 63. Each of these devices has more bytes inside, but we cannot
have them.

Steps:

1) Zero out the entire intermediary hard drive. it is now shiny clean, no MBR even.

2) Clonezilla NVMe to intermediary hard drive.

3) Run your regular backup script, pointed at intermediary. Back up intermediary to your external backup drive.

4) Immediately notice size reduction :-) Only the occupied space
is compressed in the output. White space is not taking up space.
The output from (3) can be restored to the NVMe the usual way.

This recipe assumes a lot more backups are done, than restores,
and even if the restore is inefficient (NVMe wear), you're not doing it all that
often, so it's not beating up the NVMe. The lifecycle is shown
in the diagram. The leftmost steps now take a lot of time, but
the .img.gz is smaller so you can make more of them :-)

Clonezilla d2d UsualScript UsualScript
/ \ / \ / \
NVMe Zeroed-out intermediate disk Backup drive (single file .img.gz) NVMe

<=============================== Slower than normal ==============================> <==== Normal speed =====>

Now, before someone sez "Geez Paul, that's an awful lot of effort to get a single output file".
Yes, it is. It's an illustration.

If you use Clonezilla and just image the NVMe, you get all the benefits. The difference
is, the backup sits in a folder and is a folder of stuff. Personally, I *hate* folders of
stuff for backups (that's how Windows does the "Windows 7" flavor of backups, I won't
be using that either, and that's the reason I won't use it).

I want *one* file to represent a point-in-time. And if you say
"well, you could just TAR up the folder and stop complaining".
Yes, you can do lots of things. Is random access into TAR files easy and efficient ?
Or does the software have to read the whole tape to potentially find the item ?
Would my first step be untarring the output to my scratch drive (the intermediate one) ? In the above
recipe, there is no real "handling" going on. I never see a folder full of files. One cookie
comes out of the oven at the end.

Note. I do NOT do "dd" backups on a regular basis. But, I DO know how.
Yes, I have done them. I make safety backups with dd, when testing
backup/restore on new backup software.

*******

If you use "fdisk", as part of the output at the beginning ("p" for print), it
states the total size in bytes. Say for example, it reports

4000787030016

factor 4000787030016
4000787030016: 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 7 2584007

The power-of-two size is only 8192 or 8KB. Which is unfortunate, as
this is not a natural size for the Flash device. It got truncated that
way, to make it more closely match the declaration on a 4TB hard drive.
A 4TB SSD, a 4TB WD HDD, a 4TB Seagate HDD, all reported the same size in bytes.

In terms of dd BS block sizes, in this case, you could use

bs= 4096
bs= 8192 <=== a good size for modern storage devices
bs=8192*27= 221184 <=== a good size for legacy hard drives (their cache chip doesn't really work).
This size choice may have slightly less write amplification on an SSD/NVMe.
Bigger is better, until you hit the device limit on write command size.
For example, would 8192*27*7 work ? Maybe.

Paul

On 03/11/2023 15:24, Paul wrote:
> On 11/2/2023 6:39 AM, Paul wrote:
>> On 11/2/2023 3:52 AM, bilsch01 wrote:
>>> Hello, I was never very good at linux scripting, and now I've forgotten much what I did know. I hope somebody here can help me incorporate gzip with the dd copy lines I use for backing up laptop to an external drive. The two different thumb drives I use to boot laptop and copy SDD are
>>> 1) Ubuntu Installer and 2) Knoppix.  Both of those come with gzip.
>>> I've never actually ever used gzip.
>>>
>>> Here's the lines I use to copy the SDD contents to an external drive:
>>>
<...snipped...>
>
> It is possible, for example, that only certain areas of the thing need to be backed up.
>
> https://clonezilla.org/show-live-doc-content.php?topic=clonezilla-live/doc/01_Save_disk_image
>
> Say, for example, we purchased a HDD exactly the same size as the NVMe. It
> turns out, that the "faked" size of the devices, can be the same, for commonality.
> I checked a 4TB SSD, a 4TB WD drive, a 4TB Seagate drive, all were 4,000,787,030,016 bytes.
> And that is a fake size based on CHS principles. When CHS is involved, the number is
> evenly divisible by 63. Each of these devices has more bytes inside, but we cannot
> have them.
>
<...snipped...>

Did I misunderstand, or did you mean
https://clonezilla.org/fine-print-live-doc.php?path=clonezilla-live/doc/03_Disk_to_disk_clone
(works for me; without the TORAM option makes a clone --identical -- HDD)

-- No Micro$oft products were used in the URLs above, or in preparing this message. Recommended reading:
http://www.catb.org/~esr/faqs/smart-questions.html#befor

On 11/3/2023 11:46 AM, Henry Crun wrote:
> On 03/11/2023 15:24, Paul wrote:
>> On 11/2/2023 6:39 AM, Paul wrote:
>>> On 11/2/2023 3:52 AM, bilsch01 wrote:
>>>> Hello, I was never very good at linux scripting, and now I've forgotten much what I did know. I hope somebody here can help me incorporate gzip with the dd copy lines I use for backing up laptop to an external drive. The two different thumb drives I use to boot laptop and copy SDD are
>>>> 1) Ubuntu Installer and 2) Knoppix.  Both of those come with gzip.
>>>> I've never actually ever used gzip.
>>>>
>>>> Here's the lines I use to copy the SDD contents to an external drive:
>>>>
> <...snipped...>
>>
>> It is possible, for example, that only certain areas of the thing need to be backed up.
>>
>>     https://clonezilla.org/show-live-doc-content.php?topic=clonezilla-live/doc/01_Save_disk_image
>>
>> Say, for example, we purchased a HDD exactly the same size as the NVMe. It
>> turns out, that the "faked" size of the devices, can be the same, for commonality.
>> I checked a 4TB SSD, a 4TB WD drive, a 4TB Seagate drive, all were 4,000,787,030,016 bytes.
>> And that is a fake size based on CHS principles. When CHS is involved, the number is
>> evenly divisible by 63. Each of these devices has more bytes inside, but we cannot
>> have them.
>>
> <...snipped...>
>
> Did I misunderstand, or did you mean
> https://clonezilla.org/fine-print-live-doc.php?path=clonezilla-live/doc/03_Disk_to_disk_clone
> (works for me; without the TORAM option makes a clone --identical -- HDD)
>
> -- No Micro$oft products were used in the URLs above, or in preparing this message. Recommended reading:
> http://www.catb.org/~esr/faqs/smart-questions.html#befor
>

I mean this. I have a maintenance stick I boot for this sort of work.
sudo apt install... and away you go.

https://clonezilla.org/clonezilla-live/doc/03_Disk_to_disk_clone/images/ocs-05-2-device-device-clone.png

You can also do it, by having a second disk in the computer, and
running Clonezilla from that, and treating the second device as
a "data disk at rest". I have two hard drives with multiple Linux
distros on them, and others that aren't in regular rotation/usage.

At this point, we don't know anything about the machine, or even
how the OP is getting on.

Backing up with just dd, means you are copying white space, which
could have "noise" in it. And the noise is hard for GZIP to
compress, wasting space on the destination. There is more
than one strategy for cleaning white space. Apparently
some of the methods are better than others, but I don't
know the details well enough to pick one. Using the cloning
strategies of Clonezilla, should clean the white space when
you make a clone copy. And without the long discussions about
which cleaning utility is a bad idea. You don't really want to
run a cleaner on an NVMe anyway. That's why I'm cloning to a hard
drive as an intermediate step, and storing on an external hard drive.

Paul

This capability will be so much better than straight dd copy.
I can always just do dd and then gzip later. Of course.
Info regarding gzip you provided will save me a lot of time and
potential headaches.
Thanks so much.
Bill S.

On 11/2/2023 3:39 AM, Paul wrote:
> On 11/2/2023 3:52 AM, bilsch01 wrote:
>> Hello, I was never very good at linux scripting, and now I've forgotten much what I did know. I hope somebody here can help me incorporate gzip with the dd copy lines I use for backing up laptop to an external drive. The two different thumb drives I use to boot laptop and copy SDD are
>> 1) Ubuntu Installer and 2) Knoppix.  Both of those come with gzip.
>> I've never actually ever used gzip.
>>
>> Here's the lines I use to copy the SDD contents to an external drive:
>>
>> ## first do fdisk -l to sort out the devices
>> sudo mkdir -p /mnt/sdb1  (creating a mount point)
>> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
>> sudo mkdir -p /mnt/sdb1/images (if u don't already have that directory)
>> sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096 conv=notrunc,noerror
>>
>> Here's the lines to restore the SDD if I ever need to:
>>
>> sudo mkdir -p /mnt/sdb1  (creating a mount point)
>> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
>> sudo dd if=/mnt/sdb1/images/asyymmdd.img of=/dev/nvme0n1 bs=4096 conv=notrunc,noerrer
>>
>> Thanks in advance.
>> Bill S.
>>
>
> sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096 conv=notrunc,noerror
>
> becomes
>
> sudo dd if=/dev/nvme0n1 bs=4096 conv=notrunc,noerror | gzip -3 > /mnt/sdb1/images/asyymmdd.img.gz
>
> The dd program, when you remove the "of" specification, uses STDOUT.
>
> GZIP accepts piped input. There are options of --fast and --slow which
> correspond to -1 and -9. The value -3 for compression is the default,
> and is shown as an example of a passed parameter. You might want to
> test the output, and see if the size changes or not
>
> ... | gzip -1 | wc -c # count the bytes of output
>
> ... | gzip -9 | wc -c # count the bytes of output
>
> With gzip, you have to be careful of the syntax,
> so the original file is not deleted. That's what I spent
> most of my time playing with, is checking for deletion.

Can you be more specific about what syntax will result in deletion of
original file?
Thank you.

>
> *******
>
> Later:
>
> gzip -c -d /mnt/sdb1/images/asyymmdd.img.gz | sudo dd of=/dev/nvme0n1 bs=4096 conv=notrunc,noerrer
>
> The -c sends the output to STDOUT.
> The -d says "switch to decompression mode".
> The input file is a compressed one.
> Sending to STDOUT, also stops deletion of the input file.
>
> The dd command, when there is no "if" specification, it inputs from STDIN.
>
> The output file, the target, is the nvme.
>
> *******
>
> An alternate compressor you can try, is "pigz".
>
> sudo apt install pigz
> man pigz
>
> Syntax is the same as gzip, and it accepts a -p 4
> parameter, which is the number of processor
> cores to use for compression.
>
> If you use exactly the same compression syntax as the gzip -3
> example, it uses all the cores on your CPU.
>
> pigz -3 # On a 4C 8T processor, the default is -p 8
>
> pigz output is ".gz" as before. gzip can be used for decompression.
> pigz can be used for decompression. But during decompression, it
> only uses one core. One claim I've seen, is decompression is typically
> faster, so the lack of multithreading is not the end of the world.
> It is compression that needs the horsepower.
>
> You can use "top" or "gnome-system-monitor" to monitor the amount
> of CPU usage during compression.
>
> Anyway, I'm sure you have a whole collection of compressor commands.
> There are compressors to produce smaller output, but they don't
> run as fast.
>
> Paul

On 2023-11-05, bilsch01 wrote:
> On 11/2/2023 3:39 AM, Paul wrote:
>> [...]
>> With gzip, you have to be careful of the syntax,
>> so the original file is not deleted. That's what I spent
>> most of my time playing with, is checking for deletion.
>
> Can you be more specific about what syntax will result in deletion of
> original file?

Any invocation on a file (i.e. NOT piped input) that does not include
the '-k' switch.

gzip file.txt <-- will create file.txt.gz, and delete file.txt
gzip -k file.txt <-- will create file.txt.gz, and KEEP file.txt

Piped input (e.g. 'command | gzip ') has no file to keep in the first
place.

--
|_|O|_|
|_|_|O| Github: https://github.com/dpurgert
|O|O|O| PGP: DDAB 23FB 19FA 7D85 1CC1 E067 6D65 70E5 4CE7 2860

On 11/3/2023 6:24 AM, Paul wrote:
> On 11/2/2023 6:39 AM, Paul wrote:
>> On 11/2/2023 3:52 AM, bilsch01 wrote:
>>> Hello, I was never very good at linux scripting, and now I've forgotten much what I did know. I hope somebody here can help me incorporate gzip with the dd copy lines I use for backing up laptop to an external drive. The two different thumb drives I use to boot laptop and copy SDD are
>>> 1) Ubuntu Installer and 2) Knoppix.  Both of those come with gzip.
>>> I've never actually ever used gzip.
>>>
>>> Here's the lines I use to copy the SDD contents to an external drive:
>>>
>>> ## first do fdisk -l to sort out the devices
>>> sudo mkdir -p /mnt/sdb1  (creating a mount point)
>>> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
>>> sudo mkdir -p /mnt/sdb1/images (if u don't already have that directory)
>>> sudo dd if=/dev/nvme0n1 of=/mnt/sdb1/images/asyymmdd.img bs=4096 conv=notrunc,noerror
>>>
>>> Here's the lines to restore the SDD if I ever need to:
>>>
>>> sudo mkdir -p /mnt/sdb1  (creating a mount point)
>>> sudo mount -t ntfs /dev/sdb1 /mnt/sdb1
>>> sudo dd if=/mnt/sdb1/images/asyymmdd.img of=/dev/nvme0n1 bs=4096 conv=notrunc,noerrer
>>>
>>> Thanks in advance.
>>> Bill S.
>>>
>
> If you really do have an NVMe, you should read this page, starting from scroll-bar-50%.
>
> https://wiki.archlinux.org/title/Solid_state_drive/Memory_cell_clearing
>
> That will give you some idea, the naming convention is a little more complicated
> than expected. Some references to the drive will be /dev/nvme0n1 namespace,
> while some references are /dev/nvme device. This is most important,
> when studying what the hell all the erase options mean :-/ You may not need
> to erase your NVMe all that often.

Jeeze ... I had no idea about this special NVMe stuff.
This is my next study project.
Thanks Paul.

>
> For normal usage of the device, carry on with what you're doing.
>
> *******
>
> When making a backup of the device, there may be better ways to do it.
>
> For the recipe below, I tried to "make" an output, exactly the same as your current method.
>
> It is possible, for example, that only certain areas of the thing need to be backed up.
>
> https://clonezilla.org/show-live-doc-content.php?topic=clonezilla-live/doc/01_Save_disk_image
>
> Say, for example, we purchased a HDD exactly the same size as the NVMe. It
> turns out, that the "faked" size of the devices, can be the same, for commonality.
> I checked a 4TB SSD, a 4TB WD drive, a 4TB Seagate drive, all were 4,000,787,030,016 bytes.
> And that is a fake size based on CHS principles. When CHS is involved, the number is
> evenly divisible by 63. Each of these devices has more bytes inside, but we cannot
> have them.
>
> Steps:
>
> 1) Zero out the entire intermediary hard drive. it is now shiny clean, no MBR even.
>
> 2) Clonezilla NVMe to intermediary hard drive.
>
> 3) Run your regular backup script, pointed at intermediary. Back up intermediary to your external backup drive.
>
> 4) Immediately notice size reduction :-) Only the occupied space
> is compressed in the output. White space is not taking up space.
> The output from (3) can be restored to the NVMe the usual way.
>
> This recipe assumes a lot more backups are done, than restores,
> and even if the restore is inefficient (NVMe wear), you're not doing it all that
> often, so it's not beating up the NVMe. The lifecycle is shown
> in the diagram. The leftmost steps now take a lot of time, but
> the .img.gz is smaller so you can make more of them :-)
>
> Clonezilla d2d UsualScript UsualScript
> / \ / \ / \
> NVMe Zeroed-out intermediate disk Backup drive (single file .img.gz) NVMe
>
> <=============================== Slower than normal ==============================> <==== Normal speed =====>
>
> Now, before someone sez "Geez Paul, that's an awful lot of effort to get a single output file".
> Yes, it is. It's an illustration.
>
> If you use Clonezilla and just image the NVMe, you get all the benefits. The difference
> is, the backup sits in a folder and is a folder of stuff. Personally, I *hate* folders of
> stuff for backups (that's how Windows does the "Windows 7" flavor of backups, I won't
> be using that either, and that's the reason I won't use it).
>
> I want *one* file to represent a point-in-time. And if you say
> "well, you could just TAR up the folder and stop complaining".
> Yes, you can do lots of things. Is random access into TAR files easy and efficient ?
> Or does the software have to read the whole tape to potentially find the item ?
> Would my first step be untarring the output to my scratch drive (the intermediate one) ? In the above
> recipe, there is no real "handling" going on. I never see a folder full of files. One cookie
> comes out of the oven at the end.
>
> Note. I do NOT do "dd" backups on a regular basis. But, I DO know how.
> Yes, I have done them. I make safety backups with dd, when testing
> backup/restore on new backup software.
>
> *******
>
> If you use "fdisk", as part of the output at the beginning ("p" for print), it
> states the total size in bytes. Say for example, it reports
>
> 4000787030016
>
> factor 4000787030016
> 4000787030016: 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 7 2584007
>
> The power-of-two size is only 8192 or 8KB. Which is unfortunate, as
> this is not a natural size for the Flash device. It got truncated that
> way, to make it more closely match the declaration on a 4TB hard drive.
> A 4TB SSD, a 4TB WD HDD, a 4TB Seagate HDD, all reported the same size in bytes.
>
> In terms of dd BS block sizes, in this case, you could use
>
> bs= 4096
> bs= 8192 <=== a good size for modern storage devices
> bs=8192*27= 221184 <=== a good size for legacy hard drives (their cache chip doesn't really work).
> This size choice may have slightly less write amplification on an SSD/NVMe.
> Bigger is better, until you hit the device limit on write command size.
> For example, would 8192*27*7 work ? Maybe.
>
> Paul
>

On 11/5/2023 7:07 AM, Dan Purgert wrote:
> On 2023-11-05, bilsch01 wrote:
>> On 11/2/2023 3:39 AM, Paul wrote:
>>> [...]
>>> With gzip, you have to be careful of the syntax,
>>> so the original file is not deleted. That's what I spent
>>> most of my time playing with, is checking for deletion.
>>
>> Can you be more specific about what syntax will result in deletion of
>> original file?
>
> Any invocation on a file (i.e. NOT piped input) that does not include
> the '-k' switch.
>
> gzip file.txt <-- will create file.txt.gz, and delete file.txt
> gzip -k file.txt <-- will create file.txt.gz, and KEEP file.txt
>
>
> Piped input (e.g. 'command | gzip ') has no file to keep in the first
> place.
>
Thanks for explaining. I needed that.