I figured it may be instructive to have a thread about this kind
of thing.

By policies/practices I mean not just the back-ups themselves but
also whether you have any such policies on using your back-ups
for recovering from various disaster scenarios and whether you
ever got to test those policies for real.

I'll start.

Main back-up medium for me is writable DVDs. My back-ups are not
as regular as they ought to be , it could happen that days happen
between successive back-ups. I have a BASH function called
crucial which creates a tar file of all the files (including
directories) I want on back-up. crucial accepts a single argument
which is number of days since last time the files were modified.
If I want on the tar file everything "crucial" then I will run
the function with a large enough argument that all the "crucial"
files are covered. Generally a back-up cycle is that I run
crucial with a large argument and on following days with smaller
ones until I have enough tar files so that not much of the DVD
goes to waste , so say at least 4,300,000,000 bytes total. As an
intermediate measure before burning a DVD , I may copy some of the
tar files on a memory stick.

Crucial files include all the settings for software I use often
like BASH functions and settings , vim functions and settings ,
etc. So in theory I could restore a working system exactly as I
want it from a single tar file. But I've never had to test it
because I've never had a failure of a hard disk ! Lucky I guess.

Non "crucial" files go on 1-2 back-up DVDs depending on how
important I consider them. These files tend to be larger and it
would be too inconvenient to have them on regular back-ups i.e.
every few days at most. I would certainly need to have a different
method than DVDs because the files in total would occupy a lot
more than what can fit on a DVD. Such files can be videos or audio
from youtube and other sources. Often but not always the file will
remain available at the original download link and the list of
links exist in 1-2 of the crucial files.

I want to also have hourly back-ups of files which get modified
quickly like if I'm writing some code or for a file with my daily
activities. I will probably write a function crucial2 for this
kind of thing but I haven't got around to it. RAID would be ideal
but it's not a possibility with the hardware I'm using at present.

--
My sister, also a conductor, once explained to the board of one of her
orchestras why she wouldn't let them play Mozart in her first season;
"Mozart" she said, "is the string bikini of composers, and I just think
that we, as an orchestra, don't have the body to pull it off yet."
https://kennethwoods.net/blog1/2012/06/25/which-would-you-rather-conduct-or-joining-the-mozart-protection-society/

Spiros Bousbouras <spibou@gmail.com> wrote:
> Main back-up medium for me is writable DVDs. My back-ups are not as
> regular as they ought to be , it could happen that days happen
> between successive back-ups. I have a BASH function called crucial
> which creates a tar file of all the files (including directories) I
> want on back-up. crucial accepts a single argument which is number
> of days since last time the files were modified. If I want on the
> tar file everything "crucial" then I will run the function with a
> large enough argument that all the "crucial" files are covered.
> Generally a back-up cycle is that I run crucial with a large argument
> and on following days with smaller ones until I have enough tar files
> so that not much of the DVD goes to waste , so say at least
> 4,300,000,000 bytes total. As an intermediate measure before burning
> a DVD , I may copy some of the tar files on a memory stick.

Do you also compress the tar files? (gzip, bzip2, xz, or lrzip?).

> RAID would be ideal but it's not a possibility with the hardware I'm
> using at present.

Can you explain why not?

My backup is rsnapshot, run nightly from cron, to a second machine with
a three disk Linux MD RAID5. As part of that nightly run, much of the
really important stuff is further copied from the RAID to another drive
on another machine, under the expectation that the risk of having all
of 1) the original disk, 2) two of the RAID disks, and 3) the third
disk all fail within "recovery time window X" is minisicule.

The RAID machine is a very old Pentium4 motherboard without SATA ports.
It became a RAID machine with the purchase of a 4-port to PCI (old
oridinal PCI) SATA card. That 4-port SATA to PCI adapter was, IIRC,
all of something like $25 to $35 when I bought it. It runs Linux md to
RAID the three drives into a big disk, and then runs LVM2 on top of
that big RAID disk to allow for create/resize/delete of partitions
within the RAID disk space.

I've had to recover three times so far since I set this up. Once was
failure of boot/home disk in main desktop. Second was when one of the
disks in the RAID dropped out. That event led to the "third disk" I
mention above, as I ordered it expecting to replace the disk that
dropped. When "third disk" arrived, it turned out that the dropped
disk was just a SATA cable had worked itself loose. So reseating, and
RAID rebuild onto that disk, brought the RAID back to full redundancy.
And as I now had a "spare" I turned it into the "third" backup copy.

The third time was an actual failure of one of the three disks in the
raid. Replace that failed disk with a new disk, rebuild the RAID, and
it was again back to working order.

In both of the RAID events, the RAID continued operation, albiet with
slightly degraded performance, while the single disks were dropped out.
Downtime was only the time to shutdown, swap disks (or reseat SATA
cable), blow out accumulated dust, and boot back up.

On Sun, 04 Jun 2023 12:14:30 +0000, Spiros Bousbouras wrote:

> I figured it may be instructive to have a thread about this kind of
> thing.
>
> By policies/practices I mean not just the back-ups themselves but also
> whether you have any such policies on using your back-ups for recovering
> from various disaster scenarios and whether you ever got to test those
> policies for real.

I run a backup script every night. I use UFS on BSD, so I use the dump
utility since it is arguably the best for this filesystem.

I back up complete filesystems, including the system files, for a quick
restore.

On Saturdays it does a complete backup; on other days it does a
differential backup *with the previous Saturday as a base*. In other
wprds, a restore only needs the last differential and also the last
Saturday one. The backups go to the file server, with the file server
itself being backed up elsewhere.

Every Saturday the full backup is also copied to USB storage. That gets
cycled; there are seven USBs per machine. 0, 1,2, 3, 0, 1, 2, 4, 0, 1, 2,
5, 0, 1, 2, 6. So I always have last last four weeks and beyone that the
three previous months.

Every 16 weeks (when the cycle ends) I copy the latest full backup to DVD.
I may reconsider the medium, as the file server now takes 7 DVDs, and two
others use 3 or more. I generate 2 copies of those DVDs, one being kept on
a different floor of the building, and the other being taken to a storage
facility 10 miles away.

In addition, files that I can't easily recreate (system files are not in
this category) are sent to a rolling backup on tarsnap. The last month
(every day), the last year (every month), and forever (every year, going
back 10 years so far).

I also regularly dump more static files (big ones) into Amazon S3 Deep
Archive.

--
Using UNIX since v6 (1975)...

Use the BIG mirror service in the UK:
http://www.mirrorservice.org

On Sun, 04 Jun 2023 12:54:50 +0000, Bob Eager wrote:

> On Sun, 04 Jun 2023 12:14:30 +0000, Spiros Bousbouras wrote:
>
>> I figured it may be instructive to have a thread about this kind of
>> thing.
>>
>> By policies/practices I mean not just the back-ups themselves but also
>> whether you have any such policies on using your back-ups for
>> recovering from various disaster scenarios and whether you ever got to
>> test those policies for real.
>
> I run a backup script every night. I use UFS on BSD, so I use the dump
> utility since it is arguably the best for this filesystem.
>
> I back up complete filesystems, including the system files, for a quick
> restore.
>
> On Saturdays it does a complete backup; on other days it does a
> differential backup *with the previous Saturday as a base*. In other
> wprds, a restore only needs the last differential and also the last
> Saturday one. The backups go to the file server, with the file server
> itself being backed up elsewhere.
>
> Every Saturday the full backup is also copied to USB storage. That gets
> cycled; there are seven USBs per machine. 0, 1,2, 3, 0, 1, 2, 4, 0, 1,
> 2, 5, 0, 1, 2, 6. So I always have last last four weeks and beyone that
> the three previous months.
>
> Every 16 weeks (when the cycle ends) I copy the latest full backup to
> DVD.
> I may reconsider the medium, as the file server now takes 7 DVDs, and
> two others use 3 or more. I generate 2 copies of those DVDs, one being
> kept on a different floor of the building, and the other being taken to
> a storage facility 10 miles away.
>
> In addition, files that I can't easily recreate (system files are not in
> this category) are sent to a rolling backup on tarsnap. The last month
> (every day), the last year (every month), and forever (every year, going
> back 10 years so far).
>
> I also regularly dump more static files (big ones) into Amazon S3 Deep
> Archive.

I forgot to say that the backup files are currently compressed with gzip
(moving to xz though). And the DVDs are not filled, as I add ECC for
recovery if a spot is unreadable.

--
Using UNIX since v6 (1975)...

Use the BIG mirror service in the UK:
http://www.mirrorservice.org

Spiros Bousbouras <spibou@gmail.com> writes:
> I figured it may be instructive to have a thread about this kind
> of thing.
>
> By policies/practices I mean not just the back-ups themselves but
> also whether you have any such policies on using your back-ups
> for recovering from various disaster scenarios and whether you
> ever got to test those policies for real.

I rsync everything (with exclusions for caches etc) to three USB disks,
of which one is always online at home; one is always off-site; and one
is either online at home, or in transit to my off-site location. The
mapping of physical disk to these three roles varies informally.

I keep multiple historical copies, using rsync’s ability to make hard
links between files within a backup. Past a threshold, historical
backups get thinned out, so that (for example) there’s a single backup a
week old, one two weeks old, one four weeks old, etc.

Stuff like /home gets backed up more frequently than stuff like /boot,
since the latter changes so rarely.

I’ve restored after disk failure a number of times over the years and it
went fine.

I use https://www.greenend.org.uk/rjk/rsbackup/ to manage all this.

--
https://www.greenend.org.uk/rjk/

I run (slackware 14.2) on a 2TB SSD with an unmounted 4TB HD with four
1TB partitions, as well as two 1TB USB drives. Hubby wrote some nice
scripts to write my entire working system into my choice of partitions
or HD, either from scratch or incremental. All backups are bootable.
I've only had to restore a file from a backup one or two times since
1998, plus one entire subdirectory that I deleted thinking it was a
different subdirectory. I remember something weird happening with my
working partition (before SSD) that required booting into another one to
repair the working partition (threw that 8TB drive away ASAP). I've
also got some thumb drives with tax crap and photos and stuff I want to
share.

Storage is cheap.

--
Cheers, Bev
Giving advice likely to kill the stupid is called passive eugenics.

Software-wise it varies between systems, and some data that changes
most regularly is automatically copied to another system daily that
is then manually backed up weekly. I generally prefer filesystem or
disk image based backups, but in many cases Tar is more practical.
I try to save the MBR but I always end up with the bootloader
giving me trouble during restores from filesystem or Tar archives
anyway - too many ways for things to get stuffed up, and it seems
I can't learn all the lessons. The data is usually XZ compressed,
except on slow systems.

Backup data goes onto HDDs. Primarily a USB HDD and then two remote
backups on internal HDDs that I connect via a USB adapter, but one
of those only gets accessed a few times a year. Nothing over the
internet except automatic encrypted Tar backup of a VPS.

I'm surprised that people are still using DVDs.

--
__ __
#_ < |\| |< _#

On Sun, 4 Jun 2023 12:50:34 -0000 (UTC)
Rich <rich@example.invalid> wrote:
> Spiros Bousbouras <spibou@gmail.com> wrote:
> > Main back-up medium for me is writable DVDs. My back-ups are not as
> > regular as they ought to be , it could happen that days happen
> > between successive back-ups. I have a BASH function called crucial
> > which creates a tar file of all the files (including directories) I
> > want on back-up. crucial accepts a single argument which is number
> > of days since last time the files were modified. If I want on the
> > tar file everything "crucial" then I will run the function with a
> > large enough argument that all the "crucial" files are covered.
> > Generally a back-up cycle is that I run crucial with a large argument
> > and on following days with smaller ones until I have enough tar files
> > so that not much of the DVD goes to waste , so say at least
> > 4,300,000,000 bytes total. As an intermediate measure before burning
> > a DVD , I may copy some of the tar files on a memory stick.
>
> Do you also compress the tar files? (gzip, bzip2, xz, or lrzip?).

No. A tar with all the "crucial" files occupies around 3,500,000,000 bytes
and I don't expect this to change significantly over my lifetime so it
fits comfortably on a DVD , compression would just be an extra complication.

> > RAID would be ideal but it's not a possibility with the hardware I'm
> > using at present.
>
> Can you explain why not?

Because I don't think my hardware supports it but then again I haven't really
searched. I'm not even sure which part of the hardware is responsible ;
motherboard ? Apart from that , there would be issues with physical space and
I'm happy enough with my current set up. Once I have the crucial2 function
I mention in my opening post , I should be fine. The only possibility
compared to RAID would be to lose ongoing work (like code I'm in the process
of writing) from the last hour or so. This would be irritating but fixable.
And it can only happen if the problem on the main disk is bad enough that
I can't copy the recently modified files to a different medium once I spot
the problem.

Your comments about RAID prompted me to have a look in the wikipedia article
and it turned out that I had a very simplistic view of what a RAID does. I
was under the impression that with a RAID everything simply gets copied
automatically to several disks and that made me wonder why with 1 less disk
functioning , the performance would drop as opposed to becoming higher (with
the tradeoff of lower redundancy). But it is (or can be) a lot more complicated
than this.

> I've had to recover three times so far since I set this up. Once was
> failure of boot/home disk in main desktop. Second was when one of the
> disks in the RAID dropped out. That event led to the "third disk" I
> mention above, as I ordered it expecting to replace the disk that
> dropped. When "third disk" arrived, it turned out that the dropped
> disk was just a SATA cable had worked itself loose. So reseating, and
> RAID rebuild onto that disk, brought the RAID back to full redundancy.
> And as I now had a "spare" I turned it into the "third" backup copy.

I assume there was no software based way to realise that it was the cable
rather than the disk which caused the problem ; the only way was to open
a computer case and see the loose cable.

> In both of the RAID events, the RAID continued operation, albiet with
> slightly degraded performance, while the single disks were dropped out.
> Downtime was only the time to shutdown, swap disks (or reseat SATA
> cable), blow out accumulated dust, and boot back up.

--
There are only two kinds of storage devices - those that have failed,
and those that are about to fail.
Jonathan Schwartz
http://blogs.sun.com/jonathan/entry/not_a_flash_in_the

On 4 Jun 2023 13:31:37 GMT
Bob Eager <news0009@eager.cx> wrote:
> I forgot to say that the backup files are currently compressed with gzip
> (moving to xz though). And the DVDs are not filled, as I add ECC for
> recovery if a spot is unreadable.

Is there preexisting software which does this and can also use the ECC for
recovery or have you devised your own scheme ?

On Sun, 04 Jun 2023 15:45:04 +0100
Richard Kettlewell <invalid@invalid.invalid> wrote:
> Stuff like /home gets backed up more frequently than stuff like /boot,
> since the latter changes so rarely.
>
> I’ve restored after disk failure a number of times over the years and it
> went fine.
>

What were the steps ? Were they something like

1. You diagnosed the faulty medium.
2. Replaced it and reinstalled the operating system.
3. Restored the rest from back-up.

? Or is it that your back-ups allow you to combine steps 2 and 3 in one
step ?

--
vlaho.ninja/prog

On 5 Jun 2023 09:17:37 +1000
not@telling.you.invalid (Computer Nerd Kev) wrote:
> I'm surprised that people are still using DVDs.

As long as everything fits into one DVD then it's a cheap and space
efficient way to do back-ups. Whether it's more reliable than
alternative methods of storage I don't know. Does anyone know of
any graphs which show as a function of time the probability of a
storage medium going "bad" ? Of course , there are different
levels of badness ranging from completely unrecoverable to
<everything recoverable if you jump through enough hoops>.

Am 04.06.2023 um 12:14:30 Uhr schrieb Spiros Bousbouras:

> By policies/practices I mean not just the back-ups themselves but
> also whether you have any such policies on using your back-ups
> for recovering from various disaster scenarios and whether you
> ever got to test those policies for real.

I do backups on a second disk on my system. I irregular copy the entire
disk to another disk that is offline.

> Main back-up medium for me is writable DVDs.

I don't even have a DVD drive in my PC. I also don't like write-once
media.

> Crucial files include all the settings for software I use often
> like BASH functions and settings , vim functions and settings ,
> etc. So in theory I could restore a working system exactly as I
> want it from a single tar file. But I've never had to test it
> because I've never had a failure of a hard disk ! Lucky I guess.

I backup my entire ~, but excluded cache directories.

On Mon, 05 Jun 2023 09:16:48 +0000, Spiros Bousbouras wrote:

> On 4 Jun 2023 13:31:37 GMT Bob Eager <news0009@eager.cx> wrote:
>> I forgot to say that the backup files are currently compressed with
>> gzip (moving to xz though). And the DVDs are not filled, as I add ECC
>> for recovery if a spot is unreadable.
>
> Is there preexisting software which does this and can also use the ECC
> for recovery or have you devised your own scheme ?

There's a utility called dvdisaster. I use it on FreeBSD.

I tested it by adding 20% ECC (the minimum recommended). Then I used a
Swiss Army knife on the disc, cutting grooves and scraping bits off.

I did the recovery and lost nothing.

--
Using UNIX since v6 (1975)...

Use the BIG mirror service in the UK:
http://www.mirrorservice.org

Spiros Bousbouras <spibou@gmail.com> wrote:
> On Sun, 4 Jun 2023 12:50:34 -0000 (UTC)
> Rich <rich@example.invalid> wrote:
>> Spiros Bousbouras <spibou@gmail.com> wrote:
>> > RAID would be ideal but it's not a possibility with the hardware
>> > I'm using at present.
>>
>> Can you explain why not?
>
> Because I don't think my hardware supports it but then again I haven't really
> searched. I'm not even sure which part of the hardware is responsible ;
> motherboard ?

No special hardware is required for using Linux's 'md' driver. It is
pure software RAID. Any motherboard with sufficient disk ports or to
which an expansion card can be installed to provide sufficient disk
ports can be used with the md driver to provide RAID. In fact, one
/could/ simply attach a set of disks via USB port and use md to RAID
them. Performance over USB, unless one had USB3 or better, would not
be that great, but the md driver would combine them into a RAID array.
The motherboard I'm using for my raid box has no hardware support for
raid, and only two PATA disk ports on board, yet it has been performing
RAID5 with three drives on a PCI 4-port SATA board just fine for
somewhere going on about ten years or so now.

> Your comments about RAID prompted me to have a look in the wikipedia
> article and it turned out that I had a very simplistic view of what a
> RAID does. I was under the impression that with a RAID everything
> simply gets copied automatically to several disks and that made me
> wonder why with 1 less disk functioning , the performance would drop
> as opposed to becoming higher (with the tradeoff of lower
> redundancy). But it is (or can be) a lot more complicated than this.

Yes, it is more complicated than "automatically make a second copy"
(well, other than RAID1 which is "make two copies onto two identical
disks at the same time".

>> I've had to recover three times so far since I set this up. Once was
>> failure of boot/home disk in main desktop. Second was when one of the
>> disks in the RAID dropped out. That event led to the "third disk" I
>> mention above, as I ordered it expecting to replace the disk that
>> dropped. When "third disk" arrived, it turned out that the dropped
>> disk was just a SATA cable had worked itself loose. So reseating, and
>> RAID rebuild onto that disk, brought the RAID back to full redundancy.
>> And as I now had a "spare" I turned it into the "third" backup copy.
>
> I assume there was no software based way to realise that it was the cable
> rather than the disk which caused the problem ; the only way was to open
> a computer case and see the loose cable.

What I got "software" wise was "disk disappeared" (as in /dev/sdc (for
example) was no longer accessible in any way). I had no way to know
this was simply "SATA cable came loose" from "catastrophic drive
failure resulting in dead disk". And I ordered the spare disk thinking
I'd had a "disk fail completely" only to learn when I did open the box
that it was just the cable had slipped loose. In retrospect I should
have looked around in the box first -- but then had I done that I
wouldn't have the third "redundancy" disk, so.....

Spiros Bousbouras <spibou@gmail.com> wrote:
> On 4 Jun 2023 13:31:37 GMT
> Bob Eager <news0009@eager.cx> wrote:
>> I forgot to say that the backup files are currently compressed with gzip
>> (moving to xz though). And the DVDs are not filled, as I add ECC for
>> recovery if a spot is unreadable.
>
> Is there preexisting software which does this and can also use the ECC for
> recovery or have you devised your own scheme ?

https://en.wikipedia.org/wiki/Dvdisaster

Spiros Bousbouras <spibou@gmail.com> wrote:
> On 5 Jun 2023 09:17:37 +1000
> not@telling.you.invalid (Computer Nerd Kev) wrote:
>> I'm surprised that people are still using DVDs.
>
> As long as everything fits into one DVD then it's a cheap and space
> efficient way to do back-ups.

But not necessarially "human time" efficient as you need to manually
insert fresh DVD's (unless you also have a DVD robot).

> Whether it's more reliable than alternative methods of storage I
> don't know. Does anyone know of any graphs which show as a function
> of time the probability of a storage medium going "bad" ?

For DVD's, that highly depends upon the quality of the disks when they
were manufactured. At one point some 'el cheapo' DVD-R's would fail
only a few years after being burned. Secondary effects result from the
choice of dye on the DVD, and of course manner of storage adds a huge
variable. Storing them in a hot car, in full sunlight, will degrade
them faster than in a light tight box in a cool basement. Of course
the 'cool basement' adds the possibility of a dampness factor, which
can again change the variables.

For hard drives, the most complete dataset is the periodic data
published by BackBlaze of hard disk lifetimes in their cloud backup
system.

> Of course , there are different levels of badness ranging from
> completely unrecoverable to <everything recoverable if you jump
> through enough hoops>.

For some of the el-cheapo DVD's from yesteryear, the failure mode was
indeed "completely unrecoverable".

On Mon, 05 Jun 2023 13:20:14 +0000, Rich wrote:

> Spiros Bousbouras <spibou@gmail.com> wrote:
>> On Sun, 4 Jun 2023 12:50:34 -0000 (UTC)
>> Rich <rich@example.invalid> wrote:
>>> Spiros Bousbouras <spibou@gmail.com> wrote:
>>> > RAID would be ideal but it's not a possibility with the hardware I'm
>>> > using at present.
>>>
>>> Can you explain why not?
>>
>> Because I don't think my hardware supports it but then again I haven't
>> really searched. I'm not even sure which part of the hardware is
>> responsible ; motherboard ?
>
> No special hardware is required for using Linux's 'md' driver. It is
> pure software RAID. Any motherboard with sufficient disk ports or to
> which an expansion card can be installed to provide sufficient disk
> ports can be used with the md driver to provide RAID. In fact, one
> /could/ simply attach a set of disks via USB port and use md to RAID
> them. Performance over USB, unless one had USB3 or better, would not be
> that great, but the md driver would combine them into a RAID array. The
> motherboard I'm using for my raid box has no hardware support for raid,
> and only two PATA disk ports on board, yet it has been performing RAID5
> with three drives on a PCI 4-port SATA board just fine for somewhere
> going on about ten years or so now.

Same here. I use the FreeBSD geom gmirror and have done for years. No
machine has a single disk; they are all RAID1 pairs.

Just check that the disks aren't SMR and they haven't told you.
--
Using UNIX since v6 (1975)...

Use the BIG mirror service in the UK:
http://www.mirrorservice.org

On 04/06/2023 13:14, Spiros Bousbouras wrote:
> I figured it may be instructive to have a thread about this kind
> of thing.
>
> By policies/practices I mean not just the back-ups themselves but
> also whether you have any such policies on using your back-ups
> for recovering from various disaster scenarios and whether you
> ever got to test those policies for real.
>

herewith my contribution :-)

I have several computers. Mostly windows boxes but with a few Linux ones
too.

The Win boxes all have a 2nd hard drive to back up to amd also an
automatic backup to back up to a networked NAS.

The Linux boxes can be set up with a cron job and rum rsync to the 2nd
internal drive and to the NAS.

I have a directory for each networked device on the NAS.

The NAS use either RAID 6 or RAID 10 ( 4 drives in total and can
tolerate failure of up to two drives.)

You can use Linux MD or FreeBSD's ZFS or use a hardware based RAID
controller (cheap as chips such as the adaptec 7805 whcih will support 8
SATA drives) to do a RAID array.

The adaptec 7805's have onboard 512 MB RAM with a battery backup unit so
that data in transit is not lost during a power cut due to the BBU
(battery backup unit). The NAS is aslo supported by a UPS.

you will find backing up to SATA HDDs or SSDs will be faster than DVDs
and by automating as much of it as possible makes life much easier.

HDDs and SSDs are also much bigger than DVDs, heck a 500 GB SSD will
hold 100 full DVD's worth.

NASes I've used over the years include synology, home built Freenas
(based on FreeBSD), home built Open media Vault (based on Debian)

The NAS is also set up to syncronise itself with Microsoft's own
OneDrive. I pay £5.99 a month for 1TB cloud storage.

So I have the originals in each PC, individual backups in each PC's 2nd
drives, again in the NAS and again in an off-site backup in the off site
OneDrive.

There are other off site storgae available like Backblaze etc or Dropbox
or Google Drive

There is a password locked vault within Onedrive which can be used for
more sensitive stuff or you can apply encryption or password locking to
the files you then pass to the NAS before its onward syncing to OneDrive.

Even if I lost all my computers and NAS in a house fire or flood, I can
go to any internet connected computer, log into onedrive using
Chrome/Edge/Firefox etc and download the relevant backed up files to a
USB memory stick or USB hard drive and then buy/build a new machine,
install windows or linux and then recover my backups from said USB device(s)

obviously running big backsup to Onedrive over an ADSL connection would
be painful unless you are blessed like me to to have a synchronous 1
Gbit both ways fibre broadband. :-) or do incremental backups during the
night.

Spiros Bousbouras <spibou@gmail.com> wrote:
> On 5 Jun 2023 09:17:37 +1000
> not@telling.you.invalid (Computer Nerd Kev) wrote:
>> I'm surprised that people are still using DVDs.
>
> As long as everything fits into one DVD then it's a cheap and space
> efficient way to do back-ups.

I guess so. I have lots of photos that I include, so that would
need multiple discs, which would be a pain. One disc wouldn't be so
bad if it doesn't matter how long it takes.

> Whether it's more reliable than
> alternative methods of storage I don't know. Does anyone know of
> any graphs which show as a function of time the probability of a
> storage medium going "bad" ?

Rich answered this well. As a rule I only keep the current copy and
the previous copy of backups, and that's mainly just to protect
against system failure during the backup process. It sounds like
you keep an archive of all your old backups (as opposed to using
DVD-R/W discs). I do selectively archive software and system
configurations sometimes though, such as before a major change or
upgrade, and tend to keep those for at least a few years.

--
__ __
#_ < |\| |< _#

On Mon, 5 Jun 2023 13:20:14 -0000 (UTC)
Rich <rich@example.invalid> wrote:
> Spiros Bousbouras <spibou@gmail.com> wrote:
> > On Sun, 4 Jun 2023 12:50:34 -0000 (UTC)
> > Rich <rich@example.invalid> wrote:
> >> Spiros Bousbouras <spibou@gmail.com> wrote:
> >> > RAID would be ideal but it's not a possibility with the hardware
> >> > I'm using at present.
> >>
> >> Can you explain why not?
> >
> > Because I don't think my hardware supports it but then again I haven't really
> > searched. I'm not even sure which part of the hardware is responsible ;
> > motherboard ?
>
> No special hardware is required for using Linux's 'md' driver. It is
> pure software RAID. Any motherboard with sufficient disk ports or to
> which an expansion card can be installed to provide sufficient disk
> ports can be used with the md driver to provide RAID.

Ok , something to keep in mind.

From <u5knse$bktg$3@dont-email.me> :
> Spiros Bousbouras <spibou@gmail.com> wrote:
> > On 5 Jun 2023 09:17:37 +1000
> > not@telling.you.invalid (Computer Nerd Kev) wrote:
> > Whether it's more reliable than alternative methods of storage I
> > don't know. Does anyone know of any graphs which show as a function
> > of time the probability of a storage medium going "bad" ?
>
> For DVD's, that highly depends upon the quality of the disks when they
> were manufactured. At one point some 'el cheapo' DVD-R's would fail
> only a few years after being burned.

I buy Verbatim or TDK , both respectable brands as far as I know. Come to
think of it , there was a situation some years ago where for a number of DVDs
from the middle of a spindle (I don't remember which brand) , I had to fill
them up to around 4,100,000,000 bytes (out of 4,700,000,000 maximum) or
verification would fail. That's the most serious problem I've ever
encountered. But then for most DVDs , I don't really get to test them after I
burn them ; I store them and that's it.

> Secondary effects result from the
> choice of dye on the DVD, and of course manner of storage adds a huge
> variable. Storing them in a hot car, in full sunlight, will degrade
> them faster than in a light tight box in a cool basement. Of course
> the 'cool basement' adds the possibility of a dampness factor, which
> can again change the variables.

I put them in wallets. I am slightly concerned that the DVD surface rubs
against the wallet surface as in insert them in the pocket but it's not as if
I put them in and out systematically. For most I just put them in once and
then I know they're there , just in case. The wallets are not stored in any
extreme conditions.

--
A good programmer is someone who looks both ways before crossing a one way
street.
Doug Liner

On Tue, 06 Jun 2023 10:55:47 +0000, Spiros Bousbouras wrote:

> I buy Verbatim or TDK , both respectable brands as far as I know.

I look for any brand that uses Taiyo Yuden dye, etc. Manufacturers change
what they use over time.

I always check read what I have written.

--
Using UNIX since v6 (1975)...

Use the BIG mirror service in the UK:
http://www.mirrorservice.org

On Mon, 5 Jun 2023 23:07:35 +0100
SH <i.love@spam.com> wrote:
> On 04/06/2023 13:14, Spiros Bousbouras wrote:
> > I figured it may be instructive to have a thread about this kind
> > of thing.
> >
> > By policies/practices I mean not just the back-ups themselves but
> > also whether you have any such policies on using your back-ups
> > for recovering from various disaster scenarios and whether you
> > ever got to test those policies for real.
> >
>
> herewith my contribution :-)
>
> I have several computers. Mostly windows boxes but with a few Linux ones
> too.
>
>
> The Win boxes all have a 2nd hard drive to back up to amd also an
> automatic backup to back up to a networked NAS.
>
> The Linux boxes can be set up with a cron job and rum rsync to the 2nd
> internal drive and to the NAS.
>
> I have a directory for each networked device on the NAS.
>
> The NAS use either RAID 6 or RAID 10 ( 4 drives in total and can
> tolerate failure of up to two drives.)
>
> You can use Linux MD or FreeBSD's ZFS or use a hardware based RAID
> controller (cheap as chips such as the adaptec 7805 whcih will support 8
> SATA drives) to do a RAID array.
>
> The adaptec 7805's have onboard 512 MB RAM with a battery backup unit so
> that data in transit is not lost during a power cut due to the BBU
> (battery backup unit). The NAS is aslo supported by a UPS.
>
> you will find backing up to SATA HDDs or SSDs will be faster than DVDs
> and by automating as much of it as possible makes life much easier.

Speed is not really an issue because it's not as if I'm sitting around
waiting for the burning to finish , I do my various activities as normal. So
it's just a matter of entering the DVD in the drive and typing the command.

Automation would be good. I have kind of decided that an operation mode where
every file even somewhat valuable goes on regular back-ups with no conscious
thought which files are more valuable than others and also which files are
potentially replaceable (because I downloaded them from somewhere and the
source is likely to stick around) , is impractical due to the total size of
the files involved. But this is more of a general instinct rather than the
result of precise calculations. Regardless , for substantially more storage
hardware (of a nature similar to what you describe) than what I have at
present , my first use would not be for back-ups but to store movies or music
(from DVDs and audio CDs respectively) for easy access because now there are
occasions where I'm in the mood to watch something but I can't be bothered to
try and find the DVD in some cupboard or drawer or whatever.

> HDDs and SSDs are also much bigger than DVDs, heck a 500 GB SSD will
> hold 100 full DVD's worth.

But present a single point of failure.

--
It's neat how you contain a factory for making more of you
http://xkcd.com/387

Spiros Bousbouras <spibou@gmail.com> wrote:
> I figured it may be instructive to have a thread about this kind
> of thing.
>
> By policies/practices I mean not just the back-ups themselves but
> also whether you have any such policies on using your back-ups
> for recovering from various disaster scenarios and whether you
> ever got to test those policies for real.

I have two categories of files to back up: (1) documents, databases, and Git
repos that are relatively small and frequently changed and (2) media files
that are relatively large and rarely changed when added.

Documents get backed up with Duplicity (https://duplicity.gitlab.io/) to
Linode object storage. Full backups are sent once a month, with incremental
backups sent daily in between. These backups currently use only about 16
GB. I've not yet had to recover files from this system, but I've tested it
once or twice and gotten back out what I put in…

Media get backed up to BD-R. I knocked together some scripts to manage
this:

https://gitlab.com/salfter/bdarchiver

A database that tracks what files are on what disc is maintained by the
scripts and written to each disc. I write 20 GB to single-layer BD-Rs, and
use dvdisaster (https://sourceforge.net/projects/dvdisaster/) to add
error-recovery information to the remaining free space. Discs are written
as I accumulate enough files to fill them, and are stored in (at this time)
three binders on a bookshelf at work. I've used this system for over a
decade, and had occasion once to recover the whole backup once when a second
drive failed in a RAID-5 setup. Full recovery took a couple or three weeks
of feeding discs to my computer to repopulate the server. I've also
occasionally needed to recover individual files that had somehow
disappeared. The database makes locating the right disc trivial.

--
_/_
/ v \ Scott Alfter (remove the obvious to send mail)
(IIGS( https://alfter.us/ Top-posting!
\_^_/ >What's the most annoying thing on Usenet?

Rich <rich@example.invalid> wrote:
> Spiros Bousbouras <spibou@gmail.com> wrote:
>> Whether it's more reliable than alternative methods of storage I
>> don't know. Does anyone know of any graphs which show as a function
>> of time the probability of a storage medium going "bad" ?
>
> For DVD's, that highly depends upon the quality of the disks when they
> were manufactured. At one point some 'el cheapo' DVD-R's would fail
> only a few years after being burned. Secondary effects result from the
> choice of dye on the DVD, and of course manner of storage adds a huge
> variable. Storing them in a hot car, in full sunlight, will degrade
> them faster than in a light tight box in a cool basement. Of course
> the 'cool basement' adds the possibility of a dampness factor, which
> can again change the variables.

Recordable Blu-ray improves on DVD-R and CD-R by using inorganic
composite materials instead of organic dyes as the writable material. My
oldest backup disc is over 11 years old, and it's just as readable today as
when I burned it.

(Caveat: this only applies to standard (HTL) BD-R media. In an attempt at
producing cheaper blank media, organic-dye technology was adapted to BD-R in
what are known as LTH BD-Rs. You ended up not really saving any money (I've
always been able to find HTL media for less), and they suffer the same
reliability issues that can affect CD-R and DVD-R. LTH media also don't
work in all Blu-ray drives. Fortunately, LTH BD-Rs are usually identified
as such, so they're fairly easy to avoid.)

--
_/_
/ v \ Scott Alfter (remove the obvious to send mail)
(IIGS( https://alfter.us/ Top-posting!
\_^_/ >What's the most annoying thing on Usenet?

>> you will find backing up to SATA HDDs or SSDs will be faster than DVDs
>> and by automating as much of it as possible makes life much easier.
>
> Speed is not really an issue because it's not as if I'm sitting around
> waiting for the burning to finish , I do my various activities as normal. So > it's just a matter of entering the DVD in the drive and typing the
command.

fair enough :-)

>> HDDs and SSDs are also much bigger than DVDs, heck a 500 GB SSD will
>> hold 100 full DVD's worth.
>
> But present a single point of failure.
>

Agreed with, and I found this out the hard way when I had 4 IDE drives
on an adaptec 2400A controller..... the controller failed.... so could
not access any of the 4 drives as they had been configured as a single
RAID 5 array.

I now use an Adaptec 7805 which allows me to connect up to 8 SATA drives.

I can configure these to be as 8 single drives and formatted to ext4 or
ext5.

So back up to SDA, then rsync SDA to SDB, ditto SDB to SDC, ditto SDC to
SDD, ditto SDD to SDE, ditto SDE to SDF, ditto SDF to SDG and finally
SDG to SDH.

If all 8 drives are 500 GB say, you;s be able to store 100 DVDs on each
drive, so you'd have 8 copies of the 100 DVDs. So you;d have to be
really unlucky to have all 8 drives fail simultaneously

I keep a few USB to SATA caddies spare, so if the single controller
fails, I can choose any one of those 8 HDDs and stick in a caddy and
then use my fave linux Live distro or spare Linux box to get at my
backups if I am not able to source a replacement controller.

I have had up to 3 NASes, all syncing to each other so with a 2TB drive
in each NAS, you'd be able to back up 400 DVD's thrice over :-)

the OneDrive thing is a good off-site back up as my last line of defence.