On 30/07/2023 16:25, Rich wrote:
> In comp.os.linux.misc The Natural Philosopher <tnp@invalid.invalid> wrote:
>> On 30/07/2023 10:24, Theo wrote:
>>> The directory structure may be created by the package manager on
>>> install, but the files are created on the fly. In other words I
>>> might have /var/log/foo/error.log - foo is a directory that needs to
>>> exist, but if I delete error.log it'll be created next time there's
>>> something to go in there. This is commonly used by 'logrotate',
>>> which moves old files and compresses them, allowing new files to be
>>> created for future log entries.
>>>
>>> So all you really need to do is reproduce the directory structure:
>>>
>>> find /var/log -type d -exec mkdir -p "/tmp/{}" \;
>>>
>>> which reproduces /var/log/foo/bar/ to /tmp/var/log/foo/bar/
>>>
>>> Theo
>>
>> Now that is interesting, and if I leave the old /var/log there, and
>> mount a ramdisk over it, provided that the ramdisk is *immediately*
>> populated and directories created, then all should work?
>
> You /may/ need to trigger the logging deamon to redo the files. With
> traditional syslog that would be send the log deamon's a sighub. With
> systemd I have no idea how to do this as none of my linux systems
> contain systemd.
>
A cursory glance at systemd reveals a huge potential flaw in it, or at
least that is my understanding.

There is no order of execution of systemd scripts. They all get executed
in parallel. Instead there are dependencies, which I THINK are
encapsulated in the scripts as 'depends upon', not 'is depended on by'.
So a standalone systemd script wont do the job. Instead one would
probably have to find the mount script and add a patch to that.

>> I guess I have to learn how to add a script to systemd :-(
>
> True. Hopefully there is a way.

--
"Strange as it seems, no amount of learning can cure stupidity, and
higher education positively fortifies it."

- Stephen Vizinczey

On 30/07/2023 17:40, David W. Hodgins wrote:
> On Sun, 30 Jul 2023 10:45:54 -0400, The Natural Philosopher
> <tnp@invalid.invalid> wrote:
>> Now that is interesting, and if I leave the old /var/log there,
>> and mount a ramdisk over it, provided that the ramdisk is
>> *immediately* populated and directories created, then all should
>> work?
>>
>> I guess I have to learn how to add a script to systemd :-(
>
> I would add an entry to fstab to handle the mount ... mylog /var/log
> tmpfs defaults,user,size=1G 1 1
>
Pi Zero only has 512M RAM!

> Copy the directory structure from /var/log somewhere else such as
> /my/log.
>
> copy /usr/lib/systemd/system/systemd-remount-fs.service to
> /etc/systemd/system/, (create the dir if it doesn't exist already),
> then modify the /etc service to add a second ExecStart to run a
> script that copies the directory structure from /my log to /var/log.
>

Ah. That is interesting. Patching the mount service *itself*. That
sounds like a plan, unless any updates wipe out the config...but who
bothers to update an embedded system that Just Works?

> Run "dracut -f" and reboot.
>
Can you explain why that would be necessary?

> If you later install something that creates a directory in /var/log,
> just create a directory with that name, owner, group, and
> permissions in /my/log.
>
> Make a backup on a second sd card first though, just in case! :-)
>
> Regards, Dave Hodgins

--
No Apple devices were knowingly used in the preparation of this post.

On 30/07/2023 22:13, David W. Hodgins wrote:
> On Sun, 30 Jul 2023 16:39:26 -0400, Carlos E.R.
> <robin_listas@es.invalid> wrote:
>> A systemd system might not have syslog, but journal, and non
>> permanent (not on disk). It may use temporary files, though.
>
> With systemd, systemd-journald collects messages during early boot,
> in ram. The in ram copy is flushed to persistent storage shortly
> after the / filesystem is remounted rw over the ro / filesystem from
> the initrd.
>
Ahaha. First thing that systemd does that actually makes sense.

> That's why I recommend adding an extra ExecStart to
> systemd-remount-fs.service to set up the directory structure in a
> tmpfs mount of /var/log right after the / filesystem is ready to
> accept writes. That way it's done before the journal or, if present,
> rsyslogd starts writing to /var/log.
>
Yes.
The advantage of that is that no info is lost writing to the on-disk
/var/log - the boot stuff will all be flushed to the ramdisk once it is
available.

Well it will be several weeks before I am ready to do any practical
experimentation. I try to ask first, implement later. Soi as not to
waste time on finding out what others already know, by experiment

> Regards, Dave Hodgins

Many thinks. I think your solutions is the one to use. It surgically
splices in the minimum of configuration required to do the job. It OUGHT
to be an optional part of Raspios really...

--
Any fool can believe in principles - and most of them do!

On 31/07/2023 02:54, 23k.302 wrote:
> On 7/29/23 2:42 PM, David W. Hodgins wrote:
>> On 7/30/23 10:45, The Natural Philosopher wrote:
>>> On 30/07/2023 10:24, Theo wrote:
>>>> So all you really need to do is reproduce the directory structure:
>>>>
>>>> find/var/log -type d -exec mkdir -p "/tmp/{}" \;
>>>>
>>>> which reproduces/var/log/foo/bar/  to/tmp/var/log/foo/bar/
>>>>
>>>> Theo
>>> Now that is interesting, and if I leave the old /var/log there, and
>>> mount a ramdisk over it, provided that the ramdisk is*immediately*
>>> populated and directories created, then all should work?
>> Any process that has a log file open before you mount the ramdisk won't
>> automatically start using it.  The process will still have the open file
>> handle to the original, disk-based file instead.  So you'll either need
>> to be sure to mount the ramdisk sufficiently early in the boot sequence
>> to ensure that no daemons have opened their logfiles yet, or you'll need
>> to signal those daemons reopen their logs.  Most do that in response to
>> SIGHUP.  Logrotate often uses that mechanism after rotating logs, so its
>> configuration can help you figure out what needs to be done after
>> mounting the ramdisk if programs are still logging to disk.
>
>
>   Hmm ... MIGHT be able to make that work with minimal
>   pain using systemd. It has 'depends on','run after' and
>   such params. He could init the ramdisk REALLY early in a
>   quasi-structured/quasi-safe environment.
>
>   Not everyone likes systemd - but it DOES have its uses.
>
>   Another trick is to just make the ramdisk whenever and
>   then make /var/log into a symlink/mountpoint. Yes, there'd
>   be some early stuff still in the real /var/log, but maybe
>   that's not important for him.
>
>   However I'd rec ramdisks for where they're more useful.
>   I have an app that does a lot of image manipulation
>   before the results are shown via PHP/Apache on a Pi.
>   I don't want any of that to be reading/writing to the
>   SD card - too slow AND burns it out - but I did want
>   access to be "normal", like with any 'real' drive. A
>   small ramdisk was by far the best/easiest solution.

I use them in this and my other pi-zero context as fundamentally simple
interprocess communicators. So in my Hifi Project, a daemon reads the
icecast info off radio streams and writes it to a file in a temporary
file system.

The web server then has access to that to let me know what misc is being
played, via ajax calls from the web servers radio interface.

As long as only one process is writing them, this is a very simple and
effective way to transfer data from one daemon to another.

In the current project which involves driving 4 mains relays using a
daemon that collects various bits of information in order to decide when
to switch them on, it is used to store the relay state, so again ajax
calls from the web server can determine not what *should be on, but what
is *actually* on.

I am sure the semaphores messages and pipes might have done the job, but
simply recreating a file ion a ramdisk every few seconds works fine.

--
"Nature does not give up the winter because people dislike the cold."

― Confucius

The Natural Philosopher <tnp@invalid.invalid> writes:
> There is no order of execution of systemd scripts. They all get
> executed in parallel. Instead there are dependencies, which I THINK
> are encapsulated in the scripts as 'depends upon', not 'is depended on
> by'. So a standalone systemd script wont do the job. Instead one
> would probably have to find the mount script and add a patch to that.

AFAICS all the dependency options have reverses, see the table in
systemd.unit(5).

There are both functional dependencies (e.g. Wants/WantedBy) and
ordering dependencies (e.g. After/Before). If you want to serialize
startup you’ll need both.

--
https://www.greenend.org.uk/rjk/

In comp.os.linux.misc The Natural Philosopher <tnp@invalid.invalid> wrote:
> I think that SD cards (apart from a very few) have no sophisticated
> wear levelling *at all*.

Given their race to the bottom price wise, this is a reasonably safe
assumption.

> If they are *really* cheap they wont do anything at all and there
> will be a 1:1 correlation between physical and logical sectors, in
> which case it doesn't matter how big they are, they will wear out the
> directory and sector allocation sectors without ever using the
> majority of the card.
>
> I am not sure if e.g. linux can map out bad sectors in those areas.
> I suspect not.

If the hardware does not handle bad sector replacement then you'd need
to run a Linux filesystem that does do so on top. For an SD card,
that would likely best be one of the 'flash' filesystems.
https://en.wikipedia.org/wiki/Flash_file_system#Linux_flash_filesystems

As to which might be best for an SD card that might have minimal to no
on card wear leveling I can not say.

At Mon, 31 Jul 2023 13:25:28 -0000 (UTC) Rich <rich@example.invalid> wrote:

>
> In comp.os.linux.misc The Natural Philosopher <tnp@invalid.invalid> wrote:
> > I think that SD cards (apart from a very few) have no sophisticated
> > wear levelling *at all*.
>
> Given their race to the bottom price wise, this is a reasonably safe
> assumption.
>
> > If they are *really* cheap they wont do anything at all and there
> > will be a 1:1 correlation between physical and logical sectors, in
> > which case it doesn't matter how big they are, they will wear out the
> > directory and sector allocation sectors without ever using the
> > majority of the card.
> >
> > I am not sure if e.g. linux can map out bad sectors in those areas.
> > I suspect not.
>
> If the hardware does not handle bad sector replacement then you'd need
> to run a Linux filesystem that does do so on top. For an SD card,
> that would likely best be one of the 'flash' filesystems.
> https://en.wikipedia.org/wiki/Flash_file_system#Linux_flash_filesystems
>
> As to which might be best for an SD card that might have minimal to no
> on card wear leveling I can not say.

I've been using SanDisk Class 10 uSD cards in various PIs, with the machines
running continiously for years without errors. Some of these machines have
survived multiple power failures without problems. Several of these machines
I use as "build boxes". I have been using good old Ext4 as the FS.

>
>
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

On 31/07/2023 16:11, Robert Heller wrote:
> At Mon, 31 Jul 2023 13:25:28 -0000 (UTC) Rich <rich@example.invalid> wrote:
>
>>
>> In comp.os.linux.misc The Natural Philosopher <tnp@invalid.invalid> wrote:
>>> I think that SD cards (apart from a very few) have no sophisticated
>>> wear levelling *at all*.
>>
>> Given their race to the bottom price wise, this is a reasonably safe
>> assumption.
>>
>>> If they are *really* cheap they wont do anything at all and there
>>> will be a 1:1 correlation between physical and logical sectors, in
>>> which case it doesn't matter how big they are, they will wear out the
>>> directory and sector allocation sectors without ever using the
>>> majority of the card.
>>>
>>> I am not sure if e.g. linux can map out bad sectors in those areas.
>>> I suspect not.
>>
>> If the hardware does not handle bad sector replacement then you'd need
>> to run a Linux filesystem that does do so on top. For an SD card,
>> that would likely best be one of the 'flash' filesystems.
>> https://en.wikipedia.org/wiki/Flash_file_system#Linux_flash_filesystems
>>
>> As to which might be best for an SD card that might have minimal to no
>> on card wear leveling I can not say.
>
> I've been using SanDisk Class 10 uSD cards in various PIs, with the machines
> running continiously for years without errors. Some of these machines have
> survived multiple power failures without problems. Several of these machines
> I use as "build boxes". I have been using good old Ext4 as the FS.
>
Indeed. People DO have failures, but I can find no really clear data on
*why* they fail in a Pi situation

>>
>>
>>
>

--
Renewable energy: Expensive solutions that don't work to a problem that
doesn't exist instituted by self legalising protection rackets that
don't protect, masquerading as public servants who don't serve the public.

At Mon, 31 Jul 2023 19:59:48 +0100 The Natural Philosopher <tnp@invalid.invalid> wrote:

>
> On 31/07/2023 16:11, Robert Heller wrote:
> > At Mon, 31 Jul 2023 13:25:28 -0000 (UTC) Rich <rich@example.invalid> wrote:
> >
> >>
> >> In comp.os.linux.misc The Natural Philosopher <tnp@invalid.invalid> wrote:
> >>> I think that SD cards (apart from a very few) have no sophisticated
> >>> wear levelling *at all*.
> >>
> >> Given their race to the bottom price wise, this is a reasonably safe
> >> assumption.
> >>
> >>> If they are *really* cheap they wont do anything at all and there
> >>> will be a 1:1 correlation between physical and logical sectors, in
> >>> which case it doesn't matter how big they are, they will wear out the
> >>> directory and sector allocation sectors without ever using the
> >>> majority of the card.
> >>>
> >>> I am not sure if e.g. linux can map out bad sectors in those areas.
> >>> I suspect not.
> >>
> >> If the hardware does not handle bad sector replacement then you'd need
> >> to run a Linux filesystem that does do so on top. For an SD card,
> >> that would likely best be one of the 'flash' filesystems.
> >> https://en.wikipedia.org/wiki/Flash_file_system#Linux_flash_filesystems
> >>
> >> As to which might be best for an SD card that might have minimal to no
> >> on card wear leveling I can not say.
> >
> > I've been using SanDisk Class 10 uSD cards in various PIs, with the machines
> > running continiously for years without errors. Some of these machines have
> > survived multiple power failures without problems. Several of these machines
> > I use as "build boxes". I have been using good old Ext4 as the FS.
> >
> Indeed. People DO have failures, but I can find no really clear data on
> *why* they fail in a Pi situation

Ultra "cheap" no-name brand uSD cards?

>
> >>
> >>
> >>
> >
>

--
Robert Heller -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software -- Custom Software Services
http://www.deepsoft.com/ -- Linux Administration Services
heller@deepsoft.com -- Webhosting Services

On 29/07/2023 21:14, David W. Hodgins wrote:
> On Sat, 29 Jul 2023 14:54:25 -0400, The Natural Philosopher
> <tnp@invalid.invalid> wrote:
>
>> On 29/07/2023 19:42, David W. Hodgins wrote:
>>> On Sat, 29 Jul 2023 13:34:45 -0400, The Natural Philosopher
>>> <tnp@invalid.invalid> wrote:
>>>> Embedded Raspberry Pi Zero W. Minimal logs generated but I
>>>> only use logs for debugging and why wear the SD card?
>>>
>>>> My question is simple. If I make a 50MByte or so RAMDISK and
>>>> mount it on /var/log, will the logging daemons recreate all the
>>>> files and directories they need at boot time?

> It can be done using scripts that hook into systemd, but is not easy,
> and may require changes to scripts with any package install that
> uses /var/log, and also risks losing parts of the logs in the case of
> an unclean shutdown, which is typically when you are going to want
> the logs.

On Bullseye and later the largest amount of logging activity to disc is
bloody systemd's journald, which by default will grow /var/log/journal/*
to 2GB, taking up valuable space on SD cards and severely reducing their
write life.

Two things you can do; first as fucking Peottering always knows best,
there is no way of turning off journald and just using the god given
rsyslog, but you can stop journald logging to disc and just keep it in
RAM instead by adding the following to /etc/systemd/journald.conf

Storage=volatile
RuntimeMaxUse=32M
RuntimeMaxFileSize=32M
ForwardToConsole=no
ForwardToWall=no

Secondly stop systemd spamming /var/log/syslog with all sorts of crap
you have no interest in. Create a file called
/etc/rsyslog.d/drop.systemd.conf containing

# Drop messages from f***ing systemd
:programname,startswith,"systemd" stop

If you do that, you'll be left with far less logging activity to disk,
so you may not have to bother moving it to RAM. If you still want no
logging at all to disc, you can disable rsyslog entirely and just use
journald restricted RAM. But that way madness lies.

---druck

On 29/07/2023 18:34, The Natural Philosopher wrote:
> Embedded Raspberry Pi Zero W. Minimal logs generated but  I only use
> logs for debugging and why wear the  SD card?
>
> My question is simple. If I make a 50MByte or so RAMDISK and mount it on
> /var/log, will the logging daemons recreate all the files and
> directories they need at boot time?
>
>

Works fine for me. Pi W running Raspbian (Debian 11 version)

Cron job reboots it every morning so I don't worry about ramdisk filling up.

$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 7.0G 2.1G 4.7G 31% /
devtmpfs 183M 0 183M 0% /dev
tmpfs 215M 0 215M 0% /dev/shm
tmpfs 86M 2.7M 84M 4% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 100M 8.0K 100M 1% /tmp
tmpfs 100M 176K 100M 1% /var/log
tmpfs 30M 0 30M 0% /var/spool/mqueue
tmpfs 30M 0 30M 0% /var/tmp
/dev/mmcblk0p1 253M 51M 202M 20% /boot
tmpfs 43M 0 43M 0% /run/user/1001

Logging works and everything needed seems to be created on boot.

$ ls /var/log -la
total 180
drwxr-xr-x 3 root root 260 Aug 1 05:33 .
drwxr-xr-x 11 root root 4096 Jan 11 2021 ..
-rw-r----- 1 root adm 23843 Aug 1 08:06 auth.log
-rw-rw---- 1 root utmp 3072 Aug 1 08:03 btmp
-rw-r----- 1 root adm 21318 Aug 1 08:00 daemon.log
-rw-r----- 1 root adm 1032 Aug 1 05:33 debug
-rw------- 1 root root 2559 Aug 1 08:03 fail2ban.log
-rw-r----- 1 root adm 26940 Aug 1 05:33 kern.log
-rw-rw-r-- 1 root utmp 292584 Aug 1 08:00 lastlog
-rw-r----- 1 root adm 26239 Aug 1 05:33 messages
drwx------ 2 root root 40 Aug 1 05:33 private
-rw-r----- 1 root adm 50106 Aug 1 08:00 syslog
-rw-rw-r-- 1 root utmp 1920 Aug 1 08:00 wtmp

It's been running like this on the same SDCARD for 2+ years.

On 31/07/2023 21:27, Robert Heller wrote:
> At Mon, 31 Jul 2023 19:59:48 +0100 The Natural Philosopher <tnp@invalid.invalid> wrote:
>
>> Indeed. People DO have failures, but I can find no really clear data on
>> *why* they fail in a Pi situation
>
> Ultra "cheap" no-name brand uSD cards?
>
Possibly, again, no direct evidence, sadly.

Its another of those details that Pi users probably need to know, but no
one else cares about.,

There are SD cards for data transfer - video capture - and sd cards for
multiple use like cameras.

no one sells one for a lot of micro writes...

--
“it should be clear by now to everyone that activist environmentalism
(or environmental activism) is becoming a general ideology about humans,
about their freedom, about the relationship between the individual and
the state, and about the manipulation of people under the guise of a
'noble' idea. It is not an honest pursuit of 'sustainable development,'
a matter of elementary environmental protection, or a search for
rational mechanisms designed to achieve a healthy environment. Yet
things do occur that make you shake your head and remind yourself that
you live neither in Joseph Stalin’s Communist era, nor in the Orwellian
utopia of 1984.”

Vaclav Klaus

On 31/07/2023 21:36, druck wrote:
> On 29/07/2023 21:14, David W. Hodgins wrote:
>> On Sat, 29 Jul 2023 14:54:25 -0400, The Natural Philosopher
>> <tnp@invalid.invalid> wrote:
>>
>>> On 29/07/2023 19:42, David W. Hodgins wrote:
>>>> On Sat, 29 Jul 2023 13:34:45 -0400, The Natural Philosopher
>>>> <tnp@invalid.invalid> wrote:
>>>>> Embedded Raspberry Pi Zero W. Minimal logs generated but  I
>>>>> only use logs for debugging and why wear the  SD card?
>>>>
>>>>> My question is simple. If I make a 50MByte or so RAMDISK and
>>>>> mount it on /var/log, will the logging daemons recreate all the
>>>>> files and directories they need at boot time?
>
>> It can be done using scripts that hook into systemd, but is not easy,
>>  and may require changes to scripts with any package install that
>> uses /var/log, and also risks losing parts of the logs in the case of
>> an unclean shutdown, which is typically when you are going to want
>> the logs.
>
> On Bullseye and later the largest amount of logging activity to disc is
> bloody systemd's journald, which by default will grow /var/log/journal/*
> to 2GB, taking up valuable space on SD cards and severely reducing their
> write life.
>
> Two things you can do; first as fucking Peottering always knows best,

I am already liking your world view. As I said 'designed to make life
easier for a minicomputer system admin'
Not really what we want in a Pi.

> there is no way of turning off journald and just using the god given
> rsyslog, but you can stop journald logging to disc and just keep it in
> RAM instead by adding the following to /etc/systemd/journald.conf
>
> Storage=volatile
> RuntimeMaxUse=32M
> RuntimeMaxFileSize=32M
> ForwardToConsole=no
> ForwardToWall=no
>
Ah. I had already limited its size.
Where in RAM does it keep it, or is that only known the the Great
Poettering, who has provided some buggy tool to access it?

In normal use I would have access and hopefully not error logs from
apache, but only minimal. And perhaps the odd error message being mailed
out via exim.

Mostly the machine will be, apart from a daemon waking up every few
seconds and seeing what's what, essentially idle

> Secondly stop systemd spamming /var/log/syslog with all sorts of crap
> you have no interest in. Create a file called
> /etc/rsyslog.d/drop.systemd.conf containing
>
> # Drop messages from f***ing systemd
> :programname,startswith,"systemd" stop
>
is ryslog still running? Oh, yes it is.

I think stage one is to create and mount something like /var/ramlog and
see how much standard logging can be moved there by patching
rsyslog.conf and the logrotate files.

apache2 can be moved there too. That's defined in /etc/apache2/envvars

The only hardwired one I can see so far is exim4, but I can cope with that.
Certainly should be able to push a huge amount off the SD

> If you do that, you'll be left with far less logging activity to disk,
> so you may not have to bother moving it to RAM. If you still want no
> logging at all to disc, you can disable rsyslog entirely and just use
> journald restricted RAM. But that way madness lies.
>
Logging is phenomenally useful for debugging a *development* project,
but there is absolutely no need for persistence

So a ram disk plus limiting what goes on it, is handy.
....a few moments later...
successfuly got systemd journal to cease to exist on disk, and moved
nearly everything to a ram log.
As the man falling past the 13th floor yelled: 'so far. so good!'

--
Religion is regarded by the common people as true, by the wise as
foolish, and by the rulers as useful.

(Seneca the Younger, 65 AD)

On Tue, 1 Aug 2023 11:03:45 +0100
The Natural Philosopher <tnp@invalid.invalid> wrote:

> I think stage one is to create and mount something like /var/ramlog and
> see how much standard logging can be moved there by patching
> rsyslog.conf and the logrotate files.

Why don't you just use overlayfs to overlay a ramdisk on /var ?

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/
Host: Beautiful Theory meet Inconvenient Fact
Obit: Beautiful Theory died today of factual inconsistency

On 01/08/2023 09:58, The Natural Philosopher wrote:
> Its another of those details that Pi users probably need to know, but no
> one else cares about.,
>
> There are SD cards for data transfer - video capture - and sd cards for
> multiple use like cameras.
>
> no one sells one for a lot of micro writes...

There are "Application Performance Class" cards now available which
focus on I/O operations per second instead of raw streaming read/write
speeds. They are identified as class A1 and A2. Sandisk seems to be the
main supplier.

One would hope that these cards would have wear levelling set up to
handle lots of small I/O operations but I suspect that's not part of the
spec...

On Tue, 1 Aug 2023 14:25:33 +0100
Dave <dave@cyw.uklinux.net> wrote:

> One would hope that these cards would have wear levelling set up to
> handle lots of small I/O operations but I suspect that's not part of the
> spec...

SD cards have had wear levelling for about a decade now, I doubt
there are many without it.

--
Steve O'Hara-Smith
Odds and Ends at http://www.sohara.org/
Host: Beautiful Theory meet Inconvenient Fact
Obit: Beautiful Theory died today of factual inconsistency

On 7/31/23 16:36, druck wrote:
>
> On Bullseye and later the largest amount of logging activity to disc is
> bloody systemd's journald, which by default will grow /var/log/journal/*
> to 2GB, taking up valuable space on SD cards and severely reducing their
> write life.
>
> Two things you can do; first as fucking Peottering always knows best,
> there is no way of turning off journald and just using the god given
> rsyslog, but you can stop journald logging to disc and just keep it in
> RAM instead by adding the following to /etc/systemd/journald.conf
>
> Storage=volatile
> RuntimeMaxUse=32M
> RuntimeMaxFileSize=32M
> ForwardToConsole=no
> ForwardToWall=no
>
> Secondly stop systemd spamming /var/log/syslog with all sorts of crap
> you have no interest in. Create a file called
> /etc/rsyslog.d/drop.systemd.conf containing
>
> # Drop messages from f***ing systemd
> :programname,startswith,"systemd" stop

Thanks for posting that!

I'm not the OP, but I still found that info immensely useful for a
totally different reason. And I'm probably not the only one who doesn't
want to waste a day sifting through the systemd and journalctl docs to
figure it out for myself.

On Tue, 1 Aug 2023 14:25:33 +0100, Dave <dave@cyw.uklinux.net> declaimed
the following:

>There are "Application Performance Class" cards now available which
>focus on I/O operations per second instead of raw streaming read/write
>speeds. They are identified as class A1 and A2. Sandisk seems to be the
>main supplier.
>
>One would hope that these cards would have wear levelling set up to
>handle lots of small I/O operations but I suspect that's not part of the
>spec...

I'd expect the main feature to be: how many allocation units can be
kept open at one time.

Class 10 cards are rated for streaming to a newly formatted card -- and
with FAT file systems, they only need to keep two allocation units open.
The FAT and the streamed file, and cheaper cards only allow for that.
Opening a second file for I/O requires flushing the current allocation unit
to the actual flash rather than RAM-type buffer, finding a free allocation
unit (erasing it to all 1s), loading it (copying any data from the old
allocation unit if the file is being /reopened/ before marking that old one
free).

Class 2/4/6 were rated for still image operations on cards with
deletions. They often support 4-6 parallel open allocation units.

On a journal ling file system, having lots of open allocation units is
desirable.

On 01/08/2023 11:03, The Natural Philosopher wrote:
> Where in RAM does it keep it, or is that only known the the Great
> Poettering, who has provided some buggy tool to access it?

It's stored in /run/log/journal (/run is tmpfs in RAM)

Despite the configuration specifying a maximum journal size of 32M and
even 512MB Pi's having a 64MB /run, if you do a df /run and see it at
100% it's down to bloody journald, and you have to use Peottering,s
buggy tool to manually clear out the crap.

journal --vacuum-files=1

> successfuly got systemd journal to cease to exist on disk, and moved
> nearly everything to a ram log.
> As the man falling past the 13th floor yelled: 'so far. so good!'

If you aiming for uptimes of 80+ days or more, then bung that command in
a weekly or monthly cron job, as things can get a bit temperamental if
/run is full.

---druck

On 7/29/23 19:45, Robert Heller wrote:
> This is in fact the default behaviour for Armbian on my Banana Pi M64.
>

+1, :-) I was just thinking, I spent ages undoing this, in order to
find out why my system was crashing. But I'm running Armbian.

TNP, FFS, SD cards cost buttons. Pack of two 32GB for less than a
tenner. If you are worried about wear, clone a new one every year, or
whatever.

On 01/08/2023 08:08, mm0fmf wrote:
> Cron job reboots it every morning so I don't worry about ramdisk filling
> up.
>
> It's been running like this on the same SDCARD for 2+ years.

Lucky. I had an older Pi which I had set to reboot every day, and I had
lots of occasions when it didn't come back up by itself. Often just
needed power cycling, occasionally filesystem corruption needed fixing.
Turned out it was a dodgy WiFi dongle, and after replacing that I was
able to abandon the daily reboots, and the reliability was rock solid
after that.

My suggestion is to look at the logging levels of various processes, and
ensure everything has a suitable log rotate configuration, so the size
of your logging directory is kept pretty static over time without having
to reboot. Particularly as booting does lots of logging, and once that
is rotated out, the size may go down slightly.

I do nightly differential backups across my fleet of Pi's and the first
thing I look for in the monitoring graphs is the size of data
transferred, any significant increase is invariably due to some problem
causing a log to grow massively. A quick intervention has often avoided
considerable downtime.

---druck

On 01/08/2023 21:48, druck wrote:
> On 01/08/2023 08:08, mm0fmf wrote:
>> Cron job reboots it every morning so I don't worry about ramdisk
>> filling up.
>>
>> It's been running like this on the same SDCARD for 2+ years.
>
[snip]

> My suggestion is to look at the logging levels of various processes, and
> ensure everything has a suitable log rotate configuration, so the size
> of your logging directory is kept pretty static over time without having
> to reboot. Particularly as booting does lots of logging, and once that
> is rotated out, the size may go down slightly.
>

I think you are right. I really should set it up better than it is. It's
a classic "round tuit" problem as it works acceptably as is. I suppose
the problem with Pi's and the like is they feel disposable along with
their SD Cards and once you have something working the urge to do the
job properly wanes. And you can always just stick the backup SD Card in
and reboot!

On 7/31/23 5:32 AM, The Natural Philosopher wrote:
> On 31/07/2023 02:54, 23k.302 wrote:
>> On 7/29/23 2:42 PM, David W. Hodgins wrote:
>>> On 7/30/23 10:45, The Natural Philosopher wrote:
>>>> On 30/07/2023 10:24, Theo wrote:
>>>>> So all you really need to do is reproduce the directory structure:
>>>>>
>>>>> find/var/log -type d -exec mkdir -p "/tmp/{}" \;
>>>>>
>>>>> which reproduces/var/log/foo/bar/  to/tmp/var/log/foo/bar/
>>>>>
>>>>> Theo
>>>> Now that is interesting, and if I leave the old /var/log there, and
>>>> mount a ramdisk over it, provided that the ramdisk is*immediately*
>>>> populated and directories created, then all should work?
>>> Any process that has a log file open before you mount the ramdisk won't
>>> automatically start using it.  The process will still have the open file
>>> handle to the original, disk-based file instead.  So you'll either need
>>> to be sure to mount the ramdisk sufficiently early in the boot sequence
>>> to ensure that no daemons have opened their logfiles yet, or you'll need
>>> to signal those daemons reopen their logs.  Most do that in response to
>>> SIGHUP.  Logrotate often uses that mechanism after rotating logs, so its
>>> configuration can help you figure out what needs to be done after
>>> mounting the ramdisk if programs are still logging to disk.
>>
>>
>>    Hmm ... MIGHT be able to make that work with minimal
>>    pain using systemd. It has 'depends on','run after' and
>>    such params. He could init the ramdisk REALLY early in a
>>    quasi-structured/quasi-safe environment.
>>
>>    Not everyone likes systemd - but it DOES have its uses.
>>
>>    Another trick is to just make the ramdisk whenever and
>>    then make /var/log into a symlink/mountpoint. Yes, there'd
>>    be some early stuff still in the real /var/log, but maybe
>>    that's not important for him.
>>
>>    However I'd rec ramdisks for where they're more useful.
>>    I have an app that does a lot of image manipulation
>>    before the results are shown via PHP/Apache on a Pi.
>>    I don't want any of that to be reading/writing to the
>>    SD card - too slow AND burns it out - but I did want
>>    access to be "normal", like with any 'real' drive. A
>>    small ramdisk was by far the best/easiest solution.
>
> I use them in this and my other pi-zero  context as fundamentally simple
> interprocess communicators. So in my Hifi Project, a daemon reads the
> icecast info off radio streams and writes it to a file in a temporary
> file system.
>
> The web server then has access to that to let me know what misc is being
> played, via ajax calls from the web servers radio interface.
>
> As long as only one process is writing them, this is a very simple and
> effective way to transfer data from one daemon to another.

Agreed. If you CAN keep it simple, DO keep it simple.
Remember YOU might have to debug/enhance it a few
years LATER :-)

Things like Pi's are special cases - not blazing fast,
not a lot of RAM, run off SD cards/eMMC that have to be
protected from re-write fatigue. If the need for space
isn't excessive, and simplicity is worthwhile (almost
always) then a RAMdisk is often your best solution.

Because of the filesystem overhead, RAMdisks just MIGHT
not be fast enough ... then you have to go to "less simple"
approaches alas. To each app, its own.

> In the current project which involves driving 4 mains relays using a
> daemon that collects various bits of information in order to decide when
> to switch them on, it is used to store the relay state, so again ajax
> calls from the web server can determine not what *should be on, but what
> is *actually* on.

Cool stuff

> I am sure the semaphores messages and pipes might have done the job, but
> simply recreating  a file ion a ramdisk every few seconds works fine.

"Pipes" are a kind of cheat - really just "invisible" R/W files.
I used them kinda extensively on some forking servers I wrote a
couple of years ago in Python and 'C' to make data gathered by
new forks accessible to the mother process. They WORK just fine
and are commonly used and I *think* a tad faster than a RAMdisk,
but still ...

I'm fond of such 'servers' - even cobbled together a good
"pre-forked" one - supposedly the highest-capacity/speed -
for both TCP and UDP - but never had a good high-volume
reason to use them. The best was almost like a 'chat' app,
non-sync bi-directional where the queen server process could
initiate, even push, tasks and messages to the clients. Fun !

Hmm ... now apparently some possible servants of Xi here
are calling me a "troll" because I repeated a few things
(said by HS/NSA/CIA) they didn't WANT to hear about
CCP sabotage-ware infiltrating almost everything in
our important infrastructure/military systems and
'devices' (which are almost entirely Linux/Unix-based).
One day soon they ARE gonna get a big wake-up call ... but
meanwhile denial means they won't DO anything about it :-)

ANYway, /var/log CAN be moved to a RAMdisk if you want.
Not 100% sure WHY you'd want to, but it CAN. If a few
very early logs get 'lost' as you re-direct /var/log
then that MIGHT not be all so important. If you want
it all on RAMdisk then you don't CARE if it all
vanishes on reboot. I very rarely look in /var/log
anyhow so ....

On 01/08/2023 14:25, Dave wrote:
> On 01/08/2023 09:58, The Natural Philosopher wrote:
>> Its another of those details that Pi users probably need to know, but
>> no one else cares about.,
>>
>> There are SD cards for data transfer - video capture - and sd cards
>> for multiple use like cameras.
>>
>> no one sells one for a lot of micro writes...
>
> There are "Application Performance Class" cards now available which
> focus on I/O operations per second instead of raw streaming read/write
> speeds. They are identified as class A1 and A2. Sandisk seems to be the
> main supplier.
>
> One would hope that these cards would have wear levelling set up to
> handle lots of small I/O operations but I suspect that's not part of the
> spec...
>
Some info I gleaned suggested that [some of] the better cards do have
full wear levelling.

--
“It is not the truth of Marxism that explains the willingness of
intellectuals to believe it, but the power that it confers on
intellectuals, in their attempts to control the world. And since...it is
futile to reason someone out of a thing that he was not reasoned into,
we can conclude that Marxism owes its remarkable power to survive every
criticism to the fact that it is not a truth-directed but a
power-directed system of thought.”
Sir Roger Scruton

On 01/08/2023 14:46, Ahem A Rivet's Shot wrote:
> On Tue, 1 Aug 2023 14:25:33 +0100
> Dave <dave@cyw.uklinux.net> wrote:
>
>> One would hope that these cards would have wear levelling set up to
>> handle lots of small I/O operations but I suspect that's not part of the
>> spec...
>
> SD cards have had wear levelling for about a decade now, I doubt
> there are many without it.
>
I could find no clear evidence of that.
--
“It is not the truth of Marxism that explains the willingness of
intellectuals to believe it, but the power that it confers on
intellectuals, in their attempts to control the world. And since...it is
futile to reason someone out of a thing that he was not reasoned into,
we can conclude that Marxism owes its remarkable power to survive every
criticism to the fact that it is not a truth-directed but a
power-directed system of thought.”
Sir Roger Scruton