On 2023-09-14, The Natural Philosopher <tnp@invalid.invalid> wrote:
> On 14/09/2023 16:29, Ralf Fassel wrote:
>> * The Natural Philosopher <tnp@invalid.invalid>
>> | One possibility is that it is opening and reading a file at the
>> | precise time another process is writing it...in both cases the read
>> | and write
>> | operations are atomic and done with C code.
>>>
>> | READ
>> | ====
>> | fp=fopen(fullname, "r");
>> | len=fread(filbuf,1,255,fp); // read entire file
>>
>> Check for fp != NULL is missing here in this example code before
>> fread(). If this also in the production version, it might be a problem
>> if the file is not accessible for any reason.
>>
>> R'
> Ralf, I already put that in this morning, re compiled the code and after
> an hour, it crashed again.
>
> The filename is built by scanning a directory so the filename must exist.

Maybe not applicable in this situation, but if something deleted
the file between the time of the scan and the time of the fopen
call, it might/would not exist.

--
Robert Riches
spamtrap42@jacob21819.net
(Yes, that is one of my email addresses.)

"David W. Hodgins" <dwhodgins@nomail.afraid.org> writes:
> The Natural Philosopher <tnp@invalid.invalid> wrote:
>> I am leaning towards possibly a cracked solder joint or board.

Again, I agree with Theo. Reported behavior is not really consistent
with a hardware fault.

> Have you run fsck on the file system since the power loss? Make sure the fstab
> entry does not have a zero in the sixth field for the file system(s) in use.
> If using systemd, run dracut -f after any fstab changes. Then reboot.

Reported behavior is also not consistent with a corrupt filesystem.

--
https://www.greenend.org.uk/rjk/

The Natural Philosopher <tnp@invalid.invalid> writes:
> On 14/09/2023 09:23, Richard Kettlewell wrote:
>> Also:
>> * I would also have a look at the kernel log; if it’s a
>> kernel-generated signal then there’s usually a log message about it.
>>
> Nothing in kern.log after the boot process finishes.

Most likely a bug in your program then.

>> * Run the application under valgrind; depending what the issue is, that
>> will provide a backtrace and perhaps more detailed information. If it
>> is a memory corruption issue then it may identify where the corruption
>> happens, rather than the later point where malloc failed a consistency
>> check (or whatever it is).
>>
>> Using valgrind (and/or compiler sanitizer features) is a good idea
>> even before running into trouble, really.
>
> The strange thing is that it failed once after a minute, then I
> rebooted and it failed after 20 minutes, and its been running several
> days now with no issues at all.
>
> I am not sure valgrind would actually help unless it failed.

It’s extremely good at identifying memory corruption even in cases where
that doesn’t immediately lead to a crash; that’s what it’s for. But if
it doesn’t, you leave it running until the crash happens.

Up to you, of course, whether you use the tools available, or debug with
one hand tied behind your back.

--
https://www.greenend.org.uk/rjk/

* The Natural Philosopher <tnp@invalid.invalid>
| On 14/09/2023 16:29, Ralf Fassel wrote:
| > * The Natural Philosopher <tnp@invalid.invalid>
| > | One possibility is that it is opening and reading a file at the
| > | precise time another process is writing it...in both cases the read
| > | and write
| > | operations are atomic and done with C code.
| >>
| > | READ
| > | ====
| > | fp=fopen(fullname, "r");
| > | len=fread(filbuf,1,255,fp); // read entire file
| > Check for fp != NULL is missing here in this example code before
| > fread(). If this also in the production version, it might be a problem
| > if the file is not accessible for any reason.
| > R'
| Ralf, I already put that in this morning, re compiled the code and
| after an hour, it crashed again.
>
| The filename is built by scanning a directory so the filename must exist.

That assumption does not hold. Since scanning and opening are separated
by a time gap (albeit a 'small' one), there is a non-zero chance that
the file vanished between scan and open.

Further possibilities:
- how is 'filbuf' used after the fread()? If you use it as C-string, make
sure it is 0-terminated (fread() won't do that for you). Maybe use
fgets(3) instead?

| I am leaning towards possibly a cracked solder joint or board.

Well, since the Raspi is cheap, that should be easily checked by simply
using another one. I bet 1 beer that it is *not* a cracked board, since
with that many more processes should run into trouble, not only this
particular one.

R' (.sig not from me .-)
--
echo '[ bottles of beer]sa[ bottle of beer]sb[ take one down, pass it around
]sd[ on the wall]sc[no more]se99snlc[lalnpsnPplalnp1-snpldPln1=ylnpsnPp[]pst
ln0<x]sx[salblnpsnPplblnpsnpldPleplaPlcpq]sylxx' | dc

On 14/09/2023 20:57, David W. Hodgins wrote:
> On Thu, 14 Sep 2023 14:57:36 -0400, The Natural Philosopher
> <tnp@invalid.invalid> wrote:
>
>> On 14/09/2023 19:53, David W. Hodgins wrote:
>>> journalctl -b --no-h|grep fsck
>>
>> Sep 14 14:17:03 systemd[1]: Created slice system-systemd\x2dfsck.slice.
>> Sep 14 14:17:03 systemd[1]: Listening on fsck to fsckd communication
>> Socket.
>> Sep 14 14:17:04 systemd-fsck[109]: e2fsck 1.46.2 (28-Feb-2021)
>> Sep 14 14:17:04 systemd-fsck[109]: rootfs: clean, 51075/932256 files,
>> 460111/3822976 blocks
>> Sep 14 14:17:14 systemd-fsck[178]: fsck.fat 4.2 (2021-01-31)
>> Sep 14 14:17:14 systemd-fsck[178]: There are differences between boot
>> sector and its backup.
>> Sep 14 14:17:14 systemd-fsck[178]: This is mostly harmless. Differences:
>> (offset:original/backup)
>> Sep 14 14:17:14 systemd-fsck[178]:   65:01/00
>> Sep 14 14:17:14 systemd-fsck[178]:   Not automatically fixing this.
>> Sep 14 14:17:14 systemd-fsck[178]: Dirty bit is set. Fs was not properly
>> unmounted and some data may be corrupt.
>> Sep 14 14:17:14 systemd-fsck[178]:  Automatically removing dirty bit.
>> Sep 14 14:17:14 systemd-fsck[178]: *** Filesystem was changed ***
>> Sep 14 14:17:14 systemd-fsck[178]: Writing changes.
>> Sep 14 14:17:14 systemd-fsck[178]: /dev/mmcblk0p1: 330 files,
>> 25815/130554 clusters
>> Sep 14 14:30:12 systemd[1]: systemd-fsckd.service: Succeeded.
>
> If there are any corrupted files, diagnosing any problems they cause
> will be
> difficult. I strongly recommend re-installing.
>
> Regards, Dave Hodgins

If it persists I may do that, but now it is been rock steady for 20 hours.

The actual code has been replaced because I recompiled it anyway, but
the problem persisted after that.

Then I twisted the board a bit, and now it hasn't failed since, No
guarantees of course.

Does anyone else remember Tracy Kidder's 'Soul of a New Machine'* where
they had a wire wrapped backplane on the prototype and a strange
intermittent bug? And the director came in, twisted the backplane and
the bug instantly reappeared?

One of the more curious 'bugs' I encountered was early in my software
career, when code that I wrote suddenly went crazy, in a way in which
the actual software as written could not possibly have caused. And only
on one machine, equipped with a custom video capture card. We removed
the card, but it made no difference.

I then compared the code on the machine with the code as compiled. Two
bytes were FFH

I burned a new floppy and transferred the code again, and the code ran
correctly.

Then we reinstalled the video card. The code ran correctly. Then we
copied over the code again with the video card installed. The code again
was corrupted.

Then the hardware guys looked at the address decide in the video card.
It was a mass of gates one after the other. The total delay was well out
of spec. It dawned on us that what was happening was that the DMA
controller on the floppy was using bus addresses that were being decoded
by the card, and then the IO request came along to access the floppy and
those addresses were still on the bus as far as the sluglike video card
was concerned, so it grabbed the data bus and shoved FFH on it.

Hardware is not perfect. That is the lesson. And chasing software when
its really hardware is a losing game.

Anyway, I have in reserve all the great techniques suggested, but for
now I am playing a wait and see game to see if any pattern emerges. My
experience suggests that the same code running a loop in the same memory
wont crash and burn unless there is a malloc/free mismatch, and that
happens fairly quickly and shows in 'top'.

This kind of weird utterly asynchronous behaviour is often hardware.
And. since I trod on the bloody PCB, I may simple get another one and
test that. It doesn't need to be installed till winter. There is time.
And my PCB design for the relay and PSU module isn't back from China yet...

*https://en.wikipedia.org/wiki/The_Soul_of_a_New_Machine . Definitely
recommended if you haven't read it.

--
"When one man dies it's a tragedy. When thousands die it's statistics."

Josef Stalin

On 14/09/2023 20:40, candycanearter07 wrote:
> On 9/14/23 13:42, The Natural Philosopher wrote:
>> I assumed that the thing would have done its own fsck on every boot
>> anyway...isnt that a debian default?
>
> Pretty sure it's a standard, my arch install has it set.
>>
>> (The sixth fields are 2 and 1 respectively for the file systems)
>>
>>
>> PARTUUID=b8c9fbb7-01  /boot           vfat    defaults          0       2
>> PARTUUID=b8c9fbb7-02  /               ext4    defaults,noatime  0       1
>>
>
> 1 is fsck check for the root partition and 2 is for others, right
>
I looked it up, it merely specifies the order I think, so you are right
in practice.

--
"Corbyn talks about equality, justice, opportunity, health care, peace,
community, compassion, investment, security, housing...."
"What kind of person is not interested in those things?"

"Jeremy Corbyn?"

On 15/09/2023 01:40, Robert Riches wrote:
> On 2023-09-14, The Natural Philosopher <tnp@invalid.invalid> wrote:
>> On 14/09/2023 16:29, Ralf Fassel wrote:
>>> * The Natural Philosopher <tnp@invalid.invalid>
>>> | One possibility is that it is opening and reading a file at the
>>> | precise time another process is writing it...in both cases the read
>>> | and write
>>> | operations are atomic and done with C code.
>>>>
>>> | READ
>>> | ====
>>> | fp=fopen(fullname, "r");
>>> | len=fread(filbuf,1,255,fp); // read entire file
>>>
>>> Check for fp != NULL is missing here in this example code before
>>> fread(). If this also in the production version, it might be a problem
>>> if the file is not accessible for any reason.
>>>
>>> R'
>> Ralf, I already put that in this morning, re compiled the code and after
>> an hour, it crashed again.
>>
>> The filename is built by scanning a directory so the filename must exist.
>
> Maybe not applicable in this situation, but if something deleted
> the file between the time of the scan and the time of the fopen
> call, it might/would not exist.
>

Exactly. That is a possibility, which I have now covered. It made no
difference.

In practice the write code that *replaces* the file is very simple. It is
fopen( "w") immediately followed by
fwrite()

without knowing the exact code involved with the fopen("w"); I cant say
if that actually deletes the file and creates a new one, or merely
truncates it to zero length, or indeed just opens it and trips the
length *after* the new data is written..

--
WOKE is an acronym... Without Originality, Knowledge or Education.

On 14/09/2023 21:51, Theo wrote:
> In comp.sys.raspberry-pi The Natural Philosopher <tnp@invalid.invalid> wrote:
>> both already done. Not closng it was the cause of a memory leak but I
>> fixed that a fortnight ago.
>>
>> I am beginning to wonder if I did more damage than just the power socket
>> when I trod on it.
>
> SIGABRT is a problem in your code.

Very definite.

Are you sure about that?

If you aren't seeing stuff in the kernel
> log then it almost certainly isn't a hardware fault. It is a very special
> skill to have a hardware fault without spewing lots of stuff there.
>
Even a corrupted bit in a ram disk?

> Post the code somewhere and someone can take a look. Otherwise you need to
> use the development tools available to you to debug the problem.
>

I can post the code, but it may not help. You need the whole system
including the perpiherals that write, to the daemon that writes the data
files that the daemon that crashes reads.

At the moment it is behaving perfectly. Without a reproducible bug I can
see no point in using a debugger.

> Theo

--
There is nothing a fleet of dispatchable nuclear power plants cannot do
that cannot be done worse and more expensively and with higher carbon
emissions and more adverse environmental impact by adding intermittent
renewable energy.

On 15/09/2023 08:30, Richard Kettlewell wrote:
> The Natural Philosopher <tnp@invalid.invalid> writes:
>> On 14/09/2023 09:23, Richard Kettlewell wrote:
>>> Also:
>>> * I would also have a look at the kernel log; if it’s a
>>> kernel-generated signal then there’s usually a log message about it.
>>>
>> Nothing in kern.log after the boot process finishes.
>
> Most likely a bug in your program then.
>
>>> * Run the application under valgrind; depending what the issue is, that
>>> will provide a backtrace and perhaps more detailed information. If it
>>> is a memory corruption issue then it may identify where the corruption
>>> happens, rather than the later point where malloc failed a consistency
>>> check (or whatever it is).
>>>
>>> Using valgrind (and/or compiler sanitizer features) is a good idea
>>> even before running into trouble, really.
>>
>> The strange thing is that it failed once after a minute, then I
>> rebooted and it failed after 20 minutes, and its been running several
>> days now with no issues at all.
>>
>> I am not sure valgrind would actually help unless it failed.
>
> It’s extremely good at identifying memory corruption even in cases where
> that doesn’t immediately lead to a crash; that’s what it’s for. But if
> it doesn’t, you leave it running until the crash happens.
>
Well that is an option for sure.

> Up to you, of course, whether you use the tools available, or debug with
> one hand tied behind your back.
>

Tell me in what way a corrupted - say - libc file, or a faulty bit of
memory would show up in the kernel logs?

The problem is that this thing is looping very frequently.
loop()
{
while (1)
{
int i;
readThermometers();
readZones();
readOverrides();
readTimerData();
setRelayState();
setRelays();
usleep (1120000);
}
}

And that means thousands of faultless iterations in a day.

So this bug ( if it is a bug) is a one in a million or worse.

I suppose I could make the thing loop ten times a second (or even
faster) and see if it happens more often..

its not as though its chewing up CPU...

The problem I have is that these crashes only recently started
happening: prior to that the code ran for days. And two things happened,
a massive brownout, and then a full power cut, and I trod on it.

And I made systemd start it...

I see it crashed again last night, again with zero errors apart from
SIGABRT...

I will start it manually and cut systemd out.

--
The lifetime of any political organisation is about three years before
its been subverted by the people it tried to warn you about.

Anon.

On 15/09/2023 10:11, Ralf Fassel wrote:
> * The Natural Philosopher <tnp@invalid.invalid>
> | On 14/09/2023 16:29, Ralf Fassel wrote:
> | > * The Natural Philosopher <tnp@invalid.invalid>
> | > | One possibility is that it is opening and reading a file at the
> | > | precise time another process is writing it...in both cases the read
> | > | and write
> | > | operations are atomic and done with C code.
> | >>
> | > | READ
> | > | ====
> | > | fp=fopen(fullname, "r");
> | > | len=fread(filbuf,1,255,fp); // read entire file
> | > Check for fp != NULL is missing here in this example code before
> | > fread(). If this also in the production version, it might be a problem
> | > if the file is not accessible for any reason.
> | > R'
> | Ralf, I already put that in this morning, re compiled the code and
> | after an hour, it crashed again.
>>
> | The filename is built by scanning a directory so the filename must exist.
>
> That assumption does not hold. Since scanning and opening are separated
> by a time gap (albeit a 'small' one), there is a non-zero chance that
> the file vanished between scan and open.
>
> Further possibilities:
> - how is 'filbuf' used after the fread()? If you use it as C-string, make
> sure it is 0-terminated (fread() won't do that for you). Maybe use
> fgets(3) instead?
>

dir = opendir(VOLATILE_DIR);

if(!dir)
return;
while ((dp = readdir (dir)) != NULL)
{
filename=dp->d_name;
// skip known bollocks
if(!strcmp(filename, "." ) || !strcmp(filename, ".." ) ||
!strcmp(filename, "relays.dat" ))
continue;
// construct full path
sprintf(fullname,"%s/%s",VOLATILE_DIR,filename);
stat(fullname,&stats);// get tfile times
if(time(NULL)-stats.st_ctime >1800) // skip files older than half an hour
continue;
len=strlen(filename);
if(strncmp(filename+len-4, ".dat",4)) // .dat file but not relays.dat
continue;
fp=fopen(fullname, "r");
if(fp==0) //file has disappeared?
continue;
len=fread(filbuf,1,255,fp);
if(len==0) // file has zero length
goto baddata;
filbuf[len]=0;
if(len=strncmp(filbuf,"ZONE",4)) //supposed to reject a file whose
contents do not start with ZONE
goto baddata;

// looking very much like a temperature file
i=(int)filbuf[4] -'1'; // this is our zone from "ZONE2" etc. 1-4 is
zone but index is 0-3 so subtract '1'
p=strstr(filbuf,"\n");
if(p)
{
p++;
if(q=strstr(p,"\n"))
{
*q++=0;
thermometers[i].name=strdup(p); // make a copy of the name and
attach it to our thermometer structure
p=q;
}
else goto baddata;
// now to fetch the temp data.
if(q=strstr(p,"\n"))
{
*q++=0;
thermometers[i].temp=atof(p);
p=q;
}
else goto baddata;
// what's left is the voltage. To hell with any crap after it
thermometers[i].battery=atof(p);
}
baddata:fclose(fp);
} // end of directory scan loop
> | I am leaning towards possibly a cracked solder joint or board.
>
> Well, since the Raspi is cheap, that should be easily checked by simply
> using another one. I bet 1 beer that it is *not* a cracked board, since
> with that many more processes should run into trouble, not only this
> particular one.
>
> R' (.sig not from me .-)

--
There is something fascinating about science. One gets such wholesale
returns of conjecture out of such a trifling investment of fact.

Mark Twain

The Natural Philosopher <tnp@invalid.invalid> writes:
> On 15/09/2023 08:30, Richard Kettlewell wrote:
>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>> I am not sure valgrind would actually help unless it failed.
>> It’s extremely good at identifying memory corruption even in cases
>> where that doesn’t immediately lead to a crash; that’s what it’s for.
>> But if it doesn’t, you leave it running until the crash happens.
>
> Well that is an option for sure.
>
>> Up to you, of course, whether you use the tools available, or debug with
>> one hand tied behind your back.
>
> Tell me in what way a corrupted - say - libc file, or a faulty bit of
> memory would show up in the kernel logs?

Very dependent on the nature of the corruption. But you’ve already told
us there’s nothing in the kernel logs.

Anyway, not responsible for advice not taken.

--
https://www.greenend.org.uk/rjk/

In comp.sys.raspberry-pi The Natural Philosopher <tnp@invalid.invalid> wrote:
> Tell me in what way a corrupted - say - libc file, or a faulty bit of
> memory would show up in the kernel logs?

Well, it could be a cosmic ray. The Pi doesn't have ECC memory to it's
possible to bit-flip in RAM or storage without it noticing. I don't know
which part of the galaxy you inhabit, but cosmic rays are rare enough down
here that random bit flips like this don't happen often - ballpark once a
year for a server (which has a much greater surface area to absorb them than
a Pi).

It is also possible to be marginal on signal integrity for PCB interconnect,
but that would mostly be a design fault: either they all work or many of
them don't. Since we don't have a lot of people complaining of the same
problem, we can assume the design is not marginal in that respect.

If computers were that unreliable they would be failing all the time - and
we'd fit ECC to everything. That they aren't suggests bit-flip corruption
isn't a problem. In general random bit-flip errors are not a statistically
major source of crashes, unless you're running a hyper-redundant mainframe
and have eliminated all the other sources.

What are a well-known class of bugs are concurrency/timing races and memory
safety violations. Which is odds-on what's happening here, especially given
we've already picked up on potentially risky code like failing to check for
NULL from fopen().

> And that means thousands of faultless iterations in a day.
>
> So this bug ( if it is a bug) is a one in a million or worse.
>
> I suppose I could make the thing loop ten times a second (or even
> faster) and see if it happens more often..

That would be a useful thing to try.

> its not as though its chewing up CPU...
>
> The problem I have is that these crashes only recently started
> happening: prior to that the code ran for days. And two things happened,
> a massive brownout, and then a full power cut, and I trod on it.

Most of those things would cause it to fail hard (ie not power up), rather
than have a very rare random fault.

> And I made systemd start it...

It is possible that being run from systemd changes the timing or environment
that provokes the fault in some way, but I doubt it would be the cause of
the fault.

Theo

The Natural Philosopher <tnp@invalid.invalid> writes:
> dir = opendir(VOLATILE_DIR);
>
> if(!dir)
> return;
> while ((dp = readdir (dir)) != NULL)
> {
> filename=dp->d_name;
> // skip known bollocks
> if(!strcmp(filename, "." ) || !strcmp(filename, ".." )
> || !strcmp(filename, "relays.dat" ))
> continue;
> // construct full path
> sprintf(fullname,"%s/%s",VOLATILE_DIR,filename);

Possible write overrun here.

> stat(fullname,&stats);// get tfile times
> if(time(NULL)-stats.st_ctime >1800) // skip files older than half an hour
> continue;
> len=strlen(filename);
> if(strncmp(filename+len-4, ".dat",4)) // .dat file but not relays.dat
> continue;

Possible read under-run here. (But if it crashes then you’d expect
SIGSEGV rather than SIGABRT, so that’s probably not the issue.)

> fp=fopen(fullname, "r");
> if(fp==0) //file has disappeared?
> continue;
> len=fread(filbuf,1,255,fp);

I don’t think the declaration of filbuf has been posted, so there’s a
possible write overrun if it’s less than 255 bytes.

--
https://www.greenend.org.uk/rjk/

You trust the contents of 'outside'-files very much, do you? ;-)
I don't know who can create files in the directory you're scanning, but
not *assuring* the input you expect is another possible cause for
problems...

* The Natural Philosopher <tnp@invalid.invalid>
| > Further possibilities:
| > - how is 'filbuf' used after the fread()? If you use it as C-string, make
| > sure it is 0-terminated (fread() won't do that for you). Maybe use
| > fgets(3) instead?
| >
| dir = opendir(VOLATILE_DIR);
>
| if(!dir)
| return;
| while ((dp = readdir (dir)) != NULL)
[looks good, error checks for stat() et al couldn't hurt]
--<snip-snip>--
| if(len=strncmp(filbuf,"ZONE",4)) //supposed to reject
| a file whose contents do not start with ZONE
| goto baddata;
|
| // looking very much like a temperature file
| i=(int)filbuf[4] -'1'; // this is our zone from
| "ZONE2" etc. 1-4 is zone but index is 0-3 so subtract
| '1'

The access of filbuf[4] is ok (since you checked that there are at least
4 characters in the file), but what if nothing follows after the 'ZONE',
or ZONE is followed by anything but [1-4]?
=> Assert that 'i' is in the valid index range here, before using it as
index into other arrays.

| p=strstr(filbuf,"\n");
| if(p)
| {
| p++;
| if(q=strstr(p,"\n"))
| {
| *q++=0;
| thermometers[i].name=strdup(p); //
| make a copy of the name and attach it
| to our thermometer structure

Memory leak if thermometers[i].name already contains something.

Other than that, I really would have it running under a debugger or
valgrind, since then *if* it crashes, you *know* *where* in your code it
crashes.

Good luck hunting!
R'

Theo <theom+news@chiark.greenend.org.uk> writes:
> The Natural Philosopher <tnp@invalid.invalid> wrote:
>> Tell me in what way a corrupted - say - libc file, or a faulty bit of
>> memory would show up in the kernel logs?
>
> Well, it could be a cosmic ray. The Pi doesn't have ECC memory to it's
> possible to bit-flip in RAM or storage without it noticing. I don't know
> which part of the galaxy you inhabit, but cosmic rays are rare enough down
> here that random bit flips like this don't happen often - ballpark once a
> year for a server (which has a much greater surface area to absorb them than
> a Pi).

I’ve seen one inarguable random bit flip in several decades. In that
case the behavior was deterministic - chiark’s /bin/ls had got a
single-bit error, and caching meant it crashed _every_ time anyone ran
it.

Maybe TNP has taken a trip to Sizewell?

--
https://www.greenend.org.uk/rjk/

On 15/09/2023 10:46, The Natural Philosopher wrote:
> On 15/09/2023 08:30, Richard Kettlewell wrote:
>> The Natural Philosopher <tnp@invalid.invalid> writes:
>>> On 14/09/2023 09:23, Richard Kettlewell wrote:
>>>> Also:
>>>> * I would also have a look at the kernel log; if it’s a
>>>>    kernel-generated signal then there’s usually a log message about it.
>>>>
>>> Nothing in kern.log after the boot process finishes.
>>
>> Most likely a bug in your program then.
>>
>>>> * Run the application under valgrind; depending what the issue is, that
>>>>     will provide a backtrace and perhaps more detailed information.
>>>> If it
>>>>     is a memory corruption issue then it may identify where the
>>>> corruption
>>>>     happens, rather than the later point where malloc failed a
>>>> consistency
>>>>     check (or whatever it is).
>>>>
>>>> Using valgrind (and/or compiler sanitizer features) is a good idea
>>>> even before running into trouble, really.
>>>
>>> The strange thing is that it failed once after a minute, then I
>>> rebooted and it failed after 20 minutes, and its been running several
>>> days now with no issues at all.
>>>
>>> I am not sure valgrind would actually help unless it failed.
>>
>> It’s extremely good at identifying memory corruption even in cases where
>> that doesn’t immediately lead to a crash; that’s what it’s for.  But if
>> it doesn’t, you leave it running until the crash happens.
>>
> Well that is an option for sure.
>

Valgrind seems to be a modern version of Purify, which was absolutely
essential, when I programmed C 30 years ago.

Personally, I want to run with full debug, stack trace, logging,
exception handling, and bounds checking turned on all the time, even in
production. Which is why I generally use a modern language like C# or Java.

I'm with you on Python being rubbish, but have you considered something
like Rust? That gives you the benefit of a modern language, without
Garbage Collection pauses (if you care), or the need for a runtime
environment (like Python, C#, and Java).

Even using C++, would give you exception handling. C++ won't force you
to go too far, If you don't want to.

On 15/09/2023 11:58, Theo wrote:
> In comp.sys.raspberry-pi The Natural Philosopher <tnp@invalid.invalid> wrote:
>> Tell me in what way a corrupted - say - libc file, or a faulty bit of
>> memory would show up in the kernel logs?
>
> Well, it could be a cosmic ray. The Pi doesn't have ECC memory to it's
> possible to bit-flip in RAM or storage without it noticing. I don't know
> which part of the galaxy you inhabit, but cosmic rays are rare enough down
> here that random bit flips like this don't happen often - ballpark once a
> year for a server (which has a much greater surface area to absorb them than
> a Pi).

Lol! I thought cosmic rays when I read this thread.

Decades of having my nose rubbed in the shit of my own stupidity, I
guess. :-)

On 15/09/2023 11:58, Theo wrote:
> What are a well-known class of bugs are concurrency/timing races and memory
> safety violations. Which is odds-on what's happening here, especially given
> we've already picked up on potentially risky code like failing to check for
> NULL from fopen().

No, I do check it.

--
“It is dangerous to be right in matters on which the established
authorities are wrong.”

― Voltaire, The Age of Louis XIV

On 15/09/2023 12:12, Richard Kettlewell wrote:
> Theo <theom+news@chiark.greenend.org.uk> writes:
>> The Natural Philosopher <tnp@invalid.invalid> wrote:
>>> Tell me in what way a corrupted - say - libc file, or a faulty bit of
>>> memory would show up in the kernel logs?
>>
>> Well, it could be a cosmic ray. The Pi doesn't have ECC memory to it's
>> possible to bit-flip in RAM or storage without it noticing. I don't know
>> which part of the galaxy you inhabit, but cosmic rays are rare enough down
>> here that random bit flips like this don't happen often - ballpark once a
>> year for a server (which has a much greater surface area to absorb them than
>> a Pi).
>
> I’ve seen one inarguable random bit flip in several decades. In that
> case the behavior was deterministic - chiark’s /bin/ls had got a
> single-bit error, and caching meant it crashed _every_ time anyone ran
> it.
>
> Maybe TNP has taken a trip to Sizewell?
>

LOL!

Nope.

I am trying some stuff out to try and get it to fail *consistently*.

I dont feel its hugely profitable to attempt to debug it when most of
the time its not doing anything wrong

--
“It is dangerous to be right in matters on which the established
authorities are wrong.”

― Voltaire, The Age of Louis XIV

On 15/09/2023 12:07, Richard Kettlewell wrote:
> The Natural Philosopher <tnp@invalid.invalid> writes:
>> dir = opendir(VOLATILE_DIR);
>>
>> if(!dir)
>> return;
>> while ((dp = readdir (dir)) != NULL)
>> {
>> filename=dp->d_name;
>> // skip known bollocks
>> if(!strcmp(filename, "." ) || !strcmp(filename, ".." )
>> || !strcmp(filename, "relays.dat" ))
>> continue;
>> // construct full path
>> sprintf(fullname,"%s/%s",VOLATILE_DIR,filename);
>
> Possible write overrun here.
The filenames never change length.

>
>> stat(fullname,&stats);// get tfile times
>> if(time(NULL)-stats.st_ctime >1800) // skip files older than half an hour
>> continue;
>> len=strlen(filename);
>> if(strncmp(filename+len-4, ".dat",4)) // .dat file but not relays.dat
>> continue;
>
> Possible read under-run here. (But if it crashes then you’d expect
> SIGSEGV rather than SIGABRT, so that’s probably not the issue.)
>
>> fp=fopen(fullname, "r");
>> if(fp==0) //file has disappeared?
>> continue;
>> len=fread(filbuf,1,255,fp);
>
> I don’t think the declaration of filbuf has been posted, so there’s a
> possible write overrun if it’s less than 255 bytes.
>
>
char filbuf[256];
char fullname[256];

The fullname is of the form

/var/www/data/volatile/192.168.0.xx.dat

There are no other files apart from 'relay.dat' in that directory.

I mean you are all throwing noob bugs at me. Yes, in 1984 that's the
sort of shit I used to write. Not these days.

I have a drawer full of T shirts marked 'buffer overrun' 'alloc without
free' 'fopen without fclose'.

The fact is the memory footprint does not increase. So there are no
obvious or simple memory leaks.

I've absolutely covered every error case mentioned here in the one case
of the files that get written and read every few seconds.

It occurs to me that this behaviour started when I made it autoboot
under systemd as well.

Since the consensus seems to be it isn't hardware, or file corruption, I
am trying it launched manually to see if it crashes or not.

Systemd does seem to wrap things in resource limits, and start with a
slightly different ENV although I cant see that any are being exceeded.

If it wasn't a daemon I would expect it to segfault and show that on
screen. I could run it without daemonising it as well.

So lots of options to try.

As well as soft debuggers.

--
“It is dangerous to be right in matters on which the established
authorities are wrong.”

― Voltaire, The Age of Louis XIV

On 15/09/2023 12:12, Ralf Fassel wrote:
> You trust the contents of 'outside'-files very much, do you? ;-)
> I don't know who can create files in the directory you're scanning, but
> not *assuring* the input you expect is another possible cause for
> problems...
>
> * The Natural Philosopher <tnp@invalid.invalid>
> | > Further possibilities:
> | > - how is 'filbuf' used after the fread()? If you use it as C-string, make
> | > sure it is 0-terminated (fread() won't do that for you). Maybe use
> | > fgets(3) instead?
> | >
> | dir = opendir(VOLATILE_DIR);
>>
> | if(!dir)
> | return;
> | while ((dp = readdir (dir)) != NULL)
> [looks good, error checks for stat() et al couldn't hurt]
> --<snip-snip>--
> | if(len=strncmp(filbuf,"ZONE",4)) //supposed to reject
> | a file whose contents do not start with ZONE
> | goto baddata;
> |
> | // looking very much like a temperature file
> | i=(int)filbuf[4] -'1'; // this is our zone from
> | "ZONE2" etc. 1-4 is zone but index is 0-3 so subtract
> | '1'
>
> The access of filbuf[4] is ok (since you checked that there are at least
> 4 characters in the file), but what if nothing follows after the 'ZONE',
> or ZONE is followed by anything but [1-4]?

That cannot happen. Its hard wired into the code that writes the file

> => Assert that 'i' is in the valid index range here, before using it as
> index into other arrays.
>
> | p=strstr(filbuf,"\n");
> | if(p)
> | {
> | p++;
> | if(q=strstr(p,"\n"))
> | {
> | *q++=0;
> | thermometers[i].name=strdup(p); //
> | make a copy of the name and attach it
> | to our thermometer structure
>
> Memory leak if thermometers[i].name already contains something.
>
further up the line...

bzero(filbuf,sizeof(filbuf));
/** first thing to do is clean any allocated memory used to store
values. **/
for(i=0;i<NUMBER_RELAYS;i++)
free(thermometers[i].name);

> Other than that, I really would have it running under a debugger or
> valgrind, since then *if* it crashes, you *know* *where* in your code it
> crashes.
>
Last resort. I have to learn how to *use* those tools.
Right now I am working on other stuff and am content to change one thing
at a time to see if that makes any difference.

That is a low user time strategy.

> Good luck hunting!
> R'

Thank you. The input has been valuable. And I now have further
strategies in reserve.

As with all intermittent faults, the thing you need most is a reliable
way to make the fault occur.

--
"The great thing about Glasgow is that if there's a nuclear attack it'll
look exactly the same afterwards."

Billy Connolly

On 9/15/23 04:16, The Natural Philosopher wrote:
> On 14/09/2023 20:40, candycanearter07 wrote:
>> On 9/14/23 13:42, The Natural Philosopher wrote:
>>> I assumed that the thing would have done its own fsck on every boot
>>> anyway...isnt that a debian default?
>>
>> Pretty sure it's a standard, my arch install has it set.
>>>
>>> (The sixth fields are 2 and 1 respectively for the file systems)
>>>
>>>
>>> PARTUUID=b8c9fbb7-01  /boot           vfat    defaults
>>> 0       2
>>> PARTUUID=b8c9fbb7-02  /               ext4    defaults,noatime
>>> 0       1
>>>
>>
>> 1 is fsck check for the root partition and 2 is for others, right
>>
> I looked it up, it merely specifies the order I think, so you are right
> in practice.
>
>

Oh, the thing I learned was that you should always put root as 1 and
everything else as 2 ^^" but that makes more sense

--
--
user <candycane> is generated from /dev/urandom

In comp.sys.raspberry-pi The Natural Philosopher <tnp@invalid.invalid> wrote:
> On 15/09/2023 12:12, Ralf Fassel wrote:
> > | {
> > | *q++=0;
> > | thermometers[i].name=strdup(p); //
> > | make a copy of the name and attach it
> > | to our thermometer structure
> >
> > Memory leak if thermometers[i].name already contains something.
> >
> further up the line...
>
> bzero(filbuf,sizeof(filbuf));
> /** first thing to do is clean any allocated memory used to store
> values. **/
> for(i=0;i<NUMBER_RELAYS;i++)
> free(thermometers[i].name);

You could get a SIGABRT if you were trying to free something that was
already freed. Are you sure those are interlocked such that for each i you
call strdup() exactly once, and subsequently free() exactly once? If there
was some code path that was breaking out of the loop or similar you might
get such behaviour.

Theo

On 15/09/2023 14:23, Theo wrote:
> In comp.sys.raspberry-pi The Natural Philosopher <tnp@invalid.invalid> wrote:
>> On 15/09/2023 12:12, Ralf Fassel wrote:
>>> | {
>>> | *q++=0;
>>> | thermometers[i].name=strdup(p); //
>>> | make a copy of the name and attach it
>>> | to our thermometer structure
>>>
>>> Memory leak if thermometers[i].name already contains something.
>>>
>> further up the line...
>>
>> bzero(filbuf,sizeof(filbuf));
>> /** first thing to do is clean any allocated memory used to store
>> values. **/
>> for(i=0;i<NUMBER_RELAYS;i++)
>> free(thermometers[i].name);
>
> You could get a SIGABRT if you were trying to free something that was
> already freed. Are you sure those are interlocked such that for each i you
> call strdup() exactly once, and subsequently free() exactly once? If there
> was some code path that was breaking out of the loop or similar you might
> get such behaviour.
>
Hmm. I free the pointers even for relay zones that don't have
thermometers, whose pointers are 0. That isn't an issue.

But that might be a remotely possible issue. I dont zero the pointers
after freeing them as far as I can tell. The silly thing is that this
program doesn't use the name anyway.

Its used elsewhere
Well I don't think its an issue, but I can zero the pointers anyway
after free()ing

> Theo

--
"I guess a rattlesnake ain't risponsible fer bein' a rattlesnake, but ah
puts mah heel on um jess the same if'n I catches him around mah chillun".

* The Natural Philosopher <tnp@invalid.invalid>
| > | if(len=strncmp(filbuf,"ZONE",4)) //supposed to reject
| > | a file whose contents do not start with ZONE
| > | goto baddata;
| > |
| > | // looking very much like a temperature file
| > | i=(int)filbuf[4] -'1'; // this is our zone from
| > | "ZONE2" etc. 1-4 is zone but index is 0-3 so subtract
| > | '1'
| > The access of filbuf[4] is ok (since you checked that there are at
| > least
| > 4 characters in the file), but what if nothing follows after the 'ZONE',
| > or ZONE is followed by anything but [1-4]?
>
| That cannot happen. Its hard wired into the code that writes the file

Depending on the permissions of VOLATILE_DIR, it *might* be possible
that *anybody* can drop files in there. Save some "// skip known
bollocks", you just scan every file in VOLATILE_DIR. If I were an
attacker, I sure would try to use that vector, regardless whether the
program in question runs with elevated permissions or not ;-)

| > Other than that, I really would have it running under a debugger or
| > valgrind, since then *if* it crashes, you *know* *where* in your code it
| > crashes.
| >
| Last resort. I have to learn how to *use* those tools.

With valgrind, it is as easy as putting 'valgrind' in front of the
commandline you use to start your program. With gdb, it is a tiny bit
more complicated, agreed. But since these tools are worth learning
anyway for any programmer, the time invested in learning them is not
wasted.

R'