scott@slp53.sl.home (Scott Lurndal) writes:
> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>Am 11.05.2023 um 19:18 schrieb Scott Lurndal:
>>> Linux only overcommits if the administrator configures it to
>>> overcommit. ...
>>
>>This is the default-behaviour which is almost never changed.
>
> And you base your opinion on what experiences, exactly?

I had thought about commenting on this myself.

In my experience, which is certainly not exhaustive, most Linux systems
have overcommit enabled by default. I've never seen one where it's
disabled (though I haven't always checked).

On my Ubuntu system, `cat /proc/sys/vm/overcommit_memory` shows 0,
which according to the proc(5) man page means:

In mode 0, calls of mmap(2) with MAP_NORESERVE are not checked, and
the default check is very weak, leading to the risk of getting a
process "OOM-killed".

Have you seen it set by default to a different value? (This is
admittedly only marginally topical, but it does affect the behavior of
memory allocation in C++ programs.)

--
Keith Thompson (The_Other_Keith) Keith.S.Thompson+u@gmail.com
Working, but not speaking, for XCOM Labs
void Void(void) { Void(); } /* The recursive call of the void */

Am 12.05.2023 um 19:58 schrieb Scott Lurndal:

> Stack exchange? Not representative of production unix or linux system
> operations, that's certain.

Stack Exchange is the most qualified source for such discussions
because they delete unqualified comments.

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 12.05.2023 um 19:58 schrieb Scott Lurndal:
>
>> Stack exchange? Not representative of production unix or linux system
>> operations, that's certain.
>
>Stack Exchange is the most qualified source for such discussions
>because they delete unqualified comments.
>

Now that is truely self-referential. Sigh.

Keith Thompson <Keith.S.Thompson+u@gmail.com> writes:
>scott@slp53.sl.home (Scott Lurndal) writes:
>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>>Am 11.05.2023 um 19:18 schrieb Scott Lurndal:
>>>> Linux only overcommits if the administrator configures it to
>>>> overcommit. ...
>>>
>>>This is the default-behaviour which is almost never changed.
>>
>> And you base your opinion on what experiences, exactly?
>
>I had thought about commenting on this myself.
>
>In my experience, which is certainly not exhaustive, most Linux systems
>have overcommit enabled by default. I've never seen one where it's
>disabled (though I haven't always checked).
>
>On my Ubuntu system, `cat /proc/sys/vm/overcommit_memory` shows 0,
>which according to the proc(5) man page means:
>
> In mode 0, calls of mmap(2) with MAP_NORESERVE are not checked, and
> the default check is very weak, leading to the risk of getting a
> process "OOM-killed".
>
>Have you seen it set by default to a different value? (This is
>admittedly only marginally topical, but it does affect the behavior of
>memory allocation in C++ programs.)

All of our simulation systems (many hundred linux servers) run with
overcommit set to 2 (always check). They're running RTL simulations
and it's much better to prevent a long-running simulation from starting rather
than killing one randomly after a few hours of cpu time.

The queue management system that distributes jobs to the servers
understands the server capacities and trusts the user's estimate
of required memory, but those estimates are often faulty.

On 12/05/2023 21:13, Keith Thompson wrote:
> scott@slp53.sl.home (Scott Lurndal) writes:
>> Bonita Montero <Bonita.Montero@gmail.com> writes:
>>> Am 11.05.2023 um 19:18 schrieb Scott Lurndal:
>>>> Linux only overcommits if the administrator configures it to
>>>> overcommit. ...
>>>
>>> This is the default-behaviour which is almost never changed.
>>
>> And you base your opinion on what experiences, exactly?
>
> I had thought about commenting on this myself.
>
> In my experience, which is certainly not exhaustive, most Linux systems
> have overcommit enabled by default. I've never seen one where it's
> disabled (though I haven't always checked).
>
> On my Ubuntu system, `cat /proc/sys/vm/overcommit_memory` shows 0,
> which according to the proc(5) man page means:
>
> In mode 0, calls of mmap(2) with MAP_NORESERVE are not checked, and
> the default check is very weak, leading to the risk of getting a
> process "OOM-killed".
>
> Have you seen it set by default to a different value? (This is
> admittedly only marginally topical, but it does affect the behavior of
> memory allocation in C++ programs.)
>

It would not surprise me if the default value for this is different for
desktop-oriented distributions and server-oriented distributions - or
for kernels aimed at these different uses. There have traditionally
been some differences in how these are tuned - desktops aim for greater
responsiveness, servers for more stable throughput, amongst other things.

For most of my Linux systems, I don't bother about that kind of thing -
Linux adapts well enough as it goes along. But if I were running
servers with high continuous loads or other special needs, I'd look
closely at a variety of this kind of tunable.

On Sat, 13 May 2023 15:50:47 +0200
David Brown <david.brown@hesbynett.no> wrote:
>For most of my Linux systems, I don't bother about that kind of thing -
>Linux adapts well enough as it goes along. But if I were running
>servers with high continuous loads or other special needs, I'd look
>closely at a variety of this kind of tunable.

For mission critical systems you really don't want processes being killed
randomly due to lack of real memory, you want to know at startup.

Years ago part of my then job was 2nd line support for a system based on
Sybase which unfortunately unlike Oracle couldn't resolve deadlocks and so just
killed one of the 2 processes involved. Usually at 3am when the batch was
halfway through which would sometimes mean a phone call and trying to log on
via an ISDN line. Not fun.

Am 13.05.2023 um 16:54 schrieb Muttley@dastardlyhq.com:

> For mission critical systems you really don't want processes
> being killed randomly due to lack of real memory, you want
> to know at startup.

When the computer starts swapping, it usually happens that the
swapping slows down the software so much that it stops responding
and it ultimately makes no difference as if the operating system
had simply killed the critical process.

On Sat, 13 May 2023 17:56:17 +0200
Bonita Montero <Bonita.Montero@gmail.com> wrote:
>Am 13.05.2023 um 16:54 schrieb Muttley@dastardlyhq.com:
>
>> For mission critical systems you really don't want processes
>> being killed randomly due to lack of real memory, you want
>> to know at startup.
>
>When the computer starts swapping, it usually happens that the
>swapping slows down the software so much that it stops responding
>and it ultimately makes no difference as if the operating system
>had simply killed the critical process.

That depends on what hardware your swap space is on. If its an SSD then that
shouldn't happen, if its spinning rust then yes that is often the outcome
and because of that I don't bother with swap on either of my 2 linux systems.
I'd sooner the kernel just killed the process rather than bring the whole
system to its knees but obviously with 24/7/365 systems that might not be an
option.

Am 13.05.2023 um 18:05 schrieb Muttley@dastardlyhq.com:

> That depends on what hardware your swap space is on. If its an SSD then that
> shouldn't happen, ...

LOL. On my Samsung PCIe 4.0 SSD synchonous I/O - like with swapping
- is about 70us per 4kB block. Random-access memory accesses with at
most page hits as possible is about 25ns. That's factor 2.800. To
rely on swapping to keep the machine running is silly.

Am 13.05.2023 um 18:18 schrieb Bonita Montero:
> Am 13.05.2023 um 18:05 schrieb Muttley@dastardlyhq.com:
>
>> That depends on what hardware your swap space is on. If its an SSD
>> then that
>> shouldn't happen, ...
>
> LOL. On my Samsung PCIe 4.0 SSD synchonous I/O - like with swapping
> - is about 70us per 4kB block. Random-access memory accesses with at
> most page hits as possible is about 25ns. That's factor 2.800. ...

That was an estimate; here are exact numbers:

C:\Users\Bonita\Documents\Source\ParallelIo>x64\Release\ParallelIo
--synch --threads 1 --degree 1
synchronous I/O, filename: "testfile", file-size: 1GB, block-size: 4kB,
threads: 1
53.6 MB/s, 13232 IOs/s (76.2us), avg 53.0 MB/s, 13086 IOs/s (75.4us),
max 53.0 MB/s, 13232 IOs/s (76.2us)
....

C:\Users\Bonita\Documents\Source\memLatency>x64\Release\memLatency.exe
--paged --lower 32gb --upper 32gb
32GiB 31.46ns

Factor 2422.

On 13/05/2023 17:56, Bonita Montero wrote:
> Am 13.05.2023 um 16:54 schrieb Muttley@dastardlyhq.com:
>
>> For mission critical systems you really don't want processes
>> being killed randomly due to lack of real memory, you want
>> to know at startup.
>
> When the computer starts swapping, it usually happens that the
> swapping slows down the software so much that it stops responding
> and it ultimately makes no difference as if the operating system
> had simply killed the critical process.

That is not always the case - and certainly hasn't always been the case.
Modern high speed memory is proportionally much faster than even
modern SSD disks, but there was a time when it was quite reasonable to
use swap space for many purposes - the latency was high, but the
throughput could be relatively good for the sizes of memory common at
the time.

There can also be a lot of convenience in swapping. Sometimes programs
have a lot of code or data that is only needed occasionally, such as at
startup. Letting that swap out again is a benefit for everyone. This
is in fact how Linux handles program or library loading - it does not
read the executable into ram, it just maps it into virtual memory space,
effectively starting it fully swapped out. It then swaps in as needed.

But unexpected or unplanned swapping is a bad thing - if a process
starts taking so much memory that it forces most of the rest of the
system onto swap, then I agree things are going to go /really/ badly.

Am 13.05.2023 um 19:03 schrieb David Brown:

> That is not always the case - and certainly hasn't always been the case.
>  Modern high speed memory is proportionally much faster than even
> modern SSD disks, but there was a time when it was quite reasonable to
> use swap space for many purposes - the latency was high, but the
> throughput could be relatively good for the sizes of memory common at
> the time.

That's not true.
This random access I/O from 100 Threads to a normal 10ms 4TB Disk:

C:\Users\Bonita\Documents\Source\ParallelIo>x64\Release\ParallelIo.exe
--degree 32 --file x:\testfile --synch
synchronous I/O, filename: "x:\testfile", file-size: 1GB, block-size:
4kB, threads: 32
1.8 MB/s, 453 IOs/s (64.3ms), avg 1.8 MB/s, 447 IOs/s (63.5ms), max 1.8
MB/s, 453 IOs/s (64.3ms)

1.8 MB/s from 32 threads. Compare that if you'd have 100 threads
accessing raw memory with mostly page hits which is about 32ns,
i.e. 2GB/s (counting the whole fetched cachelines).

Swap does make sense only to back overcomitting which is the default
for Windows or what can be configured under Linux.

On 13/05/2023 19:23, Bonita Montero wrote:
> Am 13.05.2023 um 19:03 schrieb David Brown:
>
>> That is not always the case - and certainly hasn't always been the
>> case.   Modern high speed memory is proportionally much faster than
>> even modern SSD disks, but there was a time when it was quite
>> reasonable to use swap space for many purposes - the latency was high,
>> but the throughput could be relatively good for the sizes of memory
>> common at the time.
>
> That's not true.

You /do/ understand that I was referring to past usage, not typical
modern machines?

> This random access I/O from 100 Threads to a normal 10ms 4TB Disk:
>
> C:\Users\Bonita\Documents\Source\ParallelIo>x64\Release\ParallelIo.exe
> --degree 32 --file x:\testfile --synch
> synchronous I/O, filename: "x:\testfile", file-size: 1GB, block-size:
> 4kB, threads: 32
> 1.8 MB/s, 453 IOs/s (64.3ms), avg 1.8 MB/s, 447 IOs/s (63.5ms), max 1.8
> MB/s, 453 IOs/s (64.3ms)
>
> 1.8 MB/s from 32 threads. Compare that if you'd have 100 threads
> accessing raw memory with mostly page hits which is about 32ns,
> i.e. 2GB/s (counting the whole fetched cachelines).
>
> Swap does make sense only to back overcomitting which is the default
> for Windows or what can be configured under Linux.
>
>

Am 13.05.2023 um 21:08 schrieb David Brown:

> You /do/ understand that I was referring to past usage, not typical
> modern machines?

There are not much differences in access times on old HDDs.
And DRAM access times aren't much different the last 20 yrs.

On Sat, 13 May 2023 19:03:24 +0200
David Brown <david.brown@hesbynett.no> wrote:
>But unexpected or unplanned swapping is a bad thing - if a process
>starts taking so much memory that it forces most of the rest of the
>system onto swap, then I agree things are going to go /really/ badly.

Unfortunately Linux (and probably Windows and Mac) doesn't AFAIK prioritise
processes that the user needs to rescue the system when it tanks due to
swapping. Any shell and its child processes spawned from getty/login (or
whatever fucked up crap systemDire replaced them with) should never be swapped
out if the shell is running as root but I guess thats tricky to sort out.

On 14/05/2023 02:46, Bonita Montero wrote:
> Am 13.05.2023 um 21:08 schrieb David Brown:
>
>> You /do/ understand that I was referring to past usage, not typical
>> modern machines?
>
> There are not much differences in access times on old HDDs.
> And DRAM access times aren't much different the last 20 yrs.
>
Remind me what was the memory cycle time on an IBM360?

I started off on mainframes. The system would quite happily perform IO
on several disc drives simultaneously without affecting the CPU at all.
IIRC our big system had ~40 disc drives.

The speed of all these things have gone up, but the ratios between the
speeds have changed too. Some of us do go back a long way.

Andy

Vir Campestris <vir.campestris@invalid.invalid> writes:
>On 14/05/2023 02:46, Bonita Montero wrote:
> > Am 13.05.2023 um 21:08 schrieb David Brown:
> >
> >> You /do/ understand that I was referring to past usage, not typical
> >> modern machines?
> >
> > There are not much differences in access times on old HDDs.
> > And DRAM access times aren't much different the last 20 yrs.
> >
>Remind me what was the memory cycle time on an IBM360?
>
>I started off on mainframes. The system would quite happily perform IO
>on several disc drives simultaneously without affecting the CPU at all.

I think that depends on the model. The lower-end models didn't have
real DMA IIRC.

>IIRC our big system had ~40 disc drives.

The more important consideration for bandwidth is the channel
(bus/tag) capacity and the ability for the host storage controller
(channel program) to interleave accesses to multiple drives on
a channel (sometimes splitting seek from read to allow the heads to
position while transfering from a different drive on the string).

Burroughs disk subsystems could multiplex eight (1MByte/s) host channels to a
string of 16 drives, so any eight could be transferring into memory
simultaneously (well, interleaved by the I/O controller when
accessing main memory).

Am 16.05.2023 um 22:43 schrieb Vir Campestris:

> I started off on mainframes. The system would quite happily perform IO
> on several disc drives simultaneously without affecting the CPU at all.
> IIRC our big system had ~40 disc drives.

I can transfer 150MB/s vom my 4TB HDD nearly without havin any CPU-load.
If I do random access I/O with I/O completion ports and 512 outstanding
operations I get a CPU load of three to four percent with about 4GB/s.

On 5/16/2023 7:46 PM, Bonita Montero wrote:
> Am 16.05.2023 um 22:43 schrieb Vir Campestris:
>
>> I started off on mainframes. The system would quite happily perform IO
>> on several disc drives simultaneously without affecting the CPU at
>> all. IIRC our big system had ~40 disc drives.
>
> I can transfer 150MB/s vom my 4TB HDD nearly without havin any CPU-load.
> If I do random access I/O with I/O completion ports and 512 outstanding
> operations I get a CPU load of three to four percent with about 4GB/s.
>

Be careful with those IO completion ports for they use non-paged memory!

Am 17.05.2023 um 05:24 schrieb Chris M. Thomasson:

> Be careful with those IO completion ports for they use non-paged memory!

The kernel uses non-paged memory for a lot of data structures.

On 5/16/2023 9:22 PM, Bonita Montero wrote:
> Am 17.05.2023 um 05:24 schrieb Chris M. Thomasson:
>
>> Be careful with those IO completion ports for they use non-paged memory!
>
> The kernel uses non-paged memory for a lot of data structures.
>

You need to be weary about how many in progress IOCP actions there are
in the system at the same time. They will eat into your non-paged pool.

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 16.05.2023 um 22:43 schrieb Vir Campestris:
>
>> I started off on mainframes. The system would quite happily perform IO
>> on several disc drives simultaneously without affecting the CPU at all.
>> IIRC our big system had ~40 disc drives.
>
>I can transfer 150MB/s vom my 4TB HDD nearly without havin any CPU-load.
>If I do random access I/O with I/O completion ports and 512 outstanding
>operations I get a CPU load of three to four percent with about 4GB/s.
>

So what? DMA has been around for decades.

Am 17.05.2023 um 15:37 schrieb Scott Lurndal:

> So what? DMA has been around for decades.

Vir thinks I/O is more efficient with Mainframes.
I gave hin an example of how eficient I/O is on a 1.500$ PC.

Bonita Montero <Bonita.Montero@gmail.com> writes:
>Am 17.05.2023 um 15:37 schrieb Scott Lurndal:
>
>> So what? DMA has been around for decades.
>
>Vir thinks I/O is more efficient with Mainframes.
>I gave hin an example of how eficient I/O is on a 1.500$ PC.

Apples are red, oranges are, well, orange.

How well did your PC handle I/O in 1975?

Am 17.05.2023 um 17:11 schrieb Scott Lurndal:

> Bonita Montero <Bonita.Montero@gmail.com> writes:

>> I gave hin an example of how eficient I/O is on a 1.500$ PC.

> Apples are red, oranges are, well, orange.
> How well did your PC handle I/O in 1975?

There are still mainframes and they're
for sure not more efficient with I/O.