Hello all.

I wrote a shell scraper of a news website and one of the options is to
keep re-accessing the initial webpage (while loop) at regular intervals,
grab all news links and scrape text of the ones which are new. This
option keeps the script running for days straight. When stdout is
redirected to a *file*, it works as expected.

Instead if we pipe output to `less', there is a buffering stand. When
`less' buffering is full at about 8-64KB, the whole while loop in the
script hangs and only continues to run when we scroll down `less'
display buffer. But if it hangs for a few hours, it means the scraping
tool will only resume scraping some hours later, too, failing to scrape
a lot of news from the initial webpage during that time.

I tried piping to `less -B -b4096' and `less -B -b-1' to no avail:

-B or --auto-buffers
By default, when data is read from a pipe, buffers are allocated
automatically as needed. If a large amount of data is read from
the pipe, this can cause a large amount of memory to be allo‐
cated. The -B option disables this automatic allocation of buf‐
fers for pipes, so that only 64 KB (or the amount of space spec‐
ified by the -b option) is used for the pipe.

-bn or --buffers=n
Specifies the amount of buffer space less will use for each
file, in units of kilobytes (1024 bytes). By default 64 KB of
buffer space is used for each file (unless the file is a pipe; see
the -B option). The -b option specifies instead that n kilobytes
of buffer space should be used for each file. If n is -1, buffer
space is unlimited; that is, the entire file can be read into
memory.

I use GNU coreutils and Arch Linux. My understanding is that `less
-Bb-1' should have worked but there is some internal buffering system
that still holds. Any suggestion? I know of `stdbuf', tried a little but
could not make it work. If I was not clear, please let me know.

Thanks,
JSN

frogger <somebody@invalid.com> writes:

> I wrote a shell scraper of a news website and one of the options is to
> keep re-accessing the initial webpage (while loop) at regular
> intervals, grab all news links and scrape text of the ones which are
> new. This option keeps the script running for days straight. When
> stdout is redirected to a *file*, it works as expected.
>
> Instead if we pipe output to `less', there is a buffering stand. When
> `less' buffering is full at about 8-64KB, the whole while loop in the
> script hangs and only continues to run when we scroll down `less'
> display buffer. But if it hangs for a few hours, it means the scraping
> tool will only resume scraping some hours later, too, failing to
> scrape a lot of news from the initial webpage during that time.

I'm not 100% sure what you want, but does:

$ scraper >file & less file

and then using the F command do something like you want?

--
Ben.

On 19/03/2022 19:06, Ben Bacarisse wrote:
> frogger <somebody@invalid.com> writes:
>
>> I wrote a shell scraper of a news website and one of the options is to
>> keep re-accessing the initial webpage (while loop) at regular
>> intervals, grab all news links and scrape text of the ones which are
>> new. This option keeps the script running for days straight. When
>> stdout is redirected to a *file*, it works as expected.
>>
>> Instead if we pipe output to `less', there is a buffering stand. When
>> `less' buffering is full at about 8-64KB, the whole while loop in the
>> script hangs and only continues to run when we scroll down `less'
>> display buffer. But if it hangs for a few hours, it means the scraping
>> tool will only resume scraping some hours later, too, failing to
>> scrape a lot of news from the initial webpage during that time.
>
> I'm not 100% sure what you want, but does:
>
> $ scraper >file & less file
>
> and then using the F command do something like you want?
>

Hey Ben!

That is *exactly* what I am trying just now. Was not sure it would be a
good solution but it seems straightforward that is, as it was so obvious
to you.

So what I just did and works is:

scraperFunction >file & tail -f file | less

Also added a trap inside the script to kill forks and remove temp file.

Thanks, Ben!
JSN

I am a little drunk but stop wars, you english-american people.
English language is very funny. That is why programmer languages are
funny, too.

Cheers, english people! Stop the war. Americans and Europeans are very
funny people, wish they would be better than us, tho.

Because I did not see it here in usenet, UKRAINE wins the wars.