Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

"It's not just a computer -- it's your ass." -- Cal Keegan


computers / news.software.nntp / Re: CNFS buffer consuming more disk space than the file size?

SubjectAuthor
* CNFS buffer consuming more disk space than the file size?Jesse Rehmer
+* Re: CNFS buffer consuming more disk space than the file size?Julien ÉLIE
|`* Re: CNFS buffer consuming more disk space than the file size?Jesse Rehmer
| `- Re: CNFS buffer consuming more disk space than the file size?Julien ÉLIE
`* Re: CNFS buffer consuming more disk space than the file size?Billy G. (go-while)
 +- Re: CNFS buffer consuming more disk space than the file size?Billy G. (go-while)
 `* Re: CNFS buffer consuming more disk space than the file size?Jesse Rehmer
  `* Re: CNFS buffer consuming more disk space than the file size?Billy G. (go-while)
   +- Re: CNFS buffer consuming more disk space than the file size?Jesse Rehmer
   `* Re: CNFS buffer consuming more disk space than the file size?Jesse Rehmer
    +* Re: CNFS buffer consuming more disk space than the file size?Billy G. (go-while)
    |`* Re: CNFS buffer consuming more disk space than the file size?Jesse Rehmer
    | `* Re: CNFS buffer consuming more disk space than the file size?Jesse Rehmer
    |  `* Re: CNFS buffer consuming more disk space than the file size?Billy G. (go-while)
    |   `- Re: CNFS buffer consuming more disk space than the file size?Jesse Rehmer
    `- Re: CNFS buffer consuming more disk space than the file size?Russ Allbery

1
CNFS buffer consuming more disk space than the file size?

<ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2259&group=news.software.nntp#2259

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: CNFS buffer consuming more disk space than the file size?
Date: Thu, 5 Oct 2023 14:25:25 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 5 Oct 2023 14:25:25 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="77150"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:awo8tNykhV2QjIrCTXB2/gDJz/g= sha256:mLv1Klx05yyZ0KmWSalArB5hdcHorRyoadd6eGMdi6M=
sha1:LmL7pmykI8Mc/xgjVb7636YEtXg= sha256:7vAn3XUukPWloBhAlnuAawHTAasTIOP0KpRMA1lASto=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Thu, 5 Oct 2023 14:25 UTC

This could be a ZFS/FreeBSD issue, but figure I'd start here to see if anyone
has seen this behavior before... I started to run tight on disk space on one
of my servers and noticed that the CNFS buffers were using an amount of disk
space that is larger than the 'actual' file size of the buffer.

I have this cycbuff.conf entry:

cycbuff:SBIN2:/usr/local/news/spool/bin1:102400000

Looking at this file with 'ls' the size lines up:

-rw-r--r-- 1 news news 104857600000 Oct 4 15:14 bin1

However, when I check actual disk space used with du it is much greater:

# du -h bin1
129G bin1

# du bin1
135118061 bin1

This buffer contained 'junk' so I decided to remove it, and did reclaim 129GB
of space on the filesystem. How is this possible?

Re: CNFS buffer consuming more disk space than the file size?

<ufn058$76br$1@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2267&group=news.software.nntp#2267

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176-143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Thu, 5 Oct 2023 20:43:52 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ufn058$76br$1@news.trigofacile.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 5 Oct 2023 18:43:52 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176-143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="235899"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla Thunderbird
Cancel-Lock: sha1:N1mWC42hEPdLSHEB4XZO/qlw97Y= sha256:K86qDkossRLd0JtKZjvmJxCQeDTJGydApiTlwFyO+cA=
sha1:C3UbArlt4oRIaaNIxzoTWyFPC0o= sha256:A0G8AGmpMlZW9p2w1q/9xxyKrCFAtO4dlEM2I0fnTlA=
In-Reply-To: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Thu, 5 Oct 2023 18:43 UTC

Hi Jesse,

> I have this cycbuff.conf entry:
>
> cycbuff:SBIN2:/usr/local/news/spool/bin1:102400000
>
> Looking at this file with 'ls' the size lines up:
>
> -rw-r--r-- 1 news news 104857600000 Oct 4 15:14 bin1
>
> However, when I check actual disk space used with du it is much greater:
>
> # du -h bin1
> 129G bin1
>
> # du bin1
> 135118061 bin1

I've checked my CNFS buffers on Debian, and do not see such a difference.

cycbuff:FR1:/home/news/spool/cycbuffs/frcnfs:4194304

% du frcnfs
4194308 frcnfs

% ls -l
-rw-r--r-- 1 news news 4294967296 5 oct. 20:36 frcnfs

> This buffer contained 'junk' so I decided to remove it, and did reclaim 129GB
> of space on the filesystem. How is this possible?

I don't know. Is it the same for your other CNFS buffers?

It would have been interesting to see the output of "cnfsstat" on your
bin1. Do you still have a checkpoint for SBIN2 in news.notice to see
its reported length?

--
Julien ÉLIE

« Un homme plein de vices finit un jour ou l'autre sous écrou. » (Marc
Escayrol)

Re: CNFS buffer consuming more disk space than the file size?

<ufn0j7$l9f$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2269&group=news.software.nntp#2269

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!rocksolid2!news.neodome.net!weretis.net!feeder8.news.weretis.net!3.eu.feeder.erje.net!1.us.feeder.erje.net!feeder.erje.net!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Thu, 5 Oct 2023 18:51:19 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ufn0j7$l9f$1@nnrp.usenet.blueworldhosting.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com> <ufn058$76br$1@news.trigofacile.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 5 Oct 2023 18:51:19 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="21807"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:1wj8OANigXRdDSyxf2MkTUzyQQs= sha256:D4lPrUxT4hCZ58ZXaHwvZUl95I4RTt3irfjfFUxH2JM=
sha1:4iOOnqC+kyBAvPstMLrBQ0SxGbQ= sha256:jumg6G5FHDJJriP/FqBbtJnMjhpuRh6G7BC3gh5rLOM=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Thu, 5 Oct 2023 18:51 UTC

On Oct 5, 2023 at 1:43:52 PM CDT, "Julien ÉLIE"
<iulius@nom-de-mon-site.com.invalid> wrote:

> Hi Jesse,
>
>> I have this cycbuff.conf entry:
>>
>> cycbuff:SBIN2:/usr/local/news/spool/bin1:102400000
>>
>> Looking at this file with 'ls' the size lines up:
>>
>> -rw-r--r-- 1 news news 104857600000 Oct 4 15:14 bin1
>>
>> However, when I check actual disk space used with du it is much greater:
>>
>> # du -h bin1
>> 129G bin1
>>
>> # du bin1
>> 135118061 bin1
>
> I've checked my CNFS buffers on Debian, and do not see such a difference.
>
>
> cycbuff:FR1:/home/news/spool/cycbuffs/frcnfs:4194304
>
> % du frcnfs
> 4194308 frcnfs
>
> % ls -l
> -rw-r--r-- 1 news news 4294967296 5 oct. 20:36 frcnfs
>
>
>
>> This buffer contained 'junk' so I decided to remove it, and did reclaim 129GB
>> of space on the filesystem. How is this possible?
>
> I don't know. Is it the same for your other CNFS buffers?
>
> It would have been interesting to see the output of "cnfsstat" on your
> bin1. Do you still have a checkpoint for SBIN2 in news.notice to see
> its reported length?

There was another small 1GB buffer that was 1.2GB on disk, I deleted it too,
unfortunately. The only thing I see in news.notice about CNFS are lines like
this:

innd[74417]: CNFS: CNFSflushallheads: flushing SBIN2

I've made a new buffer and will wait until it wraps to see. So far the size is
in line with what I expect:

$ cnfsstat
Class SBIN for groups matching "control.cancel,junk,alt.binaries.*"
Buffer SBIN1, size: 9.54 GBytes, position: 9.21 GBytes 0.97 cycles
Newest: 2023-10-05 13:47:44, 0 days, 0:00:06 ago

$ du -h cnfs2
9.3G cnfs2

Re: CNFS buffer consuming more disk space than the file size?

<ufn0u8$76bs$3@news.trigofacile.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2270&group=news.software.nntp#2270

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!news.trigofacile.com!.POSTED.176.143-2-105.abo.bbox.fr!not-for-mail
From: iul...@nom-de-mon-site.com.invalid (Julien ÉLIE)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Thu, 5 Oct 2023 20:57:12 +0200
Organization: Groupes francophones par TrigoFACILE
Message-ID: <ufn0u8$76bs$3@news.trigofacile.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
<ufn058$76br$1@news.trigofacile.com>
<ufn0j7$l9f$1@nnrp.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 5 Oct 2023 18:57:12 -0000 (UTC)
Injection-Info: news.trigofacile.com; posting-account="julien"; posting-host="176.143-2-105.abo.bbox.fr:176.143.2.105";
logging-data="235900"; mail-complaints-to="abuse@trigofacile.com"
User-Agent: Mozilla Thunderbird
Cancel-Lock: sha1:646oOvFfXeofvZ7cDRrHEsw4w74= sha256:InHfO1zfhNmMlFJXfuxbieFIKS7uF4nhjbLNJ2RDyI4=
sha1:eyNU/WgZb1SLEb5gwOGESHDdM0c= sha256:oPvU7GQ+thEnsaDnwToR5X4Ie6AflyF+tgNr4DQALu0=
In-Reply-To: <ufn0j7$l9f$1@nnrp.usenet.blueworldhosting.com>
 by: Julien ÉLIE - Thu, 5 Oct 2023 18:57 UTC

Hi Jesse,

> The only thing I see in news.notice about CNFS are lines like this:
>
> innd[74417]: CNFS: CNFSflushallheads: flushing SBIN2

Ah yes, there's an inn.conf parameter (docnfsstat) which is *false* by
default, to regularly log the results of cnfsstat in news.notice.

As I activated it a long time ago, I thought it was normal to have these
logs :)

> I've made a new buffer and will wait until it wraps to see. So far the size is
> in line with what I expect

OK, thanks. Keep us informed!

--
Julien ÉLIE

« C'est tout un art que de parvenir à couper un gâteau de telle sorte
que chacun croit en avoir eu la plus grosse part. »

Re: CNFS buffer consuming more disk space than the file size?

<XSFTM.7805$4rg.2638@fx09.ams4>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2271&group=news.software.nntp#2271

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!feeder1.feed.usenet.farm!feed.usenet.farm!peer03.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx09.ams4.POSTED!not-for-mail
MIME-Version: 1.0
User-Agent: NoZilla/3.11 (Hackint; Unicorn; rv:0.8.15) go-while/19720229
NewsRW/4.2.0
Subject: Re: CNFS buffer consuming more disk space than the file size?
Newsgroups: news.software.nntp
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
Content-Language: en-US
From: no-re...@no.spam (Billy G. (go-while))
Organization: github.com/go-while
In-Reply-To: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 176
Message-ID: <XSFTM.7805$4rg.2638@fx09.ams4>
X-Complaints-To: abuse@blocknews.net
NNTP-Posting-Date: Thu, 05 Oct 2023 21:27:19 UTC
Date: Thu, 5 Oct 2023 23:39:32 +0200
X-Received-Bytes: 6231
 by: Billy G. (go-while) - Thu, 5 Oct 2023 21:39 UTC

On 05.10.23 16:25, Jesse Rehmer wrote:
> This could be a ZFS/FreeBSD issue, but figure I'd start here to see if anyone
> has seen this behavior before... I started to run tight on disk space on one
> of my servers and noticed that the CNFS buffers were using an amount of disk
> space that is larger than the 'actual' file size of the buffer.
>
> I have this cycbuff.conf entry:
>
> cycbuff:SBIN2:/usr/local/news/spool/bin1:102400000
>
> Looking at this file with 'ls' the size lines up:
>
> -rw-r--r-- 1 news news 104857600000 Oct 4 15:14 bin1
>
> However, when I check actual disk space used with du it is much greater:
>
> # du -h bin1
> 129G bin1
>
> # du bin1
> 135118061 bin1
>
> This buffer contained 'junk' so I decided to remove it, and did reclaim 129GB
> of space on the filesystem. How is this possible?

you have compression enabled?
how did you create the cycbuf?
as sparse via fallocate or filled via dd from /dev/zero?

i have zfs with compression lz4 and as you see 'ls -l' reports this size
but 'du -h' reports real used spaced from file.

ls -l
total 717894363
-rw-r--r-- 1 news news 107374182400 Oct 25 2022 BIG1.cyc
-rw-rw---- 1 news news 107374182400 Oct 27 2022 BIG2.cyc
-rw-rw---- 1 news news 107374182400 Dec 10 2022 BIG3.cyc
-rw-r--r-- 1 news news 107374182400 May 21 03:27 BIG4.cyc
-rw-r--r-- 1 news news 107374182400 Nov 1 2022 BIG5.cyc
-rw-r--r-- 1 news news 107374182400 Nov 1 2022 BIG6.cyc
-rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW10.cyc
-rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW11.cyc
-rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW12.cyc
-rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW13.cyc
-rw-rw---- 1 news news 107374182400 Oct 28 2022 LOW14.cyc
-rw-rw---- 1 news news 107374182400 Nov 4 2022 LOW15.cyc
-rw-rw---- 1 news news 107374182400 Apr 10 18:08 LOW16.cyc
-rw-rw---- 1 news news 107374182400 May 21 03:27 LOW17.cyc
-rw-rw---- 1 news news 107374182400 Nov 6 2022 LOW18.cyc
-rw-rw---- 1 news news 107374182400 Nov 6 2022 LOW19.cyc
-rw-r--r-- 1 news news 107374182400 Oct 23 2022 LOW1.cyc
-rw-rw---- 1 news news 107374182400 Nov 6 2022 LOW20.cyc
-rw-r--r-- 1 news news 107374182400 Oct 23 2022 LOW2.cyc
-rw-rw---- 1 news news 107374182400 Oct 24 2022 LOW3.cyc
-rw-rw---- 1 news news 107374182400 Oct 24 2022 LOW4.cyc
-rw-rw---- 1 news news 107374182400 Oct 25 2022 LOW5.cyc
-rw-rw---- 1 news news 107374182400 Oct 25 2022 LOW6.cyc
-rw-rw---- 1 news news 107374182400 Oct 26 2022 LOW7.cyc
-rw-rw---- 1 news news 107374182400 Oct 26 2022 LOW8.cyc
-rw-rw---- 1 news news 107374182400 Oct 26 2022 LOW9.cyc
-rw-r--r-- 1 news news 107374182400 May 21 03:26 MID1.cyc
-rw-rw---- 1 news news 107374182400 Oct 15 2022 MID2.cyc
-rw-rw---- 1 news news 107374182400 Oct 15 2022 MID3.cyc
-rw-rw---- 1 news news 107374182400 Oct 15 2022 MID4.cyc
-rw-rw---- 1 news news 107374182400 Oct 15 2022 MID5.cyc
-rw-rw---- 1 news news 107374182400 Oct 15 2022 MID6.cyc

du -h *
59G BIG1.cyc
50G BIG2.cyc
52G BIG3.cyc
4.2G BIG4.cyc
33K BIG5.cyc
33K BIG6.cyc
34G LOW10.cyc
34G LOW11.cyc
35G LOW12.cyc
36G LOW13.cyc
36G LOW14.cyc
29G LOW15.cyc
29G LOW16.cyc
4.7G LOW17.cyc
11K LOW18.cyc
11K LOW19.cyc
22G LOW1.cyc
11K LOW20.cyc
25G LOW2.cyc
21G LOW3.cyc
25G LOW4.cyc
33G LOW5.cyc
30G LOW6.cyc
31G LOW7.cyc
34G LOW8.cyc
36G LOW9.cyc
35G MID1.cyc
11K MID2.cyc
11K MID3.cyc
11K MID4.cyc
11K MID5.cyc
11K MID6.cyc

and 'du -b' reports other size...

du -b *
107374182400 BIG1.cyc
107374182400 BIG2.cyc
107374182400 BIG3.cyc
107374182400 BIG4.cyc
107374182400 BIG5.cyc
107374182400 BIG6.cyc
107374182400 LOW10.cyc
107374182400 LOW11.cyc
107374182400 LOW12.cyc
107374182400 LOW13.cyc
107374182400 LOW14.cyc
107374182400 LOW15.cyc
107374182400 LOW16.cyc
107374182400 LOW17.cyc
107374182400 LOW18.cyc
107374182400 LOW19.cyc
107374182400 LOW1.cyc
107374182400 LOW20.cyc
107374182400 LOW2.cyc
107374182400 LOW3.cyc
107374182400 LOW4.cyc
107374182400 LOW5.cyc
107374182400 LOW6.cyc
107374182400 LOW7.cyc
107374182400 LOW8.cyc
107374182400 LOW9.cyc
107374182400 MID1.cyc
107374182400 MID2.cyc
107374182400 MID3.cyc
107374182400 MID4.cyc
107374182400 MID5.cyc
107374182400 MID6.cyc

and finaly du *

du *
61314603 BIG1.cyc
51517375 BIG2.cyc
53918660 BIG3.cyc
4385048 BIG4.cyc
33 BIG5.cyc
33 BIG6.cyc
35449737 LOW10.cyc
35492501 LOW11.cyc
36521650 LOW12.cyc
37057675 LOW13.cyc
36731237 LOW14.cyc
30020730 LOW15.cyc
29718199 LOW16.cyc
4829014 LOW17.cyc
11 LOW18.cyc
11 LOW19.cyc
22134082 LOW1.cyc
11 LOW20.cyc
26145542 LOW2.cyc
21492682 LOW3.cyc
25516461 LOW4.cyc
33670575 LOW5.cyc
31290129 LOW6.cyc
31834280 LOW7.cyc
35086535 LOW8.cyc
37314320 LOW9.cyc
36453183 MID1.cyc
11 MID2.cyc
11 MID3.cyc
11 MID4.cyc
11 MID5.cyc
11 MID6.cyc

the size of the cycbuf directory 'du -hs' is 685G, this also reports
'zfs list' for the dataset. with compressratio 2.97x there :)

Re: CNFS buffer consuming more disk space than the file size?

<uVFTM.7806$4rg.6201@fx09.ams4>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2272&group=news.software.nntp#2272

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!news.furie.org.uk!usenet.goja.nl.eu.org!weretis.net!feeder8.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!peer01.ams1!peer.ams1.xlned.com!news.xlned.com!peer03.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx09.ams4.POSTED!not-for-mail
MIME-Version: 1.0
User-Agent: NoZilla/3.11 (Hackint; Unicorn; rv:0.8.15) go-while/19720229
NewsRW/4.2.0
Subject: Re: CNFS buffer consuming more disk space than the file size?
Content-Language: en-US
Newsgroups: news.software.nntp
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
<XSFTM.7805$4rg.2638@fx09.ams4>
From: no-re...@no.spam (Billy G. (go-while))
Organization: github.com/go-while
In-Reply-To: <XSFTM.7805$4rg.2638@fx09.ams4>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 2
Message-ID: <uVFTM.7806$4rg.6201@fx09.ams4>
X-Complaints-To: abuse@blocknews.net
NNTP-Posting-Date: Thu, 05 Oct 2023 21:30:02 UTC
Date: Thu, 5 Oct 2023 23:42:17 +0200
X-Received-Bytes: 976
 by: Billy G. (go-while) - Thu, 5 Oct 2023 21:42 UTC

to add a note, the buffers holding about 35G are 100% full. but du -hs
shows the compressed size.

Re: CNFS buffer consuming more disk space than the file size?

<ufnakv$14o$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2273&group=news.software.nntp#2273

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Thu, 5 Oct 2023 21:42:55 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ufnakv$14o$1@nnrp.usenet.blueworldhosting.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com> <XSFTM.7805$4rg.2638@fx09.ams4>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Thu, 5 Oct 2023 21:42:55 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="1176"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:P/yB4CrZ8OsDluQsYJBsrKGRgHw= sha256:prh3UIzyDIkpcDoYcs4s8SNSRWeCYbX/p+BFsRN1fd8=
sha1:HcYs7bS9LO3uqBJmOfHiz+5Fxwg= sha256:ZVLE+TRK0te9aCB6MI/nX0VRfvI8Jxi2BpybLzEQpKs=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Thu, 5 Oct 2023 21:42 UTC

On Oct 5, 2023 at 4:39:32 PM CDT, ""Billy G." <go-while)" <no-reply@no.spam>
wrote:

> On 05.10.23 16:25, Jesse Rehmer wrote:
>> This could be a ZFS/FreeBSD issue, but figure I'd start here to see if anyone
>> has seen this behavior before... I started to run tight on disk space on one
>> of my servers and noticed that the CNFS buffers were using an amount of disk
>> space that is larger than the 'actual' file size of the buffer.
>>
>> I have this cycbuff.conf entry:
>>
>> cycbuff:SBIN2:/usr/local/news/spool/bin1:102400000
>>
>> Looking at this file with 'ls' the size lines up:
>>
>> -rw-r--r-- 1 news news 104857600000 Oct 4 15:14 bin1
>>
>> However, when I check actual disk space used with du it is much greater:
>>
>> # du -h bin1
>> 129G bin1
>>
>> # du bin1
>> 135118061 bin1
>>
>> This buffer contained 'junk' so I decided to remove it, and did reclaim 129GB
>> of space on the filesystem. How is this possible?
>
> you have compression enabled?
> how did you create the cycbuf?
> as sparse via fallocate or filled via dd from /dev/zero?
>
> i have zfs with compression lz4 and as you see 'ls -l' reports this size
> but 'du -h' reports real used spaced from file.
>
> ls -l
> total 717894363
> -rw-r--r-- 1 news news 107374182400 Oct 25 2022 BIG1.cyc
> -rw-rw---- 1 news news 107374182400 Oct 27 2022 BIG2.cyc
> -rw-rw---- 1 news news 107374182400 Dec 10 2022 BIG3.cyc
> -rw-r--r-- 1 news news 107374182400 May 21 03:27 BIG4.cyc
> -rw-r--r-- 1 news news 107374182400 Nov 1 2022 BIG5.cyc
> -rw-r--r-- 1 news news 107374182400 Nov 1 2022 BIG6.cyc
> -rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW10.cyc
> -rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW11.cyc
> -rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW12.cyc
> -rw-rw---- 1 news news 107374182400 Oct 27 2022 LOW13.cyc
> -rw-rw---- 1 news news 107374182400 Oct 28 2022 LOW14.cyc
> -rw-rw---- 1 news news 107374182400 Nov 4 2022 LOW15.cyc
> -rw-rw---- 1 news news 107374182400 Apr 10 18:08 LOW16.cyc
> -rw-rw---- 1 news news 107374182400 May 21 03:27 LOW17.cyc
> -rw-rw---- 1 news news 107374182400 Nov 6 2022 LOW18.cyc
> -rw-rw---- 1 news news 107374182400 Nov 6 2022 LOW19.cyc
> -rw-r--r-- 1 news news 107374182400 Oct 23 2022 LOW1.cyc
> -rw-rw---- 1 news news 107374182400 Nov 6 2022 LOW20.cyc
> -rw-r--r-- 1 news news 107374182400 Oct 23 2022 LOW2.cyc
> -rw-rw---- 1 news news 107374182400 Oct 24 2022 LOW3.cyc
> -rw-rw---- 1 news news 107374182400 Oct 24 2022 LOW4.cyc
> -rw-rw---- 1 news news 107374182400 Oct 25 2022 LOW5.cyc
> -rw-rw---- 1 news news 107374182400 Oct 25 2022 LOW6.cyc
> -rw-rw---- 1 news news 107374182400 Oct 26 2022 LOW7.cyc
> -rw-rw---- 1 news news 107374182400 Oct 26 2022 LOW8.cyc
> -rw-rw---- 1 news news 107374182400 Oct 26 2022 LOW9.cyc
> -rw-r--r-- 1 news news 107374182400 May 21 03:26 MID1.cyc
> -rw-rw---- 1 news news 107374182400 Oct 15 2022 MID2.cyc
> -rw-rw---- 1 news news 107374182400 Oct 15 2022 MID3.cyc
> -rw-rw---- 1 news news 107374182400 Oct 15 2022 MID4.cyc
> -rw-rw---- 1 news news 107374182400 Oct 15 2022 MID5.cyc
> -rw-rw---- 1 news news 107374182400 Oct 15 2022 MID6.cyc
>
>
> du -h *
> 59G BIG1.cyc
> 50G BIG2.cyc
> 52G BIG3.cyc
> 4.2G BIG4.cyc
> 33K BIG5.cyc
> 33K BIG6.cyc
> 34G LOW10.cyc
> 34G LOW11.cyc
> 35G LOW12.cyc
> 36G LOW13.cyc
> 36G LOW14.cyc
> 29G LOW15.cyc
> 29G LOW16.cyc
> 4.7G LOW17.cyc
> 11K LOW18.cyc
> 11K LOW19.cyc
> 22G LOW1.cyc
> 11K LOW20.cyc
> 25G LOW2.cyc
> 21G LOW3.cyc
> 25G LOW4.cyc
> 33G LOW5.cyc
> 30G LOW6.cyc
> 31G LOW7.cyc
> 34G LOW8.cyc
> 36G LOW9.cyc
> 35G MID1.cyc
> 11K MID2.cyc
> 11K MID3.cyc
> 11K MID4.cyc
> 11K MID5.cyc
> 11K MID6.cyc
>
> and 'du -b' reports other size...
>
> du -b *
> 107374182400 BIG1.cyc
> 107374182400 BIG2.cyc
> 107374182400 BIG3.cyc
> 107374182400 BIG4.cyc
> 107374182400 BIG5.cyc
> 107374182400 BIG6.cyc
> 107374182400 LOW10.cyc
> 107374182400 LOW11.cyc
> 107374182400 LOW12.cyc
> 107374182400 LOW13.cyc
> 107374182400 LOW14.cyc
> 107374182400 LOW15.cyc
> 107374182400 LOW16.cyc
> 107374182400 LOW17.cyc
> 107374182400 LOW18.cyc
> 107374182400 LOW19.cyc
> 107374182400 LOW1.cyc
> 107374182400 LOW20.cyc
> 107374182400 LOW2.cyc
> 107374182400 LOW3.cyc
> 107374182400 LOW4.cyc
> 107374182400 LOW5.cyc
> 107374182400 LOW6.cyc
> 107374182400 LOW7.cyc
> 107374182400 LOW8.cyc
> 107374182400 LOW9.cyc
> 107374182400 MID1.cyc
> 107374182400 MID2.cyc
> 107374182400 MID3.cyc
> 107374182400 MID4.cyc
> 107374182400 MID5.cyc
> 107374182400 MID6.cyc
>
> and finaly du *
>
> du *
> 61314603 BIG1.cyc
> 51517375 BIG2.cyc
> 53918660 BIG3.cyc
> 4385048 BIG4.cyc
> 33 BIG5.cyc
> 33 BIG6.cyc
> 35449737 LOW10.cyc
> 35492501 LOW11.cyc
> 36521650 LOW12.cyc
> 37057675 LOW13.cyc
> 36731237 LOW14.cyc
> 30020730 LOW15.cyc
> 29718199 LOW16.cyc
> 4829014 LOW17.cyc
> 11 LOW18.cyc
> 11 LOW19.cyc
> 22134082 LOW1.cyc
> 11 LOW20.cyc
> 26145542 LOW2.cyc
> 21492682 LOW3.cyc
> 25516461 LOW4.cyc
> 33670575 LOW5.cyc
> 31290129 LOW6.cyc
> 31834280 LOW7.cyc
> 35086535 LOW8.cyc
> 37314320 LOW9.cyc
> 36453183 MID1.cyc
> 11 MID2.cyc
> 11 MID3.cyc
> 11 MID4.cyc
> 11 MID5.cyc
> 11 MID6.cyc
>
> the size of the cycbuf directory 'du -hs' is 685G, this also reports
> 'zfs list' for the dataset. with compressratio 2.97x there :)

I do have compression enabled and created the buffers with dd from /dev/zero.

In your examples the actual size is smaller, which is what I would expect, but
my situation is reverse where the actual size used is larger by about 30%.

Re: CNFS buffer consuming more disk space than the file size?

<PPZTM.104749$vEK1.10266@fx08.ams4>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2287&group=news.software.nntp#2287

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!peer02.ams1!peer.ams1.xlned.com!news.xlned.com!peer02.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx08.ams4.POSTED!not-for-mail
MIME-Version: 1.0
User-Agent: NoZilla/3.11 (Hackint; Unicorn; rv:0.8.15) go-while/19720229
NewsRW/4.2.0
Subject: Re: CNFS buffer consuming more disk space than the file size?
Content-Language: en-US
Newsgroups: news.software.nntp
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
<XSFTM.7805$4rg.2638@fx09.ams4>
<ufnakv$14o$1@nnrp.usenet.blueworldhosting.com>
From: no-re...@no.spam (Billy G. (go-while))
Organization: github.com/go-while
In-Reply-To: <ufnakv$14o$1@nnrp.usenet.blueworldhosting.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 19
Message-ID: <PPZTM.104749$vEK1.10266@fx08.ams4>
X-Complaints-To: abuse@blocknews.net
NNTP-Posting-Date: Fri, 06 Oct 2023 20:09:19 UTC
Date: Fri, 6 Oct 2023 22:22:12 +0200
X-Received-Bytes: 1808
 by: Billy G. (go-while) - Fri, 6 Oct 2023 20:22 UTC

On 05.10.23 23:42, Jesse Rehmer wrote:
> I do have compression enabled and created the buffers with dd from /dev/zero.
>
> In your examples the actual size is smaller, which is what I would expect, but
> my situation is reverse where the actual size used is larger by about 30%.

weird and can't get my head around.
inn would not start if filesize from conf and disk dont match.
i believe, i had such issue once while playing and copied wrong values
and inn failed to start or initialize the buffer.
maybe there is no check if it is already initialized and file has grown
somehow.
but how could it grow bigger, i have no idea. should not be possible.
cycbuff puts data in some kind of blocks but did not dive deep enough to
get a better understanding.

would be cool if you did not delete the file, so we could further
investigate it.

Re: CNFS buffer consuming more disk space than the file size?

<ufpq3p$13kp$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2288&group=news.software.nntp#2288

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Fri, 6 Oct 2023 20:19:05 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ufpq3p$13kp$1@nnrp.usenet.blueworldhosting.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com> <XSFTM.7805$4rg.2638@fx09.ams4> <ufnakv$14o$1@nnrp.usenet.blueworldhosting.com> <PPZTM.104749$vEK1.10266@fx08.ams4>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Fri, 6 Oct 2023 20:19:05 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="36505"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:Nda3LkY5+xjUr8xkhbo0cnQWrZM= sha256:sYWkjS2/UV8qzIRLK14qJBiFrZoWq38pVMj+wEMhKAA=
sha1:TmxYarxv5cXar3BYrSGELg+hwrw= sha256:Yk9Y2BarM0xqw17U0eYNaHDywxxqMnJdtp4Tv8jLs6c=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Fri, 6 Oct 2023 20:19 UTC

On Oct 6, 2023 at 3:22:12 PM CDT, ""Billy G." <go-while)" <no-reply@no.spam>
wrote:

> On 05.10.23 23:42, Jesse Rehmer wrote:
>> I do have compression enabled and created the buffers with dd from /dev/zero.
>>
>> In your examples the actual size is smaller, which is what I would expect, but
>> my situation is reverse where the actual size used is larger by about 30%.
>
>
> weird and can't get my head around.
> inn would not start if filesize from conf and disk dont match.
> i believe, i had such issue once while playing and copied wrong values
> and inn failed to start or initialize the buffer.
> maybe there is no check if it is already initialized and file has grown
> somehow.
> but how could it grow bigger, i have no idea. should not be possible.
> cycbuff puts data in some kind of blocks but did not dive deep enough to
> get a better understanding.
>
> would be cool if you did not delete the file, so we could further
> investigate it.

I know, I know, but I needed the space and was frustrated. I created a smaller
one in its place, I'll monitor the size.

I do not understand very much about how filesystems work, just the very basic
things taught to me 30 years ago, but I went looking for ZFS related problems
online. I found a case or two that on the surface sounded similar but were
referring to odd situations where compression overhead would end being greater
than the compression provided, and these issues seemed to stem from something
relating to block sizes that I didn't understand.

I also wondered if it could be some weird issue with ZFS snapshots and the way
CNFS stores data, maybe the snapshot removal isn't clearing up all the
referenced blocks it should, but this seems far-fetched.

Re: CNFS buffer consuming more disk space than the file size?

<ug236i$1am2$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2342&group=news.software.nntp#2342

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Mon, 9 Oct 2023 23:43:14 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ug236i$1am2$1@nnrp.usenet.blueworldhosting.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com> <XSFTM.7805$4rg.2638@fx09.ams4> <ufnakv$14o$1@nnrp.usenet.blueworldhosting.com> <PPZTM.104749$vEK1.10266@fx08.ams4>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 9 Oct 2023 23:43:14 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="43714"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:Do16tCny/daI/G1cYFFW2vkZmp4= sha256:uMi8+pcYzQU2fE+st8DayjK1d77ueuJpnhuUIuQwMWg=
sha1:mKC7BzOWkkL3i1NRIR58FoHXKtk= sha256:c7teNtxisug+cBH7aASrZaBwLVFO3k2RNIEtLLPAgG8=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Mon, 9 Oct 2023 23:43 UTC

On Oct 6, 2023 at 3:22:12 PM CDT, ""Billy G." <go-while)" <no-reply@no.spam>
wrote:

> On 05.10.23 23:42, Jesse Rehmer wrote:
>> I do have compression enabled and created the buffers with dd from /dev/zero.
>>
>> In your examples the actual size is smaller, which is what I would expect, but
>> my situation is reverse where the actual size used is larger by about 30%.
>
>
> weird and can't get my head around.
> inn would not start if filesize from conf and disk dont match.
> i believe, i had such issue once while playing and copied wrong values
> and inn failed to start or initialize the buffer.
> maybe there is no check if it is already initialized and file has grown
> somehow.
> but how could it grow bigger, i have no idea. should not be possible.
> cycbuff puts data in some kind of blocks but did not dive deep enough to
> get a better understanding.
>
> would be cool if you did not delete the file, so we could further
> investigate it.

The new buffer has wrapped some times and is now larger than its 'apparent'
size:

[news@spool1 ~/spool]$ du cnfs2
11739697 cnfs2
[news@spool1 ~/spool]$ du -A cnfs2
10000000 cnfs2

$ cnfsstat
Class SBIN for groups matching "control.cancel,junk,alt.binaries.*"
Buffer SBIN1, size: 9.54 GBytes, position: 7.32 GBytes 12.77 cycles
Newest: 2023-10-09 18:42:21, 0 days, 0:00:02 ago

Re: CNFS buffer consuming more disk space than the file size?

<Yb7VM.126743$iKfd.14563@fx05.ams4>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2343&group=news.software.nntp#2343

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!newsreader4.netcologne.de!news.netcologne.de!peer02.ams1!peer.ams1.xlned.com!news.xlned.com!peer01.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx05.ams4.POSTED!not-for-mail
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: CNFS buffer consuming more disk space than the file size?
Content-Language: en-US
Newsgroups: news.software.nntp
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
<XSFTM.7805$4rg.2638@fx09.ams4>
<ufnakv$14o$1@nnrp.usenet.blueworldhosting.com>
<PPZTM.104749$vEK1.10266@fx08.ams4>
<ug236i$1am2$1@nnrp.usenet.blueworldhosting.com>
From: no-re...@no.spam (Billy G. (go-while))
Organization: github.com/go-while
In-Reply-To: <ug236i$1am2$1@nnrp.usenet.blueworldhosting.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 27
Message-ID: <Yb7VM.126743$iKfd.14563@fx05.ams4>
X-Complaints-To: abuse@blocknews.net
NNTP-Posting-Date: Tue, 10 Oct 2023 07:38:32 UTC
Date: Tue, 10 Oct 2023 09:38:12 +0200
X-Received-Bytes: 1829
 by: Billy G. (go-while) - Tue, 10 Oct 2023 07:38 UTC

On 10.10.23 01:43, Jesse Rehmer wrote:
> The new buffer has wrapped some times and is now larger than its 'apparent'
> size:
>
> [news@spool1 ~/spool]$ du cnfs2
> 11739697 cnfs2
> [news@spool1 ~/spool]$ du -A cnfs2
> 10000000 cnfs2
>
> $ cnfsstat
> Class SBIN for groups matching "control.cancel,junk,alt.binaries.*"
> Buffer SBIN1, size: 9.54 GBytes, position: 7.32 GBytes 12.77 cycles
> Newest: 2023-10-09 18:42:21, 0 days, 0:00:02 ago

my OS lacks 'du -A'. i'm not on xBSD :/

what does 'du -b cnfs2' (bytes size) or 'du -h cnfs2' (human read.)?

as well as 'ls -l cnfs2' and 'ls -lh cnfs2'?

i#d mostly bet it is an optical illusion.

the file is only 10g, can you produce a copy for us?

or just move a copy to another filesystem, ext4?

what size value you set in cycbuff.conf to this buffer?

Re: CNFS buffer consuming more disk space than the file size?

<ug3e7l$22us$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2344&group=news.software.nntp#2344

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Tue, 10 Oct 2023 11:57:41 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ug3e7l$22us$1@nnrp.usenet.blueworldhosting.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com> <PPZTM.104749$vEK1.10266@fx08.ams4> <ug236i$1am2$1@nnrp.usenet.blueworldhosting.com> <Yb7VM.126743$iKfd.14563@fx05.ams4>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 10 Oct 2023 11:57:41 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="68572"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:xFOtdw/77uojQ/AbhqUzXuXd3r0= sha256:nIUg7GLtTapBvcHEdnRUCxNUJ/27VD/koCn7OIBEoUc=
sha1:kFpXEXvmKaZLC6KDRWzewmYByQE= sha256:HSGxdYslE+KgZCWE7b+i/x9j7ueZ025uNRlLJKS/JvA=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Tue, 10 Oct 2023 11:57 UTC

On Oct 10, 2023 at 2:38:12 AM CDT, ""Billy G." <go-while)" <no-reply@no.spam>
wrote:

> On 10.10.23 01:43, Jesse Rehmer wrote:
>> The new buffer has wrapped some times and is now larger than its 'apparent'
>> size:
>>
>> [news@spool1 ~/spool]$ du cnfs2
>> 11739697 cnfs2
>> [news@spool1 ~/spool]$ du -A cnfs2
>> 10000000 cnfs2
>>
>> $ cnfsstat
>> Class SBIN for groups matching "control.cancel,junk,alt.binaries.*"
>> Buffer SBIN1, size: 9.54 GBytes, position: 7.32 GBytes 12.77 cycles
>> Newest: 2023-10-09 18:42:21, 0 days, 0:00:02 ago
>
> my OS lacks 'du -A'. i'm not on xBSD :/
>
> what does 'du -b cnfs2' (bytes size) or 'du -h cnfs2' (human read.)?
>
> as well as 'ls -l cnfs2' and 'ls -lh cnfs2'?
>
> i#d mostly bet it is an optical illusion.
>
> the file is only 10g, can you produce a copy for us?
>
> or just move a copy to another filesystem, ext4?
>
> what size value you set in cycbuff.conf to this buffer?

The size in cycbuff.conf is 10000000.

From the FreeBSD/ZFS box:

$ ls -laF cnfs2
-rw-r--r-- 1 news news 10240000000 Oct 10 06:08 cnfs2

$ ls -lh cnfs2
-rw-r--r-- 1 news news 9.5G Oct 10 06:08 cnfs2

FreeBSD's du does not have -b (it's the default)

$ du cnfs2
11967209 cnfs2

$ du -A cnfs2
10000000 cnfs2

$ du -h cnfs2
11G cnfs2

$ du -Ah cnfs2
9.5G cnfs2

I scp'd the file over to a Linux box with ext4, there it reports as I would
expect:

# du -h cnfs2
9.6G cnfs2

# du -b cnfs2
10240000000 cnfs2

# ls -lb cnfs2
-rw-r--r--. 1 jesse jesse 10240000000 Oct 10 06:52 cnfs2

Re: CNFS buffer consuming more disk space than the file size?

<ug3k4l$2clq$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2345&group=news.software.nntp#2345

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!news.1d4.us!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Tue, 10 Oct 2023 13:38:29 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ug3k4l$2clq$1@nnrp.usenet.blueworldhosting.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com> <PPZTM.104749$vEK1.10266@fx08.ams4> <ug236i$1am2$1@nnrp.usenet.blueworldhosting.com> <Yb7VM.126743$iKfd.14563@fx05.ams4> <ug3e7l$22us$1@nnrp.usenet.blueworldhosting.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 10 Oct 2023 13:38:29 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="78522"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:QUIlwRyLpZjBnAhylG7DyMz0wVY= sha256:wwejuMLgLtZgRxjpDyfDaKy3CHHslTohcgGsZionM9g=
sha1:wS35Y3E/C1j7Y9RcF6XBzsUFF/Q= sha256:NvgeXWMc7Jx2x+kZABrF33SzGuiwrKaGDOt7Ps+QbGI=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Tue, 10 Oct 2023 13:38 UTC

On Oct 10, 2023 at 6:57:41 AM CDT, "Jesse Rehmer"
<jesse.rehmer@blueworldhosting.com> wrote:

> On Oct 10, 2023 at 2:38:12 AM CDT, ""Billy G." <go-while)" <no-reply@no.spam>
> wrote:
>
>> On 10.10.23 01:43, Jesse Rehmer wrote:
>>> The new buffer has wrapped some times and is now larger than its 'apparent'
>>> size:
>>>
>>> [news@spool1 ~/spool]$ du cnfs2
>>> 11739697 cnfs2
>>> [news@spool1 ~/spool]$ du -A cnfs2
>>> 10000000 cnfs2
>>>
>>> $ cnfsstat
>>> Class SBIN for groups matching "control.cancel,junk,alt.binaries.*"
>>> Buffer SBIN1, size: 9.54 GBytes, position: 7.32 GBytes 12.77 cycles
>>> Newest: 2023-10-09 18:42:21, 0 days, 0:00:02 ago
>>
>> my OS lacks 'du -A'. i'm not on xBSD :/
>>
>> what does 'du -b cnfs2' (bytes size) or 'du -h cnfs2' (human read.)?
>>
>> as well as 'ls -l cnfs2' and 'ls -lh cnfs2'?
>>
>> i#d mostly bet it is an optical illusion.
>>
>> the file is only 10g, can you produce a copy for us?
>>
>> or just move a copy to another filesystem, ext4?
>>
>> what size value you set in cycbuff.conf to this buffer?
>
> The size in cycbuff.conf is 10000000.
>
> From the FreeBSD/ZFS box:
>
> $ ls -laF cnfs2
> -rw-r--r-- 1 news news 10240000000 Oct 10 06:08 cnfs2
>
> $ ls -lh cnfs2
> -rw-r--r-- 1 news news 9.5G Oct 10 06:08 cnfs2
>
> FreeBSD's du does not have -b (it's the default)
>
> $ du cnfs2
> 11967209 cnfs2
>
> $ du -A cnfs2
> 10000000 cnfs2
>
> $ du -h cnfs2
> 11G cnfs2
>
> $ du -Ah cnfs2
> 9.5G cnfs2
>
>
> I scp'd the file over to a Linux box with ext4, there it reports as I would
> expect:
>
> # du -h cnfs2
> 9.6G cnfs2
>
> # du -b cnfs2
> 10240000000 cnfs2
>
> # ls -lb cnfs2
> -rw-r--r--. 1 jesse jesse 10240000000 Oct 10 06:52 cnfs2

I realized after I sent this that bytes isn't the default of du in FreeBSD,
derp. Interestingly, I recopied this file back to the original box, in a
different folder, and it's a smaller size.

There must be something with compression overhead or something that ZFS isn't
letting go of?

# du -A cnfs2
10000000 cnfs2

# du cnfs2
9781913 cnfs2

# du -h cnfs2
9.3G cnfs2

Re: CNFS buffer consuming more disk space than the file size?

<87wmvucwc7.fsf@hope.eyrie.org>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2349&group=news.software.nntp#2349

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!paganini.bofh.team!news.killfile.org!news.eyrie.org!.POSTED!not-for-mail
From: eag...@eyrie.org (Russ Allbery)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Tue, 10 Oct 2023 08:48:24 -0700
Organization: The Eyrie
Message-ID: <87wmvucwc7.fsf@hope.eyrie.org>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
<XSFTM.7805$4rg.2638@fx09.ams4>
<ufnakv$14o$1@nnrp.usenet.blueworldhosting.com>
<PPZTM.104749$vEK1.10266@fx08.ams4>
<ug236i$1am2$1@nnrp.usenet.blueworldhosting.com>
Mime-Version: 1.0
Content-Type: text/plain
Injection-Info: hope.eyrie.org;
logging-data="4795"; mail-complaints-to="news@eyrie.org"
User-Agent: Gnus/5.13 (Gnus v5.13)
Cancel-Lock: sha1:AZ0+1jC+gLeBS2sVegEqZBfQyjY=
 by: Russ Allbery - Tue, 10 Oct 2023 15:48 UTC

Jesse Rehmer <jesse.rehmer@blueworldhosting.com> writes:

> The new buffer has wrapped some times and is now larger than its 'apparent'
> size:

> [news@spool1 ~/spool]$ du cnfs2
> 11739697 cnfs2
> [news@spool1 ~/spool]$ du -A cnfs2
> 10000000 cnfs2

> $ cnfsstat
> Class SBIN for groups matching "control.cancel,junk,alt.binaries.*"
> Buffer SBIN1, size: 9.54 GBytes, position: 7.32 GBytes 12.77 cycles
> Newest: 2023-10-09 18:42:21, 0 days, 0:00:02 ago

On an ext4 file system, I only see the expected file system overhead and
block size rounding.

$ du *
10485764 B01
10485764 B02
10485764 B03
10485764 B04
$ du --apparent-size *
10485760 B01
10485760 B02
10485760 B03
10485760 B04

That's just one data point, but that does support my feeling that whatever
is going on here is something related to ZFS specifically, as opposed to,
say, a bug in INN that's causing it to write past the end of the cycbuff
or something.

--
Russ Allbery (eagle@eyrie.org) <https://www.eyrie.org/~eagle/>

Please post questions rather than mailing me directly.
<https://www.eyrie.org/~eagle/faqs/questions.html> explains why.

Re: CNFS buffer consuming more disk space than the file size?

<UbfVM.47671$8qf7.19118@fx06.ams4>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2350&group=news.software.nntp#2350

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!feeder1.feed.usenet.farm!feed.usenet.farm!peer01.ams4!peer.am4.highwinds-media.com!news.highwinds-media.com!fx06.ams4.POSTED!not-for-mail
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: CNFS buffer consuming more disk space than the file size?
Content-Language: en-US
Newsgroups: news.software.nntp
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com>
<PPZTM.104749$vEK1.10266@fx08.ams4>
<ug236i$1am2$1@nnrp.usenet.blueworldhosting.com>
<Yb7VM.126743$iKfd.14563@fx05.ams4>
<ug3e7l$22us$1@nnrp.usenet.blueworldhosting.com>
<ug3k4l$2clq$1@nnrp.usenet.blueworldhosting.com>
From: no-re...@no.spam (Billy G. (go-while))
Organization: github.com/go-while
In-Reply-To: <ug3k4l$2clq$1@nnrp.usenet.blueworldhosting.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Lines: 9
Message-ID: <UbfVM.47671$8qf7.19118@fx06.ams4>
X-Complaints-To: abuse@blocknews.net
NNTP-Posting-Date: Tue, 10 Oct 2023 16:44:36 UTC
Date: Tue, 10 Oct 2023 18:44:33 +0200
X-Received-Bytes: 1200
 by: Billy G. (go-while) - Tue, 10 Oct 2023 16:44 UTC

On 10.10.23 15:38, Jesse Rehmer wrote:
>
> There must be something with compression overhead or something that ZFS isn't
> letting go of?

you do zfs snapshots for example, daily or hourly?

maybe does bsd refer to snapshot size as filesize?

Re: CNFS buffer consuming more disk space than the file size?

<ug3vrb$12q3$1@nnrp.usenet.blueworldhosting.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=2351&group=news.software.nntp#2351

 copy link   Newsgroups: news.software.nntp
Path: i2pn2.org!i2pn.org!weretis.net!feeder6.news.weretis.net!1.us.feeder.erje.net!feeder.erje.net!usenet.blueworldhosting.com!diablo1.usenet.blueworldhosting.com!nnrp.usenet.blueworldhosting.com!.POSTED!not-for-mail
From: jesse.re...@blueworldhosting.com (Jesse Rehmer)
Newsgroups: news.software.nntp
Subject: Re: CNFS buffer consuming more disk space than the file size?
Date: Tue, 10 Oct 2023 16:58:19 -0000 (UTC)
Organization: BWH Usenet (https://usenet.blueworldhosting.com)
Message-ID: <ug3vrb$12q3$1@nnrp.usenet.blueworldhosting.com>
References: <ufmh0l$2bau$1@nnrp.usenet.blueworldhosting.com> <ug3e7l$22us$1@nnrp.usenet.blueworldhosting.com> <ug3k4l$2clq$1@nnrp.usenet.blueworldhosting.com> <UbfVM.47671$8qf7.19118@fx06.ams4>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=fixed
Content-Transfer-Encoding: 8bit
Injection-Date: Tue, 10 Oct 2023 16:58:19 -0000 (UTC)
Injection-Info: nnrp.usenet.blueworldhosting.com;
logging-data="35651"; mail-complaints-to="usenet@blueworldhosting.com"
User-Agent: Usenapp for MacOS
Cancel-Lock: sha1:HqyDAw73dl7Uce4bXsONnhe0Htw= sha256:Pb6A4MCWPb8CmtILQXo+1cJeXWQqzWfFx7BmyW/Szqk=
sha1:WXC5GHjQcxyu4tjME+YO6cA4vpY= sha256:4DEGt0flWYuss91MDx5wDGMBorW3Yf7mdjtmak0lOUk=
X-Usenapp: v1.27.1/d - Full License
 by: Jesse Rehmer - Tue, 10 Oct 2023 16:58 UTC

On Oct 10, 2023 at 11:44:33 AM CDT, ""Billy G." <go-while)" <no-reply@no.spam>
wrote:

> On 10.10.23 15:38, Jesse Rehmer wrote:
>>
>> There must be something with compression overhead or something that ZFS isn't
>> letting go of?
>
> you do zfs snapshots for example, daily or hourly?
>
> maybe does bsd refer to snapshot size as filesize?

The only time I take snapshots is the automatic ones freebsd-update performs.

Something is wonky with ZFS, but what I have no idea, I just copied the file
back from Linux to FreeBSD:

# du -A cnfs2
10000000 cnfs2

# du cnfs2
20224841 cnfs2

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor