Is INN the only usenet news software for linux? I've done some searching
around and can't seam to find much of anything else. There is citadel
but that is a entire groupware sweet in it self. There is leafnode but I
don't know if you can host local groups with that or not.

INN
Diablo
DNews
C News
WendzelNNTPd

Historic Software

B News
A News

--
Diese E-Mail wurde von AVG auf Viren geprüft.
http://www.avg.com

On 7/29/22 07:22, Matthias Meyser wrote:
> INN
> Diablo
> DNews
> C News
> WendzelNNTPd
>
> Historic Software
>
> B News
> A News

# Rslite
has PHP front end and automated email registration system
usenet group : rocksolid.nodes
web : https://novabbs.com
faq : https://novabbs.com/common/faq.txt
code : https://github.com/novabbs/rocksolid-light

# Papercut NNTP server
has forum front end and automated registration system
web : http://pessoal.org/papercut/
code : https://github.com/jpm/papercut

--

G.K.

On Fri, 29 Jul 2022 14:22:38 +0200, Matthias Meyser wrote:

> DNews
>

This one is excellent, fast, ultra nice and light with system resources.
Has a couple cons however, not ipv6 capable, nor 64bit capable.

We still run it (since the 90s) but all our attempts in past couple years
to move the spool into INN which is ipv6 and 64bit capable, has thus far
failed, it bogs down, and trying to move near TB of data would take
months, as there is no apparent way to convert DNews's spool into a
format INN understands so we have shelved the idea, not too worried since
ipv4 despite exhaustion will be round for a very long time yet anyway.

Le 04/08/2022 à 11:08, noel a écrit :

> format INN understands so we have shelved the idea, not too worried since
> ipv4 despite exhaustion will be round for a very long time yet anyway.

not so long. My Linux User Group FAI (REDbySFR) just set our box as full
IPV6 - no more IPV4 usable address for servers :-( - and no warning :-(

jdd

Hi,

noel a tapoté le 04/08/2022 11:08:
> We still run it (since the 90s) but all our attempts in past couple years
> to move the spool into INN which is ipv6 and 64bit capable, has thus far
> failed, it bogs down, and trying to move near TB of data would take
> months, as there is no apparent way to convert DNews's spool into a
> format INN understands so we have shelved the idea, not too worried since
> ipv4 despite exhaustion will be round for a very long time yet anyway.

The only simple solution would to find how to feed all articles from
Dnews server to the INN2 server.

Or with suck on the new server?
<https://netwinsite.com/dnews/faq3.htm#10>

I had never used suck but there may some users here.

--
Stéphane

On Aug 4, 2022 at 4:49:14 AM CDT, "yamo'" in <tcg4ng$3fr$1@rasp.pasdenom.info>
wrote:

> Hi,
>
> noel a tapoté le 04/08/2022 11:08:
>> We still run it (since the 90s) but all our attempts in past couple years
>> to move the spool into INN which is ipv6 and 64bit capable, has thus far
>> failed, it bogs down, and trying to move near TB of data would take
>> months, as there is no apparent way to convert DNews's spool into a
>> format INN understands so we have shelved the idea, not too worried since
>> ipv4 despite exhaustion will be round for a very long time yet anyway.
>
> The only simple solution would to find how to feed all articles from
> Dnews server to the INN2 server.
>
> Or with suck on the new server?
> <https://netwinsite.com/dnews/faq3.htm#10>
>
> I had never used suck but there may some users here.

Using any of the single threaded tools like suck or pullnews to move TB's
worth of articles is tedious, highly error-prone, and takes forever. There is
a multi-threaded app called multisuck, but there is zero documentation about
it (looks incomplete perhaps) and I couldn't figure out how to make it work.
The main issue is using anything that uses a single thread or connection is
going to be a huge bottleneck when you have millions/billions of articles to
move.

I opted to use multiple instances of pullnews to initially seed the spool. It
crashed a lot and had difficulty with very large groups. After doing as much
as I could with pullnews I started using suck since it uses the history
database and deduplicates before downloading every article. This made filling
in the holes that were left with pullnews less taxing.

It's been a few months of going heavy at it and then a week in between of
validating, testing, etc. I think I'm about 1/3 of the way through the Big8
hierarchies.

Hi Jesse,

> I opted to use multiple instances of pullnews to initially seed the spool. It
> crashed a lot and had difficulty with very large groups.

Do you remember the errors you got?
They are maybe worthwhile fixing in pullnews.

What kind of "difficulty" is there with large groups? (resuming after a
crash? normally the article numbers of latest downloaded articles are
saved in the pullnews.marks file, making it easier to continue)

> After doing as much
> as I could with pullnews I started using suck since it uses the history
> database and deduplicates before downloading every article.

"pullnews -O" does that too.

--
Julien ÉLIE

« Ibi etsi uis te non esse sed es ibi. »

On Aug 4, 2022 at 10:55:29 AM CDT, "Julien ÉLIE" in
<tcgq5i$31omo$1@news.trigofacile.com> wrote:

> Hi Jesse,
>
>> I opted to use multiple instances of pullnews to initially seed the spool. It
>> crashed a lot and had difficulty with very large groups.
>
> Do you remember the errors you got?
> They are maybe worthwhile fixing in pullnews.

It could be related to the server on the other end (pulling from a commercial
entity). The one situation that happened a lot, especially with large groups,
was at some point in the session pullnews would just start spewing x's to the
terminal in a loop and not stop until killed. It never gave me an error.
Killing and resuming always seemed to work.

In comparison, suck seems to handle whatever issue pullnews runs into by
eventually cleaning up and reporting a socket error. This made it easier to
see which instances weren't running anymore. When I had 20-50 pullnews
instances running at the same time I would have to connect to each screen to
make sure it wasn't barfing.

>> After doing as much
>> as I could with pullnews I started using suck since it uses the history
>> database and deduplicates before downloading every article.
>
> "pullnews -O" does that too.

It does but I did a few comparisons with pullnews and suck, even when using
the -O flag with pullnews, suck is a lot faster in the overall runtime of
pulling an entire group (tested the same group on empty servers). Internally I
don't know how the NNTP operations compare, but pullnews wasn't as efficient
as suck's method of building the list of articles, deduping, grabbing all
articles, and posting as serial operations led to getting the job done faster.

Hi Jesse,

> It could be related to the server on the other end (pulling from a commercial
> entity). The one situation that happened a lot, especially with large groups,
> was at some point in the session pullnews would just start spewing x's to the
> terminal in a loop and not stop until killed. It never gave me an error.
> Killing and resuming always seemed to work.

Interesting.
I see in the code that if pullnews receives an unexpected response code
when asking for article #i, it will print "x" and try #(i+1).

I've tested a timeout, as you suggested, and I confirm "pullnews -d2"
(debug mode) prints "x" and goes on...

x
DEBUGGING 1 421 [Net::NNTP] Connection closed
x DEBUGGING 2 421 [Net::NNTP] Connection closed
x DEBUGGING 3 421 [Net::NNTP] Connection closed
[...]

This is a bug.
I would just suggest in that case to verify whether the connection were
still active by sending a DATE command through Net::NNTP::date(). If
there is no answer, then we can consider the connection is no longer
here, and we just bail out.
And if there is an answer to DATE, hmmm, we can still be in a deadloop
with a repeated unknown error. Maybe we could also bail out after 10
attempts for instance?

Unfortunately, I do not see how to check that the connection is no
longer active with the Net::NNTP module pullnews uses :-/
It seems to just answer "undef" to each command, but it could be for
another reason. That's why I suggest to ask for the DATE.

> I did a few comparisons with pullnews and suck, even when using
> the -O flag with pullnews, suck is a lot faster in the overall runtime of
> pulling an entire group (tested the same group on empty servers). Internally I
> don't know how the NNTP operations compare, but pullnews wasn't as efficient
> as suck's method of building the list of articles, deduping, grabbing all
> articles, and posting as serial operations led to getting the job done faster.

I've just had a glance at suck source code. It directly runs the native
C function provided by INN (dbzexists) on the history file. No wonder
it is more efficient than pullnews which just runs an NNTP command
(STAT) on the local news server...

Thanks again for your valuable bug report!
Any other issue you remember with pullnews?

--
Julien ÉLIE

« Ibi etsi uis te non esse sed es ibi. »

On Aug 4, 2022 at 1:57:05 PM CDT, "Julien ÉLIE" in
<tch4q1$31vln$1@news.trigofacile.com> wrote:

> I've just had a glance at suck source code. It directly runs the native
> C function provided by INN (dbzexists) on the history file. No wonder
> it is more efficient than pullnews which just runs an NNTP command
> (STAT) on the local news server...
>
> Thanks again for your valuable bug report!
> Any other issue you remember with pullnews?

I think that was the primary issue I ran into. It would be great if an option
could be used to start a new connection should one be considered dead or
unusable.

Resource-wise, pullnews uses exponentially less memory and CPU than suck, so
there's that. Probably a memory leak, but suck really sucks up the memory. ;)

On Thu, 04 Aug 2022 11:32:07 +0200, jdanield wrote:

> Le 04/08/2022 à 11:08, noel a écrit :
>
>> format INN understands so we have shelved the idea, not too worried
>> since ipv4 despite exhaustion will be round for a very long time yet
>> anyway.
>
> not so long. My Linux User Group FAI (REDbySFR) just set our box as full
> IPV6 - no more IPV4 usable address for servers :-( - and no warning :-(
>
> jdd

Most carriers and ISP's have hords of them, some RIR's still issue to new
customers only and a max of /22, else, there are plenty wanting to sell
their unused excess - but do not jump at the first offer, because many of
them are charging criminal rates for them, thhose muppets can keep them :)

Yes, it might be hard for someone just wanting to start out in some parts
of the world, but as for end users, their ipv4 will be prevelaint for a
ver long time.

On Thu, 04 Aug 2022 11:49:14 +0200, yamo' wrote:

> Hi,
>
> noel a tapoté le 04/08/2022 11:08:
>> We still run it (since the 90s) but all our attempts in past couple
>> years to move the spool into INN which is ipv6 and 64bit capable, has
>> thus far failed, it bogs down, and trying to move near TB of data would
>> take months, as there is no apparent way to convert DNews's spool into
>> a format INN understands so we have shelved the idea, not too worried
>> since ipv4 despite exhaustion will be round for a very long time yet
>> anyway.
>
> The only simple solution would to find how to feed all articles from
> Dnews server to the INN2 server.
>
> Or with suck on the new server?
> <https://netwinsite.com/dnews/faq3.htm#10>
>
> I had never used suck but there may some users here.

We tried suck, its the only way they might get into INN, thats where its
bogging down

Hi Noel,

> We tried suck, its the only way they might get into INN, thats where its
> bogging down

I've found out in the Dnews documentation
<https://netwinsite.com/dnews/tellnews.htm> the following command:

"""
tellnews refeed feed.name

Description: This command will re-send all current articles to a
particular news feed. This command is fairly cpu intensive as the
entire history file (maybe 80MB) must be parsed. Also, it does not
apply all the rules associated with the feed you have specified, it only
checks that the group name matches. This command is particularly useful
when setting up the FTS (Full Text Searching) system on a running
server. It allows you to 're-index' all existing items. It should not
be used for other purposes.
"""

Did you try it? Maybe by setting the feed.name entry in newsfeeds.conf
and testing it with a single newsgroup to check whether it works? (and
if it is, then "group *")

Also, DNews seems to store all the articles in files named "db*.itm".
Did you try to have a look at the contents of these files? Maybe the
articles could be extracted from them with an appropriate (simple) script?

I've found out a message of the DNews support:
https://groups.google.com/g/netwin.dnews/c/ISgH6dc5baM/m/CrLz-bU4hDMJ

"""
Create a feed entry like this:
site xxx
groups *
And then write a program to read 'xxx.feed' or 'xxx.send' files
which list each new item, and gives the
offset into the bucket and item length so you can read direct from teh
..itm files, this method is not gauranteed to
work with future versiosn of dnews :-)
"""

Well, that does not seem easy... I'm very surprised by such a response
from the official DNews support. They could at least have natively
provided a tool to export their proprietary data spool. Seems like they
wanted their customes to be captive :-/

--
Julien ÉLIE

« La grippe, ça dure huit jours si on la soigne et une semaine si on ne
fait rien. » (Raymond Devos)

Hi Noel,

> We tried suck, its the only way they might get into INN, thats where its
> bogging down

What problem do you encounter with suck? (except that it may take a long
time to run)

Did you eventually try to run suck in its multifile mode "suck -m"
directly on your DNews news server to extract the articles?
It can also generate batch files suitable for innxmit or rnews with
respectively the "-bi" and "-br" flags.

--
Julien ÉLIE

« Un voyage de mille lieues commence toujours par un premier pas. » (Lao
Zi)

On 8/6/22 06:51, Julien ÉLIE wrote:
> Hi Noel,
>
>> We tried suck, its the only way they might get into INN, thats where its
>> bogging down
>
> What problem do you encounter with suck? (except that it may take a long
> time to run)
>
> Did you eventually try to run suck in its multifile mode "suck -m"
> directly on your DNews news server to extract the articles?
> It can also generate batch files suitable for innxmit or rnews with
> respectively the "-bi" and "-br" flags.
>
I would like to try Dnews running arch linux. How much does it cost and
how would you go about installing it. I was using INN running 1 group
for a nntp only access bbs. Could use wendzelnntp but that does not
allow for multiple servers or linking. I was using mystic bbs to link
the news server to fidonet.

On Sat, 06 Aug 2022 12:39:06 +0200, Julien ÉLIE wrote:

> Hi Noel,
>
>> We tried suck, its the only way they might get into INN, thats where
>> its bogging down
>
> I've found out in the Dnews documentation
> <https://netwinsite.com/dnews/tellnews.htm> the following command:
>
> """
> tellnews refeed feed.name
>
> Description: This command will re-send all current articles to a
> particular news feed. This command is fairly cpu intensive as the
> entire history file (maybe 80MB) must be parsed. Also, it does not
> apply all the rules associated with the feed you have specified, it only
> checks that the group name matches. This command is particularly useful
> when setting up the FTS (Full Text Searching) system on a running
> server. It allows you to 're-index' all existing items. It should not
> be used for other purposes.
> """
>
> Did you try it? Maybe by setting the feed.name entry in newsfeeds.conf
> and testing it with a single newsgroup to check whether it works? (and
> if it is, then "group *")
>
>

I'm very aware of that command and have used it before over the 25 years
of running dnews, it was the first thing I tried and was also a failure
in this case, perhaps it was the INN storage I selected, cant remember
what it was.

>
>
>
> Also, DNews seems to store all the articles in files named "db*.itm".
> Did you try to have a look at the contents of these files? Maybe the
> articles could be extracted from them with an appropriate (simple)
> script?
>

the buckets are plain text articles with internal id's, one bucket can
contain any number of different groups articles

> I've found out a message of the DNews support:
> https://groups.google.com/g/netwin.dnews/c/ISgH6dc5baM/m/CrLz-bU4hDMJ
>
> """
> Create a feed entry like this:
> site xxx groups *
> And then write a program to read 'xxx.feed' or 'xxx.send' files which
> list each new item, and gives the offset into the bucket and item length
> so you can read direct from teh .itm files, this method is not
> gauranteed to work with future versiosn of dnews :-)
> """
>
> Well, that does not seem easy... I'm very surprised by such a response
> from the official DNews support. They could at least have natively
> provided a tool to export their proprietary data spool. Seems like they
> wanted their customes to be captive :-/

yeah, then end its life leaving as is and have told me they have no
intention of open sourcing it so someone might be able to bring it into
the 21st century, thanlkfully modern hardware can run in 32bit mode, and
well, I think I'll have retired by the time ipv4 is no longer of any
importance :)

On Sat, 06 Aug 2022 12:51:31 +0200, Julien ÉLIE wrote:

> Hi Noel,
>
>> We tried suck, its the only way they might get into INN, thats where
>> its bogging down
>
> What problem do you encounter with suck? (except that it may take a long
> time to run)

Its time to run, strangely, and this is likely memory related, it starts
off fast enough, but over time, its just gets slow, and not sure I can
pause new articles for the 6 months it would take to be done, I'd have no
one left using it with that long no new posts lol, I say that because I
dpont want to upset counters, I dont want the sucking or pulling to be
through 2010 articles trhen import this hours new stuff and go back to
2010 :)

>
> Did you eventually try to run suck in its multifile mode "suck -m"
> directly on your DNews news server to extract the articles?
> It can also generate batch files suitable for innxmit or rnews with
> respectively the "-bi" and "-br" flags.

to be honest, I cant recall, so maybe not.

On Sat, 06 Aug 2022 07:58:25 -0400, hurst wrote:

> On 8/6/22 06:51, Julien ÉLIE wrote:
>> Hi Noel,
>>
>>> We tried suck, its the only way they might get into INN, thats where
>>> its bogging down
>>
>> What problem do you encounter with suck? (except that it may take a
>> long time to run)
>>
>> Did you eventually try to run suck in its multifile mode "suck -m"
>> directly on your DNews news server to extract the articles?
>> It can also generate batch files suitable for innxmit or rnews with
>> respectively the "-bi" and "-br" flags.
>>
> I would like to try Dnews running arch linux. How much does it cost and
> how would you go about installing it. I was using INN running 1 group
> for a nntp only access bbs. Could use wendzelnntp but that does not
> allow for multiple servers or linking. I was using mystic bbs to link
> the news server to fidonet.

I dont advise new users try dnews, it has been EOL for 15 years, and the
captain obvious reasons are currently being discussed, the cost wasnt
that much back in hte day I understand, I was lucky, as an early tester
with dnews, I had a free key issued to me, I have it backed up in 5
different places because if it gets corrupted, I wont be able to get a
new one - I dont think anyway, and I think Chris would be amazed someone
was still running it in 2022 :)

On Sat, 06 Aug 2022 12:39:06 +0200, Julien ÉLIE wrote:

> Also, DNews seems to store all the articles in files named "db*.itm".
> Did you try to have a look at the contents of these files? Maybe the
> articles could be extracted from them with an appropriate (simple)
> script?
many of these bucket files are up to 195mb in zise, the majorty though
are around 60-90mb, I've found one from 2018 thats about 10mb gz'd to 2mb
if your curious to see what one looks like.

https://members.ausics.net/noelb/db_1615_2.itm.gz

cheers

Hi Noel,

>> Also, DNews seems to store all the articles in files named "db*.itm".
>> Did you try to have a look at the contents of these files? Maybe the
>> articles could be extracted from them with an appropriate (simple)
>> script?
>
> many of these bucket files are up to 195mb in zise, the majorty though
> are around 60-90mb, I've found one from 2018 thats about 10mb gz'd to 2mb
> if your curious to see what one looks like.
>
> https://members.ausics.net/noelb/db_1615_2.itm.gz

Very interesting, thanks for this sample!

These .itm files contain plain-text articles, dot-stuffed, and each line
ending with CRLF, separated with:

^~00001957:0000389823:017829:

The first field is the number of bytes of the article.
The second field is the article number the article has in the newsgroup
internally represented in DNews by the ID in the third field.

Converting these files to a format understandable by INN is not complicated.

For instance, each .itm file could be converted to an rnews batch file.
The tricky part is that one cannot directly rewrite the DNews
beginning line to "#! rnews 1957" because rnews expects articles in
native format (not dot-stuffed, and each line ending with LF) so the
number of bytes has to be recomputed after the translation.

Would it be something that would interest you, or do you prefer to keep
your DNews installation that already fits your needs?
At least you now know that you can transfer your spool to INN if you
wish for whatever reason in the future:
1/ create the conversion script itm -> rnews format (less than half a
day of work)
2/ run that script to convert all your itm files to rnews batches
3/ transfer these rnews batches to another server (INN or another one)
4/ locally run rnews, which will happily inject your spool in the new
news server

--
Julien ÉLIE

« – Heureusement, nous aurons bientôt le TGV !
– C'est quoi le tégévé ?
– C'est le Transport Gaulois Véloce. » (Astérix)

Hi Noel,

>> Did you try it? Maybe by setting the feed.name entry in newsfeeds.conf
>> and testing it with a single newsgroup to check whether it works? (and
>> if it is, then "group *")
>
> I'm very aware of that command and have used it before over the 25 years
> of running dnews, it was the first thing I tried and was also a failure
> in this case, perhaps it was the INN storage I selected, cant remember
> what it was.

It would have been interesting to have INN logs about your attempt, to
see if it really got it and why it rejected it (unrecognized command?
missing configuration somewhere? ...)

Maybe the refeed command works as-is, and could really be usable.

If you want, I can create a newsgroup on my news server and you can try
to refeed this newsgroup to me (a small newsgroup of a few articles
would be enough to see what's going on - or you could create a
local.refeed testing group, write a few articles there, and refeed it to
me afterwards).

--
Julien ÉLIE

« – Ouvre l'œil, et le bon !
– L'autre, je peux pas encore l'ouvrir, je risque pas de me
tromper ! » (Astérix)

Hi Noel,

>> What problem do you encounter with suck? (except that it may take a long
>> time to run)
>
> Its time to run, strangely, and this is likely memory related, it starts
> off fast enough, but over time, its just gets slow

Like Jesse found out too (high memory consumption).

> and not sure I can
> pause new articles for the 6 months it would take to be done, I'd have no
> one left using it with that long no new posts lol, I say that because I
> dpont want to upset counters, I dont want the sucking or pulling to be
> through 2010 articles trhen import this hours new stuff and go back to
> 2010 :)

Also exactly what Jesse wishes for article numbers being "time-ordered" :)

--
Julien ÉLIE

« Faites le mur, pas la guerre. » (Astérix)

On Aug 7, 2022 at 5:12:29 AM CDT, "Julien ÉLIE" in
<tco36d$3724r$1@news.trigofacile.com> wrote:

> Hi Noel,
>
>>> What problem do you encounter with suck? (except that it may take a long
>>> time to run)
>>
>> Its time to run, strangely, and this is likely memory related, it starts
>> off fast enough, but over time, its just gets slow
>
> Like Jesse found out too (high memory consumption).

Most suck processes I've watched use 2+GB of memory, even with a small list of
groups. I found you can decrease its consumption by disabling the killfiles
(-K).

You may benefit from smaller batch postings. Here is the suck command I'm
using to pull from commercial NNTP servers that do strange things with
connection handling at times:

suck source.host.name -U username -P 'password' -bP 100 -hl localhost -HF
/usr/local/news/db/history -f -lr

These options will have suck do the following:

- Establish connections
- Build the list of articles from each group defined in sucknewsrc
- De-duplicate against the local history database (on an empty server you
should skip this)
- Re-establish the remote connection after de-duplication operations (I found
this was required because the de-dupe can take a long time)
- Download articles from oldest to newest (depending on source article
numbering this may be unreliable)
- After downloading 100 articles it will batch post them to the localhost

If suck gets 'stuck', or otherwise dies due to some error, when you run it
again, add the '-R' flag, it will skip steps two through four above.

>> and not sure I can
>> pause new articles for the 6 months it would take to be done, I'd have no
>> one left using it with that long no new posts lol, I say that because I
>> dpont want to upset counters, I dont want the sucking or pulling to be
>> through 2010 articles trhen import this hours new stuff and go back to
>> 2010 :)
>
> Also exactly what Jesse wishes for article numbers being "time-ordered" :)

It not the best user experience when they load 10,000 articles and get a mix
of date ranges all incomplete until the entire list of articles is processed
by the newsreader.

Cheers,

Jesse

Hi Jesse and Noel,

>>>> What problem do you encounter with suck? (except that it may take a long
>>>> time to run)
>>>
>>> Its time to run, strangely, and this is likely memory related, it starts
>>> off fast enough, but over time, its just gets slow
>>
>> Like Jesse found out too (high memory consumption).
>
> Most suck processes I've watched use 2+GB of memory, even with a small list of
> groups. I found you can decrease its consumption by disabling the killfiles
> (-K).

A recent discussion on a French newsgroup about the newsx program makes
me think that it could maybe interest you?

http://www.kvaleberg.no/newsx.html
http://www.kvaleberg.no/newsx-man.html

It seems to perform like suck, and perhaps better. It could be
worthwhile giving it a try. (It is unfortunately not packaged on all
distributions.)

--
Julien ÉLIE

« À vaincre sans péril, on évite des ennuis ! » (légionnaire romain)