REQUEST FOR DISCUSSION (RFD)
moderated group news.groups.proposals

This is a formal Request For Discussion (RFD) to suspend the charter and
moderation policy of the Usenet newsgroup news.groups.proposals.

BACKGROUND

The unmoderated news.groups newsgroup (formerly known as net.news.group)
historically served as the main venue for discussion of potential new
newsgroups. These discussions were sometimes difficult to follow due to
noise, flames, sporgeries, poorly tagged or structured proposals, etc.
As a result of these issues, the moderated group news.groups.proposals
was created in November 2006. According to its creation RFD, the group
was intended to serve as a "healthy environment where ideas can be
raised, discussed, and developed" without the disruption of "personal
attacks, flames, and other inappropriate content".

news.groups.proposals now serves as the sole "official" venue for all
discussions pertaining to existing or potential proposals to create,
remove, or modify newsgroups in the Big-8 hierarchies (comp, humanities,
misc, news, rec, sci, soc, and talk). What this means in practice is
that while anyone is free to discuss RFDs elsewhere on Usenet, there is
no guarantee that the Big-8 Management Board will monitor those discussions.

PROPOSAL

The Big-8 Management Board proposes to redesignate the unmoderated
news.groups newsgroup as the sole "official" venue for all public
discussions pertaining to existing or potential proposals to create,
remove, or modify newsgroups in the Big-8 hierarchies. (Again, by
"official" we mean only that the Board is guaranteed to monitor
discussions there; users are of course free to hold discussions
elsewhere.) The Board would update its public documentation relating to
Big-8 workflows and policies accordingly, and the charter and moderation
policy for news.groups.proposals would be indefinitely suspended.
Thenceforth all submissions to news.groups.proposals would be
automatically rejected with an explanatory note referring to the outcome
of this RFD and with a suggestion to resubmit to news.groups.

Provided news.groups remains a viable venue, the Board may eventually
issue a subsequent RFD to remove news.groups.proposals. Otherwise, the
Board may issue a subsequent RFD to restore the status quo ante.

RATIONALE

Since 2006, the Big-8 hierarchies have undergone an overall reduction in
their active user base and article traffic. The news.groups newsgroup
has followed this general trend; the past few years have seen some
measure of spam and other off-topic messages, but little of the
acrimonious content that was the main impetus behind the creation of
news.groups.proposals. There is therefore reason to believe that
news.groups could once again function as "a healthy environment" for the
discussion of RFDs.

By contrast, in the past few years news.groups.proposals has had
problems of its own, mostly stemming from its convoluted and antiquated
moderation system. Many submissions have gone missing or unnoticed by
the moderators due to breakdowns in the submission pipeline. While the
current Board members have been working to streamline and modernize the
moderation system they inherited, and to put better fault detection and
prevention measures in place, there is always the risk of further
unexpected technical issues. Technical issues aside, the Board sees no
need to act as gatekeepers for discussions that are, by and large, civil
and constructive.

Although it would be technically possible to designate both news.groups
and news.groups.proposals as "co-official" venues for the discussion of
RFDs, there are obvious benefits to keeping discussions centralized.

PROCEDURE

Those who wish to influence the development of this RFD and its final
resolution should subscribe to news.groups.proposals and participate in
the relevant threads there. To this end, the followup header of this
RFD has been set to news.groups.proposals.

Alternatively, comments may be sent to the Board privately by e-mail at
board@big-8.org. The Board may summarize such comments in subsequent
iterations of this RFD but will not disclose the identities of the
commenters without their express permission.

FURTHER INFORMATION

Home page for news.groups.proposals:
<https://www.big-8.org/wiki/News.groups.proposals>

Charter for news.groups.proposals:
<https://www.big-8.org/wiki/Charter_for_news.groups.proposals>

FAQ for news.groups.proposals:
<https://www.big-8.org/wiki/Frequently_Asked_Questions_for_news.groups.proposals>

Moderation policy for news.groups.proposals:
<https://www.big-8.org/wiki/Moderation_policy_for_news.groups.proposals>

RFD for creation of news.groups.proposals:
<https://ftp.isc.org/usenet/news.announce.newgroups/news/news.groups.proposals>

General information on news.groups: <https://www.big-8.org/wiki/News.groups>

History of news.groups:
<https://www.big-8.org/wiki/Big-8_Usenet_hierarchies#History_of_news.groups>

DISTRIBUTION

This document has been posted to the following newsgroups:

news.announce.newgroups
news.groups.proposals
news.groups
news.admin.hierarchies

PROPONENT

Usenet Big-8 Management Board <board@big-8.org>

CHANGE HISTORY

2022-05-20 1st RFD

--
Usenet Big-8 Management Board
https://www.big-8.org/
board@big-8.org

Except for the occasional technical difficulties with moderation, this
newsgroup is working fine. Moderation ensures that the off-topic posts
which clutter news.groups and many other newsgroups do not appear in
news.groups.proposals.

Would recommend that Big-8 continue their work on streamlining and
modernizing the moderation system.

Also important is to consider the objective of Usenet: to facilitate
discussion. Everything else is just overhead.

Finally, I wonder if there are enough users posting to news.groups and
news.groups.proposals to generate any meaningful discussion on what
course of action should be taken: new groups, removing old groups, etc.

A lot of the discussion is repetitive and predictable, coming from the
same small group of users.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In <t680m4$pgt$2@dont-email.me> Usenet Big-8 Management Board <board@big-8.org> writes:

> REQUEST FOR DISCUSSION (RFD)
> moderated group news.groups.proposals

>This is a formal Request For Discussion (RFD) to suspend the charter and
>moderation policy of the Usenet newsgroup news.groups.proposals.

[...]

>RATIONALE

>Since 2006, the Big-8 hierarchies have undergone an overall reduction in
>their active user base and article traffic. The news.groups newsgroup
>has followed this general trend; the past few years have seen some
>measure of spam and other off-topic messages, but little of the
>acrimonious content that was the main impetus behind the creation of
>news.groups.proposals. There is therefore reason to believe that
>news.groups could once again function as "a healthy environment" for the
>discussion of RFDs.

I am reminded of what is considered the worst sequel in a movie
franchise. The classic 1941 movie "The Maltese Falcon" with Humphrey
Bogart was later followed up in 1975 with "The Black Bird" with George
Segal as Sam Spade, Jr. Intended as as a spoof, at best it falls into
the "so bad, it's good" category, and its politically-incorrect elements
including midget Nazi's (shades of Godwin's Law!) also help ensure it
won't be seen again, save for an old VHS copy (never released on DVD) or
maybe a rare 3 AM showing on the Turner movie channel.

In one scene, George Segal is seeking help from a prim, slightly
frustrated librarian to translate some ancient texts about the black
bird. She does so, but also notes that one of the texts is about
"fornication." He asks, is there anything else in the text besides the
fornication? No, she replies, just fornication. However, she has
helpfully translated the fornication and put it an envelope for him to
read later.

The lesson here appears to be that the wisdom of the ancients holds that
we should have some "fornication" in the sacred texts, but in the
present day, helpful librarians will translate the fornication for you,
and put it in an envelope to read later. Analogously, for Usenet, the
potential messiness of an unmoderated configging newsgroup like
news.groups is considered by some to be a feature, not a bug. If you
don't understand, we would be happy to translate this wisdom for you, to
put in an envelope to read later.

Even without the past run-on, often ad-hominem, arguments about
newsgroups from a small number of individuals, there are other current
problems with news.groups, including many articles advertising illegal
drugs and sex trafficking. If the intention is to grow participation
and increase article activity on Usenet, how many serious-minded
individuals would want to post alongside such content?

Even without the problem off-topic content, I wonder if the intention of
some advocates for returning configging discussion to news.groups is to
have an unrestricted forum to argue along the lines of common fallacious
arguments against moderated newsgroups:

https://www.reddit.com/r/ClassicUsenet/comments/udu379/common_fallacious_arguments_against_moderated/

especially arguments 14, 15, and 16:

14: The proponents of a moderated newsgroup represent only "one side" of
an argument, where all participants in the argument are assumed to be
all at the same level of rationality and moral justification. A
moderated newsgroup will just allow this "one side" to establish an echo
chamber or bunker where they can ignore, or even actively ridicule
without rebuttal, other sides of the argument and the individuals who
make them. Corollary: What you call trolls are just posting "facts and
truth" and you just don't want to listen to them. (A troll could very
well say, "The sky is blue" but also say a lot of other things,
including attacks against others, that are inappropriate and offensive
by objective standards. Trolls are also well capable of asserting, "The
sky is red" then ridiculing those who reply that the sky is blue with,
"Of course, I meant on Mars.")

15: This well-written and edited Request for Discussion (RFD) for a
proposed moderated newsgroup, with a clear charter and sensible
moderation policies, named moderators, and practical plans for
moderation software, that seemingly sprang out of "nowhere," is
suspicious. Must be a conspiracy, likely enabled by outside
agitators. Corollary: The planning should have been conducted out in the
open, in an unlimited "Battle Royale" of argument, overwhelming the
unmoderated newsgroup(s), so that we could criticize it into oblivion,
get nowhere with consensus-building, and run off the proponents so that
they would learn not to submit such foolish ideas again.

16: There is "Standard Advice" not published anywhere, but with which
(of course) all sane and sensible people agree, that all newsgroups
should be unmoderated, anyway. If you can't succeed with wildly
impractical suggestions to make them better, you should just live with
their shortcomings.

The moderated news.groups.proposals newsgroup was created for very
specific reasons to solve very specific problems. Some of these
problems were discussed in Russ Allbery's farewell article from 2006:

https://groups.google.com/g/news.groups/c/7U9Up4l_7MY/m/ibm4-XJAUPwJ

There's no assurance that these problems won't emerge again even in a
smaller Usenet.

>By contrast, in the past few years news.groups.proposals has had
>problems of its own, mostly stemming from its convoluted and antiquated
>moderation system. Many submissions have gone missing or unnoticed by
>the moderators due to breakdowns in the submission pipeline. While the
>current Board members have been working to streamline and modernize the
>moderation system they inherited, and to put better fault detection and
>prevention measures in place, there is always the risk of further
>unexpected technical issues.

[...]
The technical issues are solvable. The moderation system for
news.groups.proposals can run in a stable and reliable fashion, with
prompt error detection and notification, if it is installed on reliable
hosting, such as Panix.com, and with many software improvements made to
the version of moderation software used there. Several other
newsgroups, including news.announce.newgroups, use this option. If
independence of this team from the Big-8 Board is desired, it can be ran
from a separate account.

[...]

- --
Paul W. Schleck
pschleck@panix.com

-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAmKKTDAACgkQ6Pj0az779o55cwCfZW9WLntc4DNHnUuPluTf+SFK
iNMAn1wMezDLbNnQ+4KyUFJ2dq6IaP5v
=LG2T
-----END PGP SIGNATURE-----

On Sun, 22 May 2022 04:54:33 CST, D Finnigan wrote:

> Would recommend that Big-8 continue their work on streamlining and
> modernizing the moderation system.

There's not much that the board can do when it comes to modernizing
moderation. If you're not familiar with how moderation works, here's a
very simplified explanation:

A user submits an article to a moderated newsgroup. The news server that
the user uses sends an email to the moderator.

The moderator sends the approved article to a news server with an
"approved" header. Many times, the user will also sign the message with a
PGP/GPG key.

The news server that the moderator uses accepts the article and it then
goes online.

If the moderator denies the article, they may reply to the user with a
reason or they can simply delete it.

Changing how moderation works might require a change to the nntp standard
and changes to the software usenet providers use for their servers.

What we need more than updating moderation is modern moderation software.

> Finally, I wonder if there are enough users posting to news.groups and
> news.groups.proposals to generate any meaningful discussion on what
> course of action should be taken: new groups, removing old groups, etc.

We've had people ask why we continue to use the moderated group when there
is so little traffic on news.groups that could be considered to be
actually trolling (besides to obnoxious drug spam). Is it really necessary
or just causing more work for the board?

That's why this post was made. Does moderating group-related discussions
help in 2022 or is it just more work? Those of us on the board can see
things both ways and that's why we're opening this up for discussion.

Jason

Jason Evans <jsevans@mailfence.com> wrote:
> Changing how moderation works might require a change to the nntp standard
> and changes to the software usenet providers use for their servers.
>
> What we need more than updating moderation is modern moderation software.

I wonder what the blockers on that are, and whether the Board and others
might encourage somebody to do it.

It seems to me to not be super complicated:

- a program that receives emails of submissions
- applies group-specific filter rules (eg 'all messages containing XXX are
banned')
- if the message is not auto-approved put it in a queue for a moderator
- offer a web/other interface for moderators to approve/reject messages
(similar to mailing list moderation)
- sign the message and post it to an NNTP server
- handle moderator user accounts / password resets / etc etc
- maybe a web editor for filter rules and auto-messages
- maybe a web page that provides status (submissions in the queue, rejected
submissions, etc)
- maybe mails the Board if there are pending messages and no moderator has
taken any action for N days
- run as a hosted platform on a trivial server resource (given traffic
levels, a $1/month server would probably do it) so moderators don't have to
set it up (although they could selfhost if they wanted)

Basically mostly backend based on mail processing, with a simple web frontend
that doesn't do much client-side.

I'm not the person to do it, but I imagine writing something like that in a
modern language with decent libraries to handle the mail/web/NNTP side of
things (eg Python) wouldn't be a lot of work.

> That's why this post was made. Does moderating group-related discussions
> help in 2022 or is it just more work? Those of us on the board can see
> things both ways and that's why we're opening this up for discussion.

On the general issue I suppose the question is: if moderation were easy and
trivial to set up (via better tools as above), would there be a downside to
keeping a group moderated but with a laissez-faire moderation policy
("anyone can post anything, we auto-approve everything apart from messages
with 'drugs' in the subject")? That offers the ability to adjust the
moderation policy to block spam, without requiring much in the way of
moderator work.

I suppose an argument against is that crossposts to moderated groups aren't
possible, which makes some cases awkward.

As to news.groups.proposals specifically, I don't have strong opinions one
way or the other.

Theo

On 5/22/22 11:39 AM, Jason Evans wrote:
> On Sun, 22 May 2022 04:54:33 CST, D Finnigan wrote:
>
>> Would recommend that Big-8 continue their work on streamlining and
>> modernizing the moderation system.
>
> There's not much that the board can do when it comes to modernizing
> moderation.
I made my statement of recommendation in response to the RFD author's
remark that "While the current Board members have been working to
streamline and modernize the moderation system they inherited, and to
put better fault detection and prevention measures in place..."

>
> We've had people ask why we continue to use the moderated group when
there
> is so little traffic on news.groups that could be considered to be
> actually trolling (besides to obnoxious drug spam). Is it really
necessary
> or just causing more work for the board?

The volume of off-topic posts is probably a deterrent to users who wish
to have discussions in conformance with the newsgroup's charter. I have
anecdotal evidence, just from one newsgroup, comp.sys.apple2
In the past few years there's been the ITALIAN SPAMMER (you know who I'm
talking about). And a few months ago, probably last fall, his volume of
messages became so great that on-topic posts to comp.sys.apple2 fell to
almost none for a period of several weeks. He comes and goes, and his
posting volume rises and falls, and I've noticed the trend that when his
off-topic posting volume rises, on-topic Apple II posts tend to fall.

I don't have any more evidence for other newsgroups, but I doubt that
this phenomenon is unique to comp.sys.apple2:

The volume of off-topic posts will deter users who wish to participate
in conformance with the newsgroup's charter.

Greetings.

On 28/05/2022 00.00, Theo wrote:
> Jason Evans <jsevans@mailfence.com> wrote:
>> Changing how moderation works might require a change to the nntp standard
>> and changes to the software usenet providers use for their servers.
>>
>> What we need more than updating moderation is modern moderation software.
>
> I wonder what the blockers on that are, and whether the Board and others
> might encourage somebody to do it.
>
> It seems to me to not be super complicated:

It may not seem so, but there are countless details and special (albeit
not entirely uncommon) cases that need to be accounted for and
documented. Some examples:

> - a program that receives emails of submissions

Receives them how? Do we build the software in a way that the moderated
group must have a dedicated e-mail account? Or is it sufficient for
there to simply be a dedicated alias that auto-forwards to a real e-mail
account, which is possibly used for other purposes? Or is there no
requirement at all that the "To:" address be unique for a given group?

How is the e-mail account to be accessed? POP? IMAP? Unix mail spool?
Local mbox/maildir folder? In the case where the account is shared,
what software is responsible for ensuring that article submissions get
passed to the moderation software? Is it the moderation software itself
which examines all incoming messages, or do they get pre-filtered by an
MDA? If the latter, which MDA? Previous moderation packages have
relied on procmail, and usually include documentation that instructs
users on how to interface with it, but many users don't want to use
procmail nowadays due to its arcane, practically unmaintainable syntax
and its history of upstream maintenance problems.

> - applies group-specific filter rules (eg 'all messages containing XXX are
> banned')

Before you can even start filtering, you need to get the submission in a
format that can be processed by a filter. What happens if you receive a
MIME-encoded message? What if it's multi-part MIME? You may need to
separate the message into parts, find the plain-text one, and decode it
if necessary.

And before you start applying group-specific filters, you need to apply
ones that are universal to Usenet. For this you need a good knowledge
of the relevant RFCs. For example, RFC 5536 and 5537 prescribe special
handling of the Injection-Date, Injection-Info, and Xref headers. If
the submission contains these headers (and it probably does), and if you
simply pass them on as-is, your NNTP posting agent is likely to reject
the submission. You should probably also make sure that all mandatory
headers (Newsgroups, Subject, Date, From, etc.) are present and
well-formed, and that certain special optional headers (Approved,
Control, etc.) are absent. If not, you need to implement and document
some way of dealing with such cases.

You will probably also want to implement some default, common-sense
filters that nearly all moderators will want, such as stripping MIME
attachments (possibly except for things like OpenPGP signatures),
stripping non-plain-text "alternative" versions of multi-part MIME
messages, and wrapping long lines (or simply detecting these cases and
rejecting the message with an appropriate message to the submitter).

> - offer a web/other interface for moderators to approve/reject messages
> (similar to mailing list moderation)

How should the software authenticate a moderator? Does it need to
implement some sort of access control, or does it delegate this to the
web server (as with HTTP authentication)? How should the authentication
state be preserved -- via cookies or an HTTP request method?

> - sign the message and post it to an NNTP server

Sign it how? Does the software depend on a library for OpenPGP
signatures, or does it make a system call to GnuPG/PGP or the like?
Does the software assume that it has unrestricted access to the private
keys or does it require the user to decrypt them with a passphrase for
each signature?

> - handle moderator user accounts / password resets / etc etc

This goes hand-in-hand with the authentication issues above. Probably
nowadays there are libraries that securely handle authentication and
credentials for web accounts. (About 25 years ago, when STUMP and other
moderation packages were being developed, there weren't.)

> I'm not the person to do it, but I imagine writing something like that in a
> modern language with decent libraries to handle the mail/web/NNTP side of
> things (eg Python) wouldn't be a lot of work.

Well, none of the issues I've listed above are insurmountable -- in
fact, they're not even technically difficult ones. But they do require
getting acquainted with the relevant RFCs and mail/web/security/NNTP
frameworks, making a lot of well-informed design decisions, and
documenting everything in sufficient detail to let (a)
technically-inclined folk install and configure the system, (b)
non-technically-inclined moderators use the system, and (c) current and
future programmers further develop or maintain the system.

If any of the Board members had the time, we'd love to dive in and write
a brand-new moderation system from the ground up, using today's
state-of-the-art libraries and best practices in security, UI design,
etc. But we've all got day jobs, and other real-life commitments, and
plenty of other mundane Board-related administration work to do, and so
at the moment the best we're able to manage is to apply the occasional
enhancement or bugfix to STUMP to keep it limping along on modern OSes.

Still, we've had some offers to help with moderation software in the
past, and some of these people have actually followed through, so a new
moderation system might indeed happen one day. (It might even happen
without us, though as far as I know we're the only organized group
actively maintaining moderation software nowadays.)

Regards,
Tristan

--
Usenet Big-8 Management Board
https://www.big-8.org/
board@big-8.org

Tristan Miller wrote:

> If any of the Board members had the time, we'd love to dive in and write
> a brand-new moderation system from the ground up, using today's
> state-of-the-art libraries and best practices in security, UI design,
> etc.  But we've all got day jobs, and other real-life commitments, and
> plenty of other mundane Board-related administration work to do, and so
> at the moment the best we're able to manage is to apply the occasional
> enhancement or bugfix to STUMP to keep it limping along on modern OSes.

None of the infrastructure used to moderate Usenet newsgroups has
changed in decades. Why is there a need for new state-of-the-art
software with flashy GUIs? It's not like having wonderful new
moderation software is going to attract a new crowd of people eager to
moderate newsgroups . . . and even if it did, the problem is more one of
a crowd of people to participate in the group.

> Still, we've had some offers to help with moderation software in the
> past, and some of these people have actually followed through, so a new
> moderation system might indeed happen one day.  (It might even happen
> without us, though as far as I know we're the only organized group
> actively maintaining moderation software nowadays.)

What's the problem you're trying to solve? There are a few moderated
newsgroups that are still in operation, and by definition they have
working moderation software. Realistically, you're not going to revive
dead groups; the users have long ago found other options. If the Board
really is interested in helping Usenet, offer a moderation platform for
cases when the existing moderator must quit. Even that is going to have
vanishingly few takers.

Today's Usenet is such a tiny shadow of its former self that the
traditional task of administration is essentially gone.

Greetings.

On 03/06/2022 15.07, Steve Bonine wrote:
> None of the infrastructure used to moderate Usenet newsgroups has
> changed in decades.

Sure it has:

* Programming languages and their compilers/interpreters, software
libraries, and operating systems have all changed. The latest stable
release of STUMP, from more than 20 years ago, no longer runs on modern
versions of Perl. Procmail, upon which STUMP indirectly depends, is
being abandoned or phased out by some hosting services.

* Security infrastructure, threats, mitigation tactics, and best
practices have evolved significantly. Passing sensitive information in
HTTP GET query strings, and storing usernames and passwords in
plaintext, were commonplace in the 1990s but a big no-no today. Modern
versions of GnuPG, necessary for signing messages, can't read keys that
were created decades ago, which are in any case small enough to be
brute-forced with today's computing power.

* Netnews protocols have been continuously updated by the IETF.
Moderation software today needs to correctly process headers that didn't
exist 15 years ago.

The Board has already updated STUMP (its own instance and/or the version
in the public source code repository) to address some of these issues.
But some others -- in particular some of those involving security -- are
not trivial to fix.

I would love to have a modern moderation package simply for my own use
-- I'm tired of troubleshooting the current system when it breaks. And
I know that STUMP is also causing headaches for other current or
would-be moderators. A new system isn't going to result in a "crowd of
people" rushing to moderate groups, but it will probably enable a few
people who are interested in moderating to start up or to continue.

Regards,
Tristan

--
Usenet Big-8 Management Board
https://www.big-8.org/
board@big-8.org

Tristan Miller <tmiller@big-8.org> wrote:
> Greetings.
>
> On 28/05/2022 00.00, Theo wrote:
> > Jason Evans <jsevans@mailfence.com> wrote:
> >> Changing how moderation works might require a change to the nntp standard
> >> and changes to the software usenet providers use for their servers.
> >>
> >> What we need more than updating moderation is modern moderation software.
> >
> > I wonder what the blockers on that are, and whether the Board and others
> > might encourage somebody to do it.
> >
> > It seems to me to not be super complicated:
>
>
> It may not seem so, but there are countless details and special (albeit
> not entirely uncommon) cases that need to be accounted for and
> documented. Some examples:

Some high level sketches below. I'm assuming it's a standalone server (or
Docker container / whatever) and naming some Python libraries that look
plausible to do the heavy lifting. No doubt there are many issues and
awkwardnesses I haven't thought of.

> > - a program that receives emails of submissions
>
>
> Receives them how? Do we build the software in a way that the moderated
> group must have a dedicated e-mail account? Or is it sufficient for
> there to simply be a dedicated alias that auto-forwards to a real e-mail
> account, which is possibly used for other purposes? Or is there no
> requirement at all that the "To:" address be unique for a given group?

I'm assuming that the moderation server operates its own MDA which receives
messages addressed to news.groups-submission@mod.example.com
or similar addresses. This is a list maintained by the mod software (ie add
a new group, get a new mailbox in /var/spool/mail/news.groups-submission).

Said MDA delivers messages into a mailspool that the mod software picks up.
Library for that:
https://docs.python.org/3/library/mailbox.html

> How is the e-mail account to be accessed? POP? IMAP? Unix mail spool?
> Local mbox/maildir folder? In the case where the account is shared,
> what software is responsible for ensuring that article submissions get
> passed to the moderation software? Is it the moderation software itself
> which examines all incoming messages, or do they get pre-filtered by an
> MDA? If the latter, which MDA? Previous moderation packages have
> relied on procmail, and usually include documentation that instructs
> users on how to interface with it, but many users don't want to use
> procmail nowadays due to its arcane, practically unmaintainable syntax
> and its history of upstream maintenance problems.

In general, the fewer external tools the less there is to break. So minimal
bits of shell, glue and string. Config of necessary external tools (eg MDA)
would be managed by the mod software (ie the aliases above).

> > - applies group-specific filter rules (eg 'all messages containing XXX are
> > banned')
>
>
> Before you can even start filtering, you need to get the submission in a
> format that can be processed by a filter. What happens if you receive a
> MIME-encoded message? What if it's multi-part MIME? You may need to
> separate the message into parts, find the plain-text one, and decode it
> if necessary.

https://docs.python.org/3/library/email.html

> And before you start applying group-specific filters, you need to apply
> ones that are universal to Usenet. For this you need a good knowledge
> of the relevant RFCs. For example, RFC 5536 and 5537 prescribe special
> handling of the Injection-Date, Injection-Info, and Xref headers. If
> the submission contains these headers (and it probably does), and if you
> simply pass them on as-is, your NNTP posting agent is likely to reject
> the submission. You should probably also make sure that all mandatory
> headers (Newsgroups, Subject, Date, From, etc.) are present and
> well-formed, and that certain special optional headers (Approved,
> Control, etc.) are absent. If not, you need to implement and document
> some way of dealing with such cases.

https://docs.python.org/3/library/email.parser.html
Is there anything about incoming messages that are different to regular
email MIME such that parsing Usenet messages as email won't work?
(eg apart from different headers, eg differences in message formatting)

> You will probably also want to implement some default, common-sense
> filters that nearly all moderators will want, such as stripping MIME
> attachments (possibly except for things like OpenPGP signatures),
> stripping non-plain-text "alternative" versions of multi-part MIME
> messages, and wrapping long lines (or simply detecting these cases and
> rejecting the message with an appropriate message to the submitter).

https://docs.python.org/3/library/email.contentmanager.html

> > - offer a web/other interface for moderators to approve/reject messages
> > (similar to mailing list moderation)
>
>
> How should the software authenticate a moderator? Does it need to
> implement some sort of access control, or does it delegate this to the
> web server (as with HTTP authentication)? How should the authentication
> state be preserved -- via cookies or an HTTP request method?

Here diverges a bit depending on what kind of web framework you're using,
but essentially:

Authenticate users via OAuth:
https://oauth.net/code/python/
(using their password on another service, eg Github / Google / Facebook /
Microsoft / Apple / whatever they prefer)

Authentication is via a cookie set in their browser, like every other
service these days. The web framework knows who you are. (eg the Django
framework might be a fit for Python, although I'm not familiar with it)

Someone from the Board maintains the list of who moderates which groups, via
their login.

Users are able to login and see the groups they moderate.

> > - sign the message and post it to an NNTP server
>
>
> Sign it how? Does the software depend on a library for OpenPGP
> signatures, or does it make a system call to GnuPG/PGP or the like?
> Does the software assume that it has unrestricted access to the private
> keys or does it require the user to decrypt them with a passphrase for
> each signature?

I don't have a strong opinion either way - an integrated library
would be easier, but using a distro-provided GnuPG might be better
maintained. Some examples of both:
https://pypi.org/project/py-pgp/

Key management is an interesting question, but in general I'd expect that
users with moderation rights also have posting rights. You wouldn't need to
unlock the keys by hand each time.

It is possible the keys could be held in some more secure way (secure
enclave, SGX?) but I'm not au fait with the current state of software for
that. It would add another dependency which might risk longevity.

> > - handle moderator user accounts / password resets / etc etc
>
>
> This goes hand-in-hand with the authentication issues above. Probably
> nowadays there are libraries that securely handle authentication and
> credentials for web accounts. (About 25 years ago, when STUMP and other
> moderation packages were being developed, there weren't.)

OAuth is common these days - you don't need to manage usernames and
passwords, you just ask Google 'can you confirm this really is
xyz@gmail.com' (or whoever), and once that's confirmed they are logged in.
Once you are logged in, the web pages you see are generated by software that
knows who you are and customises your view of the content.

Basically you could sketch it as composed of several parts:

1) A web site with login, like every other web site in the world today
2) A part to handle incoming email traffic
3) A part to apply rules to messages
4) A part to interact with moderators what to do with messages
5) A part to inject messages into NNTP and handle PGP
6) A part to record activity and display status

Many of those features are shared with existing email mailing list software,
and that may be a suitable starting point. Apart from 5) and maybe 6), most
of this seems similar.

> > I'm not the person to do it, but I imagine writing something like that in a
> > modern language with decent libraries to handle the mail/web/NNTP side of
> > things (eg Python) wouldn't be a lot of work.
>
>
> Well, none of the issues I've listed above are insurmountable -- in
> fact, they're not even technically difficult ones. But they do require
> getting acquainted with the relevant RFCs and mail/web/security/NNTP
> frameworks, making a lot of well-informed design decisions, and
> documenting everything in sufficient detail to let (a)
> technically-inclined folk install and configure the system, (b)
> non-technically-inclined moderators use the system, and (c) current and
> future programmers further develop or maintain the system.

On 6/6/2022 6:24 AM, Theo wrote:

>
> How easy is it to get STUMP+WebSTUMP up and running today, on a fresh server?
> (I see some instructions from 1990s, but not sure if there are up to date)
>

I built a STUMP server a few years ago. Had to tweak a few things but
it wasn't bad. I didn't bother with WebSTUMP. It administers
misc.legal.moderated.

These is a group around that has the STUMP code and run by Big8

https://github.com/UsenetBig8/STUMP

I applied for membership in February but haven't heard anything.

I have also been chatting with owenrees@fastmail.fm on STUMP

Roy

On 2022-06-06, Theo <theom+news@chiark.greenend.org.uk> wrote:
> Tristan Miller <tmiller@big-8.org> wrote:
>> Greetings.
>>
>> On 28/05/2022 00.00, Theo wrote:
>> > Jason Evans <jsevans@mailfence.com> wrote:
>> >> Changing how moderation works might require a change to the nntp standard
>> >> and changes to the software usenet providers use for their servers.
>> >>
>> >> What we need more than updating moderation is modern moderation software.
>> >
>> > I wonder what the blockers on that are, and whether the Board and others
>> > might encourage somebody to do it.
>> >
>> > It seems to me to not be super complicated:
>>
>>
>> It may not seem so, but there are countless details and special (albeit
>> not entirely uncommon) cases that need to be accounted for and
>> documented. Some examples:
>
> Some high level sketches below. I'm assuming it's a standalone server (or
> Docker container / whatever) and naming some Python libraries that look
> plausible to do the heavy lifting. No doubt there are many issues and
> awkwardnesses I haven't thought of.
>
>> > - a program that receives emails of submissions
>>
>>
>> Receives them how? Do we build the software in a way that the moderated
>> group must have a dedicated e-mail account? Or is it sufficient for
>> there to simply be a dedicated alias that auto-forwards to a real e-mail
>> account, which is possibly used for other purposes? Or is there no
>> requirement at all that the "To:" address be unique for a given group?
>
> I'm assuming that the moderation server operates its own MDA which receives
> messages addressed to news.groups-submission@mod.example.com
> or similar addresses. This is a list maintained by the mod software (ie add
> a new group, get a new mailbox in /var/spool/mail/news.groups-submission).
>
> Said MDA delivers messages into a mailspool that the mod software picks up.
> Library for that:
> https://docs.python.org/3/library/mailbox.html
>
>> How is the e-mail account to be accessed? POP? IMAP? Unix mail spool?
>> Local mbox/maildir folder? In the case where the account is shared,
>> what software is responsible for ensuring that article submissions get
>> passed to the moderation software? Is it the moderation software itself
>> which examines all incoming messages, or do they get pre-filtered by an
>> MDA? If the latter, which MDA? Previous moderation packages have
>> relied on procmail, and usually include documentation that instructs
>> users on how to interface with it, but many users don't want to use
>> procmail nowadays due to its arcane, practically unmaintainable syntax
>> and its history of upstream maintenance problems.
>
> In general, the fewer external tools the less there is to break. So minimal
> bits of shell, glue and string. Config of necessary external tools (eg MDA)
> would be managed by the mod software (ie the aliases above).

I think it's "safe" to say at this point that most mail is stored in
Mbox or Maildir formats, so picking Maildir probably makes sense
here. Though if I think about it from a dependency perspective, both
Mbox and Maildir rely on having a local mailbox of sorts.

>
>> > - applies group-specific filter rules (eg 'all messages containing XXX are
>> > banned')
>>
>>
>> Before you can even start filtering, you need to get the submission in a
>> format that can be processed by a filter. What happens if you receive a
>> MIME-encoded message? What if it's multi-part MIME? You may need to
>> separate the message into parts, find the plain-text one, and decode it
>> if necessary.
>
> https://docs.python.org/3/library/email.html
>
>
>> And before you start applying group-specific filters, you need to apply
>> ones that are universal to Usenet. For this you need a good knowledge
>> of the relevant RFCs. For example, RFC 5536 and 5537 prescribe special
>> handling of the Injection-Date, Injection-Info, and Xref headers. If
>> the submission contains these headers (and it probably does), and if you
>> simply pass them on as-is, your NNTP posting agent is likely to reject
>> the submission. You should probably also make sure that all mandatory
>> headers (Newsgroups, Subject, Date, From, etc.) are present and
>> well-formed, and that certain special optional headers (Approved,
>> Control, etc.) are absent. If not, you need to implement and document
>> some way of dealing with such cases.
>
> https://docs.python.org/3/library/email.parser.html
> Is there anything about incoming messages that are different to regular
> email MIME such that parsing Usenet messages as email won't work?
> (eg apart from different headers, eg differences in message formatting)
>
>> You will probably also want to implement some default, common-sense
>> filters that nearly all moderators will want, such as stripping MIME
>> attachments (possibly except for things like OpenPGP signatures),
>> stripping non-plain-text "alternative" versions of multi-part MIME
>> messages, and wrapping long lines (or simply detecting these cases and
>> rejecting the message with an appropriate message to the submitter).
>
> https://docs.python.org/3/library/email.contentmanager.html

The above pieces of hygeine that Tristan discussed are still needed
but all are simple tweaks.

>
>
>> > - offer a web/other interface for moderators to approve/reject messages
>> > (similar to mailing list moderation)
>>
>>
>> How should the software authenticate a moderator? Does it need to
>> implement some sort of access control, or does it delegate this to the
>> web server (as with HTTP authentication)? How should the authentication
>> state be preserved -- via cookies or an HTTP request method?
>
> Here diverges a bit depending on what kind of web framework you're using,
> but essentially:
>
> Authenticate users via OAuth:
> https://oauth.net/code/python/
> (using their password on another service, eg Github / Google / Facebook /
> Microsoft / Apple / whatever they prefer)

I think the folks using Usenet may not like the identity of many big
OAuth providers but this is immaterial to me.

>
> Authentication is via a cookie set in their browser, like every other
> service these days. The web framework knows who you are. (eg the Django
> framework might be a fit for Python, although I'm not familiar with it)
>
> Someone from the Board maintains the list of who moderates which groups, via
> their login.
>
> Users are able to login and see the groups they moderate.

Would the Web portion be centrally run moderation software then?

>
>> > - sign the message and post it to an NNTP server
>>
>>
>> Sign it how? Does the software depend on a library for OpenPGP
>> signatures, or does it make a system call to GnuPG/PGP or the like?
>> Does the software assume that it has unrestricted access to the private
>> keys or does it require the user to decrypt them with a passphrase for
>> each signature?
>
> I don't have a strong opinion either way - an integrated library
> would be easier, but using a distro-provided GnuPG might be better
> maintained. Some examples of both:
> https://pypi.org/project/py-pgp/

What happens if GPG versions rev? Seems like there's anxiety in this
group about GPG key types. FWIW I don't have much GPG experience.

>
> Key management is an interesting question, but in general I'd expect that
> users with moderation rights also have posting rights. You wouldn't need to
> unlock the keys by hand each time.
>
> It is possible the keys could be held in some more secure way (secure
> enclave, SGX?) but I'm not au fait with the current state of software for
> that. It would add another dependency which might risk longevity.
>
>> > - handle moderator user accounts / password resets / etc etc
>>
>>
>> This goes hand-in-hand with the authentication issues above. Probably
>> nowadays there are libraries that securely handle authentication and
>> credentials for web accounts. (About 25 years ago, when STUMP and other
>> moderation packages were being developed, there weren't.)
>
> OAuth is common these days - you don't need to manage usernames and
> passwords, you just ask Google 'can you confirm this really is
> xyz@gmail.com' (or whoever), and once that's confirmed they are logged in.
> Once you are logged in, the web pages you see are generated by software that
> knows who you are and customises your view of the content.
>
> Basically you could sketch it as composed of several parts:
>
> 1) A web site with login, like every other web site in the world today
> 2) A part to handle incoming email traffic
> 3) A part to apply rules to messages
> 4) A part to interact with moderators what to do with messages
> 5) A part to inject messages into NNTP and handle PGP
> 6) A part to record activity and display status
>
> Many of those features are shared with existing email mailing list software,
> and that may be a suitable starting point. Apart from 5) and maybe 6), most
> of this seems similar.
>
>> > I'm not the person to do it, but I imagine writing something like that in a
>> > modern language with decent libraries to handle the mail/web/NNTP side of
>> > things (eg Python) wouldn't be a lot of work.
>>
>>
>> Well, none of the issues I've listed above are insurmountable -- in
>> fact, they're not even technically difficult ones. But they do require
>> getting acquainted with the relevant RFCs and mail/web/security/NNTP
>> frameworks, making a lot of well-informed design decisions, and
>> documenting everything in sufficient detail to let (a)
>> technically-inclined folk install and configure the system, (b)
>> non-technically-inclined moderators use the system, and (c) current and
>> future programmers further develop or maintain the system.
>
> I agree, nailing down the requirements is a key part of the brief, as is
> documentation and making the thing easy to run. There's some useful
> documentation here:
> https://www.algebra.com/~ichudov/stump/spec.html
> (the comment 'Maybe I'll do it after getting 32 meg of memory and installing
> Postgres.' suggests this text is very old)
>
>> If any of the Board members had the time, we'd love to dive in and write
>> a brand-new moderation system from the ground up, using today's
>> state-of-the-art libraries and best practices in security, UI design,
>> etc. But we've all got day jobs, and other real-life commitments, and
>> plenty of other mundane Board-related administration work to do, and so
>> at the moment the best we're able to manage is to apply the occasional
>> enhancement or bugfix to STUMP to keep it limping along on modern OSes.
>
> Understood... I am not volunteering either, for the above reasons.
> (this post is by no means intended as complaining, btw. Just trying to
> better understand some of those requirements)

On Mon, 6 Jun 2022 11:34:40 -0700, Roy wrote:

> These is a group around that has the STUMP code and run by Big8
>
> https://github.com/UsenetBig8/STUMP
>
> I applied for membership in February but haven't heard anything.

You can find the latest code here:
https://savannah.gnu.org/projects/stump/

Because STUMP is a FSF project the main source code is on their page. We
just mirror it on Github.

Jason

Jason Evans wrote:

> You can find the latest code here:
> https://savannah.gnu.org/projects/stump/

Is the ReadyStump service described on this site still offered / supported?

On Wed, 8 Jun 2022 09:12:58 -0500, Steve Bonine wrote:

> Is the ReadyStump service described on this site still offered /
> supported?

No, Igor Chudov stopped offering that service years ago.

On 06 Jun 2022 14:24:00 +0100 (BST), Theo
<theom+news@chiark.greenend.org.uk> wrote in
<OLs*8-3Py@news.chiark.greenend.org.uk>:

>I'm assuming that the moderation server operates its own MDA which receives
>messages addressed to news.groups-submission@mod.example.com
>or similar addresses. This is a list maintained by the mod software (ie add
>a new group, get a new mailbox in /var/spool/mail/news.groups-submission).

Traditionally, STUMP and WebSTUMP use procmail to send incoming messages
that match the relevant criteria to various scripts that process them.
Messages do not necessarily end up in a mailbox - you can send them to
mailboxes or other scripts if you want but that is not necessary for
basic operation (but it is very helpful for debugging).

There are some installations that use exim for that part of the delivery
according to what I have been able to discover.

Unfortunately, that does not eliminate the depencency on procmail as
some of the processing is done by calling formail which is part of the
procmail package. It would not be particularly difficult to eliminate
that dependency for someone reasonably familiar with perl and cpan.

Although STUMP is designed to handle only a single group it is not
diffficult to deploy multiple instances under a single account and have
procmail or an alternative send incoming submissions to the relevant
instance.

WebSTUMP is designed to handle multiple groups so only a single instance
is needed.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In <oS3oK.173712$b21.44677@fx11.ams1> Jason Evans <jsevans@mailfence.com> writes:

>On Wed, 8 Jun 2022 09:12:58 -0500, Steve Bonine wrote:

>> Is the ReadyStump service described on this site still offered /
>> supported?

>No, Igor Chudov stopped offering that service years ago.

Igor was also charging $360 per newsgroup per year. I'm not judging, as
some moderation teams were willing to pay that, even up until the end,
and it may have arguably been market price for Igor's time and trouble
to run a fully supported, completely turnkey, service over the
long-term.

Cheaper alternatives for Internet access and hosting, including STUMP,
are now available, however. For example, there has been a long-standing
(15+ year) moderator courtesy to set up an instance of STUMP and
WebSTUMP at Public Access Internet and UNIX, NYC (Panix.com). Setup
would involve copying a known stable installation, with many
enhancements over Igor's 1999 version, and changing the configuration
settings to the new newsgroup. This can be set up in a couple of hours.
Moderators would own the installation, could download it for
safekeeping, could modify it themselves or ask for assistance, and it
would be available to them indefinitely. Panix does have all of the
necessary dependencies, including Procmail, Perl, GPG, and Apache
installed. Multiple newsgroups per account, up to a reasonable limit
(likely less than a half-dozen) would be permitted. The catch is the
need to sign up for a shell account at Panix, which is $10/month, or
$100/year. Options like cost-sharing among a moderation team, or
seeking crowdsourced funding, are also possible, as long as someone pays
the bill to Panix.

I once joked on this newsgroup that Usenetters don't have any money,
except those that claim to be successful, self-made, multi-millionaires.

- --
Paul W. Schleck
pschleck@panix.com

-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAmKk5DgACgkQ6Pj0az779o5eDwCgpJlkPwKyB0DE5MEvP+Dv6ShD
bFMAn1r2ZD6dn7/zYjyk9KyiP0E2npGs
=wrqJ
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

In <t7d9hk$2c6$1@dont-email.me> Tristan Miller <tmiller@big-8.org> writes:

>Greetings.

>On 03/06/2022 15.07, Steve Bonine wrote:
> > None of the infrastructure used to moderate Usenet newsgroups has
> > changed in decades.

>Sure it has:

[...]

Also:

* Moderator submission addresses, and moderator relays, are receiving
more off-topic SPAM. Even sophisticated SPAM reporting and tracking
sites like Spamcop may incorrectly identify moderator relays as sources
of SPAM, rather than as innocent bystanders. By custom and standard,
moderation relays are not to do any filtering of submissions, relying on
the receiving moderated newsgroups and moderators to filter out
inappropriate content. Many of the current peered moderator relay sites
are now on shared hosting, often provided as a courtesy by the hosts.
With the decline of Usenet for text-only discussion, both in traffic and
perceived importance, sites hosting moderator relays are less tolerant
of misdirected abuse reports, and may choose to shut their hosted relays
down if they cause excessive workload, or even jeopardize the
connectivity of their sites for other uses. So, approaches to SPAM
filtering and reporting by moderated newsgroups need to be more careful,
and more accurate.

- --
Paul W. Schleck
pschleck@panix.com

-----BEGIN PGP SIGNATURE-----

iEYEARECAAYFAmKnvtsACgkQ6Pj0az779o4uRwCffwdqKzSsknn7Fv8oK2fDDCx2
k8cAoLc/alv5Wy/rEQJP1HEqeZ3Y6cnm
=lERu
-----END PGP SIGNATURE-----

meff <email@example.com> wrote:
> On 2022-06-06, Theo <theom+news@chiark.greenend.org.uk> wrote:
> > Tristan Miller <tmiller@big-8.org> wrote:
> >
> > In general, the fewer external tools the less there is to break. So minimal
> > bits of shell, glue and string. Config of necessary external tools (eg MDA)
> > would be managed by the mod software (ie the aliases above).
>
> I think it's "safe" to say at this point that most mail is stored in
> Mbox or Maildir formats, so picking Maildir probably makes sense
> here. Though if I think about it from a dependency perspective, both
> Mbox and Maildir rely on having a local mailbox of sorts.

Agreed. There's good reason not to write your own MDA, and so having a
standard interchange format with an existing one makes sense. I think
maildir has the edge over mbox, since there are some performance issues with
mbox files.

> The above pieces of hygeine that Tristan discussed are still needed
> but all are simple tweaks.

Agreed. It's not enough just to strip off all non text/plain attachments
(some systems may transmit everything in base64) but it's not hard to cover
those corners.

(I suppose there's a question about HTML, but probably a first cut would say
no HTML since I'm not aware of any groups that actually use it)

> > Here diverges a bit depending on what kind of web framework you're using,
> > but essentially:
> >
> > Authenticate users via OAuth:
> > https://oauth.net/code/python/
> > (using their password on another service, eg Github / Google / Facebook /
> > Microsoft / Apple / whatever they prefer)
>
> I think the folks using Usenet may not like the identity of many big
> OAuth providers but this is immaterial to me.

I'm sure there is a privacy-preserving OAuth service out there. There's no
dependency on any specific service, user just says 'use my user@xyz.com
credential' and any xyz.com that offers an OAuth service will do. Point
being it takes away a lot of the hassles about user authentication -
verifying emails, forgetting passwords etc etc - and also covers 2FA and
similar as well.

> Would the Web portion be centrally run moderation software then?

My guess would be that a lot of moderators don't want to run moderation
servers, nor know how. So they probably want to use a service run by
somebody else.

Which is not to say there must be exactly one service. If the software is
freely available and not too complicated to install, people are free to
install it on a server and run their own service. With tools like Docker
it's not too complicated to 'docker pull usenet-moderator' and you can run a
container with everything set up. Maybe a small fraction of people are
going to do that, but it leaves options open if they want. The server
resources to do that are pretty minimal: in the $10/year kind of ballpark.

In other words it's a bit like email: most people just have an account on
somebody else's server, but if you really want your own server there's
nothing stopping you.

I suppose the Board have two roles here:
1. Maintain the list of target addresses for moderated groups, ie which
server submissions get routed to
2. If the Board were to run their own server, they could have a more active
role in monitoring the health of groups beyond 'nothing was posted for a
year, maybe the moderator is dead?'.

I'm not 100% sure how #2 would square with decentralised servers, but
perhaps #2 is not essential. Or maybe it would be an optional feature of
the Board's instance of the server which wouldn't be accessible on
third-party instances.

> What happens if GPG versions rev? Seems like there's anxiety in this
> group about GPG key types. FWIW I don't have much GPG experience.

I'm not sure how Usenet handles GPG keys at the moment? Anyone know?
I presume for messages to be accepted you'd need to use whatever version the
servers are expecting.

> As I mentioned above, I'd be interested in writing moderation software
> but before that I think we'd need to decide on a few things:
>
> 1. The technical details as discussed in this thread
> 2. A timeframe to begin implementation so I can actually block time
> out to work on it.
> 3. What the future of maintaining the software would look like.
>
> I'm both familiar with and willing to implement this in Python or Go,
> but would depend on the above points first.

I'm not part of the Board so just a mere civilian, but this sounds like a
good approach.

Maintenance is a bit of a headache - and the existing stack has done well to
run (just about) for the past 20 years. My general thinking is to minimise
dependencies on external frameworks and libraries: I've had lots of problems
with tools like Mediawiki, that depend on mysql and PHP, which is very fussy
about which specific version you have. Keeping things in pure Python / etc,
files rather than databases, and so on means it's more likely to work as
dependencies get upgraded.

Perhaps it's worth specifically approaching the board with a proposal of
what you're thinking of doing?

Theo