>-attachment is lost (guess the b64 way of doing things is not overall appreciated)

actually, the attached pic displays on rslight, but the name is lost. bbs and ff cannot apparently not read it correctly and don't recognize that there is an attachment.
--
Posted on def2
z5bqfv5v75kxy7pj.onion

ok, and threading works as well, but the start of the tread must be present for it to work, as all messages in the thread refer to it (not to each other).

looks really good. now the other way round.

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

while this should be easy enough, it is not. they used a get request to sent over all the parameters for the message, including the body. and also the attachments.
to use a get request to post information is somewhat bad style, but ok. but for the hell of it, i cannot figure out how they did binary uploads with a get request. guess i will use post instead of get, and insert the commands somewhere in spoolnews.
--
Posted on def2
z5bqfv5v75kxy7pj.onion

this will take me a while to figure out, the function in question is >800 lines.
i hoped to look up the code that nntpchan used, but to my surprise it is no longer on github,
--
Posted on def2
z5bqfv5v75kxy7pj.onion

anonymous wrote:

> ok, just inserting those two into the outspool of rslight made them appear on rocksolid.shared.test (this was the defined group). works out of the box.

Nice! The format of the messages looks good, shouldn't be a problem.

> to do:

> -threading is broken (guess the header is not correct by our standards)

As I posted in r.s.test, it may be that vichan only wants one Reference.

> -attachment is lost (guess the b64 way of doing things is not overall appreciated)

I tried a header change in a test spool on the message, but still 'unnamed'. I'll look a bit more into this.

Retro Guy

--
Posted on: Rocksolid Light
news.novabbs.com

anonymous wrote:

> this will take me a while to figure out, the function in question is >800 lines.
> i hoped to look up the code that nntpchan used, but to my surprise it is no longer on github,

Is this post.php in vichan?

Looking at this file, I agree it's probably better to send it as vichan expects (via http, whether get or post), instead of trying to insert a complete message as this would require major modification to this file (not fun).

If we can write one function to take a complete article, and break it into what's needed to send it as a post request for vichan, spoolnews can call that function as it's writing to it's own spool. A simple:

if($isset($enable_vichan) && $enable_vichan === true) {
vichan_post($article;
}

function vichan_post($article) {
read $article and break it into pieces necessary to send to vichan;
send to vichan;
}

We could also modify any headers as necessary here, before sending so vichan doesn't reject the post (like stripping down to one Reference and one Newsgroup if necessary).

Retro Guy

--
Posted on: Rocksolid Light
news.novabbs.com

>Is this post.php in vichan?

yes.

>Looking at this file, I agree it's probably better to send it as vichan expects (via http, whether get or post), instead of trying to insert a complete message as this would require major modification to this file (not fun).

yes, i concur.

>If we can write one function to take a complete article, and break it into what's needed to send it as a post request for vichan, spoolnews can call that function as it's writing to it's own spool.

yes, i had that in mind too. it comes down to :
-parsing the message
-sending a post stream including the attachments (if any), probably using curl
-modying post.php on vichan to accept a message number from the post stream instead of creating it itself

>doesn't reject the post (like stripping down to one Reference and one group

either that or we modify vichan. guess it is better to modify the message.

--
Posted on def2
z5bqfv5v75kxy7pj.onion

maybe i will go the full 9 yards and make a hacked version of vichans post.php, i am just not sure yet what it would mean to include and run it from rslights side. i guess part of the environment would be different...but i just like the idea to skip the webserver and operate directly on the php code of vichan (via the webserver it is easier, of course...).
--
Posted on def2
z5bqfv5v75kxy7pj.onion

these are the parameters that post.php expects for importing messages:
$_GET['Newsgroups']
$_GET['Message-Id']
$_GET['References']
$_GET['Date'])
$_GET['Content-Type']
$_GET['Subject']
$_GET['From']
$_GET['X_Sage']
So no need to change anything for the message id, it already uses it.
Even for text only messages though the content is written raw in the request body, just as files, i guess.
Seeing that both the files and the message are already present as local files, maybe i can redirect this.
--
Posted on def2
z5bqfv5v75kxy7pj.onion

i guess the best would be to insert the function to post to vichan in spoolnews.php, function get_articles line 233 and following. all the data from the current message is nicely parsed at this point and can just be forwarded to vichan.
--
Posted on def2
z5bqfv5v75kxy7pj.onion

anonymous wrote:

> i guess the best would be to insert the function to post to vichan in spoolnews.php, function get_articles line 233 and following. all the data from the current message is nicely parsed at this point and can just be forwarded to vichan.

These are the variables that contain the data for these requests in
spoolnews.php (function get_articles) near where you specified:

$_GET['Newsgroups'] = $group
$_GET['Message-Id'] = $mid[1]
$_GET['References'] = $references (all of them)
$_GET['Date'] = $finddate[1] is a date string from the message
$article_date is a unix timestamp of same date
$_GET['Content-Type'] = not currently extracted in spoolnews, but can be
$_GET['Subject'] = $subject[1]
$_GET['From'] = $from[1]
$_GET['X_Sage'] = I'm guessing this header won't be in messages sent to vichan
unless from another vichan instance

Do you want to switch to a POST request? Since GET has length restrictions, it
might be best, but requres changes in vichan (post.php)

Retro Guy
--
Posted on: Rocksolid Light
news.novabbs.com

thanks for digging that out.
x-sage is vichan specific and can be omitted.

i decided to try the GET method first, just to test quickly if this would work.
after that i guess it must be post (unless i figure how to put the message and attachments in the GET request body).
--
Posted on def2
z5bqfv5v75kxy7pj.onion

ok, will skip the testing bullshit and get right to POST method. i could cut post.php to some 730 lines that are needed for this, out of which i need to modify some 13 lines to read POST instead of GET. Then will use something like this here:
https://www.codexworld.com/post-receive-json-data-using-php-curl/
to send over the data. so this means b64 encoding for the attachments.

i checked some sources now, and while it is technically feasible to put large texts and files into GET request bodies, it goes against all standards and almost everybody advised against it. never mind that a GET request per definition should have not have any impact on the servers state.
so i really wonder why it was designed like that. maybe there is some sneaky scheme here that i just don't get :-).

i also dug out the old sources of nntp chan (which was a little tricky), and checked the way they do it. but to my surprise all i could find was the nntp way of exchanging messages (so between two nntp instances), but not the way that is implemented in vichan, using http.

well never mind, i will hack that into shape, i think. :-)
--
Posted on def2
z5bqfv5v75kxy7pj.onion

anonymous wrote:

> i also dug out the old sources of nntp chan (which was a little tricky), and checked the way they do it. but to my surprise all i could find was the nntp way of exchanging messages (so between two nntp instances), but not the way that is implemented in vichan, using http.

If you plan to run vichan as dependent on the rslight nntp server, you would have a news server local that you could talk to. rslight would be the "other" nntp instance. Is that of any value?

This is how rslight works. Newsportal could only send/receive from a nntp server, so now it sends/receives from it's own built in nntp server, but the code for the interface is really just talking to any nntp server, I just happen to have placed one on the same machine. That's why you can run it without the built in server, and it doesn't know the difference (it's just way slower).

> well never mind, i will hack that into shape, i think. :-)

Good luck! We're all counting on you! (movie quote, lol)

Retro Guy
--
Posted on: Rocksolid Light
news.novabbs.com

ok, i did a proof of concept for posting, this works. in the end i skipped the webserver and stuff with curl, and just handed over the message as a file which the modified post.php parses (json encoded). so far so good. as long as both nntp server and vichan run on the same machine, this is a viable solution.

now for the files (so far it is text only).

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

>If you plan to run vichan as dependent on the rslight nntp server,

Yes, that is the plan. And I would also keep rslight running on the tor side.

> you would have a news server local that you could talk to. rslight would be the "other" nntp instance. Is that of any value?

Yes, absolutely. Right now the syncing from vichan to the news server works by just dropping the message in the outspool. Works well, and will most like stay that way.
The original way (for which I only have half the code from the vichan sources) worked based on ip addresses, which is not good in my case anyway.
The other way round (so from rslight to vichan) will work like I described in the message before, there is just no need for any complicated constructs if both applications are running on the same machine.

The nntp to nntp connection to update vichan is something I would not use if I have easier ways, and it seems I do (I leave it to rslight, and just read and write files).
So eventually this mod for vichan would be a plugin to an rslight installation.

And about forking: I guess that is little heavy for me, but I will upload my mods somewhere eventually (once this it is done well enough to see the light of day, so a little down the road).

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

anonymous wrote:

> The nntp to nntp connection to update vichan is something I would not use if I have easier ways, and it seems I do (I leave it to rslight, and just read and write files).
> So eventually this mod for vichan would be a plugin to an rslight installation.

So if I understand correctly, vichan will read a message from a file and apply it to it's database. If that is correct, how will vichan know the file exists, will it poll a directory?

--
Posted on: Rocksolid Light
news.novabbs.com

>So if I understand correctly, vichan will read a message from a file and apply it to it's database. If that is correct, how will vichan know the file exists, will it poll a directory?

that's correct. i was planning to call the vichan script from rslight in case of updates (guess from spoolnews).

Posted on def4

anon wrote:

>>So if I understand correctly, vichan will read a message from a file and apply it to it's database. If that is correct, how will vichan know the file exists, will it poll a directory?

> that's correct. i was planning to call the vichan script from rslight in case of updates (guess from spoolnews).

> Posted on def4

If you are reading each message as a file, would it work for rslight to just send vichan basically the equivalent of what you saw in /var/spool/rslight/rocksolid/local_groups.txt:
rocksolid.nodes:158
rocksolid.nodes.help:53
etc...

This could tell vichan where rslight has set message pointers and vichan can keep it's own list, then it can go pull the articles it doesn't have and update it's pointer list. If the pointers all match, there are no articles to import.

Is this idea of any use?

Retro Guy

--
Posted on: Rocksolid Light
news.novabbs.com

Yes, that approach could work as well. The downside would be that you would have to parse the messages again, while spoolnews is doing that already. Think I'd rather hand something very simple to vichan from the spoolnews script, like this:
{
"Newsgroups": "rocksolid.shared.test",
"Message-ID": "<83434435e73f113dcce2d23d237b6237$1@z5bqfv5v75kxy7pj.onion>",
"References": "",
"Date": "Thu, 14 May 2020 16:16 UTC",
"Subject": "nother post",
"From": "anonymous@anon.com (anonymous)",
"Content-Type": "text/plain; charset=utf-8; format=flowed",
"Content": "This is a test text"
} It is way easier to just do json_decode and have all the values, than to parse the files from the spool (this somewhat messy part is already taken care of, so I would not like to do it again).
Of course it is just a question of taste where the code should reside, at rslight or at vichan.

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

ok, that was easy, i took message_read and put all articles into something like this:

{
"header": {
"number": 100,
"id": "<96042908114f73b92a4b25ecaf31371d@def4>",
"from": "anon@anon.com",
"name": "anon",
"subject": "Re: sync trest def2",
"newsgroups": "rocksolid.shared.test",
"followup": null,
"date": 1589462807,
"date_thread": null,
"organization": "def5",
"xnoarchive": null,
"references": [
"<46f0e0c1993dc510ec8653ac72cb4445$1@z5bqfv5v75kxy7pj.onion>"
],
"bestreference": null,
"content_transfer_encoding": "8bit",
"mime_version": null,
"content_type": [
"text/plain"
],
"content_type_charset": [
"utf-8"
],
"content_type_name": [
"unnamed"
],
"content_type_boundary": null,
"content_type_format": [
"flowed"
],
"answers": null,
"isAnswer": false,
"username": "anon",
"user_agent": null,
"isReply": null,
"threadsize": null
},
"body": [
"nok, seen on def4nnPosted on def4n"
]
}

that is the json encoded array that you get when you typecaste the object that message_read returns.
now i only read this into my import routine from vichan.

right now i work on a batch import function, and try to wrap my head around some of the details of vichan (which is harder for me than it probably sounds). :-)

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

ok, i got a kind of lame import routine working (lame because it has to be called per message). but at least it works. got all the boards created too.

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

one thing vichan is clearly not prepared to do is to deal with replies to (yet) non existing messages. so no automagic fixing of threads or anything, like ff does.
so the options are: either delete the reference and import such message as a new thread start, or refuse to take it.
the second one looks cleaner to me, and gives the chance to later try it again, so I guess I will opt for this one.
In any event, vichan is sensitive here (should not be a problem once the thing is steadily updated).
Another thing I am contemplating is how many messages to keep. By design I think it keeps 100 messages per board (or something like that), and after starts to delete the oldest ones.
Would be different from the rest of the nodes, which keep the messages forever. Of course, that could be done with vichan as well. But I kind of like the idea of a volatile message base on one of the nodes.

What do you think, Retro ?

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion

anonymous wrote:

> one thing vichan is clearly not prepared to do is to deal with replies to (yet) non existing messages. so no automagic fixing of threads or anything, like ff does.
> so the options are: either delete the reference and import such message as a new thread start, or refuse to take it.
> the second one looks cleaner to me, and gives the chance to later try it again, so I guess I will opt for this one.

Do you mean put the message into some holding location and try again in the future to apply it to vichan's database of articles? One thing you can do is grep overview for the message-id of the referenced article. If the server has it, it will be in overview and you can find it. Every group containing articles has a groupname-overview (rocksolid.nodes-overview). Reading this file, you can crossreference message-id to article number, etc. It is tab separated:
160 Re: ddos defense for novabbs.com anonymous@anon.com (anonymous) Fri, 22 May 2020 10:50 UTC <e4cfbed07f97837a031302573d5ebefc$1@z5bqfv5v75kxy7pj.onion> <733615c1436ceffa04cd2d7eb3ec2546$1@news.novabbs.com> <f0ba1870346ce5f36eb9365850311bf7$1@z5bqfv5v75kxy7pj.onion> <715287e8bd4cc83e628a4e69ce6bd304$1@news.novabbs.com> 1709 12 Xref: debiantest rocksolid.nodes:160

> In any event, vichan is sensitive here (should not be a problem once the thing is steadily updated).
> Another thing I am contemplating is how many messages to keep. By design I think it keeps 100 messages per board (or something like that), and after starts to delete the oldest ones.
> Would be different from the rest of the nodes, which keep the messages forever. Of course, that could be done with vichan as well. But I kind of like the idea of a volatile message base on one of the nodes.

> What do you think, Retro ?

Expiring my count of messages is nicer than by date, so a group never becomes empty. This is also what Synchronet does and it works well. I like it.

Retro Guy

--
Posted on: Rocksolid Light
news.novabbs.com

>Do you mean put the message into some holding location and try again in the future to apply it to vichan's database of articles?

Yes, this is more or less what I meant.

>One thing you can do is grep overview for the message-id of the referenced article.

I used message_read for now, and made one new file to follow the status of the exports. Actually, this kind of thing would be better done in one of the existing code bases, this was just the fastest way for me to test the concept.

The code lines below export any new message that is found into a format that is translated later in the new post.php of vichan.

So I guess I could call this script at the end of spoolnews (or maybe just insert it).

On the other end, I have a modified post.php, which reads the oldest message (by name) and imports it. I could not modify it in a way yet that would allow to cycle through all existing messages, so the whole setup is just an interim solution yet (but it is importing the messages right now).

later more,

trw

<?php

include "config.inc.php";
include "$file_newsportal";
include 'message.inc.php';

$groups = array(
'rocksolid.nodes',
'rocksolid.nodes.help',
'rocksolid.nodes.announce',
'rocksolid.shared.encryption',
'rocksolid.shared.entertainment',
'rocksolid.shared.freenet',
'rocksolid.shared.general',
'rocksolid.shared.hacking',
'rocksolid.shared.helpdesk',
'rocksolid.shared.i2p',
'rocksolid.shared.linux',
'rocksolid.shared.news',
'rocksolid.shared.offtopic',
'rocksolid.programming',
'rocksolid.shared.rocksolid',
'rocksolid.shared.security',
'rocksolid.social',
'rocksolid.shared.test',
'rocksolid.shared.tor',
);
$watermarks = json_decode(file_get_contents('watermarks.json'), true);
$update = false;
foreach($groups as $group){
echo($group . "nr");
for ($i = 0; $i < 1000; $i++){
if(!in_array($i,$watermarks[$group])){
$message = message_read($i, 0, $group);
if($message){
echo("processing message # " . $i . " from group " . $group . "nr");
$paddedI = str_pad($i, 5, '0', STR_PAD_LEFT);
$jsonFile = fopen('/var/www/html/vi/vichan-master/debug/' . $paddedI . '_' . $group . '.json', 'w');
fwrite($jsonFile, json_encode((array) $message, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonFile);
array_push($watermarks[$group],$i);
$update = true;
}
}
}
} if($update){
echo("saving updated watermarksnr");
$jsonWatermarks = fopen('watermarks.json', 'w');
fwrite($jsonWatermarks, json_encode((array) $watermarks, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonWatermarks);
} else {
echo("no new files foundnr");
} exit;
?>

--
Posted on def2
z5bqfv5v75kxy7pj.onion