Rocksolid Light

Welcome to Rocksolid Light

register   nodelist   faq  


rocksolid / rocksolid.nodes / Re: hey, what do know ? vichan supports nntp

SubjectAuthor
* hey, what do know ? vichan supports nntpanon
+* Re: hey, what do know ? vichan supports nntpanon
|`* Re: hey, what do know ? vichan supports nntpRetro Guy
| `* Re: hey, what do know ? vichan supports nntpanon
|  `* Re: hey, what do know ? vichan supports nntpRetro Guy
|   `* Re: hey, what do know ? vichan supports nntpanon
|    `* Re: hey, what do know ? vichan supports nntpanon
|     `* Re: hey, what do know ? vichan supports nntpRetro Guy
|      `* Re: hey, what do know ? vichan supports nntpanon
|       `* Re: hey, what do know ? vichan supports nntpanon
|        +* Re: hey, what do know ? vichan supports nntpanon
|        |`* Re: hey, what do know ? vichan supports nntpanonymous
|        | +* Re: hey, what do know ? vichan supports nntpRetro Guy
|        | |`* Re: hey, what do know ? vichan supports nntpanon
|        | | `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        | |  `* Re: hey, what do know ? vichan supports nntpanonymous
|        | |   `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        | |    `* Re: hey, what do know ? vichan supports nntpanonymous
|        | |     `* Re: hey, what do know ? vichan supports nntpanon
|        | |      +* Re: hey, what do you know: vichan supports nntpanon
|        | |      |+- Re: hey, what do you know: vichan supports nntpRetro Guy
|        | |      |`* Re: hey, what do you know: vichan supports nntpanon
|        | |      | `- Re: hey, what do you know: vichan supports nntpRetro Guy
|        | |      `- Re: hey, what do know ? vichan supports nntpRetro Guy
|        | `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |  `* Re: hey, what do know ? vichan supports nntpanonymous
|        |   `* Re: hey, what do know ? vichan supports nntpanonymous
|        |    `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     +* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |`* Re: hey, what do know ? vichan supports nntpanonymous
|        |     | `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |  `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |   `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |    `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |     `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |      `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |       `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |        `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |         `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |          `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |           `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |            `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |             `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |              `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |               `* Re: hey, what do know ? vichan supports nntpanon
|        |     |                `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                 `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                  `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                   `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                    `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                     `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                      `* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                       +* Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                       |+- Re: hey, what do know ? vichan supports nntpanonymous
|        |     |                       |`- Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                       `* Re: hey, what do know ? vichan supports nntpRetro Guy
|        |     |                        `- Re: hey, what do know ? vichan supports nntpanonymous
|        |     `- Re: hey, what do know ? vichan supports nntpRetro Guy
|        `- Re: hey, what do know ? vichan supports nntpRetro Guy
+- Re: hey, what do know ? vichan supports nntpanon
+* Re: hey, what do know ? vichan supports nntpanon
|`* Re: hey, what do know ? vichan supports nntpanonymous
| +- Re: hey, what do you know....anon
| `* Re: hey, what do know ? vichan supports nntpanonymous
|  `* Re: hey, what do know ? vichan supports nntpRetro Guy
|   `- Re: hey, what do know ? vichan supports nntpanon
+* basics are workingAnonymous
|`- Re: basics are workingRetro Guy
+* guess i will open for business soonAnonymous
|+* Re: guess i will open for business soonMei
||`- Re: guess i will open for business soonRetro Guy
|`- Re: guess i will open for business soonRetro Guy
+- opsec check is goodAnonymous
+- > goo to go.Anonymous
+* yes that should workAnonymous
|`- Re: yes that should workRetro Guy
+* cron or not ?Anonymous
|`- Re: cron or not ?Retro Guy
+- themes on def2.i2pAnonymous
`* ok, ok, i switched the default themeAnonymous
 `* Re: ok, ok, i switched the default themeMei
  `- Re: ok, ok, i switched the default themeanon

Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Fri, 22 May 2020 19:24 UTC
ok, that was easy, i took message_read and put all articles into something like this:

{       "header": {
        "number": 100,
        "id": "<96042908114f73b92a4b25ecaf31371d@def4>",
        "from": "anon@anon.com",
        "name": "anon",
        "subject": "Re: sync trest def2",
        "newsgroups": "rocksolid.shared.test",
        "followup": null,
        "date": 1589462807,
        "date_thread": null,
        "organization": "def5",
        "xnoarchive": null,
        "references": [
            "<46f0e0c1993dc510ec8653ac72cb4445$1@z5bqfv5v75kxy7pj.onion>"
        ],
        "bestreference": null,
        "content_transfer_encoding": "8bit",
        "mime_version": null,
        "content_type": [
            "text/plain"
        ],
        "content_type_charset": [
            "utf-8"
        ],
        "content_type_name": [
            "unnamed"
        ],
        "content_type_boundary": null,
        "content_type_format": [
            "flowed"
        ],
        "answers": null,
        "isAnswer": false,
        "username": "anon",
        "user_agent": null,
        "isReply": null,
        "threadsize": null
    },
    "body": [
        "nok, seen on def4nnPosted on def4n"
    ]
}

that is the json encoded array that you get when you typecaste the object that message_read returns.
now i only read this into my import routine from vichan. right now i work on a batch import function, and try to wrap my head around some of the details of vichan (which is harder for me than it probably sounds). :-)

cheers

trw --
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Fri, 22 May 2020 20:53 UTC
ok, i got a kind of lame import routine working (lame because it has to be called per message). but at least it works. got all the boards created too.

cheers

trw
--
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Fri, 22 May 2020 21:50 UTC
one thing vichan is clearly not prepared to do is to deal with replies to (yet) non existing messages. so no automagic fixing of threads or anything, like ff does.
so the options are: either delete the reference and import such message as a new thread start, or refuse to take it.
the second one looks cleaner to me, and gives the chance to later try it again, so I guess I will opt for this one.
In any event, vichan is sensitive here (should not be a problem once the thing is steadily updated).
Another thing I am contemplating is how many messages to keep. By design I think it keeps 100 messages per board (or something like that), and after starts to delete the oldest ones.
Would be different from the rest of the nodes, which keep the messages forever. Of course, that could be done with vichan as well. But I kind of like the idea of a volatile message base on one of the nodes.

What do you think, Retro ?

cheers

trw  --
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do know ? vichan supports nntp
From: Retro Guy@rslight.i2p (Retro Guy)
Newsgroups: rocksolid.nodes
Organization: Rocksolid Light
Date: Sat, 23 May 2020 08:17 UTC
anonymous wrote:

one thing vichan is clearly not prepared to do is to deal with replies to (yet) non existing messages. so no automagic fixing of threads or anything, like ff does.
so the options are: either delete the reference and import such message as a new thread start, or refuse to take it.
the second one looks cleaner to me, and gives the chance to later try it again, so I guess I will opt for this one.

Do you mean put the message into some holding location and try again in the future to apply it to vichan's database of articles? One thing you can do is grep overview for the message-id of the referenced article. If the server has it, it will be in overview and you can find it. Every group containing articles has a groupname-overview (rocksolid.nodes-overview). Reading this file, you can crossreference message-id to article number, etc. It is tab separated:
160     Re: ddos defense for novabbs.com        anonymous@anon.com (anonymous)  Fri, 22 May 2020 10:50 UTC      <e4cfbed07f97837a031302573d5ebefc$1@z5bqfv5v75kxy7pj.onion> <733615c1436ceffa04cd2d7eb3ec2546$1@news.novabbs.com> <f0ba1870346ce5f36eb9365850311bf7$1@z5bqfv5v75kxy7pj.onion> <715287e8bd4cc83e628a4e69ce6bd304$1@news.novabbs.com>     1709    12      Xref: debiantest rocksolid.nodes:160

In any event, vichan is sensitive here (should not be a problem once the thing is steadily updated).
Another thing I am contemplating is how many messages to keep. By design I think it keeps 100 messages per board (or something like that), and after starts to delete the oldest ones.
Would be different from the rest of the nodes, which keep the messages forever. Of course, that could be done with vichan as well. But I kind of like the idea of a volatile message base on one of the nodes.

What do you think, Retro ?

Expiring my count of messages is nicer than by date, so a group never becomes empty. This is also what Synchronet does and it works well. I like it.

Retro Guy

--
Posted on: Rocksolid Light
news.novabbs.com



Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Sat, 23 May 2020 15:34 UTC
Do you mean put the message into some holding location and try again in the future to apply it to vichan's database of articles?

Yes, this is more or less what I meant.


One thing you can do is grep overview for the message-id of the referenced article.

I used message_read for now, and made one new file to follow the status of the exports. Actually, this kind of thing would be better done in one of the existing code bases, this was just the fastest way for me to test the concept.

The code lines below export any new message that is found into a format that is translated later in the new post.php of vichan.

So I guess I could call this script at the end of spoolnews (or maybe just insert it).

On the other end, I have a modified post.php, which reads the oldest message (by name) and imports it. I could not modify it in a way yet that would allow to cycle through all existing messages, so the whole setup is just an interim solution yet (but it is importing the messages right now).

later more,

trw



<?php

  include "config.inc.php";
  include "$file_newsportal";
  include 'message.inc.php';

$groups = array(
'rocksolid.nodes',
'rocksolid.nodes.help',
'rocksolid.nodes.announce',
'rocksolid.shared.encryption',
'rocksolid.shared.entertainment',
'rocksolid.shared.freenet',
'rocksolid.shared.general',
'rocksolid.shared.hacking',
'rocksolid.shared.helpdesk',
'rocksolid.shared.i2p',
'rocksolid.shared.linux',
'rocksolid.shared.news',
'rocksolid.shared.offtopic',
'rocksolid.programming',
'rocksolid.shared.rocksolid',
'rocksolid.shared.security',
'rocksolid.social',
'rocksolid.shared.test',
'rocksolid.shared.tor',
);
$watermarks = json_decode(file_get_contents('watermarks.json'), true);
$update = false;
foreach($groups as $group){
echo($group . "nr");
for ($i = 0; $i < 1000; $i++){
if(!in_array($i,$watermarks[$group])){
$message = message_read($i, 0, $group);
if($message){
echo("processing message # " . $i . " from group " . $group . "nr");
$paddedI = str_pad($i, 5, '0', STR_PAD_LEFT);
$jsonFile = fopen('/var/www/html/vi/vichan-master/debug/' . $paddedI . '_' . $group . '.json', 'w');
fwrite($jsonFile, json_encode((array) $message, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonFile);
array_push($watermarks[$group],$i);
$update = true; }
}
}
}
if($update){
echo("saving updated watermarksnr");
$jsonWatermarks = fopen('watermarks.json', 'w');
fwrite($jsonWatermarks, json_encode((array) $watermarks, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES));
fclose($jsonWatermarks);
} else {
echo("no new files foundnr");
}
exit;
?>


--
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Sat, 23 May 2020 21:33 UTC
so, i have got it basically running, so messages gets synced to and from vichan. it is still sort of clumsy right now...
the fields "date" and "from" are not acknowledged for some reason, so posts get the timestamp of the import, and are "forced anonymous". guess i can live with both, but will try to get it done right.
the work is a bit tedious, but will be definitely be worth it in the end. vichan is super easy to configure, modify, moderate, and it is just looking very nice (still playing with all the css that come with it, but will most likely stay on the dark one).

the main challenge to modify the code from vichan and run it differently seems to be that the context changes depending on how the scripts are run, and in most cases this lets any file inclusions fail (calling from the cli works as well as calling from the webserver, but running it from another script fails, even if the directory is changed).

cheers

trw  --
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Sat, 23 May 2020 22:12 UTC
   Do you mean put the message into some holding location and try again in >>the future to apply it to vichan's database of articles?
Yes, this is more or less what I meant.

Tried that one, and for sure it did not work. Either a message which was rejected once is not considered anymore, or there is some serious issue with the references.
While the first would not be an issue for new threads, the second one would require some coding to rewrite the references (shudder...).
--
Posted on def2
z5bqfv5v75kxy7pj.onion


Subject: Re: hey, what do know ? vichan supports nntp
From: Retro Guy@rslight.i2p (Retro Guy)
Newsgroups: rocksolid.nodes
Organization: Rocksolid Light
Date: Sun, 24 May 2020 00:51 UTC
anonymous wrote:

so, i have got it basically running, so messages gets synced to and from vichan. it is still sort of clumsy right now...
the fields "date" and "from" are not acknowledged for some reason, so posts get the timestamp of the import, and are "forced anonymous". guess i can live with both, but will try to get it done right.

Did you change something since you wrote the above? I'm seeing usernames in r.s.test starting with this post:
Re: Does this encryption thing work ? Retro Guy@rslight.i2p 05/23/20 (Sat) 23:29:02 No.141

the work is a bit tedious, but will be definitely be worth it in the end. vichan is super easy to configure, modify, moderate, and it is just looking very nice (still playing with all the css that come with it, but will most likely stay on the dark one).

It is looking great!

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com



Subject: Re: hey, what do know ? vichan supports nntp
From: Retro Guy@rslight.i2p (Retro Guy)
Newsgroups: rocksolid.nodes
Organization: Rocksolid Light
Date: Sun, 24 May 2020 00:57 UTC
anonymous wrote:

One thing you can do is grep overview for the message-id of the referenced article.

I used message_read for now, and made one new file to follow the status of the exports. Actually, this kind of thing would be better done in one of the existing code bases, this was just the fastest way for me to test the concept.

Same as the way I test. I just write a test script and get it working, then apply to existing codebase. Much easier to not break existing code that way.

The code lines below export any new message that is found into a format that is translated later in the new post.php of vichan.

So I guess I could call this script at the end of spoolnews (or maybe just insert it).

What I'd like to see, if it's possible, is calling from spoolnews. We just put an $enable_vichan variable in config so if using vichan it will run the code and send the data needed by vichan. If disabled, it does nothing.

On the other end, I have a modified post.php, which reads the oldest message (by name) and imports it. I could not modify it in a way yet that would allow to cycle through all existing messages, so the whole setup is just an interim solution yet (but it is importing the messages right now).

Since spoolnews loops through every message that it imports, it should work to just make the call to format and send to vichan in that loop. Every message rslight decides it wants (let rslight determine if a message is new or already existing), will then get sent to vichan. Thoughts?

Retro Guy

--
Posted on: Rocksolid Light
www.novabbs.com



Subject: Re: hey, what do know ? vichan supports nntp
From: anonymous@anon.com (anonymous)
Newsgroups: rocksolid.nodes
Organization: def2
Date: Sun, 24 May 2020 11:23 UTC
Since spoolnews loops through every message that it imports, it should work to just make the call to format and send to vichan in that loop. Every message rslight decides it wants (let rslight determine if a message is new or already existing), will then get sent to vichan. Thoughts?


Yes, that is the right approach imo. I just need to dig out the id of the message that started the thread before I can send it, and insert it as the first reference.


The usernames in the messages where easy to switch on.
--
Posted on def2
def2.i2p


Pages:123456789
rocksolid light 0.6.5e
clearnet i2p tor