Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  newsreader  groups  login

Message-ID:  

You have a tendency to feel you are superior to most computers.


computers / comp.os.linux.misc / Suppress exit status of system() command

SubjectAuthor
* Suppress exit status of system() commandJohn Forkosh
+* Re: Suppress exit status of system() commandThe Natural Philosopher
|`- Re: Suppress exit status of system() commandJohn Forkosh
+* Re: Suppress exit status of system() commandLew Pitcher
|`* Re: Suppress exit status of system() commandJohn Forkosh
| `* Re: Suppress exit status of system() commandLew Pitcher
|  `- Re: Suppress exit status of system() commandJohn Forkosh
`* Re: Suppress exit status of system() commandRich
 `* Re: Suppress exit status of system() commandJohn Forkosh
  +* Re: Suppress exit status of system() commandThe Natural Philosopher
  |`* Re: Suppress exit status of system() commandJohn Forkosh
  | +* Re: Suppress exit status of system() commandRich
  | |`* Re: Suppress exit status of system() commandThe Natural Philosopher
  | | `* Re: Suppress exit status of system() commandRich
  | |  `- Re: Suppress exit status of system() commandThe Natural Philosopher
  | `- Re: Suppress exit status of system() commandThe Natural Philosopher
  +* Re: Suppress exit status of system() commandRich
  |`* Re: Suppress exit status of system() commandJohn Forkosh
  | `- Re: Suppress exit status of system() commandRich
  `* Re: Suppress exit status of system() commandRich
   `* Re: Suppress exit status of system() commandJohn Forkosh
    +* Re: Suppress exit status of system() commandRich
    |`- Re: Suppress exit status of system() commandThe Natural Philosopher
    `- Re: Suppress exit status of system() commandAndreas Kohlbach

1
Suppress exit status of system() command

<s6letp$hu1$1@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4905&group=comp.os.linux.misc#4905

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!news.niel.me!news.gegeweb.eu!gegeweb.org!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Suppress exit status of system() command
Date: Sun, 2 May 2021 05:56:10 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 51
Message-ID: <s6letp$hu1$1@reader1.panix.com>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1619934970 18369 166.84.1.3 (2 May 2021 05:56:10 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Sun, 2 May 2021 05:56:10 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Sun, 2 May 2021 05:56 UTC

I have a little one-line awk/shell script like this...
awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
which analyzes my website logs, producing output like this...
672 62.210.98.10
178 54.173.189.222
116 23.100.232.233
116 101.19.4.45
88 151.38.64.253
etc
which is merely a count (on the left) of the number of times
each ip-address (on the right) has accessed my site, highest
count first.

Problem with that is there are many accesses with the same
first two aaa.bbb.. and different ..ccc.ddd. So they're
treated separately, even though they most likely should
be aggregated. So I wrote a little C program, ipprefix.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main ( int argc, char *argv[] ) {
char ipstr[999]="\000", *delim=NULL;
int ndots=2, idot=0;
if ( argc > 1 ) strcpy(ipstr,argv[1]);
if ( (delim=strchr(ipstr,',')) != NULL ) {
*delim = '\000'; ndots=atoi(delim+1); }
for ( delim=ipstr-1,idot=1; idot<=ndots; idot++ ) {
delim = strchr(delim+1,'.');
if ( delim == NULL ) break; }
if ( delim != NULL ) *delim = '\000';
printf("%s",ipstr);
} /* --- end-of-job --- */
whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb

And then I modified the awk/shell script like this...
awk '{split($0,ip,"- -"); print (system("ipprefix "ip[1]))}' |
sort | uniq -c | sort -nr

So the annoying little problem is that its output is now...
1986 116.1790
672 62.2100
576 114.1190
355 140.820
etc
I've checked, and the ip addresses are all correct except for
that extra "0" at the end of every line. So it's obviously, I think,
the exit status from system(), and I can just ignore it. But I'd
prefer to suppress system()'s exit status so there is no
extraneous "0" to begin with. Is there any way to do that?
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Re: Suppress exit status of system() command

<s6lr9t$3kh$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4906&group=comp.os.linux.misc#4906

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: tnp...@invalid.invalid (The Natural Philosopher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Sun, 2 May 2021 10:27:24 +0100
Organization: A little, after lunch
Lines: 195
Message-ID: <s6lr9t$3kh$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Injection-Date: Sun, 2 May 2021 09:27:25 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="513566011fd3537c42111001871a144f";
logging-data="3729"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+4jx7hgDCBToE/hq6IVJJ9LAMhHk8F5jo="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Thunderbird/60.6.1
Cancel-Lock: sha1:nXqWxoH0AWTeYFoCQgq6Op4uwLo=
In-Reply-To: <s6letp$hu1$1@reader1.panix.com>
Content-Language: en-GB
 by: The Natural Philosop - Sun, 2 May 2021 09:27 UTC

On 02/05/2021 06:56, John Forkosh wrote:
> I have a little one-line awk/shell script like this...
> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
> which analyzes my website logs, producing output like this...
> 672 62.210.98.10
> 178 54.173.189.222
> 116 23.100.232.233
> 116 101.19.4.45
> 88 151.38.64.253
> etc
> which is merely a count (on the left) of the number of times
> each ip-address (on the right) has accessed my site, highest
> count first.
>
> Problem with that is there are many accesses with the same
> first two aaa.bbb.. and different ..ccc.ddd. So they're
> treated separately, even though they most likely should
> be aggregated. So I wrote a little C program, ipprefix.c
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> int main ( int argc, char *argv[] ) {
> char ipstr[999]="\000", *delim=NULL;
> int ndots=2, idot=0;
> if ( argc > 1 ) strcpy(ipstr,argv[1]);
> if ( (delim=strchr(ipstr,',')) != NULL ) {
> *delim = '\000'; ndots=atoi(delim+1); }
> for ( delim=ipstr-1,idot=1; idot<=ndots; idot++ ) {
> delim = strchr(delim+1,'.');
> if ( delim == NULL ) break; }
> if ( delim != NULL ) *delim = '\000';
> printf("%s",ipstr);
> } /* --- end-of-job --- */
> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb
>
> And then I modified the awk/shell script like this...
> awk '{split($0,ip,"- -"); print (system("ipprefix "ip[1]))}' |
> sort | uniq -c | sort -nr
>
> So the annoying little problem is that its output is now...
> 1986 116.1790
> 672 62.2100
> 576 114.1190
> 355 140.820
> etc
> I've checked, and the ip addresses are all correct except for
> that extra "0" at the end of every line. So it's obviously, I think,
> the exit status from system(), and I can just ignore it. But I'd
> prefer to suppress system()'s exit status so there is no
> extraneous "0" to begin with. Is there any way to do that?
>

Honestly, if you have gone so far with C, why not finish the job?
This is juts one more example of 'oh here a tool that looks like it will
make te job easier, after I have fixed this issue, and that issue, and
this other issue', ...and in the end you look and see that 'the short
cut was longer, by far'

The way I did all this is as follows

Firslty pipe apache logs to a program -n te apoache conf file...

CustomLog "|/usr/local/bin/ipcounter" iponly env=!loopback

The program that accepts the pipe logs ip addresses and hits in a mysql
databse

/*
This takes IP addresses as input from a pipe from apache
It waits till the stream of addresses changes, then logs the result to the
sql table hits with the date of the last access in database gridwatch.
*/
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>
#include <time.h>
#include <mysql/mysql.h>

#define DBUSER "user"
#define DBPASS "password"
#define DBNAME "database"
#define DBTABLE "hits"

//#define DEBUG
#ifdef DEBUG
#define LOGFILE "/var/log/apache2/ipcounter.log"
FILE *logfp=0;

void logwrite(char *data)
{
if(!logfp)
logfp=fopen(LOGFILE,"w");
while(*data)
{
fputc(*data++,logfp);
}
fputc('\n',logfp);
fflush(logfp);
}
#endif

MYSQL mysql;
MYSQL_RES *result;
MYSQL_ROW row;

int get_line(char *buf)
{
char c;
char *p;
int count;
count=0;
p=buf;
while ((c = getchar()) !=EOF)
{
if(c=='\n')
{
*p++=0;
return 0;
}
if(count>16)
{
*p=0;
return (count);
}
*p++=c;
count++;
}
return (EOF);
}
void update_database(char *ip, int count)
{
char query[1024];
sprintf(query,"insert into hits set ip='%s', timestamp=now(),
count='%d' ON DUPLICATE KEY UPDATE timestamp=now(), count=count+'%d'
",ip, count, count );
while (mysql_ping(&mysql)) // check server still there...if not wait a
second and try again...and again..
{
sleep (1);
mysql_real_connect(&mysql,"127.0.0.1",DBUSER,DBPASS,DBNAME,0,"",0);
}
mysql_query(&mysql,query);
}
int main()
{
char last_ip[24];
char ip[24];
last_ip[0]=0; // initialise to null
int count=1;
// open database
if(!mysql_init(&mysql)) // initalise data structure
return 1;
if(!mysql_real_connect(&mysql,"127.0.0.1",DBUSER,DBPASS,DBNAME,0,"",0))
// connect to database
{
printf("Connect failed -%s\n",mysql_error(&mysql));
return 2;
}
// read a line from stdin
while (get_line(ip) !=EOF)
{
#ifdef DEBUG
logwrite(ip);
#endif
if(strcmp(last_ip,ip) || count > 20) // found new input line or just
too many hits from one source
{
if(*last_ip) // dont update with a null entry!
update_database(last_ip,count);
count=1; //reset counter
strcpy(last_ip,ip);
}
else count++;
}
mysql_close(&mysql);
}
--------------------------------------------
That allows me to record number of separate ip addresses that hit the site.

And build nice graphs using a cron script

e.g.

https://gridwatch.templar.co.uk/admin/

--
“Progress is precisely that which rules and regulations did not foresee,”

– Ludwig von Mises

Re: Suppress exit status of system() command

<s6lv9p$84i$1@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4908&group=comp.os.linux.misc#4908

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!aioe.org!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Sun, 2 May 2021 10:35:37 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 234
Message-ID: <s6lv9p$84i$1@reader1.panix.com>
References: <s6letp$hu1$1@reader1.panix.com> <s6lr9t$3kh$1@dont-email.me>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1619951737 8338 166.84.1.3 (2 May 2021 10:35:37 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Sun, 2 May 2021 10:35:37 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Sun, 2 May 2021 10:35 UTC

The Natural Philosopher <tnp@invalid.invalid> wrote:
> John Forkosh wrote:
>> I have a little one-line awk/shell script like this...
>> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
>> which analyzes my website logs, producing output like this...
>> 672 62.210.98.10
>> 178 54.173.189.222
>> 116 23.100.232.233
>> 116 101.19.4.45
>> 88 151.38.64.253
>> etc
>> which is merely a count (on the left) of the number of times
>> each ip-address (on the right) has accessed my site, highest
>> count first.
>>
>> Problem with that is there are many accesses with the same
>> first two aaa.bbb.. and different ..ccc.ddd. So they're
>> treated separately, even though they most likely should
>> be aggregated. So I wrote a little C program, ipprefix.c
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <string.h>
>> int main ( int argc, char *argv[] ) {
>> char ipstr[999]="\000", *delim=NULL;
>> int ndots=2, idot=0;
>> if ( argc > 1 ) strcpy(ipstr,argv[1]);
>> if ( (delim=strchr(ipstr,',')) != NULL ) {
>> *delim = '\000'; ndots=atoi(delim+1); }
>> for ( delim=ipstr-1,idot=1; idot<=ndots; idot++ ) {
>> delim = strchr(delim+1,'.');
>> if ( delim == NULL ) break; }
>> if ( delim != NULL ) *delim = '\000';
>> printf("%s",ipstr);
>> } /* --- end-of-job --- */
>> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb
>>
>> And then I modified the awk/shell script like this...
>> awk '{split($0,ip,"- -"); print (system("ipprefix "ip[1]))}' |
>> sort | uniq -c | sort -nr
>>
>> So the annoying little problem is that its output is now...
>> 1986 116.1790
>> 672 62.2100
>> 576 114.1190
>> 355 140.820
>> etc
>> I've checked, and the ip addresses are all correct except for
>> that extra "0" at the end of every line. So it's obviously, I think,
>> the exit status from system(), and I can just ignore it. But I'd
>> prefer to suppress system()'s exit status so there is no
>> extraneous "0" to begin with. Is there any way to do that?
>
> Honestly, if you have gone so far with C, why not finish the job?
> This is just one more example of 'oh here's a tool that looks like
> it will make the job easier, after I have fixed this issue, and
> that issue, and this other issue', ...and in the end you look and see
> that 'the short cut was longer, by far'

Thanks, Natural Philosopher, for your remarks and the program below.
One pragmatic reason I did it this way is because it's not yet clear
to me what data needs to be collected from the logs (maybe what tables
in a mysql kind of view), what variables sorted by, summed over, etc.
The ultimate problem that needs solving is some dos and ddos attacks
hitting some of my .cgi programs over-and-over-and-over-and...,
and just some "deny from"'s in .htaccess, and some other stuff,
hasn't sufficiently reduced the problem. So I've been looking at
the logs this-way and that-way and any-other-way, just looking for
ideas about what might work better. That is, who are the main
culprits, exactly what are they doing, etc.

So these scripts are just one-line throwaways to help that effort.
If I knew what I actually wanted/needed, then it might be worth
the effort to program it more elaborately. But lots of effort at
this point in the "development cycle" would be wasted; they're
just "prototypes" at best. So I can easily live with that
concatanated "0". I only asked because for some reason the
question interested me, not because I really needed the problem
solved. I figured there'd either be a really easy answer, or
a followup of the form "it can't be done", and either way would
be okay.

Another, not-progmatic reason, is because I gotta say that
I can't quite agree with your
" This is just one more example of 'oh here's a tool that looks
like it will make the job easier, after I have fixed this issue,
and that issue, and this other issue', ...and in the end you look
and see that 'the short cut was longer, by far' "
At least, in this kind of case I can't quite agree. All these
so-called "little languages" are meant for just this kind of
purpose I have here, and they work quite well in the problem domain
they were designed for, e.g., just munging through some data and
spitting out the result. Rather than your "the short cut was longer,
by far", I'd suggest "don't swat a fly with a sledgehammer".
I use C a lot, and love it, but don't think it's always the best
solution to address every problem. And note that in this particular
case, I hadn't even initially anticipated the need for that little
ipprefix program. Once I realized it was needed, the five minutes
it took to write it (way less time than writing these posts) was
quite a lot less than starting from scratch in C.

> The way I did all this is as follows
>
> Firstly pipe apache logs to a program -n te apoache conf file...
>
> CustomLog "|/usr/local/bin/ipcounter" iponly env=!loopback
>
>
> The program that accepts the pipe logs ip addresses and hits in a mysql
> databse
>
> /*
> This takes IP addresses as input from a pipe from apache
> It waits till the stream of addresses changes, then logs the result to the
> sql table hits with the date of the last access in database gridwatch.
> */
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <unistd.h>
> #include <time.h>
> #include <mysql/mysql.h>
>
> #define DBUSER "user"
> #define DBPASS "password"
> #define DBNAME "database"
> #define DBTABLE "hits"
>
> //#define DEBUG
> #ifdef DEBUG
> #define LOGFILE "/var/log/apache2/ipcounter.log"
> FILE *logfp=0;
>
> void logwrite(char *data)
> {
> if(!logfp)
> logfp=fopen(LOGFILE,"w");
> while(*data)
> {
> fputc(*data++,logfp);
> }
> fputc('\n',logfp);
> fflush(logfp);
> }
> #endif
>
> MYSQL mysql;
> MYSQL_RES *result;
> MYSQL_ROW row;
>
> int get_line(char *buf)
> {
> char c;
> char *p;
> int count;
> count=0;
> p=buf;
> while ((c = getchar()) !=EOF)
> {
> if(c=='\n')
> {
> *p++=0;
> return 0;
> }
> if(count>16)
> {
> *p=0;
> return (count);
> }
> *p++=c;
> count++;
> }
> return (EOF);
> }
> void update_database(char *ip, int count)
> {
> char query[1024];
> sprintf(query,"insert into hits set ip='%s', timestamp=now(),
> count='%d' ON DUPLICATE KEY UPDATE timestamp=now(), count=count+'%d'
> ",ip, count, count );
> while (mysql_ping(&mysql)) // check server still there...if not wait a
> second and try again...and again..
> {
> sleep (1);
> mysql_real_connect(&mysql,"127.0.0.1",DBUSER,DBPASS,DBNAME,0,"",0);
> }
> mysql_query(&mysql,query);
> }
> int main()
> {
> char last_ip[24];
> char ip[24];
> last_ip[0]=0; // initialise to null
> int count=1;
> // open database
> if(!mysql_init(&mysql)) // initalise data structure
> return 1;
> if(!mysql_real_connect(&mysql,"127.0.0.1",DBUSER,DBPASS,DBNAME,0,"",0))
> // connect to database
> {
> printf("Connect failed -%s\n",mysql_error(&mysql));
> return 2;
> }
> // read a line from stdin
> while (get_line(ip) !=EOF)
> {
> #ifdef DEBUG
> logwrite(ip);
> #endif
> if(strcmp(last_ip,ip) || count > 20) // found new input line or just
> too many hits from one source
> {
> if(*last_ip) // dont update with a null entry!
> update_database(last_ip,count);
> count=1; //reset counter
> strcpy(last_ip,ip);
> }
> else count++;
> }
> mysql_close(&mysql);
> }
>
> --------------------------------------------
> That allows me to record number of separate ip addresses that hit the site.
>
> And build nice graphs using a cron script
>
> e.g.
>
> https://gridwatch.templar.co.uk/admin/


Click here to read the complete article
Re: Suppress exit status of system() command

<s6mca3$k0n$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4909&group=comp.os.linux.misc#4909

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: lew.pitc...@digitalfreehold.ca (Lew Pitcher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Sun, 2 May 2021 14:17:39 -0000 (UTC)
Organization: The Pitcher Digital Freehold
Lines: 83
Message-ID: <s6mca3$k0n$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Date: Sun, 2 May 2021 14:17:39 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="8926c305f077fcb4c5b787e6907ef049";
logging-data="20503"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1884eBctdvP4gV2KGHT/SyXcF125Kt6+0o="
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
git://git.gnome.org/pan2)
Cancel-Lock: sha1:/b9RjriwNV29Pv/QNvOim9qCjBY=
 by: Lew Pitcher - Sun, 2 May 2021 14:17 UTC

On Sun, 02 May 2021 05:56:10 +0000, John Forkosh wrote:

> I have a little one-line awk/shell script like this...
> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
> which analyzes my website logs, producing output like this...
> 672 62.210.98.10
> 178 54.173.189.222
> 116 23.100.232.233
> 116 101.19.4.45
> 88 151.38.64.253
> etc
> which is merely a count (on the left) of the number of times
> each ip-address (on the right) has accessed my site, highest
> count first.
>
> Problem with that is there are many accesses with the same
> first two aaa.bbb.. and different ..ccc.ddd. So they're
> treated separately, even though they most likely should
> be aggregated. So I wrote a little C program, ipprefix.c
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> int main ( int argc, char *argv[] ) {
> char ipstr[999]="\000", *delim=NULL;
> int ndots=2, idot=0;
> if ( argc > 1 ) strcpy(ipstr,argv[1]);
> if ( (delim=strchr(ipstr,',')) != NULL ) {
> *delim = '\000'; ndots=atoi(delim+1); }
> for ( delim=ipstr-1,idot=1; idot<=ndots; idot++ ) {
> delim = strchr(delim+1,'.');
> if ( delim == NULL ) break; }
> if ( delim != NULL ) *delim = '\000';
> printf("%s",ipstr);
> } /* --- end-of-job --- */
> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb
>
> And then I modified the awk/shell script like this...
> awk '{split($0,ip,"- -"); print (system("ipprefix "ip[1]))}' |
/
Why, there's your problem -----+

The awk function system() "execute[s] the command ... and return[s]
the exit status"

The awk function print() (in this usage) "print[s] expressions"

Taken as
print(system("Some command"))
the print() function will print the result of the system() function,
which will be it's exit status.

(FWIW, the system() function probably uses the system(3) stdlib function
under the covers, so the exit status will likely be one of those documented
under system(3).

To fix this, just do not use the awk print function in this manner. Instead,
(after looking at your ipprefix.c code) you probably want to rephrase the awk
code from
print (system("ipprefix "ip[1]))
to
system("ipprefix "ip[1]); print;

> sort | uniq -c | sort -nr
>
> So the annoying little problem is that its output is now...
> 1986 116.1790
> 672 62.2100
> 576 114.1190
> 355 140.820
> etc
> I've checked, and the ip addresses are all correct except for
> that extra "0" at the end of every line. So it's obviously, I think,
> the exit status from system(), and I can just ignore it. But I'd
> prefer to suppress system()'s exit status so there is no
> extraneous "0" to begin with. Is there any way to do that?

--
Lew Pitcher
"In Skills, We Trust"

Re: Suppress exit status of system() command

<s6mfjd$an5$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4910&group=comp.os.linux.misc#4910

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ric...@example.invalid (Rich)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Sun, 2 May 2021 15:13:49 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 58
Message-ID: <s6mfjd$an5$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com>
Injection-Date: Sun, 2 May 2021 15:13:49 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="bb4a155c58f76f9b4f9d780028691327";
logging-data="10981"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18x1qQ3OrgEgHCzSWxF3Far"
User-Agent: tin/2.0.1-20111224 ("Achenvoir") (UNIX) (Linux/3.10.17 (x86_64))
Cancel-Lock: sha1:PnhSdRa30FbnOuXsGGPrKv2O0UQ=
 by: Rich - Sun, 2 May 2021 15:13 UTC

John Forkosh <forkosh@panix.com> wrote:
> I have a little one-line awk/shell script like this...
> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
> which analyzes my website logs, producing output like this...
> 672 62.210.98.10
> 178 54.173.189.222
> 116 23.100.232.233
> 116 101.19.4.45
> 88 151.38.64.253
> ...

> Problem with that is there are many accesses with the same
> first two aaa.bbb.. and different ..ccc.ddd. So they're
> treated separately, even though they most likely should
> be aggregated. So I wrote a little C program, ipprefix.c
> ...
> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb

In your response to TNP, you indicated that this was/is just some
"one-offs" because you are still exploring exactly how you want to
slice/dice/summarize this.

So, given that, the output of your current, simple, awk script is IP
addresses. Your count of duplicates is done by "uniq -c" and the
sorting is by sort.

Now you want to convert ip addresses of aaa.bbb.ccc.ddd to aaa.bbb (I'm
not sure that is really what you'll want long term, but it is what you
think will be useful now).

In this, self expressed, experimentation stage, converting
aaa.bbb.ccc.ddd into aaa.bbb is a job for the 'cut' utility, not a C
program:

$ echo aaa.bbb.ccc.ddd | cut -f 1-2 -d .
aaa.bbb

So your pipeline would become:

awk '{split($0,ip,"- -"); print ip[1]}' | cut -f 1-2 -d . | sort | uniq -c | sort -nr

And you get the same output as with your C program, without writing any
code at all. And you get the added benefit that should you decide you
wanted aaa.bbb.ccc.ddd to become ccc.ddd instead, you just change the
cut invocation to be:

$ echo aaa.bbb.ccc.ddd | cut -f 3-4 -d .
ccc.ddd

And you get ccc.ddd out. You can even doing things like extract aaa
and ccc:

$ echo aaa.bbb.ccc.ddd | cut -f 1,3 -d .
aaa.ccc

So experimenting which which prefix/suffix/portion works well for
clustering becomes simply changing the cut invocation.

Re: Suppress exit status of system() command

<s6nu1n$4ld$1@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4911&group=comp.os.linux.misc#4911

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!weretis.net!feeder8.news.weretis.net!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Mon, 3 May 2021 04:26:31 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 89
Message-ID: <s6nu1n$4ld$1@reader1.panix.com>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1620015991 4781 166.84.1.3 (3 May 2021 04:26:31 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Mon, 3 May 2021 04:26:31 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Mon, 3 May 2021 04:26 UTC

Rich <rich@example.invalid> wrote:
> John Forkosh <forkosh@panix.com> wrote:
>> I have a little one-line awk/shell script like this...
>> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
>> which analyzes my website logs, producing output like this...
>> 672 62.210.98.10
>> 178 54.173.189.222
>> 116 23.100.232.233
>> 116 101.19.4.45
>> 88 151.38.64.253
>> ...
>
>> Problem with that is there are many accesses with the same
>> first two aaa.bbb.. and different ..ccc.ddd. So they're
>> treated separately, even though they most likely should
>> be aggregated. So I wrote a little C program, ipprefix.c
>> ...
>> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb
>
> In your response to TNP, you indicated that this was/is just some
> "one-offs" because you are still exploring exactly how you want to
> slice/dice/summarize this.
>
> So, given that, the output of your current, simple, awk script is IP
> addresses. Your count of duplicates is done by "uniq -c" and the
> sorting is by sort.
>
> Now you want to convert ip addresses of aaa.bbb.ccc.ddd to aaa.bbb (I'm
> not sure that is really what you'll want long term, but it is what you
> think will be useful now).
>
> In this, self expressed, experimentation stage, converting
> aaa.bbb.ccc.ddd into aaa.bbb is a job for the 'cut' utility, not a C
> program:
>
> $ echo aaa.bbb.ccc.ddd | cut -f 1-2 -d .
> aaa.bbb
>
> So your pipeline would become:
>
> awk '{split($0,ip,"- -"); print ip[1]}' | cut -f 1-2 -d . | sort | uniq -c | sort -nr
>
> And you get the same output as with your C program, without writing any
> code at all. And you get the added benefit that should you decide you
> wanted aaa.bbb.ccc.ddd to become ccc.ddd instead, you just change the
> cut invocation to be:
>
> $ echo aaa.bbb.ccc.ddd | cut -f 3-4 -d .
> ccc.ddd
>
> And you get ccc.ddd out. You can even doing things like extract aaa
> and ccc:
>
> $ echo aaa.bbb.ccc.ddd | cut -f 1,3 -d .
> aaa.ccc
>
> So experimenting which which prefix/suffix/portion works well for
> clustering becomes simply changing the cut invocation.

Yup! Thanks, Rich. That worked immediately, if not sooner.
And by that I mean it saved the million-or-so runs of my
now-unnecessary little C program, which was taking 30-60 secs
to load and load and..., once for each line in the log file.
Now the whole thing runs in the blink of an eye.

And cut ...who knew? So many utilities, so little time.
Thanks, again.
P.S. re "clustering", 1-2 looks about right. whois shows
most of the attacks coming from sources with a pretty large
range of ip addresses, and using many different ..ccc.ddd's
all from the same source, e.g.,
1986 116.179 (using your terrific fix)
"resolves" into
19 116.179.37.39
17 116.179.37.44
16 116.179.37.6
16 116.179.37.55
16 116.179.37.12
15 116.179.37.122
14 116.179.37.9
14 116.179.37.8
etc (lots and lots more with fewer and fewer hits from each)
presumably adding up to 1986 (but I didn't manually check:).
So, e.g., "deny from 116.179." seems like the appropriate fix.
Actually, "deny from china" seems like the even better fix,
given all the other info from the logs, but I haven't figured out
how to do that at one fell swoop.
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Re: Suppress exit status of system() command

<s6nv3v$3vh$1@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4912&group=comp.os.linux.misc#4912

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!paganini.bofh.team!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Mon, 3 May 2021 04:44:47 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 97
Message-ID: <s6nv3v$3vh$1@reader1.panix.com>
References: <s6letp$hu1$1@reader1.panix.com> <s6mca3$k0n$1@dont-email.me>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1620017087 4081 166.84.1.3 (3 May 2021 04:44:47 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Mon, 3 May 2021 04:44:47 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Mon, 3 May 2021 04:44 UTC

Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
> John Forkosh wrote:
>> I have a little one-line awk/shell script like this...
>> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
>> which analyzes my website logs, producing output like this...
>> 672 62.210.98.10
>> 178 54.173.189.222
>> 116 23.100.232.233
>> 116 101.19.4.45
>> 88 151.38.64.253
>> etc
>> which is merely a count (on the left) of the number of times
>> each ip-address (on the right) has accessed my site, highest
>> count first.
>>
>> Problem with that is there are many accesses with the same
>> first two aaa.bbb.. and different ..ccc.ddd. So they're
>> treated separately, even though they most likely should
>> be aggregated. So I wrote a little C program, ipprefix.c
>> #include <stdio.h>
>> #include <stdlib.h>
>> #include <string.h>
>> int main ( int argc, char *argv[] ) {
>> char ipstr[999]="\000", *delim=NULL;
>> int ndots=2, idot=0;
>> if ( argc > 1 ) strcpy(ipstr,argv[1]);
>> if ( (delim=strchr(ipstr,',')) != NULL ) {
>> *delim = '\000'; ndots=atoi(delim+1); }
>> for ( delim=ipstr-1,idot=1; idot<=ndots; idot++ ) {
>> delim = strchr(delim+1,'.');
>> if ( delim == NULL ) break; }
>> if ( delim != NULL ) *delim = '\000';
>> printf("%s",ipstr);
>> } /* --- end-of-job --- */
>> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb
>>
>> And then I modified the awk/shell script like this...
>> awk '{split($0,ip,"- -"); print (system("ipprefix "ip[1]))}' |
> /
> Why, there's your problem -----+
>
> The awk function system() "execute[s] the command ... and return[s]
> the exit status"
>
> The awk function print() (in this usage) "print[s] expressions"
>
> Taken as
> print(system("Some command"))
> the print() function will print the result of the system() function,
> which will be it's exit status.
>
> (FWIW, the system() function probably uses the system(3) stdlib function
> under the covers, so the exit status will likely be one of those documented
> under system(3).
>
> To fix this, just do not use the awk print function in this manner. Instead,
> (after looking at your ipprefix.c code) you probably want to rephrase the awk
> code from
> print (system("ipprefix "ip[1]))
> to
> system("ipprefix "ip[1]); print;

Thanks, Lew. I agree that's the problem, but unfortunately
what prints using the above fix is somewhat goofy. Seems to be
a subset of lines from the log, apparently not aggregated correctly,
and each printed line is munged from the original.
For example, the very first printed line is

3 54.24954.249.8.242 - - [30/Apr/2021:18:09:31 +0000] "GET /mimetexmanual.html HTTP/1.1" 301 530 "-" "Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)"

But that seems to correspond to the 25th line in the access log,
which reads,

54.249.8.242 - - [30/Apr/2021:02:08:39 +0000] "GET /mimetexmanual.html HTTP/1.1" 301 530 "-" "Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)"

So that leading "3" in the output suggests the log contains 3 such
lines, and they somehow sort first. But God knows what's going on.
Nevertheless, Rich's fix downthread solved the problem without
my little C program at all, so no need to further analyze the
situation. Thanks again.

>> sort | uniq -c | sort -nr
>>
>> So the annoying little problem is that its output is now...
>> 1986 116.1790
>> 672 62.2100
>> 576 114.1190
>> 355 140.820
>> etc
>> I've checked, and the ip addresses are all correct except for
>> that extra "0" at the end of every line. So it's obviously, I think,
>> the exit status from system(), and I can just ignore it. But I'd
>> prefer to suppress system()'s exit status so there is no
>> extraneous "0" to begin with. Is there any way to do that?

--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Re: Suppress exit status of system() command

<s6o69s$g8o$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4913&group=comp.os.linux.misc#4913

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: tnp...@invalid.invalid (The Natural Philosopher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Mon, 3 May 2021 07:47:24 +0100
Organization: A little, after lunch
Lines: 14
Message-ID: <s6o69s$g8o$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me>
<s6nu1n$4ld$1@reader1.panix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Mon, 3 May 2021 06:47:24 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="76c44b13aa85f35c74fd4771f69c8a18";
logging-data="16664"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18ZfhzDjZbUu+zX34U2rkoA3kcGcVyVktw="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Thunderbird/60.6.1
Cancel-Lock: sha1:GRijP9qmKL5oSaaxud6MHaYNR98=
In-Reply-To: <s6nu1n$4ld$1@reader1.panix.com>
Content-Language: en-GB
 by: The Natural Philosop - Mon, 3 May 2021 06:47 UTC

On 03/05/2021 05:26, John Forkosh wrote:
> Actually, "deny from china" seems like the even better fix,
> given all the other info from the logs, but I haven't figured out
> how to do that at one fell swoop.

I did a 'deny from india' filter on my email system. Unfortunately an
indian friend is one of the few people using it.

--
"Anyone who believes that the laws of physics are mere social
conventions is invited to try transgressing those conventions from the
windows of my apartment. (I live on the twenty-first floor.) "

Alan Sokal

Re: Suppress exit status of system() command

<s6otsi$kc5$2@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4915&group=comp.os.linux.misc#4915

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ric...@example.invalid (Rich)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Mon, 3 May 2021 13:29:54 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 6
Message-ID: <s6otsi$kc5$2@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com>
Injection-Date: Mon, 3 May 2021 13:29:54 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="25788d1e077f06f4ff49a3885421e490";
logging-data="20869"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19p5dkRY7Gy+NhcB7h/v7pn"
User-Agent: tin/2.0.1-20111224 ("Achenvoir") (UNIX) (Linux/3.10.17 (x86_64))
Cancel-Lock: sha1:YB8Qk4+yuyTovWNGTqJ6HzZnYdw=
 by: Rich - Mon, 3 May 2021 13:29 UTC

John Forkosh <forkosh@panix.com> wrote:
> And cut ...who knew?

https://en.wikipedia.org/wiki/Cut_(Unix)

Initial release February 1985; 36 years ago

Re: Suppress exit status of system() command

<s6ou45$kc5$3@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4916&group=comp.os.linux.misc#4916

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ric...@example.invalid (Rich)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Mon, 3 May 2021 13:33:57 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 92
Message-ID: <s6ou45$kc5$3@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com>
Injection-Date: Mon, 3 May 2021 13:33:57 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="25788d1e077f06f4ff49a3885421e490";
logging-data="20869"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+Qvd0uMWqI0lKUv1/rCoEe"
User-Agent: tin/2.0.1-20111224 ("Achenvoir") (UNIX) (Linux/3.10.17 (x86_64))
Cancel-Lock: sha1:lwAyfqc7Ebe4HavN2KE3rGdAshc=
 by: Rich - Mon, 3 May 2021 13:33 UTC

John Forkosh <forkosh@panix.com> wrote:
> 1986 116.179 (using your terrific fix)
> "resolves" into
> 19 116.179.37.39
> 17 116.179.37.44
> 16 116.179.37.6
> 16 116.179.37.55
> 16 116.179.37.12
> 15 116.179.37.122
> 14 116.179.37.9
> 14 116.179.37.8
> etc (lots and lots more with fewer and fewer hits from each)
> presumably adding up to 1986 (but I didn't manually check:).
> So, e.g., "deny from 116.179." seems like the appropriate fix.
> Actually, "deny from china" seems like the even better fix,
> given all the other info from the logs, but I haven't figured out
> how to do that at one fell swoop.

Once you have a reasonable pattern for where the accesses are arriving,
then you can block them using iptables. For the example above you'd
block the 116.179.*.* range (which becomes a single firewall rule).

Block from china is often very effective, but knowing which ip's are
"china" is the harder part. Blocking netblocks like 116.179.*.* is
somewhat easer to do.

And, your example above is from china:

$ whois 116.179.37.39
% [whois.apnic.net]
% Whois data copyright terms http://www.apnic.net/db/dbcopyright.html

% Information related to '116.128.0.0 - 116.191.255.255'

% Abuse contact for '116.128.0.0 - 116.191.255.255' is 'ipas@cnnic.cn'

inetnum: 116.128.0.0 - 116.191.255.255
netname: NGTC
descr: China Unicom
descr: No.21 Financial Street,Xicheng District,
descr: Beijing 100140 ,P.R.China
country: CN
admin-c: YW6851-AP
tech-c: YW6851-AP
mnt-by: MAINT-CNNIC-AP
mnt-lower: MAINT-CNNIC-AP
mnt-routes: MAINT-CNCGROUP-RR
mnt-irt: IRT-CNNIC-CN
status: ALLOCATED PORTABLE
last-modified: 2017-09-21T04:13:13Z
source: APNIC

irt: IRT-CNNIC-CN
address: Beijing, China
e-mail: ipas@cnnic.cn
abuse-mailbox: ipas@cnnic.cn
admin-c: IP50-AP
tech-c: IP50-AP
auth: # Filtered
remarks: Please note that CNNIC is not an ISP and is not
remarks: empowered to investigate complaints of network abuse.
remarks: Please contact the tech-c or admin-c of the network.
mnt-by: MAINT-CNNIC-AP
last-modified: 2017-11-01T08:57:39Z
source: APNIC

person: Yuzhen Zhao
address: No.21 Financial Street,Xicheng District,
address: Beijing 100140 ,P.R.China
country: CN
phone: +86-10-66258500
fax-no: +86-10-66259626
e-mail: zhaoyz3@chinaunicom.cn
e-mail: hqs-ipabuse@chinaunicom.cn
nic-hdl: YW6851-AP
mnt-by: MAINT-CNNIC-AP
last-modified: 2017-09-21T03:48:02Z
source: APNIC

% Information related to '116.128.0.0/10AS4837'

route: 116.128.0.0/10
descr: China Unicom CHINA169 Network
country: CN
origin: AS4837
mnt-by: MAINT-CNCGROUP-RR
last-modified: 2017-10-20T06:12:02Z
source: APNIC

% This query was served by the APNIC Whois Service version 1.88.15-SNAPSHOT (WHOIS-US4)

Re: Suppress exit status of system() command

<s6pdgn$918$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4917&group=comp.os.linux.misc#4917

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: lew.pitc...@digitalfreehold.ca (Lew Pitcher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Mon, 3 May 2021 17:56:39 -0000 (UTC)
Organization: The Pitcher Digital Freehold
Lines: 90
Message-ID: <s6pdgn$918$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mca3$k0n$1@dont-email.me>
<s6nv3v$3vh$1@reader1.panix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Injection-Date: Mon, 3 May 2021 17:56:39 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="7af2cb5d1006b69a85863175ad347448";
logging-data="9256"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19wdBnvt7B0BIN7iW6pgg8k+Q+BEvDrQ9s="
User-Agent: Pan/0.139 (Sexual Chocolate; GIT bf56508
git://git.gnome.org/pan2)
Cancel-Lock: sha1:x8od4M45KWGbK9w9+KOVnmVzdgQ=
 by: Lew Pitcher - Mon, 3 May 2021 17:56 UTC

On Mon, 03 May 2021 04:44:47 +0000, John Forkosh wrote:

> Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
>> John Forkosh wrote:
>>> I have a little one-line awk/shell script like this...
>>> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
>>> which analyzes my website logs, producing output like this...
>>> 672 62.210.98.10
>>> 178 54.173.189.222
>>> 116 23.100.232.233
>>> 116 101.19.4.45
>>> 88 151.38.64.253
>>> etc
>>> which is merely a count (on the left) of the number of times
>>> each ip-address (on the right) has accessed my site, highest
>>> count first.
>>>
>>> Problem with that is there are many accesses with the same
>>> first two aaa.bbb.. and different ..ccc.ddd. So they're
>>> treated separately, even though they most likely should
>>> be aggregated. So I wrote a little C program, ipprefix.c
>>> #include <stdio.h>
>>> #include <stdlib.h>
>>> #include <string.h>
>>> int main ( int argc, char *argv[] ) {
>>> char ipstr[999]="\000", *delim=NULL;
>>> int ndots=2, idot=0;
>>> if ( argc > 1 ) strcpy(ipstr,argv[1]);
>>> if ( (delim=strchr(ipstr,',')) != NULL ) {
>>> *delim = '\000'; ndots=atoi(delim+1); }
>>> for ( delim=ipstr-1,idot=1; idot<=ndots; idot++ ) {
>>> delim = strchr(delim+1,'.');
>>> if ( delim == NULL ) break; }
>>> if ( delim != NULL ) *delim = '\000';
>>> printf("%s",ipstr);
>>> } /* --- end-of-job --- */
>>> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb
>>>
>>> And then I modified the awk/shell script like this...
>>> awk '{split($0,ip,"- -"); print (system("ipprefix "ip[1]))}' |
>> /
>> Why, there's your problem -----+
>>
>> The awk function system() "execute[s] the command ... and return[s]
>> the exit status"
>>
>> The awk function print() (in this usage) "print[s] expressions"
>>
>> Taken as
>> print(system("Some command"))
>> the print() function will print the result of the system() function,
>> which will be it's exit status.
>>
>> (FWIW, the system() function probably uses the system(3) stdlib function
>> under the covers, so the exit status will likely be one of those documented
>> under system(3).
>>
>> To fix this, just do not use the awk print function in this manner. Instead,
>> (after looking at your ipprefix.c code) you probably want to rephrase the awk
>> code from
>> print (system("ipprefix "ip[1]))
>> to
>> system("ipprefix "ip[1]); print;
>
> Thanks, Lew. I agree that's the problem, but unfortunately
> what prints using the above fix is somewhat goofy. Seems to be
> a subset of lines from the log, apparently not aggregated correctly,
> and each printed line is munged from the original.
> For example, the very first printed line is
>
> 3 54.24954.249.8.242 - - [30/Apr/2021:18:09:31 +0000] "GET /mimetexmanual.html HTTP/1.1" 301 530 "-" "Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)"
[snip]

Mea culpa. My code was faulty.

Where I said
system("ipprefix "ip[1]); print;
I /should have/ said
system("ipprefix "ip[1]); print "";

The naked print will print the entire input line, while
the print "" will only print a newline.

You could eliminate the awk print call entirely if you recoded
your ipprefix.c to output newlines for itself.

Sorry for the confusion.
--
Lew Pitcher
"In Skills, We Trust"

Re: Suppress exit status of system() command

<s6q7d0$ad1$1@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4924&group=comp.os.linux.misc#4924

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!news.uzoreto.com!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 01:18:24 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 95
Message-ID: <s6q7d0$ad1$1@reader1.panix.com>
References: <s6letp$hu1$1@reader1.panix.com> <s6mca3$k0n$1@dont-email.me> <s6nv3v$3vh$1@reader1.panix.com> <s6pdgn$918$1@dont-email.me>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1620091104 10657 166.84.1.3 (4 May 2021 01:18:24 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Tue, 4 May 2021 01:18:24 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Tue, 4 May 2021 01:18 UTC

Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
> John Forkosh wrote:
>> Lew Pitcher <lew.pitcher@digitalfreehold.ca> wrote:
>>> John Forkosh wrote:
>>>> I have a little one-line awk/shell script like this...
>>>> awk '{split($0,ip,"- -"); print ip[1]}' | sort | uniq -c | sort -nr
>>>> which analyzes my website logs, producing output like this...
>>>> 672 62.210.98.10
>>>> 178 54.173.189.222
>>>> 116 23.100.232.233
>>>> 116 101.19.4.45
>>>> 88 151.38.64.253
>>>> etc
>>>> which is merely a count (on the left) of the number of times
>>>> each ip-address (on the right) has accessed my site, highest
>>>> count first.
>>>>
>>>> Problem with that is there are many accesses with the same
>>>> first two aaa.bbb.. and different ..ccc.ddd. So they're
>>>> treated separately, even though they most likely should
>>>> be aggregated. So I wrote a little C program, ipprefix.c
>>>> #include <stdio.h>
>>>> #include <stdlib.h>
>>>> #include <string.h>
>>>> int main ( int argc, char *argv[] ) {
>>>> char ipstr[999]="\000", *delim=NULL;
>>>> int ndots=2, idot=0;
>>>> if ( argc > 1 ) strcpy(ipstr,argv[1]);
>>>> if ( (delim=strchr(ipstr,',')) != NULL ) {
>>>> *delim = '\000'; ndots=atoi(delim+1); }
>>>> for ( delim=ipstr-1,idot=1; idot<=ndots; idot++ ) {
>>>> delim = strchr(delim+1,'.');
>>>> if ( delim == NULL ) break; }
>>>> if ( delim != NULL ) *delim = '\000';
>>>> printf("%s",ipstr);
>>>> } /* --- end-of-job --- */
>>>> whose argv[1] input is aaa.bbb.ccc.ddd and whose output is aaa.bbb
>>>>
>>>> And then I modified the awk/shell script like this...
>>>> awk '{split($0,ip,"- -"); print (system("ipprefix "ip[1]))}' |
>>> /
>>> Why, there's your problem -----+
>>>
>>> The awk function system() "execute[s] the command ... and return[s]
>>> the exit status"
>>>
>>> The awk function print() (in this usage) "print[s] expressions"
>>>
>>> Taken as
>>> print(system("Some command"))
>>> the print() function will print the result of the system() function,
>>> which will be it's exit status.
>>>
>>> (FWIW, the system() function probably uses the system(3) stdlib function
>>> under the covers, so the exit status will likely be one of those documented
>>> under system(3).
>>>
>>> To fix this, just do not use the awk print function in this manner. Instead,
>>> (after looking at your ipprefix.c code) you probably want to rephrase the awk
>>> code from
>>> print (system("ipprefix "ip[1]))
>>> to
>>> system("ipprefix "ip[1]); print;
>>
>> Thanks, Lew. I agree that's the problem, but unfortunately
>> what prints using the above fix is somewhat goofy. Seems to be
>> a subset of lines from the log, apparently not aggregated correctly,
>> and each printed line is munged from the original.
>> For example, the very first printed line is
>>
>> 3 54.24954.249.8.242 - - [30/Apr/2021:18:09:31 +0000] "GET /mimetexmanual.html HTTP/1.1" 301 530 "-" "Hatena Antenna/0.5 (http://a.hatena.ne.jp/help)"
> [snip]
>
> Mea culpa. My code was faulty.
>
> Where I said
> system("ipprefix "ip[1]); print;
> I /should have/ said
> system("ipprefix "ip[1]); print "";
>
> The naked print will print the entire input line, while
> the print "" will only print a newline.
>
> You could eliminate the awk print call entirely if you recoded
> your ipprefix.c to output newlines for itself.
>
> Sorry for the confusion.

Thanks for the fix, which now works perfectly.
Guess I should've seen it myself, since I was looking at the output,
and the problem's obvious even to me ... now that you've mentioned it.
(And yeah, I'd have written ipprefix with that trailing "%s\n"
if I'd realized your better way of using it with awk.)
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Re: Suppress exit status of system() command

<s6q7mj$ad1$2@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4925&group=comp.os.linux.misc#4925

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!paganini.bofh.team!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 01:23:31 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 16
Message-ID: <s6q7mj$ad1$2@reader1.panix.com>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com> <s6otsi$kc5$2@dont-email.me>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1620091411 10657 166.84.1.3 (4 May 2021 01:23:31 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Tue, 4 May 2021 01:23:31 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Tue, 4 May 2021 01:23 UTC

Rich <rich@example.invalid> wrote:
> John Forkosh <forkosh@panix.com> wrote:
>> And cut ...who knew?
>
> https://en.wikipedia.org/wiki/Cut_(Unix)
>
> Initial release February 1985; 36 years ago

Oh, sure, man cut works pretty much just as well.
That remark was just kidding around, as per the <snip>'ed
sentence after it... "So many utilities, so little time."
In case you weren't familiar, that's a paraphrase of the
kidding-around saying, "So many women, so little time."
(I only wish that were my problem:)
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Re: Suppress exit status of system() command

<s6q8ke$ovr$1@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4926&group=comp.os.linux.misc#4926

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!paganini.bofh.team!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 01:39:26 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 51
Message-ID: <s6q8ke$ovr$1@reader1.panix.com>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com> <s6ou45$kc5$3@dont-email.me>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1620092366 25595 166.84.1.3 (4 May 2021 01:39:26 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Tue, 4 May 2021 01:39:26 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Tue, 4 May 2021 01:39 UTC

Rich <rich@example.invalid> wrote:
> John Forkosh <forkosh@panix.com> wrote:
>> 1986 116.179 (using your terrific fix)
>> "resolves" into
>> 19 116.179.37.39
>> 17 116.179.37.44
>> 16 116.179.37.6
>> 16 116.179.37.55
>> 16 116.179.37.12
>> 15 116.179.37.122
>> 14 116.179.37.9
>> 14 116.179.37.8
>> etc (lots and lots more with fewer and fewer hits from each)
>> presumably adding up to 1986 (but I didn't manually check:).
>> So, e.g., "deny from 116.179." seems like the appropriate fix.
>> Actually, "deny from china" seems like the even better fix,
>> given all the other info from the logs, but I haven't figured out
>> how to do that at one fell swoop.
>
> Once you have a reasonable pattern for where the accesses are arriving,
> then you can block them using iptables. For the example above you'd
> block the 116.179.*.* range (which becomes a single firewall rule).

Thanks, I'd just been using .htaccess so far, but now that you
mention it, an iptables script would likely be the better (and
probably necessary) approach.

> Block from china is often very effective, but knowing which ip's are
> "china" is the harder part.

Yeah, there are a bunch of ip's besides 116.179., all from China,
that are becoming a big pain in the elbow, and I'd just block
all "china" if I could figure that out.

> Blocking netblocks like 116.179.*.* is somewhat easer to do.

Lots easier, and already done.
Actually, rather than blocking, I've now tried redirecting
them to an online video site, hoping the tons of bytes
they'll download with each hit might eventually deter them.
But hasn't worked so far.

> And, your example above is from china:
> $ whois 116.179.37.39
<<snip>>

Oh, sure, I'd already whois'ed it myself, as well as
all other ip addresses whose activity seems suspicious.

--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Re: Suppress exit status of system() command

<s6q8v4$ovr$2@reader1.panix.com>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4927&group=comp.os.linux.misc#4927

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!paganini.bofh.team!goblin3!goblin.stu.neva.ru!panix!not-for-mail
From: fork...@panix.com (John Forkosh)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 01:45:08 +0000 (UTC)
Organization: PANIX Public Access Internet and UNIX, NYC
Lines: 15
Message-ID: <s6q8v4$ovr$2@reader1.panix.com>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com> <s6o69s$g8o$1@dont-email.me>
NNTP-Posting-Host: panix3.panix.com
X-Trace: reader1.panix.com 1620092708 25595 166.84.1.3 (4 May 2021 01:45:08 GMT)
X-Complaints-To: abuse@panix.com
NNTP-Posting-Date: Tue, 4 May 2021 01:45:08 +0000 (UTC)
User-Agent: tin/2.4.5-20201224 ("Glen Albyn") (NetBSD/9.0 (amd64))
 by: John Forkosh - Tue, 4 May 2021 01:45 UTC

The Natural Philosopher <tnp@invalid.invalid> wrote:
> John Forkosh wrote:
>> Actually, "deny from china" seems like the even better fix,
>> given all the other info from the logs, but I haven't figured out
>> how to do that at one fell swoop.
>
> I did a 'deny from india' filter on my email system. Unfortunately an
> indian friend is one of the few people using it.

If you're using procmail, you can "whitelist" your friend's email
before you "blacklist" india generally. Anyway, I'd have no problem
blocking china entirely; just can't figure out how to do it
all at once.
--
John Forkosh ( mailto: j@f.com where j=john and f=forkosh )

Re: Suppress exit status of system() command

<s6qa8o$l7n$2@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4928&group=comp.os.linux.misc#4928

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ric...@example.invalid (Rich)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 02:07:20 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 22
Message-ID: <s6qa8o$l7n$2@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com> <s6o69s$g8o$1@dont-email.me> <s6q8v4$ovr$2@reader1.panix.com>
Injection-Date: Tue, 4 May 2021 02:07:20 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="4f89ea7ef03d24f55852b150a8d7af41";
logging-data="21751"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18etZU/F83VoK2XnAgYrlGb"
User-Agent: tin/2.0.1-20111224 ("Achenvoir") (UNIX) (Linux/3.10.17 (x86_64))
Cancel-Lock: sha1:zfpJcZg8V70wsiVz+Mzv9D29puE=
 by: Rich - Tue, 4 May 2021 02:07 UTC

John Forkosh <forkosh@panix.com> wrote:
> The Natural Philosopher <tnp@invalid.invalid> wrote:
>> John Forkosh wrote:
>>> Actually, "deny from china" seems like the even better fix, given
>>> all the other info from the logs, but I haven't figured out how to
>>> do that at one fell swoop.
>>
>> I did a 'deny from india' filter on my email system. Unfortunately
>> an indian friend is one of the few people using it.
>
> If you're using procmail, you can "whitelist" your friend's email
> before you "blacklist" india generally. Anyway, I'd have no problem
> blocking china entirely; just can't figure out how to do it all at
> once.

There is no "block china" filter. You just have to figure out what
ranges of IP addresses are from china, and block each one you encounter
as you encounter them.

Eventually you'll have blocked the ones causing you trouble, and it
won't matter hat other non-blocked chinese IP addresses are not
blocked.

Re: Suppress exit status of system() command

<s6qaq3$l7n$3@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4929&group=comp.os.linux.misc#4929

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ric...@example.invalid (Rich)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 02:16:36 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 71
Message-ID: <s6qaq3$l7n$3@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com> <s6ou45$kc5$3@dont-email.me> <s6q8ke$ovr$1@reader1.panix.com>
Injection-Date: Tue, 4 May 2021 02:16:36 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="4f89ea7ef03d24f55852b150a8d7af41";
logging-data="21751"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18mLNheoV1+yQ6fbR8sih3X"
User-Agent: tin/2.0.1-20111224 ("Achenvoir") (UNIX) (Linux/3.10.17 (x86_64))
Cancel-Lock: sha1:UjGOeKfV6SHR0eH21zKSfTjt4AQ=
 by: Rich - Tue, 4 May 2021 02:16 UTC

John Forkosh <forkosh@panix.com> wrote:
> Rich <rich@example.invalid> wrote:
>> John Forkosh <forkosh@panix.com> wrote:
>>> 1986 116.179 (using your terrific fix)
>>> "resolves" into
>>> 19 116.179.37.39
>>> 17 116.179.37.44
>>> 16 116.179.37.6
>>> 16 116.179.37.55
>>> 16 116.179.37.12
>>> 15 116.179.37.122
>>> 14 116.179.37.9
>>> 14 116.179.37.8
>>> etc (lots and lots more with fewer and fewer hits from each)
>>> presumably adding up to 1986 (but I didn't manually check:).
>>> So, e.g., "deny from 116.179." seems like the appropriate fix.
>>> Actually, "deny from china" seems like the even better fix,
>>> given all the other info from the logs, but I haven't figured out
>>> how to do that at one fell swoop.
>>
>> Once you have a reasonable pattern for where the accesses are arriving,
>> then you can block them using iptables. For the example above you'd
>> block the 116.179.*.* range (which becomes a single firewall rule).
>
> Thanks, I'd just been using .htaccess so far, but now that you
> mention it, an iptables script would likely be the better (and
> probably necessary) approach.

Works well. I had some china IP's bothering a bulletin board that an
org's website I run used. They kept posting ads for rock crushers and
such. Eventually I tired of it, and just started adding their
netblocks to an iptables chain that simply "DROP"ed them. Problem went
away.

In any case, no value in letting them get as far as your webserver to
then be dropped by an .htaccess file. Just drop them the moment they
enter the network stack and be done with them.

>> Block from china is often very effective, but knowing which ip's are
>> "china" is the harder part.
>
> Yeah, there are a bunch of ip's besides 116.179., all from China,
> that are becoming a big pain in the elbow, and I'd just block all
> "china" if I could figure that out.

The only real way is what you are doing. Parsing the logs, extracting
the patterns, then deciding which to "DROP" (that's an iptables target,
by the way).

>> Blocking netblocks like 116.179.*.* is somewhat easer to do.
>
> Lots easier, and already done.
> Actually, rather than blocking, I've now tried redirecting them to an
> online video site, hoping the tons of bytes they'll download with
> each hit might eventually deter them. But hasn't worked so far.

Not likely to work well. But if you do want to "bother" them, do some
googling for "network tar pits". You could setup a "very slow tar pit"
that keeps their connections busy for an exceedingly long time. But,
do note that you'll need to increase the connection tables sizes to
keep a lot of them around.

>> And, your example above is from china:
>> $ whois 116.179.37.39
> <<snip>>
>
> Oh, sure, I'd already whois'ed it myself, as well as all other ip
> addresses whose activity seems suspicious.

That is the best way to find an approximate "where" -- then you can
decide if it is enough trough to add to the iptables DROP chain.

Re: Suppress exit status of system() command

<s6qarc$l7n$4@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4930&group=comp.os.linux.misc#4930

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ric...@example.invalid (Rich)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 02:17:17 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 18
Message-ID: <s6qarc$l7n$4@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com> <s6otsi$kc5$2@dont-email.me> <s6q7mj$ad1$2@reader1.panix.com>
Injection-Date: Tue, 4 May 2021 02:17:17 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="4f89ea7ef03d24f55852b150a8d7af41";
logging-data="21751"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+j/hfMV33akNQmpMCtzFov"
User-Agent: tin/2.0.1-20111224 ("Achenvoir") (UNIX) (Linux/3.10.17 (x86_64))
Cancel-Lock: sha1:E12+0vYFrdjMy9zt94hZySJBpYI=
 by: Rich - Tue, 4 May 2021 02:17 UTC

John Forkosh <forkosh@panix.com> wrote:
> Rich <rich@example.invalid> wrote:
>> John Forkosh <forkosh@panix.com> wrote:
>>> And cut ...who knew?
>>
>> https://en.wikipedia.org/wiki/Cut_(Unix)
>>
>> Initial release February 1985; 36 years ago
>
> Oh, sure, man cut works pretty much just as well.
> That remark was just kidding around, as per the <snip>'ed
> sentence after it... "So many utilities, so little time."
> In case you weren't familiar, that's a paraphrase of the
> kidding-around saying, "So many women, so little time."
> (I only wish that were my problem:)

Yes, I know, I was trying to do a subtle kid back. Maybe too subtle...

Re: Suppress exit status of system() command

<87k0ofw68q.fsf@usenet.ankman.de>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4931&group=comp.os.linux.misc#4931

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ank...@spamfence.net (Andreas Kohlbach)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Mon, 03 May 2021 23:03:01 -0400
Organization: https://news-commentaries.blogspot.com/
Lines: 11
Message-ID: <87k0ofw68q.fsf@usenet.ankman.de>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me>
<s6nu1n$4ld$1@reader1.panix.com> <s6ou45$kc5$3@dont-email.me>
<s6q8ke$ovr$1@reader1.panix.com>
Mime-Version: 1.0
Content-Type: text/plain
Injection-Info: reader02.eternal-september.org; posting-host="fff6e8d7c2320c4c8d070b0529fe992a";
logging-data="4566"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/VsCgbtf9e8gFAeYgh+1Zj"
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.1 (gnu/linux)
Cancel-Lock: sha1:Tp7039Tqa+Kwvi49KhsutPiHmuM=
sha1:Z1k7L3LvetGJou05GVfhVl8b4cQ=
X-No-Archive: Yes
X-Face: '#R~-oJz-_!iXhczPJ;=w1(`5-uQ2$0qHB7KKDV,]VoAC!P?swaa#m|eB<DkOt*XH=~9C[g S^w)b,)1q,{P\7Z3H,N(^m.YKuYM//B{X:PvbDk.|:g:$wVr*3*)[K6F+k\z-s32+oB]YJPy11wuGGz'bQAk~1.b1[;M{^A2@bboIENBB:Wd:<Fm~r7OuiJA1g}7KC-T'>Du+
X-Face-What-Is-It: Capture Bee from Galaga
 by: Andreas Kohlbach - Tue, 4 May 2021 03:03 UTC

On Tue, 4 May 2021 01:39:26 +0000 (UTC), John Forkosh wrote:
>
> Yeah, there are a bunch of ip's besides 116.179., all from China,
> that are becoming a big pain in the elbow, and I'd just block
> all "china" if I could figure that out.

A decade ago or so it was mentioned 220. and 221. should go to /dev/null.

No idea, if this is still true today.
--
Andreas

Re: Suppress exit status of system() command

<s6re7q$knj$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4933&group=comp.os.linux.misc#4933

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: tnp...@invalid.invalid (The Natural Philosopher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 13:21:13 +0100
Organization: A little, after lunch
Lines: 19
Message-ID: <s6re7q$knj$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me>
<s6nu1n$4ld$1@reader1.panix.com> <s6ou45$kc5$3@dont-email.me>
<s6q8ke$ovr$1@reader1.panix.com> <s6qaq3$l7n$3@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 4 May 2021 12:21:14 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="b0c04a065bdfe1a7e3b8e808b8b29509";
logging-data="21235"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1/QZveemcD5KsI9XI09pMB54ZRugKQTsJo="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Thunderbird/60.6.1
Cancel-Lock: sha1:qv8FEiShhKjZrCQnTdeKYwj4rn4=
In-Reply-To: <s6qaq3$l7n$3@dont-email.me>
Content-Language: en-GB
 by: The Natural Philosop - Tue, 4 May 2021 12:21 UTC

On 04/05/2021 03:16, Rich wrote:
> Just drop them the moment they
> enter the network stack and be done with them.

yes. net attacks should be dropped as early and as often as possible.
I have a website behind a firewall. It is https only - port 80 simply
doesnt work, except that once every few months I need to open it up for
a minute or two for letsencrypt's certbot to verify the site exists.

In that minute or two I got four connections made to it looking for
backdoors.

On several of my websites there are ip address blacklists as well that
simply return html errors if people try and connect from those
addresses. Iptables is a possible, but I haven't run out of CPU cycles
yet...

--
Climate Change: Socialism wearing a lab coat.

Re: Suppress exit status of system() command

<s6rei0$n8u$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4934&group=comp.os.linux.misc#4934

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!news.niel.me!news.gegeweb.eu!gegeweb.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: tnp...@invalid.invalid (The Natural Philosopher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 13:26:39 +0100
Organization: A little, after lunch
Lines: 30
Message-ID: <s6rei0$n8u$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me>
<s6nu1n$4ld$1@reader1.panix.com> <s6o69s$g8o$1@dont-email.me>
<s6q8v4$ovr$2@reader1.panix.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 4 May 2021 12:26:40 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="b0c04a065bdfe1a7e3b8e808b8b29509";
logging-data="23838"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX18ooDcw1DZmw66D4cZ2tKx2u4IzUXJpiUE="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Thunderbird/60.6.1
Cancel-Lock: sha1:8p4MG8EN5m9LYvTnmfTgcp/I8as=
In-Reply-To: <s6q8v4$ovr$2@reader1.panix.com>
Content-Language: en-GB
 by: The Natural Philosop - Tue, 4 May 2021 12:26 UTC

On 04/05/2021 02:45, John Forkosh wrote:
> The Natural Philosopher <tnp@invalid.invalid> wrote:
>> John Forkosh wrote:
>>> Actually, "deny from china" seems like the even better fix,
>>> given all the other info from the logs, but I haven't figured out
>>> how to do that at one fell swoop.
>>
>> I did a 'deny from india' filter on my email system. Unfortunately an
>> indian friend is one of the few people using it.
>
> If you're using procmail,

I am not, sadly...

> you can "whitelist" your friend's email
> before you "blacklist" india generally. Anyway, I'd have no problem
> blocking china entirely; just can't figure out how to do it
> all at once.
>
Indeed. Every couple of months I go through my spam mailbox and simply
blacklist every sender - lots of whole top level domains like .tv.
..biz .marketing and so on

I have over 1250 domains in there to date.

--
Renewable energy: Expensive solutions that don't work to a problem that
doesn't exist instituted by self legalising protection rackets that
don't protect, masquerading as public servants who don't serve the public.

Re: Suppress exit status of system() command

<s6rent$p0b$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4935&group=comp.os.linux.misc#4935

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: tnp...@invalid.invalid (The Natural Philosopher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 13:29:48 +0100
Organization: A little, after lunch
Lines: 34
Message-ID: <s6rent$p0b$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me>
<s6nu1n$4ld$1@reader1.panix.com> <s6o69s$g8o$1@dont-email.me>
<s6q8v4$ovr$2@reader1.panix.com> <s6qa8o$l7n$2@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 4 May 2021 12:29:49 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="b0c04a065bdfe1a7e3b8e808b8b29509";
logging-data="25611"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19OVksIXvqNkBNe4C8umut7OKae6/jALdI="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Thunderbird/60.6.1
Cancel-Lock: sha1:bS/wArs6PTeOhq2WrwbOMGK0JW0=
In-Reply-To: <s6qa8o$l7n$2@dont-email.me>
Content-Language: en-GB
 by: The Natural Philosop - Tue, 4 May 2021 12:29 UTC

On 04/05/2021 03:07, Rich wrote:
> John Forkosh <forkosh@panix.com> wrote:
>> The Natural Philosopher <tnp@invalid.invalid> wrote:
>>> John Forkosh wrote:
>>>> Actually, "deny from china" seems like the even better fix, given
>>>> all the other info from the logs, but I haven't figured out how to
>>>> do that at one fell swoop.
>>>
>>> I did a 'deny from india' filter on my email system. Unfortunately
>>> an indian friend is one of the few people using it.
>>
>> If you're using procmail, you can "whitelist" your friend's email
>> before you "blacklist" india generally. Anyway, I'd have no problem
>> blocking china entirely; just can't figure out how to do it all at
>> once.
>
> There is no "block china" filter. You just have to figure out what
> ranges of IP addresses are from china, and block each one you encounter
> as you encounter them.
>
here having access to global BGP routing tables would be an advantage...

> Eventually you'll have blocked the ones causing you trouble, and it
> won't matter hat other non-blocked chinese IP addresses are not
> blocked.
>

+1 to that.

Just keep blocking...

--
Gun Control: The law that ensures that only criminals have guns.

Re: Suppress exit status of system() command

<s6rfom$ua5$2@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4936&group=comp.os.linux.misc#4936

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: ric...@example.invalid (Rich)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 12:47:18 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 35
Message-ID: <s6rfom$ua5$2@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me> <s6nu1n$4ld$1@reader1.panix.com> <s6o69s$g8o$1@dont-email.me> <s6q8v4$ovr$2@reader1.panix.com> <s6qa8o$l7n$2@dont-email.me> <s6rent$p0b$1@dont-email.me>
Injection-Date: Tue, 4 May 2021 12:47:18 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="4f89ea7ef03d24f55852b150a8d7af41";
logging-data="31045"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX188Qxh0jH6t/bgXG9h2XSI2"
User-Agent: tin/2.0.1-20111224 ("Achenvoir") (UNIX) (Linux/3.10.17 (x86_64))
Cancel-Lock: sha1:k+J6t4xIwaJGQijzGUNvUlXQfIo=
 by: Rich - Tue, 4 May 2021 12:47 UTC

The Natural Philosopher <tnp@invalid.invalid> wrote:
> On 04/05/2021 03:07, Rich wrote:
>> John Forkosh <forkosh@panix.com> wrote:
>>> The Natural Philosopher <tnp@invalid.invalid> wrote:
>>>> John Forkosh wrote:
>>>>> Actually, "deny from china" seems like the even better fix, given
>>>>> all the other info from the logs, but I haven't figured out how
>>>>> to do that at one fell swoop.
>>>>
>>>> I did a 'deny from india' filter on my email system.
>>>> Unfortunately an indian friend is one of the few people using it.
>>>
>>> If you're using procmail, you can "whitelist" your friend's email
>>> before you "blacklist" india generally. Anyway, I'd have no
>>> problem blocking china entirely; just can't figure out how to do it
>>> all at once.
>>
>> There is no "block china" filter. You just have to figure out what
>> ranges of IP addresses are from china, and block each one you encounter
>> as you encounter them.
>>
> here having access to global BGP routing tables would be an
> advantage...

Few of us here have such access. But yes, having that data would let
one pick off "china" from the rest.

>> Eventually you'll have blocked the ones causing you trouble, and it
>> won't matter hat other non-blocked chinese IP addresses are not
>> blocked.
>
> +1 to that.
>
> Just keep blocking...

Re: Suppress exit status of system() command

<s6rnnh$t1g$1@dont-email.me>

 copy mid

https://www.novabbs.com/computers/article-flat.php?id=4937&group=comp.os.linux.misc#4937

 copy link   Newsgroups: comp.os.linux.misc
Path: i2pn2.org!i2pn.org!eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: tnp...@invalid.invalid (The Natural Philosopher)
Newsgroups: comp.os.linux.misc
Subject: Re: Suppress exit status of system() command
Date: Tue, 4 May 2021 16:03:12 +0100
Organization: A little, after lunch
Lines: 46
Message-ID: <s6rnnh$t1g$1@dont-email.me>
References: <s6letp$hu1$1@reader1.panix.com> <s6mfjd$an5$1@dont-email.me>
<s6nu1n$4ld$1@reader1.panix.com> <s6o69s$g8o$1@dont-email.me>
<s6q8v4$ovr$2@reader1.panix.com> <s6qa8o$l7n$2@dont-email.me>
<s6rent$p0b$1@dont-email.me> <s6rfom$ua5$2@dont-email.me>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
Injection-Date: Tue, 4 May 2021 15:03:13 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="b0c04a065bdfe1a7e3b8e808b8b29509";
logging-data="29744"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX19UkThwUh4HY5DHuyc91EkXsWU4LDkWVRM="
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101
Thunderbird/60.6.1
Cancel-Lock: sha1:ZmyhKEt8CZalqS4Ec2pvRDSGxEo=
In-Reply-To: <s6rfom$ua5$2@dont-email.me>
Content-Language: en-GB
 by: The Natural Philosop - Tue, 4 May 2021 15:03 UTC

On 04/05/2021 13:47, Rich wrote:
> The Natural Philosopher <tnp@invalid.invalid> wrote:
>> On 04/05/2021 03:07, Rich wrote:
>>> John Forkosh <forkosh@panix.com> wrote:
>>>> The Natural Philosopher <tnp@invalid.invalid> wrote:
>>>>> John Forkosh wrote:
>>>>>> Actually, "deny from china" seems like the even better fix, given
>>>>>> all the other info from the logs, but I haven't figured out how
>>>>>> to do that at one fell swoop.
>>>>>
>>>>> I did a 'deny from india' filter on my email system.
>>>>> Unfortunately an indian friend is one of the few people using it.
>>>>
>>>> If you're using procmail, you can "whitelist" your friend's email
>>>> before you "blacklist" india generally. Anyway, I'd have no
>>>> problem blocking china entirely; just can't figure out how to do it
>>>> all at once.
>>>
>>> There is no "block china" filter. You just have to figure out what
>>> ranges of IP addresses are from china, and block each one you encounter
>>> as you encounter them.
>>>
>> here having access to global BGP routing tables would be an
>> advantage...
>
> Few of us here have such access. But yes, having that data would let
> one pick off "china" from the rest.

actually i think the tables are on line somewhere

>
>>> Eventually you'll have blocked the ones causing you trouble, and it
>>> won't matter hat other non-blocked chinese IP addresses are not
>>> blocked.
>>
>> +1 to that.
>>
>> Just keep blocking...
>

--
How fortunate for governments that the people they administer don't think.

Adolf Hitler

1
server_pubkey.txt

rocksolid light 0.9.7
clearnet tor