Rocksolid Light

Welcome to novaBBS (click a section below)

mail  files  register  nodelist  faq  login

If loving linux is wrong, I dont wanna be right. -- Topic for #LinuxGER


computers / comp.security.ssh / Looking for help / advice against aggressive BotNet triggering sshd errors which fill auth.log rapidly....

SubjectAuthor
* Looking for help / advice against aggressive BotNet triggering sshdJT High
`* Re: Looking for help / advice against aggressive BotNet triggeringWilliam Unruh
 `- Re: Looking for help / advice against aggressive BotNet triggeringJT High

1
Subject: Looking for help / advice against aggressive BotNet triggering sshd errors which fill auth.log rapidly....
From: JT High
Newsgroups: comp.security.ssh
Date: Fri, 18 Sep 2020 14:53 UTC
X-Received: by 2002:ac8:1c82:: with SMTP id f2mr34478286qtl.305.1600440791245;
Fri, 18 Sep 2020 07:53:11 -0700 (PDT)
X-Received: by 2002:a05:6830:124d:: with SMTP id s13mr22514890otp.12.1600440790112;
Fri, 18 Sep 2020 07:53:10 -0700 (PDT)
Path: i2pn2.org!i2pn.org!aioe.org!peer01.ams4!peer.am4.highwinds-media.com!peer03.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.security.ssh
Date: Fri, 18 Sep 2020 07:53:09 -0700 (PDT)
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2600:1700:ba0:d2b8:88a3:546c:755:b749;
posting-account=6nq-igoAAABCZUe4jhBCYapcvMbDdZ5b
NNTP-Posting-Host: 2600:1700:ba0:d2b8:88a3:546c:755:b749
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <7335f450-7669-49c2-816d-745aa04674bfn@googlegroups.com>
Subject: Looking for help / advice against aggressive BotNet triggering sshd
errors which fill auth.log rapidly....
From: jth...@gmail.com (JT High)
Injection-Date: Fri, 18 Sep 2020 14:53:11 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 11723
X-Received-Body-CRC: 2173259917
View all headers
I'm looking for any help / advice against aggressive BotNet triggering sshd errors which fill auth.log rapidly.... What I am seeing could be a legitimate sshd bug -- or it could be a misconfiguration on my part that I've never encountered before...

I have a small linux host which is now and has been kept up to date, and has to the best of my ability been hardened for SSH and had fail2ban installed on it.  Generally this host deals with the typical SSH Internet brute force attempts fine, and fail2ban bans recurring source IP attempts as planned..

This past week something "New and unexpected" happened... a botnet came after this host, and although I don't see any sign they ever successfully authenticated (which is also protected by google-authenticator-libpam), they nevertheless managed to do something that FILLed auth.log very rapidly, and effectively crashed the host until I realized it had a problem.  None of what they were doing triggered jail from Fail2Ban which I get email notifications for...

The auth.log on this host (a sample is shown below) grew to 11GB rapidly and shows more than 250 source IPs from all around the world converging on this host with two or three at a time making attempts -- not just the typical script kiddie / dictionary attacks either - whoever this is intentionally never stayed on the same IPs long enough the get banned and the recycling of source IPs was carefully done...

In /var/log/auth.log I see this:

Please note the topmost line, and the lines at the bottom, that look like this:
Sep 15 17:35:26 rp6 sshd[21265]: error: moduli:1: type is not 2

There are millions of these lines with different integers, and an astonishing number of them were written quickly into auth.log, but none of this activity seemed to generate a ban out of fail2ban

Can someone more skilled than me - please eyeball this log snippet and tell me if I should bother anyone with a bug report, or if I am the one with the 'config' problem?

This host does not have any file system corruption and regenerating the primes doesn't seem to fix this issue, as this botnet can illicit the same effect over and over again... manually banning the offending source IPs just slows things down for a short while and they are right back at me from other IPs.  Sadly - whoever has control of this botnet is not just running out of China, Russian, Iran, etc. they also have numerous source IPs in the USA -- New York, California, Washington State, etc. so blocking incoming connections outside of North America may not be a complete solution....

---------------------------------------------------

Sep 15 17:34:45 rp6 sshd[21258]: WARNING: no suitable primes in /etc/ssh/moduli
Sep 15 17:34:45 rp6 sshd[21258]: debug2: monitor_read: 0 used once, disabling now
Sep 15 17:34:45 rp6 sshd[21258]: debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug2: bits set: 1053/2048 [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: Connection closed by <REDACTED IP> port 58879 [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: do_cleanup [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: monitor_read_log: child log fd closed
Sep 15 17:34:45 rp6 sshd[21258]: debug1: do_cleanup
Sep 15 17:34:45 rp6 sshd[21258]: debug1: Killing privsep child 21259
Sep 15 17:34:45 rp6 sshd[21258]: debug1: audit_event: unhandled event 12
Sep 15 17:35:03 rp6 sshd[660]: debug1: Forked child 21262.
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Set /proc/self/oom_score_adj to 0
Sep 15 17:35:03 rp6 sshd[21262]: debug1: rexec start in 7 out 7 newsock 7 pipe 9 sock 10
Sep 15 17:35:03 rp6 sshd[21262]: debug1: inetd sockets after dupping: 3, 3
Sep 15 17:35:03 rp6 sshd[21262]: Connection from <REDACTED IP> port 57168 on 192.168.86.25 port 22
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Client protocol version 2.0; client software version libssh-0.6.3
Sep 15 17:35:03 rp6 sshd[21262]: debug1: no match: libssh-0.6.3
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2
Sep 15 17:35:03 rp6 sshd[21262]: debug2: fd 3 setting O_NONBLOCK
Sep 15 17:35:03 rp6 sshd[21262]: debug2: Network child is on pid 21263
Sep 15 17:35:03 rp6 sshd[21262]: debug1: permanently_set_uid: 107/65534 [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: list_hostkey_types: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: Connection closed by 138.197.222.141 port 57168 [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: do_cleanup [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: monitor_read_log: child log fd closed
Sep 15 17:35:03 rp6 sshd[21262]: debug1: do_cleanup
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Killing privsep child 21263
Sep 15 17:35:03 rp6 sshd[21262]: debug1: audit_event: unhandled event 12
Sep 15 17:35:25 rp6 sshd[660]: debug1: Forked child 21265.
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Set /proc/self/oom_score_adj to 0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: rexec start in 7 out 7 newsock 7 pipe 9 sock 10
Sep 15 17:35:25 rp6 sshd[21265]: debug1: inetd sockets after dupping: 3, 3
Sep 15 17:35:25 rp6 sshd[21265]: Connection from <REDACTED IP> port 58879 on 192.168.86.25 port 22
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Client protocol version 2.0; client software version libssh2_1.8.0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: no match: libssh2_1.8.0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2
Sep 15 17:35:25 rp6 sshd[21265]: debug2: fd 3 setting O_NONBLOCK
Sep 15 17:35:25 rp6 sshd[21265]: debug2: Network child is on pid 21266
Sep 15 17:35:25 rp6 sshd[21265]: debug1: permanently_set_uid: 107/65534 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: list_hostkey_types: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: SSH2_MSG_KEXINIT received [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: local server KEXINIT proposal [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: host key algorithms: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression ctos: none,zlib@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression stoc: none,zlib@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages ctos:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages stoc:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: first_kex_follows 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: reserved 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: peer client KEXINIT proposal [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: host key algorithms: ssh-rsa,ssh-dss [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,blowfish-cbc,arcfour128,arcfour,cast128-cbc,3des-cbc [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,blowfish-cbc,arcfour128,arcfour,cast128-cbc,3des-cbc [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs ctos: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,hmac-ripemd160@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs stoc: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,hmac-ripemd160@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression ctos: none [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression stoc: none [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages ctos:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages stoc:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: first_kex_follows 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: reserved 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: host key algorithm: ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: expecting SSH2_MSG_KEX_DH_GEX_REQUEST [preauth]
Sep 15 17:35:26 rp6 sshd[21265]: debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth]

Click here to read the complete article
Subject: Re: Looking for help / advice against aggressive BotNet triggering sshd errors which fill auth.log rapidly....
From: William Unruh
Newsgroups: comp.security.ssh
Organization: A noiseless patient Spider
Date: Fri, 18 Sep 2020 20:30 UTC
References: 1
Path: i2pn2.org!i2pn.org!eternal-september.org!feeder.eternal-september.org!reader02.eternal-september.org!.POSTED!not-for-mail
From: unr...@invalid.ca (William Unruh)
Newsgroups: comp.security.ssh
Subject: Re: Looking for help / advice against aggressive BotNet triggering
sshd errors which fill auth.log rapidly....
Date: Fri, 18 Sep 2020 20:30:45 -0000 (UTC)
Organization: A noiseless patient Spider
Lines: 129
Message-ID: <rk35dk$84b$1@dont-email.me>
References: <7335f450-7669-49c2-816d-745aa04674bfn@googlegroups.com>
Injection-Date: Fri, 18 Sep 2020 20:30:45 -0000 (UTC)
Injection-Info: reader02.eternal-september.org; posting-host="8ce4a14a8826a5070263838878524bdf";
logging-data="8331"; mail-complaints-to="abuse@eternal-september.org"; posting-account="U2FsdGVkX1+Vr29jkak0m3VlkUESaEx2"
User-Agent: slrn/1.0.3 (Linux)
Cancel-Lock: sha1:lfCELH5Vw39HTE8cSrlgf5BGzIA=
View all headers
On 2020-09-18, JT High <jthigh@gmail.com> wrote:
I'm looking for any help / advice against aggressive BotNet triggering sshd errors which fill auth.log rapidly.... What I am seeing could be a legitimate sshd bug -- or it could be a misconfiguration on my part that I've never encountered before...

I have a small linux host which is now and has been kept up to date, and has to the best of my ability been hardened for SSH and had fail2ban installed on it.  Generally this host deals with the typical SSH Internet brute force attempts fine, and fail2ban bans recurring source IP attempts as planned.

Why do you have debug (-dd)  enabled for sshd?

fail2ban looks for bad usernames/failed passwords as far as I know, not
debug messages.



This past week something "New and unexpected" happened... a botnet came after this host, and although I don't see any sign they ever successfully authenticated (which is also protected by google-authenticator-libpam), they nevertheless managed to do something that FILLed auth.log very rapidly, and effectively crashed the host until I realized it had a problem.  None of what they were doing triggered jail from Fail2Ban which I get email notifications for...

The auth.log on this host (a sample is shown below) grew to 11GB rapidly and shows more than 250 source IPs from all around the world converging on this host with two or three at a time making attempts -- not just the typical script kiddie / dictionary attacks either - whoever this is intentionally never stayed on the same IPs long enough the get banned and the recycling of source IPs was carefully done...

In /var/log/auth.log I see this:

Please note the topmost line, and the lines at the bottom, that look like this:
Sep 15 17:35:26 rp6 sshd[21265]: error: moduli:1: type is not 2

There are millions of these lines with different integers, and an astonishing number of them were written quickly into auth.log, but none of this activity seemed to generate a ban out of fail2ban

Can someone more skilled than me - please eyeball this log snippet and tell me if I should bother anyone with a bug report, or if I am the one with the 'config' problem?

This host does not have any file system corruption and regenerating the primes doesn't seem to fix this issue, as this botnet can illicit the same effect over and over again... manually banning the offending source IPs just slows things down for a short while and they are right back at me from other IPs.  Sadly - whoever has control of this botnet is not just running out of China, Russian, Iran, etc. they also have numerous source IPs in the USA -- New York, California, Washington State, etc. so blocking incoming connections outside of North America may not be a complete solution....

---------------------------------------------------

Sep 15 17:34:45 rp6 sshd[21258]: WARNING: no suitable primes in /etc/ssh/moduli
Sep 15 17:34:45 rp6 sshd[21258]: debug2: monitor_read: 0 used once, disabling now
Sep 15 17:34:45 rp6 sshd[21258]: debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug2: bits set: 1053/2048 [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: Connection closed by <REDACTED IP> port 58879 [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: do_cleanup [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: monitor_read_log: child log fd closed
Sep 15 17:34:45 rp6 sshd[21258]: debug1: do_cleanup
Sep 15 17:34:45 rp6 sshd[21258]: debug1: Killing privsep child 21259
Sep 15 17:34:45 rp6 sshd[21258]: debug1: audit_event: unhandled event 12
Sep 15 17:35:03 rp6 sshd[660]: debug1: Forked child 21262.
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Set /proc/self/oom_score_adj to 0
Sep 15 17:35:03 rp6 sshd[21262]: debug1: rexec start in 7 out 7 newsock 7 pipe 9 sock 10
Sep 15 17:35:03 rp6 sshd[21262]: debug1: inetd sockets after dupping: 3, 3
Sep 15 17:35:03 rp6 sshd[21262]: Connection from <REDACTED IP> port 57168 on 192.168.86.25 port 22
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Client protocol version 2.0; client software version libssh-0.6.3
Sep 15 17:35:03 rp6 sshd[21262]: debug1: no match: libssh-0.6.3
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2
Sep 15 17:35:03 rp6 sshd[21262]: debug2: fd 3 setting O_NONBLOCK
Sep 15 17:35:03 rp6 sshd[21262]: debug2: Network child is on pid 21263
Sep 15 17:35:03 rp6 sshd[21262]: debug1: permanently_set_uid: 107/65534 [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: list_hostkey_types: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: Connection closed by 138.197.222.141 port 57168 [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: do_cleanup [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: monitor_read_log: child log fd closed
Sep 15 17:35:03 rp6 sshd[21262]: debug1: do_cleanup
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Killing privsep child 21263
Sep 15 17:35:03 rp6 sshd[21262]: debug1: audit_event: unhandled event 12
Sep 15 17:35:25 rp6 sshd[660]: debug1: Forked child 21265.
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Set /proc/self/oom_score_adj to 0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: rexec start in 7 out 7 newsock 7 pipe 9 sock 10
Sep 15 17:35:25 rp6 sshd[21265]: debug1: inetd sockets after dupping: 3, 3
Sep 15 17:35:25 rp6 sshd[21265]: Connection from <REDACTED IP> port 58879 on 192.168.86.25 port 22
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Client protocol version 2.0; client software version libssh2_1.8.0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: no match: libssh2_1.8.0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2
Sep 15 17:35:25 rp6 sshd[21265]: debug2: fd 3 setting O_NONBLOCK
Sep 15 17:35:25 rp6 sshd[21265]: debug2: Network child is on pid 21266
Sep 15 17:35:25 rp6 sshd[21265]: debug1: permanently_set_uid: 107/65534 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: list_hostkey_types: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: SSH2_MSG_KEXINIT received [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: local server KEXINIT proposal [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: KEX algorithms: curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: host key algorithms: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression ctos: none,zlib@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression stoc: none,zlib@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages ctos:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages stoc:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: first_kex_follows 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: reserved 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: peer client KEXINIT proposal [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: host key algorithms: ssh-rsa,ssh-dss [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,blowfish-cbc,arcfour128,arcfour,cast128-cbc,3des-cbc [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,aes256-cbc,rijndael-cbc@lysator.liu.se,aes192-cbc,aes128-cbc,blowfish-cbc,arcfour128,arcfour,cast128-cbc,3des-cbc [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs ctos: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,hmac-ripemd160@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs stoc: hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96,hmac-ripemd160,hmac-ripemd160@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression ctos: none [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression stoc: none [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages ctos:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages stoc:  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: first_kex_follows 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: reserved 0  [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: host key algorithm: ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none [preauth]

Click here to read the complete article
Subject: Re: Looking for help / advice against aggressive BotNet triggering sshd errors which fill auth.log rapidly....
From: JT High
Newsgroups: comp.security.ssh
Date: Sun, 20 Sep 2020 02:18 UTC
References: 1 2
X-Received: by 2002:ac8:4e0b:: with SMTP id c11mr27759013qtw.37.1600568292389;
Sat, 19 Sep 2020 19:18:12 -0700 (PDT)
X-Received: by 2002:a05:6808:a05:: with SMTP id n5mr14102413oij.154.1600568291906;
Sat, 19 Sep 2020 19:18:11 -0700 (PDT)
Path: i2pn2.org!i2pn.org!aioe.org!peer02.ams4!peer.am4.highwinds-media.com!peer01.iad!feed-me.highwinds-media.com!news.highwinds-media.com!news-out.google.com!nntp.google.com!postnews.google.com!google-groups.googlegroups.com!not-for-mail
Newsgroups: comp.security.ssh
Date: Sat, 19 Sep 2020 19:18:11 -0700 (PDT)
In-Reply-To: <rk35dk$84b$1@dont-email.me>
Complaints-To: groups-abuse@google.com
Injection-Info: google-groups.googlegroups.com; posting-host=2600:1700:ba0:d2b8:88a3:546c:755:b749;
posting-account=6nq-igoAAABCZUe4jhBCYapcvMbDdZ5b
NNTP-Posting-Host: 2600:1700:ba0:d2b8:88a3:546c:755:b749
References: <7335f450-7669-49c2-816d-745aa04674bfn@googlegroups.com> <rk35dk$84b$1@dont-email.me>
User-Agent: G2/1.0
MIME-Version: 1.0
Message-ID: <2bf53a14-3515-4459-ad47-c36381c89140n@googlegroups.com>
Subject: Re: Looking for help / advice against aggressive BotNet triggering
sshd errors which fill auth.log rapidly....
From: jth...@gmail.com (JT High)
Injection-Date: Sun, 20 Sep 2020 02:18:12 +0000
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
X-Received-Bytes: 14707
X-Received-Body-CRC: 1190267918
View all headers
William - thank you so much for the direction!

I'm not sure how debug was turned on for sshd, (maybe some prior troubleshooting that was left turned on) but you are right - it was on and that certainly caused auth.log to fill up much more quickly than normal.

Fail2Ban as you suggested, out of the box, doesn't jail anything untoward going on in auth.log for sshd, in this case the BotNet with > 250 source IPs were able
to generate 10GB of auth.log (albeit with DEBUG on sshd) in a short period without technically getting a failed password, as they never made it to the prompt to enter a 2nd factor code, much less a correct password.

I see there are discussions / feature requests on github where people desire more regex filters to find some of these patterns and to jail them also, even if the source IP is not technically making it to, and failing with a bad password type entry in auth.log -> https://github.com/fail2ban/fail2ban/issues; If you are going to use multi-factor, and require that code before the password that definitely changes what Fail2Ban sees in Auth.log

* Turned debug on sshd down to verbose, cleaned up the moduli file, and did a refresh of the sshd hardening via https://www.sshaudit.com/ and all seems well now.







On Friday, September 18, 2020 at 4:30:47 PM UTC-4, William Unruh wrote:
On 2020-09-18, JT High <jth...@gmail.com> wrote:
I'm looking for any help / advice against aggressive BotNet triggering sshd errors which fill auth.log rapidly.... What I am seeing could be a legitimate sshd bug -- or it could be a misconfiguration on my part that I've never encountered before...

I have a small linux host which is now and has been kept up to date, and has to the best of my ability been hardened for SSH and had fail2ban installed on it. Generally this host deals with the typical SSH Internet brute force attempts fine, and fail2ban bans recurring source IP attempts as planned.
Why do you have debug (-dd) enabled for sshd?

fail2ban looks for bad usernames/failed passwords as far as I know, not
debug messages.

This past week something "New and unexpected" happened... a botnet came after this host, and although I don't see any sign they ever successfully authenticated (which is also protected by google-authenticator-libpam), they nevertheless managed to do something that FILLed auth.log very rapidly, and effectively crashed the host until I realized it had a problem. None of what they were doing triggered jail from Fail2Ban which I get email notifications for...

The auth.log on this host (a sample is shown below) grew to 11GB rapidly and shows more than 250 source IPs from all around the world converging on this host with two or three at a time making attempts -- not just the typical script kiddie / dictionary attacks either - whoever this is intentionally never stayed on the same IPs long enough the get banned and the recycling of source IPs was carefully done...

In /var/log/auth.log I see this:

Please note the topmost line, and the lines at the bottom, that look like this:
Sep 15 17:35:26 rp6 sshd[21265]: error: moduli:1: type is not 2

There are millions of these lines with different integers, and an astonishing number of them were written quickly into auth.log, but none of this activity seemed to generate a ban out of fail2ban

Can someone more skilled than me - please eyeball this log snippet and tell me if I should bother anyone with a bug report, or if I am the one with the 'config' problem?

This host does not have any file system corruption and regenerating the primes doesn't seem to fix this issue, as this botnet can illicit the same effect over and over again... manually banning the offending source IPs just slows things down for a short while and they are right back at me from other IPs. Sadly - whoever has control of this botnet is not just running out of China, Russian, Iran, etc. they also have numerous source IPs in the USA -- New York, California, Washington State, etc. so blocking incoming connections outside of North America may not be a complete solution....

---------------------------------------------------

Sep 15 17:34:45 rp6 sshd[21258]: WARNING: no suitable primes in /etc/ssh/moduli
Sep 15 17:34:45 rp6 sshd[21258]: debug2: monitor_read: 0 used once, disabling now
Sep 15 17:34:45 rp6 sshd[21258]: debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug2: bits set: 1053/2048 [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: Connection closed by <REDACTED IP> port 58879 [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: do_cleanup [preauth]
Sep 15 17:34:45 rp6 sshd[21258]: debug1: monitor_read_log: child log fd closed
Sep 15 17:34:45 rp6 sshd[21258]: debug1: do_cleanup
Sep 15 17:34:45 rp6 sshd[21258]: debug1: Killing privsep child 21259
Sep 15 17:34:45 rp6 sshd[21258]: debug1: audit_event: unhandled event 12
Sep 15 17:35:03 rp6 sshd[660]: debug1: Forked child 21262.
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Set /proc/self/oom_score_adj to 0
Sep 15 17:35:03 rp6 sshd[21262]: debug1: rexec start in 7 out 7 newsock 7 pipe 9 sock 10
Sep 15 17:35:03 rp6 sshd[21262]: debug1: inetd sockets after dupping: 3, 3
Sep 15 17:35:03 rp6 sshd[21262]: Connection from <REDACTED IP> port 57168 on 192.168.86.25 port 22
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Client protocol version 2.0; client software version libssh-0.6.3
Sep 15 17:35:03 rp6 sshd[21262]: debug1: no match: libssh-0.6.3
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2
Sep 15 17:35:03 rp6 sshd[21262]: debug2: fd 3 setting O_NONBLOCK
Sep 15 17:35:03 rp6 sshd[21262]: debug2: Network child is on pid 21263
Sep 15 17:35:03 rp6 sshd[21262]: debug1: permanently_set_uid: 107/65534 [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: list_hostkey_types: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: Connection closed by 138.197.222.141 port 57168 [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: do_cleanup [preauth]
Sep 15 17:35:03 rp6 sshd[21262]: debug1: monitor_read_log: child log fd closed
Sep 15 17:35:03 rp6 sshd[21262]: debug1: do_cleanup
Sep 15 17:35:03 rp6 sshd[21262]: debug1: Killing privsep child 21263
Sep 15 17:35:03 rp6 sshd[21262]: debug1: audit_event: unhandled event 12
Sep 15 17:35:25 rp6 sshd[660]: debug1: Forked child 21265.
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Set /proc/self/oom_score_adj to 0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: rexec start in 7 out 7 newsock 7 pipe 9 sock 10
Sep 15 17:35:25 rp6 sshd[21265]: debug1: inetd sockets after dupping: 3, 3
Sep 15 17:35:25 rp6 sshd[21265]: Connection from <REDACTED IP> port 58879 on 192.168.86.25 port 22
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Client protocol version 2.0; client software version libssh2_1.8.0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: no match: libssh2_1.8.0
Sep 15 17:35:25 rp6 sshd[21265]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2
Sep 15 17:35:25 rp6 sshd[21265]: debug2: fd 3 setting O_NONBLOCK
Sep 15 17:35:25 rp6 sshd[21265]: debug2: Network child is on pid 21266
Sep 15 17:35:25 rp6 sshd[21265]: debug1: permanently_set_uid: 107/65534 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: list_hostkey_types: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug1: SSH2_MSG_KEXINIT received [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: local server KEXINIT proposal [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: KEX algorithms: curve255...@libssh.org,diffie-hellman-group-exchange-sha256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: host key algorithms: ssh-ed25519,ssh-rsa [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers ctos: chacha20...@openssh.com,aes25...@openssh.com,aes12...@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers stoc: chacha20...@openssh.com,aes25...@openssh.com,aes12...@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs ctos: hmac-sha...@openssh..com,hmac-sha...@openssh.com,umac-1...@openssh.com,hmac-sha2-512,hmac-sha2-256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: MACs stoc: hmac-sha...@openssh..com,hmac-sha...@openssh.com,umac-1...@openssh.com,hmac-sha2-512,hmac-sha2-256 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression ctos: none,zl...@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: compression stoc: none,zl...@openssh.com [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages ctos: [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: languages stoc: [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: first_kex_follows 0 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: reserved 0 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: peer client KEXINIT proposal [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: host key algorithms: ssh-rsa,ssh-dss [preauth]
Sep 15 17:35:25 rp6 sshd[21265]: debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,aes256-cbc,rijnda...@lysator.liu.se,aes192-cbc,aes128-cbc,blowfish-cbc,arcfour128,arcfour,cast128-cbc,3des-cbc [preauth]

Click here to read the complete article
1
rocksolid light 0.7.2
clearneti2ptor