With all the fancy scientists in the world, why can't they just once build a nuclear balm?

devel / comp.security.ssh / Putty combined with CentOS 8 with 802.3ad bonding - lost keystrokes

Hello!
I have been following an issue with OpenSSH server on CentOS 8 - package openssh-server-8.0p1-5.el8.x86_64.rpm. The issue happens in combination with Putty (0.71, 0.73 and 0.74 tested) but not with other clients (Cygwin ssh and Windows 10 ssh).
When I press key 'a' nothing appears in the terminal. I have to press again several times till 'a' actually appears in Putty terminal.
This happens usually during the work day. At the time of writing I am not able to reproduce the issue. It happens more often when using Putty over DirectAccess tunnel, less likely over WiFi connection and someties at the start of SSH session even over corporate LAN.
At the same time when this issue happens with Putty other ssh clients have no problems.
First I thought that the issue is somewhere in the switching fiber because the CentOS 8 servers in question all have two network interfaces bonded with 802.3ad. But further investigation showed that the sshd doesn't respond with the packet as would be expected.
These are the packet capture from the server.
No response from server:
No. Time Source Destination Proto Length Info
3574933 4521.403882417 172.28.128.57 16.122.12.184 SSHv2 118 Client: Encrypted packet (len=64)
3574934 4521.403904725 16.122.12.184 172.28.128.57 TCP 54 22 -> 26758 [ACK] Seg:4494 Ack=4621 Win=43008 Len=0
With response from the server:
No. Time Source Destination Proto Length Info
3575606 4521.509405156 172.28.128.57 16.122.12.184 SSHv2 118 Client: Encrypted packet (len=64)
3575607 4521.509430159 16.122.12.184 172.28.128.57 TCP 54 22 -> 26758 [ACK] Seg:4494 Ack=4685 Win=43008 Len=0
3575608 4521.509765698 16.122.12.184 172.28.128.57 SSHv2 118 Server: Encrypted packet (len=64)
3575619 4521.594057396 172.28.128.57 16.122.12.184 TCP 60 26758 -> 22 [ACK] Seg:4685 Ack=4558 Win=65792 Len=0
From ltrace of sshd one can see that the server is not encripting and sending the response but I don't really know what the real problem is.
No response from the server:
----------------------------------------------------------------------------
sigemptyset(<>) = 0
sigaddset(<16>, SIGCHLD) = 0
sigprocmask(0, <16>, <>) = 0
sigprocmask(SIG_UNBLOCK, <>, nil) = 0
clock_gettime(7, 0x7ffc86124880, 0x55ce81041720, 0x55ce810414b0) = 0
read(5, "\301\r\335Q\224,a;\274Y<\312", 16384) = 64
memcpy(0x55ce810687e0, "\301\r\335Q\224,a;\274Y<\312\0\202\332\f\267/\275\224\333U.-\213XJl\0+j\270\236\376\275{q\224M\365\233K\334\346\200rd\215J\274yS\vx\305:\251\243\315\222\360\003\374k", 64) = 0x55ce810687e0
__explicit_bzero_chk(0x55ce81069a70, 256, -1, 5) = 0x55ce81069a70
EVP_Cipher(0x55ce8101e490, 0x55ce81069a70, 0x55ce810687e0, 16) = 1
EVP_Cipher(0x55ce8101e490, 0x55ce81069a80, 0x55ce810687f0, 16) = 1
EVP_MD_CTX_copy_ex(0x55ce81041800, 0x55ce8103fca0, 0, 32) = 1
EVP_DigestUpdate(0x55ce81041800, 0x7ffc861246fc, 4, 32) = 1
EVP_DigestUpdate(0x55ce81041800, 0x55ce81069a70, 32, 0xa4010000) = 1
EVP_DigestFinal_ex(0x55ce81041800, 0x55ce81045e00, 0x7ffc861246a4, 0xffffffff) = 1
EVP_MD_CTX_copy_ex(0x55ce81041800, 0x55ce81060b30, 0, 0x987b) = 1
EVP_DigestUpdate(0x55ce81041800, 0x55ce81045e00, 32, 0x987b) = 1
EVP_DigestFinal_ex(0x55ce81041800, 0x55ce801ae3c0, 0x7ffc861246a4, 0xffffffff) = 1
memcpy(0x7ffc86124750, "\236\376\275{q\224M\365\233K\334\346\200rd\215J\274yS\vx\305:\251\243\315\222\360\003\374k", 32) = 0x7ffc86124750
memcpy(0x55ce81044c80, "a", 1) = 0x55ce81044c80
memset(0x55ce81041720, '\0', 8) = 0x55ce81041720
memset(0x55ce810414b0, '\0', 8) = 0x55ce810414b0
clock_gettime(7, 0x7ffc861247d0, 0x55ce81041720, 0x55ce810414b0) = 0
select(21, 0x55ce81041720, 0x55ce810414b0, 0) = 1
sigemptyset(<>) = 0
sigaddset(<16>, SIGCHLD) = 0
sigprocmask(0, <16>, <>) = 0
sigprocmask(SIG_UNBLOCK, <>, nil) = 0
clock_gettime(7, 0x7ffc86124880, 0x55ce81041720, 0x55ce810414b0) = 0
write(16, "a", 1) = 1
tcgetattr(16, 0x7ffc86124820) = 0
memset(0x55ce81041720, '\0', 8) = 0x55ce81041720
memset(0x55ce810414b0, '\0', 8) = 0x55ce810414b0
clock_gettime(7, 0x7ffc861247d0, 0x55ce81041720, 0x55ce810414b0) = 0
select(21, 0x55ce81041720, 0x55ce810414b0, 0
----------------------------------------------------------------------------
With response from the server:
----------------------------------------------------------------------------
sigemptyset(<>) = 0
sigaddset(<16>, SIGCHLD) = 0
sigprocmask(0, <16>, <>) = 0
sigprocmask(SIG_UNBLOCK, <>, nil) = 0
clock_gettime(7, 0x7ffc86124880, 0x55ce81041720, 0x55ce810414b0) = 0
read(5, "y\351*)\n\264\261A\260\237\265\300;iR\244\266\217\340\a\204(n\017Roy\003W\300\367=\324R\227{\375c`W\201'\020\274\t\256`\251\350\304\266dDfo!\212\261u\341\216\346\177\216", 16384) = 64
memcpy(0x55ce810687e0, "y\351*)\n\264\261A\260\237\265\300;iR\244\266\217\340\a\204(n\017Roy\003W\300\367=\324R\227{\375c`W\201'\020\274\t\256`\251\350\304\266dDfo!\212\261u\341\216\346\177\216", 64) = 0x55ce810687e0
__explicit_bzero_chk(0x55ce81069a70, 256, -1, 5) = 0x55ce81069a70
EVP_Cipher(0x55ce8101e490, 0x55ce81069a70, 0x55ce810687e0, 16) = 1
EVP_Cipher(0x55ce8101e490, 0x55ce81069a80, 0x55ce810687f0, 16) = 1
EVP_MD_CTX_copy_ex(0x55ce81041800, 0x55ce8103fca0, 0, 32) = 1
EVP_DigestUpdate(0x55ce81041800, 0x7ffc861246fc, 4, 32) = 1
EVP_DigestUpdate(0x55ce81041800, 0x55ce81069a70, 32, 0xa3010000) = 1
EVP_DigestFinal_ex(0x55ce81041800, 0x55ce81045e00, 0x7ffc861246a4, 0xffffffff) = 1
EVP_MD_CTX_copy_ex(0x55ce81041800, 0x55ce81060b30, 0, 0x4ecc) = 1
EVP_DigestUpdate(0x55ce81041800, 0x55ce81045e00, 32, 0x4ecc) = 1
EVP_DigestFinal_ex(0x55ce81041800, 0x55ce801ae3c0, 0x7ffc861246a4, 0xffffffff) = 1
memcpy(0x7ffc86124750, "\324R\227{\375c`W\201'\020\274\t\256`\251\350\304\266dDfo!\212\261u\341\216\346\177\216", 32) = 0x7ffc86124750
memcpy(0x55ce81044c80, "a", 1) = 0x55ce81044c80
memset(0x55ce81041720, '\0', 8) = 0x55ce81041720
memset(0x55ce810414b0, '\0', 8) = 0x55ce810414b0
clock_gettime(7, 0x7ffc861247d0, 0x55ce81041720, 0x55ce810414b0) = 0
select(21, 0x55ce81041720, 0x55ce810414b0, 0) = 1
sigemptyset(<>) = 0
sigaddset(<16>, SIGCHLD) = 0
sigprocmask(0, <16>, <>) = 0
sigprocmask(SIG_UNBLOCK, <>, nil) = 0
clock_gettime(7, 0x7ffc86124880, 0x55ce81041720, 0x55ce810414b0) = 0
write(16, "a", 1) = 1
tcgetattr(16, 0x7ffc86124820) = 0
memset(0x55ce81041720, '\0', 8) = 0x55ce81041720
memset(0x55ce810414b0, '\0', 8) = 0x55ce810414b0
clock_gettime(7, 0x7ffc861247d0, 0x55ce81041720, 0x55ce810414b0) = 0
select(21, 0x55ce81041720, 0x55ce810414b0, 0) = 1
sigemptyset(<>) = 0
sigaddset(<16>, SIGCHLD) = 0
sigprocmask(0, <16>, <>) = 0
sigprocmask(SIG_UNBLOCK, <>, nil) = 0
clock_gettime(7, 0x7ffc86124880, 0x55ce81041720, 0x55ce810414b0) = 0
__errno_location() = 0x7f987ecbe738
read(20, "a", 16384) = 1
memcpy(0x55ce8101e9a0, "a", 1) = 0x55ce8101e9a0
__explicit_bzero_chk(0x55ce81045560, 256, -1, 1) = 0x55ce81045560
memcpy(0x55ce81045560, "\0\0\0\0\0^", 6) = 0x55ce81045560
memcpy(0x55ce8104556e, "a", 1) = 0x55ce8104556e
getpid() = 2596488
memcpy(0x55ce8104556f, "x\273@\0Rd\243\246O)\340@l\230jC\245", 17) = 0x55ce8104556f
memset(0x55ce801ae895, '\0', 17) = 0x55ce801ae895
EVP_MD_CTX_copy_ex(0x55ce8105da70, 0x55ce81041260, 0, 32) = 1
EVP_DigestUpdate(0x55ce8105da70, 0x7ffc8612476c, 4, 32) = 1
EVP_DigestUpdate(0x55ce8105da70, 0x55ce81045560, 32, 0x6b010000) = 1
EVP_DigestFinal_ex(0x55ce8105da70, 0x55ce81069990, 0x7ffc86124714, 0xffffffff) = 1
EVP_MD_CTX_copy_ex(0x55ce8105da70, 0x55ce8105e990, 0, 8059) = 1
EVP_DigestUpdate(0x55ce8105da70, 0x55ce81069990, 32, 8059) = 1
EVP_DigestFinal_ex(0x55ce8105da70, 0x55ce801ae3c0, 0x7ffc86124714, 0xffffffff) = 1
memcpy(0x7ffc861247f0, "\177.\v<9\314\277\276\252\177ND\335\034\322J\321\362\325\274Y+\033\302\352\336\364y\320sF\301", 32) = 0x7ffc861247f0
EVP_Cipher(0x55ce81041e90, 0x55ce81069c20, 0x55ce81045560, 32) = 1
memcpy(0x55ce81069c40, "\177.\v<9\314\277\276\252\177ND\335\034\322J\321\362\325\274Y+\033\302\352\336\364y\320sF\301", 32) = 0x55ce81069c40
__explicit_bzero_chk(0x55ce81045560, 256, -1, 0) = 0x55ce81045560
memset(0x55ce81041720, '\0', 8) = 0x55ce81041720
memset(0x55ce810414b0, '\0', 8) = 0x55ce810414b0
clock_gettime(7, 0x7ffc861247d0, 0x55ce81041720, 0x55ce810414b0) = 0
select(21, 0x55ce81041720, 0x55ce810414b0, 0) = 1
sigemptyset(<>) = 0
sigaddset(<16>, SIGCHLD) = 0
sigprocmask(0, <16>, <>) = 0
sigprocmask(SIG_UNBLOCK, <>, nil) = 0
clock_gettime(7, 0x7ffc86124880, 0x55ce81041720, 0x55ce810414b0) = 0
write(5, "\245\350\355\373\375\257N\301\025B\262\030=\223\242\244\326\246]\227\377?\304*\255d\327\211\005\314\2651\177.\v<9\314\277\276\252\177ND\335\034\322J\321\362\325\274Y+\033\302\352\336\364y\320sF\301", 64) = 64
memset(0x55ce81041720, '\0', 8) = 0x55ce81041720
memset(0x55ce810414b0, '\0', 8) = 0x55ce810414b0
clock_gettime(7, 0x7ffc861247d0, 0x55ce81041720, 0x55ce810414b0) = 0
select(21, 0x55ce81041720, 0x55ce810414b0, 0
----------------------------------------------------------------------------

Click here to read the complete article

Hello!

As it turns out the problem is using X11 Forwarding when X server is not running.

On Centos 8 (and probably on RHEL8) at login from Putty (with X11 forwarding) dbus-launches on the server (/etc/profile.d/ssh-x-forwarding.sh). If X11 server is not running the dbus-launch process remains alive on the same /dev/pts and it is stealing keystrokes from bash.

In terminal I enter keys a,b,c,d,e,f,g and in the terminal I get: abeg

strace of dbus-launch process:
root@baraba1: ~# strace -p 1276525
strace: Process 1276525 attached
select(1, [0], NULL, [0], NULL) = 1 (in [0])
read(0, "c", 512) = 1
select(1, [0], NULL, [0], NULL) = 1 (in [0])
read(0, "d", 512) = 1
select(1, [0], NULL, [0], NULL) = 1 (in [0])
read(0, "f", 512) = 1
select(1, [0], NULL, [0], NULL

strace of bash:
root@baraba1: ~# strace -p 1276497
strace: Process 1276497 attached
read(0, "a", 1) = 1
select(1, [0], NULL, [0], {tv_sec=0, tv_usec=0}) = 0 (Timeout)
write(2, "a", 1) = 1
pselect6(1, [0], NULL, NULL, NULL, {[], 8}) = 1 (in [0])
read(0, "b", 1) = 1
select(1, [0], NULL, [0], {tv_sec=0, tv_usec=0}) = 0 (Timeout)
write(2, "b", 1) = 1
pselect6(1, [0], NULL, NULL, NULL, {[], 8}) = 1 (in [0])
read(0, "e", 1) = 1
select(1, [0], NULL, [0], {tv_sec=0, tv_usec=0}) = 0 (Timeout)
write(2, "e", 1) = 1
pselect6(1, [0], NULL, NULL, NULL, {[], 8}) = 1 (in [0])
read(0, "g", 1) = 1
select(1, [0], NULL, [0], {tv_sec=0, tv_usec=0}) = 0 (Timeout)
write(2, "g", 1) = 1
pselect6(1, [0], NULL, NULL, NULL, {[], 8}

If this terminal hijacking from dbus-launch happens there is an easy remedy: just press Ctrl-C several times and dbus-launch will hijack ^C and exit :)

Well, disabling X11 forwardin when not needed is a good rule of thumb but for a better server-side solution I'll have to dig around RHEL bugzilla.

Why did the sshd run without problems on port 80? Because I ran it directly

# /usr/sbin/sshd -p 80

and X11Forwarding is disabled by default :)

I learned a lot while researching this issue and next time I can be more efficient.

Take care,
Miha

Subject	Author
Putty combined with CentOS 8 with 802.3ad bonding - lost keystrokes	MD Mikka
Re: Putty combined with CentOS 8 with 802.3ad bonding - lost keystrokes	MD Mikka