When I send mail from one of my servers to the other (both using sendmail 8.17.2), I am seeing this in my mail logs:

STARTTLS=client, cert-subject=/CN=mail.networkguild.org, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get issuer certificate

If I verify the cert by hand using:

openssl s_client -starttls smtp -showcerts -connect mail.networkguild.org:25 -servername mail.networkguild.org

it verifies OK and sees all the certificates in the chain. I am
indeed using the fullchain.pem file in my sendmail config. I've tried using both fullchain.pem and cert.pem as the CAcert file, result is the same.

My 'CN=mail.networkguild.org' depends on the 'CN=R3' cert. The
'CN=R3' cert depends on a cert named 'CN = ISRG Root X1'. I
definitely see that cert both in the chain my mail server presents and
in /etc/ssl.

I'm not sure if this is a debian issue, a sendmail issue, or a cert
issue. The fact that my cert verifies from other sites seems to
indicate it may be a sendmail sending issue, but I'm not sure.

Anyone have any ideas how to get sendmail to say this cert is valid?

Michael Grant

Claus can probably correct me, but that's what I use for SSL at both server and client settings on my .m4 file:

define(`confCACERT_PATH', `/etc/ssl/certs')
define(`confCACERT', `/etc/ssl/private/chain.pem')
define(`confSERVER_CERT', `/etc/ssl/private/cert.pem')
define(`confSERVER_KEY', `/etc/ssl/private/cert.key')
define(`confCLIENT_CERT', `/etc/ssl/private/cert.pem')
define(`confCLIENT_KEY', `/etc/ssl/private/cert.key')

Replacing paths and cert/key files by your own.

On Friday, January 5, 2024 at 2:20:42 AM UTC, HQuest wrote:
> Claus can probably correct me, but that's what I use for SSL at both server and client settings on my .m4 file:
>
> define(`confCACERT_PATH', `/etc/ssl/certs')
> define(`confCACERT', `/etc/ssl/private/chain.pem')
> define(`confSERVER_CERT', `/etc/ssl/private/cert.pem')
> define(`confSERVER_KEY', `/etc/ssl/private/cert.key')
> define(`confCLIENT_CERT', `/etc/ssl/private/cert.pem')
> define(`confCLIENT_KEY', `/etc/ssl/private/cert.key')
>
> Replacing paths and cert/key files by your own.

The errors I am seeing on the CLIENT end, as in, this is what someone running sendmail might see in their logs if they try sending me a message.

This is what I have the server:

define(`confCACERT_PATH', `/etc/ssl/certs')
define(`confCACERT', `/etc/letsencrypt/live/strange.networkguild.org/chain.pem')
define(`confSERVER_CERT', `/etc/letsencrypt/live/strange.networkguild.org/cert.pem')
define(`confSERVER_KEY', `/etc/letsencrypt/live/strange.networkguild.org/privkey.pem')
define(`confCLIENT_CERT', `/etc/letsencrypt/live/strange.networkguild.org/cert.pem')
define(`confCLIENT_KEY', `/etc/letsencrypt/live/strange.networkguild.org/privkey.pem')

This is what I see on the client side (sendmail server delivering a message to this server):

tls_clt_features=(null), relay=strange.networkguild.org [IPv6:2600:3c02:e000:dd:0:0:0:1]
tls_clt_features=empty, stat=0, relay=strange.networkguild.org [IPv6:2600:3c02:e000:dd:0:0:0:1]
STARTTLS=client, relay=strange.networkguild.org., version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
STARTTLS=client, cert-subject=/CN=strange.networkguild.org, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get issuer certificate
AUTH=client, relay=strange.networkguild.org., mech=, bits=0

Running this on the server produces:

# openssl storeutl -r -certs -noout -text /etc/letsencrypt/live/strange.networkguild.org/chain.pem | grep 'Issuer:|Subject:'
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Subject: C=US, O=Let's Encrypt, CN=R3
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1

The only thing I can see here is Sendmail is logging the subject with the space replaced with +20. Could this be making it not match? I'm wondering.... is there a bug in sendmail as an STARTTLS client where it's not properly verifying the chain cert with a space in it?

And just for completeness, I tried this in the reverse direction, sending a message in the other direction, and I get the same client unable to get issuer certificate. Again, the SSL client is Sendmail.

If I turn logging up to level 15, I see it's also failing for the root X1 cert:

tls_clt_features=(null), relay=strange.networkguild.org [IPv6:2600:3c02:e000:dd:0:0:0:1]
tls_clt_features=empty, stat=0, relay=strange.networkguild.org [IPv6:2600:3c02:e000:dd:0:0:0:1]
STARTTLS=client, start=ok
STARTTLS=client, info: fds=9/3, err=2
STARTTLS=client, info: fds=9/3, err=2
STARTTLS: TLS cert verify: depth=2 /C=US/O=Internet Security Research Group/CN=ISRG Root X1, state=0, reason=unable to get issuer certificate
STARTTLS=client, info: fds=9/3, err=2
STARTTLS=client, get_verify: 2 get_peer: 0x55bbc9147a40
STARTTLS=client, relay=strange.networkguild.org., version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
STARTTLS=client, cert-subject=/CN=strange.networkguild.org, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get issuer certificate

The cert chain is:
Issuer: C=US, O=Let's Encrypt, CN=R3
Subject: CN=strange.networkguild.org
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Subject: C=US, O=Let's Encrypt, CN=R3
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1

I can't explain where the space character in the Let's Encrypt cert is getting replaced with '+20'. And I can't explain why it's only the intermediate cert and not getting replaced in the X1 root cert as well.

I looked for other clients my sendmail connected to using Let's Encrypt certs and I always see this same error, so it's apparently not me.

Michael Grant wrote:

> I can't explain where the space character in the Let's Encrypt cert is
> getting replaced with '+20'.

Did you check the fine documentation?

6.7. Encoding of STARTTLS and AUTH related Macros

Macros that contain STARTTLS and AUTH related
data which comes from outside sources, e.g., all
macros containing information from certificates, are
encoded to avoid problems with non-printable or spe-
cial characters. The latter are '\', '<', '>', '(',
')', '"', '+', and ' '. All of these characters are
replaced by their value in hexadecimal with a leading
'+'. For example:

/C=US/ST=California/O=endmail.org/OU=private/CN=Darth Mail (Cert)/
Email=darth+cert@endmail.org

is encoded as:

/C=US/ST=California/O=endmail.org/OU=private/
CN=Darth+20Mail+20+28Cert+29/Email=darth+2Bcert@endmail.org

(line breaks have been inserted for readability). The
macros which are subject to this encoding are
{cert_subject}, {cert_issuer}, {cn_subject}, {cn_is-
suer}, as well as {auth_authen} and {auth_author}.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

Michael Grant wrote:

> STARTTLS=client, cert-subject=/CN=mail.networkguild.org,
> cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get
> issuer certificate

> My 'CN=mail.networkguild.org' depends on the 'CN=R3' cert. The
> 'CN=R3' cert depends on a cert named 'CN = ISRG Root X1'. I
> definitely see that cert both in the chain my mail server presents and
> in /etc/ssl.

Did you configure sendmail to use /etc/ssl for CACertPath?

openssl s_client refers to some certs / paths via its
config or builtin defaults.

sendmail does not do that.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

> Did you configure sendmail to use /etc/ssl for CACertPath?

Yes, I have tried both /etc/ssl and /etc/ssl/certs

> openssl s_client refers to some certs / paths via its
> config or builtin defaults.
>
> sendmail does not do that.

I'm not sure what you mean here. Do you think this might be an openssl issue?

I had a look in tls.c in sendmail's source. I have a difficult time following it, I would probably need to pull it into the debugger, but from sendmail's logging and the source, it seems to me that sendmail could be looping over the cert chain.

What I was trying to verify was whether when it did this loop, is it comparing the subject and issuer both with the special characters such as space substituted with the + notation.? (as in, is it comparing "O=Let's Encrypt" in the issuer to "O=Let's+20Encrypt" in the encoded subject)? Could there be a bug here, I wonder? Just a hypothesis.

Michael Grant wrote:
> > Did you configure sendmail to use /etc/ssl for CACertPath?

> Yes, I have tried both /etc/ssl and /etc/ssl/certs

Which of the two has the symbolic links for the CAs?
Do you use different openssl versions (library for sendmail
vs. openssl program)?

> > openssl s_client refers to some certs / paths via its
> > config or builtin defaults.

> > sendmail does not do that.

> I'm not sure what you mean here. Do you think this might be an openssl issue?

sendmail does not have any defaults for CAs,
so config must provide the required information.

> "O=Let's Encrypt" in the issuer to "O=Let's+20Encrypt" in the encoded
> subject)? Could there be a bug here, I wonder? Just a hypothesis.

"You can't be serious"...
The encoding is only used for display purposes.

The verification is done inside openssl, sendmail doesn't
"mess" with that - it invokes the proper openssl setup
functions with your CACertPath/CACertFile and leaves the rest
to the openssl code.

Anyway: I just tried it from one of my hosts using sendmail:

STARTTLS=client, relay=strange.networkguild.org., version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256

so with the right setup it works fine...

Check the openssl s_client (or verify) options: maybe it can show
which CAs it uses for verification?

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

On Tuesday, January 16, 2024 at 4:38:30 PM UTC, Claus Aßmann wrote:
> Michael Grant wrote:
> > > Did you configure sendmail to use /etc/ssl for CACertPath?
>
> > Yes, I have tried both /etc/ssl and /etc/ssl/certs
> Which of the two has the symbolic links for the CAs?
/etc/ssl/certs on debian and I also verified that the hash of the R3 cert it complained about is properly linked to the correct R3 cert.

> Do you use different openssl versions (library for sendmail
> vs. openssl program)?

No, not as far as I am aware. I did check though.

Both the sendmail executable and the openssl executable depend on libssl3, both use the same file, and this file is in the debian package libssl3- 3.0..11-1~deb12u2.

> sendmail does not have any defaults for CAs,
> so config must provide the required information.

Yup, am aware of this, I believe I have that properly configured.

> > "O=Let's Encrypt" in the issuer to "O=Let's+20Encrypt" in the encoded
> > subject)? Could there be a bug here, I wonder? Just a hypothesis.
> "You can't be serious"...
> The encoding is only used for display purposes.

Great
> Anyway: I just tried it from one of my hosts using sendmail:
>
> STARTTLS=client, relay=strange.networkguild.org., version=TLSv1.3, verify=OK, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
>
> so with the right setup it works fine...

I do see verify=OK for many other certificates for outbound connections, just not for Let's Encrypt.

> Check the openssl s_client (or verify) options: maybe it can show
> which CAs it uses for verification?

Ok, good point. Unfortunately, I didn't see such an option. I tried several, including the -debug option. Though I see the that it prints out the certs. I also think it doesn't really show me if the certs are valid, even though it says OK.

On the server side, I ran this:

$ openssl verify /etc/letsencrypt/live/strange.networkguild.org/fullchain.pem
CN = strange.networkguild.org
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/letsencrypt/live/strange.networkguild.org/fullchain.pem: verification failed

And now the plot thickens..... Unless I'm looking at the wrong thing, it looks like the X1 cert that is part of the cert chain for strange.networkguild.org is not the same X1 cert that I see in /etc/ssl/certs, yet, both have the same subject name. Both are valid.

I don't fully understand how this happened.

Thanks, you've pushed me in the right direction to sort this out.

Unfortunately, after some research, I still can't resolve this. Here's what I found so far:

All the Let's Encrypt certs signed by "C=US, O=Let's Encrypt, CN=R3" have this same error:

> cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get
> issuer certificate

Digging into these certs...

C=US, O=Let's Encrypt, CN=R3, in turn is signed by:
"C=US, O=Internet Security Research Group, CN=ISRG Root X1"

I have the self-signed ISRG Root X1 cert in /etc/ssl/certs/ on my
debian box which expires Jun 4 11:04:38 2035 GMT.

But the certificate chain that gets presented seems to have a
different ISRG Root X1 cert that's signed by:
"O=Digital Signature Trust Co., CN=DST Root CA X3" which expires
Sep 30 18:14:03 2024 GMT. (This is why I got confused yesterday).

But regardless I do have a valid self-signed ISRG Root X1 cert, so it
shouldn't matter, right? I don't understand why the Let's encrypt
certs are not being trusted.

How can I fix this? How can I accept these seemingly properly signed
Let's Encrypt certs which depend on the X1 cert which I have?

It seems quite normal to me to have a cert that is signed by multiple
different entities, one which I trust and other(s) which I don't.

I didn't use openssl verify correctly yesterday. Here's the proper usage using a chained cert:

$ openssl verify -show_chain -verbose -CApath /etc/ssl/certs/ -untrusted chain.pem cert.pem
cert.pem: OK
Chain:
depth=0: CN = strange.networkguild.org (untrusted)
depth=1: C = US, O = Let's Encrypt, CN = R3 (untrusted)
depth=2: C = US, O = Internet Security Research Group, CN = ISRG Root X1

openssl s_client returns 'Verify: OK'. I have not found a way to get it to show me which files it's using, but it does output the certificates. It outputs the certs that were presented to it in the connection. It, however, does not print out the self-signed ISRG Root X1 cert. It prints out the DST Root CA X3 signed X1 cert. I turned on all the verbose output I could find. However, it still reports Verify OK.

So I don't get it. As far as I can tell, everything is set up properly. Claus, I don't know which OS you are running sendmail on and I don't know if you happen to have this DST Root CA X3 cert which I don't have, but I really don't understand why that would matter. I also don't understand why sendmail doesn't "short-circuit" looking at the chain the way openssl-verify and openssl-s_client seem to do. I believe you when you say sendmail just calls the openssl library (which I only have one installed on my system).

I'm running sendmail 8.17.2 on debian stable. There was also some related discussion about this on the mailop mailing list back in September about the R3 cert, but it was not specific to sendmail and I didn't see any resolution that helped me in this situation.

Michael Grant <michael.grant@clinicalandherbal.com> wrote:
> [...]
> openssl s_client returns 'Verify: OK'. I have not found a way
> to get it to show me which files it's using, but it does output
> the certificates.

Unfortunately I cannot help you with your main problem, but you
can easily use Linux system call tracing to see which files
are opened. Just prefix your command with:

strace -o out -e open

br,
KK

Kalevi Kolttonen <kalevi@kolttonen.fi> wrote:
> Unfortunately I cannot help you with your main problem, but you
> can easily use Linux system call tracing to see which files
> are opened. Just prefix your command with:
>
> strace -o out -e open

Hmm, that gives no matches, but try this instead:

strace -o out -e openat

br,
KK

Michael Grant wrote:

> openssl s_client returns 'Verify: OK'. I have not found a way to get it
> to show me which files it's using, but it does output the certificates.

Write each of those certs into separate files
and check whether you have them in CACertPath used by sendmail.
Then check that the correct symbolic links exist too.
Also make sure all of the involved certs/links are readable
by everyone.

If that doesn't help:
post the output of openssl s_client,
esp. the certs that are used.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

The DST Root CA X3 certificate is not part of the default ca-certificates package from Mozilla, and likely not found into your certs/ folder - unless you manually added it *and* updated your certificate hooks (via update-ca-cerificate shell script). Alternatively, if you are using certbot to renew your certificates, add flag --preferred-chain "ISRG Root X1", so it uses the self-signed certificate and not the cross-signed ISRG Root Certificate, signed by the DST Root CA.

For more info on the LE certificate chain, see https://letsencrypt.org/certificates/ .

Hope this helps - it works like a champ over here with the self signed cert and the flags I originally sent you above.

I just tried straceing sendmail while I sent some messages.

Here are the cert files that got opened when I sent mail from bottom to strange.networkguild.org:

[bottom /etc/mail #1254] strace -p 3197777 --follow-forks 2>&1 | grep openat
....
[pid 3200438] openat(AT_FDCWD, "/etc/letsencrypt/live/bottom.networkguild.org/privkey.pem", O_RDONLY) = 13
[pid 3200438] openat(AT_FDCWD, "/etc/letsencrypt/live/bottom.networkguild.org/cert.pem", O_RDONLY) = 13
[pid 3200438] openat(AT_FDCWD, "/etc/letsencrypt/live/bottom.networkguild.org/chain.pem", O_RDONLY) = 13

and here are the cert files that got opened when I sent mail to my gmail account:

[pid 3213298] openat(AT_FDCWD, "/etc/letsencrypt/live/bottom.networkguild.org/privkey.pem", O_RDONLY) = 13
[pid 3213298] openat(AT_FDCWD, "/etc/letsencrypt/live/bottom.networkguild.org/cert.pem", O_RDONLY) = 13
[pid 3213298] openat(AT_FDCWD, "/etc/letsencrypt/live/bottom.networkguild.org/chain.pem", O_RDONLY) = 13
....
[pid 3213298] openat(AT_FDCWD, "/etc/ssl/certs/1001acf7.0", O_RDONLY) = 13

ls -l /etc/ssl/certs/1001acf7.0
lrwxrwxrwx 1 root root 15 Jun 17 2020 /etc/ssl/certs/1001acf7.0 -> GTS_Root_R1.pem

It's interesting, it does not look like sendmail even tries to open the X1 cert in /etc/ssl/certs when I send mail from bottom->strange, but when bottom tries to deliver a message to gmail, it does open Google's root cert in /etc/ssl/certs. It's as if sendmail (or libssl in this case) gives up when it sees the cross signed X1 cert signed by the non-existent DST Root CA X3 cert. But the openssl command line tools behave differently.

@HQuest,

I installed the expired DST Root CA X3 cert as you recommended. Here's what Sendmail says now:

STARTTLS=client, relay=strange.networkguild.org., version=TLSv1.3, verify=FAIL, cipher=TLS_AES_256_GCM_SHA384, bits=256/256
STARTTLS=client, cert-subject=/CN=strange.networkguild.org, cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=certificate has expired

Well, it's good news that it does not accept an exipred cert! But it's still not noticing it can use the perfectly valid X1 cert in /etc/ssl/certs.

Probably because you have both CACERT and CACERT_PATH, it is using the CACERT value first (i.e., your chain.pem certificate that contains the expired certificate) over the CACERT_PATH folder. Why certbot is issuing you a chain with an expired certificate is something you have to check with the Let's Encrypt folks, but you might want to remove the chain.pem off your configuration and use the non-expired certificate under /etc/ssl/certs, then - or, as I suggested, rely on one less certificate in the chain and use a specific root certificate whenever requesting your next certificates. But sure sendmail is not to be blamed here.

Ok so I've fixed it. (And there was much rejoicing!) And now I see the confusion.

certbot renew --force-renewal --preferred-chain "ISRG Root X1"

This command fixed things. I ran it on bottom, my outbound mail server. It updated bottom's cert and removed the dependency on the expired X3 cert.

This appears to have 2 effects:
1) when bottom presents its cert, it does not present the X3 chained cert. Totally expected.
2) when bottom receives a cert chain from a remote server, and that cert is a Let's Encrypt cert that also depends on the X3 cert in the chain, bottom now does not have it cached in memory and so does not use it. This was the confusing bit!

Also confusing is that openssl on the command line does not behave this way because it wasn't using the local chain with X3 in it to verify the remote chain. Perhaps there's some option to control that? Regardless, sendmail does not behave in the same way. It's certainly confusing to use the cached cert like this.

To be clear, anyone using sendmail who has an old letsencrypt cert that has an expired X3 in the chain should force renew this cert and prefer it to the X1 root as above. This will fix the verification problem locally when receiving certs other letsencrypt certs, regardless of whether the received cert depends on the expired X3 cert or not. Perhaps a good idea regardless of using sendmail or not.

I would definitely lay some fault on sendmail here. It could have complained about the local cert was depending on an expired cert in its chain and that would have forced me to fix the problem without hours of investigation and this long back and forth exchange. You can argue that sendmail actually didn't depend on the expired X3 cert for it's local cert or that it doesn't actually check such things because it's just sending these things, not verifying them--that's the other end's job. I would argue against that. It should try to validate these things on start and at least log this. I would classify that as validating your config.

Now, let's turn our attention to letsencrypt, or rather certbot. In reading the man page, it appears that certbot, by default, uses the original chain that the certificate was issued with, unless otherwise asked for, for example with the preferred-chain option. Presumably all new certbot certs would not get the old X3 cert? But I did actually try and delete strange's cert (my test client) and completely reissue it and it got the X3 cert in the chain, so I'm not so sure. Maybe letsencrypt cached the chain on their server? I will have to check next time I create a fresh new cert for a new domain if the X3 cert is in the chain. Honestly, it's an accident waiting to happen to automatically keep the expired X3 cert in the chain like that. This feels like a bug to me. I can't see any good reason why it would be good to default to keeping an expired cert in the chain and hope that all the software that used such a chain was smart enough to not use it and look for a better X1 cert.

Thanks everyone for helping sort this mystery out!

Michael Grant

Michael Grant wrote:

> I would definitely lay some fault on sendmail here. It could have
> complained about the local cert was depending on an expired cert in its
> chain and that would have forced me to fix the problem without hours of

Send a patch for another "DontBlameSendmail" option, including tests
which check the "correctness" of the patch.

--
Note: please read the netiquette before posting. I will almost never
reply to top-postings which include a full copy of the previous
article(s) at the end because it's annoying, shows that the poster
is too lazy to trim his article, and it's wasting the time of all readers.

I don't see the OP's messages because of all the spam from Google Groups.

Michael Grant wrote:
> I would definitely lay some fault on sendmail here. It could have
> complained about the local cert was depending on an expired cert in its
> chain and that would have forced me to fix the problem without hours of

Sendmail is in good company; Apache HTTP Server, NginX, Postfix, and IIS
will also happily start with expired certs somewhere in the chain.

Most daemons that I use trust that the system administrator knows what
they are doing and as such might have a good reason to start using an
expired cert that is beyond the daemon's understanding.

--
Grant. . . .

On Wednesday, January 24, 2024 at 3:45:50 AM UTC, Grant Taylor wrote:
> I don't see the OP's messages because of all the spam from Google Groups.
> Michael Grant wrote:
> > I would definitely lay some fault on sendmail here. It could have
> > complained about the local cert was depending on an expired cert in its
> > chain and that would have forced me to fix the problem without hours of
> Sendmail is in good company; Apache HTTP Server, NginX, Postfix, and IIS
> will also happily start with expired certs somewhere in the chain.
>
> Most daemons that I use trust that the system administrator knows what
> they are doing and as such might have a good reason to start using an
> expired cert that is beyond the daemon's understanding.

Grant, it's not about starting with an expired cert or not, though I do think sendmail should at least log that's what it's doing so the admin has the chance to notice the expired cert in the chain. I have started looking at the source to see if I can do make it do that.

The confusing thing here is that sendmail used the expired root cert of it's own server cert while verifying a remote cert. This definitely seems wrong to me, or at best confusing, especially in light it didn't log that it's own server cert was expired, or rather, one branch of it was.

I don't know if this is clear, let me try to spell this out:

Let's call the local server cert L. The local sendmail is configured with a cert which has a chain like L -> A -> B where B is a root CA cert (L is the cert in the cert file and A and B in the chain file). The L cert is issued by A and A is issued by B. But A has a second issuer called C which is in the local CA store. Hence you could also verify A by either B or C (as in L->A->B or L->A->C might both be valid chains, B in the chain file or C in the store). However, B has expired, but it's ok because C is still valid, so L -> A -> C is a verifiable good chain, you just ignore B in the chain and use C from the local cert store. All good, the server starts without complaint.

Then, at some point, the local server connects outbound to some remote server that just happens to use a cert also issued by A (the same A that issued the local cert L) and the remote proffers up the chain R -> A -> B, but as we noted, B is expired. Instead of looking in the cert store to find C, sendmail gives up and fails to verify R. And you might say, of course it does because the remote server sent the chain R -> A -> B and B has expired, of course it failed!

But wait... here's where it gets interesting! If, instead, when you configured the server's local server cert you configured it L -> A (meaning you put L in the cert file and just A in the chain file), and then when the remote server proffers up the chain R -> A -> B, it verifies good! Even though B is expired! Whoa! Sendmail somehow uses the store's A -> C chain and overlooks the expired B cert!

Therefor the second issue here I am raising--and I will try to fix it in sendmail as well as the first above-- is that when sendmail verifies R -> A -> B chain, it should not depend on the expired B cert it cached from the cert chain of it's own certificate chain. It should not be depending on some cached cert in the chain that it held onto from it's own local server cert chain. It should not co-mingle the verification chains of both the local and remote certificates.

And coming back to the comment about logging the expired cert, if sendmail had in fact logged the that the local cert depended on an expired cert in the chain as configured, that alone probably would have been enough for me to have spotted what was going on rather than depend on trial and error debugging.

Web server daemons don't normally make outbound connections and verify a remote cert, that's what a web browser does. Web browsers don't normally get configured with client certificates. This is why this situation does not arise with things like apache, nginx, and IIS. I do not know about Postfix.. I do not know if it's possible to reproduce this situation with a client and server cert using the command line openssl tools.

On 1/23/24 21:45, Grant Taylor wrote:
> I don't see the OP's messages because of all the spam from Google Groups.

Thank you for emailing me directly Michael.

I'm replying here in the hopes that what I write can help multiple people.

On 1/24/24 19:12, Michael Grant wrote:
> When I send mail from one of my servers to the other (both using
> sendmail 8.17.2), I am seeing this in my mail logs:

I believe I ran into a similar issue between some (then) 2-4 year old
Sendmail installs on a pair of my systems a few years ago.

> STARTTLS=client, cert-subject=/CN=mail.networkguild.org,
> cert-issuer=/C=US/O=Let's+20Encrypt/CN=R3, verifymsg=unable to get
> issuer certificate

This looks very familiar to me.

> If I verify the cert by hand using:
>
> openssl s_client -starttls smtp -showcerts -connect
> mail.networkguild.org:25 -servername mail.networkguild.org

Please elaborate on what "verify" means in this context. Are you
checking / comparing things by hand / script (grep, et al.)? Or are you
using the `openssl verify` (sub)command?

% time (openssl s_client -starttls smtp -showcerts -connect
mail.networkguild.org:25 -servername mail.networkguild.org < /dev/null
|& openssl verify)
stdin: OK
( openssl s_client -starttls smtp -showcerts -connect
mail.networkguild.org:2) 0.01s user 0.00s system 0% cpu 13.132 total

Aside: I'm surprised that it's taking 13 seconds to do this. I can
only assume that you have something akin to pre-greeting traffic
filtering. Though I would think that would be after the TCP+TLS
connection establishes and before the SMTP greeting banner.

> it verifies OK and sees all the certificates in the chain.

This seems remarkably similar to an issue I had between the two systems
running Sendmail alluded to above.

From memory, what I found after hours of digging is that one of the
root certificates that was then used to cross sign the Let's Encrypt
certificates had expired.

Like you I was providing the full chain.

But something about the OpenSSL install I was using still wasn't happy.

I don't have notes at hand, but I remember one of the root certificate
files in /etc/ssl/certs contained the current LE root /and/ the expired
cross signing cert (full chain of sorts). This expired cross signing
cert was being found and used by OpenSSL /before/ the current cross
signing cert in the same /etc/ssl/certs directory.

....

Yep, looking back at the script that I'm using to manage certificates
collected by acme.sh (I don't let acme.sh install things for me) I see
that the ca.cer (PEM) file that it downloads contains two certificates:

1st what is probably best described as an intermediate certificate:

subject= /C=US/O=Let's Encrypt/CN=R3
notBefore=Sep 4 00:00:00 2020 GMT
notAfter=Sep 15 16:00:00 2025 GMT
serial=912B084ACF0C18A753F6D62E25A75F5A
issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1

2nd what is probably best described as the root certificate:

subject= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT
serial=4001772137D4E942B8EE76AA3C640AB7
issuer= /O=Digital Signature Trust Co./CN=DST Root CA X3

The above output came from the following command:

cat ca.cer | awk 'BEGIN{cmd="openssl x509 -noout -subject -dates -serial
-issuer -extensions subjectAltName; echo"}/BEGIN CERTIFICATE/,/END
CERTIFICATE/{print | cmd}/END CERTIFICATE/{close(cmd)}'

So, I'm having openssl read in ca.cer and write out the first
certificate to another file. -- I then use /just/ the one certificate
(currently serial=912B084ACF0C18A753F6D62E25A75F5A) and my certificate
to generate a new chain that doesn't include the other (expired at the
time I worked on this) certs in the chain.

As best as I could tell, this was all about things in the full chain
file from acme.sh (I don't know if it amalgamated it or if it acme.sh
simply downloaded it from somewhere else, maybe Let's Encrypt
themselves) being expired and the expired root cert being used /before/
OpenSSL would use the (then) current / new root cert in /etc/ssl/certs.

Upon spot checking this, I found that there was a smattering of
different numbers of certs in different files in /etc/ssl/certs.

Fortunately for me I was able to work around the problem for Let's
Encrypt which is what I cared about.

> I am indeed using the fullchain.pem file in my sendmail config. I've
> tried using both fullchain.pem and cert.pem as the CAcert file,
> result is the same.

I currently have the following in my sendmail.cf file (set via sendmail.mc):

# CA directory
O CACertPath=/etc/ssl/certs
# CA file
O CACertFile=/etc/mail/tls/letsencrypt-ca.cer

I want to say that the CACertFile wasn't strictly necessary after I
fixed things in the CACertPath.

N.B. the CACertFile is the file I extract just the 1st certificate
(currently serial=912B084ACF0C18A753F6D62E25A75F5A) into.

> My 'CN=mail.networkguild.org' depends on the 'CN=R3' cert. The 'CN=R3'
> cert depends on a cert named 'CN = ISRG Root X1'. I definitely see
> that cert both in the chain my mail server presents and in /etc/ssl.

This seems strikingly familiar.

I suspect that if (when) you dig, you will find that there may be an
expired certificate in another file, possibly the full chain file, that
is interfering with you.

> I'm not sure if this is a debian issue, a sendmail issue, or a cert
> issue. The fact that my cert verifies from other sites seems to
> indicate it may be a sendmail sending issue, but I'm not sure.

The systems that I ran into this issue on was a (now) ~5 year old Ubuntu
install. So it may be an upstream Debian issue that Ubuntu is inheriting.

Or it may have been a bug in the acme.sh client version I was using at
the time.

> Anyone have any ideas how to get sendmail to say this cert is valid?

As you can see above, the system that I'm talking about thinks that your
server's certificate /is/ valid and /does/ verify.

This is a VERY DEEP in the weeds issue. I spent hours over a few days
debugging this.

I spent the time debugging this because my servers are using their
certificate when they are sending / relaying through each other and I'm
allowing relay based on CERTISSUER+CERTSUBJECT in AccessDB:

# Allow <SERVER> to relay based on client TLS certificate.
CERTISSUER:/C=US/O=Let's+20Encrypt/CN=R3 SUBJECT
CERTSUBJECT:/CN=<SERVER1 subject> RELAY
CERTSUBJECT:/CN=<SERVER2 subject> RELAY

Please feel free to email me again if you'd like to discuss any private
command output / investigation.

I hope my write is helpful.

--
Grant. . . .