> On 17 Aug 2021, at 19:25, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Wed, Aug 18, 2021 at 4:16 AM Barry Scott <barry@barrys-emacs.org> wrote:
>> Oh and if you have the freedom avoid Basic Auth as its not secure at all.
>>
>
> That's usually irrelevant, since the alternative is most likely to be
> form fill-out, which is exactly as secure. If you're serving over
> HTTPS, the page is encrypted, and that includes the headers; if you're
> not, then it's not encrypted, and that includes the form body.

There is digest and Ntlm that do not reveal the password.

If you are over TLS then form or base is as good as each other.

Barry
>
> There are other issues with basic auth, but security really isn't one.
>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

On 2021-08-17, Barry <barry@barrys-emacs.org> wrote:
>> That's usually irrelevant, since the alternative is most likely to be
>> form fill-out, which is exactly as secure. If you're serving over
>> HTTPS, the page is encrypted, and that includes the headers; if you're
>> not, then it's not encrypted, and that includes the form body.
>
> There is digest and Ntlm that do not reveal the password.

That's only true if you're not using HTTPS - and you should *never*
not be using HTTPS, and that goes double if forms are being filled
in and double again if passwords are being supplied.

On 17/08/2021 22:47, Jon Ribbens via Python-list wrote:
........
> That's only true if you're not using HTTPS - and you should *never*
> not be using HTTPS, and that goes double if forms are being filled
> in and double again if passwords are being supplied.
>

I think I agree with most of the replies; I understood from reading the rfc that the charset is utf8 (presumably without
':') and that basic auth is considered insecure. It is being used over https so should avoid the simplest net scanning.

I googled a bunch of ways to do this, but many come down to 1) using the requests package or 2) setting up an opener.
Both of these seem to be much more complex than is required to add the header.

I thought there might be a shortcut or more elegant way to replace the old code, but it seems not

thanks
--
Robin Becker

On 2021-08-18, Robin Becker <robin@reportlab.com> wrote:
> On 17/08/2021 22:47, Jon Ribbens via Python-list wrote:
> .......
>> That's only true if you're not using HTTPS - and you should *never*
>> not be using HTTPS, and that goes double if forms are being filled
>> in and double again if passwords are being supplied.
>
> I think I agree with most of the replies; I understood from reading
> the rfc that the charset is utf8 (presumably without ':')

The username can't contain a ':'. It shouldn't matter in the password.

> and that basic auth is considered insecure. It is being used over
> https so should avoid the simplest net scanning.

It's not insecure over HTTPS. Bear in mind the Basic Auth RFC was
written when HTTP was the standard and HTTPS was unusual. The positions
are now effectively reversed.

> I googled a bunch of ways to do this, but many come down to 1) using
> the requests package or 2) setting up an opener. Both of these seem to
> be much more complex than is required to add the header.
>
> I thought there might be a shortcut or more elegant way to replace the
> old code, but it seems not

It's only a trivial str/bytes difference, it shouldn't be any big deal.
But using 'requests' instead is likely to simplify things and doesn't
tend to be an onerous dependency.

While it is correct to say that Basic Auth without HTTPS is absolutely
insecure, using Basic Auth *and* HTTPS is not secure either.

Well, the definition of "secure" depends of your threat model.

HTTPS ensures encryption so the content, including the Basic Auth
username and password, is secret for any external observer.

But it is *not* secret for the receiver (the server): if it was
compromised an adversary will have access to your password. It is much
easier to print a captured password than cracking the hashes.

Other authentication mechanisms exist, like OAuth, which are more
"secure".

Thanks,
Martin

On Wed, Aug 18, 2021 at 11:05:46PM -0000, Jon Ribbens via Python-list wrote:
>On 2021-08-18, Robin Becker <robin@reportlab.com> wrote:
>> On 17/08/2021 22:47, Jon Ribbens via Python-list wrote:
>> .......
>>> That's only true if you're not using HTTPS - and you should *never*
>>> not be using HTTPS, and that goes double if forms are being filled
>>> in and double again if passwords are being supplied.
>>
>> I think I agree with most of the replies; I understood from reading
>> the rfc that the charset is utf8 (presumably without ':')
>
>The username can't contain a ':'. It shouldn't matter in the password.
>
>> and that basic auth is considered insecure. It is being used over
>> https so should avoid the simplest net scanning.
>
>It's not insecure over HTTPS. Bear in mind the Basic Auth RFC was
>written when HTTP was the standard and HTTPS was unusual. The positions
>are now effectively reversed.
>
>> I googled a bunch of ways to do this, but many come down to 1) using
>> the requests package or 2) setting up an opener. Both of these seem to
>> be much more complex than is required to add the header.
>>
>> I thought there might be a shortcut or more elegant way to replace the
>> old code, but it seems not
>
>It's only a trivial str/bytes difference, it shouldn't be any big deal.
>But using 'requests' instead is likely to simplify things and doesn't
>tend to be an onerous dependency.
>--
>https://mail.python.org/mailman/listinfo/python-list

On Sun, Aug 22, 2021 at 4:55 AM Martin Di Paola
<martinp.dipaola@gmail.com> wrote:
>
> While it is correct to say that Basic Auth without HTTPS is absolutely
> insecure, using Basic Auth *and* HTTPS is not secure either.
>
> Well, the definition of "secure" depends of your threat model.

Yes. Which makes statements like "not secure" rather suspect :)

> HTTPS ensures encryption so the content, including the Basic Auth
> username and password, is secret for any external observer.
>
> But it is *not* secret for the receiver (the server): if it was
> compromised an adversary will have access to your password. It is much
> easier to print a captured password than cracking the hashes.
>
> Other authentication mechanisms exist, like OAuth, which are more
> "secure".

If your server is compromised in that way, *all is lost*. If an
attacker is actually running code on your server, listening to your
sockets, after everything's decrypted, then *shut that server down*. I
don't think there is ANY security model that can handle this - if
you're using OAuth, and the server is compromised, then your client ID
and client secret are just as visible to the attacker as passwords
would be.

ChrisA

On 2021-08-22 05:04:43 +1000, Chris Angelico wrote:
> On Sun, Aug 22, 2021 at 4:55 AM Martin Di Paola
> <martinp.dipaola@gmail.com> wrote:
> >
> > While it is correct to say that Basic Auth without HTTPS is absolutely
> > insecure, using Basic Auth *and* HTTPS is not secure either.
> >
> > Well, the definition of "secure" depends of your threat model.
>
> Yes. Which makes statements like "not secure" rather suspect :)

Right.

> > HTTPS ensures encryption so the content, including the Basic Auth
> > username and password, is secret for any external observer.
> >
> > But it is *not* secret for the receiver (the server): if it was
> > compromised an adversary will have access to your password. It is much
> > easier to print a captured password than cracking the hashes.
> >
> > Other authentication mechanisms exist, like OAuth, which are more
> > "secure".

OAuth is "an /authorization protocol/, rather than an /authentication
protocol/" [Wikipedia].

> If your server is compromised in that way, *all is lost*.

If "you" are the service provider, yes. but if "you" are the user, no.

From a user's perspective "all" is much more than the data (including
username and password) associated with that particular service. So if
one service is compromised, not all is lost, but only a bit (of course,
depending on the importance of the service, that bit may be little or
big; a random web forum probably doesn't matter. Your bank account
probably does).

So assuming that many people reuse passwords (which of course they
shouldn't and thanks to password is becoming rarer, but is still
distressingly common), there are three levels of security (from highest
to lowest) in this scenario:

1: The secret known to the user is never transmitted to the server at
all, the client only proves that the secret is known. This is the
case for TLS client authentication (which AFAIK all browsers support
but is a real pain in the ass to set up, so it's basically never
used) and for SCRAM (which isn't part of HTTP(S) but could be
implemented in JavaScript).

2: The secret is transmitted on login but never stored. This limits the
damage to users who logged in while the server was compromised. This
is the case for Basic Authentication combined with a probperly salted
hashed storage.

3: The secret is stored on the server. When the server is compromised,
all user's passwords are known. This is (AFAIK) the case for Digest
and NTLM.

So given the choice between Basic Auth and Digest or NTLM (over HTTPS in
all cases) I would prefer Basic Auth. Ideally I would use SCRAM or a
public key method, but I admit that my security requirements were never
high enough to actually bother to do that (actually, I used SSL client
side auth once, 20 years ago, ...).

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

On Sun, Aug 22, 2021 at 6:45 PM Peter J. Holzer <hjp-python@hjp.at> wrote:
>
> On 2021-08-22 05:04:43 +1000, Chris Angelico wrote:
> > On Sun, Aug 22, 2021 at 4:55 AM Martin Di Paola
> > <martinp.dipaola@gmail.com> wrote:
> > > HTTPS ensures encryption so the content, including the Basic Auth
> > > username and password, is secret for any external observer.
> > >
> > > But it is *not* secret for the receiver (the server): if it was
> > > compromised an adversary will have access to your password. It is much
> > > easier to print a captured password than cracking the hashes.
> > >
> > > Other authentication mechanisms exist, like OAuth, which are more
> > > "secure".
>
> OAuth is "an /authorization protocol/, rather than an /authentication
> protocol/" [Wikipedia].
>
> > If your server is compromised in that way, *all is lost*.
>
> If "you" are the service provider, yes. but if "you" are the user, no.

If "your server" is compromised, then you are the service provider,
are you not? I'm not sure what "your server" would mean if "you" are
the user.

But okay. Suppose I log in to Random Service 1, using a user name and
password, and also to Random Service 2, using OAuth. What happens if
those servers get compromised?

1) Someone knows the login credentials that I created for that
service. If I've used the same password that I also use at my bank,
then I am in big trouble. It is, largely, my fault.

2) Someone has access to my login token and the client ID/secret
associated with it. That attacker can now impersonate me to the OAuth
provider, to the exact extent that the scopes permit. At absolute
least, the attacker gets to know a lot about who I am on some entirely
separate service.

I'm talking here about a complete and utter compromise, the sort where
neither SSL encryption nor proper password hashing would protect my
details, since that's what was being claimed.

Which is actually worse? Is it as clear-cut?

> From a user's perspective "all" is much more than the data (including
> username and password) associated with that particular service. So if
> one service is compromised, not all is lost, but only a bit (of course,
> depending on the importance of the service, that bit may be little or
> big; a random web forum probably doesn't matter. Your bank account
> probably does).
>
> So assuming that many people reuse passwords (which of course they
> shouldn't and thanks to password is becoming rarer, but is still
> distressingly common),

True, but reuse of passwords is something under the user's control.
OAuth scope selection is partly under the service's control, and
partly under the provider's (some providers have extremely coarse
scopes, widening the attack).

> there are three levels of security (from highest
> to lowest) in this scenario:
>
> 1: The secret known to the user is never transmitted to the server at
> all, the client only proves that the secret is known. This is the
> case for TLS client authentication (which AFAIK all browsers support
> but is a real pain in the ass to set up, so it's basically never
> used) and for SCRAM (which isn't part of HTTP(S) but could be
> implemented in JavaScript).

This would be great, if nobody minded (a) setting up a unique client
certificate for every site, or (b) allowing the ultimate in remote
tracking cookie whereby any server could recognize you by your TLS
certificate.

> 2: The secret is transmitted on login but never stored. This limits the
> damage to users who logged in while the server was compromised. This
> is the case for Basic Authentication combined with a probperly salted
> hashed storage.

Current best prac, and what I'd generally recommend to most people.

> 3: The secret is stored on the server. When the server is compromised,
> all user's passwords are known. This is (AFAIK) the case for Digest
> and NTLM.

I'm not sure what the effects of wide-spread Digest/NTLM usage would
have on password managers and the risks of compromise to them, but the
way things currently are, I would prefer salted/hashed passwords, such
that a data breach doesn't mean compromise of all historical data.

> So given the choice between Basic Auth and Digest or NTLM (over HTTPS in
> all cases) I would prefer Basic Auth. Ideally I would use SCRAM or a
> public key method, but I admit that my security requirements were never
> high enough to actually bother to do that (actually, I used SSL client
> side auth once, 20 years ago, ...).
>

I would, of course, prefer something like form fill-out over Basic,
but that's due to UI concerns rather than security ones.

SCRAM seems tempting, but in a context of web browsers, I'm not sure
that it would be worth the hassle.

When it comes to security, one thing I'm very curious about is why we
don't have any sort of certificate renewal verification. My browser
could retain the certificates of some web site (or of all web sites,
even - they're not THAT large), and if the site presents a different
cert, it could show the previously retained one and challenge the
server "prove that you're the same guy". This proof would consist of
the latest cert, signed by the older cert's key (or possibly a chain
that can construct such a proof, which would allow the server to
simply retain each new cert signed by the one previous cert, forming a
line - or a tree if necessary). My suspicion is that it'd add little
above simply having a valid cert, but if people are paranoid, surely
that's a better place to look?

ChrisA

> On 22 Aug 2021, at 10:37, Chris Angelico <rosuav@gmail.com> wrote:
>
> When it comes to security, one thing I'm very curious about is why we
> don't have any sort of certificate renewal verification. My browser
> could retain the certificates of some web site (or of all web sites,
> even - they're not THAT large), and if the site presents a different
> cert, it could show the previously retained one and challenge the
> server "prove that you're the same guy". This proof would consist of
> the latest cert, signed by the older cert's key (or possibly a chain
> that can construct such a proof, which would allow the server to
> simply retain each new cert signed by the one previous cert, forming a
> line - or a tree if necessary). My suspicion is that it'd add little
> above simply having a valid cert, but if people are paranoid, surely
> that's a better place to look?

The web site proves it owners the hostname and/or IP address using its certificate.
You use your trust store to show that you can trust that certificate.

The fact that a certificate changes is not a reason to stop trusting a site.

So it does not add anything.

The pain point in PKI is revocation. The gold standard is for a web site to use OCSP stapling.
But that is rare sadly. And because of issues with revocation lists, (privacy, latency, need to
fail open on failiure, DoD vector, etc) this is where the paranoid should look.

Barry

On Sun, Aug 22, 2021 at 8:30 PM Barry Scott <barry@barrys-emacs.org> wrote:
>
>
>
> On 22 Aug 2021, at 10:37, Chris Angelico <rosuav@gmail.com> wrote:
>
> When it comes to security, one thing I'm very curious about is why we
> don't have any sort of certificate renewal verification. My browser
> could retain the certificates of some web site (or of all web sites,
> even - they're not THAT large), and if the site presents a different
> cert, it could show the previously retained one and challenge the
> server "prove that you're the same guy". This proof would consist of
> the latest cert, signed by the older cert's key (or possibly a chain
> that can construct such a proof, which would allow the server to
> simply retain each new cert signed by the one previous cert, forming a
> line - or a tree if necessary). My suspicion is that it'd add little
> above simply having a valid cert, but if people are paranoid, surely
> that's a better place to look?
>
>
> The web site proves it owners the hostname and/or IP address using its certificate.
> You use your trust store to show that you can trust that certificate.
>
> The fact that a certificate changes is not a reason to stop trusting a site.
>
> So it does not add anything.
>
> The pain point in PKI is revocation. The gold standard is for a web site to use OCSP stapling.
> But that is rare sadly. And because of issues with revocation lists, (privacy, latency, need to
> fail open on failiure, DoD vector, etc) this is where the paranoid should look.
>

Fair point. Let me give you a bit of context.

Recently, the owner/operator of a site (I'll call it
https://demo.example/ ) died. Other people, who have been using the
site extensively, wish for it to continue. If the domain registration
expires, anyone can reregister it, and can then generate a completely
new certificate for the common name "demo.example", and web browsers
will accept that. The old cert may or may not have expired, but it
won't be revoked.

As far as I can tell, a web browser with default settings will happily
accept the change of ownership. It won't care that the IP address,
certificate, etc, have all changed. It just acknowledges that some CA
has signed some certificate with the right common name. And therein is
the vulnerability. (NOTE: I'm not saying that this is a real and
practical vulnerability - this is theoretical only, and a focus for
the paranoid.)

This is true even if the old cert were one of those enhanced
certificates that some CAs try to upsell you to ("Extended Validation"
and friends). Even if, in the past, your bank was secured by one of
those certs, your browser will still accept a perfectly standard cert
next time. Which, in my opinion, renders those (quite pricey)
certificates no more secure than something from Let's Encrypt that has
no validation beyond ownership of DNS.

Of course, you can pin a certificate. You can ask your browser to warn
you if it's changed *at all*. But since certs expire, that's highly
impractical, hence wondering why we don't have a system for using the
old cert to prove ownership of the new one.

So how is a web browser supposed to distinguish between (a) normal
operation in which certs expire and are replaced, and (b) legit or
non-legit ownership changes? (Of course the browser can't tell you
whether the ownership change is legit, but out-of-band info can help
with that.)

Or does it really matter that little?

ChrisA

On Wed, Aug 25, 2021 at 5:20 PM Barry Scott <barry@barrys-emacs.org> wrote:
>
> Only if this threat model matters to you or your organisation.
> Personal its low down of the threats I watch out for.
>
> The on-line world and the real-world are the same here.
>
> If a business changes hands then do you trust the new owners?
>
> Nothing we do with PKI certificates will answer that question.

Fair enough; but a closer parallel would be walking up to a
previously-familiar street vendor and seeing a different person there.
Did the business change hands, or did some random dude hop over the
counter and pretend to be a new owner?

But you're right, it's not usually a particularly high risk threat.
Still, it does further weaken the value of named SSL certificates and
certificate authorities; there's not actually that much difference if
the server just gave you a self-signed cert. In theory, the CA is
supposed to protect you against someone doing a DNS hack and
substituting a different server, in practice, anyone capable of doing
a large-scale DNS hack is probably capable of getting a very
legit-looking SSL cert for the name as well.

ChrisA

On 2021-08-25, Chris Angelico <rosuav@gmail.com> wrote:
> On Wed, Aug 25, 2021 at 5:20 PM Barry Scott <barry@barrys-emacs.org> wrote:
>> Only if this threat model matters to you or your organisation.
>> Personal its low down of the threats I watch out for.
>>
>> The on-line world and the real-world are the same here.
>>
>> If a business changes hands then do you trust the new owners?
>>
>> Nothing we do with PKI certificates will answer that question.
>
> Fair enough; but a closer parallel would be walking up to a
> previously-familiar street vendor and seeing a different person there.
> Did the business change hands, or did some random dude hop over the
> counter and pretend to be a new owner?
>
> But you're right, it's not usually a particularly high risk threat.
> Still, it does further weaken the value of named SSL certificates and
> certificate authorities; there's not actually that much difference if
> the server just gave you a self-signed cert. In theory, the CA is
> supposed to protect you against someone doing a DNS hack and
> substituting a different server, in practice, anyone capable of doing
> a large-scale DNS hack is probably capable of getting a very
> legit-looking SSL cert for the name as well.

There are so many trusted CAs these days that the chances of them all
being secure approaches zero - they are not all equal yet they are all
equally trusted. Which is why a change of CA on a site you have visited
before is potentially suspicious.

> On 22 Aug 2021, at 12:03, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Sun, Aug 22, 2021 at 8:30 PM Barry Scott <barry@barrys-emacs.org <mailto:barry@barrys-emacs.org>> wrote:
>>
>>
>>
>> On 22 Aug 2021, at 10:37, Chris Angelico <rosuav@gmail.com> wrote:
>>
>> When it comes to security, one thing I'm very curious about is why we
>> don't have any sort of certificate renewal verification. My browser
>> could retain the certificates of some web site (or of all web sites,
>> even - they're not THAT large), and if the site presents a different
>> cert, it could show the previously retained one and challenge the
>> server "prove that you're the same guy". This proof would consist of
>> the latest cert, signed by the older cert's key (or possibly a chain
>> that can construct such a proof, which would allow the server to
>> simply retain each new cert signed by the one previous cert, forming a
>> line - or a tree if necessary). My suspicion is that it'd add little
>> above simply having a valid cert, but if people are paranoid, surely
>> that's a better place to look?
>>
>>
>> The web site proves it owners the hostname and/or IP address using its certificate.
>> You use your trust store to show that you can trust that certificate.
>>
>> The fact that a certificate changes is not a reason to stop trusting a site.
>>
>> So it does not add anything.
>>
>> The pain point in PKI is revocation. The gold standard is for a web site to use OCSP stapling.
>> But that is rare sadly. And because of issues with revocation lists, (privacy, latency, need to
>> fail open on failiure, DoD vector, etc) this is where the paranoid should look.
>>
>
> Fair point. Let me give you a bit of context.
>
> Recently, the owner/operator of a site (I'll call it
> https://demo.example/ <https://demo.example/> ) died. Other people, who have been using the
> site extensively, wish for it to continue. If the domain registration
> expires, anyone can reregister it, and can then generate a completely
> new certificate for the common name "demo.example", and web browsers
> will accept that. The old cert may or may not have expired, but it
> won't be revoked.
>
> As far as I can tell, a web browser with default settings will happily
> accept the change of ownership. It won't care that the IP address,
> certificate, etc, have all changed. It just acknowledges that some CA
> has signed some certificate with the right common name. And therein is
> the vulnerability. (NOTE: I'm not saying that this is a real and
> practical vulnerability - this is theoretical only, and a focus for
> the paranoid.)
>
> This is true even if the old cert were one of those enhanced
> certificates that some CAs try to upsell you to ("Extended Validation"
> and friends). Even if, in the past, your bank was secured by one of
> those certs, your browser will still accept a perfectly standard cert
> next time. Which, in my opinion, renders those (quite pricey)
> certificates no more secure than something from Let's Encrypt that has
> no validation beyond ownership of DNS.
>
> Of course, you can pin a certificate. You can ask your browser to warn
> you if it's changed *at all*. But since certs expire, that's highly
> impractical, hence wondering why we don't have a system for using the
> old cert to prove ownership of the new one.
>
> So how is a web browser supposed to distinguish between (a) normal
> operation in which certs expire and are replaced, and (b) legit or
> non-legit ownership changes? (Of course the browser can't tell you
> whether the ownership change is legit, but out-of-band info can help
> with that.)
>
> Or does it really matter that little?

Only if this threat model matters to you or your organisation.
Personal its low down of the threats I watch out for.

The on-line world and the real-world are the same here.

If a business changes hands then do you trust the new owners?

Nothing we do with PKI certificates will answer that question.

For web sites that we care a lot about, like banks, we trust that
the site owners take care to protect that site. This includes making
sure that they do not lose control of its DNS name and certificates.

One of the biggest issues for PKI in recent years has been
Certificate Authorities (CA) that issued certificates for web sites without
checking that ownership. In these cases it lead to the browsers
removing those CA's from the trust stores and also putting extra
rules on all CA's to do a higher quality job.

Barry

>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list <https://mail.python.org/mailman/listinfo/python-list>

On Thu, Aug 26, 2021 at 12:16 AM Jon Ribbens via Python-list
<python-list@python.org> wrote:
>
> On 2021-08-25, Chris Angelico <rosuav@gmail.com> wrote:
> > On Wed, Aug 25, 2021 at 5:20 PM Barry Scott <barry@barrys-emacs.org> wrote:
> >> Only if this threat model matters to you or your organisation.
> >> Personal its low down of the threats I watch out for.
> >>
> >> The on-line world and the real-world are the same here.
> >>
> >> If a business changes hands then do you trust the new owners?
> >>
> >> Nothing we do with PKI certificates will answer that question.
> >
> > Fair enough; but a closer parallel would be walking up to a
> > previously-familiar street vendor and seeing a different person there.
> > Did the business change hands, or did some random dude hop over the
> > counter and pretend to be a new owner?
> >
> > But you're right, it's not usually a particularly high risk threat.
> > Still, it does further weaken the value of named SSL certificates and
> > certificate authorities; there's not actually that much difference if
> > the server just gave you a self-signed cert. In theory, the CA is
> > supposed to protect you against someone doing a DNS hack and
> > substituting a different server, in practice, anyone capable of doing
> > a large-scale DNS hack is probably capable of getting a very
> > legit-looking SSL cert for the name as well.
>
> There are so many trusted CAs these days that the chances of them all
> being secure approaches zero - they are not all equal yet they are all
> equally trusted. Which is why a change of CA on a site you have visited
> before is potentially suspicious.

Do any popular web browsers notify you if that happens? I've certainly
never noticed it with any that I use (and I've transitioned several
sites from one CA to another).

I've come to the conclusion that most security threats don't bother
most people, and that security *warnings* bother nearly everyone, so
real authentication of servers doesn't really matter all that much.
*Encryption* does still have value, but you'd get that with a
self-signed cert too.

ChrisA

On 2021-08-25, Chris Angelico <rosuav@gmail.com> wrote:
> On Thu, Aug 26, 2021 at 12:16 AM Jon Ribbens via Python-list
><python-list@python.org> wrote:
>> There are so many trusted CAs these days that the chances of them all
>> being secure approaches zero - they are not all equal yet they are all
>> equally trusted. Which is why a change of CA on a site you have visited
>> before is potentially suspicious.
>
> Do any popular web browsers notify you if that happens? I've certainly
> never noticed it with any that I use (and I've transitioned several
> sites from one CA to another).

There was, if the site was using "HTTP Public Key Pinning". But
that appears to have now been removed in favour of "Certificate
Transparency", which to me seems to be a system very much based
on the "problem: horse gone; solution: shut stable door" principle.

Another attempt at combatting this problem is DNS CAA records,
which are a way of politely asking all CAs in the world except the
ones you choose "please don't issue a certificate for my domain".
By definition someone who had hacked a CA would pay no attention
to that request, of course.

> I've come to the conclusion that most security threats don't bother
> most people, and that security *warnings* bother nearly everyone, so
> real authentication of servers doesn't really matter all that much.
> *Encryption* does still have value, but you'd get that with a
> self-signed cert too.

Encryption without knowing who you're encrypting *to* is worthless,
it's pretty much functionally equivalent to not encrypting.

On Thu, Aug 26, 2021 at 12:48 AM Jon Ribbens via Python-list
<python-list@python.org> wrote:
>
> On 2021-08-25, Chris Angelico <rosuav@gmail.com> wrote:
> > On Thu, Aug 26, 2021 at 12:16 AM Jon Ribbens via Python-list
> ><python-list@python.org> wrote:
> >> There are so many trusted CAs these days that the chances of them all
> >> being secure approaches zero - they are not all equal yet they are all
> >> equally trusted. Which is why a change of CA on a site you have visited
> >> before is potentially suspicious.
> >
> > Do any popular web browsers notify you if that happens? I've certainly
> > never noticed it with any that I use (and I've transitioned several
> > sites from one CA to another).
>
> There was, if the site was using "HTTP Public Key Pinning". But
> that appears to have now been removed in favour of "Certificate
> Transparency", which to me seems to be a system very much based
> on the "problem: horse gone; solution: shut stable door" principle.
>
> Another attempt at combatting this problem is DNS CAA records,
> which are a way of politely asking all CAs in the world except the
> ones you choose "please don't issue a certificate for my domain".
> By definition someone who had hacked a CA would pay no attention
> to that request, of course.

True, but that would still prevent legit CAs from unwittingly
contributing to an attack. But it still wouldn't help if someone can
do any sort of large-scale DNS attack, which is kinda essential for
most of this to matter anyway (it doesn't matter if an attacker has a
fake cert if all traffic goes to the legit site anyway).

> > I've come to the conclusion that most security threats don't bother
> > most people, and that security *warnings* bother nearly everyone, so
> > real authentication of servers doesn't really matter all that much.
> > *Encryption* does still have value, but you'd get that with a
> > self-signed cert too.
>
> Encryption without knowing who you're encrypting *to* is worthless,
> it's pretty much functionally equivalent to not encrypting.

Somewhat. It does prevent various forms of MitM attack. It's all about
adding extra difficulties on an attacker, so I wouldn't say
"worthless", just because it isn't 100% reliable.

Earlier I posited a hypothetical approach wherein the server would
sign a new cert using the old cert, and would then be able to present
that upon request. Are there any massive glaring problems with that?
(Actually, I'm pretty sure there will be. Lemme reword. What massive
glaring problems can you see with that?) It would require servers to
retain a chain of certificates, and to be able to provide that upon
request. It wouldn't even need a change to HTTP per se - could be
something like "https://your.host.example/cert_proof.txt" the same way
that robots.txt is done. In theory, that would allow a client to, at
the cost of retaining the one last-seen cert for each site, have
confidence that the site is the same one that was previously seen.

But, maybe we're just coming back to "it doesn't matter and nobody
really cares".

ChrisA

On 2021-08-25, Chris Angelico <rosuav@gmail.com> wrote:
> On Thu, Aug 26, 2021 at 12:48 AM Jon Ribbens via Python-list
><python-list@python.org> wrote:
>> Another attempt at combatting this problem is DNS CAA records,
>> which are a way of politely asking all CAs in the world except the
>> ones you choose "please don't issue a certificate for my domain".
>> By definition someone who had hacked a CA would pay no attention
>> to that request, of course.
>
> True, but that would still prevent legit CAs from unwittingly
> contributing to an attack. But it still wouldn't help if someone can
> do any sort of large-scale DNS attack, which is kinda essential for
> most of this to matter anyway (it doesn't matter if an attacker has a
> fake cert if all traffic goes to the legit site anyway).

That depends whether it's a large-scale attack or targeted at some
particular person or organisation, I suppose.

> Earlier I posited a hypothetical approach wherein the server would
> sign a new cert using the old cert, and would then be able to present
> that upon request. Are there any massive glaring problems with that?

That's a very similar idea to HTTP Public Key Pinning, and apparently
there were enough problems with that that they discontinued it.

> But, maybe we're just coming back to "it doesn't matter and nobody
> really cares".

People don't care until something goes wrong, and then suddenly they
care a great deal...

On 2021-08-22 19:37:24 +1000, Chris Angelico wrote:
> On Sun, Aug 22, 2021 at 6:45 PM Peter J. Holzer <hjp-python@hjp.at> wrote:
> >
> > On 2021-08-22 05:04:43 +1000, Chris Angelico wrote:
> > > On Sun, Aug 22, 2021 at 4:55 AM Martin Di Paola
> > > <martinp.dipaola@gmail.com> wrote:
> > > > HTTPS ensures encryption so the content, including the Basic Auth
> > > > username and password, is secret for any external observer.
> > > >
> > > > But it is *not* secret for the receiver (the server): if it was
> > > > compromised an adversary will have access to your password. It is much
> > > > easier to print a captured password than cracking the hashes.
> > > >
> > > > Other authentication mechanisms exist, like OAuth, which are more
> > > > "secure".
> >
> > OAuth is "an /authorization protocol/, rather than an /authentication
> > protocol/" [Wikipedia].
> >
> > > If your server is compromised in that way, *all is lost*.
> >
> > If "you" are the service provider, yes. but if "you" are the user, no.
>
> If "your server" is compromised, then you are the service provider,
> are you not? I'm not sure what "your server" would mean if "you" are
> the user.

The server you use. The same way you talk about "your bank" or "your
school" etc.

But that's besides the point. What I wanted to point out is that as a
service provider you take some responsibility for the user's data and
should take the user's POV into account.

> > there are three levels of security (from highest
> > to lowest) in this scenario:
> >
> > 1: The secret known to the user is never transmitted to the server at
> > all, the client only proves that the secret is known. This is the
> > case for TLS client authentication (which AFAIK all browsers support
> > but is a real pain in the ass to set up, so it's basically never
> > used) and for SCRAM (which isn't part of HTTP(S) but could be
> > implemented in JavaScript).
>
> This would be great, if nobody minded (a) setting up a unique client
> certificate for every site, or (b) allowing the ultimate in remote
> tracking cookie whereby any server could recognize you by your TLS
> certificate.

Which is one of reasons why TLS client authentication wasn't really
practical. It doesn't apply to SCRAM.

> > So given the choice between Basic Auth and Digest or NTLM (over HTTPS in
> > all cases) I would prefer Basic Auth. Ideally I would use SCRAM or a
> > public key method, but I admit that my security requirements were never
> > high enough to actually bother to do that (actually, I used SSL client
> > side auth once, 20 years ago, ...).
> >
>
> I would, of course, prefer something like form fill-out over Basic,
> but that's due to UI concerns rather than security ones.

Me too. But since that is irrelevant from a security POV I chose not to
complicate matters by bringing that up.

> When it comes to security, one thing I'm very curious about is why we
> don't have any sort of certificate renewal verification. My browser
> could retain the certificates of some web site (or of all web sites,
> even - they're not THAT large), and if the site presents a different
> cert, it could show the previously retained one and challenge the
> server "prove that you're the same guy". This proof would consist of
> the latest cert, signed by the older cert's key (or possibly a chain
> that can construct such a proof, which would allow the server to
> simply retain each new cert signed by the one previous cert, forming a
> line - or a tree if necessary).

I think this wasn't done because it violated the strictly hierarchical
model of X.509, where CAs are fully trusted and the server operators
aren't trusted at all. Letting them add signatures smacked too much of
an egalitarian web of trust.

Of course, technically, these days website certificates do form a web of
trust (with multiple chains and CA cross-signing), but the mental model
is still the same (and web site keys cannot be used for signing).

Apart from that I can see two problems: The first is that if the private
key is compromised, an attacker can use it to sign their own successor
key. The second is that if the private key is lost, a successor key
cannot be signed. So the absence or presence of such a signature is just
corroborating evidence, not proof.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

In comp.lang.python, Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
> Another attempt at combatting this problem is DNS CAA records,
> which are a way of politely asking all CAs in the world except the
> ones you choose "please don't issue a certificate for my domain".
> By definition someone who had hacked a CA would pay no attention
> to that request, of course.

Yeah, but it works for the case of forgotten hostnames, a rare but
real attack. Basically it works like this:

$COMPANY puts out a lot of things on different IP addresses from
a shared public(ish) pool like AWS and assigns different names
to them. Later $COMPANY discontinues one or more of those things,
terminates the host, and lets the IP address rejoin the public(ish)
pool.

$ATTACKER notices the domain name pointing to an unused IP address
and works to acquire it for their own server. $ATTACKER then gets
a cert for that domain, since they can easily prove ownership of
the server through http content challenges. $ATTACKER now has a
host in $COMPANY's name to launch phishing attacks.

This probably has some clever infosec name that I don't know.

Elijah
------
or a clever infosec name now forgotten

On 2021-08-25, Eli the Bearded <*@eli.users.panix.com> wrote:
> In comp.lang.python, Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
>> Another attempt at combatting this problem is DNS CAA records,
>> which are a way of politely asking all CAs in the world except the
>> ones you choose "please don't issue a certificate for my domain".
>> By definition someone who had hacked a CA would pay no attention
>> to that request, of course.
>
> Yeah, but it works for the case of forgotten hostnames, a rare but
> real attack. Basically it works like this:
>
> $COMPANY puts out a lot of things on different IP addresses from
> a shared public(ish) pool like AWS and assigns different names
> to them. Later $COMPANY discontinues one or more of those things,
> terminates the host, and lets the IP address rejoin the public(ish)
> pool.
>
> $ATTACKER notices the domain name pointing to an unused IP address
> and works to acquire it for their own server. $ATTACKER then gets
> a cert for that domain, since they can easily prove ownership of
> the server through http content challenges. $ATTACKER now has a
> host in $COMPANY's name to launch phishing attacks.

How does CAA help with this? Unless the domain owner knows in advance
that they're going to forget about the hostname and prepares for it
by setting a CAA record that denies all CAs, the attacker will simply
get a certificate from one of the permitted CAs - since, as you point
out, they genuinely own and control the relevant IP address.

In comp.lang.python, Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
> On 2021-08-25, Eli the Bearded <*@eli.users.panix.com> wrote:
>> $COMPANY puts out a lot of things on different IP addresses from
>> a shared public(ish) pool like AWS and assigns different names
>> to them. Later $COMPANY discontinues one or more of those things,
>> terminates the host, and lets the IP address rejoin the public(ish)
>> pool.
>>
>> $ATTACKER notices the domain name pointing to an unused IP address
>> and works to acquire it for their own server. $ATTACKER then gets
>> a cert for that domain, since they can easily prove ownership of
>> the server through http content challenges. $ATTACKER now has a
>> host in $COMPANY's name to launch phishing attacks.
> How does CAA help with this? Unless the domain owner knows in advance
> that they're going to forget about the hostname and prepares for it
> by setting a CAA record that denies all CAs, the attacker will simply
> get a certificate from one of the permitted CAs - since, as you point
> out, they genuinely own and control the relevant IP address.

I believe the way it helps is by limiting to a CA that will insist
all cert requests come through the right channel, not some random
one off somewhere. This doesn't prevent issues, but does raise the
complexity on an already niche attack.

It does aid in knocking out the easy random one-offs from Let's Encrypt.

Elijah
------
using LE for all his personal sites these days

> On 25 Aug 2021, at 20:34, Eli the Bearded <*@eli.users.panix.com> wrote:
>
> In comp.lang.python, Jon Ribbens <jon+usenet@unequivocal.eu> wrote:
>> Another attempt at combatting this problem is DNS CAA records,
>> which are a way of politely asking all CAs in the world except the
>> ones you choose "please don't issue a certificate for my domain".
>> By definition someone who had hacked a CA would pay no attention
>> to that request, of course.
>
> Yeah, but it works for the case of forgotten hostnames, a rare but
> real attack. Basically it works like this:
>
> $COMPANY puts out a lot of things on different IP addresses from
> a shared public(ish) pool like AWS and assigns different names
> to them. Later $COMPANY discontinues one or more of those things,
> terminates the host, and lets the IP address rejoin the public(ish)
> pool.
>
> $ATTACKER notices the domain name pointing to an unused IP address
> and works to acquire it for their own server. $ATTACKER then gets
> a cert for that domain, since they can easily prove ownership of
> the server through http content challenges. $ATTACKER now has a
> host in $COMPANY's name to launch phishing attacks.
>
> This probably has some clever infosec name that I don't know.

It is possible to sign an ip address in a certificate, but that is not often done.
Getting to reuse the IP address that example.com was using will not help
the attacker unless they can make a cert that signs the dns name.
And that means they hacked the CA which is a big problem.

Barry

>
> Elijah
> ------
> or a clever infosec name now forgotten
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

In comp.lang.python, Barry <barry@barrys-emacs.org> wrote:
> It is possible to sign an ip address in a certificate, but that is not
> often done.

It's bad practice. I've never seen one in the wild.

> Getting to reuse the IP address that example.com was using will not help
> the attacker unless they can make a cert that signs the dns name.
> And that means they hacked the CA which is a big problem.

You misunderstand the attack. Some web searching suggests the term is
"dangling DNS record".

Big co Acme Example, with example.com, has a website for the regular
public on www.example.com, gets mail at mail.example.com, serves
DNS from ns1., ns2. and ns3.example.com. The IT staff watch those
domaines very carefully.

One day marketing says, "We've got a big CES show this year, let's
make a website for the press at ces.example.com." They tell execs
the execs tell the IT guys the IT guys say "Okay, what does it point
to?" and Marketing gives them the IP address of the blog site they
just rented. IT sets up an A record. IT does not watch _that_
carefully. Two years later Marketing stops paying the bill on the
blog site, and ces.example.com has a "dangling" DNS record, it
exists but no longer points to a valid resource.

Attacker gets the IP address that points to (maybe they churn
through a bunch of temporary accounts until they do) and now with
the right IP to match ces.example.com they go off to get a SSL
cert for that.

$500 bug bounty write up here for someone who found a dangling
record, but didn't churn for the record to exploit it:

https://gist.github.com/TheBinitGhimire/9ebcd27086a11df1d7ec925e5f604e03

Another variant of this, which probably doesn't get you an SSL
cert, is a dangling CNAME. These can be easier to get. If
ces.example.com was a CNAME to cesdemosite2017.com then when
cesdemosite2017.com expires, it's trivial to re-register it and
squat "as" ces.example.com.

The most insidious version is a DNS delegation. If ces.example.com is an
NS record (unlikely for a marketing site, but plausible for some other
scenarios) and it goes to ns1.parternership.net, when parternership.net
expires the attacker can grab that, create a new ns1.parternership.net
and give themselves finan.ces.example.com then start spitting out bogus
bills with it.

The CAA record adds a smidgen more protection against those attacks.
(I don't think that's what it is designed for, but a good defense
works against more than just the original attack method.)

I also found this in my search, which is exactly the sort of threat
CAA was meant to handle:

https://en.wikipedia.org/wiki/Comodo_Cybersecurity#Dangling_markup_injection_vulnerability
On 25 July 2016, Matthew Bryant showed that Comodo's website is
vulnerable to dangling markup injection attacks and can send emails
to system administrators from Comodo's servers to approve a wildcard
certificate issue request which can be used to issue arbitrary
wildcard certificates via Comodo's 30-Day PositiveSSL product.

Bugs in automated systems that give out arbitrary certs are not
common, but very very nasty.

Elijah
------
DNS: the cause of, and solution to, all our Internet problems