On Sat, 7 May 2022 at 19:02, MRAB <python@mrabarnett.plus.com> wrote:
>
> On 2022-05-07 17:28, Marco Sulla wrote:
> > On Sat, 7 May 2022 at 16:08, Barry <barry@barrys-emacs.org> wrote:
> >> You need to handle the file in bin mode and do the handling of line endings and encodings yourself. It’s not that hard for the cases you wanted.
> >
> >>>> "\n".encode("utf-16")
> > b'\xff\xfe\n\x00'
> >>>> "".encode("utf-16")
> > b'\xff\xfe'
> >>>> "a\nb".encode("utf-16")
> > b'\xff\xfea\x00\n\x00b\x00'
> >>>> "\n".encode("utf-16").lstrip("".encode("utf-16"))
> > b'\n\x00'
> >
> > Can I use the last trick to get the encoding of a LF or a CR in any encoding?
>
> In the case of UTF-16, it's 2 bytes per code unit, but those 2 bytes
> could be little-endian or big-endian.
>
> As you didn't specify which you wanted, it defaulted to little-endian
> and added a BOM (U+FEFF).
>
> If you specify which endianness you want with "utf-16le" or "utf-16be",
> it won't add the BOM:
>
> >>> # Little-endian.
> >>> "\n".encode("utf-16le")
> b'\n\x00'
> >>> # Big-endian.
> >>> "\n".encode("utf-16be")
> b'\x00\n'

Well, ok, but I need a generic method to get LF and CR for any
encoding an user can input.
Do you think that

"\n".encode(encoding).lstrip("".encode(encoding))

is good for any encoding? Furthermore, is there a way to get the
encoding of an opened file object?

Marco Sulla <Marco.Sulla.Python@gmail.com> writes:
>Well, ok, but I need a generic method to get LF and CR for any
>encoding an user can input.

"LF" and "CR" come from US-ASCII. It is theoretically
possible that there might be some encodings out there
(not for Unicode) that are not based on US-ASCII and
have no LF or no CR.

>is good for any encoding? Furthermore, is there a way to get the
>encoding of an opened file object?

I have written a function that might be able to detect one
of few encodings based on a heuristic algorithm.

def encoding( name ):
path = pathlib.Path( name )
for encoding in( "utf_8", "latin_1", "cp1252" ):
try:
with path.open( encoding=encoding, errors="strict" )as file:
text = file.read()
return encoding
except UnicodeDecodeError:
pass
return "ascii"

Yes, it's potentially slow and might be wrong.
The result "ascii" might mean it's a binary file.

On Sat, 7 May 2022 20:35:34 +0200, Marco Sulla
<Marco.Sulla.Python@gmail.com> declaimed the following:

>Well, ok, but I need a generic method to get LF and CR for any
>encoding an user can input.

Other than EBCDIC, <lf> and <cr> AS BYTES should appear as x0A and x0D
in any of the 8-bit encodings (ASCII, ISO-8859-x, CPxxxx, UTF-8). I believe
those bytes also appear in UTF-16 -- BUT, they will have a null (x00) byte
associated with them as padding; as a result, you can not search for just
x0Dx0A (Windows line end convention -- they may be x00x0Dx00x0A or
x0Dx00x0Ax00 depending on endianness cf:
https://docs.microsoft.com/en-us/cpp/text/support-for-unicode?view=msvc-170
)

For EBCDIC <cr> is still x0D, but <lf> is x25 (and there is a separate
<nl> [new line] at x15)

--
Wulfraed Dennis Lee Bieber AF6VN
wlfraed@ix.netcom.com http://wlfraed.microdiversity.freeddns.org/

On 2022-05-07 19:47, Stefan Ram wrote:
> Marco Sulla <Marco.Sulla.Python@gmail.com> writes:
>>Well, ok, but I need a generic method to get LF and CR for any
>>encoding an user can input.
>
> "LF" and "CR" come from US-ASCII. It is theoretically
> possible that there might be some encodings out there
> (not for Unicode) that are not based on US-ASCII and
> have no LF or no CR.
>
>>is good for any encoding? Furthermore, is there a way to get the
>>encoding of an opened file object?
>
> I have written a function that might be able to detect one
> of few encodings based on a heuristic algorithm.
>
> def encoding( name ):
> path = pathlib.Path( name )
> for encoding in( "utf_8", "latin_1", "cp1252" ):
> try:
> with path.open( encoding=encoding, errors="strict" )as file:
> text = file.read()
> return encoding
> except UnicodeDecodeError:
> pass
> return "ascii"
>
> Yes, it's potentially slow and might be wrong.
> The result "ascii" might mean it's a binary file.
>
"latin-1" will decode any sequence of bytes, so it'll never try
"cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
anyway because the file could contain 0x80..0xFF, which aren't supported
by that encoding.

MRAB <python@mrabarnett.plus.com> writes:
>On 2022-05-07 19:47, Stefan Ram wrote:
....
>>def encoding( name ):
>> path = pathlib.Path( name )
>> for encoding in( "utf_8", "latin_1", "cp1252" ):
>> try:
>> with path.open( encoding=encoding, errors="strict" )as file:
>> text = file.read()
>> return encoding
>> except UnicodeDecodeError:
>> pass
>> return "ascii"
>>Yes, it's potentially slow and might be wrong.
>>The result "ascii" might mean it's a binary file.
>"latin-1" will decode any sequence of bytes, so it'll never try
>"cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
>anyway because the file could contain 0x80..0xFF, which aren't supported
>by that encoding.

Thank you! It's working for my specific application where
I'm reading from a collection of text files that should be
encoded in either utf_8, latin_1, or ascii.

On Sun, 8 May 2022 at 07:19, Stefan Ram <ram@zedat.fu-berlin.de> wrote:
>
> MRAB <python@mrabarnett.plus.com> writes:
> >On 2022-05-07 19:47, Stefan Ram wrote:
> ...
> >>def encoding( name ):
> >> path = pathlib.Path( name )
> >> for encoding in( "utf_8", "latin_1", "cp1252" ):
> >> try:
> >> with path.open( encoding=encoding, errors="strict" )as file:
> >> text = file.read()
> >> return encoding
> >> except UnicodeDecodeError:
> >> pass
> >> return "ascii"
> >>Yes, it's potentially slow and might be wrong.
> >>The result "ascii" might mean it's a binary file.
> >"latin-1" will decode any sequence of bytes, so it'll never try
> >"cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
> >anyway because the file could contain 0x80..0xFF, which aren't supported
> >by that encoding.
>
> Thank you! It's working for my specific application where
> I'm reading from a collection of text files that should be
> encoded in either utf_8, latin_1, or ascii.
>

In that case, I'd exclude ASCII from the check, and just check UTF-8,
and if that fails, decode as Latin-1. Any ASCII files will decode
correctly as UTF-8, and any file will decode as Latin-1.

I've used this exact fallback system when decoding raw data from
Unicode-naive servers - they accept and share bytes, so it's entirely
possible to have a mix of encodings in a single stream. As long as you
can define the span of a single "unit" (say, a line, or a chunk in
some form), you can read as bytes and do the exact same "decode as
UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not
perfectly ideal, but it's about as good as you'll get with a lot of
US-based servers. (Depending on context, you might use CP-1252 instead
of Latin-1, but you might need errors="replace" there, since
Windows-1252 has some undefined byte values.)

ChrisA

> On 7 May 2022, at 22:31, Chris Angelico <rosuav@gmail.com> wrote:
>
> On Sun, 8 May 2022 at 07:19, Stefan Ram <ram@zedat.fu-berlin.de> wrote:
>>
>> MRAB <python@mrabarnett.plus.com> writes:
>>> On 2022-05-07 19:47, Stefan Ram wrote:
>> ...
>>>> def encoding( name ):
>>>> path = pathlib.Path( name )
>>>> for encoding in( "utf_8", "latin_1", "cp1252" ):
>>>> try:
>>>> with path.open( encoding=encoding, errors="strict" )as file:
>>>> text = file.read()
>>>> return encoding
>>>> except UnicodeDecodeError:
>>>> pass
>>>> return "ascii"
>>>> Yes, it's potentially slow and might be wrong.
>>>> The result "ascii" might mean it's a binary file.
>>> "latin-1" will decode any sequence of bytes, so it'll never try
>>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
>>> anyway because the file could contain 0x80..0xFF, which aren't supported
>>> by that encoding.
>>
>> Thank you! It's working for my specific application where
>> I'm reading from a collection of text files that should be
>> encoded in either utf_8, latin_1, or ascii.
>>
>
> In that case, I'd exclude ASCII from the check, and just check UTF-8,
> and if that fails, decode as Latin-1. Any ASCII files will decode
> correctly as UTF-8, and any file will decode as Latin-1.
>
> I've used this exact fallback system when decoding raw data from
> Unicode-naive servers - they accept and share bytes, so it's entirely
> possible to have a mix of encodings in a single stream. As long as you
> can define the span of a single "unit" (say, a line, or a chunk in
> some form), you can read as bytes and do the exact same "decode as
> UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not
> perfectly ideal, but it's about as good as you'll get with a lot of
> US-based servers. (Depending on context, you might use CP-1252 instead
> of Latin-1, but you might need errors="replace" there, since
> Windows-1252 has some undefined byte values.)

There is a very common error on Windows that files and especially web pages that
claim to be utf-8 are in fact CP-1252.

There is logic in the HTML standards to try utf-8 and if it fails fall back to CP-1252.

Its usually the left and "smart" quote chars that cause the issue as they code
as an invalid utf-8.

Barry

>
> ChrisA
> --
> https://mail.python.org/mailman/listinfo/python-list
>

On Mon, 9 May 2022 at 04:15, Barry Scott <barry@barrys-emacs.org> wrote:
>
>
>
> > On 7 May 2022, at 22:31, Chris Angelico <rosuav@gmail.com> wrote:
> >
> > On Sun, 8 May 2022 at 07:19, Stefan Ram <ram@zedat.fu-berlin.de> wrote:
> >>
> >> MRAB <python@mrabarnett.plus.com> writes:
> >>> On 2022-05-07 19:47, Stefan Ram wrote:
> >> ...
> >>>> def encoding( name ):
> >>>> path = pathlib.Path( name )
> >>>> for encoding in( "utf_8", "latin_1", "cp1252" ):
> >>>> try:
> >>>> with path.open( encoding=encoding, errors="strict" )as file:
> >>>> text = file.read()
> >>>> return encoding
> >>>> except UnicodeDecodeError:
> >>>> pass
> >>>> return "ascii"
> >>>> Yes, it's potentially slow and might be wrong.
> >>>> The result "ascii" might mean it's a binary file.
> >>> "latin-1" will decode any sequence of bytes, so it'll never try
> >>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
> >>> anyway because the file could contain 0x80..0xFF, which aren't supported
> >>> by that encoding.
> >>
> >> Thank you! It's working for my specific application where
> >> I'm reading from a collection of text files that should be
> >> encoded in either utf_8, latin_1, or ascii.
> >>
> >
> > In that case, I'd exclude ASCII from the check, and just check UTF-8,
> > and if that fails, decode as Latin-1. Any ASCII files will decode
> > correctly as UTF-8, and any file will decode as Latin-1.
> >
> > I've used this exact fallback system when decoding raw data from
> > Unicode-naive servers - they accept and share bytes, so it's entirely
> > possible to have a mix of encodings in a single stream. As long as you
> > can define the span of a single "unit" (say, a line, or a chunk in
> > some form), you can read as bytes and do the exact same "decode as
> > UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not
> > perfectly ideal, but it's about as good as you'll get with a lot of
> > US-based servers. (Depending on context, you might use CP-1252 instead
> > of Latin-1, but you might need errors="replace" there, since
> > Windows-1252 has some undefined byte values.)
>
> There is a very common error on Windows that files and especially web pages that
> claim to be utf-8 are in fact CP-1252.
>
> There is logic in the HTML standards to try utf-8 and if it fails fall back to CP-1252.
>
> Its usually the left and "smart" quote chars that cause the issue as they code
> as an invalid utf-8.
>

Yeah, or sometimes, there isn't *anything* in UTF-8, and it has some
sort of straight-up lie in the form of a meta tag. It's annoying. But
the same logic still applies: attempt one decode (UTF-8) and if it
fails, there's one fallback. Fairly simple.

ChrisA

On 2022-05-08 19:15, Barry Scott wrote:
>
>
>> On 7 May 2022, at 22:31, Chris Angelico <rosuav@gmail.com> wrote:
>>
>> On Sun, 8 May 2022 at 07:19, Stefan Ram <ram@zedat.fu-berlin.de> wrote:
>>>
>>> MRAB <python@mrabarnett.plus.com> writes:
>>>> On 2022-05-07 19:47, Stefan Ram wrote:
>>> ...
>>>>> def encoding( name ):
>>>>> path = pathlib.Path( name )
>>>>> for encoding in( "utf_8", "latin_1", "cp1252" ):
>>>>> try:
>>>>> with path.open( encoding=encoding, errors="strict" )as file:
>>>>> text = file.read()
>>>>> return encoding
>>>>> except UnicodeDecodeError:
>>>>> pass
>>>>> return "ascii"
>>>>> Yes, it's potentially slow and might be wrong.
>>>>> The result "ascii" might mean it's a binary file.
>>>> "latin-1" will decode any sequence of bytes, so it'll never try
>>>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
>>>> anyway because the file could contain 0x80..0xFF, which aren't supported
>>>> by that encoding.
>>>
>>> Thank you! It's working for my specific application where
>>> I'm reading from a collection of text files that should be
>>> encoded in either utf_8, latin_1, or ascii.
>>>
>>
>> In that case, I'd exclude ASCII from the check, and just check UTF-8,
>> and if that fails, decode as Latin-1. Any ASCII files will decode
>> correctly as UTF-8, and any file will decode as Latin-1.
>>
>> I've used this exact fallback system when decoding raw data from
>> Unicode-naive servers - they accept and share bytes, so it's entirely
>> possible to have a mix of encodings in a single stream. As long as you
>> can define the span of a single "unit" (say, a line, or a chunk in
>> some form), you can read as bytes and do the exact same "decode as
>> UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not
>> perfectly ideal, but it's about as good as you'll get with a lot of
>> US-based servers. (Depending on context, you might use CP-1252 instead
>> of Latin-1, but you might need errors="replace" there, since
>> Windows-1252 has some undefined byte values.)
>
> There is a very common error on Windows that files and especially web pages that
> claim to be utf-8 are in fact CP-1252.
>
> There is logic in the HTML standards to try utf-8 and if it fails fall back to CP-1252.
>
> Its usually the left and "smart" quote chars that cause the issue as they code
> as an invalid utf-8.
>
Is it CP-1252 or ISO-8859-1 (Latin-1)?

Le dimanche 8 mai 2022 à 20:50:50 UTC+2, MRAB a écrit :
> On 2022-05-08 19:15, Barry Scott wrote:
> >
> >
> >> On 7 May 2022, at 22:31, Chris Angelico <ros...@gmail.com> wrote:
> >>
> >> On Sun, 8 May 2022 at 07:19, Stefan Ram <r...@zedat.fu-berlin.de> wrote:
> >>>
> >>> MRAB <pyt...@mrabarnett.plus.com> writes:
> >>>> On 2022-05-07 19:47, Stefan Ram wrote:
> >>> ...
> >>>>> def encoding( name ):
> >>>>> path = pathlib.Path( name )
> >>>>> for encoding in( "utf_8", "latin_1", "cp1252" ):
> >>>>> try:
> >>>>> with path.open( encoding=encoding, errors="strict" )as file:
> >>>>> text = file.read()
> >>>>> return encoding
> >>>>> except UnicodeDecodeError:
> >>>>> pass
> >>>>> return "ascii"
> >>>>> Yes, it's potentially slow and might be wrong.
> >>>>> The result "ascii" might mean it's a binary file.
> >>>> "latin-1" will decode any sequence of bytes, so it'll never try
> >>>> "cp1252", nor fall back to "ascii", and falling back to "ascii" is wrong
> >>>> anyway because the file could contain 0x80..0xFF, which aren't supported
> >>>> by that encoding.
> >>>
> >>> Thank you! It's working for my specific application where
> >>> I'm reading from a collection of text files that should be
> >>> encoded in either utf_8, latin_1, or ascii.
> >>>
> >>
> >> In that case, I'd exclude ASCII from the check, and just check UTF-8,
> >> and if that fails, decode as Latin-1. Any ASCII files will decode
> >> correctly as UTF-8, and any file will decode as Latin-1.
> >>
> >> I've used this exact fallback system when decoding raw data from
> >> Unicode-naive servers - they accept and share bytes, so it's entirely
> >> possible to have a mix of encodings in a single stream. As long as you
> >> can define the span of a single "unit" (say, a line, or a chunk in
> >> some form), you can read as bytes and do the exact same "decode as
> >> UTF-8 if possible, otherwise decode as Latin-1" dance. Sure, it's not
> >> perfectly ideal, but it's about as good as you'll get with a lot of
> >> US-based servers. (Depending on context, you might use CP-1252 instead
> >> of Latin-1, but you might need errors="replace" there, since
> >> Windows-1252 has some undefined byte values.)
> >
> > There is a very common error on Windows that files and especially web pages that
> > claim to be utf-8 are in fact CP-1252.
> >
> > There is logic in the HTML standards to try utf-8 and if it fails fall back to CP-1252.
> >
> > Its usually the left and "smart" quote chars that cause the issue as they code
> > as an invalid utf-8.
> >
> Is it CP-1252 or ISO-8859-1 (Latin-1)?

In good software, latin-1 / ISO-8859-1 does not exist.
This also the case for (and in) unicode.