novaBBS - comp.lang.python - Strange UnicodeEncodeError in Windows image on Azure DevOps and Github

Strange UnicodeEncodeError in Windows image on Azure DevOps and Github

<mailman.922.1668194743.20444.python-list@python.org>

https://www.novabbs.com/devel/article-flat.php?id=20285&group=comp.lang.python#20285

copy link Newsgroups: comp.lang.python

Path: i2pn2.org!i2pn.org!news.swapon.de!fu-berlin.de!uni-berlin.de!not-for-mail
From: 12jessic...@gmail.com (Jessica Smith)
Newsgroups: comp.lang.python
Subject: Strange UnicodeEncodeError in Windows image on Azure DevOps and Github
Date: Thu, 10 Nov 2022 18:14:15 -0600
Lines: 123
Message-ID: <mailman.922.1668194743.20444.python-list@python.org>
References: <CAHYhOdEWSXWwt174AoYwZN59R2dbirLAH=-_TkyUnyWE-XUYLg@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
reason="2048-bit key; unprotected key"
header.d=gmail.com header.i=@gmail.com header.b=J1/h1Wa5;
dkim-adsp=pass; dkim-atps=neutral
X-Spam-Status: OK 0.002
X-Spam-Evidence: '*H*': 1.00; '*S*': 0.00; 'url-ip:140.82/16': 0.03;
'3.7': 0.03; 'traceback': 0.04; '(most': 0.05; '2022': 0.05;
'last):': 0.05; 'variable': 0.05; 'subject:Windows': 0.07;
'subject:image': 0.07; 'utf-8': 0.07; '8bit%:41': 0.09; 'fails':
0.09; 'hosted': 0.09; 'page:': 0.09; 'help,': 0.14; 'url:github':
0.14; 'url-ip:140/8': 0.15; '"import': 0.16; '1252': 0.16; '3.9':
0.16; 'because,': 0.16; 'cleaned': 0.16; 'easiest': 0.16;
'encoding': 0.16; 'great.': 0.16; 'logs': 0.16;
'received:209.85.217': 0.16; 'researching': 0.16; 'runner': 0.16;
'shell:': 0.16; 'skip:" 70': 0.16; 'unicode': 0.16; 'url:imgur':
0.16; 'url:releases': 0.16; 'url:tag': 0.16; 'python': 0.16;
'github': 0.17; "can't": 0.17; 'to:addr:python-list': 0.20;
'issue': 0.21; "i've": 0.22; 'code': 0.23; 'python,': 0.25;
'tried': 0.26; "isn't": 0.27; 'ideas': 0.28; 'printed': 0.28;
'seem': 0.31; '"",': 0.32; 'actions': 0.32; 'validate': 0.32;
'message-id:@mail.gmail.com': 0.32; 'but': 0.32; "i'm": 0.33;
'there': 0.33; 'server': 0.33; 'windows': 0.34; 'same': 0.34;
'printing': 0.34; 'received:google.com': 0.34; 'running': 0.34;
'trying': 0.35; 'particularly': 0.35; 'skip:2 20': 0.35;
'from:addr:gmail.com': 0.35; 'processes': 0.36; 'really': 0.37;
'using': 0.37; 'received:209.85': 0.37; 'hard': 0.37; 'this.':
0.37; 'file': 0.38; 'thanks': 0.38; 'received:209': 0.39; 'rest':
0.39; 'hello,': 0.40; 'included': 0.61; 'stay': 0.61; 'skip:i 20':
0.62; 'here': 0.62; 'come': 0.62; 'email': 0.63; 'skip:b 10':
0.63; 'url:a': 0.64; 'upon': 0.64; 'well': 0.65; 'shows': 0.67;
'url:%0': 0.67; 'skip:# 10': 0.68; 'azure': 0.69; 'jessica': 0.69;
'screenshot': 0.69; 'waiting': 0.73; 'url:%2f': 0.73; 'skip:f 20':
0.75; 'requested': 0.76; 'stumbled': 0.76; 'position': 0.81;
'url:click': 0.83; 'happens': 0.84; 'url-ip:199/8': 0.84;
'".': 0.84; 'characters': 0.84; 'hair': 0.84; 'job...': 0.84;
'mapped': 0.84; 'maps': 0.84; 'weird': 0.84
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112;
h=to:subject:message-id:date:from:mime-version:from:to:cc:subject
:date:message-id:reply-to;
bh=wsvwCz/+UVeutTNhGvYwP6jPr11Sco4xhIzsWePGqsE=;
b=J1/h1Wa5YeSRc6/dy7K1lz2XXjK+MXMfZ8fH+UGlWxlPqamaKEfz20kyFjed81u4wU
/zeAZkCvmMXtojP4/Ul50mlRguy8QgcpLe+3VXCPT6Y/gr15k4T9RyU+lFwAbN74kPzk
aXls3W1zJ4t+07F97jr4xSA1QQXlpGjwYNJxL/rQDDKhMgBle/8WIPqvZ9w6XQx8Ro6/
ydYO6meWf5ueftnsD9qV9KAm0YZO6Ksu1Hw+4XzLGMA1ExdQpLDB0bYT/rHGQh4TQG/G
xwkFbysdVYf8fZ+U0Fhbgtt7eIAvhx3J2r7lgFAyteeWmh1LUFahuvGUXI2+oiLFXIky
KVIw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20210112;
h=to:subject:message-id:date:from:mime-version:x-gm-message-state
:from:to:cc:subject:date:message-id:reply-to;
bh=wsvwCz/+UVeutTNhGvYwP6jPr11Sco4xhIzsWePGqsE=;
b=voRNTXa+JoMZuxudHD0mzWIgB7ah7Dce+t6oPuxEZ1QUwWdTceaeWqxfWSYVjooNUp
s7j2bstayC/ctDZh6X1cCBCu54wE+hvHaaToYRScXRuLEKfTh1rW7m7sp/39hfkyc9YT
EuQTrvHsFDq13PNbKR2uqzZgOIgBmYmdsm4JLe7u6fM6EoFq3ZHpzDxjFC4CNiQyCv0W
xnLJSNmue4OR4XgcppnE3SPj6IL92AX1ZUqsV+TSDN32MpFO08HyIkpYcN171EWyWOXt
Hr3QRgDGMQO7LhKi2SDimSBoHkt/KQsMSb/RIqPiW1pE2MbL+EhABdb7jMLz2FcMrsqL
puWQ==
X-Gm-Message-State: ACrzQf2ENk/ejxn2/fOonsnBAbc0Q0xYDmD8khPOQmoCP5iBmTRuPOey
2CnOULASL0D2AvDLnb3Gaa4ZDnKSp48ZJHmB9LRafP5nEaxDsg==
X-Google-Smtp-Source: AMsMyM6U7sKtinqlvKVw0Oe2EHPdkGTOQpK6J2+MQ2flWUDU+/DTbRXe7coIeUZhF4u12cq1uH6i531SS0uuJjTgq0k=
X-Received: by 2002:a67:f985:0:b0:3a6:8ef4:e285 with SMTP id
b5-20020a67f985000000b003a68ef4e285mr4201564vsq.20.1668125665501; Thu, 10 Nov
2022 16:14:25 -0800 (PST)
X-Mailman-Approved-At: Fri, 11 Nov 2022 14:25:41 -0500
X-Content-Filtered-By: Mailman/MimeDel 2.1.39
X-BeenThere: python-list@python.org
X-Mailman-Version: 2.1.39
Precedence: list
List-Id: General discussion list for the Python programming language
<python-list.python.org>
List-Unsubscribe: <https://mail.python.org/mailman/options/python-list>,
<mailto:python-list-request@python.org?subject=unsubscribe>
List-Archive: <https://mail.python.org/pipermail/python-list/>
List-Post: <mailto:python-list@python.org>
List-Help: <mailto:python-list-request@python.org?subject=help>
List-Subscribe: <https://mail.python.org/mailman/listinfo/python-list>,
<mailto:python-list-request@python.org?subject=subscribe>
X-Mailman-Original-Message-ID: <CAHYhOdEWSXWwt174AoYwZN59R2dbirLAH=-_TkyUnyWE-XUYLg@mail.gmail.com>

by: Jessica Smith - Fri, 11 Nov 2022 00:14 UTC

Hello,

Weird issue I've found on Windows images in Azure Devops Pipelines and
Github actions. Printing Unicode characters fails on these images because,
for some reason, the encoding is mapped to cp1252. What is particularly
weird about the code page being set to 1252 is that if you execute "chcp"
it shows that the code page is 65001.

At the end of this email are the cleaned up logs from GH actions. The
actions are very simple - print out unicode characters using echo to prove
the characters can be printed to the console. The rest of the commands are
in Python, and they include printing out the "encoding" variable of
sys.stdout, as well as printing sys.flags and sys.getfilesystemencoding.
Then print the same unicode character using print, which causes a
UnicodEncodeError because the character isn't in the cp1252 charmap.

I've also uploaded the logs to pastebin here: https://pastebin.com/ExzGRHav
I also uploaded a screenshot to imgur, since the logs are not the easiest
to read. https://imgur.com/a/dhvLWOJ

I'm trying to determine why this issue only happens on these images - I can
replicate it on multiple versions of Python (from 3.9 to 3.7 at least,
haven't tried more), but I can't replicate this on my own machines.

There are a few issues on GH regarding this issue but they seem to stay
open since they are hard to replicate. Here are the ones I have stumbled
upon while researching this.

https://github.com/databrickslabs/dbx/issues/455
https://github.com/PrefectHQ/prefect/issues/5754
https://github.com/pallets/click/issues/2121

Any insight or ideas on how to test and validate the cause would be great.
I'm pulling my hair out trying to find the root cause of this - not because
it really matters to any of my processes but because it is weird and broken..

Thanks for any help,

Jessica

Begin Logs:

2022-11-10T23:54:51.7272453Z Requested labels: windows-latest
2022-11-10T23:54:51.7272494Z Job defined at:
NodeJSmith/wsl_home/.github/workflows/blank.yml@refs/heads/main
2022-11-10T23:54:51.7272514Z Waiting for a runner to pick up this job...
2022-11-10T23:54:52.3387510Z Job is waiting for a hosted runner to come
online.
2022-11-10T23:55:04.8574435Z Job is about to start running on the hosted
runner: Hosted Agent (hosted)
2022-11-10T23:55:15.8332600Z Current runner version: '2.298.2'

2022-11-10T23:55:15.8366947Z ##[group]Operating System
2022-11-10T23:55:15.8367650Z Microsoft Windows Server 2022
2022-11-10T23:55:15.8367954Z 10.0.20348
2022-11-10T23:55:15.8368389Z Datacenter
2022-11-10T23:55:15.8368696Z ##[endgroup]

2022-11-10T23:55:15.8369023Z ##[group]Runner Image
2022-11-10T23:55:15.8369654Z Image: windows-2022
2022-11-10T23:55:15.8369931Z Version: 20221027.1
2022-11-10T23:55:15.8370539Z Included Software:
https://github.com/actions/runner-images/blob/win22/20221027.1/images/win/Windows2022-Readme.md
2022-11-10T23:55:15.8371174Z Image Release:
https://github.com/actions/runner-images/releases/tag/win22%2F20221027.1
2022-11-10T23:55:15.8371622Z ##[endgroup]

2022-11-10T23:55:15.8371955Z ##[group]Runner Image Provisioner
2022-11-10T23:55:15.8372277Z 2.0.91.1
2022-11-10T23:55:15.8372514Z ##[endgroup]

2022-11-10T23:55:16.3619998Z ##[group]Run echo " └── ID:"
2022-11-10T23:55:16.3620626Z echo " └── ID:"
2022-11-10T23:55:16.3927292Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:16.3927894Z ##[endgroup]
2022-11-10T23:55:32.9958751Z └── ID:

2022-11-10T23:55:34.0835652Z ##[group]Run chcp
2022-11-10T23:55:34.0836104Z chcp
2022-11-10T23:55:34.0878901Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:34.0879350Z ##[endgroup]
2022-11-10T23:55:34.4878247Z Active code page: 65001

2022-11-10T23:55:34.7917219Z ##[group]Run python -c "import sys;
print('sys.stdout.encoding', sys.stdout.encoding);
print('sys.flags',sys.flags);print('sys.getfilesystemencoding',sys.getfilesystemencoding())"
2022-11-10T23:55:34.7918148Z python -c "import sys;
print('sys.stdout.encoding', sys.stdout.encoding);
print('sys.flags',sys.flags);print('sys.getfilesystemencoding',sys.getfilesystemencoding())"
2022-11-10T23:55:34.7960873Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:34.7961202Z ##[endgroup]
2022-11-10T23:55:36.2324642Z sys.stdout.encoding cp1252
2022-11-10T23:55:36.2325910Z sys.flags sys.flags(debug=0, inspect=0,
interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0,
no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0,
hash_randomization=1, isolated=0, dev_mode=False, utf8_mode=0)
2022-11-10T23:55:36.2327055Z sys.getfilesystemencoding utf-8

2022-11-10T23:55:36.4553957Z ##[group]Run python -c "print('└── ID:')"
2022-11-10T23:55:36.4554395Z python -c "print('└── ID:')"
2022-11-10T23:55:36.4595413Z shell: C:\Program Files\PowerShell\7\pwsh.EXE
-command ". '{0}'"
2022-11-10T23:55:36.4595740Z ##[endgroup]
2022-11-10T23:55:36.8739309Z Traceback (most recent call last):
2022-11-10T23:55:37.1316425Z File "<string>", line 1, in <module>
2022-11-10T23:55:37.1317452Z File
"C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\encodings\cp1252.py",
line 19, in encode
2022-11-10T23:55:37.1324632Z return
codecs.charmap_encode(input,self.errors,encoding_table)[0]
2022-11-10T23:55:37.1325913Z UnicodeEncodeError: 'charmap' codec can't
encode characters in position 0-2: character maps to <undefined>

Swap read error. You lose your mind.

devel / comp.lang.python / Strange UnicodeEncodeError in Windows image on Azure DevOps and Github

Subject	Author
Strange UnicodeEncodeError in Windows image on Azure DevOps and Github	Jessica Smith