Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

python build_tools/update_deps.py fails when the default locale is cp932 and dotnet tool shows messages in Japanese #1203

Open
yukawa opened this issue Mar 3, 2025 · 0 comments · May be fixed by #1204
Assignees

Comments

@yukawa
Copy link
Collaborator

yukawa commented Mar 3, 2025

Description

python build_tools/update_deps.py fails when the default locale is cp932 and dotnet tool shows messages in Japanese.

Rrerequisite

  • dotnet tool --help shows messages in Japanese
  • python -c "import locale; print(locale.getpreferredencoding())" shows cp932

Steps to reproduce

Steps to reproduce the behavior:

  1. git clone https://github.com/google/mozc.git
  2. cd mozc
  3. python build_tools/update_deps.py

Expected behavior

At the step 3, the command succeeds.

Actual behavior

At the step 3, the command fails with the following error.

PS D:\mozc\src> python build_tools/update_deps.py
Traceback (most recent call last):
  File "D:\mozc\src\build_tools\update_deps.py", line 499, in <module>
    main()
    ~~~~^^
  File "D:\mozc\src\build_tools\update_deps.py", line 485, in main
    restore_dotnet_tools(args.dryrun)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "D:\mozc\src\build_tools\update_deps.py", line 443, in restore_dotnet_tools
    exec_command(args, cwd=ABS_MOZC_SRC_DIR)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\mozc\src\build_tools\update_deps.py", line 427, in exec_command
    _, _ = process.communicate()
           ~~~~~~~~~~~~~~~~~~~^^
  File "C:\Users\testuser\AppData\Local\Programs\Python\Python313\Lib\subprocess.py", line 1206, in communicate
    stdout = self.stdout.read()
UnicodeDecodeError: 'cp932' codec can't decode byte 0x84 in position 2: illegal multibyte sequence

Screenshots

Image Image

Version or commit-id

14afac9

Environment

  • OS: Windows 24H2

Additional context

We can fix the issue by adding encoding='utf-8' here.

process = subprocess.Popen(
args,
cwd=cwd,
shell=False,
stdout=subprocess.PIPE,
text=True,
)

@yukawa yukawa self-assigned this Mar 6, 2025
yukawa added a commit to yukawa/mozc that referenced this issue Mar 6, 2025
If 'dotnet tool' outputs non-ASCII characters and the system locale is
not 'utf-8', then 'update_deps.py' can fail due to UnicodeDecodeError.
This commit addresses the above issue by explicitly setting 'utf-8' when
calling subprocess.Popen.

Closes google#1203
@yukawa yukawa linked a pull request Mar 6, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant