Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added embedding default fonts in the theme wheel #232

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .readthedocs.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,6 @@ sphinx:
# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/requirements.txt
- method: setuptools
path: .
- requirements: docs/requirements.txt
2 changes: 2 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
sphinx_immaterial[json,clang-format,keys,cpp]
sphinxcontrib-details-directive
sphinx-jinja
pydantic
importlib_resources
1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,4 @@ pydantic
typing-extensions
appdirs
requests
importlib-resources
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like there's a importlib.resources module that has been available as part of the std libs since python v3.7. Since python 3.6 has reached end-of-life and this theme requires v3.8+, does this mean we don't need a new external dependency? Is there some advantage to using the thrid-party importlib_resources lib?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After some more thought, I don't think the cached default fonts needs to be an importable resource. We could just use

# from google_fonts.py or external_resource_cache.py
pathlib.PurePath(__file__).parent / f"{default_font_location}" / f"{default_font_name}"

The importlib.resources module seems designed to let external API easily find a non-python asset from our pkg.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we should not add an extra dependency --- we can just use importlib.resources that is built in. Basically importlib.resources is primarily useful to support cases where the package is stored not as plain files, e.g. in a zip file. In that case it just works transparently, while using __file__ will not. However, we already locate some files using __file__ so I think it is fine to just do that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

importlib-resources is added due to the fact that Python versions below 3.9 do not have files in importlib.resources. But sure, I can do a workaround that will not use importlib-resources dependency

13 changes: 13 additions & 0 deletions setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,12 @@
import setuptools.command.develop
import setuptools.command.install
import setuptools.command.sdist
from sphinx_immaterial.google_fonts import install_google_fonts

from importlib_resources import files
from sphinx_immaterial import resources

from sphinx_immaterial import DEFAULT_THEME_OPTIONS
Comment on lines +33 to +38
Copy link
Collaborator

@2bndy5 2bndy5 Mar 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is causing a "chicken before the egg" problem because you are trying to import from the very pkg that this installs. The CI seems fine because the pkg deps are installed before the pkg is built. However, when I run pip install . in a fresh env, I get ModuleNotFoundError.

full traceback
Traceback (most recent call last):
  File "path\to\repo\.venv-temp\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
    main()
  File "path\to\repo\.venv-temp\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\repo\.venv-temp\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
           ^^^^^^^^^^^^^^^^^^^^^
  File "path\to\temp\pip-build-env-415k6g8g\overlay\Lib\site-packages\setuptools\build_meta.py", line 338, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\temp\pip-build-env-415k6g8g\overlay\Lib\site-packages\setuptools\build_meta.py", line 320, in _get_build_requires
    self.run_setup()
  File "path\to\temp\pip-build-env-415k6g8g\overlay\Lib\site-packages\setuptools\build_meta.py", line 485, in run_setup
    self).run_setup(setup_script=setup_script)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "path\to\temp\pip-build-env-415k6g8g\overlay\Lib\site-packages\setuptools\build_meta.py", line 335, in run_setup
    exec(code, locals())
  File "<string>", line 33, in <module>
  File "path\to\repo\sphinx-immaterial\sphinx_immaterial\__init__.py", line 6, in <module>
    import docutils.nodes
ModuleNotFoundError: No module named 'docutils'

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid this we could set a special environment variable that causes sphinx_immaterial/__init__.py not to import other stuff.


with open("requirements.txt", encoding="utf-8") as reqs:
REQUIREMENTS = [reqs.readlines()]
Expand Down Expand Up @@ -146,6 +152,11 @@ def run(self):
target = {"min": "build", "dev": "build:dev"}

try:
install_google_fonts(
files(resources),
files(resources),
DEFAULT_THEME_OPTIONS["font"].values(),
)
Comment on lines +155 to +159
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this will really solve the problem reported in #222. If a CI installs this theme, then the install process will error out if the CI runner doesn't have access to Google's font API (as similarly reported in #222).

I was under the impression that we'd have to include copies of the default fonts with the theme src to properly address #222. I do appreciate the attempt to update the default fonts at pkg install time, but it isn't really feasible (with respect to #222).

#222 would be better resolved by updating the default fonts' cache when npm run build is performed.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When installing from the git repository directly, we have to download the npm packages (which requires internet access). However, the wheel package that we publish to PyPI includes the pre-built javascript and css bundles, and icons, and I think the idea here is that it could also include the Google fonts. So it would in fact solve the problem reported in #222 as long as the user does not wish to change the fonts from their default values.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that was my thinking exactly.

I'm not sure if installing from git is preferred when the users' env has limited access to other domains.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, those are two different use cases:

  • Installing the wheel is meant for users of the tool which want to consume a pre-built artifact.
  • Installing from git|zip|tar is meant for developers and contributors which want to build the tool from sources.

Those two use cases are common in any "compiled|elaborated" software project. sphinx-immaterial does need "compilation|elaboration", despite the languages being both "interpreted". The elaboration is not only related to the fonts, but the set of dependencies (tools) is different from runtime deps.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Installing from git/zip is also common for envs that don't have python's pip tool. For example, the python executable shipped with Ubuntu usually doesn't have pip or venv and the admin group (needed to use apt) isn't viable for the active user(s). Sure, there are alternative install methods (docker containers), but it usually falls on pkg maintainers when all else fails.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I completely missed that comment, so I'm answering it in parts...

Where would that sdist zip be hosted?

Currently, we don't publish releases on github, but we could upload the archive as a release asset if we do. Maybe that is something that should be discussed as part of #223. However, installing from git does require npm and stuff.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@2bndy5 any update to this feature? Is there an alternative implementation?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I got stumped on how to do this from setup.py without breaking installs in a fresh python venv. See this comment (and the following response) which talks about switching to fontsource instead of fetching fonts directly from the GoogleFonts API.

My only real problem with this PR (aside from unnecessary new dependency on importlib.resources -- see review discussion here) is not being able to install it in a fresh venv because it would require all theme deps be install-time deps as well. See this review discussion.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you or @glatosinski could satisfy the review concerns here, then I'd be happy to move this PR forward and implement the switch to fontsource CDN in a separate PR.

I'm currently enthralled with exploring my idea for #261.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I'll look into it

tgt = target[self.bundle_type]
node_modules_path = os.path.join(root_dir, "node_modules")
if self.skip_npm_reinstall and os.path.exists(node_modules_path):
Expand Down Expand Up @@ -187,6 +198,8 @@ def run(self):
"*.html",
"custom_admonitions.css",
"theme.conf",
"resources/*.response",
"resources/*/*.response",
],
"sphinx_immaterial.apidoc.cpp.cppreference_data": ["*.xml"],
},
Expand Down
20 changes: 18 additions & 2 deletions sphinx_immaterial/external_resource_cache.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,16 @@
import os
import tempfile
from typing import Dict, Optional
from importlib_resources import files

import appdirs
import requests
import sphinx.application
import sphinx.config
import sphinx.util.logging

from sphinx_immaterial import resources

logger = sphinx.util.logging.getLogger(__name__)


Expand All @@ -22,13 +25,23 @@ def get_url(
req_json_encoded = json.dumps(req_json).encode("utf-8")
req_key = hashlib.sha256(req_json_encoded).hexdigest()

resp_path = os.path.join(cache_dir, f"{req_key}.response")
# First try the in-module resources
mod_res_path = files(resources) / f"{req_key}.response"
try:
with open(resp_path, "rb") as f:
with open(str(mod_res_path), "rb") as f:
return f.read()
except FileNotFoundError:
pass

# Secondly, look at the cache
resp_path = os.path.join(cache_dir, f"{req_key}.response")
if cache_dir:
try:
with open(resp_path, "rb") as f:
return f.read()
except FileNotFoundError:
pass

logger.info("Fetching: %s with %r", url, headers)
r = requests.get( # pylint: disable=missing-timeout
url, headers=headers, stream=True
Expand All @@ -37,6 +50,9 @@ def get_url(

response_content = r.content

if not cache_dir:
return response_content

# Write request.
req_path = os.path.join(cache_dir, f"{req_key}.request")
os.makedirs(cache_dir, exist_ok=True)
Expand Down
35 changes: 29 additions & 6 deletions sphinx_immaterial/google_fonts.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,25 @@ def _adjust_css_urls(css_content: bytes, renamed_fonts: Dict[str, str]) -> str:
_TTF_FONT_PATHS_KEY = "sphinx_immaterial_ttf_font_paths"


def add_google_fonts(app: sphinx.application.Sphinx, fonts: List[str]):
cache_dir = os.path.join(get_cache_dir(app), "google_fonts")
static_dir = os.path.join(app.outdir, "_static")
# _static path
font_dir = os.path.join(static_dir, "fonts")
def install_google_fonts(cache_dir: str, font_dir: str, fonts: List[str]):
"""
Saves google fonts to given directory.

Firstly, it tries to load fonts from cache directory.
If it fails, it downloads fonts from remote locations.

The font files are saved to font_dir.

Parameters
----------
cache_dir : str
Directory with cached downloaded files.
If cache_dir is empty, skip checking cache
font_dir : str
Target directory where fonts are saved
fonts : List[str]
List of fonts to save
"""
os.makedirs(font_dir, exist_ok=True)

with concurrent.futures.ThreadPoolExecutor(max_workers=32) as executor:
Expand Down Expand Up @@ -166,7 +180,16 @@ async def do_fetch():
css_content = dict(zip(css_future_keys, await asyncio.gather(*css_futures)))
return css_content

css_content = asyncio.run(do_fetch())
return asyncio.run(do_fetch())


def add_google_fonts(app: sphinx.application.Sphinx, fonts: List[str]):
cache_dir = os.path.join(get_cache_dir(app), "google_fonts")
static_dir = os.path.join(app.outdir, "_static")
# _static path
font_dir = os.path.join(static_dir, "fonts")

css_content = install_google_fonts(cache_dir, font_dir, fonts)

# Write fonts css file
ttf_font_paths = {}
Expand Down
3 changes: 3 additions & 0 deletions sphinx_immaterial/resources/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
"""
Module holding additional resources, i.e. built-in fonts
"""