feat: Implement HTML pre-processing option

Author: pawamoy

Makefile

@@ -17,6 +17,7 @@ BASIC_DUTIES = \
 	docs \
 	docs-deploy \
 	format \
+	manpage \
 	release
 
 QUALITY_DUTIES = \

README.md

@@ -16,13 +16,13 @@ Pandoc must be [installed](https://pandoc.org/installing.html) and available as
 
 With `pip`:
 ```bash
-pip install mkdocs-manpage
+pip install mkdocs-manpage[preprocess]
 ```
 
 With [`pipx`](https://github.com/pipxproject/pipx):
 ```bash
 python3.7 -m pip install --user pipx
-pipx install mkdocs-manpage
+pipx install mkdocs-manpage[preprocess]
 ```
 
 ## Usage
@@ -31,15 +31,85 @@ pipx install mkdocs-manpage
 # mkdocs.yml
 plugins:
 - manpage:
-    enabled: !ENV [MANPAGE, false]
     pages:
     - index.md
     - usage.md
     - reference/api.md
 ```
 
-We also recommend disabling some options from other plugins/extensions
-to improve the final manual page:
+To enable/disable the plugin with an environment variable:
+
+```yaml
+# mkdocs.yml
+plugins:
+- manpage:
+    enabled: !ENV [MANPAGE, false]
+```
+
+Then set the environment variable and run MkDocs:
+
+```bash
+MANPAGE=true mkdocs build
+```
+
+The manpage will be written into the root of the site directory
+and named `manpage.1`.
+
+### Pre-processing HTML
+
+This plugin works by concatenating the HTML from all selected pages
+into a single file that is then converted to a manual page using Pandoc.
+
+With a complete conversion, the final manual page will not look so good.
+For example images and SVG will be rendered as long strings of data and/or URIs.
+So this plugin allows users to pre-process the HTML, to remove unwanted
+HTML elements before converting the whole thing to a manpage.
+
+First, you must make sure to install the `preprocess` extra:
+
+```bash
+pip install mkdocs-manpage[preprocess]
+```
+
+To pre-process the HTML, we use [BeautifulSoup](https://pypi.org/project/beautifulsoup4/).
+Users have to write their own `preprocess` function in order to modify the soup
+returned by BeautifulSoup:
+
+```python title="scripts/preprocess.py"
+from bs4 import BeautifulSoup, Tag
+
+
+def to_remove(tag: Tag) -> bool:
+    # remove images and SVGs
+    if tag.name in {"img", "svg"}:
+        return True
+    # remove links containing images or SVGs
+    if tag.name == "a" and tag.img and to_remove(tag.img):
+        return True
+    # remove permalinks
+    if tag.name == "a" and "headerlink" in tag.get("class", ()):
+        return True
+    return False
+
+
+def preprocess(soup: BeautifulSoup) -> None:
+    for element in soup.find_all(to_remove):
+        element.decompose()
+```
+
+Then, instruct the plugin to use this module and its `preprocess` function:
+
+```yaml title="mkdocs.yml"
+plugins:
+- manpage:
+    preprocess: scripts/preprocess.py
+```
+
+See the documentation of both [`BeautifulSoup`][bs4.BeautifulSoup] and [`Tag`][bs4.Tag]
+to know what methods are available to correctly select the elements to remove.
+
+The alternative to HTML processing for improving the final manpage
+is disabling some options from other plugins/extensions:
 
 - no source code through `mkdocstrings`:
 
@@ -67,5 +137,4 @@ export MANPAGE=true
 export PERMALINK=false
 export SHOW_SOURCE=false
 mkdocs build
-# manpage is in site dir: ./site/manpage.1
 ```

docs/insiders/changelog.md

@@ -2,6 +2,10 @@
 
 ## MkDocs Manpage Insiders
 
+### 1.1.0 <small>June 07, 2023</small> { id="1.1.0" }
+
+- Implement an HTML pre-processing option, to improve manpage rendering
+
 ### 1.0.0 <small>June 06, 2023</small> { id="1.0.0" }
 
 - Release first Insiders version

docs/license.md

@@ -1,3 +1,5 @@
+# License
+
 ```
 --8<-- "LICENSE"
 ```

duties.py

@@ -309,9 +309,5 @@ def manpage(ctx: Context) -> None:
     Parameters:
         ctx: The context instance (passed automatically).
     """
-    os.environ["MANPAGE"] = "true"
-    os.environ["SHOW_SOURCE"] = "false"
-    os.environ["PERMALINK"] = "false"
-    os.environ["DEPLOY"] = "false"
     ctx.run(mkdocs.build(), title="Building docs and manpage")
     ctx.run("man ./site/manpage.1", capture=False)

mkdocs.yml

@@ -108,6 +108,7 @@ plugins:
       python:
         import:
         - https://docs.python.org/3/objects.inv
+        - https://www.crummy.com/software/BeautifulSoup/bs4/doc/objects.inv
         options:
           separate_signature: true
           merge_init_into_class: true
@@ -118,7 +119,7 @@ plugins:
     enabled: !ENV [DEPLOY, false]
     repository: pawamoy/mkdocs-manpage
 - manpage:
-    enabled: !ENV [MANPAGE, false]
+    preprocess: scripts/preprocess.py
     pages:
     - index.md
     - changelog.md

pyproject.toml

@@ -27,7 +27,15 @@ classifiers = [
     "Topic :: Utilities",
     "Typing :: Typed",
 ]
-dependencies = []
+dependencies = [
+    "importlib-metadata>=4; python_version < '3.8'"
+]
+
+[project.optional-dependencies]
+preprocess = [
+    "beautifulsoup4>=4.12",
+    "lxml>=4.9",
+]
 
 [project.urls]
 Homepage = "https://pawamoy.github.io/mkdocs-manpage"

scripts/gen_credits.py

@@ -89,6 +89,8 @@ def _render_credits() -> str:
     }
     template_text = dedent(
         """
+        # Credits
+
         These projects were used to build *{{ project_name }}*. **Thank you!**
 
         [`python`](https://www.python.org/) |

scripts/preprocess.py

@@ -0,0 +1,37 @@
+"""HTML pre-processing module."""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from bs4 import BeautifulSoup as Soup
+    from bs4 import Tag
+
+
+def to_remove(tag: Tag) -> bool:
+    """Tell whether a tag should be removed from the soup.
+
+    Parameters:
+        tag: The tag to check.
+
+    Returns:
+        True or false.
+    """
+    # Remove images and SVGs.
+    if tag.name in {"img", "svg"}:
+        return True
+    # Remove permalinks.
+    if tag.name == "a" and ("headerlink" in tag.get("class", "") or tag.img and to_remove(tag.img)):
+        return True
+    return False
+
+
+def preprocess(soup: Soup) -> None:
+    """Pre-process the soup by removing elements.
+
+    Parameters:
+        soup: The soup to modify.
+    """
+    for element in soup.find_all(to_remove):
+        element.decompose()

src/mkdocs_manpage/config.py

@@ -2,69 +2,13 @@
 
 from __future__ import annotations
 
-import contextlib
-from typing import Any, Dict, Generic, TypeVar
-
 from mkdocs.config import config_options as mkconf
-from mkdocs.config.base import Config
 from mkdocs.config.base import Config as BaseConfig
-from mkdocs.config.config_options import BaseConfigOption, LegacyConfig, ValidationError
-
-T = TypeVar("T")
-
-
-# TODO: remove once https://github.com/mkdocs/mkdocs/pull/3242 is merged and released
-class DictOfItems(Generic[T], BaseConfigOption[Dict[str, T]]):
-    """Validates a dict of items. Keys are always strings.
-
-    E.g. for `config_options.DictOfItems(config_options.Type(int))` a valid item is `{"a": 1, "b": 2}`.
-    """
-
-    required: bool | None = None  # Only for subclasses to set.
-
-    def __init__(self, option_type: BaseConfigOption[T], default: Any = None) -> None:  # noqa: D107
-        super().__init__()
-        self.default = default
-        self.option_type = option_type
-        self.option_type.warnings = self.warnings
-
-    def __repr__(self) -> str:
-        return f"{type(self).__name__}: {self.option_type}"
-
-    def pre_validation(self, config: Config, key_name: str) -> None:  # noqa: D102
-        self._config = config
-        self._key_name = key_name
-
-    def run_validation(self, value: object) -> dict[str, T]:  # noqa: D102
-        if value is None:
-            if self.required or self.default is None:
-                raise ValidationError("Required configuration not provided.")
-            value = self.default
-        if not isinstance(value, dict):
-            raise ValidationError(f"Expected a dict of items, but a {type(value)} was given.")
-        if not value:  # Optimization for empty list
-            return value
-
-        fake_config = LegacyConfig(())
-        with contextlib.suppress(AttributeError):
-            fake_config.config_file_path = self._config.config_file_path
-
-        # Emulate a config-like environment for pre_validation and post_validation.
-        fake_config.data = value
-
-        for key_name in fake_config:
-            self.option_type.pre_validation(fake_config, key_name)
-        for key_name in fake_config:
-            # Specifically not running `validate` to avoid the OptionallyRequired effect.
-            fake_config[key_name] = self.option_type.run_validation(fake_config[key_name])
-        for key_name in fake_config:
-            self.option_type.post_validation(fake_config, key_name)
-
-        return value
 
 
 class PluginConfig(BaseConfig):
     """Configuration options for the plugin."""
 
     enabled = mkconf.Type(bool, default=True)
     pages = mkconf.ListOfItems(mkconf.Type(str))
+    preprocess = mkconf.File(exists=True)