Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finalize switch flow #207

Merged
merged 59 commits into from
Jan 31, 2025
Merged
Changes from 21 commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
4d1a825
refactor: Introduce generator class to handle global flow
Hellgartner Jan 27, 2025
fee2d18
refactor: Introduce argument class
Hellgartner Jan 27, 2025
3a118fd
refactor: Remove unnecessary comment
Hellgartner Jan 27, 2025
168ddc1
refactor: Introduce core package
Hellgartner Jan 27, 2025
1ade466
refactor: Introduce file list
Hellgartner Jan 27, 2025
0441c60
refactor: Make internal method "private"
Hellgartner Jan 28, 2025
2c734e6
refactor: Make scancode converter publish to the internal model
Hellgartner Jan 28, 2025
e471c65
refactor: Make opossum converter publish to the internal model
Hellgartner Jan 28, 2025
dd7e456
refactor: Extract reading into the internal format
Hellgartner Jan 28, 2025
08b8a67
refactor: Define interface and implement for opossum file
Hellgartner Jan 28, 2025
95b5523
refactor: Implement for interface for scancode file
Hellgartner Jan 28, 2025
996483a
Merge branch 'main' into finalize_switch_flow
Hellgartner Jan 28, 2025
57adfde
fix: remove spdx remainings
Hellgartner Jan 28, 2025
86ea7cf
refactor: introduce can handle pattern
Hellgartner Jan 28, 2025
5ab9635
refactor: rename
Hellgartner Jan 28, 2025
ed40898
refactor: use Path instead of strings for file paths
Hellgartner Jan 28, 2025
1051de7
refactor: rename
Hellgartner Jan 28, 2025
854e16e
refactor: remove not used merging functionality
Hellgartner Jan 28, 2025
fd3a285
refactor: use common generator helpers for testing
Hellgartner Jan 28, 2025
b1d0e2d
feat: introduce import linter
Hellgartner Jan 28, 2025
b58ba84
refactor: dedicated package for input formats
Hellgartner Jan 28, 2025
18bb9d4
refactor: review-comment: Rename input file type
Hellgartner Jan 28, 2025
308d175
refactor: review-comment: Using property annotation
Hellgartner Jan 28, 2025
c46f1a2
feat: Remove misleading log statement
Hellgartner Jan 28, 2025
51a4603
refactor: review comment: further improve the readability of the gene…
Hellgartner Jan 28, 2025
93ecfe1
refactor: review comment: remove for loop to select reader
Hellgartner Jan 28, 2025
f58490a
refactor: move opossum file model to dedicated package
Hellgartner Jan 29, 2025
d24c105
refactor: move also file writing into the package
Hellgartner Jan 29, 2025
5dd9461
refactor: DIY dependency injection: Define input readers at top level
Hellgartner Jan 29, 2025
780a80b
feat: Add importlinter rule to keep file format independent
Hellgartner Jan 29, 2025
bf6ab6e
fix: Add missing license header
Hellgartner Jan 29, 2025
137a3d0
fix: Remove surplus pass
Hellgartner Jan 29, 2025
9d5f9c9
feat: review-comment: New package structure
Hellgartner Jan 29, 2025
7707e8f
refactor: review-comment: Filename to match main class name
Hellgartner Jan 29, 2025
2348a1a
refactor: review-comment: Move file writer to core
Hellgartner Jan 29, 2025
72dd03e
refactor: review-comment: Fuse closely connected classes
Hellgartner Jan 29, 2025
b8848d5
refactor: review-comment: Fix typo
Hellgartner Jan 29, 2025
7c95235
refactor: review-comment: Smaller fixes
Hellgartner Jan 29, 2025
7b1547a
refactor: review-comment: rename opossum model files
Hellgartner Jan 29, 2025
bc8a7d6
refactor: review-comment: rename all file specific models to end with…
Hellgartner Jan 29, 2025
378c276
refactor: review-comment: use the renamings in converters
Hellgartner Jan 29, 2025
c4fd699
refactor: review-comment: Make tests adhere to the new structure
Hellgartner Jan 29, 2025
ea536f1
refactor: review -- inline validation logic
Hellgartner Jan 30, 2025
30bd42a
refactor: review -- static only classes to methods
Hellgartner Jan 30, 2025
df51d5a
refactor: review -- expand opossum model to multiple files
Hellgartner Jan 30, 2025
c06bb6d
refactor: review -- improve writing to opossum files
Hellgartner Jan 30, 2025
927b3eb
refactor: review -- rename scancode data model
Hellgartner Jan 30, 2025
d774d13
refactor: review -- improve scancode file reader
Hellgartner Jan 30, 2025
a2afebb
refactor: review -- extract shared CamelBaseModel
Hellgartner Jan 30, 2025
5087102
refactor: review -- adapt test structure
Hellgartner Jan 30, 2025
938c94e
refactor: review -- group tests in class structures
Hellgartner Jan 30, 2025
c21de0c
refactor: review -- unify closely related classes
Hellgartner Jan 31, 2025
17e603d
refactor: review -- remove no longer necessary linter exceptions
Hellgartner Jan 31, 2025
47f0aad
refactor: review -- delete dead code
Hellgartner Jan 31, 2025
768d9c0
refactor: review -- move primary class in file to top
Hellgartner Jan 31, 2025
dd4cf38
refactor: review -- rename method to match target type
Hellgartner Jan 31, 2025
c36ea7d
refactor: review -- improve docs
Hellgartner Jan 31, 2025
9d04580
refactor: review -- remove unused noqa
Hellgartner Jan 31, 2025
15d9a59
refactor: review -- static methods to private methods
Hellgartner Jan 31, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/lint_and_run_tests.yml
Original file line number Diff line number Diff line change
@@ -48,6 +48,9 @@ jobs:
- name: Run mypy
run: uv run python -m mypy src/ tests/

- name: Run import linter
run: uv run lint-imports

test:
runs-on: ${{ matrix.os }}
if: |
17 changes: 17 additions & 0 deletions .importlinter
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0

[importlinter]
root_package = opossum_lib
# Optional:
include_external_packages = True
exclude_type_checking_imports = True

[importlinter:contract:file-formats-independent]
name = The different file formats should be independent
type = independence
modules =
opossum_lib.input_formats.*
ignore_imports =
** -> opossum_lib.core.opossum_model
4 changes: 2 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -12,7 +12,6 @@ dependencies = [
"click>=8.1.8,<9",
"pydantic>=2.10.6",
"pyinstaller>=6.11.1",
"faker>=35.0.0",
]

[project.urls]
@@ -22,11 +21,12 @@ Repository = "https://github.com/opossum-tool/opossum-file"
opossum-file = "opossum_lib.cli:opossum_file"

[dependency-groups]
test = ["pytest>=8.3.4,<9"]
test = ["pytest>=8.3.4,<9", "faker>=35.0.0",]
dev = [
"mypy>=1.14.1,<2",
"pre-commit>=4.1.0,<5",
"ruff>=0.9.3",
"import-linter>=2.1",
]

[tool.uv]
61 changes: 13 additions & 48 deletions src/opossum_lib/cli.py
Original file line number Diff line number Diff line change
@@ -1,21 +1,14 @@
#!/usr/bin/env python3
from pathlib import Path

# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0


import logging
import sys
from pathlib import Path

import click

from opossum_lib.opossum.file_generation import OpossumFileWriter
from opossum_lib.opossum.opossum_file_content import OpossumFileContent
from opossum_lib.opossum.read_opossum_file import read_opossum_file
from opossum_lib.scancode.convert_scancode_to_opossum import (
convert_scancode_file_to_opossum,
from opossum_lib.core.opossum_generator import (
OpossumGenerationArguments,
OpossumGenerator,
)


@@ -50,9 +43,9 @@ def opossum_file() -> None:
'If appropriate, the extension ".opossum" will be appended.',
)
def generate(
scancode_json_files: list[str],
opossum_files: list[str],
outfile: str,
scancode_json_files: list[Path],
opossum_files: list[Path],
outfile: Path,
) -> None:
"""
Generate an Opossum file from various other file formats.
@@ -62,42 +55,14 @@ def generate(
- ScanCode
- Opossum
"""
validate_input_and_exit_on_error(scancode_json_files, opossum_files)
opossum_file_content = convert_after_valid_input(scancode_json_files, opossum_files)

if not outfile.endswith(".opossum"):
outfile += ".opossum"

if Path.is_file(Path(outfile)):
logging.warning(f"{outfile} already exists and will be overwritten.")

OpossumFileWriter.write_opossum_information_to_file(
opossum_file_content, Path(outfile)
OpossumGenerator().generate(
opossum_generation_arguments=OpossumGenerationArguments(
opossum_files=opossum_files,
scancode_json_files=scancode_json_files,
outfile=outfile,
)
)


def validate_input_and_exit_on_error(
scancode_json_files: list[str], opossum_files: list[str]
) -> None:
total_number_of_files = len(scancode_json_files) + len(opossum_files)
if total_number_of_files == 0:
logging.warning("No input provided. Exiting.")
sys.exit(1)
if total_number_of_files > 1:
logging.error("Merging of multiple files not yet supported!")
sys.exit(1)


def convert_after_valid_input(
scancode_json_files: list[str], opossum_files: list[str]
) -> OpossumFileContent:
if len(scancode_json_files) == 1:
scancode_json_input_file = scancode_json_files[0]
return convert_scancode_file_to_opossum(scancode_json_input_file)
else:
opossum_input_file = opossum_files[0]
return read_opossum_file(opossum_input_file)


if __name__ == "__main__":
opossum_file()
File renamed without changes.
17 changes: 17 additions & 0 deletions src/opossum_lib/core/input_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0
from enum import Enum, auto
from pathlib import Path

from pydantic import BaseModel


class FileType(Enum):
OPOSSUM = (auto(),)
SCAN_CODE = (auto(),)


class InputFile(BaseModel):
path: Path
type: FileType
18 changes: 18 additions & 0 deletions src/opossum_lib/core/input_format_reader.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0

from abc import abstractmethod
from pathlib import Path
from typing import Protocol

from opossum_lib.core.input_file import FileType
from opossum_lib.core.opossum_model import Opossum


class InputFormatReader(Protocol):
@abstractmethod
def can_handle(self, file_type: FileType) -> bool: ...

@abstractmethod
def read(self, path: Path) -> Opossum: ...
24 changes: 24 additions & 0 deletions src/opossum_lib/core/input_reader.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0

from opossum_lib.core.input_file import InputFile
from opossum_lib.core.input_format_reader import InputFormatReader
from opossum_lib.core.opossum_model import Opossum
from opossum_lib.input_formats.opossum.opossum_format_reader import OpossumFormatReader
from opossum_lib.input_formats.scancode.scancode_format_reader import (
ScancodeFormatReader,
)


class InputReader:
input_format_readers: list[InputFormatReader] = [
OpossumFormatReader(),
ScancodeFormatReader(),
]

def read(self, input_file: InputFile) -> Opossum:
for input_format_reader in self.input_format_readers:
if input_format_reader.can_handle(input_file.type):
return input_format_reader.read(input_file.path)
raise NotImplementedError(f"Unsupported file type: {input_file.type}")
44 changes: 44 additions & 0 deletions src/opossum_lib/core/opossum_generation_arguments.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0

import logging
import sys
from pathlib import Path

from pydantic import BaseModel

from opossum_lib.core.input_file import FileType, InputFile


class OpossumGenerationArguments(BaseModel):
scancode_json_files: list[Path]
opossum_files: list[Path]
outfile: Path

def validate_and_exit_on_error(self) -> None:
total_number_of_files = +len(self.scancode_json_files) + len(self.opossum_files)
if total_number_of_files == 0:
logging.warning("No input provided. Exiting.")
sys.exit(1)
if total_number_of_files > 1:
logging.error("Merging of multiple files not yet supported!")
sys.exit(1)

def add_outfile_ending_and_warn_on_existing_outfile(self) -> None:
if self.outfile.suffix != ".opossum":
self.outfile = self.outfile.with_suffix(".opossum")

if Path.is_file(Path(self.outfile)):
logging.warning(f"{self.outfile} already exists and will be overwritten.")

def input_files(self) -> list[InputFile]:
result = []
result += [
InputFile(path=path, type=FileType.SCAN_CODE)
for path in self.scancode_json_files
]
result += [
InputFile(path=path, type=FileType.OPOSSUM) for path in self.opossum_files
]
return result
26 changes: 26 additions & 0 deletions src/opossum_lib/core/opossum_generator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# SPDX-FileCopyrightText: TNG Technology Consulting GmbH <https://www.tngtech.com>
#
# SPDX-License-Identifier: Apache-2.0


from opossum_lib.core.input_reader import InputReader
from opossum_lib.core.opossum_generation_arguments import OpossumGenerationArguments
from opossum_lib.input_formats.opossum.opossum_file_writer import OpossumFileWriter


class OpossumGenerator:
input_reader: InputReader = InputReader()

def generate(
self, opossum_generation_arguments: OpossumGenerationArguments
) -> None:
opossum_generation_arguments.validate_and_exit_on_error()
input_files = opossum_generation_arguments.input_files()

opossum = self.input_reader.read(input_files[0])

opossum_file_content = opossum.to_opossum_file_format()
opossum_generation_arguments.add_outfile_ending_and_warn_on_existing_outfile()
OpossumFileWriter.write(
opossum_file_content, opossum_generation_arguments.outfile
)
Original file line number Diff line number Diff line change
@@ -15,9 +15,9 @@

from pydantic import BaseModel, ConfigDict

import opossum_lib.opossum.opossum_file as opossum_file
from opossum_lib.opossum.opossum_file_content import OpossumFileContent
from opossum_lib.opossum.output_model import OpossumOutputFile
import opossum_lib.input_formats.opossum.opossum_file as opossum_file
from opossum_lib.input_formats.opossum.opossum_file_content import OpossumFileContent
from opossum_lib.input_formats.opossum.output_model import OpossumOutputFile

type OpossumPackageIdentifier = str
type ResourcePath = str
File renamed without changes.
Empty file.
File renamed without changes.
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
from copy import deepcopy
from dataclasses import field
from enum import Enum, auto
from typing import Literal, cast
from typing import Literal

from pydantic import BaseModel, ConfigDict, model_serializer
from pydantic.alias_generators import to_camel
@@ -189,24 +189,3 @@ class ExternalAttributionSource(CamelBaseModel):
name: str
priority: int
is_relevant_for_preferred: bool | None = None


def _build_resource_tree(resource: ResourceInFile) -> Resource:
if isinstance(resource, int):
return Resource(type=ResourceType.FILE)
else:
result = Resource(type=ResourceType.FOLDER)
for name, child_resource in resource.items():
result.children[name] = _build_resource_tree(child_resource)
return result


def convert_resource_in_file_to_resource(resource: ResourceInFile) -> Resource:
root_node = Resource(type=ResourceType.TOP_LEVEL)

if isinstance(resource, dict):
dict_resource = cast(dict[str, ResourceInFile], resource)
for name, child_resource in dict_resource.items():
root_node.children[name] = _build_resource_tree(child_resource)

return root_node
Original file line number Diff line number Diff line change
@@ -6,26 +6,30 @@
import json
import logging
import sys
from pathlib import Path
from zipfile import ZipFile

from pydantic import BaseModel, TypeAdapter

from opossum_lib.opossum.constants import INPUT_JSON_NAME, OUTPUT_JSON_NAME
from opossum_lib.opossum.opossum_file import OpossumInformation
from opossum_lib.opossum.output_model import OpossumOutputFile
from opossum_lib.input_formats.opossum.constants import (
INPUT_JSON_NAME,
OUTPUT_JSON_NAME,
)
from opossum_lib.input_formats.opossum.opossum_file import OpossumInformation
from opossum_lib.input_formats.opossum.output_model import OpossumOutputFile


class OpossumFileContent(BaseModel):
input_file: OpossumInformation
output_file: OpossumOutputFile | None = None

@staticmethod
def from_file(file_name: str) -> OpossumFileContent:
logging.info(f"Converting opossum to opossum {file_name}")
def from_file(path: Path) -> OpossumFileContent:
logging.info(f"Converting opossum to opossum {path}")

try:
with (
ZipFile(file_name, "r") as input_zip_file,
ZipFile(path, "r") as input_zip_file,
):
OpossumFileContent._validate_zip_file_contents(input_zip_file)
input_file = OpossumFileContent._read_input_json_from_zip_file(
@@ -38,7 +42,7 @@ def from_file(file_name: str) -> OpossumFileContent:
),
)
except Exception as e:
print(f"Error reading file {file_name}: {e}")
print(f"Error reading file {path}: {e}")
sys.exit(1)

@staticmethod
Loading
Loading