T7147: Import Large Network Sets to Firewall Group From File

[sskaje]

Change summary

• add source-file to firewall/group/network-group (ipv6-network-group)
• add op command update firewall-groups to update sets from file

Op command updates:

• nat (nftables_nat.conf)
• policy (nftables_policy.conf)

Following with group_tmpl not updated:

• firewall (nftables.conf)
• nat66 (nftables_nat66.nft)
• conntrack (nftables-ct.conf)
• bridge (nftables-bridge)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Code style update (formatting, renaming)
• Refactoring (no functional changes)
• Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
• Other (please describe):

Related Task(s)

https://vyos.dev/T7147

Related PR(s)

How to test / Smoketest result

Passed

• test_firewall.py
• test_policy.py
• test_policy_route.py
• test_policy_local-route.py
• test_policy_local-route.py

Checklist:

• I have read the CONTRIBUTING document
• I have linked this PR to one or more Phabricator Task(s)
• I have run the components SMOKETESTS if applicable
• My commit headlines contain a valid Task id
• My change requires a change to the documentation
• I have updated the documentation accordingly

[github-actions]

👍
No issues in PR Title / Commit Title

[sskaje]

For the op command, I don't add all because:

1. different templates use different vars for group_tmpl, to build a config tree for all templates but using different array keys looks weird
2. building config tree for these templates like what I have done is not that good, but I don't find a better way rebuild all nftable files and re-apply in op mode.

[sskaje]

When I make changes to firewall groups in config mode, all nftables using groups are updated. Is there any mechanism I can invoke in op mode?

[c-po]

How is this different from T5493 ?

• firewall: T5493: Implement remote-group #4326

[sskaje]

How is this different from T5493 ?

• firewall: T5493: Implement remote-group #4326

I read the changes in that PR, he added a new type of group, but I don't see how he uses such group.
(also read #1648, T6040, etc.)

I would say, the so-called 'remote' should just be a different source for each type of group(embedded in config tree, local file or remote url), but not a new group type.

In my case I planned to say from file or url, but I see many different file formats for china route lists, and there is no standards.

So I create this PR with least thing which I think VyOS should support, which is loading from a file under /config and update nftables.
Those who need to update, they can write their own scripts and setup their own task schedulers.

Other type of groups can be also added in similar way.

[github-actions]

CI integration ❌ failed!

Details

CI logs

• CLI Smoketests (no interfaces) ❌ failed
• CLI Smoketests (interfaces only) 👍 passed
• Config tests 👍 passed
• RAID1 tests 👍 passed
• TPM tests 👍 passed

PR Overview

This PR introduces a new feature to import large network sets into firewall groups from a file and adds an operation command to update firewall groups from file data. Key changes include:

• Adding a source-file parameter to relevant firewall and network-group configurations.
• Implementing the op command update firewall-groups for nat and policy configurations.
• Updating validation in firewall group configuration to prevent mutually exclusive use of network and file source data.

Reviewed Changes

File	Description
src/helpers/firewall-group-update.py	New helper for retrieving and merging firewall group configuration.
python/vyos/firewall.py	New implementation of firewall_group_update for op command support.
src/conf_mode/firewall.py	Extended configuration verification to enforce exclusive parameters in network groups.
python/vyos/template.py	Added a new Jinja2 filter for nested group processing with file-sourced lists.

Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

src/conf_mode/firewall.py:445

• Error message on line 449 refers to 'source-file' while the code checks for 'source_file'. Consider using consistent naming for clarity.

if 'network_group' in firewall['group']: