Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Randomized Smoothing Variations Implementation #2218

Merged

Conversation

f4str
Copy link
Collaborator

@f4str f4str commented Jul 10, 2023

Description

Implement the following variations of randomized smoothing:

  • MACER for PyTorch
  • MACER for TensorFlow v2
  • SmoothAdv for PyTorch
  • SmoothAdv for TensorFlow 2
  • SmoothMix for PyTorch

Additionally implement the following changes

  • Rewrite the randomized smoothing test cases from unittest to pytest
  • Add support to schedulers for all TensorFlow v2 classifiers (following the implementation from TensorFlowV2DeRandomizedSmoothing)
  • Add the verbose parameter to all randomized smoothing classifiers to hide/show the progress bar

This PR is continued and consolidated from

Links to Paper and Authors' code repository:

Type of change

Please check all relevant options.

  • Improvement (non-breaking)
  • Bug fix (non-breaking)
  • New feature (non-breaking)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

  • Updated unit tests for Randomized Smoothing
  • New unit tests for MACER
  • New unit tests for SmoothAdv
  • New unit tests for SmoothMix

Test Configuration:

  • OS
  • Python version
  • ART version or commit number
  • TensorFlow / Keras / PyTorch / MXNet version

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

f4str added 14 commits July 6, 2023 14:02
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
@codecov-commenter
Copy link

codecov-commenter commented Jul 10, 2023

Codecov Report

Merging #2218 (63612d2) into dev_1.16.0 (82f8fa2) will decrease coverage by 4.25%.
The diff coverage is 95.02%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Impacted file tree graph

@@              Coverage Diff               @@
##           dev_1.16.0    #2218      +/-   ##
==============================================
- Coverage       85.61%   81.36%   -4.25%     
==============================================
  Files             308      313       +5     
  Lines           27448    27782     +334     
  Branches         5044     5082      +38     
==============================================
- Hits            23499    22606     -893     
- Misses           2669     3932    +1263     
+ Partials         1280     1244      -36     
Files Changed Coverage Δ
art/estimators/classification/tensorflow.py 82.72% <50.00%> (-0.61%) ⬇️
...s/certification/randomized_smoothing/tensorflow.py 87.09% <60.00%> (-2.74%) ⬇️
...ification/randomized_smoothing/macer/tensorflow.py 91.89% <91.89%> (ø)
...tion/randomized_smoothing/smooth_adv/tensorflow.py 94.11% <94.11%> (ø)
...ication/randomized_smoothing/smooth_mix/pytorch.py 97.22% <97.22%> (ø)
...ication/randomized_smoothing/smooth_adv/pytorch.py 98.36% <98.36%> (ø)
...ertification/randomized_smoothing/macer/pytorch.py 98.38% <98.38%> (ø)
...ors/certification/randomized_smoothing/__init__.py 100.00% <100.00%> (ø)
...tors/certification/randomized_smoothing/pytorch.py 85.54% <100.00%> (ø)
...ation/randomized_smoothing/randomized_smoothing.py 97.36% <100.00%> (+0.03%) ⬆️

... and 21 files with indirect coverage changes

@f4str f4str marked this pull request as ready for review July 11, 2023 15:45
@beat-buesser beat-buesser self-requested a review July 14, 2023 10:42
@beat-buesser beat-buesser self-assigned this Jul 14, 2023
@beat-buesser beat-buesser added the enhancement New feature or request label Jul 14, 2023
@beat-buesser beat-buesser added this to the ART 1.16.0 milestone Jul 14, 2023
Copy link
Collaborator

@beat-buesser beat-buesser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @f4str Thank you very much for your pull request! The code looks great! I have a dded a few suggestions for minor improvements, what do you think?

loss: "torch.nn.modules.loss._Loss",
input_shape: Tuple[int, ...],
nb_classes: int,
optimizer: Optional["torch.optim.Optimizer"] = None, # type: ignore
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this type ignore required? The None should be covered by the Optional type.

Copy link
Collaborator Author

@f4str f4str Aug 4, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type: ignore is not required. The original randomized_smoothing/pytorch.py had this so I assumed it was needed. It can be removed from all variations including the original.

loss: "torch.nn.modules.loss._Loss",
input_shape: Tuple[int, ...],
nb_classes: int,
optimizer: Optional["torch.optim.Optimizer"] = None, # type: ignore
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
optimizer: Optional["torch.optim.Optimizer"] = None, # type: ignore
optimizer: Optional["torch.optim.Optimizer"] = None,

loss: "torch.nn.modules.loss._Loss",
input_shape: Tuple[int, ...],
nb_classes: int,
optimizer: Optional["torch.optim.Optimizer"] = None, # type: ignore
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
optimizer: Optional["torch.optim.Optimizer"] = None, # type: ignore
optimizer: Optional["torch.optim.Optimizer"] = None,

) -> Tuple["torch.Tensor", "torch.Tensor"]:
"""
The authors' implementation of the SmoothMixPGD attack.
Code modified from https://github.com/jh-jeong/smoothmix/code/train.py
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's include their MIT License text (https://github.com/jh-jeong/smoothmix/blob/main/LICENSE) at the top of our new file after ART's MIT license text.

Suggested change
Code modified from https://github.com/jh-jeong/smoothmix/code/train.py
Code modified from https://github.com/jh-jeong/smoothmix/blob/main/code/train.py

Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
@f4str f4str requested a review from beat-buesser August 4, 2023 23:22
@f4str
Copy link
Collaborator Author

f4str commented Aug 4, 2023

Hi @beat-buesser thank you for the review. I've addressed your comments and made the corresponding changes. Feel free to let me know if there's any other issues.

Copy link
Collaborator

@beat-buesser beat-buesser left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @f4str Thank you very much for extending the randomised smoothing capabilities in ART! The changes look good to me.

@beat-buesser beat-buesser merged commit cb4deda into Trusted-AI:dev_1.16.0 Aug 14, 2023
@f4str f4str deleted the randomized-smoothing-additions branch August 14, 2023 19:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants