-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ObjectSeeker Defense Implementation #2246
Conversation
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Codecov Report
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. @@ Coverage Diff @@
## dev_1.16.0 #2246 +/- ##
==============================================
+ Coverage 85.28% 85.79% +0.50%
==============================================
Files 315 318 +3
Lines 28054 28329 +275
Branches 5123 5157 +34
==============================================
+ Hits 23927 24304 +377
+ Misses 2833 2711 -122
- Partials 1294 1314 +20
... and 11 files with indirect coverage changes 📢 Have feedback on the report? Share it here. |
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @f4str Thank you very much for your well prepared pull request implementing ObjectSeeker in ART! I have only a few questions and suggestions, but otherwise the code looks good.
class ObjectSeekerMixin(abc.ABC): | ||
""" | ||
Implementation of the ObjectSeeker certifiable robust defense applied to object detection models. | ||
The original implementation is https://github.com/inspire-group/ObjectSeeker |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need to include any licence information?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The original code has an MIT license, so we can probably include their license below the ART one.
|
||
|
||
@pytest.fixture() | ||
def get_pytorch_yolo(get_default_cifar10_subset): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this a fixture that we also could use in other tests and move to contest.py?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea, this can be moved outside to a conftest.py
since it is used by multiple tests now.
# Compute loss after training | ||
loss2 = object_seeker.compute_loss(x=x_test, y=y_test) | ||
|
||
assert loss1 != loss2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should loss2
be smaller than loss1
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I borrowed the same unit test from test_pytorch_yolo.py
. Logically it should be less than, but I believe it was not equal in case of stochasticity that causes the loss to increase. I can look into this and see if it can be changed to less than for both tests.
Signed-off-by: Farhan Ahmed <Farhan.Ahmed@ibm.com>
Hi @beat-buesser thank you for the review. I've addressed the changes by adding the license from the original ObjectSeeker repo and extracting the fixtures into a As for the training loss not-equal vs less-than, it needs to remain a not-equal condition since due to stochasticity, the loss does not always decrease, especially with the "incorrect" dataset. The same applies to the PyTorchYolo and PyTorchFasterRCNN models. |
Hi @f4str Thank you very much for the updates as well as checking and documenting! We'll merge this PR as soon as possible. |
Description
Implement the ObjectSeeker certifiably robust defense for object detectors. This is an estimator located in
art.estimators.certification.object_seeker
as thePyTorchObjectSeeker
class. It is constructed in a similar way as thePyTorchObjectDetector
orPyTorchYolo
. Currently, there is only a PyTorch implementation, but a generic and TensorFlow implementation is possible in the future.Additionally created utility functions for
intersection_over_union
,intersection_over_area
, andnon_maximum_supression
underart.utils
.Note that unit tests are in
tests/estimators/object_detection
rather thantests/estimators/certification
since a YOLO model is required.Relevant links:
Fixes #2193
Type of change
Please check all relevant options.
Testing
Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.
Test Configuration:
Checklist