Error in **attack_adversarial_patch_pytorch_yolo.ipynb**

[Louquinze]

Describe the bug
I am trying to run the example notebook attack_adversarial_patch_pytorch_yolo.ipynb. In cell 7 I get a KeyError.

Adversarial Patch PyTorch:   0%|                                                                                            | 0/1000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/ad_patch_torch.py", line 185, in <module>
    patch, patch_mask = ap.generate(x=x, y=target)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/attacks/evasion/adversarial_patch/adversarial_patch_pytorch.py", line 606, in generate
    _ = self._train_step(images=images, target=target, mask=None)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/attacks/evasion/adversarial_patch/adversarial_patch_pytorch.py", line 188, in _train_step
    loss = self._loss(images, target, mask)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/attacks/evasion/adversarial_patch/adversarial_patch_pytorch.py", line 251, in _loss
    loss = self.estimator.compute_loss(x=patched_input, y=target)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/estimators/object_detection/pytorch_yolo.py", line 632, in compute_loss
    loss_components, _ = self._get_losses(x=x, y=y)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/estimators/object_detection/pytorch_yolo.py", line 380, in _get_losses
    x_preprocessed, y_preprocessed = self._preprocess_and_convert_inputs(x=x, y=y, fit=False, no_grad=False)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/estimators/object_detection/pytorch_yolo.py", line 309, in _preprocess_and_convert_inputs
    if y is not None and isinstance(y[0]["boxes"], np.ndarray):
KeyError: 0

I was also checking the content of y:

{'boxes': tensor([[[  0.00000,   0.00000,   8.17627,   8.57283],
         [  0.00000,   0.00000,  27.70321,   8.76418],
         [  1.62518,   0.00000,  34.54520,   9.07174],
         ...,
         [495.75159, 549.26038, 636.07825, 640.00000],
         [539.20776, 543.38226, 640.00000, 640.00000],
         [557.95374, 531.07214, 640.00000, 640.00000]]]), 'labels': tensor([[0, 0, 0,  ..., 0, 0, 0]]), 'scores': tensor([[2.62016e-05, 2.61472e-05, 1.82523e-05,  ..., 3.00497e-06, 6.19944e-06, 4.10037e-06]])}

So it looks like that y isn't a list or array but a dictionary and therefore does not feature the label 0.

When i change the condition to:

if y is not None and isinstance(y["boxes"], np.ndarray):
    ....

I get the following error:

Traceback (most recent call last):
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/ad_patch_torch.py", line 185, in <module>
    patch, patch_mask = ap.generate(x=x, y=target)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/attacks/evasion/adversarial_patch/adversarial_patch_pytorch.py", line 606, in generate
    _ = self._train_step(images=images, target=target, mask=None)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/attacks/evasion/adversarial_patch/adversarial_patch_pytorch.py", line 188, in _train_step
    loss = self._loss(images, target, mask)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/attacks/evasion/adversarial_patch/adversarial_patch_pytorch.py", line 251, in _loss
    loss = self.estimator.compute_loss(x=patched_input, y=target)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/estimators/object_detection/pytorch_yolo.py", line 634, in compute_loss
    loss_components, _ = self._get_losses(x=x, y=y)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/estimators/object_detection/pytorch_yolo.py", line 382, in _get_losses
    x_preprocessed, y_preprocessed = self._preprocess_and_convert_inputs(x=x, y=y, fit=False, no_grad=False)
  File "/home/lukas/PycharmProjects/ExperimentalAttack/adversarial-robustness-toolbox/art/estimators/object_detection/pytorch_yolo.py", line 325, in _preprocess_and_convert_inputs
    x_tensor.requires_grad = True
RuntimeError: you can only change requires_grad flags of leaf variables.

To Reproduce
Steps to reproduce the behavior:
Simple run the jupyter notebook

Expected behavior
Create a adverserial Patch

System information (please complete the following information):

• Ubuntu 22.04
• Python 3.10
• ART version or commit number: Merge pull request Update pytest requirement from ~=7.2.2 to ~=7.3.1 #2133
• PyTorch

[beat-buesser]

Hi @Louquinze Thank you very much for using ART and the detailed investigation! We'll try to reproduce the issue as soon as possible and come back.

[beat-buesser]

@kieranfraser Are you able to reproduce this issue?

[kieranfraser]

Hi @Louquinze, @beat-buesser,

Yes, I was able to reproduce this issue - due to recent updates to PyTorchYolo, the format of targets passed to the estimator in AdversarialPatchPyTorch were incorrect and gradients of input tensors were not set. I have created a PR addressing this issue which contains a notebook demonstrating generation of patches for YOLO using the AdversarialPatchPyTorch and RobustDPatch classes.

Whilst debugging this, I identified another possible issue related to the loss computation. The pytorchyolo and yolov5 libraries do not have a relatively recent fix for 0 division when calculating alpha. Investigating this further. But you should be able to generate patches with the notebook referenced @Louquinze - let us know if you run into any issues with it?