Question: Is parallel attack support a useful feature within ART?

[simplymathematics]

I'll admit that I'm a novice user of Tools like Pytorch and Tensorflow for back-end optimization. However, I've added support locally for two types of parallelization:

1. Handling massively parallel attack parameter optimization by running models in parallel on different cores within a cpu.
2. Handling large datasets during attack generation by loading the data in parallel across cores in a CPU.

I understand that these only give modest speed increases, but implementing them was very useful to my own understanding and research. If you find that either of these features would be useful, I would happily open a pull request. However, I totally understand if the design goals are orthogonal to this kind of optimization and if your intent is hand it off to a back-end framework like those mentioned above.

I appreciate your input and your time spent developing this tool.

[beat-buesser]

Hi @simplymathematics Thank you very, very much for opening this issue! I was very happy to read your positive message.

I think both, parallel attack parameter optimisation and parallel loading of data, are of great interest to ART. We are actually planning to introduce a new module for evaluations in future releases and on a first look the parameter optimisation could be a great fit there but also in other modules.

I would like to suggest that your next step would be to open a pull request to branch dev_1.5.0 and we'll be very interested to take closer look and continue the discussion on integrating your code into ART based on the PR.

[simplymathematics]

Wonderful. Unfortunately I have some pressing deadlines over the next few weeks (am a Phd student with articles to publish), so I will have to table the development for some point in the future (November, most likely). In addition, I have written wrapper functions and a command line tool to aid in model/attack generation and evaluation(with further support planned for defense mechanisms). If you think those too would be useful in the evaluations module, then I would love to hear more about your goals for that module and how I can contribute. I had already planned on further developing my tooling after these deadlines and would love your input so that my work can be as useful as possible.

[beat-buesser]

That's ok, no worries, we are also familiar with the challenges of publications and deadlines. I would recommend, if possible, to open the pull request of your parallelisation work for optimisation and data loading as it is now as soon as you can, because that way the ART team can take a look and get familiar with your developments in the meantime and we will be ready to provide feedback after the deadlines.

Your command line tool sounds interesting and I would be interested to learn more. Our plans for a new evaluations module go into the direction of providing new tools that automate the application of the attacks for evaluation purposes which I think probably will include aspects of optimising attack parameters. One of the goals of the evaluations module is to enable evaluations of machine learning models with reduced requirements on the user knowing how optimise the application of the attacks. I also think the evaluations module should provide easier access to evaluations which could take advantage of your command line tool.

[saigontrade88]

Hello ART,
May I know there is any update on this. Many thanks.