| title | tags | authors | affiliations | date | bibliography | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
DIANNA: Deep Insight And Neural Network Analysis |
|
|
|
22 March 2022 |
paper.bib |
The growing demand from science and industry inspired rapid advances in Artificial Intelligence (AI). The increased use of AI and the reliability and trust required by automated decision making and standards of scientific rigour, led to the boom of eXplainable Artificial Intelligence (XAI). DIANNA (Deep Insight And Neural Network Analysis) is the first Python package of systematically selected XAI methods supporting the Open Neural Networks Exchange (ONNX) format. DIANNA is built by and designed for all research software engineers and researchers, also non-AI experts.
AI systems have been increasingly used in a wide variety of fields, including such data-sensitive areas as healthcare [@alshehrifatima], renewable energy [@kuzlumurat], supply chain [@toorajipourreza] and finance. Automated decision-making and scientific research standards require reliability and trust of the AI technology [@xuwei]. Especially in AI-enhanced research, a scientist need to be able to trust a high-performant, but opaque AI model used for automation of their data processing pipeline. In addition, XAI has the potential for helping any scientist to "find new scientific discoveries in the analysis of their data” [@hey]. Furthermore, tools for supporting repeatable science are of high demand [@Feger2020InteractiveTF].
DIANNA addresses these needs of researchers in various scientific domains who make use of AI models, especially supporting non-AI experts. DIANNA provides a Python-based, user-friendly, and uniform interface to several XAI methods. To the best of our knowledge, it is the only library using Open Neural Network Exchange (ONNX) [@onnx], the open-source, framework-agnostic standard for AI models, which supports repeatability of scientific research.
There are numerous Python XAI libraries, many are listed in the Awesome explainable AI [@awesomeai] repository. Popular and widely used packages are Pytorch [@pytorch], LIME [@ribeirolime], Captum [@kokhlikyan2020captum], Lucid [@tflucid], SHAP [@lundbergshap], InterpretML [@nori2019interpretml], PyTorch CNN visualizations [@uozbulak_pytorch_vis_2021] Pytorch GradCAM [@jacobgilpytorchcam], Deep Visualization Toolbox [@yosinski-2015-ICML-DL-understanding-neural-networks], ELI5 [@eli5]. However, these libraries have limitations that complicate adoption by scientific communities:
- Single XAI method or single data modality. While libraries such as SHAP, LIME, Pytorch GradCAM. have gained great popularity, their methods are not always suitable for the research task and/or data modality. For example, GradCAM is applicable only to images. Most importantly, each library in that class addresses AI explainability with a different method, complicating comparison between methods.
- Single Deep Neural Network (DNN) format/framework/architecture. Many XAI libraries support a single DNN format: Lucid supports only TensorFlow [@tf], Captum - PyTorch [@pytorch] and iNNvestigate [@innvestigatenn] is aimed at Keras users exclusively. Pytorch GradCAM supports a single method for a single format and Convolutional Neural Network Visualizations even limits the choice to a single DNN type. While this is not an issue for the current most popular framework communities, not all mature libraries support a spectrum of XAI methods. Most importantly, tools that support a single framework are not "future-proof". For instance, Caffe [@jia2014caffe] was the most popular framework in the computer vision (CV) community in 2018, but it has since been abandoned.
- Unclear choice of supported XAI methods. ELI5 supports multiple frameworks/formats and XAI methods, but it is unclear how the selection of these methods was made. Furthermore, the library has not been maintained since 2020, so any methods in the rapidly changing XAI field proposed since then are missing.
- AI expertise is necessary. The Deep Visualization Toolbox requires DNN knowledge and is only used by AI experts mostly within the CV community.
In addition, on more fundamental level, the results of XAI research does not help to make the technology understandable and trustworthy for non (X)AI experts:
- Properties of the output of the explainer. There is no commonly accepted methodology to systematically study XAI methods and their output.
- Human interpretation intertwined with the one of the explainer. This is a major problem in the current XAI literature, and there has been limited research to define what constitutes a meaningful explanation in the context of AI systems [@joylu].
- Lack of suitable (scientific) datasets. The most popular and simplest dataset used as "toy-example" is the MNIST dataset of handwritten digits [@mnistdataset], composed of 10 classes and with no structural variation in the content. Such a dataset is too complex for non-AI experts to intuitively understand the XAI output and simultaneously too far from scientific research data.
- Plethora of current AI model formats. The amount and the speed with which they become obsolete is another important show-stopper for reproducibility of AI-enabled science.
- Lack of funding. Some libraries, such as iNNvestigate, are becoming obsolete due to the lack of support for research projects that sponsored their creation.
DIANNA is an open source XAI Python package with the following key characteristics:
- Systematically chosen diverse set of XAI methods. We have used a relevant subset of the thorough objective and systematic evaluation criteria defined in [@peterflatch]. Several complementary and model-architecture agnostic state-of-the-art XAI methods have been chosen and included in DIANNA [@ranguelova_how_2022].
- Multiple data modalities. DIANNA supports images and text, we will extend the input data modalities to embeddings, time-series, tabular data and graphs. This is particularly important to scientific researchers, whose data are in domains different than the classical examples from CV and natural language processing communities.
- Open Neural Network Exchange (ONNX) format. ONNX is the de-facto standard format for neural network models. Not only is the use of ONNX very beneficial for interoperability, enabling reproducible science, but it is also compatible with runtimes and libraries designed to maximize performance across hardware. To the best of our knowledge, DIANNA is the first and only XAI library supporting ONNX.
- Simple, intuitive benchmark datasets. We have proposed two new datasets which enable systematic research of the properties of the XAI methods' output and understanding on an intuitive level: Simple Geometric Shapes [@oostrum_leon_2021_5012825] and LeafSnap30 [@ranguelova_elena_2021_5061353]. The classification of tree species on LeafSnap data is a great example of a simple scientific problem tackled with both classical CV and a deep learning method, where the latter outperforms, but needs explanations. DIANNA also uses well-established benchmarks: a simplified MNIST with 2 distinctive classes only and the Stanford Sentiment Treebank [@socher-etal-2013-recursive].
- User-friendly interface. DIANNA wraps all XAI methods with a common API.
- Modular architecture, extensive testing and compliance with modern software engineering practices. It is very easy for new XAI methods which do not need to access the ONNX model internals to be added to DIANNA. For relevance-propagation type of methods, more work is needed within the ONNX standard [@levitan_onnx_2020] and we hope our work will boost the development growth of ONNX (scientific) models. We welcome the XAI research community to contribute to these developments via DIANNA.
- Thorough documentation. The package includes user and developer documentation. It also provides instructions for conversion between ONNX and Tensorflow, Pytorch, Keras or Scikit-learn.
DIANNA is currently being used in the project "Recognizing symbolism in Turkish television drama" [@turkishdrama]. An important task is the development of an effective model for detecting and recognizing symbols in videos. DIANNA is being used to increase understanding of AI models in order to explore how to improve them.
DIANNA is also currently being used in the "Visually grounded models of spoken language" project, which builds on previous work by [@chrupala17representations;@alishahi+17;@chrupala18;@chrupala+19]. The goal is a multimodal model that projects image and sound data into a common embedded space. Within DIANNA, we are developing XAI methods to visualize and explain these embedded spaces in their complex multimodal network contexts.
Finally, DIANNA has also been used in the EU-funded Examode medical research project [@bryancardenas]. It deals with very large datasets and, because it aims to support physicians in their decision making, needs transparent and trustworthy models.
This work was supported by the Netherlands eScience Center and SURF.