Digging into the Landscape of Graphs Counterfactual Explainability (AAAI 2024)

Digging into the Landscape of Graphs Counterfactual Explainability (AAAI 2024)

Laboratory at the Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24) on 20-21, Feb. 2024


Organisers

Mario A. Prado-Romero
Mario A. Prado-Romero
Gran Sasso Science Institute
Dr. Bardh Prenkaj
Dr. Bardh Prenkaj
Technical University of Munich
Prof. Giovanni Stilo
Prof. Giovanni Stilo
University of L'Aquila


Slides can be downloaded HERE

CLICK ON THE LOGO TO ACCESS THE VIRTUAL RESEARCH ENVIROMENT OF THE LAB

SoBigData.it - Research Infrastructure


Table of contents

Abstract

In this lab, we provide a hands-on experience to help users develop and evaluate novel Graph Counterfactual Explanation (GCE) methods using a simple and modular framework, GRETEL. We will cover the main challenges of producing counterfactual explanations in the graph domain. We will show the participants how to build basic explanation pipelines in GRETEL. Furthermore, we show how the different categories of explainers can be implemented following the framework’s general logic and present an empirical study of their performance in different datasets. Moreover, we will instruct the participants on extending the frameworks’ main components to support custom GCE scenarios (i.e., new datasets, oracles, explainers, and evaluation measures).

The attendants will dive into an immersive lab session combining hands-on exploration and captivating demonstrations. Our lab is designed to make them active participants in cutting-edge XAI frameworks. Here are the key highlights of the interactive experience (Excluding the demonstration parts):

Duration

The laboratory will be carried on for a quarter-day, spanning 1 hour 45 minutes, as the most suitable format to introduce attendees to counterfactual explanations on graphs and the GRETEL framework [1]. This duration will provide enough room to present the basic principles of the evaluation framework, analyze how different classes of explanation methods can be easily implemented into the general design, and provide examples of how to extend the framework for bespoke solutions. Notice that this lab complements a tutorial on the same topic we propose in the same venue to satisfy those interested in more theoretical aspects of Graph Counterfactual Explanations. Tutorial Website

Scope of the laboratory

The AAAI community has exhibited considerable interest in Graph Neural Networks and Explainable AI, as evidenced by the tutorials in previous editions. Graph Counterfactual Explainability (GCE) holds the potential to captivate a significantly broad audience of researchers and practitioners due to the pervasive nature of graphs and the inherent capacity of GCE methods to provide profound insights into intricate non-linear prediction systems. This lab will cover mostly the GRETEL framework for developing and evaluating Graph Counterfactual Explanations methods. This modular and extensible framework allows users to adapt the different components to their needs.

The lab is aimed at practitioners in academia and industry interested in applying machine learning techniques to analyze graph data. Participants with a background in data mining will gain an understanding of the information provided by explanation methods to end users and how to integrate their datasets into an explanation pipeline. Those with machine learning expertise will delve deeper into state-of-the-art counterfactual explainers, critically analyzing their strengths and weaknesses. They will also learn to integrate their oracles and develop new explanation methods using the GRETEL framework.

Prerequisites & Tools

The lab is aimed at practitioners in academia and industry interested in applying machine learning techniques to analyze graph data. Familiarity with modern AI frameworks like PyTorch and scikit-learn and proficiency in Python will be beneficial.

To fully follow the hands-on experience, the participants should bring their own laptops. We recommend a minimum of 8 GB of RAM for a better experience. It is necessary that the attendants pre-install a proper virtual environment on their computers. We will provide the necessary code and environment creation script for simplicity. An internet connection will be required only for downloading and installing the required tools. We encourage the participants to install and download the framework’s source code in advance. We will provide a simple “hello world” example inside the framework so the participants can check that everything is running smoothly before the lab. The source code can be downloaded from the GRETEL GitHub https://github.com/aiim-research/GRETEL.

Outline & Contents

Counterfactual explanations shed light on the decision-making mechanisms and provide alternative scenarios that yield different outcomes, offering end users recourse—these distinctive characteristics position counterfactual explainability as a highly effective method in graph-based contexts. A brief outline of the laboratory is provided in Figure 1. In the first part of this lab, the attendants will be introduced to the challenges of developing and evaluating GCE methods [1], including a lack of standardization in metrics and oracles, as well as benchmarking studies. Then, they will be introduced to the GRETEL framework for developing and evaluating GCE methods. Furthermore, we will provide hands-on examples of how to build the explanation pipeline using GRETEL.

Figure 1: The brief outline of the laboratory with its time scheduling.

Following the introduction, we will present how the different categories of explainers can be implemented into the framework, including search-based methods [2], heuristic-based ones [3,4,5], learning-based ones [4,6,7,8,9,10,11,12,13] and global-level explainers [14]. Moreover, we will analyze an empirical comparison of some of these methods [3,5,8,10,12,15} in different datasets.

In the third part of the lab, we will focus on a hands-on experience showing the attendants how to extend the framework to fit their needs [16]. This session will allow also the participants to access and use the SoBigData.it Research Infrastructure. Industry practitioners will learn how to use the framework on their datasets and to explain their own ML models, while attendants interested in XAI will learn how to develop and evaluate their explainers using the ready-to-use datasets, oracles and evaluation metrics provided in GRETEL. Furthermore, we will show how to use the data analysis capabilities of the framework to get more insights into the explanations.

To finalize the lab session, we will discuss the challenges of providing Graph Counterfactual Explanation and possible ways to tackle them. This part will allow the attendants to participate in the discussion actively.

Highlights

The lab is primarily based on the following BibTeX rferences:

@inproceedings{prado-romero2022gretel,
  title={GRETEL: Graph Counterfactual Explanation Evaluation Framework},
  author={Prado-Romero, Mario Alfonso and Stilo, Giovanni},
  publisher = {Association for Computing Machinery},
  address = {New York, NY, USA},
  isbn = {9781450392365},
  year={2022},
  doi = {10.1145/3511808.3557608},
  booktitle={Proceedings of the 31st ACM International Conference on Information and Knowledge Management},
  location = {Atlanta, GA, USA},
  series = {CIKM '22}
}

@inproceedings{prado-romero2023developing,
author = {Prado-Romero, Mario Alfonso and Prenkaj, Bardh and Stilo, Giovanni},
title = {Developing and Evaluating Graph Counterfactual Explanation with GRETEL},
year = {2023},
isbn = {9781450394079},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
doi = {10.1145/3539597.3573026},
booktitle = {Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining},
pages = {1180–1183},
location = {Singapore, Singapore},
series = {WSDM '23}
}

@article{10.1145/3618105,
    author = {Prado-Romero, Mario Alfonso and Prenkaj, Bardh and Stilo, Giovanni and Giannotti, Fosca},
    title = {A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges},
    year = {2023},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    issn = {0360-0300},
    url = {https://doi.org/10.1145/3618105},
    doi = {10.1145/3618105},
    journal = {ACM Computing Surveys},
    month = {sep}
}

Acknowledgement

SoBigData.it - Research Infrastructure

“Digging into the Landscape of Graphs Counterfactual Explainability (Laboratory at AAAI 2024)” event was organised as part of the SoBigData.it project (Prot. IR0000013 - Call n. 3264 of 12/28/2021) initiatives aimed at training new users and communities in the usage of the research infrastructure (SoBigData.eu). SoBigData.it receives funding from European Union – NextGenerationEU – National Recovery and Resilience Plan (Piano Nazionale di Ripresa e Resilienza, PNRR) – Project: “SoBigData.it – Strengthening the Italian RI for Social Mining and Big Data Analytics” – Prot. IR0000013 – Avviso n. 3264 del 28/12/2021.

Meet the Speakers

Mario Alfonso Prado-Romero

is a PhD fellow specialising in AI at the esteemed Gran Sasso Science Institute in Italy. His primary research focuses on the confluence of Explainable AI and Graph Neural Networks. Before this, he gained experience in relevant fields such as Anomaly Detection, Data Mining, and Information Retrieval. Notably, he is the key contributor to the GRETEL project, which offers a comprehensive framework for developing and assessing Graph Counterfactual Explanations. Additionally, he was selected by NEC Laboratories Europe as the only intern of the Human-Centric AI group in 2023, where his expertise in eXplainable Artificial Intelligence (XAI) will be applied to Graph Neural Networks for Biomedical Applications.

Bardh Prenkaj

obtained his M.Sc. and PhD in Computer Science from the Sapienza University of Rome in 2018 and 2022, respectively. He then worked as a senior researcher at the Department of Computer Science in Sapienza. He led a team of four junior researchers and six software engineers in devising novel anomaly detection strategies for social isolation disorders. Since October 2022, he has been a postdoc at Sapienza in eXplainable AI and Anomaly Detection. He is part of and actively collaborates with Italian and international research groups (RDS of TUM, PINlab of Sapienza, AIIM of UnivAQ, and DMML of GMU). He serves as a Program Committee (PC) member for conferences such as ICCV, CVPR, KDD, CIKM, and ECAI. He actively contributes as a reviewer for journals, including TKDE, TKDD, VLDB, TIST, and KAIS.

Giovanni Stilo

is a Computer Science and Data Science associate professor at the University of L’Aquila, where he leads the Master’s Degree in Data Science, and he is part of the Artificial Intelligence and Information Mining collective. He received his PhD in Computer Science in 2013, and in 2014, he was a visiting researcher at Yahoo! Labs in Barcelona. His research interests are related to machine learning, data mining, and artificial intelligence, with a special interest in (but not limited to) trustworthiness aspects such as Bias, Fairness, and Explainability. Specifically, he is the head of the GRETEL project devoted to empowering the research in the Graph Counterfactual Explainability field. He has co-organized a long series (2020-2023) of top-tier International events and Journal Special Issues focused on Bias and Fairness in Search and Recommendation. He serves on the editorial boards of IEEE, ACM, Springer, and Elsevier Journals such as TITS, TKDE, DMKD, AI, KAIS, and AIIM. He is responsible for New technologies for data collection, preparation, and analysis of the Territory Aperti project and coordinator of the activities on “Responsible Data Science and Training” of PNRR SoBigData.it project, and PI of the “FAIR-EDU: Promote FAIRness in EDUcation Institutions” project. During his academic growth, he devoted much of his attention to teaching and tutoring, where he worked on more than 30 different national and international theses (of B.Sc., M.Sc., and PhD levels). In more than ten years of academia, he provided university-level courses for ten different topics and grades in the scientific field of Computer and Data Science.

Bibliography

[1] Mario Alfonso Prado-Romero and Giovanni Stilo. 2022. GRETEL: Graph Counterfactual Explanation Evaluation Framework. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management (CIKM ‘22). Association for Computing Machinery, New York, NY, USA, 4389–4393. https://doi.org/10.1145/3511808.3557608

[2] Y. Liu, C. Chen, Y. Liu, X. Zhang and S. Xie, Multi-objective Explanations of GNN Predictions, 2021 IEEE International Conference on Data Mining (ICDM), Auckland, New Zealand, 2021, pp. 409-418, https://doi.org/10.1109/ICDM51629.2021.00052

[3] Carlo Abrate and Francesco Bonchi. 2021. Counterfactual Graphs for Explainable Classification of Brain Networks. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (KDD ‘21). Association for Computing Machinery, New York, NY, USA, 2495–2504. https://doi.org/10.1145/3447548.3467154

[4] Bajaj, M., Chu, L., Xue, Z., Pei, J., Wang, L., Lam, P.,and Zhang, Y. Robust counterfactual explanations on graph neural networks. Advances in neural information processing systems, 34.

[5] Wellawatte GP, Seshadri A, White AD. Model agnostic generation of counterfactual explanations for molecules. Chem Sci. 2022 Feb 16;13(13):3697-3705. https://doi.org/10.1039/d1sc05259d

[6] Cai, R., Zhu, Y., Chen, X., Fang, Y., Wu, M., Qiao,J., and Hao, Z. 2022. On the Probability of Necessity and Sufficiency of Explaining Graph Neural Networks: A Lower Bound Optimization Approach. arXiv preprint arXiv:2212.07056

[7] Ana Lucic, Maartje A. Ter Hoeve, Gabriele Tolomei, Maarten De Rijke, Fabrizio Silvestri, CF-GNNExplainer: Counterfactual Explanations for Graph Neural Networks, in Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, PMLR 151:4499-4511, 2022.

[8] Ma, J.; Guo, R.; Mishra, S.; Zhang, A.; and Li, J., 2022. CLEAR: Generative Counterfactual Explanations on Graphs. Advances in neural information processing systems, 35, 25895–25907.

[9] T. M. Nguyen, T. P. Quinn, T. Nguyen and T. Tran, Explaining Black Box Drug Target Prediction Through Model Agnostic Counterfactual Samples, in IEEE/ACM Transactions on Computational Biology and Bioinformatics, vol. 20, no. 2, pp. 1020-1029, 1 March-April 2023, https://doi.org/10.1109/TCBB.2022.3190266

[10] Numeroso, Danilo and Bacciu, Davide, MEG: Generating Molecular Counterfactual Explanations for Deep Graph Networks, 2021 International Joint Conference on Neural Networks (IJCNN). https://doi.org/10.1109/IJCNN52387.2021.9534266

[11] Sun, Y., Valente, A., Liu, S., & Wang, D. (2021). Preserve, Promote, or Attack? GNN Explanation via Topology Perturbation. ArXiv. https://arxiv.org/abs/2103.13944

[12] Juntao Tan, Shijie Geng, Zuohui Fu, Yingqiang Ge, Shuyuan Xu, Yunqi Li, and Yongfeng Zhang. 2022. Learning and Evaluating Graph Neural Network Explanations based on Counterfactual and Factual Reasoning. In Proceedings of the ACM Web Conference 2022 (WWW ‘22).https://doi.org/10.1145/3485447.3511948

[13] Wu, H.; Chen, W.; Xu, S.; and Xu, B. 2021. Counterfactual Supporting Facts Extraction for Explainable Medical Record Based Diagnosis with Graph Network. In Proc. of the 2021 Conf. of the North American Chapter of the Assoc. for Comp. Linguistics: Human Lang. Techs., 1942–1955. https://aclanthology.org/2021.naacl-main.156/

[14] Zexi Huang, Mert Kosan, Sourav Medya, Sayan Ranu, and Ambuj Singh. 2023. Global Counterfactual Explainer for Graph Neural Networks. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining (WSDM ‘23). Association for Computing Machinery, New York, NY, USA, 141–149. https://doi.org/10.1145/3539597.3570376

[15] Mario Alfonso Prado-Romero, Bardh Prenkaj, Giovanni Stilo, and Fosca Giannotti. A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges. ACM Comput. Surv. (September 2023). https://doi.org/10.1145/3618105

[16] Mario Alfonso Prado-Romero, Bardh Prenkaj, and Giovanni Stilo. 2023. Developing and Evaluating Graph Counterfactual Explanation with GRETEL. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining (WSDM ‘23). Association for Computing Machinery, New York, NY, USA, 1180–1183. https://doi.org/10.1145/3539597.3573026

© Copyright 2019-2024 - Artificial Intelligence and Information Mining - All rights reserved.