m typo: Association for Computing Machinery (via WP:JWB) |
copy edits for conciseness and clarity |
||
Line 1: | Line 1: | ||
{{short description|AI in which the results of the solution can be understood by humans}} |
{{short description|AI in which the results of the solution can be understood by humans}} |
||
'''Explainable AI''' ('''XAI'''), or '''Interpretable AI''', or '''Explainable Machine Learning''' ('''XML'''),<ref>{{Cite journal |last1=Phillips |first1=P. Jonathon |last2=Hahn |first2=Carina A. |last3=Fontana |first3=Peter C. |last4=Yates |first4=Amy N. |last5=Greene |first5=Kristen |last6=Broniatowski |first6=David A. |last7=Przybocki |first7=Mark A. |date=2021-09-29 |title=Four Principles of Explainable Artificial Intelligence |url=https://doi.org/10.6028/NIST.IR.8312 |doi=10.6028/nist.ir.8312}}</ref> is [[artificial intelligence]] (AI) in which humans can understand the decisions or predictions made by the AI.<ref>{{Cite journal|last1=Vilone|first1=Giulia|last2=Longo|first2=Luca|title=Notions of explainability and evaluation approaches for explainable artificial intelligence|url=https://www.sciencedirect.com/science/article/pii/S1566253521001093|journal=Information Fusion|year=2021|volume= December 2021 - Volume 76 |pages=89–106|doi=10.1016/j.inffus.2021.05.009}}</ref> It contrasts with the "[[black box]]" concept in machine learning where even |
'''Explainable AI''' ('''XAI'''), or '''Interpretable AI''', or '''Explainable Machine Learning''' ('''XML'''),<ref>{{Cite journal |last1=Phillips |first1=P. Jonathon |last2=Hahn |first2=Carina A. |last3=Fontana |first3=Peter C. |last4=Yates |first4=Amy N. |last5=Greene |first5=Kristen |last6=Broniatowski |first6=David A. |last7=Przybocki |first7=Mark A. |date=2021-09-29 |title=Four Principles of Explainable Artificial Intelligence |url=https://doi.org/10.6028/NIST.IR.8312 |doi=10.6028/nist.ir.8312}}</ref> is [[artificial intelligence]] (AI) in which humans can understand the reasoning behind decisions or predictions made by the AI.<ref>{{Cite journal|last1=Vilone|first1=Giulia|last2=Longo|first2=Luca|title=Notions of explainability and evaluation approaches for explainable artificial intelligence|url=https://www.sciencedirect.com/science/article/pii/S1566253521001093|journal=Information Fusion|year=2021|volume= December 2021 - Volume 76 |pages=89–106|doi=10.1016/j.inffus.2021.05.009}}</ref> It contrasts with the "[[black box]]" concept in machine learning where even the AI's designers cannot explain why it arrived at a specific decision.<ref>{{Cite journal |last=Castelvecchi |first=Davide |date=2016-10-06 |title=Can we open the black box of AI? |url=http://www.nature.com/articles/538020a |journal=Nature |language=en |volume=538 |issue=7623 |pages=20–23 |doi=10.1038/538020a |pmid=27708329 |bibcode=2016Natur.538...20C |s2cid=4465871 |issn=0028-0836}}</ref><ref name=guardian>{{cite news|last1=Sample|first1=Ian|title=Computer says no: why making AIs fair, accountable and transparent is crucial|url=https://www.theguardian.com/science/2017/nov/05/computer-says-no-why-making-ais-fair-accountable-and-transparent-is-crucial|access-date=30 January 2018|work=The Guardian |date=5 November 2017|language=en}}</ref> |
||
XAI hopes to help users of AI-powered systems perform more effectively by improving their understanding of how those systems reason.<ref>{{Cite journal|last=Alizadeh|first=Fatemeh|date=2021|title=I Don't Know, Is AI Also Used in Airbags?: An Empirical Study of Folk Concepts and People's Expectations of Current and Future Artificial Intelligence|url=https://www.researchgate.net/publication/352638184|journal=Icom|volume=20 |issue=1 |pages=3–17 |doi=10.1515/icom-2021-0009|s2cid=233328352}}</ref> XAI may be an implementation of the social [[right to explanation]].<ref name=":0">{{Cite journal|last1=Edwards|first1=Lilian|last2=Veale|first2=Michael|date=2017|title=Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For|journal=Duke Law and Technology Review|volume=16|pages=18|ssrn=2972855}}</ref> Even if there is no such legal right or regulatory requirement, XAI can improve the [[user experience]] of a product or service by helping end users trust that the AI is making good decisions. XAI aims to explain what has been done, what is being done, and what will be done next, and to unveil which information these actions are based on.<ref name=":3">{{Cite journal|last1=Gunning|first1=D.|last2=Stefik|first2=M.|last3=Choi|first3=J.|last4=Miller|first4=T.|last5=Stumpf|first5=S.|last6=Yang|first6=G.-Z.|date=2019-12-18|title=XAI-Explainable artificial intelligence|url=https://openaccess.city.ac.uk/id/eprint/23405/|journal=Science Robotics|language=en|volume=4|issue=37|pages=eaay7120|doi=10.1126/scirobotics.aay7120|pmid=33137719|issn=2470-9476|doi-access=free}}</ref> This makes it possible to confirm existing knowledge, to challenge existing knowledge, and to generate new assumptions.<ref>{{Cite journal|last1=Rieg|first1=Thilo|last2=Frick|first2=Janek|last3=Baumgartl|first3=Hermann|last4=Buettner|first4=Ricardo|date=2020-12-17|title=Demonstration of the potential of white-box machine learning approaches to gain insights from cardiovascular disease electrocardiograms|journal=PLOS ONE|language=en|volume=15|issue=12|pages=e0243615|doi=10.1371/journal.pone.0243615|issn=1932-6203|pmc=7746264|pmid=33332440|bibcode=2020PLoSO..1543615R|doi-access=free}}</ref> |
|||
⚫ | |||
⚫ | [[Machine learning]] (ML) algorithms used in AI can be categorized as [[White-box testing|white-box]] or black-box.<ref>{{Cite journal|last1=Vilone|first1=Giulia|last2=Longo|first2=Luca|title= Classification of Explainable Artificial Intelligence Methods through Their Output Formats |journal=Machine Learning and Knowledge Extraction|year=2021|volume=3|issue=3|pages=615–661|doi=10.3390/make3030032|doi-access=free }}</ref> White-box models provide results that are understandable for experts in the domain. Black-box models, on the other hand, are extremely hard to explain and can hardly be understood even by domain experts.<ref>{{Cite journal|last=Loyola-González|first=O.|date=2019|title=Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View|journal=IEEE Access|volume=7|pages=154096–154113|doi=10.1109/ACCESS.2019.2949286|issn=2169-3536|doi-access=free}}</ref> XAI algorithms follow the three principles of transparency, interpretability, and explainability. A model is transparent “if the processes that extract model parameters from training data and generate labels from testing data can be described and motivated by the approach designer”.<ref name=":4">{{Cite journal|last1=Roscher|first1=R.|last2=Bohn|first2=B.|last3=Duarte|first3=M. F.|last4=Garcke|first4=J.|date=2020|title=Explainable Machine Learning for Scientific Insights and Discoveries|journal=IEEE Access|volume=8|pages=42200–42216|doi=10.1109/ACCESS.2020.2976199|arxiv=1905.08883 |issn=2169-3536|doi-access=free}}</ref> Interpretability describes the possibility of comprehending the ML model and presenting the underlying basis for decision-making in a way that is understandable to humans.<ref name="Interpretable machine learning: def">{{cite journal|last1=Murdoch|first1=W. James|last2=Singh|first2=Chandan|last3=Kumbier|first3=Karl|last4=Abbasi-Asl|first4=Reza|last5=Yu|first5=Bin|date=2019-01-14|title=Interpretable machine learning: definitions, methods, and applications|journal=Proceedings of the National Academy of Sciences of the United States of America|volume=116|issue=44|pages=22071–22080|arxiv=1901.04592|doi=10.1073/pnas.1900654116|pmid=31619572|pmc=6825274|bibcode=2019arXiv190104592M|doi-access=free}}</ref><ref name="Lipton 31–57">{{Cite journal|last=Lipton|first=Zachary C.|date=June 2018|title=The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery.|journal=Queue|language=en|volume=16|issue=3|pages=31–57|doi=10.1145/3236386.3241340|issn=1542-7730|doi-access=free}}</ref><ref>{{Cite web|date=2019-10-22|title=Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI|url=https://deepai.org/publication/explainable-artificial-intelligence-xai-concepts-taxonomies-opportunities-and-challenges-toward-responsible-ai|access-date=2021-01-13|website=DeepAI}}</ref> Explainability is a concept that is recognized as important, but a consensus definition is not available.<ref name=":4" /> One possibility is: “the collection of features of the interpretable domain, that have contributed for a given example to produce a decision (e.g., classification or regression)”.<ref>{{Cite journal|date=2018-02-01|title=Methods for interpreting and understanding deep neural networks|journal=Digital Signal Processing|language=en|volume=73|pages=1–15|doi=10.1016/j.dsp.2017.10.011|issn=1051-2004|doi-access=free|last1=Montavon|first1=Grégoire|last2=Samek|first2=Wojciech|last3=Müller|first3=Klaus-Robert|author-link3=Klaus-Robert Müller}}</ref> If algorithms fulfil these principles, they provide a basis for justifying decisions, tracking and thereby verifying them, improving the algorithms, and exploring new facts.<ref>{{Cite journal|last1=Adadi|first1=A.|last2=Berrada|first2=M.|date=2018|title=Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)|journal=IEEE Access|volume=6|pages=52138–52160|doi=10.1109/ACCESS.2018.2870052|issn=2169-3536|doi-access=free}}</ref> |
||
⚫ | Sometimes it is also possible to achieve a result with high accuracy with a white-box ML algorithm that is interpretable in itself.<ref name=":6">{{Cite journal|last=Rudin|first=Cynthia|date=2019|title=Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead|journal=Nature Machine Intelligence|language=en|volume=1|issue=5|pages=206–215|doi=10.1038/s42256-019-0048-x|pmid=35603010 |pmc=9122117 |arxiv=1811.10154|issn=2522-5839|doi-access=free}}</ref> This is especially important in domains like medicine, defense, finance and law, where it is crucial to understand |
||
⚫ | Sometimes it is also possible to achieve a result with high accuracy with a white-box ML algorithm that is interpretable {{clarify|text=in itself}}.<ref name=":6">{{Cite journal|last=Rudin|first=Cynthia|date=2019|title=Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead|journal=Nature Machine Intelligence|language=en|volume=1|issue=5|pages=206–215|doi=10.1038/s42256-019-0048-x|pmid=35603010 |pmc=9122117 |arxiv=1811.10154|issn=2522-5839|doi-access=free}}</ref> This is especially important in domains like medicine, defense, finance, and law, where it is crucial to understand decisions and build trust in the algorithms.<ref name=":3" /> Many researchers argue that, at least for supervised machine learning, the way forward is [[symbolic regression]], where the algorithm searches the space of mathematical expressions to find the model that best fits a given dataset.<ref name="Wenninger Kaymakci Wiethe 2022 p=118300">{{cite journal | last1=Wenninger | first1=Simon | last2=Kaymakci | first2=Can | last3=Wiethe | first3=Christian | title=Explainable long-term building energy consumption prediction using QLattice | journal=Applied Energy | publisher=Elsevier BV | volume=308 | year=2022 | issn=0306-2619 | doi=10.1016/j.apenergy.2021.118300 | page=118300| s2cid=245428233 }}</ref><ref name="Christiansen Wilstrup Hedley 2022 p. ">{{cite journal | last1=Christiansen | first1=Michael | last2=Wilstrup | first2=Casper | last3=Hedley | first3=Paula L. | title=Explainable "white-box" machine learning is the way forward in preeclampsia screening | journal=American Journal of Obstetrics and Gynecology | publisher=Elsevier BV | year=2022 | volume=227 | issue=5 | issn=0002-9378 | doi=10.1016/j.ajog.2022.06.057 | page=791| pmid=35779588 | s2cid=250160871 }}</ref><ref name="Wilstup Cave p. ">{{citation | last1=Wilstup | first1=Casper | last2=Cave | first2=Chris | title=Combining symbolic regression with the Cox proportional hazards model improves prediction of heart failure deaths | publisher=Cold Spring Harbor Laboratory | date=2021-01-15 | doi=10.1101/2021.01.15.21249874 | page=| s2cid=231609904 }}</ref> |
||
⚫ | AI systems optimize behavior to satisfy a mathematically specified goal system chosen by the system designers, such as the command "maximize accuracy of [[sentiment analysis|assessing how positive]] film reviews are in the test dataset". The AI may learn useful general rules from the test set, such as "reviews containing the word 'horrible' are likely to be negative". However, it may also learn inappropriate rules, such as "reviews containing '[[Daniel Day-Lewis]]' are usually positive"; such rules may be undesirable if they are |
||
⚫ | AI systems optimize behavior to satisfy a mathematically specified goal system chosen by the system designers, such as the command "maximize accuracy of [[sentiment analysis|assessing how positive]] film reviews are in the test dataset". The AI may learn useful general rules from the test set, such as "reviews containing the word 'horrible' are likely to be negative". However, it may also learn inappropriate rules, such as "reviews containing '[[Daniel Day-Lewis]]' are usually positive"; such rules may be undesirable if they are likely to fail to generalize outside the training set, or if people consider the rule to be "cheating" or "unfair". A human can audit rules in an XAI to get an idea how likely the system is to generalize to future real-world data outside the test-set.<ref name="science">{{cite journal|date=5 July 2017|title=How AI detectives are cracking open the black box of deep learning|url=https://www.science.org/content/article/how-ai-detectives-are-cracking-open-black-box-deep-learning|journal=Science|language=en|access-date=30 January 2018}}.</ref> |
||
== Goals == |
== Goals == |
||
Cooperation between [[Agency (sociology)|agents]], in this case algorithms and humans, depends on trust. If humans are to accept algorithmic prescriptions, they need to trust them. Incompleteness in formalization of trust criteria is a barrier to straightforward optimization approaches. |
Cooperation between [[Agency (sociology)|agents]], in this case algorithms and humans, depends on trust. If humans are to accept algorithmic prescriptions, they need to trust them. Incompleteness in formalization of trust criteria is a barrier to straightforward optimization approaches.{{copy edit inline|reason=for better clarity, consider rewriting this sentence so that its nounified verbs (formalization, optimization) are verb phrases with concrete subjects}} Transparency, interpretability and explainability are intermediate goals on the road to these more comprehensive trust criteria.<ref name=dosilovic2018>{{cite conference |url=http://docs.mipro-proceedings.com/dsdc/dsdc_11_4754.pdf |title=Explainable Artificial Intelligence: A Survey |last1=Dosilovic |first1=Filip |last2=Brcic |first2=Mario |last3=Hlupic |first3=Nikica |date=2018-05-25 |book-title=MIPRO 2018 - 41st International Convention Proceedings |pages=210–215 |location=Opatija, Croatia |conference=MIPRO 2018 |doi=10.23919/MIPRO.2018.8400040 |isbn=978-953-233-095-3 |access-date=2018-12-09 |archive-date=2018-12-10 |archive-url=https://web.archive.org/web/20181210110820/http://docs.mipro-proceedings.com/dsdc/dsdc_11_4754.pdf |url-status=dead }}</ref> This is particularly relevant in medicine,<ref>{{Cite journal |last1=Bernal |first1=Jose |last2=Mazo |first2=Claudia |date=2022-10-11 |title=Transparency of Artificial Intelligence in Healthcare: Insights from Professionals in Computing and Healthcare Worldwide |journal=Applied Sciences |language=en |volume=12 |issue=20 |pages=10228 |doi=10.3390/app122010228 |issn=2076-3417|doi-access=free }}</ref> especially [[clinical decision support system]]s (CDSS), in which medical professionals should be able to understand how and why a machine-based decision was made in order to trust the decision and augment their decision-making process.<ref>{{Cite journal |last1=Antoniadi |first1=Anna Markella |last2=Du |first2=Yuhan |last3=Guendouz |first3=Yasmine |last4=Wei |first4=Lan |last5=Mazo |first5=Claudia |last6=Becker |first6=Brett A. |last7=Mooney |first7=Catherine |date=January 2021 |title=Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review |journal=Applied Sciences |language=en |volume=11 |issue=11 |pages=5088 |doi=10.3390/app11115088 |issn=2076-3417|doi-access=free }}</ref> |
||
AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit pre-programmed goals on the training data, but that do not reflect the |
AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit pre-programmed goals on the training data, but that do not reflect the more nuanced implicit desires of the human system designers or the full complexity of the domain data. For example, a 2017 system tasked with [[image recognition]] learned to "cheat" by looking for a copyright tag that happened to be associated with horse pictures, rather than learning how to tell if a horse was actually pictured.<ref name=guardian/> In another 2017 system, a [[supervised learning]] AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.<ref>{{cite news|title=DeepMind Has Simple Tests That Might Prevent Elon Musk's AI Apocalypse|url=https://www.bloomberg.com/news/articles/2017-12-11/deepmind-has-simple-tests-that-might-prevent-elon-musk-s-ai-apocalypse|access-date=30 January 2018|work=Bloomberg.com|date=11 December 2017|language=en}}</ref><ref>{{cite news|title=Learning from Human Preferences|url=https://blog.openai.com/deep-reinforcement-learning-from-human-preferences/|access-date=30 January 2018|work=OpenAI Blog|date=13 June 2017}}</ref> |
||
One transparency project, the [[DARPA]] XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop", without greatly sacrificing AI performance. Human users |
One transparency project, the [[DARPA]] XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop", without greatly sacrificing AI performance. Human users of such a system can understand the AI's cognition (both in real-time and after the fact), and can determine whether to trust the AI.<ref>{{cite web|title=Explainable Artificial Intelligence (XAI)|url=https://www.darpa.mil/program/explainable-artificial-intelligence|website=DARPA|publisher=DARPA|access-date=17 July 2017}}</ref> Other applications of XAI are [[knowledge extraction]] from black-box models and model comparisons.<ref>{{cite journal|last=Biecek|first=Przemyslaw|title= DALEX: explainers for complex predictive models|journal=Journal of Machine Learning Research|volume=19|pages=1–5|arxiv=1806.08915|date=23 June 2018|bibcode=2018arXiv180608915B}}</ref> The term "glass box" is also used to describe tools that monitor the inputs and outputs of a system, with the purpose of verifying the system's adherence to ethical and socio-legal values and that produce value-based explanations.<ref name="Aler Tubella Theodorou Dignum Dignum 2019 p. ">{{cite conference | last1=Aler Tubella | first1=Andrea | last2=Theodorou | first2=Andreas | last3=Dignum | first3=Frank | last4=Dignum | first4=Virginia | title=Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence | chapter=Governance by Glass-Box: Implementing Transparent Moral Bounds for AI Behaviour | publisher=International Joint Conferences on Artificial Intelligence Organization | location=California | year=2019 | pages=5787–5793 | isbn=978-0-9992411-4-1 | doi=10.24963/ijcai.2019/802 | doi-access=free }}</ref> The term is also used to name a voice assistant that produces counterfactual statements as explanations.<ref name="SokolFlach2018">{{cite book|last1=Sokol|first1=Kacper|last2=Flach|first2=Peter|title=Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence|chapter=Glass-Box: Explaining AI Decisions With Counterfactual Statements Through Conversation With a Voice-enabled Virtual Assistant|year=2018|pages=5868–5870|doi=10.24963/ijcai.2018/865|isbn=9780999241127|s2cid=51608978}}</ref> |
||
== History and methods == |
== History and methods == |
||
During the 1970s to 1990s, [[Symbolic artificial intelligence|symbolic reasoning systems]], such as [[Mycin|MYCIN]],<ref>{{cite journal |last1=Fagan |first1=L. M. |last2=Shortliffe |first2=E. H. |last3=Buchanan |first3=B. G. |year=1980 |title=Computer-based medical decision making: from MYCIN to VM |journal=Automedica |volume=3 |issue=2 |pages=97–108}}</ref> GUIDON,<ref>{{Cite book| publisher = The MIT Press| last = Clancey| first = William| title = Knowledge-Based Tutoring: The GUIDON Program| location = Cambridge, Massachusetts| date = 1987}}</ref> SOPHIE,<ref>{{Cite book| publisher = Academic Press| isbn = 0-12-648680-8| last1 = Brown| first1 = John S.| last2 = Burton| first2 = R. R.| last3 = De Kleer| first3 = Johan| title = Intelligent Tutoring Systems| chapter = Pedagogical, natural language, and knowledge engineering techniques in SOPHIE I, II, and III| date = 1982}}</ref> and PROTOS<ref>{{Cite book| publisher = Morgan Kaufmann Publishers Inc.| isbn = 1-55860-119-8| volume = 3| pages = 112–139| last1 = Bareiss| first1 = Ray| last2 = Porter| first2 = Bruce| last3 = Weir| first3 = Craig| last4 = Holte| first4 = Robert| title = Machine Learning| chapter = Protos: An Exemplar-Based Learning Apprentice| date = 1990}}</ref><ref>{{Cite book| last = Bareiss, Ray| title = Exemplar-Based Knowledge Acquisition: A Unified Approach to Concept Representation, Classification, and Learning| series = Perspectives in Artificial Intelligence}}</ref> |
During the 1970s to 1990s, [[Symbolic artificial intelligence|symbolic reasoning systems]], such as [[Mycin|MYCIN]],<ref>{{cite journal |last1=Fagan |first1=L. M. |last2=Shortliffe |first2=E. H. |last3=Buchanan |first3=B. G. |year=1980 |title=Computer-based medical decision making: from MYCIN to VM |journal=Automedica |volume=3 |issue=2 |pages=97–108}}</ref> GUIDON,<ref>{{Cite book| publisher = The MIT Press| last = Clancey| first = William| title = Knowledge-Based Tutoring: The GUIDON Program| location = Cambridge, Massachusetts| date = 1987}}</ref> SOPHIE,<ref>{{Cite book| publisher = Academic Press| isbn = 0-12-648680-8| last1 = Brown| first1 = John S.| last2 = Burton| first2 = R. R.| last3 = De Kleer| first3 = Johan| title = Intelligent Tutoring Systems| chapter = Pedagogical, natural language, and knowledge engineering techniques in SOPHIE I, II, and III| date = 1982}}</ref> and PROTOS<ref>{{Cite book| publisher = Morgan Kaufmann Publishers Inc.| isbn = 1-55860-119-8| volume = 3| pages = 112–139| last1 = Bareiss| first1 = Ray| last2 = Porter| first2 = Bruce| last3 = Weir| first3 = Craig| last4 = Holte| first4 = Robert| title = Machine Learning| chapter = Protos: An Exemplar-Based Learning Apprentice| date = 1990}}</ref><ref>{{Cite book| last = Bareiss, Ray| title = Exemplar-Based Knowledge Acquisition: A Unified Approach to Concept Representation, Classification, and Learning| series = Perspectives in Artificial Intelligence}}</ref> could represent, reason about, and explain their reasoning for diagnostic, instructional, or machine-learning (explanation-based learning) purposes. MYCIN, developed in the early 1970s as a research prototype for diagnosing [[bacteremia]] infections of the bloodstream, could explain<ref>{{cite book |last1=Van Lent |first1=M. |last2=Fisher |first2=W. |last3=Mancuso |first3=M. |date=July 2004 |chapter=An explainable artificial intelligence system for small-unit tactical behavior |title=Proceedings of the National Conference on Artificial Intelligence |pages=900–907 |location=San Jose, CA |publisher=AAAI Press |isbn=0262511835}}</ref> which of its hand-coded rules contributed to a diagnosis in a specific case. Research in [[intelligent tutoring systems]] resulted in developing systems such as SOPHIE that could act as an "articulate expert", explaining problem-solving strategy at a level the student could understand, so they would know what action to take next. For instance, SOPHIE could explain the qualitative reasoning behind its electronics troubleshooting, even though it ultimately relied on the [[SPICE]] circuit simulator. Similarly, GUIDON added tutorial rules to supplement MYCIN's domain-level rules so it could explain strategy for medical diagnosis. Symbolic approaches to machine learning, especially those relying on explanation-based learning, such as PROTOS, explicitly relied on representations of explanations, both to explain their actions and to acquire new knowledge.{{clarify|reason=unclear what this last sentence means e.g. what are "representations of explanations"?}} |
||
In the 1980s through early 1990s, truth maintenance systems (TMS) |
In the 1980s through early 1990s, truth maintenance systems (TMS) extended the capabilities of causal-reasoning, [[Rule-based system|rule-based]], and logic-based inference systems.<ref>{{Cite book| edition = Second| publisher = Prentice Hall, Pearson Education| isbn = 0-13-790395-2| last1 = Russell| first1 = Stuart| last2 = Norvig| first2 = Peter| title = Artificial Intelligence: A Modern Approach| location = Upper Saddle River, New Jersey| series = Prentice Hall Series in Artificial Intelligence| date = 2003}}</ref>{{rp|360–362}} A TMS explicitly tracks alternate lines of reasoning, justifications for conclusions, and lines of reasoning that lead to contradictions, allowing future reasoning to avoid these dead ends. To provide explanation, they trace reasoning from conclusions to assumptions through rule operations or logical inferences, allowing explanations to be generated from the reasoning traces. As an example, consider a rule-based problem solver with just a few rules about Socrates that concludes he has died from poison: |
||
{{Blockquote |
{{Blockquote |
||
|text=By just tracing through the dependency structure the problem solver can construct the following explanation: "Socrates died because he was mortal and drank poison, and all mortals die when they drink poison. |
|text=By just tracing through the dependency structure the problem solver can construct the following explanation: "Socrates died because he was mortal and drank poison, and all mortals die when they drink poison. Socrates was mortal because he was a man and all men are mortal. Socrates drank poison because he held dissident beliefs, the government was conservative, and those holding conservative dissident beliefs under conservative governments must drink poison."<ref name="RMS">{{Cite book| publisher = The MIT Press| isbn = 0-262-06157-0| last1 = Forbus| first1 = Kenneth| last2 = De Kleer| first2 = Johan| title = Building Problem Solvers| location = Cambridge, Massachusetts| date = 1993}}</ref>{{rp|164–165}} |
||
}} |
}} |
||
By the 1990s researchers |
By the 1990s researchers began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks.<ref>{{Cite journal|last1=Tickle|first1=A. B.|last2=Andrews|first2=R.|last3=Golea|first3=M.|last4=Diederich|first4=J.|date=November 1998|title=The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks|journal=IEEE Transactions on Neural Networks|volume=9|issue=6|pages=1057–1068|doi=10.1109/72.728352|pmid=18255792|issn=1045-9227}}</ref> Researchers in clinical [[expert system]]s creating{{clarify|reason=who is creating? the researchers or the expert systems?}} neural network-powered decision support for clinicians sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice.<ref name=":0" /> In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence.<ref name="guardian" /> As a result, many academics and organizations are developing tools to help detect bias in their systems.<ref>{{cite news|url=https://www.bloomberg.com/news/articles/2018-06-13/accenture-unveils-tool-to-help-companies-insure-their-ai-is-fair|title=Accenture Unveils Tool to Help Companies Insure Their AI Is Fair|date=June 2018|work=Bloomberg.com|access-date=5 August 2018|language=en}}</ref> |
||
[[Marvin Minsky]] et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced "human-in-the-loop" AI.<ref>Minsky, et al., "The Society of Intelligent Veillance" IEEE ISTAS2013, pages 13-17.</ref> |
[[Marvin Minsky]] et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced "human-in-the-loop" AI.<ref>Minsky, et al., "The Society of Intelligent Veillance" IEEE ISTAS2013, pages 13-17.</ref> |
||
Modern complex AI techniques, such as [[deep learning]] and genetic algorithms are naturally opaque.<ref>{{cite magazine|last1=Mukherjee|first1=Siddhartha|title=A.I. Versus M.D.|url=https://www.newyorker.com/magazine/2017/04/03/ai-versus-md|access-date=30 January 2018|magazine=The New Yorker|date=27 March 2017}}</ref> To address this issue, |
Modern complex AI techniques, such as [[deep learning]] and genetic algorithms, are naturally opaque.<ref>{{cite magazine|last1=Mukherjee|first1=Siddhartha|title=A.I. Versus M.D.|url=https://www.newyorker.com/magazine/2017/04/03/ai-versus-md|access-date=30 January 2018|magazine=The New Yorker|date=27 March 2017}}</ref> To address this issue, methods have been developed to make new models more explainable and interpretable.<ref>{{Cite journal|date=2020-07-08|title=Interpretable neural networks based on continuous-valued logic and multicriteria decision operators|journal=Knowledge-Based Systems|language=en|volume=199|pages=105972|doi=10.1016/j.knosys.2020.105972 |arxiv=1910.02486 |issn=0950-7051|doi-access=free|last1=Csiszár|first1=Orsolya|last2=Csiszár|first2=Gábor|last3=Dombi|first3=József}}</ref><ref name="Lipton 31–57"/><ref name="Interpretable machine learning: def"/><ref>{{cite arXiv|last1=Doshi-Velez|first1=Finale|last2=Kim|first2=Been|date=2017-02-27|title=Towards A Rigorous Science of Interpretable Machine Learning|eprint=1702.08608|class=stat.ML}}</ref><ref>{{Cite arXiv |last=Abdollahi, Behnoush, and Olfa Nasraoui.|title=Explainable Restricted Boltzmann Machines for Collaborative Filtering.|eprint=1606.07129|class=stat.ML|year=2016}}</ref><ref>{{Cite journal|last1=Dombi|first1=József|last2=Csiszár|first2=Orsolya|date=2021|title=Explainable Neural Networks Based on Fuzzy Logic and Multi-criteria Decision Tools|url=https://link.springer.com/book/10.1007/978-3-030-72280-7|journal=Studies in Fuzziness and Soft Computing|volume=408|language=en-gb|doi=10.1007/978-3-030-72280-7|isbn=978-3-030-72279-1|s2cid=233486978|issn=1434-9922}}</ref> This includes layerwise relevance propagation (LRP), a technique for determining which features in a particular input vector contribute most strongly to a neural network's output.<ref name="Bach Binder Montavon Klauschen p=e0130140">{{cite journal|last1=Bach|first1=Sebastian|last2=Binder|first2=Alexander|last3=Montavon|first3=Grégoire|last4=Klauschen|first4=Frederick|last5=Müller|first5=Klaus-Robert|author-link5=Klaus-Robert Müller|last6=Samek|first6=Wojciech|date=2015-07-10|editor-last=Suarez|editor-first=Oscar Deniz|title=On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation|journal=PLOS ONE|volume=10|issue=7|page=e0130140|bibcode=2015PLoSO..1030140B|doi=10.1371/journal.pone.0130140|issn=1932-6203|pmc=4498753|pmid=26161953|doi-access=free}}</ref><ref>{{cite news|url=https://www.theguardian.com/science/2017/nov/05/computer-says-no-why-making-ais-fair-accountable-and-transparent-is-crucial|title=Computer says no: why making AIs fair, accountable and transparent is crucial|last1=Sample|first1=Ian|date=5 November 2017|work=The Guardian|access-date=5 August 2018|language=en}}</ref> Other techniques explain some particular prediction made by a (nonlinear) black-box model, a goal referred to as "local interpretability".<ref>{{Cite journal|last1=Martens|first1=David|last2=Provost|first2=Foster|title=Explaining data-driven document classifications|url=http://pages.stern.nyu.edu/~fprovost/Papers/MartensProvost_Explaining.pdf|journal=MIS Quarterly|year=2014|volume=38|pages=73–99|doi=10.25300/MISQ/2014/38.1.04|s2cid=14238842}}</ref><ref>{{Cite journal|title="Why Should I Trust You?" {{!}} Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining|language=EN|doi=10.1145/2939672.2939778|s2cid=13029170}}</ref><ref>{{Citation|last1=Lundberg|first1=Scott M|title=A Unified Approach to Interpreting Model Predictions|date=2017|url=http://papers.nips.cc/paper/7062-a-unified-approach-to-interpreting-model-predictions.pdf|work=Advances in Neural Information Processing Systems 30|pages=4765–4774|editor-last=Guyon|editor-first=I.|publisher=Curran Associates, Inc.|access-date=2020-03-13|last2=Lee|first2=Su-In|editor2-last=Luxburg|editor2-first=U. V.|editor3-last=Bengio|editor3-first=S.|editor4-last=Wallach|editor4-first=H.|bibcode=2017arXiv170507874L|arxiv=1705.07874}}</ref><ref>{{Cite journal|last1=Carter|first1=Brandon|last2=Mueller|first2=Jonas|last3=Jain|first3=Siddhartha|last4=Gifford|first4=David|date=2019-04-11|title=What made you do this? Understanding black-box decisions with sufficient input subsets|url=http://proceedings.mlr.press/v89/carter19a.html|journal=The 22nd International Conference on Artificial Intelligence and Statistics|language=en|pages=567–576}}</ref><ref>{{Cite journal|last1=Shrikumar|first1=Avanti|last2=Greenside|first2=Peyton|last3=Kundaje|first3=Anshul|date=2017-07-17|title=Learning Important Features Through Propagating Activation Differences|url=http://proceedings.mlr.press/v70/shrikumar17a.html|journal=International Conference on Machine Learning|language=en|pages=3145–3153}}</ref><ref>{{Cite journal|url=https://dl.acm.org/doi/abs/10.5555/3305890.3306024|title=Axiomatic attribution for deep networks {{!}} Proceedings of the 34th International Conference on Machine Learning - Volume 70|website=dl.acm.org|series=Icml'17|date=6 August 2017|pages=3319–3328|language=EN|access-date=2020-03-13}}</ref> The mere transposition of the concepts of local interpretability into a remote context (where the black-box model is executed at a third party) is {{vague|text=currently under scrutiny}}.{{clarify}}<ref>{{Cite journal|last1=Aivodji|first1=Ulrich|last2=Arai|first2=Hiromi|last3=Fortineau|first3=Olivier|last4=Gambs|first4=Sébastien|last5=Hara|first5=Satoshi|last6=Tapp|first6=Alain|date=2019-05-24|title=Fairwashing: the risk of rationalization|url=http://proceedings.mlr.press/v97/aivodji19a.html|journal=International Conference on Machine Learning|language=en|publisher=PMLR|pages=161–170|arxiv=1901.09749}}</ref><ref>{{Cite journal|last1=Le Merrer|first1=Erwan|last2=Trédan|first2=Gilles|date=September 2020|title=Remote explainability faces the bouncer problem|url=https://www.nature.com/articles/s42256-020-0216-z|journal=Nature Machine Intelligence|language=en|volume=2|issue=9|pages=529–539|doi=10.1038/s42256-020-0216-z|issn=2522-5839|arxiv=1910.01432|s2cid=225207140}}</ref> |
||
There has been work on making glass-box models which are more transparent to inspection.<ref name=":6"/><ref>{{cite journal |last1=Singh |first1=Chandan |last2=Nasseri |first2=Keyan |last3=Tan |first3=Yan Shuo |last4=Tang |first4=Tiffany |last5=Yu |first5=Bin |title=imodels: a python package for fitting interpretable models |journal=Journal of Open Source Software |date=4 May 2021 |volume=6 |issue=61 |pages=3192 |doi=10.21105/joss.03192 |bibcode=2021JOSS....6.3192S |s2cid=235529515 |url=https://joss.theoj.org/papers/10.21105/joss.03192 |language=en |issn=2475-9066}}</ref> This includes decision trees,<ref>{{Cite journal|last1=Vidal|first1=Thibaut|last2=Schiffer|first2=Maximilian|date=2020|title=Born-Again Tree Ensembles|url=http://proceedings.mlr.press/v119/vidal20a.html|journal=International Conference on Machine Learning|language=en|publisher=PMLR|volume=119|pages=9743–9753|arxiv=2003.11132}}</ref> [[Bayesian network]]s, sparse [[linear model]]s,<ref>{{cite journal |last1=Ustun |first1=Berk |last2=Rudin |first2=Cynthia |title=Supersparse linear integer models for optimized medical scoring systems |journal=Machine Learning |date=1 March 2016 |volume=102 |issue=3 |pages=349–391 |doi=10.1007/s10994-015-5528-6 |s2cid=207211836 |url=https://link.springer.com/article/10.1007/s10994-015-5528-6 |language=en |issn=1573-0565}}</ref> and more.<ref>Bostrom, N., & Yudkowsky, E. (2014). [https://intelligence.org/files/EthicsofAI.pdf The ethics of artificial intelligence]. ''The Cambridge Handbook of Artificial Intelligence'', 316-334.</ref> The [[ACM Conference on Fairness, Accountability, and Transparency|Association for Computing Machinery Conference on Fairness, Accountability, and Transparency (ACM FAccT)]] was established in 2018 to study transparency and explainability in the context of socio-technical systems, many of which include artificial intelligence.<ref name="FAT* conference">{{cite web | url=https://fatconference.org/ | title=FAT* Conference }}</ref><ref>{{cite news |title=Computer programs recognise white men better than black women |url=https://www.economist.com/science-and-technology/2018/02/15/computer-programs-recognise-white-men-better-than-black-women |access-date=5 August 2018 |newspaper=The Economist |date=2018 |language=en}}</ref> |
|||
Some techniques allow visualisations of the inputs which individual [[Neuron (software)|software neurons]] respond to most strongly. Several groups |
Some techniques allow visualisations of the inputs which individual [[Neuron (software)|software neurons]] respond to most strongly. Several groups found that neurons can be aggregated into circuits that perform human-comprehensible functions, some of which reliably arise across different networks trained independently.<ref name="Circuits">{{cite journal |last1=Olah |first1=Chris |last2=Cammarata |first2=Nick |last3=Schubert |first3=Ludwig |last4=Goh |first4=Gabriel |last5=Petrov |first5=Michael |last6=Carter |first6=Shan |title=Zoom In: An Introduction to Circuits |journal=Distill |date=10 March 2020 |volume=5 |issue=3 |pages=e00024.001 |doi=10.23915/distill.00024.001 |url=https://distill.pub/2020/circuits/zoom-in/ |language=en |issn=2476-0757|doi-access=free }}</ref><ref>{{cite journal |last1=Li |first1=Yixuan |last2=Yosinski |first2=Jason |last3=Clune |first3=Jeff |last4=Lipson |first4=Hod |last5=Hopcroft |first5=John |title=Convergent Learning: Do different neural networks learn the same representations? |journal=Feature Extraction: Modern Questions and Challenges |date=8 December 2015 |pages=196–212 |url=http://proceedings.mlr.press/v44/li15convergent.html |publisher=PMLR |language=en}}</ref> |
||
There are various techniques to extract compressed representations of the features of given inputs, which can then be analysed by standard [[Cluster analysis|clustering techniques]]. Alternatively, networks can be trained to output linguistic explanations of their behaviour, which are then directly human-interpretable.<ref>{{cite journal |last1=Hendricks |first1=Lisa Anne |last2=Akata |first2=Zeynep |last3=Rohrbach |first3=Marcus |last4=Donahue |first4=Jeff |last5=Schiele |first5=Bernt |last6=Darrell |first6=Trevor |title=Generating Visual Explanations |journal=Computer Vision – ECCV 2016 |series=Lecture Notes in Computer Science |date=2016 |volume=9908 |pages=3–19 |doi=10.1007/978-3-319-46493-0_1 |url=https://link.springer.com/chapter/10.1007/978-3-319-46493-0_1 |publisher=Springer International Publishing |language=en|arxiv=1603.08507 |isbn=978-3-319-46492-3 |s2cid=12030503 }}</ref> Model behaviour can also be explained with reference to training data—for example, by evaluating which training inputs influenced a given behaviour the most.<ref>{{cite journal |last1=Koh |first1=Pang Wei |last2=Liang |first2=Percy |title=Understanding Black-box Predictions via Influence Functions |journal=International Conference on Machine Learning |date=17 July 2017 |pages=1885–1894 |url=http://proceedings.mlr.press/v70/koh17a.html |publisher=PMLR |arxiv=1703.04730 |language=en}}</ref> |
|||
== Regulation == |
== Regulation == |
Revision as of 01:04, 1 April 2023
Explainable AI (XAI), or Interpretable AI, or Explainable Machine Learning (XML),[1] is artificial intelligence (AI) in which humans can understand the reasoning behind decisions or predictions made by the AI.[2] It contrasts with the "black box" concept in machine learning where even the AI's designers cannot explain why it arrived at a specific decision.[3][4]
XAI hopes to help users of AI-powered systems perform more effectively by improving their understanding of how those systems reason.[5] XAI may be an implementation of the social right to explanation.[6] Even if there is no such legal right or regulatory requirement, XAI can improve the user experience of a product or service by helping end users trust that the AI is making good decisions. XAI aims to explain what has been done, what is being done, and what will be done next, and to unveil which information these actions are based on.[7] This makes it possible to confirm existing knowledge, to challenge existing knowledge, and to generate new assumptions.[8]
Machine learning (ML) algorithms used in AI can be categorized as white-box or black-box.[9] White-box models provide results that are understandable for experts in the domain. Black-box models, on the other hand, are extremely hard to explain and can hardly be understood even by domain experts.[10] XAI algorithms follow the three principles of transparency, interpretability, and explainability. A model is transparent “if the processes that extract model parameters from training data and generate labels from testing data can be described and motivated by the approach designer”.[11] Interpretability describes the possibility of comprehending the ML model and presenting the underlying basis for decision-making in a way that is understandable to humans.[12][13][14] Explainability is a concept that is recognized as important, but a consensus definition is not available.[11] One possibility is: “the collection of features of the interpretable domain, that have contributed for a given example to produce a decision (e.g., classification or regression)”.[15] If algorithms fulfil these principles, they provide a basis for justifying decisions, tracking and thereby verifying them, improving the algorithms, and exploring new facts.[16]
Sometimes it is also possible to achieve a result with high accuracy with a white-box ML algorithm that is interpretable in itself[clarification needed].[17] This is especially important in domains like medicine, defense, finance, and law, where it is crucial to understand decisions and build trust in the algorithms.[7] Many researchers argue that, at least for supervised machine learning, the way forward is symbolic regression, where the algorithm searches the space of mathematical expressions to find the model that best fits a given dataset.[18][19][20]
AI systems optimize behavior to satisfy a mathematically specified goal system chosen by the system designers, such as the command "maximize accuracy of assessing how positive film reviews are in the test dataset". The AI may learn useful general rules from the test set, such as "reviews containing the word 'horrible' are likely to be negative". However, it may also learn inappropriate rules, such as "reviews containing 'Daniel Day-Lewis' are usually positive"; such rules may be undesirable if they are likely to fail to generalize outside the training set, or if people consider the rule to be "cheating" or "unfair". A human can audit rules in an XAI to get an idea how likely the system is to generalize to future real-world data outside the test-set.[21]
Goals
Cooperation between agents, in this case algorithms and humans, depends on trust. If humans are to accept algorithmic prescriptions, they need to trust them. Incompleteness in formalization of trust criteria is a barrier to straightforward optimization approaches.[needs copy edit] Transparency, interpretability and explainability are intermediate goals on the road to these more comprehensive trust criteria.[22] This is particularly relevant in medicine,[23] especially clinical decision support systems (CDSS), in which medical professionals should be able to understand how and why a machine-based decision was made in order to trust the decision and augment their decision-making process.[24]
AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit pre-programmed goals on the training data, but that do not reflect the more nuanced implicit desires of the human system designers or the full complexity of the domain data. For example, a 2017 system tasked with image recognition learned to "cheat" by looking for a copyright tag that happened to be associated with horse pictures, rather than learning how to tell if a horse was actually pictured.[4] In another 2017 system, a supervised learning AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.[25][26]
One transparency project, the DARPA XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop", without greatly sacrificing AI performance. Human users of such a system can understand the AI's cognition (both in real-time and after the fact), and can determine whether to trust the AI.[27] Other applications of XAI are knowledge extraction from black-box models and model comparisons.[28] The term "glass box" is also used to describe tools that monitor the inputs and outputs of a system, with the purpose of verifying the system's adherence to ethical and socio-legal values and that produce value-based explanations.[29] The term is also used to name a voice assistant that produces counterfactual statements as explanations.[30]
History and methods
During the 1970s to 1990s, symbolic reasoning systems, such as MYCIN,[31] GUIDON,[32] SOPHIE,[33] and PROTOS[34][35] could represent, reason about, and explain their reasoning for diagnostic, instructional, or machine-learning (explanation-based learning) purposes. MYCIN, developed in the early 1970s as a research prototype for diagnosing bacteremia infections of the bloodstream, could explain[36] which of its hand-coded rules contributed to a diagnosis in a specific case. Research in intelligent tutoring systems resulted in developing systems such as SOPHIE that could act as an "articulate expert", explaining problem-solving strategy at a level the student could understand, so they would know what action to take next. For instance, SOPHIE could explain the qualitative reasoning behind its electronics troubleshooting, even though it ultimately relied on the SPICE circuit simulator. Similarly, GUIDON added tutorial rules to supplement MYCIN's domain-level rules so it could explain strategy for medical diagnosis. Symbolic approaches to machine learning, especially those relying on explanation-based learning, such as PROTOS, explicitly relied on representations of explanations, both to explain their actions and to acquire new knowledge.[clarification needed]
In the 1980s through early 1990s, truth maintenance systems (TMS) extended the capabilities of causal-reasoning, rule-based, and logic-based inference systems.[37]: 360–362 A TMS explicitly tracks alternate lines of reasoning, justifications for conclusions, and lines of reasoning that lead to contradictions, allowing future reasoning to avoid these dead ends. To provide explanation, they trace reasoning from conclusions to assumptions through rule operations or logical inferences, allowing explanations to be generated from the reasoning traces. As an example, consider a rule-based problem solver with just a few rules about Socrates that concludes he has died from poison:
By just tracing through the dependency structure the problem solver can construct the following explanation: "Socrates died because he was mortal and drank poison, and all mortals die when they drink poison. Socrates was mortal because he was a man and all men are mortal. Socrates drank poison because he held dissident beliefs, the government was conservative, and those holding conservative dissident beliefs under conservative governments must drink poison."[38]: 164–165
By the 1990s researchers began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks.[39] Researchers in clinical expert systems creating[clarification needed] neural network-powered decision support for clinicians sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice.[6] In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence.[4] As a result, many academics and organizations are developing tools to help detect bias in their systems.[40]
Marvin Minsky et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI (Humanistic Intelligence) as a way to create a more fair and balanced "human-in-the-loop" AI.[41]
Modern complex AI techniques, such as deep learning and genetic algorithms, are naturally opaque.[42] To address this issue, methods have been developed to make new models more explainable and interpretable.[43][13][12][44][45][46] This includes layerwise relevance propagation (LRP), a technique for determining which features in a particular input vector contribute most strongly to a neural network's output.[47][48] Other techniques explain some particular prediction made by a (nonlinear) black-box model, a goal referred to as "local interpretability".[49][50][51][52][53][54] The mere transposition of the concepts of local interpretability into a remote context (where the black-box model is executed at a third party) is currently under scrutiny[vague].[clarification needed][55][56]
There has been work on making glass-box models which are more transparent to inspection.[17][57] This includes decision trees,[58] Bayesian networks, sparse linear models,[59] and more.[60] The Association for Computing Machinery Conference on Fairness, Accountability, and Transparency (ACM FAccT) was established in 2018 to study transparency and explainability in the context of socio-technical systems, many of which include artificial intelligence.[61][62]
Some techniques allow visualisations of the inputs which individual software neurons respond to most strongly. Several groups found that neurons can be aggregated into circuits that perform human-comprehensible functions, some of which reliably arise across different networks trained independently.[63][64]
There are various techniques to extract compressed representations of the features of given inputs, which can then be analysed by standard clustering techniques. Alternatively, networks can be trained to output linguistic explanations of their behaviour, which are then directly human-interpretable.[65] Model behaviour can also be explained with reference to training data—for example, by evaluating which training inputs influenced a given behaviour the most.[66]
Regulation
As regulators, official bodies, and general users come to depend on AI-based dynamic systems, clearer accountability will be required for automated decision-making processes to ensure trust and transparency. Evidence of this requirement gaining more momentum can be seen with the launch of the first global conference exclusively dedicated to this emerging discipline, the International Joint Conference on Artificial Intelligence: Workshop on Explainable Artificial Intelligence (XAI).[67]
The European Union introduced a right to explanation in the General Data Protection Right (GDPR) as an attempt to deal with the potential problems stemming from the rising importance of algorithms. The implementation of the regulation began in 2018. However, the right to explanation in GDPR covers only the local aspect of interpretability. In the United States, insurance companies are required to be able to explain their rate and coverage decisions.[68] In France the Loi pour une République numérique (Digital Republic Act) grants subjects the right to request and receive information pertaining to the implementation of algorithms which process data about them.
Limitations
Despite efforts to increase the explainability of AI models, they still have a number of limitations.
Adversarial parties
By making an AI system more explainable, we also reveal more of its inner workings. For example, the explainability method of feature importance identifies features or variables that are most important in determining the model's output, while the influential samples method identifies the training samples that are most influential in determining the output, given a particular input.[69] However, there are a number of adversarial parties that could take advantage of this knowledge.
For example, competitor firms could replicate aspects of the original AI system in their own product, thus reducing competitive advantage.[70] An explainable AI system is also susceptible to different parties to “game” the system, or influence the output in a way that undermines the intended purpose. One study gives the example of a predictive policing system; in this case, those who could potentially “game” the system are the criminals subject to the system's decisions. In this study, developers of the system discussed the issue of criminal gangs looking to illegally obtain passports, and they expressed concerns that, if given an idea of what factors might trigger an alert in the passport application process, those gangs would be able to “send guinea pigs” to test those triggers, eventually finding a loophole that would allow them to “reliably get passports from under the noses of the authorities”.[71]
Technical complexity
A fundamental barrier to making AI systems explainable in the first place is the technical complexity of such systems. End users often lack even the prerequisite coding knowledge required to understand software of any kind. Current methods used to explain AI are mainly technical ones, geared toward machine learning engineers for debugging purposes, rather than the end users who are ultimately affected by the system, causing “a gap between explainability in practice and the goal of transparency”.[69] Proposed solutions to address the issue of technical complexity include either promoting the coding education of the general public that would make technical explanations more accessible to end users, or developing an outward-facing component that would provide explanations in layperson terms.[70]
Regardless of the solution, however, it must avoid the pitfall of oversimplification. It is important to strike a balance between accuracy – how faithfully does the explanation reflect the actual process of the AI system – and explainability – how well end users understand the process. However, this is a difficult balance to strike, since complexity of machine learning makes it difficult for even ML engineers to fully understand, let alone non-experts.[69]
Understanding versus trust
The goal of explainability to end users of AI systems is ultimately to increase trust in the system, even “address concerns about lack of ‘fairness’ and discriminatory effects”.[70] However, even with a good understanding of an AI system, end users may not necessarily trust the system. In one study, participants were presented with combinations of white-box and black-box explanations, and static and interactive explanations of AI systems. While these explanations served to increase both their self-reported and objective understanding, it had no impact on their level of trust, which remained skeptical.[72]
This outcome was especially true for decisions which impacted the end user in a significant way, such as graduate school admissions. Participants judged algorithms to be too inflexible and unforgiving in comparison to human decision-makers; instead of rigidly adhering to a set of rules, humans are able to consider exceptional cases as well as appeals to their initial decision.[72] So for such decisions, explainability will not necessarily cause end users to accept the use of decision-making algorithms. We will need to either turn to another method to increase trust and acceptance of decision-making algorithms, or question the need to rely solely on AI for such impactful decisions in the first place.
Criticism
Recent scholarship has suggested that the pursuit of explainability in AI techniques should be considered a secondary goal to the pursuit of AI's effectiveness, and that encouraging the exclusive development of XAI may limit the functionality of AI more broadly.[73][74] Critiques of XAI rely on developed concepts of mechanistic and empiric reasoning from evidence-based medicine to suggest that AI technologies can be clinically validated even when their function cannot be understood by their operators.[73]
Moreover, XAI systems have primarily focused on making AI systems understandable to AI practitioners rather than end users, and their results on user perceptions of these systems have been somewhat fragmented.[75] Some researchers have also advocated for the use of inherently interpretable machine learning models, rather than using post-hoc explanations, where a second model is created to explain the first. This is partly because post-hoc models increase the complexity in a decision pathway and partly because it is often unclear how faithfully a post-hoc explanation can mimic the computations of an entirely separate model.[17] However, another popular view is that it only matters if the explanation accomplishes the given task at hand, and whether it is pre or post-hoc, doesn't matter. For example, if a post-hoc explanation method helps a doctor diagnose cancer better, it is of secondary importance whether it is a correct/incorrect explanation.
Analysis of the goals of XAI find that it requires a form of lossy compression that will become less effective as AI models grow in their number of parameters, and in conjunction with other factors this leads to a theoretical limit for explainability.[76]
See also
References
- ^ Phillips, P. Jonathon; Hahn, Carina A.; Fontana, Peter C.; Yates, Amy N.; Greene, Kristen; Broniatowski, David A.; Przybocki, Mark A. (2021-09-29). "Four Principles of Explainable Artificial Intelligence". doi:10.6028/nist.ir.8312.
{{cite journal}}
: Cite journal requires|journal=
(help) - ^ Vilone, Giulia; Longo, Luca (2021). "Notions of explainability and evaluation approaches for explainable artificial intelligence". Information Fusion. December 2021 - Volume 76: 89–106. doi:10.1016/j.inffus.2021.05.009.
- ^ Castelvecchi, Davide (2016-10-06). "Can we open the black box of AI?". Nature. 538 (7623): 20–23. Bibcode:2016Natur.538...20C. doi:10.1038/538020a. ISSN 0028-0836. PMID 27708329. S2CID 4465871.
- ^ a b c Sample, Ian (5 November 2017). "Computer says no: why making AIs fair, accountable and transparent is crucial". The Guardian. Retrieved 30 January 2018.
- ^ Alizadeh, Fatemeh (2021). "I Don't Know, Is AI Also Used in Airbags?: An Empirical Study of Folk Concepts and People's Expectations of Current and Future Artificial Intelligence". Icom. 20 (1): 3–17. doi:10.1515/icom-2021-0009. S2CID 233328352.
- ^ a b Edwards, Lilian; Veale, Michael (2017). "Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For". Duke Law and Technology Review. 16: 18. SSRN 2972855.
- ^ a b Gunning, D.; Stefik, M.; Choi, J.; Miller, T.; Stumpf, S.; Yang, G.-Z. (2019-12-18). "XAI-Explainable artificial intelligence". Science Robotics. 4 (37): eaay7120. doi:10.1126/scirobotics.aay7120. ISSN 2470-9476. PMID 33137719.
- ^ Rieg, Thilo; Frick, Janek; Baumgartl, Hermann; Buettner, Ricardo (2020-12-17). "Demonstration of the potential of white-box machine learning approaches to gain insights from cardiovascular disease electrocardiograms". PLOS ONE. 15 (12): e0243615. Bibcode:2020PLoSO..1543615R. doi:10.1371/journal.pone.0243615. ISSN 1932-6203. PMC 7746264. PMID 33332440.
- ^ Vilone, Giulia; Longo, Luca (2021). "Classification of Explainable Artificial Intelligence Methods through Their Output Formats". Machine Learning and Knowledge Extraction. 3 (3): 615–661. doi:10.3390/make3030032.
- ^ Loyola-González, O. (2019). "Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View". IEEE Access. 7: 154096–154113. doi:10.1109/ACCESS.2019.2949286. ISSN 2169-3536.
- ^ a b Roscher, R.; Bohn, B.; Duarte, M. F.; Garcke, J. (2020). "Explainable Machine Learning for Scientific Insights and Discoveries". IEEE Access. 8: 42200–42216. arXiv:1905.08883. doi:10.1109/ACCESS.2020.2976199. ISSN 2169-3536.
- ^ a b Murdoch, W. James; Singh, Chandan; Kumbier, Karl; Abbasi-Asl, Reza; Yu, Bin (2019-01-14). "Interpretable machine learning: definitions, methods, and applications". Proceedings of the National Academy of Sciences of the United States of America. 116 (44): 22071–22080. arXiv:1901.04592. Bibcode:2019arXiv190104592M. doi:10.1073/pnas.1900654116. PMC 6825274. PMID 31619572.
- ^ a b Lipton, Zachary C. (June 2018). "The Mythos of Model Interpretability: In machine learning, the concept of interpretability is both important and slippery". Queue. 16 (3): 31–57. doi:10.1145/3236386.3241340. ISSN 1542-7730.
- ^ "Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI". DeepAI. 2019-10-22. Retrieved 2021-01-13.
- ^ Montavon, Grégoire; Samek, Wojciech; Müller, Klaus-Robert (2018-02-01). "Methods for interpreting and understanding deep neural networks". Digital Signal Processing. 73: 1–15. doi:10.1016/j.dsp.2017.10.011. ISSN 1051-2004.
- ^ Adadi, A.; Berrada, M. (2018). "Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)". IEEE Access. 6: 52138–52160. doi:10.1109/ACCESS.2018.2870052. ISSN 2169-3536.
- ^ a b c Rudin, Cynthia (2019). "Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead". Nature Machine Intelligence. 1 (5): 206–215. arXiv:1811.10154. doi:10.1038/s42256-019-0048-x. ISSN 2522-5839. PMC 9122117. PMID 35603010.
- ^ Wenninger, Simon; Kaymakci, Can; Wiethe, Christian (2022). "Explainable long-term building energy consumption prediction using QLattice". Applied Energy. 308. Elsevier BV: 118300. doi:10.1016/j.apenergy.2021.118300. ISSN 0306-2619. S2CID 245428233.
- ^ Christiansen, Michael; Wilstrup, Casper; Hedley, Paula L. (2022). "Explainable "white-box" machine learning is the way forward in preeclampsia screening". American Journal of Obstetrics and Gynecology. 227 (5). Elsevier BV: 791. doi:10.1016/j.ajog.2022.06.057. ISSN 0002-9378. PMID 35779588. S2CID 250160871.
- ^ Wilstup, Casper; Cave, Chris (2021-01-15), Combining symbolic regression with the Cox proportional hazards model improves prediction of heart failure deaths, Cold Spring Harbor Laboratory, doi:10.1101/2021.01.15.21249874, S2CID 231609904
- ^ "How AI detectives are cracking open the black box of deep learning". Science. 5 July 2017. Retrieved 30 January 2018..
- ^ Dosilovic, Filip; Brcic, Mario; Hlupic, Nikica (2018-05-25). "Explainable Artificial Intelligence: A Survey" (PDF). MIPRO 2018 - 41st International Convention Proceedings. MIPRO 2018. Opatija, Croatia. pp. 210–215. doi:10.23919/MIPRO.2018.8400040. ISBN 978-953-233-095-3. Archived from the original (PDF) on 2018-12-10. Retrieved 2018-12-09.
- ^ Bernal, Jose; Mazo, Claudia (2022-10-11). "Transparency of Artificial Intelligence in Healthcare: Insights from Professionals in Computing and Healthcare Worldwide". Applied Sciences. 12 (20): 10228. doi:10.3390/app122010228. ISSN 2076-3417.
- ^ Antoniadi, Anna Markella; Du, Yuhan; Guendouz, Yasmine; Wei, Lan; Mazo, Claudia; Becker, Brett A.; Mooney, Catherine (January 2021). "Current Challenges and Future Opportunities for XAI in Machine Learning-Based Clinical Decision Support Systems: A Systematic Review". Applied Sciences. 11 (11): 5088. doi:10.3390/app11115088. ISSN 2076-3417.
- ^ "DeepMind Has Simple Tests That Might Prevent Elon Musk's AI Apocalypse". Bloomberg.com. 11 December 2017. Retrieved 30 January 2018.
- ^ "Learning from Human Preferences". OpenAI Blog. 13 June 2017. Retrieved 30 January 2018.
- ^ "Explainable Artificial Intelligence (XAI)". DARPA. DARPA. Retrieved 17 July 2017.
- ^ Biecek, Przemyslaw (23 June 2018). "DALEX: explainers for complex predictive models". Journal of Machine Learning Research. 19: 1–5. arXiv:1806.08915. Bibcode:2018arXiv180608915B.
- ^ Aler Tubella, Andrea; Theodorou, Andreas; Dignum, Frank; Dignum, Virginia (2019). "Governance by Glass-Box: Implementing Transparent Moral Bounds for AI Behaviour". Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence. California: International Joint Conferences on Artificial Intelligence Organization. pp. 5787–5793. doi:10.24963/ijcai.2019/802. ISBN 978-0-9992411-4-1.
- ^ Sokol, Kacper; Flach, Peter (2018). "Glass-Box: Explaining AI Decisions With Counterfactual Statements Through Conversation With a Voice-enabled Virtual Assistant". Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence. pp. 5868–5870. doi:10.24963/ijcai.2018/865. ISBN 9780999241127. S2CID 51608978.
- ^ Fagan, L. M.; Shortliffe, E. H.; Buchanan, B. G. (1980). "Computer-based medical decision making: from MYCIN to VM". Automedica. 3 (2): 97–108.
- ^ Clancey, William (1987). Knowledge-Based Tutoring: The GUIDON Program. Cambridge, Massachusetts: The MIT Press.
- ^ Brown, John S.; Burton, R. R.; De Kleer, Johan (1982). "Pedagogical, natural language, and knowledge engineering techniques in SOPHIE I, II, and III". Intelligent Tutoring Systems. Academic Press. ISBN 0-12-648680-8.
- ^ Bareiss, Ray; Porter, Bruce; Weir, Craig; Holte, Robert (1990). "Protos: An Exemplar-Based Learning Apprentice". Machine Learning. Vol. 3. Morgan Kaufmann Publishers Inc. pp. 112–139. ISBN 1-55860-119-8.
- ^ Bareiss, Ray. Exemplar-Based Knowledge Acquisition: A Unified Approach to Concept Representation, Classification, and Learning. Perspectives in Artificial Intelligence.
- ^ Van Lent, M.; Fisher, W.; Mancuso, M. (July 2004). "An explainable artificial intelligence system for small-unit tactical behavior". Proceedings of the National Conference on Artificial Intelligence. San Jose, CA: AAAI Press. pp. 900–907. ISBN 0262511835.
- ^ Russell, Stuart; Norvig, Peter (2003). Artificial Intelligence: A Modern Approach. Prentice Hall Series in Artificial Intelligence (Second ed.). Upper Saddle River, New Jersey: Prentice Hall, Pearson Education. ISBN 0-13-790395-2.
- ^ Forbus, Kenneth; De Kleer, Johan (1993). Building Problem Solvers. Cambridge, Massachusetts: The MIT Press. ISBN 0-262-06157-0.
- ^ Tickle, A. B.; Andrews, R.; Golea, M.; Diederich, J. (November 1998). "The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks". IEEE Transactions on Neural Networks. 9 (6): 1057–1068. doi:10.1109/72.728352. ISSN 1045-9227. PMID 18255792.
- ^ "Accenture Unveils Tool to Help Companies Insure Their AI Is Fair". Bloomberg.com. June 2018. Retrieved 5 August 2018.
- ^ Minsky, et al., "The Society of Intelligent Veillance" IEEE ISTAS2013, pages 13-17.
- ^ Mukherjee, Siddhartha (27 March 2017). "A.I. Versus M.D." The New Yorker. Retrieved 30 January 2018.
- ^ Csiszár, Orsolya; Csiszár, Gábor; Dombi, József (2020-07-08). "Interpretable neural networks based on continuous-valued logic and multicriteria decision operators". Knowledge-Based Systems. 199: 105972. arXiv:1910.02486. doi:10.1016/j.knosys.2020.105972. ISSN 0950-7051.
- ^ Doshi-Velez, Finale; Kim, Been (2017-02-27). "Towards A Rigorous Science of Interpretable Machine Learning". arXiv:1702.08608 [stat.ML].
- ^ Abdollahi, Behnoush, and Olfa Nasraoui. (2016). "Explainable Restricted Boltzmann Machines for Collaborative Filtering". arXiv:1606.07129 [stat.ML].
{{cite arXiv}}
: CS1 maint: multiple names: authors list (link) - ^ Dombi, József; Csiszár, Orsolya (2021). "Explainable Neural Networks Based on Fuzzy Logic and Multi-criteria Decision Tools". Studies in Fuzziness and Soft Computing. 408. doi:10.1007/978-3-030-72280-7. ISBN 978-3-030-72279-1. ISSN 1434-9922. S2CID 233486978.
- ^ Bach, Sebastian; Binder, Alexander; Montavon, Grégoire; Klauschen, Frederick; Müller, Klaus-Robert; Samek, Wojciech (2015-07-10). Suarez, Oscar Deniz (ed.). "On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation". PLOS ONE. 10 (7): e0130140. Bibcode:2015PLoSO..1030140B. doi:10.1371/journal.pone.0130140. ISSN 1932-6203. PMC 4498753. PMID 26161953.
- ^ Sample, Ian (5 November 2017). "Computer says no: why making AIs fair, accountable and transparent is crucial". The Guardian. Retrieved 5 August 2018.
- ^ Martens, David; Provost, Foster (2014). "Explaining data-driven document classifications" (PDF). MIS Quarterly. 38: 73–99. doi:10.25300/MISQ/2014/38.1.04. S2CID 14238842.
- ^ ""Why Should I Trust You?" | Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining". doi:10.1145/2939672.2939778. S2CID 13029170.
{{cite journal}}
: Cite journal requires|journal=
(help) - ^ Lundberg, Scott M; Lee, Su-In (2017), Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H. (eds.), "A Unified Approach to Interpreting Model Predictions" (PDF), Advances in Neural Information Processing Systems 30, Curran Associates, Inc., pp. 4765–4774, arXiv:1705.07874, Bibcode:2017arXiv170507874L, retrieved 2020-03-13
- ^ Carter, Brandon; Mueller, Jonas; Jain, Siddhartha; Gifford, David (2019-04-11). "What made you do this? Understanding black-box decisions with sufficient input subsets". The 22nd International Conference on Artificial Intelligence and Statistics: 567–576.
- ^ Shrikumar, Avanti; Greenside, Peyton; Kundaje, Anshul (2017-07-17). "Learning Important Features Through Propagating Activation Differences". International Conference on Machine Learning: 3145–3153.
- ^ "Axiomatic attribution for deep networks | Proceedings of the 34th International Conference on Machine Learning - Volume 70". dl.acm.org. Icml'17: 3319–3328. 6 August 2017. Retrieved 2020-03-13.
- ^ Aivodji, Ulrich; Arai, Hiromi; Fortineau, Olivier; Gambs, Sébastien; Hara, Satoshi; Tapp, Alain (2019-05-24). "Fairwashing: the risk of rationalization". International Conference on Machine Learning. PMLR: 161–170. arXiv:1901.09749.
- ^ Le Merrer, Erwan; Trédan, Gilles (September 2020). "Remote explainability faces the bouncer problem". Nature Machine Intelligence. 2 (9): 529–539. arXiv:1910.01432. doi:10.1038/s42256-020-0216-z. ISSN 2522-5839. S2CID 225207140.
- ^ Singh, Chandan; Nasseri, Keyan; Tan, Yan Shuo; Tang, Tiffany; Yu, Bin (4 May 2021). "imodels: a python package for fitting interpretable models". Journal of Open Source Software. 6 (61): 3192. Bibcode:2021JOSS....6.3192S. doi:10.21105/joss.03192. ISSN 2475-9066. S2CID 235529515.
- ^ Vidal, Thibaut; Schiffer, Maximilian (2020). "Born-Again Tree Ensembles". International Conference on Machine Learning. 119. PMLR: 9743–9753. arXiv:2003.11132.
- ^ Ustun, Berk; Rudin, Cynthia (1 March 2016). "Supersparse linear integer models for optimized medical scoring systems". Machine Learning. 102 (3): 349–391. doi:10.1007/s10994-015-5528-6. ISSN 1573-0565. S2CID 207211836.
- ^ Bostrom, N., & Yudkowsky, E. (2014). The ethics of artificial intelligence. The Cambridge Handbook of Artificial Intelligence, 316-334.
- ^ "FAT* Conference".
- ^ "Computer programs recognise white men better than black women". The Economist. 2018. Retrieved 5 August 2018.
- ^ Olah, Chris; Cammarata, Nick; Schubert, Ludwig; Goh, Gabriel; Petrov, Michael; Carter, Shan (10 March 2020). "Zoom In: An Introduction to Circuits". Distill. 5 (3): e00024.001. doi:10.23915/distill.00024.001. ISSN 2476-0757.
- ^ Li, Yixuan; Yosinski, Jason; Clune, Jeff; Lipson, Hod; Hopcroft, John (8 December 2015). "Convergent Learning: Do different neural networks learn the same representations?". Feature Extraction: Modern Questions and Challenges. PMLR: 196–212.
- ^ Hendricks, Lisa Anne; Akata, Zeynep; Rohrbach, Marcus; Donahue, Jeff; Schiele, Bernt; Darrell, Trevor (2016). "Generating Visual Explanations". Computer Vision – ECCV 2016. Lecture Notes in Computer Science. 9908. Springer International Publishing: 3–19. arXiv:1603.08507. doi:10.1007/978-3-319-46493-0_1. ISBN 978-3-319-46492-3. S2CID 12030503.
- ^ Koh, Pang Wei; Liang, Percy (17 July 2017). "Understanding Black-box Predictions via Influence Functions". International Conference on Machine Learning. PMLR: 1885–1894. arXiv:1703.04730.
- ^ "IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI)" (PDF). Earthlink. IJCAI. Archived from the original (PDF) on 4 April 2019. Retrieved 17 July 2017.
- ^ Kahn, Jeremy (12 December 2018). "Artificial Intelligence Has Some Explaining to Do". Bloomberg Businessweek. Retrieved 17 December 2018.
- ^ a b c Bhatt, Umang; Xiang, Alice; Sharma, Shubham; Weller, Adrian; Taly, Ankur; Jia, Yunhan; Ghosh, Joydeep; Puri, Richir; M.F. Moura, José; Eckersley, Peter (2022). "Explainable Machine Learning in Deployment". Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. pp. 648–657. doi:10.1145/3351095.3375624. ISBN 9781450369367. S2CID 202572724.
- ^ a b c Burrel, Jenna (2016). "How the machine 'thinks': Understanding opacity in machine learning algorithms". Big Data & Society. 3 (1). doi:10.1177/2053951715622512. S2CID 61330970.
{{cite journal}}
: Cite journal requires|journal=
(help) - ^ Veale, Michael; Van Kleek, Max; Binns, Reuben (2018). "Fairness and Accountability Design Needs for Algorithmic Support in High-Stakes Public Sector Decision-Making". Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Vol. 40. pp. 1–14. doi:10.1145/3173574.3174014. ISBN 9781450356206. S2CID 3639135.
- ^ a b Cheng, Hao-Fei; Wang, Ruotang; Zhang, Zheng; O’Connell, Fiona; Gray, Terrance; Harper, F. Maxwell; Zhu, Haiyi (2019). Explaining Decision-Making Algorithms through UI: Strategies to Help Non-Expert Stakeholders. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Vol. 559. pp. 1–12. doi:10.1145/3290605.3300789. ISBN 9781450359702. S2CID 140281803.
- ^ a b McCoy, Liam G.; Brenna, Connor T. A.; Chen, Stacy S.; Vold, Karina; Das, Sunit (2021-11-05). "Believing in black boxes: machine learning for healthcare does not need explainability to be evidence-based". Journal of Clinical Epidemiology. 142 (Online ahead of print): 252–257. doi:10.1016/j.jclinepi.2021.11.001. ISSN 0895-4356. PMID 34748907. S2CID 243810442.
- ^ Ghassemi, Marzyeh; Oakden-Rayner, Luke; Beam, Andrew L. (2021-11-01). "The false hope of current approaches to explainable artificial intelligence in health care". The Lancet Digital Health. 3 (11): e745–e750. doi:10.1016/S2589-7500(21)00208-9. ISSN 2589-7500. PMID 34711379. S2CID 239963176.
- ^ Alizadeh, Fatemeh (2020). "eXplainable AI: take one step back, move two steps forward". Mensch und Computer.
- ^ Sarkar, Advait (2022). "Is explainable AI a race against model complexity?" (PDF). Workshop on Transparency and Explanations in Smart Systems (TeXSS), in Conjunction with ACM Intelligent User Interfaces (IUI 2022): 192–199. arXiv:2205.10119 – via CEUR Workshop Proceedings.
External links
- Mazumdar, Dipankar; Neto, Mário Popolin; Paulovich, Fernando V. (2021). "Random Forest similarity maps: A Scalable Visual Representation for Global and Local Interpretation". Electronics. 10 (22): 2862. doi:10.3390/electronics10222862.
- "AI Explainability 360".
- "What is the Explainable-Ai and why is important".
- "Explainable AI Is The Next Big Thing In Accounting And Finance". Forbes.
- "FAT* Conference on Fairness, Accountability, and Transparency".
- "FATML Workshop on Fairness, Accountability, and Transparency in Machine Learning".
- "'Explainable Artificial Intelligence': Cracking open the black box of AI". Computerworld. 2017-11-02. Archived from the original on 2020-10-22. Retrieved 2017-11-02.
- Park, Dong Huk; Hendricks, Lisa Anne; Akata, Zeynep; Schiele, Bernt; Darrell, Trevor; Rohrbach, Marcus (2016-12-14). "Attentive Explanations: Justifying Decisions and Pointing to the Evidence". arXiv:1612.04757 [cs.CV].
- "Explainable AI: Making machines understandable for humans". Explainable AI: Making machines understandable for humans. Retrieved 2017-11-02.
- "End-to-End Deep Learning for Self-Driving Cars". Parallel Forall. 2016-08-17. Retrieved 2017-11-02.
- "Explaining How End-to-End Deep Learning Steers a Self-Driving Car". Parallel Forall. 2017-05-23. Retrieved 2017-11-02.
- Knight, Will (2017-03-14). "DARPA is funding projects that will try to open up AI's black boxes". MIT Technology Review. Retrieved 2017-11-02.
- Alvarez-Melis, David; Jaakkola, Tommi S. (2017-07-06). "A causal framework for explaining the predictions of black-box sequence-to-sequence models". arXiv:1707.01943 [cs.LG].
- "Similarity Cracks the Code Of Explainable AI". simMachines. 2017-10-12. Retrieved 2018-02-02.
- Bojarski, Mariusz; Yeres, Philip; Choromanska, Anna; Choromanski, Krzysztof; Firner, Bernhard; Jackel, Lawrence; Muller, Urs (2017-04-25). "Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car". arXiv:1704.07911 [cs.CV].
- "What are the methods to interpret the output of machine learning methods?". IntelligenceReborn. 2020-12-30. Retrieved 2020-12-30.