Citation bot (talk | contribs) m Add: class, eprint. Removed parameters. You can use this bot yourself. Report bugs here. |
Rolf h nelson (talk | contribs) WP:BRD trim redundancy and wordiness, clarify, expand |
||
Line 1: | Line 1: | ||
An '''Explainable AI''' ('''XAI''') or '''Transparent AI''' is an [[Artificial Intelligence]] (AI) whose actions can be easily understood by humans. It contrasts with "black box" AIs that employ complex opaque algorithms, where even their designers cannot explain why the AI arrived at a specific decision.<ref name=guardian>{{cite news|last1=Sample|first1=Ian|title=Computer says no: why making AIs fair, accountable and transparent is crucial|url=https://www.theguardian.com/science/2017/nov/05/computer-says-no-why-making-ais-fair-accountable-and-transparent-is-crucial|accessdate=30 January 2018|work=the Guardian|date=5 November 2017|language=en}}</ref> XAI can be used to implement a social [[right to explanation]].<ref name=":0">{{Cite journal|last=Edwards|first=Lilian|last2=Veale|first2=Michael|date=2017|title=Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For|url=https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2972855|journal=Duke Law and Technology Review|volume=|pages=|via=}}</ref> Transparency rarely comes for free; there are often tradeoffs between how "smart" an AI is and how transparent it is, and these tradeoffs are expected to grow larger as AI systems increase in internal complexity. The technical challenge of explaining AI decisions is sometimes known as the '''interpretability problem'''.<ref name=science>{{cite news|title=How AI detectives are cracking open the black box of deep learning|url=http://www.sciencemag.org/news/2017/07/how-ai-detectives-are-cracking-open-black-box-deep-learning|accessdate=30 January 2018|work=Science {{!}} AAAS|date=5 July 2017|language=en}}</ref> |
|||
'''Explainable AI''' ('''XAI''') is a neologism that has recently reached the parlance of [[artificial intelligence]]. Its purpose is to provide accountability when addressing technological innovations ascribed to dynamic and non-linearly programmed systems, e.g. [[artificial neural networks]], [[deep learning]], and [[genetic algorithms]]. |
|||
AI systems optimize behavior to satisfy a mathematically-specified goal system chosen by the system designers, such as "maximize accuracy of [[sentiment analysis|assessing how positive]] film reviews are in the test dataset". The AI may learn useful general rules from the testset, such as "reviews containing the word 'horrible'" are likely to be negative". However, it may also learn inappropriate rules, such as "reviews containing '[[Daniel Day-Lewis]]' are usually positive"; such rules may be undesirable if they are deemed likely to fail to generalize outside the test set, or if people consider the rule to be "cheating" or "unfair". A human can audit rules in an XAI to get an idea how likely the system is to generalize to future real-world data outside the test-set.<ref name=science/> |
|||
It is about asking the question of how algorithms arrive at their decisions. In a sense, it is a technical discipline providing operational tools that might be useful for explaining systems, such as in implementing a [[right to explanation]].<ref name=":0">{{Cite journal|last=Edwards|first=Lilian|last2=Veale|first2=Michael|date=2017|title=Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For|url=https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2972855|journal=Duke Law and Technology Review|volume=|pages=|via=}}</ref> |
|||
⚫ | |||
AI-related algorithmic (supervised and unsupervised) practices work on a model of success that orientates towards some form of correct state, with singular focus placed on an expected output. E.g., an image recognition algorithm's level of success will be based on the algorithm's ability to recognize certain objects, and failure to do so will indicate that the algorithm requires further tuning. As the tuning level is dynamic, closely correlated to function refinement and training data-set, granular understanding of the underlying operational vectors is rarely introspected. |
|||
AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit preprogrammed goals on the training data, but that do not reflect the complicated implicit desires of the human system designers. For example, a 2017 system tasked with image recognition learned to "cheat" by looking for a copyright tag that happened to be associated with horse pictures, rather than learning how to tell if a horse was actually pictured.<ref name=guardian/> In another 2017 system, a [[supervised learning]] AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.<ref>{{cite news|title=DeepMind Has Simple Tests That Might Prevent Elon Musk’s AI Apocalypse|url=https://www.bloomberg.com/news/articles/2017-12-11/deepmind-has-simple-tests-that-might-prevent-elon-musk-s-ai-apocalypse|accessdate=30 January 2018|work=Bloomberg.com|date=11 December 2017|language=en}}</ref><ref>{{cite news|title=Learning from Human Preferences|url=https://blog.openai.com/deep-reinforcement-learning-from-human-preferences/|accessdate=30 January 2018|work=OpenAI Blog|date=13 June 2017}}</ref> |
|||
⚫ | One transparency project, the [[DARPA]] XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop", without greatly sacrificing AI performance. Human users should be able to understand the AI's cognition (both in real-time and after the fact), and should be able to determine when to trust the AI and when the AI should be distrusted.<ref>{{cite web|title=Explainable Artificial Intelligence (XAI)|url=https://www.darpa.mil/program/explainable-artificial-intelligence|website=DARPA|publisher=DARPA|accessdate=17 July 2017}}</ref><ref>{{cite arxiv|last=Holzinger|first=Andreas|last2=Plass|first2=Markus|last3=Holzinger|first3=Katharina|last4=Crisan|first4=Gloria Cerasela|last5=Pintea|first5=Camelia-M.|last6=Palade|first6=Vasile|date=2017-08-03|title=A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop|eprint=1708.01104|class=cs.AI}}</ref> |
||
XAI aims to address this black-box approach and allow introspection of these dynamic systems tractable, allowing humans to understand how computational machines develop their own models for solving tasks. |
|||
== |
== History and methods == |
||
[[Mycin]], a research prototype that could explain which of its hand-coded rules contributed to a diagnosis in a specific case, was developed in the early 1970s.<ref>Van Lent, M., Fisher, W., & Mancuso, M. (2004, July). An explainable artificial intelligence system for small-unit tactical behavior. In Proceedings of the National Conference on Artificial Intelligence (pp. 900-907). Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999.</ref><ref>Fagan, L. M., Shortliffe, E. H., & Buchanan, B. G. (1980). Computer-based medical decision making: from MYCIN to VM. Automedica, 3(2), 97-108.</ref> |
|||
A universal definition of this term has yet to have been fully established; however, the [[DARPA]] XAI program defines its aims as the following: |
|||
⚫ | By the 1990s researchers also began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks.<ref>{{Cite journal|last=Tickle|first=A. B.|last2=Andrews|first2=R.|last3=Golea|first3=M.|last4=Diederich|first4=J.|date=November 1998|title=The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks|url=http://ieeexplore.ieee.org/document/728352/|journal=IEEE Transactions on Neural Networks|volume=9|issue=6|pages=1057–1068|doi=10.1109/72.728352|issn=1045-9227}}</ref>. Researchers in clinical expert systems creating neural network-powered decision support for clinicians have sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice.<ref name=":0" /> |
||
The "deep learning" methods powering cutting-edge AI in the 2010s are naturally opaque,<ref>{{cite news|last1=Mukherjee|first1=Siddhartha|title=A.I. Versus M.D.|url=https://www.newyorker.com/magazine/2017/04/03/ai-versus-md|accessdate=30 January 2018|work=The New Yorker|date=27 March 2017}}</ref> as are other complicated neural networks; genetic algorithms likewise are naturally opaque. In contrast, decision trees and Bayesian networks are more transparent to inspection.<ref>Bostrom, N., & Yudkowsky, E. (2014). The ethics of artificial intelligence. The Cambridge handbook of artificial intelligence, 316-334.</ref> |
|||
* Produce more explainable models, while maintaining a high level of learning performance (prediction accuracy) |
|||
* Enable human users to understand, appropriately trust, and effectively manage the emerging generation of artificially intelligent partners<ref>{{cite web|title=Explainable Artificial Intelligence (XAI)|url=https://www.darpa.mil/program/explainable-artificial-intelligence|website=DARPA|publisher=DARPA|accessdate=17 July 2017}}</ref> |
|||
⚫ | |||
⚫ | |||
⚫ | |||
Layerwise Relevance Propagation (LRP), first described in 2015, is a technique for determining which features in a particular input vector contribute most strongly to a neural network’s output.<ref name="Shiebler 2017">{{cite web | last=Shiebler | first=Dan | title=Understanding Neural Networks with Layerwise Relevance Propagation and Deep Taylor Series | website=Dan Shiebler | date=2017-04-16 | url=https://dshieble.github.io/2017-04-16-deep-taylor-lrp/ | access-date=2017-11-03}}</ref><ref name="Bach Binder Montavon Klauschen p=e0130140">{{cite journal | last=Bach | first=Sebastian | last2=Binder | first2=Alexander | last3=Montavon | first3=Grégoire | last4=Klauschen | first4=Frederick | last5=Müller | first5=Klaus-Robert | last6=Samek | first6=Wojciech | editor-last=Suarez | editor-first=Oscar Deniz | title=On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation | journal=Plos One | publisher=Public Library of Science (PLoS) | volume=10 | issue=7 | date=2015-07-10 | issn=1932-6203 | doi=10.1371/journal.pone.0130140 | page=e0130140| bibcode=2015PLoSO..1030140B }}</ref> |
Layerwise Relevance Propagation (LRP), first described in 2015, is a technique for determining which features in a particular input vector contribute most strongly to a neural network’s output.<ref name="Shiebler 2017">{{cite web | last=Shiebler | first=Dan | title=Understanding Neural Networks with Layerwise Relevance Propagation and Deep Taylor Series | website=Dan Shiebler | date=2017-04-16 | url=https://dshieble.github.io/2017-04-16-deep-taylor-lrp/ | access-date=2017-11-03}}</ref><ref name="Bach Binder Montavon Klauschen p=e0130140">{{cite journal | last=Bach | first=Sebastian | last2=Binder | first2=Alexander | last3=Montavon | first3=Grégoire | last4=Klauschen | first4=Frederick | last5=Müller | first5=Klaus-Robert | last6=Samek | first6=Wojciech | editor-last=Suarez | editor-first=Oscar Deniz | title=On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation | journal=Plos One | publisher=Public Library of Science (PLoS) | volume=10 | issue=7 | date=2015-07-10 | issn=1932-6203 | doi=10.1371/journal.pone.0130140 | page=e0130140| bibcode=2015PLoSO..1030140B }}</ref> |
||
In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence.<ref name=guardian/> |
|||
Newer however is the focus on explaining machine learning and AI to those whom the decisions concern, rather than the designers or direct users of decision systems. Since DARPA's introduction of its program in 2016, a number of new initiatives seek to address the issue of algorithmic accountability and provide transparency concerning how technologies within this domain function. |
|||
* 25 April 2017: [[Nvidia]] published the paper "Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car"<ref>{{Cite arxiv |title=Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car|eprint=1704.07911|author1=Bojarski|first1=Mariusz|last2=Yeres|first2=Philip|last3=Choromanska|first3=Anna|last4=Choromanski|first4=Krzysztof|last5=Firner|first5=Bernhard|last6=Jackel|first6=Lawrence|last7=Muller|first7=Urs|year=2017|class=cs.CV}}</ref> |
* 25 April 2017: [[Nvidia]] published the paper "Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car"<ref>{{Cite arxiv |title=Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car|eprint=1704.07911|author1=Bojarski|first1=Mariusz|last2=Yeres|first2=Philip|last3=Choromanska|first3=Anna|last4=Choromanski|first4=Krzysztof|last5=Firner|first5=Bernhard|last6=Jackel|first6=Lawrence|last7=Muller|first7=Urs|year=2017|class=cs.CV}}</ref> |
||
* 13 July 2017: [[Accenture]] recommended "Responsible AI: Why we need Explainable AI"<ref>{{cite web|title=Responsible AI: Why we need Explainable AI|url=https://www.youtube.com/watch?v=A668RoogabM|website=YouTube|publisher=Accenture|accessdate=17 July 2017}}</ref> |
* 13 July 2017: [[Accenture]] recommended "Responsible AI: Why we need Explainable AI"<ref>{{cite web|title=Responsible AI: Why we need Explainable AI|url=https://www.youtube.com/watch?v=A668RoogabM|website=YouTube|publisher=Accenture|accessdate=17 July 2017}}</ref> |
||
== |
== Sectors == |
||
XAI has been researched in many sectors, including: |
|||
A cross-section of industrial sectors will be affected by these requirements, as accountability is delegated to a greater or lesser extent from humans to machines. |
|||
Examples of these effects have already been seen in the following sectors: |
|||
* Neural Network Tank imaging<ref>{{cite web|title=Neual Network Tank image|url=https://neil.fraser.name/writing/tank/|website=Neil Fraser|publisher=Neil Fraser|accessdate=17 July 2017}}</ref> |
* Neural Network Tank imaging<ref>{{cite web|title=Neual Network Tank image|url=https://neil.fraser.name/writing/tank/|website=Neil Fraser|publisher=Neil Fraser|accessdate=17 July 2017}}</ref> |
||
* Antenna design ([[evolved antenna]])<ref>{{cite web|title=NASA 'Evolutionary' software automatically designs antenna|url=https://www.nasa.gov/mission_pages/st-5/main/04-55AR.html|website=NASA|publisher=NASA|accessdate=17 July 2017}}</ref> |
* Antenna design ([[evolved antenna]])<ref>{{cite web|title=NASA 'Evolutionary' software automatically designs antenna|url=https://www.nasa.gov/mission_pages/st-5/main/04-55AR.html|website=NASA|publisher=NASA|accessdate=17 July 2017}}</ref> |
Revision as of 06:21, 30 January 2018
An Explainable AI (XAI) or Transparent AI is an Artificial Intelligence (AI) whose actions can be easily understood by humans. It contrasts with "black box" AIs that employ complex opaque algorithms, where even their designers cannot explain why the AI arrived at a specific decision.[1] XAI can be used to implement a social right to explanation.[2] Transparency rarely comes for free; there are often tradeoffs between how "smart" an AI is and how transparent it is, and these tradeoffs are expected to grow larger as AI systems increase in internal complexity. The technical challenge of explaining AI decisions is sometimes known as the interpretability problem.[3]
AI systems optimize behavior to satisfy a mathematically-specified goal system chosen by the system designers, such as "maximize accuracy of assessing how positive film reviews are in the test dataset". The AI may learn useful general rules from the testset, such as "reviews containing the word 'horrible'" are likely to be negative". However, it may also learn inappropriate rules, such as "reviews containing 'Daniel Day-Lewis' are usually positive"; such rules may be undesirable if they are deemed likely to fail to generalize outside the test set, or if people consider the rule to be "cheating" or "unfair". A human can audit rules in an XAI to get an idea how likely the system is to generalize to future real-world data outside the test-set.[3]
Goals
AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit preprogrammed goals on the training data, but that do not reflect the complicated implicit desires of the human system designers. For example, a 2017 system tasked with image recognition learned to "cheat" by looking for a copyright tag that happened to be associated with horse pictures, rather than learning how to tell if a horse was actually pictured.[1] In another 2017 system, a supervised learning AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.[4][5]
One transparency project, the DARPA XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop", without greatly sacrificing AI performance. Human users should be able to understand the AI's cognition (both in real-time and after the fact), and should be able to determine when to trust the AI and when the AI should be distrusted.[6][7]
History and methods
Mycin, a research prototype that could explain which of its hand-coded rules contributed to a diagnosis in a specific case, was developed in the early 1970s.[8][9] By the 1990s researchers also began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks.[10]. Researchers in clinical expert systems creating neural network-powered decision support for clinicians have sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice.[2]
The "deep learning" methods powering cutting-edge AI in the 2010s are naturally opaque,[11] as are other complicated neural networks; genetic algorithms likewise are naturally opaque. In contrast, decision trees and Bayesian networks are more transparent to inspection.[12]
Layerwise Relevance Propagation (LRP), first described in 2015, is a technique for determining which features in a particular input vector contribute most strongly to a neural network’s output.[13][14]
In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence.[1]
- 25 April 2017: Nvidia published the paper "Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car"[15]
- 13 July 2017: Accenture recommended "Responsible AI: Why we need Explainable AI"[16]
Sectors
XAI has been researched in many sectors, including:
- Neural Network Tank imaging[17]
- Antenna design (evolved antenna)[18]
- Algorithmic trading (high-frequency trading)[19]
- Medical diagnoses[20][21]
- Autonomous vehicles[22][23]
Recent developments
As regulators, official bodies and general users dependency on AI-based dynamic systems, clearer accountability will be required for decision making processes to ensure trust and transparency. Evidence of this requirement gaining more momentum can be seen with the launch of the first global conference exclusively dedicated to this emerging discipline, the International Joint Conference on Artificial Intelligence: Workshop on Explainable Artificial Intelligence (XAI).[24]
References
- ^ a b c Sample, Ian (5 November 2017). "Computer says no: why making AIs fair, accountable and transparent is crucial". the Guardian. Retrieved 30 January 2018.
- ^ a b Edwards, Lilian; Veale, Michael (2017). "Slave to the Algorithm? Why a 'Right to an Explanation' Is Probably Not the Remedy You Are Looking For". Duke Law and Technology Review.
- ^ a b "How AI detectives are cracking open the black box of deep learning". Science | AAAS. 5 July 2017. Retrieved 30 January 2018.
- ^ "DeepMind Has Simple Tests That Might Prevent Elon Musk's AI Apocalypse". Bloomberg.com. 11 December 2017. Retrieved 30 January 2018.
- ^ "Learning from Human Preferences". OpenAI Blog. 13 June 2017. Retrieved 30 January 2018.
- ^ "Explainable Artificial Intelligence (XAI)". DARPA. DARPA. Retrieved 17 July 2017.
- ^ Holzinger, Andreas; Plass, Markus; Holzinger, Katharina; Crisan, Gloria Cerasela; Pintea, Camelia-M.; Palade, Vasile (2017-08-03). "A glass-box interactive machine learning approach for solving NP-hard problems with the human-in-the-loop". arXiv:1708.01104 [cs.AI].
- ^ Van Lent, M., Fisher, W., & Mancuso, M. (2004, July). An explainable artificial intelligence system for small-unit tactical behavior. In Proceedings of the National Conference on Artificial Intelligence (pp. 900-907). Menlo Park, CA; Cambridge, MA; London; AAAI Press; MIT Press; 1999.
- ^ Fagan, L. M., Shortliffe, E. H., & Buchanan, B. G. (1980). Computer-based medical decision making: from MYCIN to VM. Automedica, 3(2), 97-108.
- ^ Tickle, A. B.; Andrews, R.; Golea, M.; Diederich, J. (November 1998). "The truth will come to light: directions and challenges in extracting the knowledge embedded within trained artificial neural networks". IEEE Transactions on Neural Networks. 9 (6): 1057–1068. doi:10.1109/72.728352. ISSN 1045-9227.
- ^ Mukherjee, Siddhartha (27 March 2017). "A.I. Versus M.D." The New Yorker. Retrieved 30 January 2018.
- ^ Bostrom, N., & Yudkowsky, E. (2014). The ethics of artificial intelligence. The Cambridge handbook of artificial intelligence, 316-334.
- ^ Shiebler, Dan (2017-04-16). "Understanding Neural Networks with Layerwise Relevance Propagation and Deep Taylor Series". Dan Shiebler. Retrieved 2017-11-03.
- ^ Bach, Sebastian; Binder, Alexander; Montavon, Grégoire; Klauschen, Frederick; Müller, Klaus-Robert; Samek, Wojciech (2015-07-10). Suarez, Oscar Deniz (ed.). "On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation". Plos One. 10 (7). Public Library of Science (PLoS): e0130140. Bibcode:2015PLoSO..1030140B. doi:10.1371/journal.pone.0130140. ISSN 1932-6203.
{{cite journal}}
: CS1 maint: unflagged free DOI (link) - ^ Bojarski, Mariusz; Yeres, Philip; Choromanska, Anna; Choromanski, Krzysztof; Firner, Bernhard; Jackel, Lawrence; Muller, Urs (2017). "Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car". arXiv:1704.07911 [cs.CV].
- ^ "Responsible AI: Why we need Explainable AI". YouTube. Accenture. Retrieved 17 July 2017.
- ^ "Neual Network Tank image". Neil Fraser. Neil Fraser. Retrieved 17 July 2017.
- ^ "NASA 'Evolutionary' software automatically designs antenna". NASA. NASA. Retrieved 17 July 2017.
- ^ "The Flash Crash: The Impact of High Frequency Trading on an Electronic Market" (PDF). CFTC. CFTC. Retrieved 17 July 2017.
- ^ Weng, Stephen F; Reps, Jenna; Kai, Joe; Garibaldi, Jonathan M; Qureshi, Nadeem (2017). "Can machine-learning improve cardiovascular risk prediction using routine clinical data?". Plos One. 12 (4): e0174944. Bibcode:2017PLoSO..1274944W. doi:10.1371/journal.pone.0174944. PMID 28376093. Retrieved 17 July 2017.
{{cite journal}}
: CS1 maint: unflagged free DOI (link) - ^ Holzinger, Andreas; Biemann, Chris; Pattichis, Constantinos S.; Kell, Douglas B. (2017-12-28). "What do we need to build explainable AI systems for the medical domain?". arXiv:1712.09923 [cs.AI].
- ^ "Tesla says it has 'no way of knowing' if autopilot was used in fatal Chinese crash". Guardian. Guardian. Retrieved 17 July 2017.
- ^ "Joshua Brown, Who Died in Self-Driving Accident, Tested Limits of His Tesla". New York Times. New York Times. Retrieved 17 July 2017.
- ^ "IJCAI 2017 Workshop on Explainable Artificial Intelligence (XAI)". Earthlink. IJCAI. Retrieved 17 July 2017.
External links
- "'Explainable Artificial Intelligence': Cracking open the black box of AI". Computerworld. 2017-11-02. Retrieved 2017-11-02.
- Park, Dong Huk; Hendricks, Lisa Anne; Akata, Zeynep; Schiele, Bernt; Darrell, Trevor; Rohrbach, Marcus (2016-12-14). "Attentive Explanations: Justifying Decisions and Pointing to the Evidence". arXiv:1612.04757.
{{cite journal}}
: Cite journal requires|journal=
(help); Invalid|ref=harv
(help) - "Explainable AI: Making machines understandable for humans". Explainable AI: Making machines understandable for humans. Retrieved 2017-11-02.
- "End-to-End Deep Learning for Self-Driving Cars". Parallel Forall. 2016-08-17. Retrieved 2017-11-02.
- "Explaining How End-to-End Deep Learning Steers a Self-Driving Car". Parallel Forall. 2017-05-23. Retrieved 2017-11-02.
- "New isn't on its way. We're applying it right now". Accenture. 2016-10-25. Retrieved 2017-11-02.
- Knight, Will (2017-03-14). "DARPA is funding projects that will try to open up AI's black boxes". MIT Technology Review. Retrieved 2017-11-02.
{{cite web}}
: Invalid|ref=harv
(help) - Alvarez-Melis, David; Jaakkola, Tommi S. (2017-07-06). "A causal framework for explaining the predictions of black-box sequence-to-sequence models" (PDF). 1707: arXiv:1707.01943. arXiv:1707.01943. Bibcode:2017arXiv170701943A. Retrieved 2017-11-09.
{{cite journal}}
: Cite journal requires|journal=
(help); Invalid|ref=harv
(help) - Bojarski, Mariusz; Yeres, Philip; Choromanska, Anna; Choromanski, Krzysztof; Firner, Bernhard; Jackel, Lawrence; Muller, Urs (2017-04-25). "Explaining How a Deep Neural Network Trained with End-to-End Learning Steers a Car". 1704: arXiv:1704.07911. arXiv:1704.07911. Bibcode:2017arXiv170407911B.
{{cite journal}}
: Cite journal requires|journal=
(help); Invalid|ref=harv
(help)