An Explainability-Centric Requirements Analysis Framework for Machine Learning Applications

Pei Zhongyi; Liu Lin; Wang Chen; Wang Jianmin

doi:10.7544/issn1000-1239.202220794

Journal of Computer Research and Development > 2024 > 61(4): 983-1002. > DOI: 10.7544/issn1000-1239.202220794 CSTR: 32373.14.issn1000-1239.202220794

Pei Zhongyi, Liu Lin, Wang Chen, Wang Jianmin. An Explainability-Centric Requirements Analysis Framework for Machine Learning Applications[J]. Journal of Computer Research and Development, 2024, 61(4): 983-1002. DOI: 10.7544/issn1000-1239.202220794

Citation:

PDF (1278 KB)

An Explainability-Centric Requirements Analysis Framework for Machine Learning Applications

1.
National Engineering Research Center for Big Data Software (Tsinghua University), Beijing 100084
2.
School of Software, Tsinghua University, Beijing 100084

Funds: This work was supported by the National Key Research and Development Program (2021YFB1715200) and the National Natural Science Foundation of Innovation Group Project (62021002).

More Information

Author Bio:
Pei Zhongyi: born in 1987. PhD, assistant professor. Member of CCF. His main research interests include software engineering for machine learning, industrial intelligence

Liu Lin: born in 1973. PhD, professor. Her main research interests include requirements engineering, domain data and knowledge engineering

Wang Chen: born in 1981. Associate professor. His main research interests include database and industrial big data

Wang Jianmin: born in 1968. PhD, professor. His main research interests include database, and industrial big data
Received Date: September 05, 2022
Revised Date: August 14, 2023
Available Online: January 24, 2024

Graphical Abstract

Abstract

Abstract

Data-driven intelligent software based on machine learning technology is an important means to realize industrial digital transformation. The research and development processes of data-driven intelligent software require the combined use of software requirements engineering, data and domain knowledge engineering, machine learning and so on. This process involves many subjects and roles, making it extremely challenging to clearly explain why and how the domain knowledge, business logic and data semantics relate to each other. Hence, a systematic requirements engineering approach is needed to explicitly address the explainability requirements issues of data-driven intelligence applications. It is still a fast-evolving research field which requires the proper embedding of various domain models and end-to-end machine learning technology fused into a given business processes. A key research question is how to deal with explainability as a core requirement for safety-critical scenarios in industrial, medical and other applications. We provide a research overview on requirements engineering for machine learning applications, in relation to explainability. First, the research status quo, research foci and representative research progress are reviewed. Then, an explainability-centric requirements analysis framework for machine learning applications is proposed, and some open important issues are put forward. Finally, based on the proposed framework, a case study of industrial intelligence application is discussed to illustrate the proposed requirements analysis methodological framework.
- requirements engineering,
- machine learning,
- domain model,
- industrial engineering,
- explainability

FullText(HTML)

References (199)

References

[1]	Ashmore R, Calinescu R, Paterson C. Assuring the machine learning lifecycle: Desiderata, methods, and challenges[J]. ACM Computing Surveys, 2021, 54(5): 1−39
[2]	Habibullah K M, Horkoff J. Non-functional requirements for machine learning: Understanding current use and challenges in industry[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 13−23
[3]	Mohseni S, Zarei N, Ragan E D. A multidisciplinary survey and framework for design and evaluation of explainable AI systems[J]. ACM Transactions on Interactive Intelligent Systems, 2021, 11(3/4): 1−45
[4]	Mehrabi N, Morstatter F, Saxena N, et al. A survey on bias and fairness in machine learning[J]. ACM Computing Surveys, 2021, 54(6): 1−35
[5]	Floridi L. Establishing the rules for building trustworthy AI[J]. Nature Machine Intelligence, 2019, 1(6): 261−262 doi: 10.1038/s42256-019-0055-y
[6]	汪烨,陈骏武,夏鑫,等. 智能需求获取与建模研究综述[J]. 计算机研究与发展,2021,58(4):683−705 doi: 10.7544/issn1000-1239.2021.20200740 Wang Ye, Chen Junwu, Xia Xin, et al. Intelligent requirements elicitation and modeling: A literature review[J]. Journal of Computer Research and Development, 2021, 58(4): 683−705(in Chinese) doi: 10.7544/issn1000-1239.2021.20200740
[7]	Iqbal T, Elahidoost P, Lucio L. A bird’s eye view on requirements engineering and machine learning[C]//Proc of the 25th Asia-Pacific Software Engineering Conf (APSEC). Piscataway, NJ: IEEE, 2018: 11−20
[8]	Villamizar H, Escovedo T, Kalinowski M. Requirements engineering for machine learning: A systematic mapping study[C]//Proc of the 47th Euromicro Conf on Software Engineering and Advanced Applications (SEAA). Piscataway, NJ: IEEE, 2021: 29−36
[9]	Willard J, Jia Xiaowei, Xu Shaoming, et al. Integrating physics-based modeling with machine learning: A survey[J]. arXiv preprint, arXiv: 2003.04919, 2020
[10]	Harel D, Marron A, Sifakis J. Autonomics: In search of a foundation for next-generation autonomous systems[J]. Proceedings of the National Academy of Sciences, 2020, 117(30): 17491−17498 doi: 10.1073/pnas.2003162117
[11]	Horkoff J, Aydemir F B, Cardoso E, et al. Goal-oriented requirements engineering: An extended systematic mapping study[J]. Requirements Engineering, 2019, 24(2): 133−160 doi: 10.1007/s00766-017-0280-z
[12]	Wolny S, Mazak A, Carpella C, et al. Thirteen years of SysML: A systematic mapping study[J]. Software and Systems Modeling, 2020, 19(1): 111−169 doi: 10.1007/s10270-019-00735-y
[13]	Azevedo A, Santos M F. KDD, SEMMA and CRISP-DM: A parallel overview[C/OL]//Proc of the IADIS European Conf on Data Mining, 2008[2022-05-30].https://www.iadisportal.org/digital-library/kdd-semma-and-crisp-dm-a-parallel-overview
[14]	Vogelsang A, Borg M. Requirements engineering for machine learning: Perspectives from data scientists[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 245−251
[15]	Amershi S, Begel A, Bird C, et al. Software engineering for machine learning: A case study[C]//Proc of the 41st IEEE/ACM Int Conf on Software Engineering: Software Engineering in Practice (ICSE-SEIP). Piscataway, NJ: IEEE, 2019: 291−300
[16]	Silva G R S, Canedo E D. Requirements engineering challenges and techniques in building chatbots[C]//Proc of the 14th Int Conf on Agents and Artificial Intelligence. Setúbal, Portugal: SCITEPRESS, 2022: 180−187
[17]	Bencomo N, Guo J L C, Harrison R, et al. The secret to better AI and better software (is requirements engineering)[J]. IEEE Software, 2021, 39(1): 105−110
[18]	Yoshioka N, Husen J H, Tun H T, et al. Landscape of requirements engineering for machine learning-based AI systems[C]//Proc of the 28th Asia-Pacific Software Engineering Conf Workshops (APSEC Workshops). Piscataway, NJ: IEEE, 2021: 5−8
[19]	Ahmad K, Bano M, Abdelrazek M, et al. What’s up with requirements engineering for artificial intelligence systems?[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 1−12
[20]	Tukur M, Umar S, Hassine J. Requirement engineering challenges: A systematic mapping study on the academic and the industrial perspective[J]. Arabian Journal for Science and Engineering, 2021, 46(4): 3723−3748 doi: 10.1007/s13369-020-05159-1
[21]	Heyn H M, Knauss E, Muhammad A P, et al. Requirement engineering challenges for AI-intense systems development[C]//Proc of the 1st IEEE/ACM Workshop on AI Engineering-Software Engineering for AI (WAIN). Piscataway, NJ: IEEE, 2021: 89−96
[22]	Ferreira F, Silva L L, Valente M T. Software engineering meets deep learning: A mapping study[C]//Proc of the 36th Annual ACM Symp on Applied Computing. New York: ACM, 2021: 1542−1549
[23]	Dalpiaz F, Niu Nan. Requirements engineering in the days of artificial intelligence[J]. IEEE Software, 2020, 37(4): 7−10 doi: 10.1109/MS.2020.2986047
[24]	Kostova B, Gurses S, Wegmann A. On the interplay between requirements, engineering, and artificial intelligence[J]. Information and Software Technology, 2023, 158: 107176 doi: 10.1016/j.infsof.2023.107176
[25]	Belani H, Vukovic M, Car Ž. Requirements engineering challenges in building AI-based complex systems[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 252−255
[26]	Lwakatare L E, Raj A, Bosch J, et al. A taxonomy of software engineering challenges for machine learning systems: An empirical investigation[C]//Proc of the Int Conf on Agile Software Development. Berlin: Springer, 2019: 227−243
[27]	Bosch J, Olsson H H, Crnkovic I. It takes three to tango: Requirement, outcome/data, and AI driven development[C]//Proc of the 1st Int Workshop on Software-intensive Business: Start-ups, Ecosystems and Platforms (SiBW). Berlin: Springer, 2018: 177−192
[28]	Studer S, Bui T B, Drescher C, et al. Towards CRISP-ML (Q): A machine learning process model with quality assurance methodology[J]. Machine Learning and Knowledge Extraction, 2021, 3(2): 392−413 doi: 10.3390/make3020020
[29]	Nalchigar S, Yu E, Obeidi Y, et al. Solution patterns for machine learning[C]//Proc of the 31st Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 627−642
[30]	Nalchigar S, Yu E, Keshavjee K. Modeling machine learning requirements from three perspectives: A case report from the healthcare domain[J]. Requirements Engineering, 2021, 26(2): 237−254 doi: 10.1007/s00766-020-00343-z
[31]	Chuprina T, Mendez D, Wnuk K. Towards artefact-based requirements engineering for data-centric systems[J]. arXiv preprint, arXiv: 2103.05233, 2021
[32]	Barrera J M, Reina Reina A, Maté A, et al. Applying i* in conceptual modelling in machine learning[C]//Proc of the 14th Int iStar Workshop. Berlin: Springer, 2020: 56−62
[33]	Camilli M, Felderer M, Giusti A, et al. Towards risk modeling for collaborative AI[C]//Proc of the 1st IEEE/ACM Workshop on AI Engineering-Software Engineering for AI (WAIN). Piscataway, NJ: IEEE, 2021: 51−54
[34]	Alrajeh D, Cailliau A, van Lamsweerde A. Adapting requirements models to varying environments[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2020: 50−61
[35]	Deshpande A, Sharp H. Responsible AI systems: Who are the stakeholders?[C]//Proc of the 2022 AAAI/ACM Conf on AI, Ethics, and Society. New York: ACM, 2022: 227−236
[36]	Khuat T T, Kedziora D J, Gabrys B. The roles and modes of human interactions with automated machine learning systems[J]. arXiv preprint, arXiv: 2205.04139, 2022
[37]	Nahar N, Zhou Shurui, Lewis G, et al. Collaboration challenges in building ML-enabled systems: Communication, documentation, engineering, and process[C]//Proc of the 44th Int Conf on Software Engineering. New York: ACM, 2022: 413−425
[38]	Odong L A, Perini A, Susi A. Requirements engineering for collaborative artificial intelligence systems: A literature survey[C]//Proc of the 16th Int Conf on Research Challenges in Information Science. Berlin: Springer, 2022: 409−425
[39]	Barclay I, Abramson W. Identifying roles, requirements and responsibilitiesin trustworthy AI systems[C]//Proc of the 2021 ACM Int Joint Conf on Pervasive and Ubiquitous Computing and Proc of the 2021 ACM Int Symp on Wearable Computers. New York: ACM, 2021: 264−271
[40]	Piorkowski D, Park S, Wang A Y, et al. How AI developers overcome communication challenges in a multidisciplinary team: A case study[J]. Proceedings of the ACM on Human-Computer Interaction, 2021, 5(CSCW1): 131
[41]	Camilli M, Felderer M, Giusti A, et al. Risk-driven compliance assurance for collaborative AI systems: A vision paper[C]//Proc of the 27th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2021: 123−130
[42]	Zhang A X, Muller M, Wang Dakuo. How do data science workers collaborate? Roles, workflows, and tools[J]. Proceedings of the ACM on Human-Computer Interaction, 2020, 4(CSCW1): 22
[43]	Berry D M. Requirements engineering for artificial intelligence: What is a requirements specification for an artificial intelligence?[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 19−25
[44]	Al M, Ma Yihong, Alarcon P, et al. RESAM: Requirements elicitation and specification for deep-learning anomaly models with applications to UAV flight controllers[C]//Proc of the 30th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2022: 153−165
[45]	Villamizar H, Kalinowski M, Lopes H. Towards perspective-based specification of machine learning-enabled systems[J]. arXiv preprint, arXiv: 2206.09760, 2022
[46]	Gillani M, Niaz H A, Ullah A. Integration of software architecture in requirements elicitation for rapid software development[J]. IEEE Access, 2022, 10: 56158−56178 doi: 10.1109/ACCESS.2022.3177659
[47]	Gabriel S, Bentler D, Grote E M, et al. Requirements analysis for an intelligent workforce planning system: A socio-technical approach to design AI-based systems[J]. Procedia CIRP, 2022, 109: 431−436 doi: 10.1016/j.procir.2022.05.274
[48]	Wang Xi, Miao Weikai. A framework for requirements specification of machine-learning systems[C]//Proc of the 32nd Conf on Software Engineering and Knowledge Engineering (SEKE). Pittsburgh, Pennsylvania: KSI Research Inc, 2022: 7−12
[49]	Ahmad K. Human-centric requirements engineering for artificial intelligence software systems[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 468−473
[50]	Schuh G, Scholz P, Leich T, et al. Identifying and analyzing data model requirements and technology potentials of machine learning systems in the manufacturing industry of the future[C]//Proc of the 61st Int Scientific Conf on Information Technology and Management Science of Riga Technical University (ITMS). Piscataway, NJ: IEEE, 2020: 20178653
[51]	Cirqueira D, Nedbal D, Helfert M, et al. Scenario-based requirements elicitation for user-centric explainable AI[C]//Proc of the 4th Int Cross-Domain Conf for Machine Learning and Knowledge Extraction. Berlin: Springer, 2020: 321−341
[52]	d’Aloisio G, Di Marco A, Stilo G. Modeling quality and machine learning pipelines through extended feature models[J]. arXiv preprint, arXiv: 2207.07528, 2022
[53]	Levy N, Katz G. RoMA: A method for neural network robustness measurement and assessment[J]. arXiv preprint, arXiv: 2110.11088, 2021
[54]	Siebert J, Joeckel L, Heidrich J, et al. Construction of a quality model for machine learning systems[J]. Software Quality Journal, 2022, 30(2): 307−335 doi: 10.1007/s11219-021-09557-y
[55]	Haindl P, Hoch T, Dominguez J, et al. Quality characteristics of a software platform for human-AI teaming in smart manufacturing[J]. arXiv preprint, arXiv: 2205.15767, 2022
[56]	Madaio M, Egede L, Subramonyam H, et al. Assessing the fairness of AI systems: AI practitioners’ processes, challenges, and needs for support[J]. Proceedings of the ACM on Human-Computer Interaction, 2022, 6(CSCW1): 52
[57]	Perera A, Aleti A, Tantithamthavorn C, et al. Search-based fairness testing for regression-based machine learning systems[J]. Empirical Software Engineering, 2022, 27(3): 1−36
[58]	Chen Huaming, Babar M A. Security for machine learning-based software systems: A survey of threats, practices and challenges[J]. arXiv preprint, arXiv: 2201.04736, 2022
[59]	Georgieva I, Lazo C, Timan T, et al. From AI ethics principles to data science practice: A reflection and a gap analysis based on recent frameworks and practical experience[J]. AI and Ethics, 2022, 2: 697−711 doi: 10.1007/s43681-021-00127-3
[60]	Steimers A, Schneider M. Sources of risk of AI systems[J]. International Journal of Environmental Research and Public Health, 2022, 19(6): 3641
[61]	Bartels R, Dudink J, Haitjema S, et al. A perspective on a quality management system for AI/ML-based clinical decision support in hospital care[J]. Frontiers in Digital Health, 2022, 4: 942588
[62]	d’Aloisio G. Quality-driven machine learning-based data science pipeline realization: A software engineering approach[C]//Proc of the 44th ACM/IEEE Int Conf on Software Engineering: Companion Proceedings. Piscataway, NJ: IEEE, 2022: 291−293
[63]	Khan A, Siddiqui I F, Shaikh M, et al. Handling non-fuctional requirements in IoT-based machine learning systems[C]//Proc of the 7th Joint Int Conf on Digital Arts, Media and Technology with ECTI Northern Section Conf on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON). Piscataway, NJ: IEEE, 2022: 477−479
[64]	Truong H L, Nguyen T M. QoA4ML—A framework for supporting contracts in machine learning services[C]//Proc of the IEEE Int Conf on Web Services (ICWS). Piscataway, NJ: IEEE, 2021: 465−475
[65]	Muñante D, Perini A, Kifetew F M, et al. Combining risk and variability modelling for requirements analysis in SAS engineering[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 396−401
[66]	Eliyahu T, Kazak Y, Katz G, et al. Verifying learning-augmented systems[C]//Proc of the 2021 ACM SIGCOMM. New York: ACM, 2021: 305−318
[67]	Mauri L, Damiani E. STRIDE-AI: An approach to identifying vulnerabilities of machine learning assets[C]//Proc of the IEEE Int Conf on Cyber Security and Resilience (CSR). Piscataway, NJ: IEEE, 2021: 147−154
[68]	Dey S, Lee S W. Multilayered review of safety approaches for machine learning-based systems in the days of AI[J]. Journal of Systems and Software, 2021, 176: 110941
[69]	Wang Jingyi, Chen Jialuo, Sun Youcheng, et al. Robot: Robustness-oriented testing for deep learning systems[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 300−311
[70]	Baluta T, Chua Z L, Meel K S, et al. Scalable quantitative verification for deep neural networks[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 312−323
[71]	Zhang J M, Harman M. "Ignorance and Prejudice" in software fairness[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1436−1447
[72]	Hobbs R. Integrating ethically align design into agile and CRISP-DM[C]//Proc of the Southeast Conf 2021. Piscataway, NJ: IEEE, 2021: 20631991
[73]	Tariq S, Cheema S M. Approaches for non-functional requirement modeling: A literature survey[C]//Proc of the 4th Int Conf on Computing & Information Sciences (ICCIS). Piscataway, NJ: IEEE, 2021: 21573902
[74]	Cerqueira J, Acco H, Dias E. Ethical guidelines and principles in the context of artificial intelligence[C]//Proc of the 17th Brazilian Symp on Information Systems. New York: ACM, 2021: 36
[75]	Byun T, Rayadurgam S. Manifold for machine learning assurance[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). Piscataway, NJ: IEEE, 2020: 97−100
[76]	Ishikawa F, Matsuno Y. Evidence-driven requirements engineering for uncertainty of machine learning-based systems[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 346−351
[77]	Damyanova B. Quality attributes in AI-ML-based systems: Differences and challenges[D]. Stuttgart: University of Stuttgart, 2020
[78]	Balasubramaniam N, Kauppinen M, Kujala S, et al. Ethical guidelines for solving ethical issues and developing AI systems[C]//Proc of the Int Conf on Product-Focused Software Process Improvement. Berlin: Springer, 2020: 331−346
[79]	Riccio V, Tonella P. Model-based exploration of the frontier of behaviours for deep learning system testing[C]//Proc of the 28th ACM Joint Meeting on European Software Engineering Conf and Symp on the Foundations of Software Engineering. New York: ACM, 2020: 876−888
[80]	Wilhjelm C, Younis A A. A threat analysis methodology for security requirements elicitation in machine learning based systems[C]//Proc of the 20th IEEE Int Conf on Software Quality, Reliability and Security Companion (QRS-C). Piscataway, NJ: IEEE, 2020: 426−433
[81]	Nakamichi K, Ohashi K, Namba I, et al. Requirements-driven method to determine quality characteristics and measurements for machine learning software and its evaluation[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 260−270
[82]	Humbatova N, Jahangirova G, Bavota G, et al. Taxonomy of real faults in deep learning systems[C]//Proc of the 42nd ACM/IEEE Int Conf on Software Engineering. Piscataway, NJ: IEEE, 2020: 1110−1121
[83]	Chechik M. Uncertain requirements, assurance and machine learning[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 2−3
[84]	Horkoff J. Non-functional requirements for machine learning: Challenges and new directions[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 386−391
[85]	Kuwajima H, Ishikawa F. Adapting square for quality assessment of artificial intelligence systems[C]//Proc of the IEEE Int Symp on Software Reliability Engineering Workshops (ISSREW). Piscataway, NJ: IEEE, 2019: 13−18
[86]	Bhatia J, Breaux T D. Semantic incompleteness in privacy policy goals[C]//Proc of the 26th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2018: 159−169
[87]	Zhang Boyu, Magaña J C, Davoodi A. Analysis of security of split manufacturing using machine learning[C]//Proc of the 55th Annual Design Automation Conf. New York: ACM, 2018: 2767−2780
[88]	Hayrapetian A, Raje R. Empirically analyzing and evaluating security features in software requirements[C]//Proc of the 11th Innovations in Software Engineering Conf. New York: ACM, 2018: 9
[89]	Hoel T, Griffiths D, Chen Weiqin. The influence of data protection and privacy frameworks on the design of learning analytics systems[C]//Proc of the 7th Int Learning Analytics & Knowledge Conf. New York: ACM, 2017: 243−252
[90]	Burwinkel H, Matz H, Saur S, et al. Physics-aware learning and domain-specific loss design in ophthalmology[J]. Medical Image Analysis, 2022, 76: 102314
[91]	Von Rueden L, Mayer S, Beckh K, et al. Informed machine learning—A taxonomy and survey of integrating knowledge into learning systems[J]. arXiv preprint, arXiv: 1903.12394, 2019
[92]	Vo K, Nguyen T, Pham D, et al. Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media[J]. International Journal of Computational Vision and Robotics, 2019, 9(5): 458−485 doi: 10.1504/IJCVR.2019.102286
[93]	Abbasi A, Nataraj C. Physics-informed machine learning for uncertainty reduction in time response reconstruction of a dynamic system[J]. IEEE Internet Computing, 2022, 26(4): 35−44 doi: 10.1109/MIC.2022.3170736
[94]	Huang Bin, Wang Jianhui. Applications of physics-informed neural networks in power systems-a review[J]. IEEE Transactions on Power Systems, 2022, 38(1): 572−588
[95]	Zhao W L, Gentine P, Reichstein M, et al. Physics-constrained machine learning of evapotranspiration[J]. Geophysical Research Letters, 2019, 46(24): 14496−14507 doi: 10.1029/2019GL085291
[96]	Lin Jing, Zhang Yu, Khoo E. Hybrid physics-based and data-driven modeling with calibrated uncertainty for lithium-ion battery degradation diagnosis and prognosis[J]. arXiv preprint, arXiv: 2110.13661, 2021
[97]	Cornelio J, Mohd Razak S, Cho Y, et al. Residual learning to integrate neural network and physics-based models for improved production prediction in unconventional reservoirs[J]. SPE Journal, 2022, 27(6): 1−22
[98]	Soleimani M, Intezari A, Pauleen D J. Mitigating cognitive biases in developing AI-assisted recruitment systems: A knowledge-sharing approach[J]. International Journal of Knowledge Management, 2022, 18(1): 1−18
[99]	Willard J, Jia Xiaowei, Xu Shaoming, et al. Integrating scientific knowledge with machine learning for engineering and environmental systems[J]. ACM Computing Surveys, 2022, 55(4): 1−37
[100]	Luo Xing, Zhang Dongxiao, Zhu Xu. Deep learning based forecasting of photovoltaic power generation by incorporating domain knowledge[J]. Energy, 2021, 225: 120240
[101]	Park S, Wang A Y, Kawas B, et al. Facilitating knowledge sharing from domain experts to data scientists for building NLP models[C]//Proc of the 26th Int Conf on Intelligent User Interfaces. New York: ACM, 2021: 585−596
[102]	Xie Xiaozheng, Niu Jianwei, Liu Xuefeng, et al. A survey on incorporating domain knowledge into deep learning for medical image analysis[J]. Medical Image Analysis, 2021, 69: 101985
[103]	Van Oort C M. Leveraging domain knowledge in deep learning systems[D]. Burlington: The University of Vermont and State Agricultural College, 2021
[104]	Chitchyan R, Bird C. Theory as a source of software requirements[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 227−237
[105]	Vázquez-Ingelmo A, García Peñalvo F J, Therón R. Advances in the use of domain engineering to support feature identification and generation of information visualizations[C]//Proc of the 8th Int Conf on Technological Ecosystems for Enhancing Multiculturality. New York: ACM, 2020: 1053−1056
[106]	Chai Yidong, Liu Hongyan, Xu Jie. Glaucoma diagnosis based on both hidden features and domain knowledge through deep learning models[J]. Knowledge-Based Systems, 2018, 161: 147−156 doi: 10.1016/j.knosys.2018.07.043
[107]	Song Yangqiu, Roth D. Machine learning with world knowledge: The position and survey[J]. arXiv preprint, arXiv: 1705.02908, 2017
[108]	Haidry S Z, Falkner K, Szabo C. Identifying domain-specific cognitive strategies for software engineering[C]//Proc of the 2017 ACM Conf on Innovation and Technology in Computer Science Education. New York: ACM, 2017: 206−211
[109]	Laato S, Mäntymäki M, Minkkinen M, et al. Integrating machine learning with software development lifecycles: Insights from experts[C/OL]//Proc of the 30th European Conf on Information Systems. Association for Information Systems, 2022[2022-05-30].https://aisel.aisnet.org/ecis2022_rp/118
[110]	Washizaki H, Uchida H, Khomh F, et al. Machine learning architecture and design patterns[J/OL]. IEEE Software, 2020[2022-04-01]. http://g7.washi.cs.waseda.ac.jp/wp-content/uploads/2019/12/IEEE_Software_19__ML_Patterns.pdf
[111]	Lewis G A, Ozkaya I, Xu Xiwei. Software architecture challenges for ML systems[C]//Proc of the 37th IEEE Int Conf on Software Maintenance and Evolution (ICSME). Piscataway, NJ: IEEE, 2021: 634−638
[112]	Serban A, Visser J. An empirical study of software architecture for machine learning[J]. arXiv preprint, arXiv: 2105.12422, 2021
[113]	Chen Zhenpeng, Yao Huihan, Lou Yiling, et al. An empirical study on deployment faults of deep learning based mobile applications[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 674-685
[114]	Serban A, van der Blom K, Hoos H, et al. Adoption and effects of software engineering best practices in machine learning[C]//Proc of the 14th ACM/IEEE Int Symp on Empirical Software Engineering and Measurement (ESEM). New York: ACM, 2020: 3
[115]	John M M, Olsson H H, Bosch J. Developing ML/DL models: A design framework[C/OL]//Proc of the 14th Int Conf on Software and System Processes. New York: ACM, 2020[2022-05-30].https://www.diva-portal.org/smash/get/diva2:1553907/FULLTEXT01.pdf
[116]	Correia J L, Pereira J A, Mello R, et al. Brazilian data scientists: Revealing their challenges and practices on machine learning model development[C]//Proc of the 19th Brazilian Symp on Software Quality. New York: ACM, 2020: 10
[117]	Reimann L, Kniesel-Wünsche G. Achieving guidance in applied machine learning through software engineering techniques[C]//Proc of the 4th Int Conf on Art, Science, and Engineering of Programming. New York: ACM, 2020: 7−12
[118]	Kourouklidis P, Kolovos D, Matragkas N, et al. Towards a low-code solution for monitoring machine learning model performance[C]//Proc of the 23rd ACM/IEEE Int Conf on Model Driven Engineering Languages and Systems: Companion Conf. Piscataway, NJ: IEEE, 2020: 62
[119]	Prado F F, Digiampietri L A. A systematic review of automated feature engineering solutions in machine learning problems[C]//Proc of the 16th Brazilian Symp on Information Systems. New York: ACM, 2020: 12
[120]	Oppold S, Herschel M. A system framework for personalized and transparent data-driven decisions[C]//Proc of the 32nd Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2020: 153−168
[121]	Ole M, Volker G. Towards concept based software engineering for intelligent agents[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 42−48
[122]	Hesenius M, Schwenzfeier N, Meyer O, et al. Towards a software engineering process for developing data-driven applications[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 35−41
[123]	Mucha T M, Ma Sijia, Abhari K. Sustainability of machine learning-based solutions: A lifecycle perspective[C/OL]//Proc of the Pacific Asia Conf on Information Systems. Association for Information Systems, 2022[2022-05-30].https://aisel.aisnet.org/pacis2022/262/
[124]	Yu Kui, Guo Xianjie, Liu Lin, et al. Causality-based feature selection: Methods and evaluations[J]. ACM Computing Surveys, 2020, 53(5): 1−36
[125]	Cabrera Á A, Ribeiro M T, Lee B, et al. What did my AI learn? How data scientists make sense of model behavior[J]. ACM Transactions on Computer-Human Interaction, 2022, 30(1): 1−27
[126]	Sun Jiao, Liao Q V, Muller M, et al. Investigating explainability of generative AI for code through scenario-based design[C]//Proc of the 27th Int Conf on Intelligent User Interfaces. New York: ACM, 2022: 212−228
[127]	Shen M W. Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient[J]. arXiv preprint, arXiv: 2202.05302, 2022
[128]	Balasubramaniam N, Kauppinen M, Hiekkanen K, et al. Transparency and explainability of AI systems: Ethical guidelines in practice[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 3−18
[129]	Piorkowski D, Richards J, Hind M. Evaluating a methodology for increasing AI transparency: A case study[J]. arXiv preprint, arXiv: 2201.13224, 2022
[130]	Sadeghi M, Klös V, Vogelsang A. Cases for explainable software systems: Characteristics and examples[C]//Proc of the 29th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2021: 181−187
[131]	Langer M, Baum K, Hartmann K, et al. Explainability auditing for intelligent systems: A rationale for multi-disciplinary perspectives[C]//Proc of the 29th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2021: 164−168
[132]	Chazette L, Brunotte W, Speith T. Exploring explainability: A definition, a model, and a knowledge catalogue[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 197−208
[133]	Velez M, Jamshidi P, Siegmund N, et al. White-box analysis over machine learning: Modeling performance of configurable systems[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1072−1084
[134]	Narteni S, Ferretti M, Orani V, et al. From explainable to reliable artificial intelligence[C]//Proc of the 5th Int Cross-Domain Conf for Machine Learning and Knowledge Extraction. Berlin: Springer, 2021: 255−273
[135]	Zohdinasab T, Riccio V, Gambi A, et al. Deephyperion: Exploring the feature space of deep learning-based systems through illumination search[C]//Proc of the 30th ACM SIGSOFT Int Symp on Software Testing and Analysis. New York: ACM, 2021: 79−90
[136]	Liao Q V, Gruen D, Miller S. Questioning the AI: Informing design practices for explainable AI user experiences[C]//Proc of the 2020 CHI Conf on Human Factors in Computing Systems. New York: ACM, 2020: 1−15
[137]	Köhl M A, Baum K, Langer M, et al. Explainability as a non-functional requirement[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 363−368
[138]	Yang Zijiang, Al-Bahrani R, Reid A C E, et al. Deep learning based domain knowledge integration for small datasets: Illustrative applications in materials informatics[C]//Proc of the Int Joint Conf on Neural Networks (IJCNN). Piscataway, NJ: IEEE, 2019: 19028509
[139]	Wolf C T. Explainability scenarios: Towards scenario-based XAI design[C]//Proc of the 24th Int Conf on Intelligent User Interfaces. New York: ACM, 2019: 252−257
[140]	Eiband M, Schneider H, Bilandzic M, et al. Bringing transparency design into practice[C]//Proc of the 23rd Int Conf on Intelligent User Interfaces. New York: ACM, 2018: 211−223
[141]	Thakkar D, Ismail A, Kumar P, et al. When is machine learning data good?: Valuing in public health datafication[C]//Proc of the CHI Conf on Human Factors in Computing Systems. New York: ACM, 2022: 322
[142]	Jung J Y, Steinberger T, King J L, et al. How domain experts work with data: Situating data science in the practices and settings of craftwork[J]. Proceedings of the ACM on Human-Computer Interaction, 2022, 6(CSCW1): 58
[143]	Lwakatare L E, Rånge E, Crnkovic I, et al. On the experiences of adopting automated data validation in an industrial machine learning project[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering: Software Engineering in Practice (ICSE-SEIP). Piscataway, NJ: IEEE, 2021: 248−257
[144]	Hutchinson B, Smart A, Hanna A, et al. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure[C]//Proc of the 2021 ACM Conf on Fairness, Accountability, and Transparency. New York: ACM, 2021: 560-575
[145]	Shao Zhijun, Wu Ji, Zhao Wenxiao, et al. How data plays in the requirements of face recognition system: A concern driven systematic literature review[C]//Proc of the 28th Asia-Pacific Software Engineering Conf Workshops (APSEC Workshops). Piscataway, NJ: IEEE, 2021: 9−12
[146]	Astegher M, Busetta P, Perini A, et al. Specifying requirements for data collection and analysis in data-driven RE. A research preview[C]//Proc of the 27th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2021: 182−188
[147]	Hayes J H, Payne J, Leppelmeier M. Toward improved artificial intelligence in requirements engineering: Metadata for tracing datasets[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 256−262
[148]	Biffl S, Lüder A, Rinker F, et al. Efficient engineering data exchange in multi-disciplinary systems engineering[C]//Proc of the 29th Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 17−31
[149]	Mei Songzhu, Liu Cong, Wang Qinglin, et al. Model provenance management in MLOps pipeline[C]//Proc of the 8th Int Conf on Computing and Data Engineering. New York: ACM, 2022: 45−50
[150]	Kaminwar S R, Goschenhofer J, Thomas J, et al. Structured verification of machine learning models in industrial settings[J/OL]. Big Data, 2021[2022-04-01].https://www.liebertpub.com/doi/10.1089/big.2021.0112
[151]	Xiao Yan, Beschastnikh I, Rosenblum D S, et al. Self-checking deep neural networks in deployment[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 372−384
[152]	Wang Song, Shrestha N, Subburaman A K, et al. Automatic unit test generation for machine learning libraries: How far are we?[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1548−1560
[153]	Hestness J, Ardalani N, Diamos G. Beyond human-level accuracy: Computational challenges in deep learning[C/OL]//Proc of the 24th Symp on Principles and Practice of Parallel Programming. New York: ACM, 2019[2022-05-30]. https://dl.acm.org/doi/10.1145/3293883.3295710
[154]	Gharibi G, Walunj V, Rella S, et al. Modelkb: Towards automated management of the modeling lifecycle in deep learning[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 28−34
[155]	Borg M, Englund C, Wnuk K, et al. Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry[J]. arXiv preprint, arXiv: 1812.05389, 2018
[156]	Czvetkó T, Kummer A, Ruppert T, et al. Data-driven business process management-based development of Industry 4.0 solutions[J]. CIRP Journal of Manufacturing Science and Technology, 2022, 36: 117−132 doi: 10.1016/j.cirpj.2021.12.002
[157]	Merkelbach S, Von Enzberg S, Kühn A, et al. Towards a process model to enable domain experts to become citizen data scientists for industrial applications[C]//Proc of the 5th IEEE Int Conf on Industrial Cyber-Physical Systems (ICPS). Piscataway, NJ: IEEE, 2022
[158]	Heyn H M, Subbiah P, Linder J, et al. Setting AI in context: A case study on defining the context and operational design domain for automated driving[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 199−215
[159]	Zhang Ran, Albrecht A, Kausch J, et al. DDE process: A requirements engineering approach for machine learning in automated driving[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 269−279
[160]	Borg M, Bronson J, Christensson L, et al. Exploring the assessment list for trustworthy AI in the context of advanced driver-assistance systems[C]//Proc of the 2nd IEEE/ACM Int Workshop on Ethics in Software Engineering Research and Practice (SEthics). Piscataway, NJ: IEEE, 2021: 5−12
[161]	Martínez-Fernández S, Franch X, Jedlitschka A, et al. Developing and operating artificial intelligence models in trustworthy autonomous systems[C]//Proc of the 15th Int Conf on Research Challenges in Information Science. Berlin: Springer, 2021: 221−229
[162]	Kolyshkina I, Simoff S. Interpretability of machine learning solutions in public healthcare: The CRISP-ML approach[J]. Frontiers in Big Data, 2021, 4: 660206
[163]	Pinto A. Requirement specification, analysis and verification for autonomous systems[C]//Proc of the 58th ACM/IEEE Design Automation Conf (DAC). Piscataway, NJ: IEEE, 2021: 1315−1318
[164]	Leung C K, Fung D L, Mai D, et al. Explainable data analytics for disease and healthcare informatics[C]//Proc of the 25th Int Database Engineering & Applications Symp. New York: ACM, 2021: 65−74
[165]	León A, García S A, Costa M, et al. Evolution of an adaptive information system for precision medicine[C]//Proc of the 33rd Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2021: 3−10
[166]	Pereira A, Thomas C. Challenges of machine learning applied to safety-critical cyber-physical systems[J]. Machine Learning and Knowledge Extraction, 2020, 2(4): 579−602 doi: 10.3390/make2040031
[167]	Zhang Ru, Xiao Wencong, Zhang Hongyu, et al. An empirical study on program failures of deep learning jobs[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2020: 1159−1170
[168]	du Preez A, Bekker J. A machine learning decision support framework for industrial engineering purposes[C]//Proc of the 1st Int Conf on Industrial Engineering and Industrial Management. New York: ACM, 2020: 9−14
[169]	Leung C K, Fung D L X, Mushtaq S B, et al. Data science for healthcare predictive analytics[C]//Proc of the 24th Symp on Int Database Engineering & Applications. New York: ACM, 2020: 8
[170]	Loucopoulos P, Kavakli E, Chechina N. Requirements engineering for cyber physical production systems[C]//Proc of the 31st Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 276−291
[171]	Bao N, Chung S T. A rule-based smart thermostat[C]//Proc of the 1st Int Conf on Computational Intelligence and Intelligent Systems. New York: ACM, 2018: 20−25
[172]	Ma J, Park S C, Shin J H, et al. AI based intelligent system on the EDISON platform[C]//Proc of the 2018 Artificial Intelligence and Cloud Computing Conf. New York: ACM, 2018: 106−114
[173]	Gupta P, Suryavanshi A, Maheshwari S, et al. Human-machine interface system for pre-diagnosis of diseasesusing machine learning[C]//Proc of the 18th Int Conf on Machine Vision and Applications. New York: ACM, 2018: 71−75
[174]	Sothilingam R, Eric S K. Modeling agents, roles, and positions in machine learning project organizations[C]//Proc of the 15th Int iStar Workshop. Berlin: Springer, 2020: 61−66
[175]	Lim S, Henriksson A, Zdravkovic J. Data-driven requirements elicitation: A systematic literature review[J]. SN Computer Science, 2021, 2(1): 1−35 doi: 10.1007/s42979-020-00382-x
[176]	Kirikova M. Continuous requirements engineering[C/OL]//Proc of the 18th Int Conf on Computer Systems and Technologies. New York: ACM, 2017[2022-05-31].https://dl.acm.org/doi/10.1145/3134302.3134304
[177]	Hartmann T, Moawad A, Fouquet F, et al. The next evolution of MDE: A seamless integration of machine learning into domain modeling[J]. Software & Systems Modeling, 2019, 18(2): 1285−1304
[178]	Arruda D, Laigner R. Requirements engineering practices and challenges in the context of big data software development projects: Early insights from a case study[C]//Proc of the IEEE Int Conf on Big Data (Big Data). Piscataway, NJ: IEEE, 2020: 2012−2019
[179]	Alrajeh D, Van Lamsweerde A, Kramer J, et al. Risk-driven revision of requirements models[C]//Proc of the 38th Int Conf on Software Engineering. New York: ACM, 2016: 855−865
[180]	Ishikawa F, Yoshioka N. How do engineers perceive difficulties in engineering of machine-learning systems?—Questionnaire survey[C]//Proc of the 7th IEEE/ACM Joint Int Workshop on Conducting Empirical Studies in Industry (CESI) and 6th Int Workshop on Software Engineering Research and Industrial Practice (SER&IP). Piscataway, NJ: IEEE, 2019: 2−9
[181]	Muccini A, Vaidhyanathan K. Software architecture for ML-based systems: What exists and what lies ahead[C]// Proc of 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Fngineering for AI (WAIN). Piscataway, NJ: IEEE, 2021:121-128
[182]	Murdock R J, Kauwe S K, Wang A Y T, et al. Is domain knowledge necessary for machine learning materials properties?[J]. Integrating Materials and Manufacturing Innovation, 2020, 9(3): 221−227 doi: 10.1007/s40192-020-00179-z
[183]	Childs C M, Washburn N R. Embedding domain knowledge for machine learning of complex material systems[J]. MRS Communications, 2019, 9(3): 806−820 doi: 10.1557/mrc.2019.90
[184]	Latef M, Aslam T, Sehar P. Impact of domain knowledge in phase of requirement engineering[J]. International Journal of Advanced Research and Development, 2018, 3(6): 54−57
[185]	Fu Wei, Menzies T. Easy over hard: A case study on deep learning[C]//Proc of the 11th Joint Meeting on Foundations of Software Engineering. New York: ACM, 2017: 49−60
[186]	Caramujo J, Rodrigues A, Monfared S, et al. RSL-IL4Privacy: A domain-specific language for the rigorous specification of privacy policies[J]. Requirements Engineering, 2019, 24(1): 1−26 doi: 10.1007/s00766-018-0305-2
[187]	Darimont R, Delor E, Massonet P, et al. GRAIL/KAOS: An environment for goal-driven requirements engineering[C]//Proc of the 19th Int Conf on Software Engineering. New York: ACM, 1997: 612−613
[188]	Doerr J, Kerkow D, Koenig T, et al. Non-functional requirements in industry-three case studies adopting an experience-based NFR method[C]//Proc of the 13th IEEE Int Conf on Requirements Engineering (RE’05). Piscataway, NJ: IEEE, 2005: 373−382
[189]	Mesquita R, Jaqueira A, Lucena M, et al. US2StarTool: Generating i* models from user stories[C]//Proc of the 10th Int iStar Workshop. Berlin: Springer. 2015: 103−108
[190]	Robertson S, Robertson J. Mastering the Requirements Process: Getting Requirements Right[M]. Reading, MA: Addison-Wesley, 2012
[191]	Saeed W, Omlin C. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities[J]. arXiv preprint, arXiv: 2111.06420, 2021
[192]	Nazir R. Studying software architecture design challenges, best practices and main decisions for machine learning systems[D]. Gothenburg, Sweden: Department of Computer Science and Engineering, Chalmers University of Technology, University of Gothenburg, 2021
[193]	Di Stefano J S, Menzies T. Machine learning for software engineering: Case studies in foftware reuse[C]// Proc of the 14th IEEE Int Conf on Tools with Artificial Intelligence (ICTAI 2002). Piscataway, NJ: IEEE, 2002: 246−251
[194]	Crawley F, Tyler B. HAZOP: Guide to Best Practice[M]. Amsterdam: Elsevier, 2015
[195]	Chiozza M L, Ponzetti C. FMEA: A model for reducing medical errors[J]. Clinica Chimica Acta, 2009, 404(1): 75−78 doi: 10.1016/j.cca.2009.03.015
[196]	Luthra P. FMECA: An integrated approach[C]//Proc of the 37th Annual Reliability and Maintainability Symp. Piscataway, NJ: IEEE, 1991: 235−241
[197]	Willey R J. Layer of protection analysis[J]. Procedia Engineering, 2014, 84: 12−22 doi: 10.1016/j.proeng.2014.10.405
[198]	Ericson C A, Ll C. Fault tree analysis[C/OL]//Proc of the 17th System Safety Conf. Saint Paul, USA: The International System Safety Society, 1999[2022-05-31].https://ftaassociates.files.wordpress.com/2018/12/C.-Ericson-Fault-Tree-Analysis-A-History-Proceedings-of-the-17th-International-System-Safety-Conference-1999.pdf
[199]	Andrews J D, Dunnett S J. Event-tree analysis using binary decision diagrams[J]. IEEE Transactions on Reliability, 2000, 49(2): 230−238 doi: 10.1109/24.877343

[1]	Gu Tianlong, Gao Hui, Li Long, Bao Xuguang, Li Yunhui. An Approach for Training Moral Agents via Reinforcement Learning[J]. Journal of Computer Research and Development, 2022, 59(9): 2039-2050. DOI: 10.7544/issn1000-1239.20210474
[2]	Ma Ang, Yu Yanhua, Yang Shengli, Shi Chuan, Li Jie, Cai Xiuxiu. Survey of Knowledge Graph Based on Reinforcement Learning[J]. Journal of Computer Research and Development, 2022, 59(8): 1694-1722. DOI: 10.7544/issn1000-1239.20211264
[3]	Qi Faxin, Tong Xiangrong, Yu Lei. Agent Trust Boost via Reinforcement Learning DQN[J]. Journal of Computer Research and Development, 2020, 57(6): 1227-1238. DOI: 10.7544/issn1000-1239.2020.20190403
[4]	Fan Hao, Xu Guangping, Xue Yanbing, Gao Zan, Zhang Hua. An Energy Consumption Optimization and Evaluation for Hybrid Cache Based on Reinforcement Learning[J]. Journal of Computer Research and Development, 2020, 57(6): 1125-1139. DOI: 10.7544/issn1000-1239.2020.20200010
[5]	Zhang Wentao, Wang Lu, Cheng Yaodong. Performance Optimization of Lustre File System Based on Reinforcement Learning[J]. Journal of Computer Research and Development, 2019, 56(7): 1578-1586. DOI: 10.7544/issn1000-1239.2019.20180797
[6]	Zhang Kaifeng, Yu Yang. Methodologies for Imitation Learning via Inverse Reinforcement Learning: A Review[J]. Journal of Computer Research and Development, 2019, 56(2): 254-261. DOI: 10.7544/issn1000-1239.2019.20170578
[7]	Zhao Fengfei and Qin Zheng. A Multi-Motive Reinforcement Learning Framework[J]. Journal of Computer Research and Development, 2013, 50(2): 240-247.
[8]	Lin Fen, Shi Chuan, Luo Jiewen, Shi Zhongzhi. Dual Reinforcement Learning Based on Bias Learning[J]. Journal of Computer Research and Development, 2008, 45(9): 1455-1462.
[9]	Shi Chuan, Shi Zhongzhi, Wang Maoguang. Online Hierarchical Reinforcement Learning Based on Path-matching[J]. Journal of Computer Research and Development, 2008, 45(9).
[10]	Chen Zonghai, Wen Feng, Nie Jianbin, and Wu Xiaoshu. A Reinforcement Learning Method Based on Node-Growing k-Means Clustering Algorithm[J]. Journal of Computer Research and Development, 2006, 43(4): 661-666.