Ashmore R, Calinescu R, Paterson C. Assuring the machine learning lifecycle: Desiderata, methods, and challenges[J]. ACM Computing Surveys, 2021, 54(5): 1−39

Habibullah K M, Horkoff J. Non-functional requirements for machine learning: Understanding current use and challenges in industry[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 13−23

Mohseni S, Zarei N, Ragan E D. A multidisciplinary survey and framework for design and evaluation of explainable AI systems[J]. ACM Transactions on Interactive Intelligent Systems, 2021, 11(3/4): 1−45

Mehrabi N, Morstatter F, Saxena N, et al. A survey on bias and fairness in machine learning[J]. ACM Computing Surveys, 2021, 54(6): 1−35

Floridi L. Establishing the rules for building trustworthy AI[J]. Nature Machine Intelligence, 2019, 1(6): 261−262 doi: 10.1038/s42256-019-0055-y

Wang Ye, Chen Junwu, Xia Xin, et al. Intelligent requirements elicitation and modeling: A literature review[J]. Journal of Computer Research and Development, 2021, 58(4): 683−705(in Chinese) doi: 10.7544/issn1000-1239.2021.20200740

Iqbal T, Elahidoost P, Lucio L. A bird’s eye view on requirements engineering and machine learning[C]//Proc of the 25th Asia-Pacific Software Engineering Conf (APSEC). Piscataway, NJ: IEEE, 2018: 11−20

Villamizar H, Escovedo T, Kalinowski M. Requirements engineering for machine learning: A systematic mapping study[C]//Proc of the 47th Euromicro Conf on Software Engineering and Advanced Applications (SEAA). Piscataway, NJ: IEEE, 2021: 29−36

Willard J, Jia Xiaowei, Xu Shaoming, et al. Integrating physics-based modeling with machine learning: A survey[J]. arXiv preprint, arXiv: 2003.04919, 2020

Harel D, Marron A, Sifakis J. Autonomics: In search of a foundation for next-generation autonomous systems[J]. Proceedings of the National Academy of Sciences, 2020, 117(30): 17491−17498 doi: 10.1073/pnas.2003162117

Horkoff J, Aydemir F B, Cardoso E, et al. Goal-oriented requirements engineering: An extended systematic mapping study[J]. Requirements Engineering, 2019, 24(2): 133−160 doi: 10.1007/s00766-017-0280-z

Wolny S, Mazak A, Carpella C, et al. Thirteen years of SysML: A systematic mapping study[J]. Software and Systems Modeling, 2020, 19(1): 111−169 doi: 10.1007/s10270-019-00735-y

Azevedo A, Santos M F. KDD, SEMMA and CRISP-DM: A parallel overview[C/OL]//Proc of the IADIS European Conf on Data Mining, 2008[2022-05-30].https://www.iadisportal.org/digital-library/kdd-semma-and-crisp-dm-a-parallel-overview

Vogelsang A, Borg M. Requirements engineering for machine learning: Perspectives from data scientists[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 245−251

Amershi S, Begel A, Bird C, et al. Software engineering for machine learning: A case study[C]//Proc of the 41st IEEE/ACM Int Conf on Software Engineering: Software Engineering in Practice (ICSE-SEIP). Piscataway, NJ: IEEE, 2019: 291−300

Silva G R S, Canedo E D. Requirements engineering challenges and techniques in building chatbots[C]//Proc of the 14th Int Conf on Agents and Artificial Intelligence. Setúbal, Portugal: SCITEPRESS, 2022: 180−187

Bencomo N, Guo J L C, Harrison R, et al. The secret to better AI and better software (is requirements engineering)[J]. IEEE Software, 2021, 39(1): 105−110

Yoshioka N, Husen J H, Tun H T, et al. Landscape of requirements engineering for machine learning-based AI systems[C]//Proc of the 28th Asia-Pacific Software Engineering Conf Workshops (APSEC Workshops). Piscataway, NJ: IEEE, 2021: 5−8

Ahmad K, Bano M, Abdelrazek M, et al. What’s up with requirements engineering for artificial intelligence systems?[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 1−12

Tukur M, Umar S, Hassine J. Requirement engineering challenges: A systematic mapping study on the academic and the industrial perspective[J]. Arabian Journal for Science and Engineering, 2021, 46(4): 3723−3748 doi: 10.1007/s13369-020-05159-1

Heyn H M, Knauss E, Muhammad A P, et al. Requirement engineering challenges for AI-intense systems development[C]//Proc of the 1st IEEE/ACM Workshop on AI Engineering-Software Engineering for AI (WAIN). Piscataway, NJ: IEEE, 2021: 89−96

Ferreira F, Silva L L, Valente M T. Software engineering meets deep learning: A mapping study[C]//Proc of the 36th Annual ACM Symp on Applied Computing. New York: ACM, 2021: 1542−1549

Dalpiaz F, Niu Nan. Requirements engineering in the days of artificial intelligence[J]. IEEE Software, 2020, 37(4): 7−10 doi: 10.1109/MS.2020.2986047

Kostova B, Gurses S, Wegmann A. On the interplay between requirements, engineering, and artificial intelligence[J]. Information and Software Technology, 2023, 158: 107176 doi: 10.1016/j.infsof.2023.107176

Belani H, Vukovic M, Car Ž. Requirements engineering challenges in building AI-based complex systems[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 252−255

Lwakatare L E, Raj A, Bosch J, et al. A taxonomy of software engineering challenges for machine learning systems: An empirical investigation[C]//Proc of the Int Conf on Agile Software Development. Berlin: Springer, 2019: 227−243

Bosch J, Olsson H H, Crnkovic I. It takes three to tango: Requirement, outcome/data, and AI driven development[C]//Proc of the 1st Int Workshop on Software-intensive Business: Start-ups, Ecosystems and Platforms (SiBW). Berlin: Springer, 2018: 177−192

Studer S, Bui T B, Drescher C, et al. Towards CRISP-ML (Q): A machine learning process model with quality assurance methodology[J]. Machine Learning and Knowledge Extraction, 2021, 3(2): 392−413 doi: 10.3390/make3020020

Nalchigar S, Yu E, Obeidi Y, et al. Solution patterns for machine learning[C]//Proc of the 31st Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 627−642

Nalchigar S, Yu E, Keshavjee K. Modeling machine learning requirements from three perspectives: A case report from the healthcare domain[J]. Requirements Engineering, 2021, 26(2): 237−254 doi: 10.1007/s00766-020-00343-z

Chuprina T, Mendez D, Wnuk K. Towards artefact-based requirements engineering for data-centric systems[J]. arXiv preprint, arXiv: 2103.05233, 2021

Barrera J M, Reina Reina A, Maté A, et al. Applying i* in conceptual modelling in machine learning[C]//Proc of the 14th Int iStar Workshop. Berlin: Springer, 2020: 56−62

Camilli M, Felderer M, Giusti A, et al. Towards risk modeling for collaborative AI[C]//Proc of the 1st IEEE/ACM Workshop on AI Engineering-Software Engineering for AI (WAIN). Piscataway, NJ: IEEE, 2021: 51−54

Alrajeh D, Cailliau A, van Lamsweerde A. Adapting requirements models to varying environments[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2020: 50−61

Deshpande A, Sharp H. Responsible AI systems: Who are the stakeholders?[C]//Proc of the 2022 AAAI/ACM Conf on AI, Ethics, and Society. New York: ACM, 2022: 227−236

Khuat T T, Kedziora D J, Gabrys B. The roles and modes of human interactions with automated machine learning systems[J]. arXiv preprint, arXiv: 2205.04139, 2022

Nahar N, Zhou Shurui, Lewis G, et al. Collaboration challenges in building ML-enabled systems: Communication, documentation, engineering, and process[C]//Proc of the 44th Int Conf on Software Engineering. New York: ACM, 2022: 413−425

Odong L A, Perini A, Susi A. Requirements engineering for collaborative artificial intelligence systems: A literature survey[C]//Proc of the 16th Int Conf on Research Challenges in Information Science. Berlin: Springer, 2022: 409−425

Barclay I, Abramson W. Identifying roles, requirements and responsibilitiesin trustworthy AI systems[C]//Proc of the 2021 ACM Int Joint Conf on Pervasive and Ubiquitous Computing and Proc of the 2021 ACM Int Symp on Wearable Computers. New York: ACM, 2021: 264−271

Piorkowski D, Park S, Wang A Y, et al. How AI developers overcome communication challenges in a multidisciplinary team: A case study[J]. Proceedings of the ACM on Human-Computer Interaction, 2021, 5(CSCW1): 131

Camilli M, Felderer M, Giusti A, et al. Risk-driven compliance assurance for collaborative AI systems: A vision paper[C]//Proc of the 27th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2021: 123−130

Zhang A X, Muller M, Wang Dakuo. How do data science workers collaborate? Roles, workflows, and tools[J]. Proceedings of the ACM on Human-Computer Interaction, 2020, 4(CSCW1): 22

Berry D M. Requirements engineering for artificial intelligence: What is a requirements specification for an artificial intelligence?[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 19−25

Al M, Ma Yihong, Alarcon P, et al. RESAM: Requirements elicitation and specification for deep-learning anomaly models with applications to UAV flight controllers[C]//Proc of the 30th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2022: 153−165

Villamizar H, Kalinowski M, Lopes H. Towards perspective-based specification of machine learning-enabled systems[J]. arXiv preprint, arXiv: 2206.09760, 2022

Gillani M, Niaz H A, Ullah A. Integration of software architecture in requirements elicitation for rapid software development[J]. IEEE Access, 2022, 10: 56158−56178 doi: 10.1109/ACCESS.2022.3177659

Gabriel S, Bentler D, Grote E M, et al. Requirements analysis for an intelligent workforce planning system: A socio-technical approach to design AI-based systems[J]. Procedia CIRP, 2022, 109: 431−436 doi: 10.1016/j.procir.2022.05.274

Wang Xi, Miao Weikai. A framework for requirements specification of machine-learning systems[C]//Proc of the 32nd Conf on Software Engineering and Knowledge Engineering (SEKE). Pittsburgh, Pennsylvania: KSI Research Inc, 2022: 7−12

Ahmad K. Human-centric requirements engineering for artificial intelligence software systems[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 468−473

Schuh G, Scholz P, Leich T, et al. Identifying and analyzing data model requirements and technology potentials of machine learning systems in the manufacturing industry of the future[C]//Proc of the 61st Int Scientific Conf on Information Technology and Management Science of Riga Technical University (ITMS). Piscataway, NJ: IEEE, 2020: 20178653

Cirqueira D, Nedbal D, Helfert M, et al. Scenario-based requirements elicitation for user-centric explainable AI[C]//Proc of the 4th Int Cross-Domain Conf for Machine Learning and Knowledge Extraction. Berlin: Springer, 2020: 321−341

d’Aloisio G, Di Marco A, Stilo G. Modeling quality and machine learning pipelines through extended feature models[J]. arXiv preprint, arXiv: 2207.07528, 2022

Levy N, Katz G. RoMA: A method for neural network robustness measurement and assessment[J]. arXiv preprint, arXiv: 2110.11088, 2021

Siebert J, Joeckel L, Heidrich J, et al. Construction of a quality model for machine learning systems[J]. Software Quality Journal, 2022, 30(2): 307−335 doi: 10.1007/s11219-021-09557-y

Haindl P, Hoch T, Dominguez J, et al. Quality characteristics of a software platform for human-AI teaming in smart manufacturing[J]. arXiv preprint, arXiv: 2205.15767, 2022

Madaio M, Egede L, Subramonyam H, et al. Assessing the fairness of AI systems: AI practitioners’ processes, challenges, and needs for support[J]. Proceedings of the ACM on Human-Computer Interaction, 2022, 6(CSCW1): 52

Perera A, Aleti A, Tantithamthavorn C, et al. Search-based fairness testing for regression-based machine learning systems[J]. Empirical Software Engineering, 2022, 27(3): 1−36

Chen Huaming, Babar M A. Security for machine learning-based software systems: A survey of threats, practices and challenges[J]. arXiv preprint, arXiv: 2201.04736, 2022

Georgieva I, Lazo C, Timan T, et al. From AI ethics principles to data science practice: A reflection and a gap analysis based on recent frameworks and practical experience[J]. AI and Ethics, 2022, 2: 697−711 doi: 10.1007/s43681-021-00127-3

Steimers A, Schneider M. Sources of risk of AI systems[J]. International Journal of Environmental Research and Public Health, 2022, 19(6): 3641

Bartels R, Dudink J, Haitjema S, et al. A perspective on a quality management system for AI/ML-based clinical decision support in hospital care[J]. Frontiers in Digital Health, 2022, 4: 942588

d’Aloisio G. Quality-driven machine learning-based data science pipeline realization: A software engineering approach[C]//Proc of the 44th ACM/IEEE Int Conf on Software Engineering: Companion Proceedings. Piscataway, NJ: IEEE, 2022: 291−293

Khan A, Siddiqui I F, Shaikh M, et al. Handling non-fuctional requirements in IoT-based machine learning systems[C]//Proc of the 7th Joint Int Conf on Digital Arts, Media and Technology with ECTI Northern Section Conf on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON). Piscataway, NJ: IEEE, 2022: 477−479

Truong H L, Nguyen T M. QoA4ML—A framework for supporting contracts in machine learning services[C]//Proc of the IEEE Int Conf on Web Services (ICWS). Piscataway, NJ: IEEE, 2021: 465−475

Muñante D, Perini A, Kifetew F M, et al. Combining risk and variability modelling for requirements analysis in SAS engineering[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 396−401

Eliyahu T, Kazak Y, Katz G, et al. Verifying learning-augmented systems[C]//Proc of the 2021 ACM SIGCOMM. New York: ACM, 2021: 305−318

Mauri L, Damiani E. STRIDE-AI: An approach to identifying vulnerabilities of machine learning assets[C]//Proc of the IEEE Int Conf on Cyber Security and Resilience (CSR). Piscataway, NJ: IEEE, 2021: 147−154

Dey S, Lee S W. Multilayered review of safety approaches for machine learning-based systems in the days of AI[J]. Journal of Systems and Software, 2021, 176: 110941

Wang Jingyi, Chen Jialuo, Sun Youcheng, et al. Robot: Robustness-oriented testing for deep learning systems[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 300−311

Baluta T, Chua Z L, Meel K S, et al. Scalable quantitative verification for deep neural networks[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 312−323

Zhang J M, Harman M. "Ignorance and Prejudice" in software fairness[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1436−1447

Hobbs R. Integrating ethically align design into agile and CRISP-DM[C]//Proc of the Southeast Conf 2021. Piscataway, NJ: IEEE, 2021: 20631991

Tariq S, Cheema S M. Approaches for non-functional requirement modeling: A literature survey[C]//Proc of the 4th Int Conf on Computing & Information Sciences (ICCIS). Piscataway, NJ: IEEE, 2021: 21573902

Cerqueira J, Acco H, Dias E. Ethical guidelines and principles in the context of artificial intelligence[C]//Proc of the 17th Brazilian Symp on Information Systems. New York: ACM, 2021: 36

Byun T, Rayadurgam S. Manifold for machine learning assurance[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). Piscataway, NJ: IEEE, 2020: 97−100

Ishikawa F, Matsuno Y. Evidence-driven requirements engineering for uncertainty of machine learning-based systems[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 346−351

Damyanova B. Quality attributes in AI-ML-based systems: Differences and challenges[D]. Stuttgart: University of Stuttgart, 2020

Balasubramaniam N, Kauppinen M, Kujala S, et al. Ethical guidelines for solving ethical issues and developing AI systems[C]//Proc of the Int Conf on Product-Focused Software Process Improvement. Berlin: Springer, 2020: 331−346

Riccio V, Tonella P. Model-based exploration of the frontier of behaviours for deep learning system testing[C]//Proc of the 28th ACM Joint Meeting on European Software Engineering Conf and Symp on the Foundations of Software Engineering. New York: ACM, 2020: 876−888

Wilhjelm C, Younis A A. A threat analysis methodology for security requirements elicitation in machine learning based systems[C]//Proc of the 20th IEEE Int Conf on Software Quality, Reliability and Security Companion (QRS-C). Piscataway, NJ: IEEE, 2020: 426−433

Nakamichi K, Ohashi K, Namba I, et al. Requirements-driven method to determine quality characteristics and measurements for machine learning software and its evaluation[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 260−270

Humbatova N, Jahangirova G, Bavota G, et al. Taxonomy of real faults in deep learning systems[C]//Proc of the 42nd ACM/IEEE Int Conf on Software Engineering. Piscataway, NJ: IEEE, 2020: 1110−1121

Chechik M. Uncertain requirements, assurance and machine learning[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 2−3

Horkoff J. Non-functional requirements for machine learning: Challenges and new directions[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 386−391

Kuwajima H, Ishikawa F. Adapting square for quality assessment of artificial intelligence systems[C]//Proc of the IEEE Int Symp on Software Reliability Engineering Workshops (ISSREW). Piscataway, NJ: IEEE, 2019: 13−18

Bhatia J, Breaux T D. Semantic incompleteness in privacy policy goals[C]//Proc of the 26th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2018: 159−169

Zhang Boyu, Magaña J C, Davoodi A. Analysis of security of split manufacturing using machine learning[C]//Proc of the 55th Annual Design Automation Conf. New York: ACM, 2018: 2767−2780

Hayrapetian A, Raje R. Empirically analyzing and evaluating security features in software requirements[C]//Proc of the 11th Innovations in Software Engineering Conf. New York: ACM, 2018: 9

Hoel T, Griffiths D, Chen Weiqin. The influence of data protection and privacy frameworks on the design of learning analytics systems[C]//Proc of the 7th Int Learning Analytics & Knowledge Conf. New York: ACM, 2017: 243−252

Burwinkel H, Matz H, Saur S, et al. Physics-aware learning and domain-specific loss design in ophthalmology[J]. Medical Image Analysis, 2022, 76: 102314

Von Rueden L, Mayer S, Beckh K, et al. Informed machine learning—A taxonomy and survey of integrating knowledge into learning systems[J]. arXiv preprint, arXiv: 1903.12394, 2019

Vo K, Nguyen T, Pham D, et al. Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media[J]. International Journal of Computational Vision and Robotics, 2019, 9(5): 458−485 doi: 10.1504/IJCVR.2019.102286

Abbasi A, Nataraj C. Physics-informed machine learning for uncertainty reduction in time response reconstruction of a dynamic system[J]. IEEE Internet Computing, 2022, 26(4): 35−44 doi: 10.1109/MIC.2022.3170736

Huang Bin, Wang Jianhui. Applications of physics-informed neural networks in power systems-a review[J]. IEEE Transactions on Power Systems, 2022, 38(1): 572−588

Zhao W L, Gentine P, Reichstein M, et al. Physics-constrained machine learning of evapotranspiration[J]. Geophysical Research Letters, 2019, 46(24): 14496−14507 doi: 10.1029/2019GL085291

Lin Jing, Zhang Yu, Khoo E. Hybrid physics-based and data-driven modeling with calibrated uncertainty for lithium-ion battery degradation diagnosis and prognosis[J]. arXiv preprint, arXiv: 2110.13661, 2021

Cornelio J, Mohd Razak S, Cho Y, et al. Residual learning to integrate neural network and physics-based models for improved production prediction in unconventional reservoirs[J]. SPE Journal, 2022, 27(6): 1−22

Soleimani M, Intezari A, Pauleen D J. Mitigating cognitive biases in developing AI-assisted recruitment systems: A knowledge-sharing approach[J]. International Journal of Knowledge Management, 2022, 18(1): 1−18

Willard J, Jia Xiaowei, Xu Shaoming, et al. Integrating scientific knowledge with machine learning for engineering and environmental systems[J]. ACM Computing Surveys, 2022, 55(4): 1−37

Luo Xing, Zhang Dongxiao, Zhu Xu. Deep learning based forecasting of photovoltaic power generation by incorporating domain knowledge[J]. Energy, 2021, 225: 120240

Park S, Wang A Y, Kawas B, et al. Facilitating knowledge sharing from domain experts to data scientists for building NLP models[C]//Proc of the 26th Int Conf on Intelligent User Interfaces. New York: ACM, 2021: 585−596

Xie Xiaozheng, Niu Jianwei, Liu Xuefeng, et al. A survey on incorporating domain knowledge into deep learning for medical image analysis[J]. Medical Image Analysis, 2021, 69: 101985

Van Oort C M. Leveraging domain knowledge in deep learning systems[D]. Burlington: The University of Vermont and State Agricultural College, 2021

Chitchyan R, Bird C. Theory as a source of software requirements[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 227−237

Vázquez-Ingelmo A, García Peñalvo F J, Therón R. Advances in the use of domain engineering to support feature identification and generation of information visualizations[C]//Proc of the 8th Int Conf on Technological Ecosystems for Enhancing Multiculturality. New York: ACM, 2020: 1053−1056

Chai Yidong, Liu Hongyan, Xu Jie. Glaucoma diagnosis based on both hidden features and domain knowledge through deep learning models[J]. Knowledge-Based Systems, 2018, 161: 147−156 doi: 10.1016/j.knosys.2018.07.043

Song Yangqiu, Roth D. Machine learning with world knowledge: The position and survey[J]. arXiv preprint, arXiv: 1705.02908, 2017

Haidry S Z, Falkner K, Szabo C. Identifying domain-specific cognitive strategies for software engineering[C]//Proc of the 2017 ACM Conf on Innovation and Technology in Computer Science Education. New York: ACM, 2017: 206−211

Laato S, Mäntymäki M, Minkkinen M, et al. Integrating machine learning with software development lifecycles: Insights from experts[C/OL]//Proc of the 30th European Conf on Information Systems. Association for Information Systems, 2022[2022-05-30].https://aisel.aisnet.org/ecis2022_rp/118

Washizaki H, Uchida H, Khomh F, et al. Machine learning architecture and design patterns[J/OL]. IEEE Software, 2020[2022-04-01]. http://g7.washi.cs.waseda.ac.jp/wp-content/uploads/2019/12/IEEE_Software_19__ML_Patterns.pdf

Lewis G A, Ozkaya I, Xu Xiwei. Software architecture challenges for ML systems[C]//Proc of the 37th IEEE Int Conf on Software Maintenance and Evolution (ICSME). Piscataway, NJ: IEEE, 2021: 634−638

Serban A, Visser J. An empirical study of software architecture for machine learning[J]. arXiv preprint, arXiv: 2105.12422, 2021

Chen Zhenpeng, Yao Huihan, Lou Yiling, et al. An empirical study on deployment faults of deep learning based mobile applications[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 674-685

Serban A, van der Blom K, Hoos H, et al. Adoption and effects of software engineering best practices in machine learning[C]//Proc of the 14th ACM/IEEE Int Symp on Empirical Software Engineering and Measurement (ESEM). New York: ACM, 2020: 3

John M M, Olsson H H, Bosch J. Developing ML/DL models: A design framework[C/OL]//Proc of the 14th Int Conf on Software and System Processes. New York: ACM, 2020[2022-05-30].https://www.diva-portal.org/smash/get/diva2:1553907/FULLTEXT01.pdf

Correia J L, Pereira J A, Mello R, et al. Brazilian data scientists: Revealing their challenges and practices on machine learning model development[C]//Proc of the 19th Brazilian Symp on Software Quality. New York: ACM, 2020: 10

Reimann L, Kniesel-Wünsche G. Achieving guidance in applied machine learning through software engineering techniques[C]//Proc of the 4th Int Conf on Art, Science, and Engineering of Programming. New York: ACM, 2020: 7−12

Kourouklidis P, Kolovos D, Matragkas N, et al. Towards a low-code solution for monitoring machine learning model performance[C]//Proc of the 23rd ACM/IEEE Int Conf on Model Driven Engineering Languages and Systems: Companion Conf. Piscataway, NJ: IEEE, 2020: 62

Prado F F, Digiampietri L A. A systematic review of automated feature engineering solutions in machine learning problems[C]//Proc of the 16th Brazilian Symp on Information Systems. New York: ACM, 2020: 12

Oppold S, Herschel M. A system framework for personalized and transparent data-driven decisions[C]//Proc of the 32nd Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2020: 153−168

Ole M, Volker G. Towards concept based software engineering for intelligent agents[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 42−48

Hesenius M, Schwenzfeier N, Meyer O, et al. Towards a software engineering process for developing data-driven applications[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 35−41

Mucha T M, Ma Sijia, Abhari K. Sustainability of machine learning-based solutions: A lifecycle perspective[C/OL]//Proc of the Pacific Asia Conf on Information Systems. Association for Information Systems, 2022[2022-05-30].https://aisel.aisnet.org/pacis2022/262/

Yu Kui, Guo Xianjie, Liu Lin, et al. Causality-based feature selection: Methods and evaluations[J]. ACM Computing Surveys, 2020, 53(5): 1−36

Cabrera Á A, Ribeiro M T, Lee B, et al. What did my AI learn? How data scientists make sense of model behavior[J]. ACM Transactions on Computer-Human Interaction, 2022, 30(1): 1−27

Sun Jiao, Liao Q V, Muller M, et al. Investigating explainability of generative AI for code through scenario-based design[C]//Proc of the 27th Int Conf on Intelligent User Interfaces. New York: ACM, 2022: 212−228

Shen M W. Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient[J]. arXiv preprint, arXiv: 2202.05302, 2022

Balasubramaniam N, Kauppinen M, Hiekkanen K, et al. Transparency and explainability of AI systems: Ethical guidelines in practice[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 3−18

Piorkowski D, Richards J, Hind M. Evaluating a methodology for increasing AI transparency: A case study[J]. arXiv preprint, arXiv: 2201.13224, 2022

Sadeghi M, Klös V, Vogelsang A. Cases for explainable software systems: Characteristics and examples[C]//Proc of the 29th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2021: 181−187

Langer M, Baum K, Hartmann K, et al. Explainability auditing for intelligent systems: A rationale for multi-disciplinary perspectives[C]//Proc of the 29th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2021: 164−168

Chazette L, Brunotte W, Speith T. Exploring explainability: A definition, a model, and a knowledge catalogue[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 197−208

Velez M, Jamshidi P, Siegmund N, et al. White-box analysis over machine learning: Modeling performance of configurable systems[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1072−1084

Narteni S, Ferretti M, Orani V, et al. From explainable to reliable artificial intelligence[C]//Proc of the 5th Int Cross-Domain Conf for Machine Learning and Knowledge Extraction. Berlin: Springer, 2021: 255−273

Zohdinasab T, Riccio V, Gambi A, et al. Deephyperion: Exploring the feature space of deep learning-based systems through illumination search[C]//Proc of the 30th ACM SIGSOFT Int Symp on Software Testing and Analysis. New York: ACM, 2021: 79−90

Liao Q V, Gruen D, Miller S. Questioning the AI: Informing design practices for explainable AI user experiences[C]//Proc of the 2020 CHI Conf on Human Factors in Computing Systems. New York: ACM, 2020: 1−15

Köhl M A, Baum K, Langer M, et al. Explainability as a non-functional requirement[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 363−368

Yang Zijiang, Al-Bahrani R, Reid A C E, et al. Deep learning based domain knowledge integration for small datasets: Illustrative applications in materials informatics[C]//Proc of the Int Joint Conf on Neural Networks (IJCNN). Piscataway, NJ: IEEE, 2019: 19028509

Wolf C T. Explainability scenarios: Towards scenario-based XAI design[C]//Proc of the 24th Int Conf on Intelligent User Interfaces. New York: ACM, 2019: 252−257

Eiband M, Schneider H, Bilandzic M, et al. Bringing transparency design into practice[C]//Proc of the 23rd Int Conf on Intelligent User Interfaces. New York: ACM, 2018: 211−223

Thakkar D, Ismail A, Kumar P, et al. When is machine learning data good?: Valuing in public health datafication[C]//Proc of the CHI Conf on Human Factors in Computing Systems. New York: ACM, 2022: 322

Jung J Y, Steinberger T, King J L, et al. How domain experts work with data: Situating data science in the practices and settings of craftwork[J]. Proceedings of the ACM on Human-Computer Interaction, 2022, 6(CSCW1): 58

Lwakatare L E, Rånge E, Crnkovic I, et al. On the experiences of adopting automated data validation in an industrial machine learning project[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering: Software Engineering in Practice (ICSE-SEIP). Piscataway, NJ: IEEE, 2021: 248−257

Hutchinson B, Smart A, Hanna A, et al. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure[C]//Proc of the 2021 ACM Conf on Fairness, Accountability, and Transparency. New York: ACM, 2021: 560-575

Shao Zhijun, Wu Ji, Zhao Wenxiao, et al. How data plays in the requirements of face recognition system: A concern driven systematic literature review[C]//Proc of the 28th Asia-Pacific Software Engineering Conf Workshops (APSEC Workshops). Piscataway, NJ: IEEE, 2021: 9−12

Astegher M, Busetta P, Perini A, et al. Specifying requirements for data collection and analysis in data-driven RE. A research preview[C]//Proc of the 27th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2021: 182−188

Hayes J H, Payne J, Leppelmeier M. Toward improved artificial intelligence in requirements engineering: Metadata for tracing datasets[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 256−262

Biffl S, Lüder A, Rinker F, et al. Efficient engineering data exchange in multi-disciplinary systems engineering[C]//Proc of the 29th Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 17−31

Mei Songzhu, Liu Cong, Wang Qinglin, et al. Model provenance management in MLOps pipeline[C]//Proc of the 8th Int Conf on Computing and Data Engineering. New York: ACM, 2022: 45−50

Kaminwar S R, Goschenhofer J, Thomas J, et al. Structured verification of machine learning models in industrial settings[J/OL]. Big Data, 2021[2022-04-01].https://www.liebertpub.com/doi/10.1089/big.2021.0112

Xiao Yan, Beschastnikh I, Rosenblum D S, et al. Self-checking deep neural networks in deployment[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 372−384

Wang Song, Shrestha N, Subburaman A K, et al. Automatic unit test generation for machine learning libraries: How far are we?[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1548−1560

Hestness J, Ardalani N, Diamos G. Beyond human-level accuracy: Computational challenges in deep learning[C/OL]//Proc of the 24th Symp on Principles and Practice of Parallel Programming. New York: ACM, 2019[2022-05-30]. https://dl.acm.org/doi/10.1145/3293883.3295710

Gharibi G, Walunj V, Rella S, et al. Modelkb: Towards automated management of the modeling lifecycle in deep learning[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 28−34

Borg M, Englund C, Wnuk K, et al. Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry[J]. arXiv preprint, arXiv: 1812.05389, 2018

Czvetkó T, Kummer A, Ruppert T, et al. Data-driven business process management-based development of Industry 4.0 solutions[J]. CIRP Journal of Manufacturing Science and Technology, 2022, 36: 117−132 doi: 10.1016/j.cirpj.2021.12.002

Merkelbach S, Von Enzberg S, Kühn A, et al. Towards a process model to enable domain experts to become citizen data scientists for industrial applications[C]//Proc of the 5th IEEE Int Conf on Industrial Cyber-Physical Systems (ICPS). Piscataway, NJ: IEEE, 2022

Heyn H M, Subbiah P, Linder J, et al. Setting AI in context: A case study on defining the context and operational design domain for automated driving[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 199−215

Zhang Ran, Albrecht A, Kausch J, et al. DDE process: A requirements engineering approach for machine learning in automated driving[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 269−279

Borg M, Bronson J, Christensson L, et al. Exploring the assessment list for trustworthy AI in the context of advanced driver-assistance systems[C]//Proc of the 2nd IEEE/ACM Int Workshop on Ethics in Software Engineering Research and Practice (SEthics). Piscataway, NJ: IEEE, 2021: 5−12

Martínez-Fernández S, Franch X, Jedlitschka A, et al. Developing and operating artificial intelligence models in trustworthy autonomous systems[C]//Proc of the 15th Int Conf on Research Challenges in Information Science. Berlin: Springer, 2021: 221−229

Kolyshkina I, Simoff S. Interpretability of machine learning solutions in public healthcare: The CRISP-ML approach[J]. Frontiers in Big Data, 2021, 4: 660206

Pinto A. Requirement specification, analysis and verification for autonomous systems[C]//Proc of the 58th ACM/IEEE Design Automation Conf (DAC). Piscataway, NJ: IEEE, 2021: 1315−1318

Leung C K, Fung D L, Mai D, et al. Explainable data analytics for disease and healthcare informatics[C]//Proc of the 25th Int Database Engineering & Applications Symp. New York: ACM, 2021: 65−74

León A, García S A, Costa M, et al. Evolution of an adaptive information system for precision medicine[C]//Proc of the 33rd Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2021: 3−10

Pereira A, Thomas C. Challenges of machine learning applied to safety-critical cyber-physical systems[J]. Machine Learning and Knowledge Extraction, 2020, 2(4): 579−602 doi: 10.3390/make2040031

Zhang Ru, Xiao Wencong, Zhang Hongyu, et al. An empirical study on program failures of deep learning jobs[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2020: 1159−1170

du Preez A, Bekker J. A machine learning decision support framework for industrial engineering purposes[C]//Proc of the 1st Int Conf on Industrial Engineering and Industrial Management. New York: ACM, 2020: 9−14

Leung C K, Fung D L X, Mushtaq S B, et al. Data science for healthcare predictive analytics[C]//Proc of the 24th Symp on Int Database Engineering & Applications. New York: ACM, 2020: 8

Loucopoulos P, Kavakli E, Chechina N. Requirements engineering for cyber physical production systems[C]//Proc of the 31st Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 276−291

Bao N, Chung S T. A rule-based smart thermostat[C]//Proc of the 1st Int Conf on Computational Intelligence and Intelligent Systems. New York: ACM, 2018: 20−25

Ma J, Park S C, Shin J H, et al. AI based intelligent system on the EDISON platform[C]//Proc of the 2018 Artificial Intelligence and Cloud Computing Conf. New York: ACM, 2018: 106−114

Gupta P, Suryavanshi A, Maheshwari S, et al. Human-machine interface system for pre-diagnosis of diseasesusing machine learning[C]//Proc of the 18th Int Conf on Machine Vision and Applications. New York: ACM, 2018: 71−75

Sothilingam R, Eric S K. Modeling agents, roles, and positions in machine learning project organizations[C]//Proc of the 15th Int iStar Workshop. Berlin: Springer, 2020: 61−66

Lim S, Henriksson A, Zdravkovic J. Data-driven requirements elicitation: A systematic literature review[J]. SN Computer Science, 2021, 2(1): 1−35 doi: 10.1007/s42979-020-00382-x

Kirikova M. Continuous requirements engineering[C/OL]//Proc of the 18th Int Conf on Computer Systems and Technologies. New York: ACM, 2017[2022-05-31].https://dl.acm.org/doi/10.1145/3134302.3134304

Hartmann T, Moawad A, Fouquet F, et al. The next evolution of MDE: A seamless integration of machine learning into domain modeling[J]. Software & Systems Modeling, 2019, 18(2): 1285−1304

Arruda D, Laigner R. Requirements engineering practices and challenges in the context of big data software development projects: Early insights from a case study[C]//Proc of the IEEE Int Conf on Big Data (Big Data). Piscataway, NJ: IEEE, 2020: 2012−2019

Alrajeh D, Van Lamsweerde A, Kramer J, et al. Risk-driven revision of requirements models[C]//Proc of the 38th Int Conf on Software Engineering. New York: ACM, 2016: 855−865

Ishikawa F, Yoshioka N. How do engineers perceive difficulties in engineering of machine-learning systems?—Questionnaire survey[C]//Proc of the 7th IEEE/ACM Joint Int Workshop on Conducting Empirical Studies in Industry (CESI) and 6th Int Workshop on Software Engineering Research and Industrial Practice (SER&IP). Piscataway, NJ: IEEE, 2019: 2−9

Muccini A, Vaidhyanathan K. Software architecture for ML-based systems: What exists and what lies ahead[C]// Proc of 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Fngineering for AI (WAIN). Piscataway, NJ: IEEE, 2021:121-128

Murdock R J, Kauwe S K, Wang A Y T, et al. Is domain knowledge necessary for machine learning materials properties?[J]. Integrating Materials and Manufacturing Innovation, 2020, 9(3): 221−227 doi: 10.1007/s40192-020-00179-z

Childs C M, Washburn N R. Embedding domain knowledge for machine learning of complex material systems[J]. MRS Communications, 2019, 9(3): 806−820 doi: 10.1557/mrc.2019.90

Latef M, Aslam T, Sehar P. Impact of domain knowledge in phase of requirement engineering[J]. International Journal of Advanced Research and Development, 2018, 3(6): 54−57

Fu Wei, Menzies T. Easy over hard: A case study on deep learning[C]//Proc of the 11th Joint Meeting on Foundations of Software Engineering. New York: ACM, 2017: 49−60

Caramujo J, Rodrigues A, Monfared S, et al. RSL-IL4Privacy: A domain-specific language for the rigorous specification of privacy policies[J]. Requirements Engineering, 2019, 24(1): 1−26 doi: 10.1007/s00766-018-0305-2

Darimont R, Delor E, Massonet P, et al. GRAIL/KAOS: An environment for goal-driven requirements engineering[C]//Proc of the 19th Int Conf on Software Engineering. New York: ACM, 1997: 612−613

Doerr J, Kerkow D, Koenig T, et al. Non-functional requirements in industry-three case studies adopting an experience-based NFR method[C]//Proc of the 13th IEEE Int Conf on Requirements Engineering (RE’05). Piscataway, NJ: IEEE, 2005: 373−382

Mesquita R, Jaqueira A, Lucena M, et al. US2StarTool: Generating i* models from user stories[C]//Proc of the 10th Int iStar Workshop. Berlin: Springer. 2015: 103−108

Robertson S, Robertson J. Mastering the Requirements Process: Getting Requirements Right[M]. Reading, MA: Addison-Wesley, 2012

Saeed W, Omlin C. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities[J]. arXiv preprint, arXiv: 2111.06420, 2021

Nazir R. Studying software architecture design challenges, best practices and main decisions for machine learning systems[D]. Gothenburg, Sweden: Department of Computer Science and Engineering, Chalmers University of Technology, University of Gothenburg, 2021

Di Stefano J S, Menzies T. Machine learning for software engineering: Case studies in foftware reuse[C]// Proc of the 14th IEEE Int Conf on Tools with Artificial Intelligence (ICTAI 2002). Piscataway, NJ: IEEE, 2002: 246−251

Crawley F, Tyler B. HAZOP: Guide to Best Practice[M]. Amsterdam: Elsevier, 2015

Chiozza M L, Ponzetti C. FMEA: A model for reducing medical errors[J]. Clinica Chimica Acta, 2009, 404(1): 75−78 doi: 10.1016/j.cca.2009.03.015

Luthra P. FMECA: An integrated approach[C]//Proc of the 37th Annual Reliability and Maintainability Symp. Piscataway, NJ: IEEE, 1991: 235−241

Willey R J. Layer of protection analysis[J]. Procedia Engineering, 2014, 84: 12−22 doi: 10.1016/j.proeng.2014.10.405

Ericson C A, Ll C. Fault tree analysis[C/OL]//Proc of the 17th System Safety Conf. Saint Paul, USA: The International System Safety Society, 1999[2022-05-31].https://ftaassociates.files.wordpress.com/2018/12/C.-Ericson-Fault-Tree-Analysis-A-History-Proceedings-of-the-17th-International-System-Safety-Conference-1999.pdf

Andrews J D, Dunnett S J. Event-tree analysis using binary decision diagrams[J]. IEEE Transactions on Reliability, 2000, 49(2): 230−238 doi: 10.1109/24.877343