面向机器学习应用的可解释性需求分析框架

裴忠一; 刘璘; 王晨; 王建民

doi:10.7544/issn1000-1239.202220794

面向机器学习应用的可解释性需求分析框架

1.
大数据系统软件国家工程研究中心（清华大学）　北京　100084
2.
清华大学软件学院　北京　100084

基金项目: 国家重点研发计划项目（2021YFB1715200）；国家自然科学基金委创新群体项目（62021002）

详细信息

作者简介:
裴忠一: 1987年生. 博士，助理研究员. CCF会员. 主要研究方向为机器学习软件工程、工业智能化

刘璘: 1973年生. 博士，研究员. 主要研究方向为需求工程、领域数据与知识工程

王晨: 1981年生. 副研究员. 主要研究方向为数据库、工业大数据

王建民: 1968年生. 博士，教授. 主要研究方向为数据库、工业大数据

通讯作者:
刘璘（linliu@tsinghua.edu.cn）

中图分类号: TP311
计量
- 文章访问数: 347
- HTML全文浏览量: 23
- PDF下载量: 113
出版历程
- 收稿日期: 2022-09-05
- 修回日期: 2023-08-14
- 网络出版日期: 2024-01-24
- 刊出日期: 2024-04-05

An Explainability-Centric Requirements Analysis Framework for Machine Learning Applications

1.
National Engineering Research Center for Big Data Software (Tsinghua University), Beijing 100084
2.
School of Software, Tsinghua University, Beijing 100084

Funds: This work was supported by the National Key Research and Development Program (2021YFB1715200) and the National Natural Science Foundation of Innovation Group Project (62021002).

More Information

Author Bio:
Pei Zhongyi: born in 1987. PhD, assistant professor. Member of CCF. His main research interests include software engineering for machine learning, industrial intelligence

Liu Lin: born in 1973. PhD, professor. Her main research interests include requirements engineering, domain data and knowledge engineering

Wang Chen: born in 1981. Associate professor. His main research interests include database and industrial big data

Wang Jianmin: born in 1968. PhD, professor. His main research interests include database, and industrial big data

摘要

摘要:
基于大数据机器学习的智能软件研发过程需要综合运用软件工程、数据与领域知识工程、机器学习等多方面的知识和工具，涉及的研究主题和人员角色众多，技术实现手段复杂、研发难度大. 面向智能软件的需求工程需要面对领域知识、业务知识、数据科学交织带来的挑战. 然而，如何将领域知识和端到端的机器学习技术恰当地融合到给定的业务流程之中，以及如何应对工业、医疗等高可信要求场景中的可解释性需求，仍是亟待探索的重要研究问题. 调研了近年来面向机器学习应用的需求工程研究文献，对该领域的发展现状、核心问题和代表性方法进行综述. 提出了面向机器学习应用的可解释性需求分析框架. 基于该框架，通过一个工业智能应用案例分析了未来待研究的重要问题，展望了可行的研究路径.
- 需求工程 /
- 机器学习 /
- 领域模型 /
- 工业工程 /
- 可解释性
Abstract:
Data-driven intelligent software based on machine learning technology is an important means to realize industrial digital transformation. The research and development processes of data-driven intelligent software require the combined use of software requirements engineering, data and domain knowledge engineering, machine learning and so on. This process involves many subjects and roles, making it extremely challenging to clearly explain why and how the domain knowledge, business logic and data semantics relate to each other. Hence, a systematic requirements engineering approach is needed to explicitly address the explainability requirements issues of data-driven intelligence applications. It is still a fast-evolving research field which requires the proper embedding of various domain models and end-to-end machine learning technology fused into a given business processes. A key research question is how to deal with explainability as a core requirement for safety-critical scenarios in industrial, medical and other applications. We provide a research overview on requirements engineering for machine learning applications, in relation to explainability. First, the research status quo, research foci and representative research progress are reviewed. Then, an explainability-centric requirements analysis framework for machine learning applications is proposed, and some open important issues are put forward. Finally, based on the proposed framework, a case study of industrial intelligence application is discussed to illustrate the proposed requirements analysis methodological framework.
- requirements engineering /
- machine learning /
- domain model /
- industrial engineering /
- explainability

HTML全文

语音增强是语音信号处理的重要组成部分，它旨在最大限度地去除背景噪声，提高语音信号质量和可懂度. 在过去几十年中，传统的语音增强技术，例如，维纳滤波法^[1]、谱减法^[2]、基于子空间的方法^[3]等受到了研究者们的青睐，但是这些技术在处理复杂环境下的语音信号时其效果往往不尽人意.

近年来，基于深度神经网络（deep neural network, DNN）的语音增强方法已被证实可获得比传统语音增强方法更高的性能^[4-6]. 具体而言，该类方法的实质在于通过优化途径获得去噪后语音的短时傅里叶变换（short-time Fourier transform, STFT）幅度谱，并将之与原带噪语音的相位谱结合构造出完整的时频谱，再进行逆短时傅里叶变换（inverse short-time Fourier transform, ISTFT），即可生成增强后的语音信号. 其中幅度谱优化可分为2类途径：一类是间接方法，其致力于估计对带噪语音的STFT幅度谱进行掩蔽操作的时频模板，如理想二值模板（ideal binary mask，IBM）^[7]、理想比值模板（ideal ratio mask，IRM）^[8]等；另一类是直接映射方法，借助网络优化直接求取去噪后的语音STFT幅度谱. 研究表明，间接方法比直接映射方法可在语音增强上获得更好的性能^[9]，故本文采用间接方法中的IRM作为训练目标.

近年来，卷积神经网络（convolutional neural network, CNN）被广泛应用于语音增强中^[10-12]. 文献[10]提出了第1个全卷积语音增强网络，该网络证明了CNN可在消耗比DNN更少参数的情况下，获得比DNN更优越的性能.

但需要指出，DNN和CNN存在共同的缺陷，即无法捕获语音信号的相邻连续时间帧之间的长依赖关系，这使其性能受到限制. 为解决这一问题，研究者们在这些方法中融入了循环神经网络（recurrent neural network, RNN）、长短时记忆（long short-term memory, LSTM）网络，并取得了相应的性能提升. 例如，文献[13]中提出基于RNN的深度循环神经网络（deep recurrent neural network, DRNN），实验结果表明DRNN的性能优于DNN. 另外，文献[14]通过在编码器和解码器之间插入了双向长短时记忆（bidirectional long short-term memory, Bi-LSTM）网络，证明了获取相邻连续时间帧之间的长依赖关系可提升语音增强的性能，但代价是消耗更多网络参数.

为在不增加网络参数量的同时又可以有效捕获输入序列的长依赖关系，研究者们将时域卷积网络（temporal convolutional network, TCN）引入到语音增强中^[15-18]. 文献[19]指出：由于TCN由扩张卷积构成，拥有更大的感受野，从而可在不额外增加参数量的同时，获得比LSTM更长的长期有效记忆能力. 然而，以上基于TCN的工作的缺陷在于，随着网络层数的增加，梯度消失问题变得突出，使得网络收敛速度变缓.

为解决梯度消失问题和进一步提升语音增强质量，本文将扩张卷积和密集连接网络（densely connected convolutional network, DenseNet）^[20]相结合，提出频率-时间扩张密集网络（frequency-time dilated dense network, FTDDN）. 其特色在于：

1）在学习上下文信息方面，除了时间方向，扩张卷积同时被应用在频率方向. 通过所构造的时间扩张卷积单元（time dilated convolution unit, TDCU）和频率扩张卷积单元（frequency dilated convolution unit, FDCU），本文所提网络在时频域内均可获得较大的感受野，从而能有效提取出深层语音特征，达到提升语音增强性能的目的.

2）在网络效率方面，本文中各级TDCU和FDCU所提取的特征以密集连接的方式传递，不仅可缓解梯度消失问题，而且可避免经典信息论所指出的因级联信息处理模块数目增加而导致的信息丢失问题^[21].

1. 语音增强问题描述

假设含噪离散语音 ${{\boldsymbol{x}}}(k)$ 表示为

${{\boldsymbol{x}}}(k) = {{\boldsymbol{s}}}(k) + {{\boldsymbol{n}}}(k) ,$

(1)

其中 $k$ 表示时间索引， ${{\boldsymbol{s}}}(k)$ 和 ${{\boldsymbol{n}}}(k)$ 分别表示干净语音和加性噪声. 为实现语音增强从含噪语音 ${{\boldsymbol{x}}}(k)$ 中恢复出干净的语音估计 $\hat {{\boldsymbol{s}}}(k)$ 的目的，需将 ${{\boldsymbol{x}}}(k)$ 进行STFT，得到时频表示:

${{\boldsymbol{X}}}(t,f) = {{\boldsymbol{S}}}(t,f) + {{\boldsymbol{N}}}(t,f) ,$

(2)

其中

$\begin{gathered} {{\boldsymbol{X}}}(t,f) = \left| {{{\boldsymbol{X}}}(t,f)} \right|{{\rm{e}}^{j{{{\boldsymbol{\varPhi}} }_{{\boldsymbol{X}}}}\left( {t,f} \right)}}, \\ {{\boldsymbol{S}}}(t,f) = \left| {{{\boldsymbol{S}}}(t,f)} \right|{{\rm{e}}^{j{{{\boldsymbol{\varPhi}} }_{{\boldsymbol{S}}}}\left( {t,f} \right)}}, \\ {{\boldsymbol{N}}}(t,f) = \left| {{{\boldsymbol{N}}}(t,f)} \right|{{\rm{e}}^{j{{{\boldsymbol{\varPhi}} }_{{\boldsymbol{N}}}}\left( {t,f} \right)}}, \\ \end{gathered}$

(3)

其中 $t \in \left[ {0,T - 1} \right],f \in \left[ {0,F - 1} \right]$ ， $T$ 和 $F$ 分别是时间帧和观测频率的数量（为简化起见，后文将省略以上各时频表示的自变量 $t$ 和 $f$ ）. 随后将此时频表示的幅度谱 $\left| {{\boldsymbol{X}}} \right|$ 作为语音特征输入到神经网络. 经过神经网络的优化，得到时频掩蔽 ${{\boldsymbol{M}}}$ ，并将此掩蔽 ${{\boldsymbol{M}}}$ 与 $\left| {{\boldsymbol{X}}} \right|$ 相乘，得到增强后的语音幅度谱 $\left| {\hat {{\boldsymbol{S}}}} \right|$ ，最后通过对 $\left| {\hat {{\boldsymbol{S}}}} \right|$ 和含噪语音的相位谱 ${{{\boldsymbol{\varPhi }}}_{\boldsymbol{X}}}$ 进行ISTFT得到增强后的语音 $\hat {{\boldsymbol{s}}}(k)$ . 以上过程可用式（4）描述：

$\begin{gathered} \left| {\hat {{\boldsymbol{S}}}} \right| = {{\boldsymbol{M}}} \odot \left| {{\boldsymbol{X}}} \right|, \\ \hat {{\boldsymbol{s}}}(k) = \Re \left(\left| {\hat {{\boldsymbol{S}}}} \right|,{{{\boldsymbol{\varPhi}} }_{{\boldsymbol{X}}}}\right), \\ \end{gathered}$

(4)

其中 $\Re$ 表示ISTFT， $\odot$ 表示矩阵对应元素相乘.

2. 网络结构设计

2.1 融合扩张卷积与密集连接的模块设计

为能够充分捕获语音时频谱在频率、时间方向上的上下文信息，同时解决随着网络深度增加带来的信息丢失问题，本文将扩张卷积与密集连接结构相结合，分别设计了频率扩张密集模块（frequency dilated dense module, FDDM）和时间扩张密集模块（time dilated dense module, TDDM）.

FDDM的结构如所示（图中表示卷积层的方框内第1行数字依次表示扩张因子、卷积核大小和卷积核数量），其由 ${\text{6}}$ 个FDCU卷积单元以密集连接的方式组成，其中每个FDCU都包括2层2D卷积层，且每个卷积层之后都连接了1层归一化层（batch normalization, BN）和1个修正线性单元（rectified linear unit, ReLU）. 但第1个卷积层使用普通2D卷积，用以减少通道数；而第2个卷积层使用频率扩张2D卷积，其只在频率方向使用扩张因子以增大卷积核尺寸，由此增大感受野来捕获频率方向的上下文信息.

图 1 频率扩张密集模块结构

Figure 1. The structure of FDDM

下载: 全尺寸图片幻灯片

FDDM结构特色在于引入了密集连接结构：表现为每一级FDCU的输入都是整个FDDM的输入与其前面各级FDCU输出的汇集，从而各级FDCU的输入依次为 $16i \times 257 \times T,i = 1,2, … ,6$ . 为保证在频率方向获得足够大的感受野，需逐级增大FDCU的扩张因子 ${d_i}$ ，将其依次设定为 ${2^{i - 1}},i = 1,2, … ,6$ .

TDDM则借鉴了TCN^[19]的设计结构，并采用了与FDDM类似的框架结构，如图2所示：同样由6个TDCU卷积块以密集连接的方式组成，每个TDCU都包括3部分，其中前2部分的结构为1D卷积层、归一化层、带参数的线性修正单元（parametric rectified linear unit, PReLU），第3部分只有1层单独的1D卷积层. 第1部分采用普通1D卷积，用以减少通道数；第2部分使用时间扩张卷积，用以学习时间方向的上下文信息；第3部分的单独卷积层则在输出时恢复整个TDCU的通道.

图 2 时间扩张密集模块结构

Figure 2. The structure of TDDM

下载: 全尺寸图片幻灯片

与FDDM同理，TDDM也融入了密集连接结构，表现为每一级TDCU的输入都是整个TDDM的输入与其前面各级TDCU输出的汇集，从而各级TDCU的输入为 $128i \times T,i = 1,2, … ,6$ ，且其扩张因子 ${d_i}$ 设定为 ${2^{i - 1}},i = 1,2, … ,6$ .

从深层次意义上讲，正是因为图1所示的FDDM和图2所示的TDDM的各层级联的FDCU和TDCU的入口采用了密集连接，才避免了经典信息论所述及的“多处理模块级联会引起信息丢失”的现象（即信息不增性原理）^[21]，从而保证了特征重用，并促进信息流的传递.

2.2 总体频率-时间扩张密集网络的设计

综合以上拥有较大感受野的FDDM和TDDM的基本模块设计，本文提出频率-时间扩张密集网络FTDDN.

展示了本文所提出的网络的框架结构，其输入时频幅度谱 $\left| {{\boldsymbol{X}}} \right|$ 首先通过2层2D卷积层. 第1个卷积层用于增加输入特征的通道数；第2个卷积层用于学习局部信息，并将其输出反馈给FDDM，以捕获频率方向的上下文信息和学习时间方向的局部信息.图3中表示卷积层的方框内的第1行数字表示卷积核大小和卷积核数量.

图 3 频率-时间扩张密集网络结构

Figure 3. The structure of FTDDN

下载: 全尺寸图片幻灯片

FDDM之后连接了2层2D卷积层和1层1D卷积层，其共同的作用是实现维度转换以及减少通道数，使FDDM的输出的维度转换为 $128 \times T$ ，并反馈至TDDM中以学习时间方向的上下文信息.

经TDDM处理后，其输出会送到3个卷积单元中，前2个卷积单元由1D卷积层、归一化层和PReLU激活函数组成，用以聚合FDDM和TDDM学习到的频率、时间方向上的上下文信息，后1个卷积单元由1D卷积层和Sigmoid激活函数组成，其将网络估计到的时频掩蔽模板 ${{\boldsymbol{M}}}$ 的维度恢复到 $257 \times T$ 并将其值限制在[0, 1]区间内.

2.3 损失函数

图3总体网络采用了文献[22]中提出的噪声感知多任务损失函数，即加权平均绝对误差（weighted mean absolute error, WMAE），其定义为：

$\begin{split} {{WMAE}} = a \times \frac{1}{{T\times F}}\sum\limits_{t = 0}^{T - 1} {\sum\limits_{f = 0}^{F - 1} {| {| {\hat {{\boldsymbol{S}}}(t,f)} | - | {{{\boldsymbol{S}}}(t,f)} |} |} } + \\ (1 - a) \times \frac{1}{{T\times F}}\sum\limits_{t = 0}^{T - 1} {\sum\limits_{f = 0}^{F - 1} {| {| {\hat {{\boldsymbol{N}}}(t,f)} t| - | {{{\boldsymbol{N}}}(t,f)}|} |} } , \\[-10pt] \end{split}$

(5)

其中 $| {\hat {{\boldsymbol{N}}}(t,f)} | = | {{{\boldsymbol{X}}}(t,f)} | - | {\hat {{\boldsymbol{S}}}(t,f)} |$ 表示噪声的幅度谱估计，而a则为干净语音和噪声之间的能量比值，即

$a = \dfrac{{\displaystyle\sum\limits_{t = 0}^{T - 1} {\displaystyle\sum\limits_{f = 0}^{F - 1} {{{\left| {{{\boldsymbol{S}}}(t,f)} \right|}^2}} } }}{{\displaystyle\sum\limits_{t = 0}^{T - 1} {\displaystyle\sum\limits_{f = 0}^{F - 1} {{{\left| {{{\boldsymbol{S}}}(t,f)} \right|}^2}} } + \displaystyle\sum\limits_{t = 0}^{T - 1} {\displaystyle\sum\limits_{f = 0}^{F - 1} {{{\left| {{{\boldsymbol{N}}}(t,f)} \right|}^2}} } }} .$

(6)

3. 实验设置

3.1 数据集及评价指标

实验数据集之一采用开源的VCTK语料库^[23]，其训练集包括28位说话人（14位女性和14位男性），测试集则包括另外2位不同的说话人（1位女性和1位男性）. 为创建含噪语音数据集，文献[23]的作者以4种信噪比（signal-noise ratio, SNR）（15 dB，10 dB，5 dB，0 dB）向干净语音训练集添加了10种常见环境噪声和人工制造的噪声^[23]，从而生成包含有11572个语音的含噪语音训练集；以另外4种SNR（17.5 dB，12.5 dB，7.5 dB，2.5 dB）向干净语音测试集中添加了5种常见环境噪声^[23]，从而生成包含有824个语音的含噪语音测试集. 为测试网络的泛化能力，测试集与训练集中所使用的噪声均不相同. 因测试集中使用的说话人和噪声类型均与训练集不同，故也将其用作验证集以优化模型参数. 为降低计算复杂度，本文将该语料库的信号采样率由48 kHz降为16 kHz.

实验数据集之二采用LibriSpeech语料库^[24]的干净语音，其采样率为16 kHz，而噪声来源取自DEMAND噪声库^[25]和DNS Challenge中的噪声集^[26]. 为了构造实验所用数据集，在训练阶段，本文分别从LibriSpeech干净语音训练集和干净语音验证集中随机选取13976句语音和871句语音，并采用随机选择的方式，将DEMAND噪声库中的1000种噪声以10种SNR（−7.5 dB，−6.5 dB，−4 dB，−3 dB，−1 dB，1 dB，3 dB，7 dB，−9 dB，11 dB）与这些干净语音混合，以生成含噪语音训练集和含噪语音验证集. 在测试阶段，本文从LibriSpeech干净语音测试集中随机选取740句语音，并以4种SNR（−5 dB，0 dB，5 dB，10 dB）向干净语音添加4种噪声（DEMAND噪声库：Cafter噪声、Kitchen噪声、Meeting噪声、Office噪声），生成含噪语音测试集. 为测试网络的泛化能力，该数据集中，测试集、验证集与训练集中的噪声不同：有水流声、汽车声等.

本文使用业内普遍接受的语音质量客观评估（perceptual evaluation of speech quality，PESQ）^[27]和短时客观可懂度（short-time objective intelligibility，STOI）^[28]，以及主观平均意见分数——信号失真的复合测度 (CSIG)、噪声失真的复合测度 (CBAK)和语音整体质量的复合测度 (COVL)，作为实验结果的评价指标^[29].

3.2 网络参数设置

本文所提出的FTDDN网络的主要参数设置为：使用汉宁（Hanning）窗作为STFT的时间窗，窗长为32 ms（帧长点数为512），帧移为16 ms（即50%重叠），由于实信号傅里叶变换具有共轭对称性，故输入STFT幅度谱特征的尺寸为 $257 \times T$ （T取决于各条语音的长度）.

在每次训练实验中，本文将设每批处理语音的条数BatchSize=4，在每批处理中，通过补零的方式使各句语音与该Batch中最长语音长度保持一致，对于超出4 s的语音，则只取前4 s参与训练. 实验选用Adam优化器，并以学习率0.0002训练网络100次.

4. 对比实验及结果分析

实验主要包括2方面：1）开展消融实验，以探究FDDM，TDDM内部的密集连接结构及卷积块FDCU和TDCU的数量R对本文所提模型的语音增强性能的影响；2）分别针对3.1节所提的2个数据集，将本文所提模型与现有的语音增强网络做性能对比.

4.1 消融实验

为探究卷积块FDCU和TDCU的数量R和模块FDDM及TDDM中的密集连接结构对语音增强网络性能的影响，本文基于VCTK语料库^[23]进行了消融实验. 为了简洁，消融实验结果仅使用PESQ和STOI作为客观评价指标.

图4展示了在FDDM和TDDM均存在密集连接结构的情况下，不同的卷积块数量R对网络性能的影响.

图 4 FDDM和TDDM中不同卷积块数量R对网络性能的影响

Figure 4. Influence of different number of convolutional blocks R in FDDM and TDDM on network performance

下载: 全尺寸图片幻灯片

从图4可看出，随着R的增大，网络性能逐渐提高至最高点后又开始逐渐下降. 具体而言，当R从2增加到4时，PESQ和STOI分别从2.89增加到2.95和从0.9388增加到0.9442；当R从4增加到6时，PESQ和STOI虽然也呈现一定程度的增加，但增速变缓，这是由于随着R的增加，网络深度增加，感受野也随之增加，使得网络学习到的上下文信息更丰富，最终提高了网络性能；而当R从6继续增大时，可看到PESQ变化趋势平缓、STOI开始下降，这是因为当R继续增加时，网络深度也会加深，这导致信息丢失问题加剧，而密集连接结构的信息补充作用又无法完全解决这一问题，进而导致了网络性能退化.

列出了在 $R = 6$ 的情况下去除TDDM和FDDM中的密集连接结构后网络的性能变化. 可发现：当分别去除TDDM和FDDM中的密集连接时，PESQ从3.02分别下降到了2.83和2.97，STOI从0.9451分别下降到了0.9409和0.9447，这反映了密集连接结构的有效性，证实了该结构可通过信息补充加强特征传递和特征重用，达到增强网络性能的效果. 从表1的PESQ和STOI的下降比例可看出，相比较而言，消融TDDM比消融FDDM影响更大，这是因为时间方向的上下文信息比频率方向的上下文信息更加丰富，从而间接证明了时间信息融合在提高网络性能方面更重要，但频率信息也不可忽略.

表 1 密集连接对网络性能的影响

Table 1. Influence of Dense Connection on Network Performance

方法	评价指标
方法	PESQ	STOI
无密集连接TDDM	2.83（↓6.29%）	0.9409（↓0.48%）
无密集连接FDDM	2.97（↓1.66%）	0.9447（↓0.04%）
FTDDN	3.02	0.9451
注：（↓*）表示该方法的得分相比于FTDDN的得分的下降比例.

下载: 导出CSV

| 显示表格

4.2 对比实验1

针对LibriSpeech语料库^[24]在不同噪声和信噪比的情况下，将本文网络与3种已有网络进行性能对比，这3种网络分别是：基于LSTM的语音增强方法、基于卷积循环网络（convolutional recurrent network, CRN）^[30]的语音增强方法、基于时间卷积神经网络（temporal convolutional neural network, TCNN）^[31]的语音增强方法.

表2和表3分别展示了对本文模型和3种对比模型测评得到的PESQ分数和STOI分数，可以看出：除了5dB Meeting噪声条件下，本文模型的STOI分数略低于CRN以外，在其他情况下，本文模型的PESQ分数和STOI分数均要高于对比模型，这表明本文模型的语音增强性能更优越.

表 2 使用LibriSpeech语料库对FTDDN与基线模型的评测PESQ分数

Table 2. Evaluation PESQ Scores of FTDDN and Baseline Models Using LibriSpeech Corpus

噪声	SNR/dB	对比模型
噪声	SNR/dB	Noisy	LSTM	CRN	TCNN	FTDDN
Cafter	−5	1.10	1.08	1.13	1.12	1.16
	0	1.14	1.15	1.30	1.24	1.39
	5	1.28	1.30	1.58	1.42	1.80
	10	1.57	1.50	1.93	1.62	2.33
Kitchen	−5	1.07	1.29	1.45	1.35	2.00
	0	1.15	1.44	1.77	1.52	2.48
	5	1.33	1.61	2.13	1.72	2.94
	10	1.66	1.75	2.51	1.91	3.34
Meeting	−5	1.07	1.09	1.13	1.11	1.17
	0	1.14	1.15	1.26	1.21	1.31
	5	1.29	1.26	1.49	1.38	1.60
	10	1.60	1.42	1.83	1.60	2.05
Office	−5	1.31	1.50	1.81	1.55	2.18
	0	1.61	1.62	2.18	1.76	2.63
	5	1.99	1.74	2.56	1.93	3.06
	10	2.84	1.85	2.89	2.06	3.49
注：加粗的数字表示每一行中最高的PESQ分数.

下载: 导出CSV

| 显示表格

表 3 使用LibriSpeech语料库对FTDDN与基线模型的评测STOI分数

Table 3. Evaluation STOI Scores of FTDDN and Baseline Models Using LibriSpeech Corpus

噪声	SNR/dB	对比模型
噪声	SNR/dB	Noisy	LSTM	CRN	TCNN	FTDDN
Cafter	−5	0.6033	0.5816	0.6614	0.6355	0.6679
	0	0.7261	0.7228	0.8044	0.7654	0.8052
	5	0.8296	0.8116	0.8884	0.8524	0.8885
	10	0.9014	0.8564	0.9337	0.9022	0.9350
Kitchen	−5	0.8569	0.8147	0.8921	0.8523	0.9114
	0	0.9110	0.8511	0.9328	0.8964	0.9418
	5	0.9498	0.8724	0.9581	0.9236	0.9626
	10	0.9741	0.8852	0.9734	0.9391	0.9772
Meeting	−5	0.6543	0.6374	0.6746	0.6455	0.6890
	0	0.7608	0.7295	0.7909	0.7629	0.7937
	5	0.8480	0.7999	0.8741	08482	0.8733
	10	0.9097	0.8460	0.9252	0.9000	0.9254
Office	−5	0.9345	0.8644	0.9396	0.9119	0.9496
	0	0.9630	0.8802	0.9606	0.9344	0.9678
	5	0.9796	0.8889	0.9729	0.9451	0.9801
	10	0.9893	0.8944	0.9805	0.9498	0.9884
注：加粗的数字表示每一行中最高的STOI分数.

下载: 导出CSV

| 显示表格

观察表2、表3的数据可发现，所有模型在Cafter和Meeting噪声条件下的语音增强性能都低于在Kitchen和Office噪声条件下的语音增强性能，这可归结为不同噪声源的时频谱结构造成的影响. 具体解释如下：如图5所示，Cafter和Meeting噪声中以人声为主，其时频谱结构与干净语音的结构非常相似，故增加了噪声与干净语音的区分难度，导致网络去噪性能下降；而Kitchen和Office噪声结构与干净语音结构相差很大，降低了网络从含噪语音中学习干净语音结构的难度，有助于提升去噪性能.

图 5 干净语音和不同噪声的时频谱

Figure 5. The time-frequency spectrograms of clean speech and different noises

下载: 全尺寸图片幻灯片

为直观反映各对比模型与本文模型的语音增强效果，图6展示了这些模型对 5dB Kitchen的含噪语音增强后的结果（其加噪前后的时频谱如图6（a）（b）所示），从中可看出，图6（c）所示的LSTM模型只是轻微地去除了噪声，只能恢复干净语音的大致结构；相比而言，图6（d）所示的CRN模型和图6（e）所示的TCNN模型去噪更显著，但其优势主要体现在低频区，而高频细节较为模糊；而本文提出的FTDDN模型在去除噪声的同时，又最大限度地保留了语音信息，见图6（f）. 究其原因，各对比模型仅着重考虑了语音时间方向的上下文信息，而忽略了语音频率方向上下文信息间的联系，而语音的能量大部分聚集在低频部分，这导致模型对语音的高频信息关注度降低，使得增强后的语音高频信息丢失，而本文提出的FTDDN模型给予了语音频率方向和时间方向上下文信息同等关注度，并同时学习了语音时频谱高频信息与低频信息之间的相关性和时间帧之间的依赖关系，最终得以保留完整的语音时频谱信息. 需指出的是，以上实验所使用的测试集中的说话人、噪声种类以及信噪比皆与训练集和验证集中的完全不同. 故表2、表3的实验结果证实了本文模型在数据条件完全不匹配的情况下，仍可实现高性能降噪，证实了本文模型具备较高的泛化能力，可适应不同噪声条件下的复杂环境.

图 6 干净语音、含噪语音以及不同模型增强后的语音时频谱

Figure 6. The time-frequency spectrograms of clean speech, noisy speech, and enhanced by different models

下载: 全尺寸图片幻灯片

4.3 对比实验2

将本文提出的FTDDN模型与现有的SEGAN^[32]，Wave-U-Net^[33]，WaveCRN^[34]，MetricGAN^[35]，MB-TCN^[17]，NAAGN^[22]模型进行性能比较. 所有模型都使用VCTK语料库进行实验. 从表4列出的对比结果可以看出，本文提出的FTDDN模型在除STOI以外，所有指标都优于其他对比模型，这是由于SEGAN，Wave-U-Net，WaveCRN这3个网络的输入为时域波形，而本文的FTDDN则以时频幅度谱作为网络的输入，但时频域的信息往往比时域更加丰富、细致，从而使得网络可学习到更丰富的信息，这有利于网络性能的提升；MetricGAN，MB-TCN，NAAGN这3个网络虽然与本文的FTDDN一样，都以时频幅度谱作为网络的输入，但MetricGAN的设计是直接基于评价指标来优化网络，未专注于学习语音信号的细节信息，从而使网络性能受到限制；MB-TCN更多关注于学习语音信号的时间方向的上下文信息，却忽略了频率方向的上下文信息的重要性；NAAGN通过扩张卷积同时学习时间和频率方向的上下文信息，但并没有进行单独学习；而本文通过融合密集连接结构和扩张卷积将学习频率和时间方向的上下文信息分开进行，并在网络末端进行信息整合，故使网络学习到的语音信息更加细致，进而提升网络性能. 特别地，可以发现NAAGN模型的STOI分数略高于本文所提模型，这是由于NAAGN模型相对于本文模型额外引入了注意力门（attention gate, AG）模块，因此可进一步学习到输入样本中的更感兴趣的特征，并对其进行修剪，以保留相关的激活，从而可获得略高的STOI分数.

表 4 使用VCTK语料库对FTDDN与基线模型的性能评测分数

Table 4. Performance Evaluation Scores of FTDDN and Baseline Models Using VCTK Corpus

模型	评价指标
模型	PESQ	STOI	CSIG	CBAK	COVL
Noisy	1.97	0.9210	3.34	2.44	2.63
SEGAN	2.16		3.48	2.94	2.80
Wave-U-Net	2.40		3.52	3.24	2.96
WaveCRN	2.64		3.94	3.37	3.29
MetricGAN	2.86		3.99	3.18	3.42
NAAGN	2.90	0.9480	4.13	3.50	3.51
MB-TCN	2.94	0.9364	4.21	3.41	3.59
FTDDN	3.02	0.9451	4.25	3.49	3.63
注：加粗的数字表示每一列中的最高分数.

下载: 导出CSV

| 显示表格

5. 结　　论

为高质量地恢复语音信号，本文设计了频率-时间扩张密集网络（frequency-time dilated dense network, FTDDN），其包括2个最主要的模块：FDDM和TDDM，由于这2个模块均融入了扩张卷积和密集连接结构，因而FTDDN可获得较大的感受野以捕获频率方向和时间方向的上下文信息. 基于LibriSpeech和VCTK语料库与各类现有语音增强网络性能的对比实验表明：本文提出的FTDDN网络的语音增强性能更加优越，可在有效抑制噪声的同时高质量地恢复语音，故在语音识别、文本语音转换、助听器设计、网上会议等应用中有广阔应用前景.

作者贡献声明：黄翔东完善实验方案并修改论文；陈红红提出算法思路，并负责完成实验和撰写论文；甘霖提出指导意见.

图 1 从用例到机器学习任务的映射示例

Figure 1. Examples of mapping from use cases to machine learning tasks

下载: 全尺寸图片幻灯片

图 2 知识与机器学习融合方式举例

Figure 2. Examples of integration method of knowledge and machine learning

下载: 全尺寸图片幻灯片

图 3 面向机器学习的协作需求工程过程框架

Figure 3. Process framework of collaborative requirements engineering for machine learning

下载: 全尺寸图片幻灯片

图 4 基于视觉的施工设备智能监控案例

Figure 4. Intelligent monitoring case of construction equipment based on vision

下载: 全尺寸图片幻灯片

图 5 以状态图作为可解释性依据

Figure 5. The state diagram is used as an interpretable basis

下载: 全尺寸图片幻灯片

表 1 调研文献的主题统计

Table 1 Theme Statistics of Research Literature

研究主题	相关文献	文献篇数占比/%
宏观视角	[8,14−27]	9.20
需求过程模型	[28−34]	4.29
角色与协作	[35−42]	4.91
需求获取和规约	[43−51]	5.52
非功能性指标	[52−89]	23.31
数据与知识融合	[9, 90−108]	12.27
机器学习系统开发	[109−123]	9.20
可解释性	[5, 124−140]	11.04
数据流构建	[141−148]	4.91
模型溯源和验证	[149−155]	4.29
智能化应用	[156−173]	11.04

下载: 导出CSV

表 2 不同角色的关注焦点及挑战

Table 2 Focus and Challenges of Different Roles

角色	关注点	相关文献
业务专家	可信度、稳定性、公平性、可解释性	[83, 175−176]
需求工程师	非功能需求、业务建模、稳定性、风险评估	[2, 76, 177−179]
软件工程师	系统架构、快速原型、模型运用、数据正确处理	[22, 25, 173, 180−181]
领域专家	机理正确性、知识表达方式、数据使用方式、可解释性	[51, 182−184]
数据科学家	目标形式化、可用资源、评价指标、数据流	[40, 42, 117, 185]

下载: 导出CSV

表 3 各个角色参与的可解释性相关步骤

Table 3 Interpretability Related Steps of Each Role’s Participation

角色	逻辑可解释性依据	数据可解释依据	可解释性建模	可解释性评价
业务专家、领域专家	提供机理知识、提供业务规则、提供政策约束	提供数据描述	提供专家经验	符合机理知识、业务规则、政策约束、专家经验
数据科学家	收集逻辑可解释性依据	分析数据、提出数据分布假设、提出特征关联假设	融合建模、实现可解释性需求	验证可预测性、验证公平性
应用开发者	搭建反馈回路	提供相关数据、保护隐私安全	可操作性、控制成本	准备测试集
最终用户	提供使用反馈	提出隐私安全需求	提出公平性需求	验证失败案例

下载: 导出CSV

表 4 需求分析框架各步骤的产出内容

Table 4 Output Content of Each Step of the Demand Analysis Framework

产出内容	监督产出的角色	核心内容	合规要求
问题形式化	数据科学家	标准问题、约束条件、作用域、预期效果	准确无歧义，符合技术规范
上下文挖掘	业务专家	业务规则、领域机理、相关政策	逻辑完整，充分体现业务全貌
数据需求分析	数据科学家	数据语义、分布假设、标注规范、质量和特性	覆盖技术考量要素，满足伦理和公平性
评价指标转换	需求工程师	功能指标、非功能指标、可解释性依据、潜在风险点	用户可理解，覆盖可解释性依据
业务目标定义	业务专家	功能目标、作用范围、业务价值、非功能目标	满足业务需要，高技术可行性

下载: 导出CSV

参考文献(199)

[1]	Ashmore R, Calinescu R, Paterson C. Assuring the machine learning lifecycle: Desiderata, methods, and challenges[J]. ACM Computing Surveys, 2021, 54(5): 1−39
[2]	Habibullah K M, Horkoff J. Non-functional requirements for machine learning: Understanding current use and challenges in industry[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 13−23
[3]	Mohseni S, Zarei N, Ragan E D. A multidisciplinary survey and framework for design and evaluation of explainable AI systems[J]. ACM Transactions on Interactive Intelligent Systems, 2021, 11(3/4): 1−45
[4]	Mehrabi N, Morstatter F, Saxena N, et al. A survey on bias and fairness in machine learning[J]. ACM Computing Surveys, 2021, 54(6): 1−35
[5]	Floridi L. Establishing the rules for building trustworthy AI[J]. Nature Machine Intelligence, 2019, 1(6): 261−262 doi: 10.1038/s42256-019-0055-y
[6]	汪烨,陈骏武,夏鑫,等. 智能需求获取与建模研究综述[J]. 计算机研究与发展,2021,58(4):683−705 doi: 10.7544/issn1000-1239.2021.20200740 Wang Ye, Chen Junwu, Xia Xin, et al. Intelligent requirements elicitation and modeling: A literature review[J]. Journal of Computer Research and Development, 2021, 58(4): 683−705(in Chinese) doi: 10.7544/issn1000-1239.2021.20200740
[7]	Iqbal T, Elahidoost P, Lucio L. A bird’s eye view on requirements engineering and machine learning[C]//Proc of the 25th Asia-Pacific Software Engineering Conf (APSEC). Piscataway, NJ: IEEE, 2018: 11−20
[8]	Villamizar H, Escovedo T, Kalinowski M. Requirements engineering for machine learning: A systematic mapping study[C]//Proc of the 47th Euromicro Conf on Software Engineering and Advanced Applications (SEAA). Piscataway, NJ: IEEE, 2021: 29−36
[9]	Willard J, Jia Xiaowei, Xu Shaoming, et al. Integrating physics-based modeling with machine learning: A survey[J]. arXiv preprint, arXiv: 2003.04919, 2020
[10]	Harel D, Marron A, Sifakis J. Autonomics: In search of a foundation for next-generation autonomous systems[J]. Proceedings of the National Academy of Sciences, 2020, 117(30): 17491−17498 doi: 10.1073/pnas.2003162117
[11]	Horkoff J, Aydemir F B, Cardoso E, et al. Goal-oriented requirements engineering: An extended systematic mapping study[J]. Requirements Engineering, 2019, 24(2): 133−160 doi: 10.1007/s00766-017-0280-z
[12]	Wolny S, Mazak A, Carpella C, et al. Thirteen years of SysML: A systematic mapping study[J]. Software and Systems Modeling, 2020, 19(1): 111−169 doi: 10.1007/s10270-019-00735-y
[13]	Azevedo A, Santos M F. KDD, SEMMA and CRISP-DM: A parallel overview[C/OL]//Proc of the IADIS European Conf on Data Mining, 2008[2022-05-30].https://www.iadisportal.org/digital-library/kdd-semma-and-crisp-dm-a-parallel-overview
[14]	Vogelsang A, Borg M. Requirements engineering for machine learning: Perspectives from data scientists[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 245−251
[15]	Amershi S, Begel A, Bird C, et al. Software engineering for machine learning: A case study[C]//Proc of the 41st IEEE/ACM Int Conf on Software Engineering: Software Engineering in Practice (ICSE-SEIP). Piscataway, NJ: IEEE, 2019: 291−300
[16]	Silva G R S, Canedo E D. Requirements engineering challenges and techniques in building chatbots[C]//Proc of the 14th Int Conf on Agents and Artificial Intelligence. Setúbal, Portugal: SCITEPRESS, 2022: 180−187
[17]	Bencomo N, Guo J L C, Harrison R, et al. The secret to better AI and better software (is requirements engineering)[J]. IEEE Software, 2021, 39(1): 105−110
[18]	Yoshioka N, Husen J H, Tun H T, et al. Landscape of requirements engineering for machine learning-based AI systems[C]//Proc of the 28th Asia-Pacific Software Engineering Conf Workshops (APSEC Workshops). Piscataway, NJ: IEEE, 2021: 5−8
[19]	Ahmad K, Bano M, Abdelrazek M, et al. What’s up with requirements engineering for artificial intelligence systems?[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 1−12
[20]	Tukur M, Umar S, Hassine J. Requirement engineering challenges: A systematic mapping study on the academic and the industrial perspective[J]. Arabian Journal for Science and Engineering, 2021, 46(4): 3723−3748 doi: 10.1007/s13369-020-05159-1
[21]	Heyn H M, Knauss E, Muhammad A P, et al. Requirement engineering challenges for AI-intense systems development[C]//Proc of the 1st IEEE/ACM Workshop on AI Engineering-Software Engineering for AI (WAIN). Piscataway, NJ: IEEE, 2021: 89−96
[22]	Ferreira F, Silva L L, Valente M T. Software engineering meets deep learning: A mapping study[C]//Proc of the 36th Annual ACM Symp on Applied Computing. New York: ACM, 2021: 1542−1549
[23]	Dalpiaz F, Niu Nan. Requirements engineering in the days of artificial intelligence[J]. IEEE Software, 2020, 37(4): 7−10 doi: 10.1109/MS.2020.2986047
[24]	Kostova B, Gurses S, Wegmann A. On the interplay between requirements, engineering, and artificial intelligence[J]. Information and Software Technology, 2023, 158: 107176 doi: 10.1016/j.infsof.2023.107176
[25]	Belani H, Vukovic M, Car Ž. Requirements engineering challenges in building AI-based complex systems[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 252−255
[26]	Lwakatare L E, Raj A, Bosch J, et al. A taxonomy of software engineering challenges for machine learning systems: An empirical investigation[C]//Proc of the Int Conf on Agile Software Development. Berlin: Springer, 2019: 227−243
[27]	Bosch J, Olsson H H, Crnkovic I. It takes three to tango: Requirement, outcome/data, and AI driven development[C]//Proc of the 1st Int Workshop on Software-intensive Business: Start-ups, Ecosystems and Platforms (SiBW). Berlin: Springer, 2018: 177−192
[28]	Studer S, Bui T B, Drescher C, et al. Towards CRISP-ML (Q): A machine learning process model with quality assurance methodology[J]. Machine Learning and Knowledge Extraction, 2021, 3(2): 392−413 doi: 10.3390/make3020020
[29]	Nalchigar S, Yu E, Obeidi Y, et al. Solution patterns for machine learning[C]//Proc of the 31st Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 627−642
[30]	Nalchigar S, Yu E, Keshavjee K. Modeling machine learning requirements from three perspectives: A case report from the healthcare domain[J]. Requirements Engineering, 2021, 26(2): 237−254 doi: 10.1007/s00766-020-00343-z
[31]	Chuprina T, Mendez D, Wnuk K. Towards artefact-based requirements engineering for data-centric systems[J]. arXiv preprint, arXiv: 2103.05233, 2021
[32]	Barrera J M, Reina Reina A, Maté A, et al. Applying i* in conceptual modelling in machine learning[C]//Proc of the 14th Int iStar Workshop. Berlin: Springer, 2020: 56−62
[33]	Camilli M, Felderer M, Giusti A, et al. Towards risk modeling for collaborative AI[C]//Proc of the 1st IEEE/ACM Workshop on AI Engineering-Software Engineering for AI (WAIN). Piscataway, NJ: IEEE, 2021: 51−54
[34]	Alrajeh D, Cailliau A, van Lamsweerde A. Adapting requirements models to varying environments[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2020: 50−61
[35]	Deshpande A, Sharp H. Responsible AI systems: Who are the stakeholders?[C]//Proc of the 2022 AAAI/ACM Conf on AI, Ethics, and Society. New York: ACM, 2022: 227−236
[36]	Khuat T T, Kedziora D J, Gabrys B. The roles and modes of human interactions with automated machine learning systems[J]. arXiv preprint, arXiv: 2205.04139, 2022
[37]	Nahar N, Zhou Shurui, Lewis G, et al. Collaboration challenges in building ML-enabled systems: Communication, documentation, engineering, and process[C]//Proc of the 44th Int Conf on Software Engineering. New York: ACM, 2022: 413−425
[38]	Odong L A, Perini A, Susi A. Requirements engineering for collaborative artificial intelligence systems: A literature survey[C]//Proc of the 16th Int Conf on Research Challenges in Information Science. Berlin: Springer, 2022: 409−425
[39]	Barclay I, Abramson W. Identifying roles, requirements and responsibilitiesin trustworthy AI systems[C]//Proc of the 2021 ACM Int Joint Conf on Pervasive and Ubiquitous Computing and Proc of the 2021 ACM Int Symp on Wearable Computers. New York: ACM, 2021: 264−271
[40]	Piorkowski D, Park S, Wang A Y, et al. How AI developers overcome communication challenges in a multidisciplinary team: A case study[J]. Proceedings of the ACM on Human-Computer Interaction, 2021, 5(CSCW1): 131
[41]	Camilli M, Felderer M, Giusti A, et al. Risk-driven compliance assurance for collaborative AI systems: A vision paper[C]//Proc of the 27th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2021: 123−130
[42]	Zhang A X, Muller M, Wang Dakuo. How do data science workers collaborate? Roles, workflows, and tools[J]. Proceedings of the ACM on Human-Computer Interaction, 2020, 4(CSCW1): 22
[43]	Berry D M. Requirements engineering for artificial intelligence: What is a requirements specification for an artificial intelligence?[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 19−25
[44]	Al M, Ma Yihong, Alarcon P, et al. RESAM: Requirements elicitation and specification for deep-learning anomaly models with applications to UAV flight controllers[C]//Proc of the 30th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2022: 153−165
[45]	Villamizar H, Kalinowski M, Lopes H. Towards perspective-based specification of machine learning-enabled systems[J]. arXiv preprint, arXiv: 2206.09760, 2022
[46]	Gillani M, Niaz H A, Ullah A. Integration of software architecture in requirements elicitation for rapid software development[J]. IEEE Access, 2022, 10: 56158−56178 doi: 10.1109/ACCESS.2022.3177659
[47]	Gabriel S, Bentler D, Grote E M, et al. Requirements analysis for an intelligent workforce planning system: A socio-technical approach to design AI-based systems[J]. Procedia CIRP, 2022, 109: 431−436 doi: 10.1016/j.procir.2022.05.274
[48]	Wang Xi, Miao Weikai. A framework for requirements specification of machine-learning systems[C]//Proc of the 32nd Conf on Software Engineering and Knowledge Engineering (SEKE). Pittsburgh, Pennsylvania: KSI Research Inc, 2022: 7−12
[49]	Ahmad K. Human-centric requirements engineering for artificial intelligence software systems[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 468−473
[50]	Schuh G, Scholz P, Leich T, et al. Identifying and analyzing data model requirements and technology potentials of machine learning systems in the manufacturing industry of the future[C]//Proc of the 61st Int Scientific Conf on Information Technology and Management Science of Riga Technical University (ITMS). Piscataway, NJ: IEEE, 2020: 20178653
[51]	Cirqueira D, Nedbal D, Helfert M, et al. Scenario-based requirements elicitation for user-centric explainable AI[C]//Proc of the 4th Int Cross-Domain Conf for Machine Learning and Knowledge Extraction. Berlin: Springer, 2020: 321−341
[52]	d’Aloisio G, Di Marco A, Stilo G. Modeling quality and machine learning pipelines through extended feature models[J]. arXiv preprint, arXiv: 2207.07528, 2022
[53]	Levy N, Katz G. RoMA: A method for neural network robustness measurement and assessment[J]. arXiv preprint, arXiv: 2110.11088, 2021
[54]	Siebert J, Joeckel L, Heidrich J, et al. Construction of a quality model for machine learning systems[J]. Software Quality Journal, 2022, 30(2): 307−335 doi: 10.1007/s11219-021-09557-y
[55]	Haindl P, Hoch T, Dominguez J, et al. Quality characteristics of a software platform for human-AI teaming in smart manufacturing[J]. arXiv preprint, arXiv: 2205.15767, 2022
[56]	Madaio M, Egede L, Subramonyam H, et al. Assessing the fairness of AI systems: AI practitioners’ processes, challenges, and needs for support[J]. Proceedings of the ACM on Human-Computer Interaction, 2022, 6(CSCW1): 52
[57]	Perera A, Aleti A, Tantithamthavorn C, et al. Search-based fairness testing for regression-based machine learning systems[J]. Empirical Software Engineering, 2022, 27(3): 1−36
[58]	Chen Huaming, Babar M A. Security for machine learning-based software systems: A survey of threats, practices and challenges[J]. arXiv preprint, arXiv: 2201.04736, 2022
[59]	Georgieva I, Lazo C, Timan T, et al. From AI ethics principles to data science practice: A reflection and a gap analysis based on recent frameworks and practical experience[J]. AI and Ethics, 2022, 2: 697−711 doi: 10.1007/s43681-021-00127-3
[60]	Steimers A, Schneider M. Sources of risk of AI systems[J]. International Journal of Environmental Research and Public Health, 2022, 19(6): 3641
[61]	Bartels R, Dudink J, Haitjema S, et al. A perspective on a quality management system for AI/ML-based clinical decision support in hospital care[J]. Frontiers in Digital Health, 2022, 4: 942588
[62]	d’Aloisio G. Quality-driven machine learning-based data science pipeline realization: A software engineering approach[C]//Proc of the 44th ACM/IEEE Int Conf on Software Engineering: Companion Proceedings. Piscataway, NJ: IEEE, 2022: 291−293
[63]	Khan A, Siddiqui I F, Shaikh M, et al. Handling non-fuctional requirements in IoT-based machine learning systems[C]//Proc of the 7th Joint Int Conf on Digital Arts, Media and Technology with ECTI Northern Section Conf on Electrical, Electronics, Computer and Telecommunications Engineering (ECTI DAMT & NCON). Piscataway, NJ: IEEE, 2022: 477−479
[64]	Truong H L, Nguyen T M. QoA4ML—A framework for supporting contracts in machine learning services[C]//Proc of the IEEE Int Conf on Web Services (ICWS). Piscataway, NJ: IEEE, 2021: 465−475
[65]	Muñante D, Perini A, Kifetew F M, et al. Combining risk and variability modelling for requirements analysis in SAS engineering[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 396−401
[66]	Eliyahu T, Kazak Y, Katz G, et al. Verifying learning-augmented systems[C]//Proc of the 2021 ACM SIGCOMM. New York: ACM, 2021: 305−318
[67]	Mauri L, Damiani E. STRIDE-AI: An approach to identifying vulnerabilities of machine learning assets[C]//Proc of the IEEE Int Conf on Cyber Security and Resilience (CSR). Piscataway, NJ: IEEE, 2021: 147−154
[68]	Dey S, Lee S W. Multilayered review of safety approaches for machine learning-based systems in the days of AI[J]. Journal of Systems and Software, 2021, 176: 110941
[69]	Wang Jingyi, Chen Jialuo, Sun Youcheng, et al. Robot: Robustness-oriented testing for deep learning systems[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 300−311
[70]	Baluta T, Chua Z L, Meel K S, et al. Scalable quantitative verification for deep neural networks[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 312−323
[71]	Zhang J M, Harman M. "Ignorance and Prejudice" in software fairness[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1436−1447
[72]	Hobbs R. Integrating ethically align design into agile and CRISP-DM[C]//Proc of the Southeast Conf 2021. Piscataway, NJ: IEEE, 2021: 20631991
[73]	Tariq S, Cheema S M. Approaches for non-functional requirement modeling: A literature survey[C]//Proc of the 4th Int Conf on Computing & Information Sciences (ICCIS). Piscataway, NJ: IEEE, 2021: 21573902
[74]	Cerqueira J, Acco H, Dias E. Ethical guidelines and principles in the context of artificial intelligence[C]//Proc of the 17th Brazilian Symp on Information Systems. New York: ACM, 2021: 36
[75]	Byun T, Rayadurgam S. Manifold for machine learning assurance[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). Piscataway, NJ: IEEE, 2020: 97−100
[76]	Ishikawa F, Matsuno Y. Evidence-driven requirements engineering for uncertainty of machine learning-based systems[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 346−351
[77]	Damyanova B. Quality attributes in AI-ML-based systems: Differences and challenges[D]. Stuttgart: University of Stuttgart, 2020
[78]	Balasubramaniam N, Kauppinen M, Kujala S, et al. Ethical guidelines for solving ethical issues and developing AI systems[C]//Proc of the Int Conf on Product-Focused Software Process Improvement. Berlin: Springer, 2020: 331−346
[79]	Riccio V, Tonella P. Model-based exploration of the frontier of behaviours for deep learning system testing[C]//Proc of the 28th ACM Joint Meeting on European Software Engineering Conf and Symp on the Foundations of Software Engineering. New York: ACM, 2020: 876−888
[80]	Wilhjelm C, Younis A A. A threat analysis methodology for security requirements elicitation in machine learning based systems[C]//Proc of the 20th IEEE Int Conf on Software Quality, Reliability and Security Companion (QRS-C). Piscataway, NJ: IEEE, 2020: 426−433
[81]	Nakamichi K, Ohashi K, Namba I, et al. Requirements-driven method to determine quality characteristics and measurements for machine learning software and its evaluation[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 260−270
[82]	Humbatova N, Jahangirova G, Bavota G, et al. Taxonomy of real faults in deep learning systems[C]//Proc of the 42nd ACM/IEEE Int Conf on Software Engineering. Piscataway, NJ: IEEE, 2020: 1110−1121
[83]	Chechik M. Uncertain requirements, assurance and machine learning[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 2−3
[84]	Horkoff J. Non-functional requirements for machine learning: Challenges and new directions[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 386−391
[85]	Kuwajima H, Ishikawa F. Adapting square for quality assessment of artificial intelligence systems[C]//Proc of the IEEE Int Symp on Software Reliability Engineering Workshops (ISSREW). Piscataway, NJ: IEEE, 2019: 13−18
[86]	Bhatia J, Breaux T D. Semantic incompleteness in privacy policy goals[C]//Proc of the 26th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2018: 159−169
[87]	Zhang Boyu, Magaña J C, Davoodi A. Analysis of security of split manufacturing using machine learning[C]//Proc of the 55th Annual Design Automation Conf. New York: ACM, 2018: 2767−2780
[88]	Hayrapetian A, Raje R. Empirically analyzing and evaluating security features in software requirements[C]//Proc of the 11th Innovations in Software Engineering Conf. New York: ACM, 2018: 9
[89]	Hoel T, Griffiths D, Chen Weiqin. The influence of data protection and privacy frameworks on the design of learning analytics systems[C]//Proc of the 7th Int Learning Analytics & Knowledge Conf. New York: ACM, 2017: 243−252
[90]	Burwinkel H, Matz H, Saur S, et al. Physics-aware learning and domain-specific loss design in ophthalmology[J]. Medical Image Analysis, 2022, 76: 102314
[91]	Von Rueden L, Mayer S, Beckh K, et al. Informed machine learning—A taxonomy and survey of integrating knowledge into learning systems[J]. arXiv preprint, arXiv: 1903.12394, 2019
[92]	Vo K, Nguyen T, Pham D, et al. Combination of domain knowledge and deep learning for sentiment analysis of short and informal messages on social media[J]. International Journal of Computational Vision and Robotics, 2019, 9(5): 458−485 doi: 10.1504/IJCVR.2019.102286
[93]	Abbasi A, Nataraj C. Physics-informed machine learning for uncertainty reduction in time response reconstruction of a dynamic system[J]. IEEE Internet Computing, 2022, 26(4): 35−44 doi: 10.1109/MIC.2022.3170736
[94]	Huang Bin, Wang Jianhui. Applications of physics-informed neural networks in power systems-a review[J]. IEEE Transactions on Power Systems, 2022, 38(1): 572−588
[95]	Zhao W L, Gentine P, Reichstein M, et al. Physics-constrained machine learning of evapotranspiration[J]. Geophysical Research Letters, 2019, 46(24): 14496−14507 doi: 10.1029/2019GL085291
[96]	Lin Jing, Zhang Yu, Khoo E. Hybrid physics-based and data-driven modeling with calibrated uncertainty for lithium-ion battery degradation diagnosis and prognosis[J]. arXiv preprint, arXiv: 2110.13661, 2021
[97]	Cornelio J, Mohd Razak S, Cho Y, et al. Residual learning to integrate neural network and physics-based models for improved production prediction in unconventional reservoirs[J]. SPE Journal, 2022, 27(6): 1−22
[98]	Soleimani M, Intezari A, Pauleen D J. Mitigating cognitive biases in developing AI-assisted recruitment systems: A knowledge-sharing approach[J]. International Journal of Knowledge Management, 2022, 18(1): 1−18
[99]	Willard J, Jia Xiaowei, Xu Shaoming, et al. Integrating scientific knowledge with machine learning for engineering and environmental systems[J]. ACM Computing Surveys, 2022, 55(4): 1−37
[100]	Luo Xing, Zhang Dongxiao, Zhu Xu. Deep learning based forecasting of photovoltaic power generation by incorporating domain knowledge[J]. Energy, 2021, 225: 120240
[101]	Park S, Wang A Y, Kawas B, et al. Facilitating knowledge sharing from domain experts to data scientists for building NLP models[C]//Proc of the 26th Int Conf on Intelligent User Interfaces. New York: ACM, 2021: 585−596
[102]	Xie Xiaozheng, Niu Jianwei, Liu Xuefeng, et al. A survey on incorporating domain knowledge into deep learning for medical image analysis[J]. Medical Image Analysis, 2021, 69: 101985
[103]	Van Oort C M. Leveraging domain knowledge in deep learning systems[D]. Burlington: The University of Vermont and State Agricultural College, 2021
[104]	Chitchyan R, Bird C. Theory as a source of software requirements[C]//Proc of the 28th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2020: 227−237
[105]	Vázquez-Ingelmo A, García Peñalvo F J, Therón R. Advances in the use of domain engineering to support feature identification and generation of information visualizations[C]//Proc of the 8th Int Conf on Technological Ecosystems for Enhancing Multiculturality. New York: ACM, 2020: 1053−1056
[106]	Chai Yidong, Liu Hongyan, Xu Jie. Glaucoma diagnosis based on both hidden features and domain knowledge through deep learning models[J]. Knowledge-Based Systems, 2018, 161: 147−156 doi: 10.1016/j.knosys.2018.07.043
[107]	Song Yangqiu, Roth D. Machine learning with world knowledge: The position and survey[J]. arXiv preprint, arXiv: 1705.02908, 2017
[108]	Haidry S Z, Falkner K, Szabo C. Identifying domain-specific cognitive strategies for software engineering[C]//Proc of the 2017 ACM Conf on Innovation and Technology in Computer Science Education. New York: ACM, 2017: 206−211
[109]	Laato S, Mäntymäki M, Minkkinen M, et al. Integrating machine learning with software development lifecycles: Insights from experts[C/OL]//Proc of the 30th European Conf on Information Systems. Association for Information Systems, 2022[2022-05-30].https://aisel.aisnet.org/ecis2022_rp/118
[110]	Washizaki H, Uchida H, Khomh F, et al. Machine learning architecture and design patterns[J/OL]. IEEE Software, 2020[2022-04-01]. http://g7.washi.cs.waseda.ac.jp/wp-content/uploads/2019/12/IEEE_Software_19__ML_Patterns.pdf
[111]	Lewis G A, Ozkaya I, Xu Xiwei. Software architecture challenges for ML systems[C]//Proc of the 37th IEEE Int Conf on Software Maintenance and Evolution (ICSME). Piscataway, NJ: IEEE, 2021: 634−638
[112]	Serban A, Visser J. An empirical study of software architecture for machine learning[J]. arXiv preprint, arXiv: 2105.12422, 2021
[113]	Chen Zhenpeng, Yao Huihan, Lou Yiling, et al. An empirical study on deployment faults of deep learning based mobile applications[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 674-685
[114]	Serban A, van der Blom K, Hoos H, et al. Adoption and effects of software engineering best practices in machine learning[C]//Proc of the 14th ACM/IEEE Int Symp on Empirical Software Engineering and Measurement (ESEM). New York: ACM, 2020: 3
[115]	John M M, Olsson H H, Bosch J. Developing ML/DL models: A design framework[C/OL]//Proc of the 14th Int Conf on Software and System Processes. New York: ACM, 2020[2022-05-30].https://www.diva-portal.org/smash/get/diva2:1553907/FULLTEXT01.pdf
[116]	Correia J L, Pereira J A, Mello R, et al. Brazilian data scientists: Revealing their challenges and practices on machine learning model development[C]//Proc of the 19th Brazilian Symp on Software Quality. New York: ACM, 2020: 10
[117]	Reimann L, Kniesel-Wünsche G. Achieving guidance in applied machine learning through software engineering techniques[C]//Proc of the 4th Int Conf on Art, Science, and Engineering of Programming. New York: ACM, 2020: 7−12
[118]	Kourouklidis P, Kolovos D, Matragkas N, et al. Towards a low-code solution for monitoring machine learning model performance[C]//Proc of the 23rd ACM/IEEE Int Conf on Model Driven Engineering Languages and Systems: Companion Conf. Piscataway, NJ: IEEE, 2020: 62
[119]	Prado F F, Digiampietri L A. A systematic review of automated feature engineering solutions in machine learning problems[C]//Proc of the 16th Brazilian Symp on Information Systems. New York: ACM, 2020: 12
[120]	Oppold S, Herschel M. A system framework for personalized and transparent data-driven decisions[C]//Proc of the 32nd Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2020: 153−168
[121]	Ole M, Volker G. Towards concept based software engineering for intelligent agents[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 42−48
[122]	Hesenius M, Schwenzfeier N, Meyer O, et al. Towards a software engineering process for developing data-driven applications[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 35−41
[123]	Mucha T M, Ma Sijia, Abhari K. Sustainability of machine learning-based solutions: A lifecycle perspective[C/OL]//Proc of the Pacific Asia Conf on Information Systems. Association for Information Systems, 2022[2022-05-30].https://aisel.aisnet.org/pacis2022/262/
[124]	Yu Kui, Guo Xianjie, Liu Lin, et al. Causality-based feature selection: Methods and evaluations[J]. ACM Computing Surveys, 2020, 53(5): 1−36
[125]	Cabrera Á A, Ribeiro M T, Lee B, et al. What did my AI learn? How data scientists make sense of model behavior[J]. ACM Transactions on Computer-Human Interaction, 2022, 30(1): 1−27
[126]	Sun Jiao, Liao Q V, Muller M, et al. Investigating explainability of generative AI for code through scenario-based design[C]//Proc of the 27th Int Conf on Intelligent User Interfaces. New York: ACM, 2022: 212−228
[127]	Shen M W. Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficient[J]. arXiv preprint, arXiv: 2202.05302, 2022
[128]	Balasubramaniam N, Kauppinen M, Hiekkanen K, et al. Transparency and explainability of AI systems: Ethical guidelines in practice[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 3−18
[129]	Piorkowski D, Richards J, Hind M. Evaluating a methodology for increasing AI transparency: A case study[J]. arXiv preprint, arXiv: 2201.13224, 2022
[130]	Sadeghi M, Klös V, Vogelsang A. Cases for explainable software systems: Characteristics and examples[C]//Proc of the 29th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2021: 181−187
[131]	Langer M, Baum K, Hartmann K, et al. Explainability auditing for intelligent systems: A rationale for multi-disciplinary perspectives[C]//Proc of the 29th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2021: 164−168
[132]	Chazette L, Brunotte W, Speith T. Exploring explainability: A definition, a model, and a knowledge catalogue[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 197−208
[133]	Velez M, Jamshidi P, Siegmund N, et al. White-box analysis over machine learning: Modeling performance of configurable systems[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1072−1084
[134]	Narteni S, Ferretti M, Orani V, et al. From explainable to reliable artificial intelligence[C]//Proc of the 5th Int Cross-Domain Conf for Machine Learning and Knowledge Extraction. Berlin: Springer, 2021: 255−273
[135]	Zohdinasab T, Riccio V, Gambi A, et al. Deephyperion: Exploring the feature space of deep learning-based systems through illumination search[C]//Proc of the 30th ACM SIGSOFT Int Symp on Software Testing and Analysis. New York: ACM, 2021: 79−90
[136]	Liao Q V, Gruen D, Miller S. Questioning the AI: Informing design practices for explainable AI user experiences[C]//Proc of the 2020 CHI Conf on Human Factors in Computing Systems. New York: ACM, 2020: 1−15
[137]	Köhl M A, Baum K, Langer M, et al. Explainability as a non-functional requirement[C]//Proc of the 27th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2019: 363−368
[138]	Yang Zijiang, Al-Bahrani R, Reid A C E, et al. Deep learning based domain knowledge integration for small datasets: Illustrative applications in materials informatics[C]//Proc of the Int Joint Conf on Neural Networks (IJCNN). Piscataway, NJ: IEEE, 2019: 19028509
[139]	Wolf C T. Explainability scenarios: Towards scenario-based XAI design[C]//Proc of the 24th Int Conf on Intelligent User Interfaces. New York: ACM, 2019: 252−257
[140]	Eiband M, Schneider H, Bilandzic M, et al. Bringing transparency design into practice[C]//Proc of the 23rd Int Conf on Intelligent User Interfaces. New York: ACM, 2018: 211−223
[141]	Thakkar D, Ismail A, Kumar P, et al. When is machine learning data good?: Valuing in public health datafication[C]//Proc of the CHI Conf on Human Factors in Computing Systems. New York: ACM, 2022: 322
[142]	Jung J Y, Steinberger T, King J L, et al. How domain experts work with data: Situating data science in the practices and settings of craftwork[J]. Proceedings of the ACM on Human-Computer Interaction, 2022, 6(CSCW1): 58
[143]	Lwakatare L E, Rånge E, Crnkovic I, et al. On the experiences of adopting automated data validation in an industrial machine learning project[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering: Software Engineering in Practice (ICSE-SEIP). Piscataway, NJ: IEEE, 2021: 248−257
[144]	Hutchinson B, Smart A, Hanna A, et al. Towards accountability for machine learning datasets: Practices from software engineering and infrastructure[C]//Proc of the 2021 ACM Conf on Fairness, Accountability, and Transparency. New York: ACM, 2021: 560-575
[145]	Shao Zhijun, Wu Ji, Zhao Wenxiao, et al. How data plays in the requirements of face recognition system: A concern driven systematic literature review[C]//Proc of the 28th Asia-Pacific Software Engineering Conf Workshops (APSEC Workshops). Piscataway, NJ: IEEE, 2021: 9−12
[146]	Astegher M, Busetta P, Perini A, et al. Specifying requirements for data collection and analysis in data-driven RE. A research preview[C]//Proc of the 27th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2021: 182−188
[147]	Hayes J H, Payne J, Leppelmeier M. Toward improved artificial intelligence in requirements engineering: Metadata for tracing datasets[C]//Proc of the 27th IEEE Int Requirements Engineering Conf Workshops (REW). Piscataway, NJ: IEEE, 2019: 256−262
[148]	Biffl S, Lüder A, Rinker F, et al. Efficient engineering data exchange in multi-disciplinary systems engineering[C]//Proc of the 29th Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 17−31
[149]	Mei Songzhu, Liu Cong, Wang Qinglin, et al. Model provenance management in MLOps pipeline[C]//Proc of the 8th Int Conf on Computing and Data Engineering. New York: ACM, 2022: 45−50
[150]	Kaminwar S R, Goschenhofer J, Thomas J, et al. Structured verification of machine learning models in industrial settings[J/OL]. Big Data, 2021[2022-04-01].https://www.liebertpub.com/doi/10.1089/big.2021.0112
[151]	Xiao Yan, Beschastnikh I, Rosenblum D S, et al. Self-checking deep neural networks in deployment[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 372−384
[152]	Wang Song, Shrestha N, Subburaman A K, et al. Automatic unit test generation for machine learning libraries: How far are we?[C]//Proc of the 43rd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2021: 1548−1560
[153]	Hestness J, Ardalani N, Diamos G. Beyond human-level accuracy: Computational challenges in deep learning[C/OL]//Proc of the 24th Symp on Principles and Practice of Parallel Programming. New York: ACM, 2019[2022-05-30]. https://dl.acm.org/doi/10.1145/3293883.3295710
[154]	Gharibi G, Walunj V, Rella S, et al. Modelkb: Towards automated management of the modeling lifecycle in deep learning[C]//Proc of the 7th IEEE/ACM Int Workshop on Realizing Artificial Intelligence Synergies in Software Engineering (RAISE). Piscataway, NJ: IEEE, 2019: 28−34
[155]	Borg M, Englund C, Wnuk K, et al. Safely entering the deep: A review of verification and validation for machine learning and a challenge elicitation in the automotive industry[J]. arXiv preprint, arXiv: 1812.05389, 2018
[156]	Czvetkó T, Kummer A, Ruppert T, et al. Data-driven business process management-based development of Industry 4.0 solutions[J]. CIRP Journal of Manufacturing Science and Technology, 2022, 36: 117−132 doi: 10.1016/j.cirpj.2021.12.002
[157]	Merkelbach S, Von Enzberg S, Kühn A, et al. Towards a process model to enable domain experts to become citizen data scientists for industrial applications[C]//Proc of the 5th IEEE Int Conf on Industrial Cyber-Physical Systems (ICPS). Piscataway, NJ: IEEE, 2022
[158]	Heyn H M, Subbiah P, Linder J, et al. Setting AI in context: A case study on defining the context and operational design domain for automated driving[C]//Proc of the 28th Int Working Conf on Requirements Engineering: Foundation for Software Quality. Berlin: Springer, 2022: 199−215
[159]	Zhang Ran, Albrecht A, Kausch J, et al. DDE process: A requirements engineering approach for machine learning in automated driving[C]//Proc of the 29th IEEE Int Requirements Engineering Conf (RE). Piscataway, NJ: IEEE, 2021: 269−279
[160]	Borg M, Bronson J, Christensson L, et al. Exploring the assessment list for trustworthy AI in the context of advanced driver-assistance systems[C]//Proc of the 2nd IEEE/ACM Int Workshop on Ethics in Software Engineering Research and Practice (SEthics). Piscataway, NJ: IEEE, 2021: 5−12
[161]	Martínez-Fernández S, Franch X, Jedlitschka A, et al. Developing and operating artificial intelligence models in trustworthy autonomous systems[C]//Proc of the 15th Int Conf on Research Challenges in Information Science. Berlin: Springer, 2021: 221−229
[162]	Kolyshkina I, Simoff S. Interpretability of machine learning solutions in public healthcare: The CRISP-ML approach[J]. Frontiers in Big Data, 2021, 4: 660206
[163]	Pinto A. Requirement specification, analysis and verification for autonomous systems[C]//Proc of the 58th ACM/IEEE Design Automation Conf (DAC). Piscataway, NJ: IEEE, 2021: 1315−1318
[164]	Leung C K, Fung D L, Mai D, et al. Explainable data analytics for disease and healthcare informatics[C]//Proc of the 25th Int Database Engineering & Applications Symp. New York: ACM, 2021: 65−74
[165]	León A, García S A, Costa M, et al. Evolution of an adaptive information system for precision medicine[C]//Proc of the 33rd Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2021: 3−10
[166]	Pereira A, Thomas C. Challenges of machine learning applied to safety-critical cyber-physical systems[J]. Machine Learning and Knowledge Extraction, 2020, 2(4): 579−602 doi: 10.3390/make2040031
[167]	Zhang Ru, Xiao Wencong, Zhang Hongyu, et al. An empirical study on program failures of deep learning jobs[C]//Proc of the 42nd IEEE/ACM Int Conf on Software Engineering (ICSE). Piscataway, NJ: IEEE, 2020: 1159−1170
[168]	du Preez A, Bekker J. A machine learning decision support framework for industrial engineering purposes[C]//Proc of the 1st Int Conf on Industrial Engineering and Industrial Management. New York: ACM, 2020: 9−14
[169]	Leung C K, Fung D L X, Mushtaq S B, et al. Data science for healthcare predictive analytics[C]//Proc of the 24th Symp on Int Database Engineering & Applications. New York: ACM, 2020: 8
[170]	Loucopoulos P, Kavakli E, Chechina N. Requirements engineering for cyber physical production systems[C]//Proc of the 31st Int Conf on Advanced Information Systems Engineering. Berlin: Springer, 2019: 276−291
[171]	Bao N, Chung S T. A rule-based smart thermostat[C]//Proc of the 1st Int Conf on Computational Intelligence and Intelligent Systems. New York: ACM, 2018: 20−25
[172]	Ma J, Park S C, Shin J H, et al. AI based intelligent system on the EDISON platform[C]//Proc of the 2018 Artificial Intelligence and Cloud Computing Conf. New York: ACM, 2018: 106−114
[173]	Gupta P, Suryavanshi A, Maheshwari S, et al. Human-machine interface system for pre-diagnosis of diseasesusing machine learning[C]//Proc of the 18th Int Conf on Machine Vision and Applications. New York: ACM, 2018: 71−75
[174]	Sothilingam R, Eric S K. Modeling agents, roles, and positions in machine learning project organizations[C]//Proc of the 15th Int iStar Workshop. Berlin: Springer, 2020: 61−66
[175]	Lim S, Henriksson A, Zdravkovic J. Data-driven requirements elicitation: A systematic literature review[J]. SN Computer Science, 2021, 2(1): 1−35 doi: 10.1007/s42979-020-00382-x
[176]	Kirikova M. Continuous requirements engineering[C/OL]//Proc of the 18th Int Conf on Computer Systems and Technologies. New York: ACM, 2017[2022-05-31].https://dl.acm.org/doi/10.1145/3134302.3134304
[177]	Hartmann T, Moawad A, Fouquet F, et al. The next evolution of MDE: A seamless integration of machine learning into domain modeling[J]. Software & Systems Modeling, 2019, 18(2): 1285−1304
[178]	Arruda D, Laigner R. Requirements engineering practices and challenges in the context of big data software development projects: Early insights from a case study[C]//Proc of the IEEE Int Conf on Big Data (Big Data). Piscataway, NJ: IEEE, 2020: 2012−2019
[179]	Alrajeh D, Van Lamsweerde A, Kramer J, et al. Risk-driven revision of requirements models[C]//Proc of the 38th Int Conf on Software Engineering. New York: ACM, 2016: 855−865
[180]	Ishikawa F, Yoshioka N. How do engineers perceive difficulties in engineering of machine-learning systems?—Questionnaire survey[C]//Proc of the 7th IEEE/ACM Joint Int Workshop on Conducting Empirical Studies in Industry (CESI) and 6th Int Workshop on Software Engineering Research and Industrial Practice (SER&IP). Piscataway, NJ: IEEE, 2019: 2−9
[181]	Muccini A, Vaidhyanathan K. Software architecture for ML-based systems: What exists and what lies ahead[C]// Proc of 2021 IEEE/ACM 1st Workshop on AI Engineering-Software Fngineering for AI (WAIN). Piscataway, NJ: IEEE, 2021:121-128
[182]	Murdock R J, Kauwe S K, Wang A Y T, et al. Is domain knowledge necessary for machine learning materials properties?[J]. Integrating Materials and Manufacturing Innovation, 2020, 9(3): 221−227 doi: 10.1007/s40192-020-00179-z
[183]	Childs C M, Washburn N R. Embedding domain knowledge for machine learning of complex material systems[J]. MRS Communications, 2019, 9(3): 806−820 doi: 10.1557/mrc.2019.90
[184]	Latef M, Aslam T, Sehar P. Impact of domain knowledge in phase of requirement engineering[J]. International Journal of Advanced Research and Development, 2018, 3(6): 54−57
[185]	Fu Wei, Menzies T. Easy over hard: A case study on deep learning[C]//Proc of the 11th Joint Meeting on Foundations of Software Engineering. New York: ACM, 2017: 49−60
[186]	Caramujo J, Rodrigues A, Monfared S, et al. RSL-IL4Privacy: A domain-specific language for the rigorous specification of privacy policies[J]. Requirements Engineering, 2019, 24(1): 1−26 doi: 10.1007/s00766-018-0305-2
[187]	Darimont R, Delor E, Massonet P, et al. GRAIL/KAOS: An environment for goal-driven requirements engineering[C]//Proc of the 19th Int Conf on Software Engineering. New York: ACM, 1997: 612−613
[188]	Doerr J, Kerkow D, Koenig T, et al. Non-functional requirements in industry-three case studies adopting an experience-based NFR method[C]//Proc of the 13th IEEE Int Conf on Requirements Engineering (RE’05). Piscataway, NJ: IEEE, 2005: 373−382
[189]	Mesquita R, Jaqueira A, Lucena M, et al. US2StarTool: Generating i* models from user stories[C]//Proc of the 10th Int iStar Workshop. Berlin: Springer. 2015: 103−108
[190]	Robertson S, Robertson J. Mastering the Requirements Process: Getting Requirements Right[M]. Reading, MA: Addison-Wesley, 2012
[191]	Saeed W, Omlin C. Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities[J]. arXiv preprint, arXiv: 2111.06420, 2021
[192]	Nazir R. Studying software architecture design challenges, best practices and main decisions for machine learning systems[D]. Gothenburg, Sweden: Department of Computer Science and Engineering, Chalmers University of Technology, University of Gothenburg, 2021
[193]	Di Stefano J S, Menzies T. Machine learning for software engineering: Case studies in foftware reuse[C]// Proc of the 14th IEEE Int Conf on Tools with Artificial Intelligence (ICTAI 2002). Piscataway, NJ: IEEE, 2002: 246−251
[194]	Crawley F, Tyler B. HAZOP: Guide to Best Practice[M]. Amsterdam: Elsevier, 2015
[195]	Chiozza M L, Ponzetti C. FMEA: A model for reducing medical errors[J]. Clinica Chimica Acta, 2009, 404(1): 75−78 doi: 10.1016/j.cca.2009.03.015
[196]	Luthra P. FMECA: An integrated approach[C]//Proc of the 37th Annual Reliability and Maintainability Symp. Piscataway, NJ: IEEE, 1991: 235−241
[197]	Willey R J. Layer of protection analysis[J]. Procedia Engineering, 2014, 84: 12−22 doi: 10.1016/j.proeng.2014.10.405
[198]	Ericson C A, Ll C. Fault tree analysis[C/OL]//Proc of the 17th System Safety Conf. Saint Paul, USA: The International System Safety Society, 1999[2022-05-31].https://ftaassociates.files.wordpress.com/2018/12/C.-Ericson-Fault-Tree-Analysis-A-History-Proceedings-of-the-17th-International-System-Safety-Conference-1999.pdf
[199]	Andrews J D, Dunnett S J. Event-tree analysis using binary decision diagrams[J]. IEEE Transactions on Reliability, 2000, 49(2): 230−238 doi: 10.1109/24.877343

施引文献(3)

期刊类型引用(2)

1.	李静莹. 基于模糊理论和卷积神经网络的翻译机器人交互语音降噪方法研究. 自动化与仪器仪表. 2025(01): 286-289+294 . 百度学术
2.	张晨辉，原之安，钱宇华 . 结合卷积增强窗口注意力的双分支语音增强神经网络. 计算机研究与发展. 2025(04): 852-862 . 本站查看