AccSMBO: Using Hyperparameters Gradient and Meta-Learning to Accelerate SMBO
-
摘要: 为了利用最佳超参高概率范围和超参梯度，提出了加速的序列模型优化算法(sequential model-based optimization algorithms, SMBO)——AccSMBO算法.AccSMBO使用了具有良好抗噪能力的基于梯度的多核高斯过程回归方法，利用元学习数据集的meta-acquisition函数.AccSMBO自然对应的并行算法则使用了基于元学习数据集的并行算法资源调度方案.基于梯度的多核高斯过程回归可以避免超参梯度噪音对拟合高斯过程的影响，加快构建较好超参-效果模型的速度.meta-acquisition函数通过读取元学习数据集，总结最佳超参高概率范围，加快最优超参搜索.在AccSMBO自然对应的并行算法中，并行资源调度方法使更多的并行计算资源用于计算最佳超参高概率范围中的超参，更快探索最佳超参高概率范围.上述3个技术充分利用超参梯度和最佳超参高概率范围加速SMBO算法.在实验中，相比于基于传统的SMBO算法实现的SMAC(sequential model-based algorithm configuration)算法、基于梯度下降的HOAG(hyperparameter optimization with approximate gradient)算法和常用的随机搜索算法，AccSMBO使用最少的资源找到了效果最好的超参.Abstract: Current machine learning models require numbers of hyperparameters. Adjusting those hyperparameters is an exhausting job. Thus, hyperparameters optimization algorithms play important roles in machine learning application. In hyperparameters optimization algorithms, sequential model-based optimization algorithms (SMBO) and parallel SMBO algorithms are state-of-the-art hyperpara-meter optimization methods. However, (parallel) SMBO algorithms do not take the best hyperpara-meters high possibility range and gradients into considerasion. It is obvious that best hyperparameters high possibility range and hyperparameter gradients can accelerate traditional hyperparameters optimization algorithms. In this paper, we accelerate the traditional SMBO method and name our method as AccSMBO. In AccSMBO, we build a novel gradient-based multikernel Gaussian process. Our multikernel Gaussian process has a good generalization ability which reduces the gradient noise influence on SMBO algorithm. And we also design meta-acquisition function and parallel resource allocation plan which encourage that (parallel) SMBO puts more attention on the best hyperpara-meters high possibility range. In theory, our method ensures that all hyperparameter gradient information and the best hyperparameters high possibility range information are fully used. In L2 norm regularised logistic loss function experiments, on different scales datasets: small-scale dataset Pc4, middle-scale dataset Rcv1, large-scale dataset Real-sim, compared with state-of-the-art gradient based algorithm: HOAG and state-of-the-art SMBO algorithm: SMAC, our method exhibits the best performance.
-
Keywords:
- AutoML /
- SMBO /
- black box optimization /
- hypergradient /
- metalearning /
- parallel resource allocation
-
-
期刊类型引用(9)
1. 臧洁，任旭，冯艳爽，王妍，肖萍，鲁锦涛. 一种干扰系数自探测的网络事件选取方法. 小型微型计算机系统. 2024(03): 763-768 . 
百度学术
2. 路苗，门可，马永红，张海瑞，冯彦成. 基于SIS模型的群体社交网络舆情演化仿真. 吉林大学学报(信息科学版). 2023(01): 106-111 . 百度学术
3. 马帅，刘建伟，左信. 图神经网络综述. 计算机研究与发展. 2022(01): 47-80 . 本站查看
4. 夏一雪，张立红，何巍，张双狮. 自治线性风险作用下网络舆情演化建模与仿真研究. 情报杂志. 2022(05): 92-98 . 百度学术
5. 易杰，曹腾飞，黄明峰，黄肖翰，张子震. 基于时间编码LSTM的高校舆情热点趋势预测研究. 大数据. 2022(05): 124-138 . 百度学术
6. 张杨，廉吉庆，张扬，高德毅. 国内网络舆情情感研究热点分析. 网络安全与数据治理. 2022(07): 47-55 . 百度学术
7. 徐缤荣. 融媒体背景下社会热点新闻舆情传播控制模型构建. 微型电脑应用. 2022(10): 149-152 . 百度学术
8. 臧洁，任旭. 考虑兴趣偏好和多事件影响的网络事件推演模型研究. 辽宁大学学报(自然科学版). 2022(04): 298-306 . 百度学术
9. 赵剑，董文华，史丽娟，匡哲君，毕京晓，王晢宇，强文倩. 针对突发公共事件的舆情监测与可视化分析. 吉林大学学报(信息科学版). 2021(06): 712-719 . 百度学术
其他类型引用(5)
计量
- 文章访问数: 860
- HTML全文浏览量: 1
- PDF下载量: 301
- 被引次数: 14
下载:
