Online Cross-Modal Hashing with Double Structure Preserving

Kang Xiao; Liu Xingbo; Lu Pengyu; Zhao Zhijie; Nie Xiushan; Wang Shaohua; Yin Yilong

doi:10.7544/issn1000-1239.202330433

Journal of Computer Research and Development > 2024 > 61(11): 2923-2936. > DOI: 10.7544/issn1000-1239.202330433 CSTR: 32373.14.issn1000-1239.202330433

Kang Xiao, Liu Xingbo, Lu Pengyu, Zhao Zhijie, Nie Xiushan, Wang Shaohua, Yin Yilong. Online Cross-Modal Hashing with Double Structure Preserving[J]. Journal of Computer Research and Development, 2024, 61(11): 2923-2936. DOI: 10.7544/issn1000-1239.202330433

Citation:

PDF (1638 KB)

Online Cross-Modal Hashing with Double Structure Preserving

1.
School of Software, Shandong University, Jinan 250101
2.
School of Computer Science and Technology, Shandong Jianzhu University, Jinan 250101

Funds: This work was supported by the National Natural Science Foundation of China (62176141, 62176139, 62206160), the Natural Science Foundation of Shandong Province (ZR2022QF082), and the Special Funds for Distinguished Professors of Shandong Jianzhu University.

More Information

Author Bio:
Kang Xiao: born in 1997. PhD. Her main research interests include multimedia retrieval and computer vision

Liu Xingbo: born in 1995. PhD, professor. His main research interests include multimedia retrieval and computer vision

Lu Pengyu: born in 1998. PhD candidate. Her main research interests include machine learning and artificial neural network. (periispengyu@163.com)

Zhao Zhijie: born in 1999. Master candidate. Her main research interests include multimedia retrieval and computer vision. (zhaozhijie_01@163.com)

Nie Xiushan: born in 1981. PhD, professor. His main research interests include multimedia retrieval, multimedia security, and computer vision. (niexiushan@163.com)

Wang Shaohua: born in 1979. PhD, senior engineer. Member of CCF. Her main research interests include multimedia retrieval and computer vision. (wsh@sdjzu.edu.cn)

Yin Yilong: born in 1972. PhD, professor, PhD supervisor. His main research interests include machine learning and data mining. (ylyin@sdu.edu.cn)
Received Date: June 04, 2023
Revised Date: December 18, 2023
Available Online: May 26, 2024

Graphical Abstract

Abstract

Abstract

Online cross-modal hashing has received increasing attention due to its efficiency and effectiveness in handling cross-modal streaming data retrieval. Despite promising progress, most existing methods rely on accurate and clear supervised information. There are only a few online cross-modal hashing studies concentrating on unsupervised learning mode, and numerous challenges still need to be tackled. For example, streaming data usually suffer from the unbalanced distribution problem due to the limited volume of data in each chunk. Most existing methods neglect this problem, resulting in heightened sensitivity to outlier samples and compromised robustness. Moreover, existing models typically exploit global data distribution, while ignoring local neighborhood information that can promote hash learning. To solve these problems, we propose an unsupervised online cross-modal hashing method with double structure-preserving, called SPOCH（structure preserving online cross-modal hashing）. It simultaneously explores the global structure and local structure to generate the corresponding common representation; thereafter, the learned common representation can be used to guide the hash learning process. In terms of global structure-preserving, we design the loss function based on ${L}_{2,1}$ norm, which can alleviate the sensitivity of outlier samples. In terms of local structure-preserving, we reconstruct sample representation based on neighbor relations that integrates the multi-modality information. In addition, to alleviate the forgetting problem, we propose joint optimization on streaming data, and design the corresponding update strategy to improve the training efficiency. We conduct experiments on two widely-used cross-modal retrieval datasets. Compared with the existing state-of-the-art unsupervised online cross-modal hashing methods, SPOCH achieves superior retrieval accuracy within a comparable or even shorter training time, validating the effectiveness of the proposed approach.
- cross-modal retrieval,
- unsupervised learning,
- online hashing,
- double structure preserving,
- robustness

FullText(HTML)

References (34)

References

[1]	Xu Xing, Shen Fumin, Yang Yang, et al. Learning discriminative binary codes for large-scale cross-modal retrieval[J]. IEEE Transactions on Image Processing, 2017, 26(5): 2494−2507 doi: 10.1109/TIP.2017.2676345
[2]	Raginsky M, Lazebnik S. Locality-sensitive binary codes from shift-invariant kernels[C]//Proc of the 22nd Int Conf on Neural Information Processing Systems. New York: ACM, 2009, 1509–1517
[3]	刘兴波,聂秀山,尹义龙. 基于双向线性回归的监督离散跨模态散列方法[J]. 计算机研究与发展,2020,57(8):1707−1714 doi: 10.7544/issn1000-1239.2020.20200122 Liu Xingbo, Nie Xiushan, Yin Yilong. Supervised discrete cross-modal hashing method based on two-way linear regression[J]. Journal of Computer Research and Development, 2020, 57(8): 1707−1714 (in Chinese) doi: 10.7544/issn1000-1239.2020.20200122
[4]	Zheng Chaoqun, Zhu Lei, Zhang Zheng, et al. Efficient semi-supervised multimodal hashing with importance differentiation regression[J]. IEEE Transactions on Image Processing, 2022, 31: 5881−5892 doi: 10.1109/TIP.2022.3203216
[5]	卓君宝,苏驰,王树徽,等. 最小熵迁移对抗散列方法[J]. 计算机研究与发展,2020,57(4):888−896 doi: 10.7544/issn1000-1239.2020.20190476 Zhuo Junbao, Su Chi, Wang Shuhui, et al. Least entropy transfer against hashing method[J]. Journal of Computer Research and Development, 2020, 57(4): 888−896 (in Chinese) doi: 10.7544/issn1000-1239.2020.20190476
[6]	Lin Zijia, Ding Guiguang, Hu Mingqing, et al. Semantics-preserving hashing for cross-view retrieval[C]// Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2015: 3864−3872
[7]	Xie Liang, Shen Jialie, Zhu Lei, et al. Online cross-modal hashing for web image retrieval[C]// Proc of the 30th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2016: 294−300
[8]	Wang Di, Wang Quan, An Yaqiang, et al. Online collective matrix factorization hashing for large-scale cross-media retrieval[C]// Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 1409−1418
[9]	Li Xuan, Wu Wei, Yuan Yunhao, et al. Online unsupervised cross-view discrete hashing for large-scale retrieval[J]. Applied Intelligence, 2022, 52(13): 14905−14917 doi: 10.1007/s10489-021-03014-w
[10]	Yao Tao, Wang Gang, Yan Lianshan, et al. Online latent semantic hashing for cross-media retrieval[J]. Pattern Recognition, 2019, 89: 1−11 doi: 10.1016/j.patcog.2018.12.012
[11]	Wang Yongxin, Luo Xin, Xu Xinshun, et al. Label embedding online hashing for cross-modal retrieval[C]// Proc of the 28th ACM Int Conf on Multimedia. New York: ACM, 2020: 871−879
[12]	Zhan Yuwei, Wang Yongxin, Sun Yu, et al. Discrete online cross-modal hashing[J]. Pattern Recognition, 2022, 122: 108262 doi: 10.1016/j.patcog.2021.108262
[13]	Lin Mingbao, Ji Rongrong, Liu Hong, et al. Towards optimal discrete online hashing with balanced similarity[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 8722−8729
[14]	Cakir F, Sclaroff S. Online supervised hashing[C]// Proc of the IEEE Int Conf on Image Processing. Piscataway, NJ: IEEE, 2015: 2606−2610
[15]	Lin Mingbao, Ji Rongrong, Liu Hong, et al. Hadamard matrix guided online hashing[J]. International Journal of Computer Vision, 2020, 128(8): 2279−2306
[16]	Lin Mingbao, Ji Rongrong, Chen Shen, et al. Similarity-preserving linkage hashing for online image retrieval[J]. IEEE Transactions on Image Processing, 2020, 29: 5289−5300 doi: 10.1109/TIP.2020.2981879
[17]	Lin Mingbao, Ji Rongrong, Sun Xiaoshuai, et al. Fast class-wise updating for online hashing[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(5): 2453−2467
[18]	Fang Yuzhi, Liu Li. Scalable supervised online hashing for image retrieval[J]. Journal of Computational Design and Engineering, 2021, 8(5): 1391−1406 doi: 10.1093/jcde/qwab052
[19]	Hao Yanbin, Mu Tingting, Goulermas J, et al. Unsupervised t-distributed video hashing and its deep hashing extension[J]. IEEE Transactions on Image Processing, 2017, 26(11): 5531−5544 doi: 10.1109/TIP.2017.2737329
[20]	Huang Longkai, Yang Qiang, Zheng Weishi. Online hashing[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(6): 2309−2322 doi: 10.1109/TNNLS.2017.2689242
[21]	Leng Cong, Wu Jiaxiang, Cheng Jian, et al. Online sketching hashing[C]// Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE Computer Society, 2015: 2503−2511
[22]	Chen Xixian, King I, Lyu M. FROSH: Faster online sketching hashing[C/OL]// Proc of the 30th Conf on Uncertainty in Artificial Intelligence. Corvallis: AUAI, 2017[2023-11-21]. http://auai.org/uai2017/proceedings/papers/12.pdf
[23]	Chen Xixian, Yang Haiqin, Zhao Shenglin, et al. Making online sketching hashing even faster[J]. IEEE Transactions on Knowledge and Data Engineering, 2021, 33(3): 1089−1101
[24]	Ding Guiguang, Guo Yuchen, Zhou Jile. Collective matrix factorization hashing for multimodal data[C]//Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2014: 2083−2090
[25]	Sun Yuan, Wang Xu , Peng Dezhong, et al. Hierarchical hashing learning for image set classification[J]. IEEE Transactions on Image Processing, 2023, 32: 1732−1744
[26]	Hoerl A, Kennard R. Ridge regression: Applications to nonorthogonal problems[J]. Technometrics, 1970, 12(1): 69−82 doi: 10.1080/00401706.1970.10488635
[27]	Roweis T, Saul K. Nonlinear dimensionality reduction by locally linear embedding[J]. Science, 2000, 290(5500): 2323−2326 doi: 10.1126/science.290.5500.2323
[28]	He Xiaofei, Cai Deng, Yan Shuicheng, et al. Neighborhood preserving embedding[C]// Proc of the 10th IEEE Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2005: 1208−1213
[29]	Escalante H, Hernandez C, Gonzalez J, et al. The segmented and annotated IAPRTC-12 benchmark[J]. Computer Vision and Image Understanding, 2010, 144(4): 419−428
[30]	Huiskes M , Lew M. The MIR flickr retrieval evaluation[C]// Proc of the 1st ACM SIGMM Int Conf on Multimedia Information Retrieval. New York: ACM, 2008: 9−43
[31]	Wang Di, Wang Quan, Gao Xinbo. Robust and flexible discrete hashing for cross-modal similarity search[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28(10): 2703−2715 doi: 10.1109/TCSVT.2017.2723302
[32]	Wang Di, Wang Quan, He Lihuo, et al. Joint and individual matrix factorization hashing for large-scale cross-modal retrieval[J]. Pattern Recognition, 2020, 107: 107479 doi: 10.1016/j.patcog.2020.107479
[33]	Wang, Di, Caiping Zhang, Quan Wang, et al. Hierarchical semantic structure preserving hashing for cross-modal retrieval[J]. IEEE Transactions on Multimedia, 2023, 25: 1217−1229 doi: 10.1109/TMM.2022.3140656
[34]	Liu Xin, Yi Jinhan, Cheung Yiuming, et al. OMGHUN: Online manifold-guided hashing for flexible cross-modal retrieval[J]. IEEE Transactions on Multimedia, 2023, 25: 3811−3824 doi: 10.1109/TMM.2022.3166668

[1]	Zhou Yuanding, Gao Guopeng, Fang Yaodong, Qin Chuan. Perceptual Authentication Hashing with Image Feature Fusion Based on Window Self-Attention[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202330669
[2]	Gao Wei, Chen Liqun, Tang Chunming, Zhang Guoyan, Li Fei. One-Time Chameleon Hash Function and Its Application in Redactable Blockchain[J]. Journal of Computer Research and Development, 2021, 58(10): 2310-2318. DOI: 10.7544/issn1000-1239.2021.20210653
[3]	Wu Linyang, Luo Rong, Guo Xueting, Guo Qi. Partitioning Acceleration Between CPU and DRAM: A Case Study on Accelerating Hash Joins in the Big Data Era[J]. Journal of Computer Research and Development, 2018, 55(2): 289-304. DOI: 10.7544/issn1000-1239.2018.20170842
[4]	Jiang Jie, Yang Tong, Zhang Mengyu, Dai Yafei, Huang Liang, Zheng Lianqing. DCuckoo: An Efficient Hash Table with On-Chip Summary[J]. Journal of Computer Research and Development, 2017, 54(11): 2508-2515. DOI: 10.7544/issn1000-1239.2017.20160795
[5]	Wang Wendi, Tang Wen, Duan Bo, Zhang Chunming, Zhang Peiheng, Sun Ninghui. Parallel Accelerator Design for High-Throughput DNA Sequence Alignment with Hash-Index[J]. Journal of Computer Research and Development, 2013, 50(11): 2463-2471.
[6]	Yuan Xinpan, Long Jun, Zhang Zuping, Luo Yueyi, Zhang Hao, and Gui Weihua. Connected Bit Minwise Hashing[J]. Journal of Computer Research and Development, 2013, 50(4): 883-890.
[7]	Qin Chuan, Chang Chin Chen, Guo Cheng. Perceptual Robust Image Hashing Scheme Based on Secret Sharing[J]. Journal of Computer Research and Development, 2012, 49(8): 1690-1698.
[8]	Ding Zhenhua, Li Jintao, Feng Bo. Research on Hash-Based RFID Security Authentication Protocol[J]. Journal of Computer Research and Development, 2009, 46(4): 583-592.
[9]	Li Zhiqiang, Chen Hanwu, Xu Baowen, Liu Wenjie. Fast Algorithms for Synthesis of Quantum Reversible Logic Circuits Based on Hash Table[J]. Journal of Computer Research and Development, 2008, 45(12): 2162-2171.
[10]	Liu Ji. One-Way Hash Function based on Integer Coupled Tent Maps and Its Performance Analysis[J]. Journal of Computer Research and Development, 2008, 45(3): 563-569.