Research on Generating Incremental ETL Processes Automatically

Zhang Xufeng, Sun Weiwei, Wang Wei, Feng Yahui, and Shi Baile

Journal of Computer Research and Development > 2006 > 43(6): 1097-1103.

Zhang Xufeng, Sun Weiwei, Wang Wei, Feng Yahui, and Shi Baile. Research on Generating Incremental ETL Processes Automatically[J]. Journal of Computer Research and Development, 2006, 43(6): 1097-1103.

Citation:

PDF (381 KB)

Research on Generating Incremental ETL Processes Automatically

Zhang Xufeng, Sun Weiwei, Wang Wei, Feng Yahui, and Shi Baile

(Department of Computing and Information Technology, Fudan University, Shanghai 200433)

More Information

Published Date: June 14, 2006

Graphical Abstract

Abstract

Abstract

ETL processes are used for collecting data from data sources to data warehouse. ETL processes can be separated into two portions: full ETL processes and increment ETL processes. A full ETL process can be designed easily but it can only deal full data. An incremental ETL process is used for loading only those data which are newly created in the data sources, but it is difficult to design manually. In this paper, using existing methods of incremental maintenance of materialized views for reference, an approach to generate an incremental ETL process automatically from a full ETL process is put forward. Existing researches are focused on the incremental maintenance of materialized views in such circumstances which involve the operators of selection, projection, join and aggregation but not the difference operators. Since difference operators are used frequently in an ETL process, incremental maintenance of materialized views defined with difference operators is also discussed in detail.
- ETL,
- data warehouse,
- incremental maintenance,
- materialized views,
- selfmaintenance

FullText(HTML)

References (0)

[1]	Zheng Wenping, Wu Zhikang, Yang Gui. A Novel Algorithm for Identifying Critical Nodes in Networks Based on Local Centrality[J]. Journal of Computer Research and Development, 2019, 56(9): 1872-1880. DOI: 10.7544/issn1000-1239.2019.20180831
[2]	Liu Haolin, Chi Jinlong, Deng Qingyong, Peng Xin, Pei Tingrui. Multi-Objective Evolutionary Sparse Recovery Approach Based on Adaptive Local Search[J]. Journal of Computer Research and Development, 2019, 56(7): 1420-1431. DOI: 10.7544/issn1000-1239.2019.20180557
[3]	Xu Zhengguo, Zheng Hui, He Liang, Yao Jiaqi. Self-Adaptive Clustering Based on Local Density by Descending Search[J]. Journal of Computer Research and Development, 2016, 53(8): 1719-1728. DOI: 10.7544/issn1000-1239.2016.20160136
[4]	Li Guilin, Yang Yuqi, Gao Xing, and Liao Minghong. Personalized Representation and Rank Algorithm for Enterprise Search Engines[J]. Journal of Computer Research and Development, 2014, 51(1): 206-214.
[5]	Li Shaohua, Feng Qilong, Wang Jianxin, and Chen Jianer. Kernelization for Weighted 3-Set Packing Problem[J]. Journal of Computer Research and Development, 2012, 49(8): 17811-786.
[6]	Li Wenjun, Wang Jianxin, and Chen Jianer. An Improved Parameterized Algorithm for Hyperplane-Cover Problem[J]. Journal of Computer Research and Development, 2012, 49(4): 804-811.
[7]	Wang Chuyang, Li Xiaoping, Wang Qian, Yuan Yingchun. A New Local Search Algorithm for No-Wait Fowshops with Setup Time[J]. Journal of Computer Research and Development, 2010, 47(4): 653-662.
[8]	Zhang Peng. Approximation Algorithms for Generalized Multicut in Trees[J]. Journal of Computer Research and Development, 2008, 45(7): 1195-1202.
[9]	Zeng Liping and Huang Wenqi. A New Local Search Algorithm for the Job Shop Scheduling Problem[J]. Journal of Computer Research and Development, 2005, 42(4): 582-587.
[10]	Yang Jinji, Su Kaile. Improvement of Local Research in SAT Problem[J]. Journal of Computer Research and Development, 2005, 42(1): 60-65.