Abstract GPU-based node has emerged as a promising direction toward ef?cient large-scale graph processing, which is relied on the high computational power and scalable caching mechanisms of GPUs. Out-of-core graphs are the graphs that exceed main and GPU- resident memory capacity. To handle them, most existing systems using GPUs employ compact partitions of ?x-sized ordered edge sets (i.e., shards) for the data movement and computation. However, when scaling to platforms with multiple GPUs, these systems require high demand of interconnect (PCI-E) bandwidth. They suffer from GPU underutilization and represent scalability and performance bottlenecks. This paper presents GFlow, an ef?cient and scalable graph processing system to handle out-of-core graphs on multi-GPU nodes. In GFlow, we propose a novel 2-Level Streaming Windows method, which stores graph’s attribute data consecutively in shared memory of multi- GPUs, and then streams graph’s topology data (shards) to GPUs. With the novel 2-level streaming windows, GFlow streams shards dynamically from SSDs to GPU devices’ memories via PCI-E fabric and applies on-the-?y updates while processing graphs, thus reducing the amount of data movement required for computation. The detailed evaluations demonstrate that GFlow signi?cantly outperforms most other competing out-of-core systems for a wide variety of graphs and algorithms under multi-GPUs environment, i.e., yields average speedups of 25.6X and 20.3X over CPU-based GraphChi and X-Stream respectively, and 1.3-2.5X speedup against GPU-based GraphReduce (single-GPU). Meanwhile, GFlow represents excellent scalability as we increase the number of GPUs in the node.