Scaling PageRank with R on Rescale - Rescale
pagerankparallel-computingr-languagempigraph-algorithmshpc
Abstraction: Parallelizing PageRank in R using MPI on cloud HPC platform
Key points:
- R is single-threaded and doesn't scale for large datasets; Rmpi (MPI wrapper for R) enables parallelization without switching to Hadoop/MapReduce
- PageRank update: PR(A) = (1-d) + d * sum(PR(Ti)/C(Ti)) — iterates until convergence (delta < epsilon)
- Parallelization strategy: distribute adjacency matrix rows via MPI scatterv; gather partial PageRank vectors with allgatherv after each iteration
- Test dataset: Stanford SNAP High Energy Physics Citation Network (34,546 nodes, 421,578 edges); sequential runtime 653.89 seconds
- Scaling gains taper off between 12 threads (86.7s) and 16 threads (68.4s) — diminishing returns from communication overhead
- Rescale is a cloud HPC platform enabling R/Rmpi jobs without local cluster infrastructure
Connections: Rescale · Google · Pagerank · Parallel Computing · Graph Algorithms
Source: http://blog.rescale.com/scaling-pagerank-with-r-on-rescale/