rdd

Type Members

1. case class MergeJoinPartition[K, V, W](index: Int, left: Dependency[(K, V)], right: Dependency[(K, W)]) extends Partition with Product with Serializable

   

   :: @DeveloperApi ::

   :: @DeveloperApi ::

   Stores partition index and references to left and right partitions to be joined

   index

   The partition index

   left

   The left Dependency to be used in the join

   right

   The right Dependency to be used in the join

   Attributes

   protected

2. class MergeJoinRDD[K, V, W, Out] extends RDD[Out]

   

   :: @DeveloperApi ::

   :: @DeveloperApi ::

   RDD implementation for merge-join that uses a shuffle to partition and sort by keys using an implicit Ordering for K, and then delegates to an instance of MergeJoin to perform the actual merge logic.

   There is an optimization in place to avoid a shuffle in some cases where left or right are guaranteed to be partition-sorted already (ie: via repartitionAndSortWithinPartitions)

   Annotations

   @DeveloperApi()

3. class PairRDDFunctions[K, V] extends Serializable

   

   Merge-join operators that provide scalable equivalents to the existing Spark RDD join, leftOuterJoin, rightOuterJoin, fullOuterJoin operators.

   Merge-join operators that provide scalable equivalents to the existing Spark RDD join, leftOuterJoin, rightOuterJoin, fullOuterJoin operators.

   Refer to the documentation for MergeJoin for implementation details.