reduceByKey(func) converts a dataset of (K, V) pairs, into a dataset of (K, V) pairs where the values for each key are aggregated using the given reduce function.
mapPartitions and mapPartitionsWithIndex performs a map operation on an entire partition and returns a new RDD.
As per Apache Spark documentation, groupBy returns an RDD of grouped items where each group consists of a key and a sequence of elements.
groupByKey([numPartitions]) is called on a dataset of (K, V) pairs, and returns a dataset of (K, Iterable) pairs.
As per Apache Spark, filter(function) returns a new dataset formed by selecting those elements of the source on which function returns true.
flatMap(func) is similar to map, but each input item can be mapped to 0 or more output items The func should return a scala.collection.Seq
map(func) transformation returns a new distributed dataset formed by passing each element of the source through a function func.
In this post we will use textFile and wholeTextFiles in Apache Spark to read a single and multiple text files into a single Spark RDD.
We can create empty RDD in three ways using SparkContext's emptyRDD method, parallelize method with RDD of type String and using pair RDD.
Spark RDD can be created from local collection, from a text file or reading from database or by calling transformation on existing RDD.
In this article we will dive into the basic concept of broadcast variables. On a very high level broadcast variable is Spark shared variable.
Hi All, In this post I will tell you How To Install Spark And Pyspark On Centos.
In this video we will understand how to work with AVRO data in Apache Spark.For the demo we are using Spark 2.4 version and scala language.
In this video we will understand how to work with DataFrame Columns in Apache Spark.
In this video we will learn how to work with JSON data in Apache Spark.
In this video we will understand DataFrame abstraction in Spark.
How to setup Spark 2.4 cluster on Google Cloud using Dataproc. Step1 - Create a new project , Step2 - Create a new Cluster using Dataproc.
In this post I will tell you how to install Apache Spark on windows machine. By the end of this tutorial you’ll be able to use Spark with Scala on windows.