How to print the contents of RDD in Apache Spark

Question

I want to output the contents of a collection to the Spark console.

I have a type:

linesWithSessionId: org.apache.spark.rdd.RDD[String] = FilteredRDD[3]
And I use the command:

scala> linesWithSessionId.map(line => println(line))
But this is printed :

res1: org.apache.spark.rdd.RDD[Unit] = MappedRDD[4] at map at :19

How can I write the RDD to console or save it to disk so I can view its contents?

Answer 1

If you want to view the content of a RDD, one way is to use collect():

myRDD.collect().foreach(println)
That's not a good idea, though, when the RDD has billions of lines. Use take() to take just a few to print out:

myRDD.take(n).foreach(println)

Hope this will help you.

Answer 2

• The map function is a transformation, which means that Spark will not actually evaluate your RDD until you run an action on it.
• To print it, you can use foreach (which is an action):​    linesWithSessionId.foreach(println)
• To write it to disk you can use one of the saveAs... functions (still actions) from the RDD API

Answer 3

Here's another way using session:

linesWithSessionId.toArray().foreach(line => println(line))

Answer 4

You can first convert it to dataframe and then print it:

line.toDF().show()

Answer 5

Save it to a text file:

line.saveAsTextFile("alicia.txt")

Print contains of the text file:

Answer 6

print (line.take(n))

Answer 7

Simple and easy:

line.foreach(println)

Answer 8

Hi,

You can follow a similar kind of approach as shown below to see the content of an RDD.

val dept = List(("Finance",10),("Marketing",20),("Sales",30), ("IT",40))
val rdd=spark.sparkContext.parallelize(dept)
val dataColl=rdd.collect()
dataColl.foreach(println)