Spark document clearly specify that you can read gz
file automatically:
All of Spark’s file-based input methods, including textFile, support
running on directories, compressed files, and wildcards as well. For
example, you can use textFile("/my/directory"),
textFile("/my/directory/.txt"), and textFile("/my/directory/.gz").
I'd suggest running the following command, and see the result:
rdd = sc.textFile("data/label.gz")
print rdd.take(10)
Assuming that spark finds the the file data/label.gz
, it will print the 10 rows from the file.
Note, that the default location for a file like data/label.gz
will be in the hdfs folder of the spark-user. Is it there?
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…