Part A - Spark RDDs and SQL with text (8 marks) In Part A your task is to answer a question about the data in a text file, first using Spark RDDs, and then using Spark DataFrames and SQL. By using...

Please see the attached document


Part A - Spark RDDs and SQL with text (8 marks) In Part A your task is to answer a question about the data in a text file, first using Spark RDDs, and then using Spark DataFrames and SQL. By using both to answer the same question about the same file you can more readily see how the two techniques compare. When you click the panel on the right you'll get a connection to a server that has, in your home directory, the text file "graph.txt". In this text file, each line is in the format of: “EdgeId FromNodeId ToNodeId Distance”. 0 0 1 10.0 1 0 2 5.0 2 1 2 2.0 3 1 3 1.0 4 2 1 3.0 5 2 3 9.0 6 2 4 2.0 7 3 4 4.0 8 4 0 7.0 9 4 3 5.0 (There may exist an empty line in the input file, and you can use a filter to solve this.) These data represent a graph as shown below: Given a directed graph, your task is to compute the average length of the out-going edges for each node, and sort the nodes first according to the average lengths in descending order and then according to the node IDs (numeric value) in ascending order. Your results should appear like the following (The result should be rounded to two decimal places.): 0 7.5 4 6.0 2 4.67 3 4.0 1 1.5 Each line in the single output file is in the format of “NodeID\tAverage length of out-going edges”, where '\t' indicates the 'tab' character. Please remove the nodes that have no out-going edges. First (4 marks) Write a Python program that uses Spark RDDs to do this. A file called "rdd.py" has been created for you - you just need to fill in the details. You should be able to modify programs that you have already seen in this week's content. To sort the RDD results, you can use SortBy, and here is an example of it. Hint: >>> tmp = [('a', 3), ('b', 2), ('a', 1), ('d', 4), ('2', 5)] >>> sc.parallelize(tmp).sortBy(lambda x: (x[0],x[1])).collect() Output: [('2', 5), ('a', 1), ('a', 3), ('b', 2), ('d', 4)] To test your program you first need to create your default directory in Hadoop, and copy walden.txt to it: $ hdfs dfs -mkdir -p /user/user $ hdfs dfs -put graph.txt You can test your program by running the following command: $ spark-submit rdd.py Please save your results in the 'result-rdd' folder in HDFS. Second (4 marks) Write a Python program that uses Spark DataFrames and SQL to do this. A file called "sql.py" has been created for you - you just need to fill in the details. Again, you should be able to modify programs that you have already seen in this week's content. You can test your program by running the following command: $ spark-submit sql.py Please save your results in the 'result-sql' folder in HDFS. Part B - Spark RDDs and SQL with CSV (8 marks) In Part B your task is to answer a question about the data in a CSV file, first using Spark RDDs, and then using Spark DataFrames and SQL. By using both to answer the same question about the same file you can more readily see how the two techniques compare. When you click the panel on the right you'll get a connection to a server that has, in your home directory, the CSV file "orders.csv". It's one that you've seen before. Here are the fields in the file: OrderDate (date) ISBN (string) Title (string) Category (string) PriceEach (decimal) Quantity (integer) FirstName (string) LastName (string) City (string) Your task is to find the number of books ordered each day, sorted by the number of books descending, then order date ascending. Your results should appear as the following: 2009-04-03,10 2009-04-02,8 2009-04-01,7 2009-04-04,6 2009-03-31,5 2009-04-05,4 2009-04-08,4 First (4 marks) Write a Python program that uses Spark RDDs to do this. A file called "rdd.py" has been created for you - you just need to fill in the details. You should be able to modify programs that you have already seen in this week's content. To sort the RDD results, you can use SortBy, and here is an example of it. Hint: >>> tmp = [('a', 3), ('b', 2), ('a', 1), ('d', 4), ('2', 5)] >>> sc.parallelize(tmp).sortBy(lambda x: (x[0],x[1])).collect() Output: [('2', 5), ('a', 1), ('a', 3), ('b', 2), ('d', 4)] To test your program you first need to create your default directory in Hadoop, and copy orders.csv to it: $ hdfs dfs -mkdir -p /user/user $ hdfs dfs -put orders.csv You can test your program by running the following command: $ spark-submit rdd.py Please save your results in the 'result-rdd' folder in HDFS. Second (4 marks) Write a Python program that uses Spark DataFrames and SQL to do this. A file called "sql.py" has been created for you - you just need to fill in the details. Again, you should be able to modify programs that you have already seen in this week's content. You can test your program by running the following command: $ spark-submit sql.py Please save your results in the 'result-sql' folder in HDFS. 0 0 1 10.0 1 0 2 5.0 2 1 2 2.0 3 1 3 1.0 4 2 1 3.0 5 2 3 9.0 6 2 4 2.0 7 3 4 4.0 8 4 0 7.0 9 4 3 5.0