![]() You can access this spark context UI till your spark shell is open.Storage Memory it took maximum 434.4 MB.When cluster created we did not pass number of thread so it took default number as 8 based on my laptop hardware resource available.JVM is a combination of driver and executer. We are not seeing separate executer process, because we are in local cluster.Check Event Timeline Spark started and executed driver process.To monitor and investigate your spark application you can check spark context web UI. □ option("multiline","true") is important if you have JSON with multiple lines formated by prettier or any other formatterĪnalyzing Spark Jobs using Spark Context Web UI in your MAC laptop ![]() json ( "/Users/rupeshti/workdir/git-box/learning-apache-spark/src/test.json" ) df.show () # read and display json file with multiline formated json like I have in my example df = ( "multiline", "true" ). You will learn about spark shell, local cluster, driver, executor and Spark Context UI. Running small program with PySpark Shell in your MAC laptop Now run pyspark to see the spark shell in python. Step 4: Running PySpark shell in your MAC laptop
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |