Spark Submit is used to start the applications in the cluster, and it runs the same commands as the Spark Shell.

./bin/spark-submit \ --class <main-class> \ --master <master-url> \ --deploy-mode <deploy-mode> \ --conf <key>=<value> \ . # other options <application-jar> \ [application-arguments]
  • –class: Entry to the application
  • –master: master URL, which is the same as the Spark Shell
  • Application-jar: The jar package address
  • Application-arguments: Arguments to the application entry

The rest of the arguments are similar to the Spark Shell and can be viewed with $SPARK_HOME/bin/spark-submit –help. Here we use Spark’s own program to calculate Pi. See the website for similar examples

$SPARK_HOME/bin/spark-submit \ --class org.apache.spark.examples.SparkPi \ --master spark://bigdata01:7077,bigdata03:7077 \ --executor-memory 512m \ --total-executor-cores 2 \ $SPARK_HOME/examples/jars/spark - examples_2. 11-2.4.5. Jar \ 100

In the process of running a web interface, runtime by visit: http://bigdata01:4040/jobs

The running results are as follows:

http://bigdata01:8080/ You can see that two Applications have been Completed, and in Completed Applications you can also see the information about the run