How To Run Spark Submit Command?
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
To run a Spark Submit command, you will need to open a command prompt on your computer and type in the command “spark-submit <path-to-your-program>.jar”. You can also specify additional parameters such as memory, number of cores, etc. to make the job run more efficiently.
1. Open a command line interface (CLI) on the master node.
2. If needed, set up the environment for your Hadoop or Spark cluster using the relevant bin/ directory in the Spark distribution. For example, use “bin/spark–env.sh“ for the Spark setup and “bin/hadoop–env.sh“ for the Hadoop setup.
3. Execute the following command to run your application: spark–submit —class <main–class> —master <master–url> <application–jar> [application–arguments] Where [application–arguments] are arguments you want to pass to the application, <main–class> is the main class of your application, and <master–url> is the URL of the master node.
4. For example, if the application‘s main class is MyApp and you want to pass two arguments arg1 and arg2 to it, execute the following command: spark–submit —class MyApp —master spark://<master–url>:7077 my–application.jar arg1 arg2