Package org.apache.gravitino.job

Class SparkJobTemplate

java.lang.Object
org.apache.gravitino.job.JobTemplate
org.apache.gravitino.job.SparkJobTemplate
public class SparkJobTemplate extends JobTemplate
Represents a job template for executing Spark applications. This class extends the JobTemplate class and provides functionality specific to Spark job templates, including the class name, jars, files, archives, and configurations required for the Spark job.

Take Spark word count job as an example:

className: "org.apache.spark.examples.JavaWordCount" executable "https://example.com/spark-examples.jar" arguments: ["{{input_path}}", "{{output_path}}"] configs: {"spark.master": "local[*]", "spark.app.name": "WordCount"}

configs is a map of configuration parameters that will be used by the Spark application. It can be templated by using placeholders like "{{foo_value}}" and "{{bar_value}}". These placeholders will be replaced with actual values when the job is executed.

jars, files, and archives are lists of resources that will be used by the Spark application . These resources must be accessible to the Gravitino server, and can be located in the local file system, on a web server (e.g., HTTP, HTTPS, FTP). Distributed file systems like HDFS or S3 will be supported in the future.

  • Constructor Details

    • SparkJobTemplate

      protected SparkJobTemplate(SparkJobTemplate.Builder builder)
      Constructs a SparkJobTemplate with the specified builder.
      Parameters:
      builder - the builder containing the configuration for the Spark job template

  • Method Details

    • className

      public String className()
      Returns the class name of the Spark application to be executed.
      Returns:
      the class name

    • jars

      public List<String> jars()
      Returns the list of JAR files required for the Spark job.
      Returns:
      the list of JAR files

    • files

      public List<String> files()
      Returns the list of files required for the Spark job.
      Returns:
      the list of files

    • archives

      public List<String> archives()
      Returns the list of archives required for the Spark job.
      Returns:
      the list of archives

    • configs

      public Map<String,String> configs()
      Returns the configuration map for the Spark job.
      Returns:
      the configuration map

    • jobType

      public JobTemplate.JobType jobType()
      Description copied from class: JobTemplate
      Get the job type.
      Specified by:
      jobType in class JobTemplate
      Returns:
      the type of the job template

    • equals

      public boolean equals(Object o)
      Overrides:
      equals in class JobTemplate

    • hashCode

      public int hashCode()
      Overrides:
      hashCode in class JobTemplate

    • toString

      public String toString()
      Overrides:
      toString in class JobTemplate

    • builder

      public static SparkJobTemplate.Builder builder()
      Creates a new builder for constructing instances of SparkJobTemplate.
      Returns:
      a new instance of SparkJobTemplate.Builder