Skip to main content

ascii_library.orchestration.pipes.spark_script_abc module

class ascii_library.orchestration.pipes.spark_script_abc.SparkScriptPipes​

Bases: ABC

abstractmethod execute_business_logic(context, execution_mode, partition_key, spark, engine)​

  • Parameters:
    • execution_mode (ExecutionMode)
    • partition_key (str | None)
    • spark (SparkSession)
    • engine (Engine)

get_base_path_seeds(execution_mode)​

  • Parameters: execution_mode (ExecutionMode)
  • Return type: str

get_base_path_IO(execution_mode)​

  • Parameters: execution_mode (ExecutionMode)
  • Return type: str

get_base_path_commoncrawl(execution_mode)​

  • Parameters: execution_mode (ExecutionMode)
  • Return type: str

__init__()​

get_loader_writer(engine)​