ascii_library.orchestration.pipes.utils module

ascii_library.orchestration.pipes.utils.library_to_cloud_paths(lib_name, filesystem='s3')

• Parameters:
  • lib_name (str)
  • filesystem (str)

ascii_library.orchestration.pipes.utils.library_from_dbfs_paths(dbfs_path)

• Parameters: dbfs_path (str)

ascii_library.orchestration.pipes.utils.empty_dir(path)

Ensure path exists and is empty. Removes files, dirs, and symlinks under it using pathlib only.

• Parameters: path (Path)
• Return type: None

ascii_library.orchestration.pipes.utils.ensure_empty(pathlike)

• Return type: Path

ascii_library.orchestration.pipes.utils.package_library(mylib_path)

ascii_library.orchestration.pipes.utils.get_input_path(io_nodes, part_seed, part_cc, lang)

ascii_library.orchestration.pipes.utils.calculate_parallelism(spark, input_path)

ascii_library.orchestration.pipes.utils.configure_spark(spark, execution_mode, compression_codec, default_parallelism, shuffle_partitions, partitionDiscovery_parallelism)