sempy_labs.spark package

Module contents

sempy_labs.spark.create_custom_pool(pool_name: str, node_size: str, min_node_count: int, max_node_count: int, min_executors: int, max_executors: int, node_family: str = 'MemoryOptimized', auto_scale_enabled: bool = True, dynamic_executor_allocation_enabled: bool = True, workspace: str | UUID | None = None)

Creates a custom pool within a workspace.

This is a wrapper function for the following API: Custom Pools - Create Workspace Custom Pool.

Parameters:

  • pool_name (str) – The custom pool name.

  • node_size (str) – The node size.

  • min_node_count (int) – The minimum node count.

  • max_node_count (int) – The maximum node count.

  • min_executors (int) – The minimum executors.

  • max_executors (int) – The maximum executors.

  • node_family (str, default='MemoryOptimized') – The node family.

  • auto_scale_enabled (bool, default=True) – The status of auto scale.

  • dynamic_executor_allocation_enabled (bool, default=True) – The status of the dynamic executor allocation.

  • workspace (str | uuid.UUID, default=None) – The name or ID of the Fabric workspace. Defaults to None which resolves to the workspace of the attached lakehouse or if no lakehouse attached, resolves to the workspace of the notebook.

sempy_labs.spark.delete_custom_pool(pool_name: str, workspace: str | UUID | None = None)

Deletes a custom pool within a workspace.

This is a wrapper function for the following API: Custom Pools - Delete Workspace Custom Pool.

Parameters:

  • pool_name (str) – The custom pool name.

  • workspace (str | uuid.UUID, default=None) – The name or ID of the Fabric workspace. Defaults to None which resolves to the workspace of the attached lakehouse or if no lakehouse attached, resolves to the workspace of the notebook.

sempy_labs.spark.get_spark_settings(workspace: str | UUID | None = None, return_dataframe: bool = True) → DataFrame | dict

Shows the spark settings for a workspace.

This is a wrapper function for the following API: Workspace Settings - Get Spark Settings.

Parameters:

  • workspace (str | uuid.UUID, default=None) – The name or ID of the Fabric workspace. Defaults to None which resolves to the workspace of the attached lakehouse or if no lakehouse attached, resolves to the workspace of the notebook.

  • return_dataframe (bool, default=True) – If True, returns a pandas dataframe. If False, returns a json dictionary.

Returns:

A pandas dataframe showing the spark settings for a workspace.

Return type:

pandas.DataFrame | dict

sempy_labs.spark.list_custom_pools(workspace: str | UUID | None = None) → DataFrame

Lists all custom pools within a workspace.

This is a wrapper function for the following API: Custom Pools - List Workspace Custom Pools.

Parameters:

workspace (str | uuid.UUID, default=None) – The name of the Fabric workspace. Defaults to None which resolves to the workspace of the attached lakehouse or if no lakehouse attached, resolves to the workspace of the notebook.

Returns:

A pandas dataframe showing all the custom pools within the Fabric workspace.

Return type:

pandas.DataFrame

sempy_labs.spark.update_custom_pool(pool_name: str, node_size: str | None = None, min_node_count: int | None = None, max_node_count: int | None = None, min_executors: int | None = None, max_executors: int | None = None, node_family: str | None = None, auto_scale_enabled: bool | None = None, dynamic_executor_allocation_enabled: bool | None = None, workspace: str | UUID | None = None)

Updates the properties of a custom pool within a workspace.

This is a wrapper function for the following API: Custom Pools - Update Workspace Custom Pool.

Parameters:

  • pool_name (str) – The custom pool name.

  • node_size (str, default=None) –

    The node size. Defaults to None which keeps the existing property setting.

  • min_node_count (int, default=None) –

    The minimum node count. Defaults to None which keeps the existing property setting.

  • max_node_count (int, default=None) – The maximum node count. Defaults to None which keeps the existing property setting.

  • min_executors (int, default=None) –

    The minimum executors. Defaults to None which keeps the existing property setting.

  • max_executors (int, default=None) – The maximum executors. Defaults to None which keeps the existing property setting.

  • node_family (str, default=None) –

    The node family. Defaults to None which keeps the existing property setting.

  • auto_scale_enabled (bool, default=None) –

    The status of auto scale. Defaults to None which keeps the existing property setting.

  • dynamic_executor_allocation_enabled (bool, default=None) –

    The status of the dynamic executor allocation. Defaults to None which keeps the existing property setting.

  • workspace (str | uuid.UUID, default=None) – The name or ID of the Fabric workspace. Defaults to None which resolves to the workspace of the attached lakehouse or if no lakehouse attached, resolves to the workspace of the notebook.

sempy_labs.spark.update_spark_settings(automatic_log_enabled: bool | None = None, high_concurrency_enabled: bool | None = None, customize_compute_enabled: bool | None = None, default_pool_name: str | None = None, max_node_count: int | None = None, max_executors: int | None = None, environment_name: str | None = None, runtime_version: str | None = None, workspace: str | UUID | None = None)

Updates the spark settings for a workspace.

This is a wrapper function for the following API: Workspace Settings - Update Spark Settings.

Parameters:

  • automatic_log_enabled (bool, default=None) – The status of the automatic log. Defaults to None which keeps the existing property setting.

  • high_concurrency_enabled (bool, default=None) – The status of the high concurrency for notebook interactive run. Defaults to None which keeps the existing property setting.

  • customize_compute_enabled (bool, default=None) – Customize compute configurations for items. Defaults to None which keeps the existing property setting.

  • default_pool_name (str, default=None) – Default pool for workspace. Defaults to None which keeps the existing property setting.

  • max_node_count (int, default=None) – The maximum node count. Defaults to None which keeps the existing property setting.

  • max_executors (int, default=None) – The maximum executors. Defaults to None which keeps the existing property setting.

  • environment_name (str, default=None) – The name of the default environment. Empty string indicated there is no workspace default environment Defaults to None which keeps the existing property setting.

  • runtime_version (str, default=None) – The runtime version. Defaults to None which keeps the existing property setting.

  • workspace (str | uuid.UUID, default=None) – The name or ID of the Fabric workspace. Defaults to None which resolves to the workspace of the attached lakehouse or if no lakehouse attached, resolves to the workspace of the notebook.