schrodinger.seam.transforms.partitioners module¶

class schrodinger.seam.transforms.partitioners.FixedSample(n: int)¶

Bases: apache_beam.typehints.decorators.WithTypeHints, apache_beam.transforms.display.HasDisplayData, Generic[apache_beam.transforms.ptransform.InputT, apache_beam.transforms.ptransform.OutputT]

A PTransform that takes a PCollection and partitions it into two PCollections. The first PCollection is a random sample of the input PCollection, and the second PCollection is the remaining elements of the input PCollection.

This is useful for creating holdout / test sets in machine learning.

Example usage:

>>> with beam.Pipeline() as p:
...     sample, remaining = (p
...         | beam.Create(list(range(10)))
...         | FixedSample(3))
...     # sample will contain three randomly selected elements from the
...     # input PCollection
...     # remaining will contain the remaining seven elements

__init__(n: int)¶

expand(pcoll)¶

annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]¶

default_label() → str¶

default_type_hints()¶

display_data() → dict¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

Dict[str, Any]: A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

classmethod from_runner_api(proto: Optional[beam_runner_api_pb2.PTransform], context: PipelineContext) → Optional[PTransform]¶

get_resource_hints() → Dict[str, bytes]¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs: Any) → Windowing¶

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)¶

property label¶

pipeline: Optional[Pipeline] = None¶

classmethod register_urn(urn, parameter_type, constructor=None)¶

runner_api_requires_keyed_input()¶

side_inputs: Sequence[pvalue.AsSideInput] = ()¶

to_runner_api(context: PipelineContext, has_parts: bool = False, **extra_kwargs: Any) → beam_runner_api_pb2.FunctionSpec¶

to_runner_api_parameter(unused_context: PipelineContext) → Tuple[str, Optional[Union[message.Message, bytes, str]]]¶

to_runner_api_pickled(unused_context: PipelineContext) → Tuple[str, bytes]¶

type_check_inputs(pvalueish)¶

type_check_inputs_or_outputs(pvalueish, input_or_output)¶

type_check_outputs(pvalueish)¶

with_input_types(input_type_hint)¶

Annotates the input type of a PTransform with a type-hint.

Args:

input_type_hint (type): An instance of an allowed built-in type, a custom: class, or an instance of a TypeConstraint.

Raises:

TypeError: If input_type_hint is not a valid type-hint.: See apache_beam.typehints.typehints.validate_composite_type_param() for further details.

Returns:

PTransform: A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.

with_output_types(type_hint)¶

Annotates the output type of a PTransform with a type-hint.

Args:

type_hint (type): An instance of an allowed built-in type, a custom class,: or a TypeConstraint.

Raises:

TypeError: If type_hint is not a valid type-hint. See: validate_composite_type_param() for further details.

Returns:

PTransform: A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.

with_resource_hints(**kwargs) → apache_beam.transforms.ptransform.PTransform¶

Adds resource hints to the PTransform.

Resource hints allow users to express constraints on the environment where the transform should be executed. Interpretation of the resource hints is defined by Beam Runners. Runners may ignore the unsupported hints.

Args:

**kwargs: key-value pairs describing hints and their values.

Raises:

ValueError: if provided hints are unknown to the SDK. See: apache_beam.transforms.resources for a list of known hints.

Returns:

PTransform: A reference to the instance of this particular PTransform object.

class schrodinger.seam.transforms.partitioners.Top(n: int, key: Optional[Callable[[Any], Any]] = None, reverse=False)¶

Bases: apache_beam.typehints.decorators.WithTypeHints, apache_beam.transforms.display.HasDisplayData, Generic[apache_beam.transforms.ptransform.InputT, apache_beam.transforms.ptransform.OutputT]

A PTransform that takes a PCollection and partitions it into two PCollections. The first PCollection contains the largest n elements of the input PCollection, and the second PCollection contains the remaining elements of the input PCollection.

Parameters:

n: The number of elements to take from the input PCollection.
key: A function that takes an element of the input PCollection and returns
a value to compare for the purpose of determining the top n elements, similar to Python’s built-in sorted function.
reverse: If True, the top n elements will be the n smallest elements of the
input PCollection.

Example usage:

>>> with beam.Pipeline() as p:
...     top, remaining = (p
...         | beam.Create(list(range(10)))
...         | Top(3))
...     # top will contain [7, 8, 9]
...     # remaining will contain [0, 1, 2, 3, 4, 5, 6]

__init__(n: int, key: Optional[Callable[[Any], Any]] = None, reverse=False)¶

expand(pcoll)¶

annotations() → Dict[str, Union[bytes, str, google.protobuf.message.Message]]¶

default_label() → str¶

default_type_hints()¶

display_data() → dict¶

Returns the display data associated to a pipeline component.

It should be reimplemented in pipeline components that wish to have static display data.

Returns:

Dict[str, Any]: A dictionary containing key:value pairs. The value might be an integer, float or string value; a DisplayDataItem for values that have more data (e.g. short value, label, url); or a HasDisplayData instance that has more display data that should be picked up. For example:

{
  'key1': 'string_value',
  'key2': 1234,
  'key3': 3.14159265,
  'key4': DisplayDataItem('apache.org', url='http://apache.org'),
  'key5': subComponent
}

classmethod from_runner_api(proto: Optional[beam_runner_api_pb2.PTransform], context: PipelineContext) → Optional[PTransform]¶

get_resource_hints() → Dict[str, bytes]¶

get_type_hints()¶

Gets and/or initializes type hints for this object.

If type hints have not been set, attempts to initialize type hints in this order: - Using self.default_type_hints(). - Using self.__class__ type hints.

get_windowing(inputs: Any) → Windowing¶

Returns the window function to be associated with transform’s output.

By default most transforms just return the windowing function associated with the input PCollection (or the first input if several).

infer_output_type(unused_input_type)¶

property label¶

pipeline: Optional[Pipeline] = None¶

classmethod register_urn(urn, parameter_type, constructor=None)¶

runner_api_requires_keyed_input()¶

side_inputs: Sequence[pvalue.AsSideInput] = ()¶

to_runner_api(context: PipelineContext, has_parts: bool = False, **extra_kwargs: Any) → beam_runner_api_pb2.FunctionSpec¶

to_runner_api_parameter(unused_context: PipelineContext) → Tuple[str, Optional[Union[message.Message, bytes, str]]]¶

to_runner_api_pickled(unused_context: PipelineContext) → Tuple[str, bytes]¶

type_check_inputs(pvalueish)¶

type_check_inputs_or_outputs(pvalueish, input_or_output)¶

type_check_outputs(pvalueish)¶

with_input_types(input_type_hint)¶

Annotates the input type of a PTransform with a type-hint.

Args:

input_type_hint (type): An instance of an allowed built-in type, a custom: class, or an instance of a TypeConstraint.

Raises:

TypeError: If input_type_hint is not a valid type-hint.: See apache_beam.typehints.typehints.validate_composite_type_param() for further details.

Returns:

PTransform: A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.

with_output_types(type_hint)¶

Annotates the output type of a PTransform with a type-hint.

Args:

type_hint (type): An instance of an allowed built-in type, a custom class,: or a TypeConstraint.

Raises:

TypeError: If type_hint is not a valid type-hint. See: validate_composite_type_param() for further details.

Returns:

PTransform: A reference to the instance of this particular PTransform object. This allows chaining type-hinting related methods.

with_resource_hints(**kwargs) → apache_beam.transforms.ptransform.PTransform¶

Adds resource hints to the PTransform.

Resource hints allow users to express constraints on the environment where the transform should be executed. Interpretation of the resource hints is defined by Beam Runners. Runners may ignore the unsupported hints.

Args:

**kwargs: key-value pairs describing hints and their values.

Raises:

ValueError: if provided hints are unknown to the SDK. See: apache_beam.transforms.resources for a list of known hints.

Returns:

PTransform: A reference to the instance of this particular PTransform object.