butterfree.testing.dataframe package

Module contents

Methods to assert properties regarding Apache Spark Dataframes.

butterfree.testing.dataframe.assert_column_equality(output_df: DataFrame, target_df: DataFrame, output_column: Column, target_column: Column) → None

Columns comparison method.

butterfree.testing.dataframe.assert_dataframe_equality(output_df: DataFrame, target_df: DataFrame) → None

Dataframe comparison method.

butterfree.testing.dataframe.create_df_from_collection(data: List[Dict[Any, Any]], spark_context: SparkContext, spark_session: SparkSession, schema: Optional[StructType] = None) → DataFrame

Creates a dataframe from a list of dicts.