butterfree.testing.dataframe package

Module contents

Methods to assert properties regarding Apache Spark Dataframes.

butterfree.testing.dataframe.assert_column_equality(output_df: pyspark.sql.dataframe.DataFrame, target_df: pyspark.sql.dataframe.DataFrame, output_column: pyspark.sql.column.Column, target_column: pyspark.sql.column.Column)

Columns comparison method.

butterfree.testing.dataframe.assert_dataframe_equality(output_df: pyspark.sql.dataframe.DataFrame, target_df: pyspark.sql.dataframe.DataFrame)

Dataframe comparison method.

butterfree.testing.dataframe.create_df_from_collection(data: List[dict], spark_context: pyspark.context.SparkContext, spark_session: pyspark.sql.session.SparkSession, schema=None)

Creates a dataframe from a list of dicts.