butterfree.migrations.database_migration package

Submodules

Cassandra Migration entity.

class butterfree.migrations.database_migration.cassandra_migration.CassandraMigration

Bases: DatabaseMigration

Cassandra class for performing migrations.

This class implements some methods of the parent DatabaseMigration class and has specific methods for query building.

The CassandraMigration class will be used, as the name suggests, for applying changes to a given Cassandra table. There are, however, some remarks that need to be highlighted:

• If an existing feature has its type changed, then it’s extremely important to

make sure that this conversion would not result in data loss; - If new features are added to your feature set, then they’re going to be added to the corresponding Cassandra table; - Since feature sets can be written both to a feature set and an entity table, we’re not going to automatically drop features when using entity tables, since it means that some features belong to a different feature set. In summary, if data is being loaded into an entity table, then users can drop columns manually.

Migration entity.

class butterfree.migrations.database_migration.database_migration.DatabaseMigration(client: AbstractClient)

Bases: ABC

Abstract base class for Migrations.

apply_migration(feature_set: FeatureSet, writer: Writer, debug_mode: bool) → None

Apply the migration in the respective database.

Parameters:

• feature_set – the feature set.

• writer – the writer being used to load the feature set.

• debug_mode – if active, it brings up the queries generated.

create_query(fs_schema: List[Dict[str, Any]], table_name: str, db_schema: Optional[List[Dict[str, Any]]] = None, write_on_entity: Optional[bool] = None) → Any

Create a query regarding a data source.

Returns:

The desired queries for the given database.

class butterfree.migrations.database_migration.database_migration.Diff(column: str, kind: Kind, value: Any)

Bases: object

DataClass to help identifying different types of diff between schemas.

class Kind(value)

Bases: Enum

Mapping actions to take given a difference between columns of a schema.

ADD = 1

ALTER_KEY = 2

ALTER_TYPE = 3

DROP = 4

column: str

kind: Kind

value: Any

Metastore Migration entity.

class butterfree.migrations.database_migration.metastore_migration.MetastoreMigration(database: Optional[str] = None)

Bases: DatabaseMigration

MetastoreMigration class for performing migrations.

This class implements some methods of the parent DatabaseMigration class and has specific methods for query building. The MetastoreMigration class will be used, as the name suggests, for applying changes to a given Metastore table. There are, however, some remarks that need to be highlighted:

• If an existing feature has its type changed, then it’s extremely important to

make sure that this conversion would not result in data loss; - If new features are added to your feature set, then they’re going to be added to the corresponding Metastore table; - Since feature sets can be written both to a feature set and an entity table, we’re not going to automatically drop features when using entity tables, since it means that some features belong to a different feature set. In summary, if data is being loaded into an entity table, then users can drop columns manually.

Module contents

Holds available database migrations.

class butterfree.migrations.database_migration.CassandraMigration

Bases: DatabaseMigration

Cassandra class for performing migrations.

This class implements some methods of the parent DatabaseMigration class and has specific methods for query building.

The CassandraMigration class will be used, as the name suggests, for applying changes to a given Cassandra table. There are, however, some remarks that need to be highlighted:

• If an existing feature has its type changed, then it’s extremely important to

make sure that this conversion would not result in data loss; - If new features are added to your feature set, then they’re going to be added to the corresponding Cassandra table; - Since feature sets can be written both to a feature set and an entity table, we’re not going to automatically drop features when using entity tables, since it means that some features belong to a different feature set. In summary, if data is being loaded into an entity table, then users can drop columns manually.

class butterfree.migrations.database_migration.Diff(column: str, kind: Kind, value: Any)

Bases: object

DataClass to help identifying different types of diff between schemas.

class Kind(value)

Bases: Enum

Mapping actions to take given a difference between columns of a schema.

ADD = 1

ALTER_KEY = 2

ALTER_TYPE = 3

DROP = 4

column: str

kind: Kind

value: Any

class butterfree.migrations.database_migration.MetastoreMigration(database: Optional[str] = None)

Bases: DatabaseMigration

MetastoreMigration class for performing migrations.

This class implements some methods of the parent DatabaseMigration class and has specific methods for query building. The MetastoreMigration class will be used, as the name suggests, for applying changes to a given Metastore table. There are, however, some remarks that need to be highlighted:

• If an existing feature has its type changed, then it’s extremely important to

make sure that this conversion would not result in data loss; - If new features are added to your feature set, then they’re going to be added to the corresponding Metastore table; - Since feature sets can be written both to a feature set and an entity table, we’re not going to automatically drop features when using entity tables, since it means that some features belong to a different feature set. In summary, if data is being loaded into an entity table, then users can drop columns manually.