Write migration

Writing a migration

To have more information about migrations in laputa, check the topic page

Disclaimer

Warning

Migrations can be dangerous, an error will lead to Laputa not booting up (and being stuck in a restart loop in prod). They can also fail silently if the data modified is not correct and can lead to further issues on runtime. Test your migrations carefully (both local and on a guru) and follow the checklist before merging a new Laputa migration.

How to create a new migration

Migrations are python files located in the laputa/app/migrations directory. They have to respect a specific format:

Filename

The filename is composed of (each part is split with a _): - migration prefix - current date when creating the file with format yyyy_MM_dd_HH_mm_ss - the name of migration

For example migration_2022_06_16_10_00_00_migration_name.py

API

The code inside the file must implement the following API:

def apply_forward(params: dict) -> None:
  ...

def apply_backward(params: dict) -> None:
  ...

The params dict given as a parameter only contains two keys: config and database_manager

Sometimes the apply_backward() method is not applicable, you must still define it but leave it empty

How to update the current configs

To migrate Front and ETL configs, you can use existing utils to make it easier.

apply_front_config_migration(params, transform, migration_name)
apply_etl_config_migration(params, transform, migration_name)

These functions take the params, a callback to a function to migrate the config, and the name of the migration They will call the callback function with the current config and handle the save for you with its returned value.

Example:

from laputa.app.migrations.front_config_migrations import apply_front_config_migration

def transform_forward(front_config) -> dict:
    # add logic to update your front_config here
    return front_config


def apply_forward(params):
    apply_front_config_migration(params, transform_forward, 'Migrate my Front Config')

Migrating something else

You might want to migrate some other data, you can do it as you would do in Laputa

Warning

Avoid using ORMs and production models here as they validate the data and might lead to exceptions

You can use the database_manager given in the params argument, a lot of utils are already existing inside the database_manager to do these queries Otherwise, you can retrieve the collection to query the database using PyMongo using database_manager.get_small_app_database and database_manager.get_default_database

Example

def apply_forward(params):  
    database_manager = params['database_manager']
    # data = database_manager.get_small_app_database('demo')['styles'] <- same as below
    data = database_manager.get_small_app_styles_variables('demo')
    # logic to update the data
    database_manager.save_small_app_styles_variables('demo', data)

Complex migrations

You can always use Pydantic models to validate your data and their types and simplify the migration process

You can find an example at laputa/app/migrations/migration_2021_11_16_14_00_00_highlighted_kpi.py

Do not forget

• Try to test every possible case scenario in your unit testing
• It might be a good idea to globally catch exceptions if the changes of the migrations are not breaking
• Make sure the Pull Requests are reviewed carefully and do not hesitate to ask for it