Import export app

EXPORT AND IMPORT OF A V2 SMALL-APP¶

INTRODUCTION¶

As part of migrating our backend from a monolith to services, we need a way to export and import a small-app from version 2 (v2) to version 3 (v3). This will allow us to transfer all the necessary objects related to the small-app and import them into our dataset-service and layout-service in v3. In this doc, we will explore why the export and import functionality is crucial for this migration process.

WHY DO WE NEED EXPORT AND IMPORT?¶

Seamless Migration: The export and import functionality provides a seamless way to transfer a small-app from v2 to v3, ensuring a smooth migration process. It allows us to preserve the configurations and settings associated with the small-app during the transition.
Preserve Small-App Data Config: Small-apps often contain data that needs to be retained while migrating to the new backend architecture. Exporting the small-app ensures that all relevant data, such as user preferences, customizations, and configurations, are preserved.
Modularity and Reusability: By exporting a small-app, we can encapsulate its functionality, dependencies, and configurations into a single tarball file. This modular approach enables us to import the small-app on any v3 instance, enhancing reusability and backuping.
Version Control: Exporting a small-app provides an opportunity to assign it a version number or tag(today it's just the timestamp), allowing us to track changes and manage different versions effectively. This version control mechanism enables us to roll back or update the small-app as needed, ensuring flexibility and adaptability.

WHAT DO WE EXPORT AND WHAT WE DON'T (for now)¶

List of object we're extracting :

ETL config (/etl) The export process for this object is done easily, since, we have the util for it, but for the import process, we need to first insert the exported dict to mongo, then run the list of all migrations availables between v2 and v3 on it. this concerns "data-sources" and "data-providers".
Front config(/config) Just like the ETL config, we export this object using the database-manager, and import it by first having it on our mongo then running the list of migrations for all its "sub-objects"... queries, variables, layout... then import to the layout service.
assets (/assets) inside the tarball.
scheduled-actions
alerts
color-scheme
notification (email) themes templates
embeds
permissions
personal-reports

List of objects we don't export and why :

~Filters~ (code existing for mongo but not useed, so we don't export it on V2).
~datasets tags~ (in v2, it was just in POC phase for the datahub)
~comments~ (no need)
~shortened-contexts~ (no need)
~connection-secrets~ (will be manually imported)
~augment_py (needed while importing on v3, some process_input functions are saved to that collections and the code can vary depending on the small-app unfortunately, thanks V2 😬 )~ (we don't want to support this in v3, see this discussion on slack : https://toucantoco.slack.com/archives/C05NBMATSLF/p1702975684003869 ).

EXPORT PROCESS¶

The export process involves packaging all the required objects and assets with no data(for now) related to the small-app in v2 into a portable tarball that can be imported into v3. The source of truth is mongoDB. The following steps outline the export process:

Extraction of a v2 small app (the most optimized way possible, parallelism | concurrency...), ready to be imported on a v3 instance as a tarball (iteration mode for extraction because it's faster).
Cleaning an refacto of export instance (but keep the same behavior as before to not break existing feature.

NOTE: ⚠️IMPORTANT NOTE: THE SOURCE OF TRUTH IS MONGO not raw files, because from the codebase and what i saw, sometimes we have incoherence between .cson, .cson.cache, .json, .json.cache + nothing can assure us that those files really represent the current state of the app ?

HOW¶

extract from mongodb everything that can be extracted on /export-app/<small-app-uid> endpoint by a super-admin only.

DEMOS¶

REQUEST THE TARBALL AND DOWNLOAD IT¶

Extract objects for the given app and generate the tarball file

with a curl request such as :

curl -u <super-admin-username>:<super-admin-password> <API_HOST>/export-app/<small-app-id> -o exported.tar.gz

# This will download your app directly

For the video DEMO, see : https://toucantoco.slack.com/archives/C05LNL7989J/p1703095606965879 : (github refuses videos upload > a certain limit)

#!/bin/bash

# How to use:
# ./export-app.sh <instance> <small-app-id> <super-username> <super-password>
#
# For example (ps: chmod +x ./export-app.sh):
# ./export-app.sh ange-lts new-demo toucantoco TeQICYqAhcfJjffVGVCttRFnzbBdqChD

if [ ! $# -eq 4 ]; then # 4 params needed.
    echo -ne "As a super-admin, export any v2 small-app with this bash util.\n"
    echo -ne "Usage: ./export-app.sh <instance> <small-app-id> <username> <password>\n\n"
fi;

V2_INSTANCE=$1
V2_LTS_APP_ID=$2
SUPER_USER=$3
SUPER_PASSWORD=$4
CURL_REQUEST="curl -LSs -u $SUPER_USER:$SUPER_PASSWORD"

echo -ne "\n> export-app '$V2_LTS_APP_ID' on instance '$V2_INSTANCE'..." && sleep 0.7 && \
$CURL_REQUEST https://api-$V2_INSTANCE.toucantoco.guru/export-app/$V2_LTS_APP_ID -o $V2_LTS_APP_ID.tar.gz

echo -ne "\n> unziping './$V2_LTS_APP_ID.tar.gz'..." && sleep 0.7 && \
tar -xzf ./$V2_LTS_APP_ID.tar.gz
ls -alh

echo -ne "\n< small-app exported successfully :)"

The implementation is available here : https://github.com/ToucanToco/laputa/pull/6292

IMPORT PROCESS¶

The import process involves transferring the exported small-app (as tarball) from v2 to v3 and integrating all relative objects into the dataset- service and layout-service. This process is async and you can follow it witht he operation_id returned on the first call.

Import of a v2 tarball's small-app to a v3 instance (/import-app/v2) Create all objects related to that small-app
To achieve this goal, a set (model -> migration -> import) strategy has been implemented depending on the object we're dealing with and it consist on running all migrations made between v2 and v3 on all related objects we're dealing with.
A rollback procedure is trigger on any error detected while importing the app.

TL&DR: To have an exploitable endpoint to import an app in v2 with all necessary calls to layout-service and dataset-service. :warning: unfortunatelly, to have everything sync such as models and insertion process, we needed to insert some object to mongo first before taking them and create them on services side.

NOTE: this PR is linked to the export part : https://github.com/ToucanToco/laputa/pull/6292

DEMOS¶

See https://toucantoco.slack.com/archives/C05LNL7989J/p1703095606965879 : (github refuses videos upload > a certain limit)

BEARER_TOKEN="_0XBPWQQ_fb7ee6ae-f87a-43c8-a55a-6bd0975ea40a"
TENANT_ID="2814c680-afb1-4d64-95f2-c271b6a9a3e4"
TARBALL_PATH="/tmp/test-import-app/v2-app-as-demo.tar.gz"

curl -X POST \
-H "authorization: Bearer $BEARER_TOKEN" \
-H "Content-Type: multipart/form-data" \
-H "x-toucan-tenant-id: $TENANT_ID" \
-F "file=@$TARBALL_PATH" \
https://api.toucan.local/import-app

NOTE: The implementation is available here : https://github.com/ToucanToco/laputa/pull/6292