Skip to content

Laputa documentation

Laputa (a.k.a. the "backend") exposes web services to handle Tucana's (a.k.a. the "frontend") most intimate needs. It uses a mongo database to store two kinds of data:

  • application's metadata (e.g. users, groups, permissions),
  • application's data, i.e. the data used to feed Tucana's charts.

Laputa main roles are to:

  • fetch application's data from database,
  • manage metadata,
  • handle static assets (logos, application configuration files, etc.),
  • launch jobs (data processing, publication of staged modifications, screenshot via gotenberg),

10000 feet overview

We use Flask to handle HTTP requests. All routes are defined in laputa/api/server.py and most of them rely on Flask restful.

Users can be authenticated with standard login / password credentials that are tested against the instance's local database. We also support external identity providers that conform to either SAMLv2 or OIDC protocols. We use JWT to convey the session data. Want to know more? Have look at the authentication documentation for more details.

All data, even laputa's own mongo backend data, are requested through toucan-connectors. Data can be cached with a redis cache server. Some permissions can be defined and associated to user profiles and data domains (i.e. collections in our mongo backend). When a query is performed, permissions corresponding to the current context (user profile and data domain) are injected as additional restrictions to the original query when possible. Otherwise, they are used once the data is fetched by removing unauthorized rows. Want to know more? Have a look at the permissions documentation for more details.

Some HTTP requests are meant to trigger long running tasks such as "data processing" (a.k.a. the "augment" or the T in ETL). We use celery to perform those tasks asynchronously and use redis as a broker. Want to know more? Have a look at the async tasks documentation for more details.

Configuration

A laputa instance is configured through a configuration file that will most notably specify:

  • the mongo database connection parameters,
  • the redis cache connection parameters,
  • the celery settings,
  • default users' credentials,
  • static assets locations on the filesystem.

Laputa's stack processes

In production, laputa's python processes are managed by supervisor, and are:

  • gunicorn (1 master process + 5 workers) to handle HTTP request
  • celery (2 pools of processes, for heavy & fast jobs) to handle async jobs
  • the websocket server (1 process, using asyncio)
  • the scheduler (1 process)

Documentation

We use mkdocs to build the documentation. To run it locally:

  • install mkdocs:

  • on debian derivatives:

bash sudo apt install mkdocs

  • on macos with brew:

bash brew install mkdocs

  • on any platform:

bash pip install mkdocs mkdocstrings mkdocs-material

  • build and run the documentation:
mkdocs serve
  • open your browser to the address printed by mkdocs in your terminal

Going further…

Install, read the detailed documentation on each topic, explore the howtos sections and enjoy!