Repo Structure

List of folders and their utility :

• dags
  • Contains all relevant code.
  • Airflow will scan this folder to find any declared DAGS to run.
  • This is structured based on business logic : Everything that is needed for one publisher is in a directory with its name.
  • Everything that is generic to multiple publishers is in the common folder.
• data
  • Contains all data that we want to add to the docker-compose services.
• tests
  • Tests folder.
  • The conftest.py file contains an import of airflow with only a print in it. This might seems useless but it's needed to init the Airflow environment during tests and have everything running properly.
• airflow.cfg
  • Configuration for Airflow. Please note that for QA and production, some configuration might be overridden with ENV variables.