Skip to content

Remove unnecessary stage table check in load_realtime_to_bq DAG #26

@VMois

Description

@VMois

check_if_tmp_vehicle_table_has_no_data = BigQueryCheckOperator(
task_id="check_if_tmp_vehicle_table_has_no_data",
gcp_conn_id=GCP_CONN_ID,
sql=f"SELECT NOT EXISTS (SELECT 1 FROM {RAW_DATASET_NAME}.{STAGE_VEHICLE_TABLE_NAME})",
use_legacy_sql=False,
)

WRITE_TRUNCATE in the next PythonOperator task will do the job of cleaning up previous data.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    Status

    ⏳ In Progress

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions