|
check_if_tmp_vehicle_table_has_no_data = BigQueryCheckOperator( |
|
task_id="check_if_tmp_vehicle_table_has_no_data", |
|
gcp_conn_id=GCP_CONN_ID, |
|
sql=f"SELECT NOT EXISTS (SELECT 1 FROM {RAW_DATASET_NAME}.{STAGE_VEHICLE_TABLE_NAME})", |
|
use_legacy_sql=False, |
|
) |
WRITE_TRUNCATE in the next PythonOperator task will do the job of cleaning up previous data.
miwaitway/dags/load_realtime_to_bq.py
Lines 99 to 104 in 1e77fde
WRITE_TRUNCATE in the next PythonOperator task will do the job of cleaning up previous data.