Skip to content
Merged
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,34 @@
# dbt_ad_reporting v2.7.0

[PR #177](https://github.com/fivetran/dbt_ad_reporting/pull/177) includes the following updates:

## Schema/Data Changes (--full-refresh required after upgrading)
**1 total change • 1 possible breaking change**

| Data Model(s) | Change type | Old | New | Notes |
| ------------- | ----------- | --- | --- | ----- |
| All models | `source_relation` column (when not using the union connections of the same type feature) | Empty string (`''`) | `<database>.<schema>` | |

## Feature Updates
- Introduces the new (recommended) `{connector_name}_sources` variables for more robust union data configuration. The old `{connector_name}_union_schemas` and `{connector_name}_union_databases` variables will still be supported. See the [README](https://github.com/fivetran/dbt_ad_reporting/tree/main#define-database-and-schema-variables) for specific details.

## Upstream Dependency Changes
- Increases the required Amazon Ads package to version [1.3.0](https://github.com/fivetran/dbt_amazon_ads/releases/tag/v1.3.0)
- Increases the required Apple Search Ads package to version [1.3.0](https://github.com/fivetran/dbt_apple_search_ads/releases/tag/v1.3.0)
- Increases the required Facebook Ads package to version [1.5.0](https://github.com/fivetran/dbt_facebook_ads/releases/tag/v1.5.0)
- Increases the required Google Ads package to version [1.4.0](https://github.com/fivetran/dbt_google_ads/releases/tag/v1.4.0)
- Increases the required LinkedIn Ads package to version [1.3.0](https://github.com/fivetran/dbt_linkedin/releases/tag/v1.3.0)
- Increases the required Microsoft Ads package to version [1.3.0](https://github.com/fivetran/dbt_microsoft_ads/releases/tag/v1.3.0)
- Increases the required Pinterest Ads package to version [1.3.0](https://github.com/fivetran/dbt_pinterest/releases/tag/v1.3.0)
- Increases the required Reddit Ads package to version [1.5.0](https://github.com/fivetran/dbt_reddit_ads/releases/tag/v1.5.0)
- Increases the required Snapchat Ads package to version [1.3.0](https://github.com/fivetran/dbt_snapchat_ads/releases/tag/v1.3.0)
- Increases the required TikTok Ads package to version [1.3.0](https://github.com/fivetran/dbt_tiktok_ads/releases/tag/v1.3.0)
- Increases the required Twitter Ads package to version [1.3.0](https://github.com/fivetran/dbt_twitter/releases/tag/v1.3.0)

## Under the Hood
- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_ad_reporting/#source-casing-for-case-sensitive-destinations) section of the README for details.
- Introduces `fivetran_utils.partition_by_source_relation` to conditionally include `source_relation` in partition clauses only when multiple sources are configured.

# dbt_ad_reporting v2.6.1

[PR #181](https://github.com/fivetran/dbt_ad_reporting/pull/181) includes the following updates:
Expand Down
197 changes: 139 additions & 58 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ Include the following github package version in your `packages.yml`
```yaml
packages:
- package: fivetran/ad_reporting
version: [">=2.6.0", "<2.7.0"] # we recommend using ranges to capture non-breaking changes automatically
version: [">=2.7.0", "<2.8.0"] # we recommend using ranges to capture non-breaking changes automatically
```

Do NOT include the individual ad platform packages in this file. The ad reporting package itself has dependencies on these packages and will install them as well.
Expand All @@ -152,8 +152,9 @@ dispatch:
search_order: ['google_ads_source', 'dbt_expectations']
```

### Configure Database and Schema Variables
By default, this package looks for your ad platform data in your target database. If this is not where your app platform data is stored, add the relevant `<connector>_database` variables to your `dbt_project.yml` file (see below).
### Define database and schema variables
#### Option A: Single connection(s)
By default, this package looks for your ad platform data in your target database. If this is not where your ad platform data is stored, add the relevant `<connector>_database` variables to your `dbt_project.yml` file (see below).
> Please note, cross-database querying, where the `*_database` variable differs from the database specified in your `profiles.yml`, is not supported by all dbt adapters (e.g., dbt-redshift). Refer to the documentation for your specific destination adapter for more details on its capabilities.

```yml
Expand Down Expand Up @@ -182,8 +183,8 @@ vars:
reddit_ads_schema: reddit_ads
reddit_ads_database: your_database_name

snapchat_schema: snapchat_ads
snapchat_database: your_database_name
snapchat_ads_schema: snapchat_ads
snapchat_ads_database: your_database_name

tiktok_ads_schema: tiktok_ads
tiktok_ads_database: your_database_name
Expand All @@ -192,6 +193,120 @@ vars:
twitter_ads_database: your_database_name
```

#### Option B: Union multiple connections
If you have multiple ad platform connections of the same type in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.

To use this functionality, you will need to set the below variables in your root `dbt_project.yml` file:
```yml
# dbt_project.yml

vars:
amazon_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

apple_search_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

facebook_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

google_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

linkedin_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

microsoft_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

pinterest_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

reddit_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

snapchat_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

tiktok_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name

twitter_ads_sources:
- database: connection_1_destination_name # Required
schema: connection_1_schema_name # Required
name: connection_1_source_name # Required only if following the step in the following subsection

- database: connection_2_destination_name
schema: connection_2_schema_name
name: connection_2_source_name
```

> Previous versions of this package employed two separate, mutually exclusive variables for unioning for each platform: (eg. `google_ads_union_schemas` and `google_ads_union_databases`). While these variables are still supported, the new approach shared above are the recommended variables to configure.

#### Optional: Incorporate unioned sources into DAG

If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Ad Platform connections of the same type, you can define your sources in a property `.yml` file. Set the variable `has_defined_sources: true` in your `dbt_project.yml`. Otherwise, your connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.

### Enabling/Disabling Models
This package takes into consideration that not every account will have every feature enabled per platform. If your syncs exclude certain tables, it is because you either don't use that functionality in your respective ad platforms or have actively excluded some tables from your syncs.

Expand Down Expand Up @@ -351,48 +466,6 @@ models:
### (Optional) Additional configurations
<details open><summary>Expand/Collapse details</summary>

#### Union multiple connections
If you have multiple ad reporting connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `<package_name>_union_schemas` OR `<package_name>_union_databases` variables (cannot do both) in your root `dbt_project.yml` file. Below are the variables and examples for each connection:

```yml
vars:
amazon_ads_union_schemas: ['amazon_ads_usa','amazon_ads_canada']
amazon_ads_union_databases: ['amazon_ads_usa','amazon_ads_canada']

apple_search_ads_union_schemas: ['apple_search_ads_usa','apple_search_ads_canada']
apple_search_ads_union_databases: ['apple_search_ads_usa','apple_search_ads_canada']

facebook_ads_union_schemas: ['facebook_ads_usa','facebook_ads_canada']
facebook_ads_union_databases: ['facebook_ads_usa','facebook_ads_canada']

google_ads_union_schemas: ['google_ads_usa','google_ads_canada']
google_ads_union_databases: ['google_ads_usa','google_ads_canada']

linkedin_ads_union_schemas: ['linkedin_usa','linkedin_canada']
linkedin_ads_union_databases: ['linkedin_usa','linkedin_canada']

microsoft_ads_union_schemas: ['microsoft_ads_usa','microsoft_ads_canada']
microsoft_ads_union_databases: ['microsoft_ads_usa','microsoft_ads_canada']

pinterest_ads_union_schemas: ['pinterest_usa','pinterest_canada']
pinterest_ads_union_databases: ['pinterest_usa','pinterest_canada']

reddit_ads_union_schemas: ['reddit_ads_usa','reddit_ads_canada']
reddit_ads_union_databases: ['reddit_ads_usa','reddit_ads_canada']

snapchat_ads_union_schemas: ['snapchat_ads_usa','snapchat_ads_canada']
snapchat_ads_union_databases: ['snapchat_ads_usa','snapchat_ads_canada']

tiktok_ads_union_schemas: ['tiktok_ads_usa','tiktok_ads_canada']
tiktok_ads_union_databases: ['tiktok_ads_usa','tiktok_ads_canada']

twitter_ads_union_schemas: ['twitter_usa','twitter_canada']
twitter_ads_union_databases: ['twitter_usa','twitter_canada']
```
> NOTE: The native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.

To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.

#### Configure types of conversion events
For the following platforms, conversion data is sent along with the type of event that you may or may not consider to be a conversion. For the most part, the packages consider **leads**, **purchases**, and **custom** user-defined events as conversions by default, but this can be configured via the appropriate variables.

Expand Down Expand Up @@ -640,6 +713,14 @@ vars:
<default_source_table_name>_identifier: your_table_name
```

#### Source casing for case-sensitive destinations
By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:

```yml
vars:
fivetran_using_source_casing: true
```

</details>
<br>

Expand Down Expand Up @@ -732,37 +813,37 @@ packages:
version: [">=1.0.0", "<2.0.0"]

- package: fivetran/amazon_ads
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]

- package: fivetran/apple_search_ads
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]

- package: fivetran/facebook_ads
version: [">=1.4.0", "<1.5.0"]
version: [">=1.5.0", "<1.6.0"]

- package: fivetran/google_ads
version: [">=1.3.0", "<1.4.0"]
version: [">=1.4.0", "<1.5.0"]

- package: fivetran/linkedin
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]

- package: fivetran/microsoft_ads
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]

- package: fivetran/pinterest
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]

- package: fivetran/reddit_ads
version: [">=1.4.0", "<1.5.0"]
version: [">=1.5.0", "<1.6.0"]

- package: fivetran/snapchat_ads
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]

- package: fivetran/tiktok_ads
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]

- package: fivetran/twitter_ads
version: [">=1.2.0", "<1.3.0"]
version: [">=1.3.0", "<1.4.0"]
```

### Other Dependencies
Expand Down
2 changes: 1 addition & 1 deletion dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'ad_reporting'
version: '2.6.1'
version: '2.7.0'

config-version: 2

Expand Down
2 changes: 1 addition & 1 deletion docs/catalog.json

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/manifest.json

Large diffs are not rendered by default.

6 changes: 6 additions & 0 deletions integration_tests/ci/test_scenarios.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,3 +41,9 @@ test_scenarios:
ad_reporting__tiktok_ads_enabled: false
ad_reporting__twitter_ads_enabled: false
include_incremental: false

- name: "MDLS"
vars:
fivetran_using_source_casing: true
include_incremental: false
include_dbt_seed: true
4 changes: 2 additions & 2 deletions integration_tests/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
name: 'ad_reporting_integration_tests'
version: '2.6.1'
version: '2.7.0'
profile: 'integration_tests'
config-version: 2

Expand Down Expand Up @@ -577,7 +577,7 @@ seeds:
ad_reporting_integration_tests:
+docs:
show: false
+quote_columns: "{{ true if target.type in ('redshift') else false }}"
+quote_columns: "{{ target.type in ('redshift') or var('fivetran_using_source_casing', false) }}"

### Microsoft Seeds
microsoft_ads_account_history_data:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
ad_group_id,ad_id,campaign_id,date,_fivetran_synced,campaign_budget_amount,campaign_budget_currency_code,campaign_budget_type,clicks,cost,impressions, advertised_asin, advertised_sku
ad_group_id,ad_id,campaign_id,date,_fivetran_synced,campaign_budget_amount,campaign_budget_currency_code,campaign_budget_type,clicks,cost,impressions,advertised_asin,advertised_sku
501,8108,5555,2022-12-10,2022-12-13 17:17:13.167000,250.0,USD,DAILY_BUDGET,2,2.53,166,,
501,8108,5555,2022-12-06,2022-12-13 17:17:13.017000,250.0,USD,DAILY_BUDGET,3,5.29,75,,
421,77,2187,2022-08-20,2022-10-14 11:39:03.648000,2000.0,USD,DAILY_BUDGET,3,3.79,1310,,
Expand Down
2 changes: 1 addition & 1 deletion integration_tests/seeds/tiktok_ad_report_hourly_data.csv
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
AD_ID,STAT_TIME_HOUR,COST_PER_CONVERSION,CPC,VIDEO_PLAY_ACTIONS,CONVERSION_RATE,VIDEO_VIEWS_P_75,RESULT,VIDEO_VIEWS_P_50,IMPRESSIONS,COMMENTS,REAL_TIME_COST_PER_RESULT,CONVERSION,REAL_TIME_RESULT,VIDEO_VIEWS_P_100,SHARES,REAL_TIME_CONVERSION_RATE,COST_PER_SECONDARY_GOAL_RESULT,SECONDARY_GOAL_RESULT_RATE,CLICKS,COST_PER_1000_REACHED,VIDEO_VIEWS_P_25,REACH,REAL_TIME_COST_PER_CONVERSION,PROFILE_VISITS_RATE,AVERAGE_VIDEO_PLAY,PROFILE_VISITS,CPM,CTR,VIDEO_WATCHED_2_S,FOLLOWS,RESULT_RATE,VIDEO_WATCHED_6_S,SECONDARY_GOAL_RESULT,COST_PER_RESULT,AVERAGE_VIDEO_PLAY_PER_USER,REAL_TIME_RESULT_RATE,SPEND,LIKES,REAL_TIME_CONVERSION,TOTAL_PURCHASE_VALUE,TOTAL_SALES_LEAD_VALUE,_FIVETRAN_SYNCED
ad_id,stat_time_hour,cost_per_conversion,cpc,video_play_actions,conversion_rate,video_views_p_75,result,video_views_p_50,impressions,comments,real_time_cost_per_result,conversion,real_time_result,video_views_p_100,shares,real_time_conversion_rate,cost_per_secondary_goal_result,secondary_goal_result_rate,clicks,cost_per_1000_reached,video_views_p_25,reach,real_time_cost_per_conversion,profile_visits_rate,average_video_play,profile_visits,cpm,ctr,video_watched_2_s,follows,result_rate,video_watched_6_s,secondary_goal_result,cost_per_result,average_video_play_per_user,real_time_result_rate,spend,likes,real_time_conversion,total_purchase_value,total_sales_lead_value,_fivetran_synced
1,2021-09-15 10:00:00,0,0,22,0,2,0,2,23,0,0,0,0,0,0,0,-,-,0,0,22,0,0,0,2.22,0,32.22,0,2,0,0,22,-,0,0,0,2.22,0,0,0.00,0.00,2021-09-15 15:56:34.862
2,2021-09-15 10:00:00,0,0,33,0,0,0,0,22,0,0,0,0,0,0,0,-,-,0,0,3,0,0,0,2.22,0,2.22,0,2,0,0,2,-,0,0,0,0.22,0,0,0.01,0.01,2021-09-15 15:56:34.953
3,2021-09-15 10:00:00,0,0,2,0,0,0,0,2,0,0,0,0,0,0,0,-,-,0,0,0,0,0,0,2.22,0,0,0,0,0,0,0,-,0,0,0,0,0,0,0.00,0.00,2021-09-15 15:56:34.923
Expand Down
Loading
Loading