Standardize unioning macros to use fivetran_utils versions

.gitignore

@@ -1,4 +1,5 @@
 # dbt
+worktrees/
 **/package-lock.yml
 package-lock.yml
 .dbt/

CHANGELOG.md

@@ -1,3 +1,11 @@
+# dbt_jira v1.10.0
+
+[PR #191](https://github.com/fivetran/dbt_jira/pull/191) includes the following updates:
+
+## Under the Hood
+- Migrates the `union_connections`, `apply_source_relation`, and `partition_by_source_relation` macros to the `dbt_fivetran_utils` package.
+- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_jira/#source-casing-for-case-sensitive-destinations) section of the README for details.
+
 # dbt_jira v1.9.0
 
 [PR #186](https://github.com/fivetran/dbt_jira/pull/186) includes the following updates:
@@ -25,7 +33,7 @@
 [PR #182](https://github.com/fivetran/dbt_jira/pull/182) includes the following updates:
 
 ## Bug Fix
-- For warehouses using the `insert_overwrite` strategy, updates the incremental filter in `jira__daily_issue_field_history` to align with the model’s partition grain. This helps prevent issues with incomplete partition coverage.
+- For warehouses using the `insert_overwrite` strategy, updates the incremental filter in `jira__daily_issue_field_history` to align with the model's partition grain. This helps prevent issues with incomplete partition coverage.
   - You must run a `--full-refresh` after upgrading to apply this fix to existing data.
 
 ## Under the Hood
@@ -601,7 +609,7 @@ This release includes the following updates.
   - Please note, if you are installing a version of `dbt_utils` in your `packages.yml` that is not in the range above then you will encounter a package dependency error.
 
 # dbt_jira v0.6.0
-## 🚨 Breaking Changes 🚨
+## 🚨 Breaking Changes
 - This release of the `dbt_jira` packages implements changes to the incremental logic within various models highlighted in the Bug Fixes section below. As such, a `dbt run --full-refresh` will be required after upgrading this dependency for this package in your `packages.yml`.
 ## Bug Fixes
 - Corrected CTE references within `int_jira__issue_assignee_resolution`. The final cte referenced was previously selecting from `issue_field_history` when it should have been selecting from `filtered`. ([#45](https://github.com/fivetran/dbt_jira/pull/45))
@@ -613,4 +621,4 @@ This release includes the following updates.
 - @thibonacci ([#45](https://github.com/fivetran/dbt_jira/pull/45))
 
 # dbt_jira v0.1.0 -> v0.5.1
-Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!
+Refer to the relevant release notes on the Github repository for specific details for the previous releases. Thank you!

README.md

@@ -66,7 +66,7 @@ Include the following jira package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/jira
-    version: [">=1.9.0", "<1.10.0"]
+    version: [">=1.10.0", "<1.11.0"]
 ```
 
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/jira_source` in your `packages.yml` since this package has been deprecated.
@@ -90,7 +90,6 @@ For **Snowflake**, **Redshift**, and **Postgres** databases, we have chosen `del
 > Regardless of strategy, we recommend that users periodically run a `--full-refresh` to ensure a high level of data quality.
 
 ### Define database and schema variables
-
 #### Option A: Single connection
 By default, this package runs using your destination and the `jira` schema. If this is not where your Jira data is (for example, if your Jira schema is named `jira_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
@@ -120,42 +119,9 @@ vars:
         name: connection_2_source_name
 ```
 
-##### Recommended: Incorporate unioned sources into DAG
-> *If you are running the package through [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore), the below step is necessary in order to synchronize model runs with your Jira connections. Alternatively, you may choose to run the package through Fivetran [Quickstart](https://fivetran.com/docs/transformations/quickstart), which would create separate sets of models for each Jira source rather than one set of unioned models.*
-
-By default, this package defines one single-connection source, called `jira`, which will be disabled if you are unioning multiple connections. This means that your DAG will not include your Jira sources, though the package will run successfully.
-
-To properly incorporate all of your Jira connections into your project's DAG:
-1. Define each of your sources in a `.yml` file in your project. Utilize the following template for the `source`-level configurations, and, **most importantly**, copy and paste the table and column-level definitions from the package's `src_jira.yml` [file](https://github.com/fivetran/dbt_jira/blob/main/models/staging/src_jira.yml).
+#### Optional: Incorporate unioned sources into DAG
 
-```yml
-# a .yml file in your root project
-
-version: 2
-
-sources:
-  - name: <name> # ex: Should match name in jira_sources
-    schema: <schema_name>
-    database: <database_name>
-    loader: fivetran
-    config:
-      loaded_at_field: _fivetran_synced
-      freshness: # feel free to adjust to your liking
-        warn_after: {count: 72, period: hour}
-        error_after: {count: 168, period: hour}
-
-    tables: # copy and paste from jira/models/staging/src_jira.yml - see https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/ for how to use anchors to only do so once
-```
-
-> **Note**: If there are source tables you do not have (see [Disable models for non existent sources](https://github.com/fivetran/dbt_jira?tab=readme-ov-file#disable-models-for-non-existent-sources)), you may still include them, as long as you have set the right variables to `False`.
-
-2. Set the `has_defined_sources` variable (scoped to the `jira` package) to `True`, like such:
-```yml
-# dbt_project.yml
-vars:
-  jira:
-    has_defined_sources: true
-```
+If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Jira connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_jira/blob/main/models/staging/src_jira.yml). Set the variable `has_defined_sources: true` under the Jira namespace in your `dbt_project.yml`. Otherwise, your Jira connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.
 
 ### Disable models for non-existent sources
 Your Jira connection may not sync every table that this package expects. If you do not have the `SPRINT`, `COMPONENT`, `VERSION`, `PRIORITY` or `TEAM` tables synced, add the respective variables to your root `dbt_project.yml` file. Additionally, if you want to remove comment aggregations from your `jira__issue_enhanced` model,  add the `jira_include_comments` variable to your root `dbt_project.yml`:
@@ -254,6 +220,14 @@ vars:
     jira_<default_source_table_name>_identifier: your_table_name 
 ```
 
+#### Source casing for case-sensitive destinations
+By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:
+
+```yml
+vars:
+    fivetran_using_source_casing: true
+```
+
 #### Lookback Window
 Records from the source may occasionally arrive late. To handle this, we implement a one-week lookback in our incremental models to capture late arrivals without requiring frequent full refreshes. The lookback is structured in weekly increments, as the incremental logic is based on weekly periods. While the frequency of full refreshes can be reduced, we still recommend running `dbt --full-refresh` periodically to maintain data quality of the models. 
 
@@ -303,4 +277,4 @@ We highly encourage and welcome contributions to this package. Learn how to cont
 
 ## Are there any resources available?
 - If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_jira/issues/new/choose) section to find the right avenue of support for you.
-- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
+- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira'
-version: '1.9.0'
+version: '1.10.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 

integration_tests/ci/test_scenarios.yml

@@ -26,3 +26,9 @@ test_scenarios:
       jira_using_versions: false
       jira_field_grain: 'field_name'
     include_incremental: true
+
+  - name: "MDLS"
+    vars:
+      fivetran_using_source_casing: true
+    include_incremental: false
+    include_dbt_seed: true

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'jira_integration_tests'
-version: '1.9.0'
+version: '1.10.0'
 config-version: 2
 profile: 'integration_tests'
 
@@ -39,12 +39,12 @@ models:
     relation: true
     columns: "{{ false if target.type in ('databricks') else true }}"
   jira:
-    +schema: "{{ 'jira_integrations_tests_sqlw' if target.name == 'databricks-sql' else 'jira' }}"
-    # +schema: "jira_{{ var('directed_schema','dev') }}"
+    +schema: "jira_{{ var('directed_schema','dev') }}"
 
 seeds:
   +docs:
     show: False
+  +quote_columns: "{{ target.type == 'redshift' or var('fivetran_using_source_casing', false) }}"
   jira_integration_tests:
     +column_types:
       _fivetran_synced: timestamp

macros/staging/get_issue_field_history_columns.sql

@@ -10,11 +10,11 @@
 ] %}
 
 {% if target.type == 'redshift' %}
- {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp(), "quote": True } ) }}
-{% elif target.type == 'snowflake' %}
- {{ columns.append( {"name": "TIME", "datatype": dbt.type_timestamp(), "quote": True } ) }}
+    {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp(), "quote": True } ) }}
+{% elif target.type == 'snowflake' and not var('fivetran_using_source_casing', false) %}
+    {{ columns.append( {"name": "TIME", "datatype": dbt.type_timestamp(), "quote": True } ) }}
 {% else %}
- {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp()} ) }}
+    {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp()} ) }}
 {% endif %}
 
 

macros/staging/get_issue_multiselect_history_columns.sql

@@ -11,11 +11,11 @@
 ] %}
 
 {% if target.type == 'redshift' %}
- {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp(), "quote": True } ) }}
-{% elif target.type == 'snowflake' %}
- {{ columns.append( {"name": "TIME", "datatype": dbt.type_timestamp(), "quote": True } ) }}
+    {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp(), "quote": True } ) }}
+{% elif target.type == 'snowflake' and not var('fivetran_using_source_casing', false) %}
+    {{ columns.append( {"name": "TIME", "datatype": dbt.type_timestamp(), "quote": True } ) }}
 {% else %}
- {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp()} ) }}
+    {{ columns.append( {"name": "time", "datatype": dbt.type_timestamp()} ) }}
 {% endif %}
 
 {{ return(columns) }}