Standardize unioning macros to use fivetran_utils versions

CHANGELOG.md

@@ -1,3 +1,21 @@
+# dbt_google_play v1.3.0
+
+[PR #34](https://github.com/fivetran/dbt_google_play/pull/34) includes the following updates:
+
+## Schema/Data Changes (--full-refresh required after upgrading)
+**1 total change • 1 possible breaking change**
+
+| Data Model(s) | Change type | Old | New | Notes |
+| ------------- | ----------- | --- | --- | ----- |
+| All models | `source_relation` column (when using a single Google Play schema) | Empty string (`''`) | `<database>.<schema>` |  |
+
+## Feature Updates
+- Introduces the new (recommended) `google_play_sources` variable for more robust union data configuration. The old `google_play_union_schemas` and `google_play_union_databases` variables will still be supported. See the [README](https://github.com/fivetran/dbt_google_play/tree/main#define-database-and-schema-variables) for specific details.
+
+## Under the Hood
+- Adds the `fivetran_using_source_casing` variable for case-sensitive destination support. When enabled, downstream transformations respect source casing to ensure consistent results. See the [Additional Configurations](https://github.com/fivetran/dbt_google_play/#source-casing-for-case-sensitive-destinations) section of the README for details.
+- Introduces `fivetran_utils.partition_by_source_relation` to conditionally include `source_relation` in partition clauses only when multiple sources are configured.
+
 # dbt_google_play v1.2.0
 
 [PR #31](https://github.com/fivetran/dbt_google_play/pull/31) includes the following updates:

README.md

@@ -64,20 +64,47 @@ Include the following Google Play package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/google_play
-    version: [">=1.2.0", "<1.3.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=1.3.0", "<1.4.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 > All required sources and staging models are now bundled into this transformation package. Do not include `fivetran/google_play_source` in your `packages.yml` since this package has been deprecated.
 
 ### Define database and schema variables
+#### Option A: Single connection
 By default, this package runs using your destination and the `google_play` schema. If this is not where your Google Play data is (for example, if your Google Play schema is named `google_play_fivetran`), add the following configuration to your root `dbt_project.yml` file:
 
 ```yml
 vars:
     google_play_database: your_destination_name
-    google_play_schema: your_schema_name 
+    google_play_schema: your_schema_name
 ```
 
+#### Option B: Union multiple connections
+If you have multiple Google Play connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. For each source table, the package will union all of the data together and pass the unioned table into the transformations. The `source_relation` column in each model indicates the origin of each record.
+
+To use this functionality, you will need to set the `google_play_sources` variable in your root `dbt_project.yml` file:
+
+```yml
+# dbt_project.yml
+
+vars:
+  google_play:
+    google_play_sources:
+      - database: connection_1_destination_name # Required
+        schema: connection_1_schema_name # Required
+        name: connection_1_source_name # Required only if following the step in the following subsection
+
+      - database: connection_2_destination_name
+        schema: connection_2_schema_name
+        name: connection_2_source_name
+```
+
+> Previous versions of this package employed two separate, mutually exclusive variables for unioning: `google_play_union_schemas` and `google_play_union_databases`. While these variables are still supported, `google_play_sources` is the recommended variable to configure.
+
+#### Optional: Incorporate unioned sources into DAG
+
+If you use [Fivetran Transformations for dbt Core™](https://fivetran.com/docs/transformations/dbt#transformationsfordbtcore) and are unioning multiple Google Play connections, you can define your sources in a property `.yml` file, [using this as a template](https://github.com/fivetran/dbt_google_play/blob/main/models/staging/src_google_play.yml). Set the variable `has_defined_sources: true` under the Google Play namespace in your `dbt_project.yml`. Otherwise, your Google Play connections won't appear in your DAG. See the `union_connections` macro [documentation](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#optional-union-connections-defined-sources-configuration) for full configuration details.
+
 ### Disable or enable source tables
 Your Google Play connection might not sync every table that this package expects. If you have financial and/or subscriptions data, namely the `earnings` and `financial_stats_subscriptions_country` tables, add the following variable(s) to your `dbt_project.yml` file:
 
@@ -96,18 +123,6 @@ You will need to `dbt seed` the `google_play__country_codes` [file](https://gith
 ### (Optional) Additional configurations
 <details open><summary>Expand/collapse configurations</summary>
 
-#### Union multiple connections
-If you have multiple google_play connections in Fivetran and would like to use this package on all of them simultaneously, we have provided functionality to do so. The package will union all of the data together and pass the unioned table into the transformations. You will be able to see which source it came from in the `source_relation` column of each model. To use this functionality, you will need to set either the `google_play_union_schemas` OR `google_play_union_databases` variables (cannot do both) in your root `dbt_project.yml` file:
-
-```yml
-vars:
-    google_play_union_schemas: ['google_play_usa','google_play_canada'] # use this if the data is in different schemas/datasets of the same database/project
-    google_play_union_databases: ['google_play_usa','google_play_canada'] # use this if the data is in different databases/projects but uses the same schema name
-```
-> NOTE: The native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.
-
-To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
-
 #### Change the build schema
 By default, this package builds the Google Play staging models within a schema titled (`<target_schema>` + `_google_play_source`) and your Google Play modeling models within a schema titled (`<target_schema>` + `_google_play`) in your destination. If this is not where you would like your Google Play data to be written to, add the following configuration to your root `dbt_project.yml` file:
 
@@ -128,6 +143,14 @@ If an individual source table has a different name than the package expects, add
 vars:
     google_play_<default_source_table_name>_identifier: your_table_name 
 ```
+
+#### Source casing for case-sensitive destinations
+By default, the package applies case-insensitive comparisons when resolving `source_relation` values. If your destination is case-sensitive and you want downstream transformations to respect the exact casing of your source database and schema names, set the following variable:
+
+```yml
+vars:
+    fivetran_using_source_casing: true
+```
 </details>
 
 <br>
@@ -171,4 +194,4 @@ We highly encourage and welcome contributions to this package. Learn how to cont
 
 ## Are there any resources available?
 - If you have questions or want to reach out for help, see the [GitHub Issue](https://github.com/fivetran/dbt_google_play/issues/new/choose) section to find the right avenue of support for you.
-- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).
+- If you would like to provide feedback to the dbt package team at Fivetran or would like to request a new dbt package, fill out our [Feedback Form](https://www.surveymonkey.com/r/DQ7K7WW).

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'google_play'
-version: '1.2.0'
+version: '1.3.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<3.0.0"]
 vars:

integration_tests/ci/test_scenarios.yml

@@ -21,4 +21,10 @@ test_scenarios:
     vars:
       google_play__using_earnings: true
       google_play__using_subscriptions: true
-    include_incremental: false
+    include_incremental: false
+
+  - name: "MDLS"
+    vars:
+      fivetran_using_source_casing: true
+    include_incremental: false
+    include_dbt_seed: true

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'google_play_integration_tests'
-version: '1.2.0'
+version: '1.3.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -41,6 +41,7 @@ seeds:
   +docs:
     show: False
   google_play_integration_tests:
+    +quote_columns: "{{ target.type == 'redshift' or var('fivetran_using_source_casing', false) }}"
     stats_installs_overview:
       +column_types:
         active_device_installs: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
@@ -49,4 +50,4 @@ seeds:
         app_version_code: "{{ 'int64' if target.type == 'bigquery' else 'bigint' }}"
 flags:
   send_anonymous_usage_stats: False
-  use_colors: True
+  use_colors: True

models/google_play__app_version_report.sql

@@ -20,8 +20,8 @@ install_metrics as (
 
     select
         *,
-        sum(device_installs) over (partition by source_relation, app_version_code, package_name order by date_day asc rows between unbounded preceding and current row) as total_device_installs,
-        sum(device_uninstalls) over (partition by source_relation, app_version_code, package_name order by date_day asc rows between unbounded preceding and current row) as total_device_uninstalls
+        sum(device_installs) over (partition by app_version_code, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_device_installs,
+        sum(device_uninstalls) over (partition by app_version_code, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_device_uninstalls
     from installs 
 ), 
 
@@ -79,7 +79,7 @@ create_partitions as (
 
     {% for metric in rolling_metrics -%}
         , sum(case when {{ metric }} is null 
-                then 0 else 1 end) over (partition by source_relation, app_version_code, package_name order by date_day asc rows unbounded preceding) as {{ metric | lower }}_partition
+                then 0 else 1 end) over (partition by app_version_code, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows unbounded preceding) as {{ metric | lower }}_partition
     {%- endfor %}
     from app_version_join
 ), 
@@ -108,7 +108,7 @@ fill_values as (
         {% for metric in rolling_metrics -%}
 
         , first_value( {{ metric }} ) over (
-            partition by source_relation, {{ metric | lower }}_partition, app_version_code, package_name order by date_day asc rows between unbounded preceding and current row) as {{ metric }}
+            partition by {{ metric | lower }}_partition, app_version_code, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as {{ metric }}
 
         {%- endfor %}
     from create_partitions

models/google_play__country_report.sql

@@ -26,17 +26,17 @@ install_metrics as (
 
     select
         *,
-        sum(device_installs) over (partition by source_relation, country, package_name order by date_day asc rows between unbounded preceding and current row) as total_device_installs,
-        sum(device_uninstalls) over (partition by source_relation, country, package_name order by date_day asc rows between unbounded preceding and current row) as total_device_uninstalls
+        sum(device_installs) over (partition by country, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_device_installs,
+        sum(device_uninstalls) over (partition by country, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_device_uninstalls
     from installs 
 ), 
 
 store_performance_metrics as (
 
     select
         *,
-        sum(store_listing_acquisitions) over (partition by source_relation, country_region, package_name order by date_day asc rows between unbounded preceding and current row) as total_store_acquisitions,
-        sum(store_listing_visitors) over (partition by source_relation, country_region, package_name order by date_day asc rows between unbounded preceding and current row) as total_store_visitors
+        sum(store_listing_acquisitions) over (partition by country_region, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_store_acquisitions,
+        sum(store_listing_visitors) over (partition by country_region, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_store_visitors
     from store_performance
 ), 
 
@@ -98,7 +98,7 @@ create_partitions as (
 
     {% for metric in rolling_metrics -%}
         , sum(case when {{ metric }} is null 
-                then 0 else 1 end) over (partition by source_relation, country, package_name order by date_day asc rows unbounded preceding) as {{ metric | lower }}_partition
+                then 0 else 1 end) over (partition by country, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows unbounded preceding) as {{ metric | lower }}_partition
     {%- endfor %}
     from country_join
 ), 
@@ -128,7 +128,7 @@ fill_values as (
         {% for metric in rolling_metrics -%}
 
         , first_value( {{ metric }} ) over (
-            partition by source_relation, {{ metric | lower }}_partition, country, package_name order by date_day asc rows between unbounded preceding and current row) as {{ metric }}
+            partition by {{ metric | lower }}_partition, country, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as {{ metric }}
 
         {%- endfor %}
     from create_partitions

models/google_play__device_report.sql

@@ -14,8 +14,8 @@ install_metrics as (
 
     select
         *,
-        sum(device_installs) over (partition by source_relation, device, package_name order by date_day asc rows between unbounded preceding and current row) as total_device_installs,
-        sum(device_uninstalls) over (partition by source_relation, device, package_name order by date_day asc rows between unbounded preceding and current row) as total_device_uninstalls
+        sum(device_installs) over (partition by device, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_device_installs,
+        sum(device_uninstalls) over (partition by device, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_device_uninstalls
     from installs 
 ), 
 
@@ -66,7 +66,7 @@ create_partitions as (
 
     {% for metric in rolling_metrics -%}
         , sum(case when {{ metric }} is null 
-                then 0 else 1 end) over (partition by source_relation, device, package_name order by date_day asc rows unbounded preceding) as {{ metric | lower }}_partition
+                then 0 else 1 end) over (partition by device, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows unbounded preceding) as {{ metric | lower }}_partition
     {%- endfor %}
     from device_join
 ), 
@@ -93,7 +93,7 @@ fill_values as (
         {% for metric in rolling_metrics -%}
 
         , first_value( {{ metric }} ) over (
-            partition by source_relation, {{ metric | lower }}_partition, device, package_name order by date_day asc rows between unbounded preceding and current row) as {{ metric }}
+            partition by {{ metric | lower }}_partition, device, package_name {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as {{ metric }}
 
         {%- endfor %}
     from create_partitions

models/google_play__finance_report.sql

@@ -69,7 +69,7 @@ create_partitions as (
     select 
         *,
         sum(case when total_active_subscriptions is null 
-                then 0 else 1 end) over (partition by source_relation, country_short, sku_id order by date_day asc rows unbounded preceding) as total_active_subscriptions_partition
+                then 0 else 1 end) over (partition by country_short, sku_id {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows unbounded preceding) as total_active_subscriptions_partition
     from daily_join
 ), 
 
@@ -94,7 +94,7 @@ fill_values as (
 
         -- now we'll take the non-null value for each partitioned batch and propagate it across the rows included in the batch
         first_value( total_active_subscriptions ) over (
-            partition by source_relation, total_active_subscriptions_partition, country_short, sku_id order by date_day asc rows between unbounded preceding and current row) as total_active_subscriptions
+            partition by total_active_subscriptions_partition, country_short, sku_id {{ fivetran_utils.partition_by_source_relation(package_name='google_play') }} order by date_day asc rows between unbounded preceding and current row) as total_active_subscriptions
     from create_partitions
 ),