[WIP] Testing by wagnerlmichael · Pull Request #1048 · ccao-data/data-architecture

wagnerlmichael · 2026-06-12T16:03:42Z

No description provided.

wagnerlmichael · 2026-06-12T18:43:26Z

View diff at a glance

SELECT 
  (SELECT count(*) FROM "z_ci_add_additional_mydec_sales_to_vw_pin_sale_default"."vw_pin_sale")
  -
  (SELECT count(*) FROM "default"."vw_pin_sale")
  AS row_difference;

#	row_difference
1	9882

SELECT
  COUNT(*) AS sales_matching_missing_list,
  COUNT_IF(sale.sale_key IS NULL) AS from_injected_cte,
  COUNT_IF(sale.sale_key IS NOT NULL) AS already_in_iasworld
FROM "z_ci_add_additional_mydec_sales_to_vw_pin_sale_default"."vw_pin_sale" AS sale
INNER JOIN "ccao"."additional_mydec_sales" AS ams
  ON sale.doc_no = ams.doc_no

#	sales_matching_missing_list	from_injected_cte	already_in_iasworld
1	10,050	9,887	163

wagnerlmichael · 2026-06-12T19:21:18Z

+            WHEN md.line_5_instrument_type = 'Beneficial interest'
+                THEN '06'
+            ELSE '05'
+        END AS instrtyp,


To have the new mydec sales pass the sale_filter_deed_type gates, we grab values that the filter understands (01, 02...).

SELECT md.line_5_instrument_type, COUNT(*) AS n_all_mydec, COUNT(ams.doc_no) AS n_in_injection_list FROM "sale"."mydec" AS md LEFT JOIN "ccao"."additional_mydec_sales" AS ams ON REPLACE(md.document_number, 'D', '') = ams.doc_no GROUP BY md.line_5_instrument_type ORDER BY n_all_mydec DESC

# line_5_instrument_type n_all_mydec n_in_injection_list

1 Warranty Deed 706392 8304

2 (NULL) 396265 2

3 Trustee Deed 127690 1548

4 Special Warranty Deed 118141 1668

5 Quit Claim Deed 16292 137

6 Judicial Sale 12217 89

7 Executor Deed 8613 84

8 Administrator's Deed 6637 112

9 Deed in Trust 5330 53

10 Other 4240 44

11 Beneficial interest 1523 15

12 Guardian's Deed 932 9

13 Limited Warranty Deed 811 9

14 Judge's Deed 551 4

15 Sheriff's Deed 361 5

. . .

wagnerlmichael · 2026-06-12T19:27:29Z

+        -- Non-multisale mydec sales are single-parcel by definition
+        CAST(1 AS DECIMAL(4, 0)) AS nopar,
+        NULLIF(TRIM(md.seller_name), '') AS oldown,
+        NULLIF(TRIM(md.buyer_name), '') AS own1,


buyer and seller name technically needed for sales val outputs

wagnerlmichael · 2026-06-12T20:23:29Z

+            AND cur = 'Y'
+    ) AS ias
+        ON REPLACE(md.document_number, 'D', '') = ias.doc_no
+    WHERE NOT md.is_multisale


Multi-sale exclusion is done here, then we put '1' as nopar for compatibility

wagnerlmichael · 2026-06-12T20:38:30Z

+        FROM {{ source('iasworld', 'sales') }}
+        WHERE deactivat IS NULL
+            AND cur = 'Y'
+    ) AS ias


Match ias.doc_No onto the inner joined mydec sales and then we filter AND ias.doc_no IS NULL to remove sales that already are present in iasworld

wagnerlmichael · 2026-06-15T13:07:50Z

+        NULLIF(TRIM(md.buyer_name), '') AS own1,
+        CAST(NULL AS VARCHAR) AS saletype,
+        CAST(NULL AS VARCHAR) AS deactivat,
+        'Y' AS cur


more stub columns so that the downstream logic doesn't break

wagnerlmichael · 2026-06-15T13:43:49Z

+-- Constructing the rows here, upstream of all other logic, means the
+-- additional sales flow through the same dedupe windows and filters as
+-- iasworld sales
+sales_unioned AS (


The idea here is to put mydec sales into the iasworld structure upstream of all the filters

wagnerlmichael · 2026-06-15T15:04:20Z

+                    'Administrator''s Deed',
+                    'Guardian''s Deed'


I think this is the right filter for these but I'm not positive

wrridgeway

Okay, this is genuinely great work. Super impressed at the attention to detail you put in here, you caught stuff I probably wouldn't even have. I just have two minor requests:

Can we move removing the 'D' in ccao.additional_mydec_sales.document_number into the warehouse script like we do for the normal mydec sales and out of default.vw_pin_sale? Just for the sake of consistency.
Can we double check that we've mapped deed types as thoroughly as possible? I think using something like the query below could help investigate how it's being done currently:

select distinct sales.instrtyp,
	mydec.line_5_instrument_type
from iasworld.sales
	left join sale.mydec on sales.instruno = mydec.document_number
where sales.cur = 'Y'
	and sales.deactivat is null
	and sales.instrtyp in ('01', '02', '03', '04', '05', '06', '99')
	and trim(mydec.line_5_instrument_type) != ''
	and substr(sales.saledt, 1, 4) = '2025'
order by sales.instrtyp

There's definitely some overlap in how these map to each other, but we can just use our better judgement to decide how they should map in those cases.

Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com>

wrridgeway

This looks great! Thanks @wagnerlmichael. We may need to update the deed type mapping if Derrick informs us of anything new, but otherwise I think we're good to do.

I'll drop a bit more review here. Checking to make sure we only have row diffs for 2025:

with old as (
	select year,
		count(*) as old_count
	from default.vw_pin_sale
	group by year
),
new as (
	select year,
		count(*) as new_count
	from z_ci_add_additional_mydec_sales_to_vw_pin_sale_default.vw_pin_sale
	group by year
)
select old.*,
	new.new_count,
	abs(old.old_count - new.new_count) as diff
from old
	left join new on old.year = new.year
where abs(old.old_count - new.new_count) != 0

year	old_count	new_count	diff
2025	66131	76013	9882

Checking we're still unique by doc_no for non-multisales:

select doc_no,
	count(*)
from z_ci_add_additional_mydec_sales_to_vw_pin_sale_default.vw_pin_sale
where not is_multisale
group by doc_no
having count(*) > 1

returns nothing.

jeancochrane

A few thoughts on the dbt architecture here! I know dbt refs and sources can be confusing, so I'm happy to chat it out if it would be faster.

I suspect you may just be waiting until later on to add more context, but to be clear: Even though this work is likely temporary and won't get merged to the main branch, we should still give the PR a clear title and description so that we can find this PR in the future and remind ourselves what we did and why we did it.

jeancochrane · 2026-06-16T16:31:55Z

 LEFT JOIN mydec_sales
    ON unique_sales.doc_no = mydec_sales.doc_no
-LEFT JOIN {{ ref('sale.vw_outlier') }} AS outlier
+LEFT JOIN z_ci_add_additional_mydec_sales_to_vw_pin_sale_sale.vw_outlier AS outlier


[Question, non-blocking] When the build-and-test-dbt GitHub workflow builds a staging version of default.vw_pin_sale using the new view definition in this branch, it should automatically resolve {{ ref('sale.vw_outlier') }} to z_ci_add_additional_mydec_sales_to_vw_pin_sale_sale.vw_outlier, since dbt will be running in a CI environment on the add-additional-mydec-sales-to-vw-pin-sale branch. So I wouldn't expect you to have to make this change in order for default.vw_pin_sale to build correctly. Is that not the case here?

Ah, right. I made the z_dev_miwagne_sale.flag AS sf change first and then just edited the other two in the same way without thinking very hard. I'm sure that is how it will work when I switch it back

jeancochrane · 2026-06-16T16:38:00Z

    sf.run_id,
    sf.version
-FROM {{ source('sale', 'flag') }} AS sf
+FROM z_dev_miwagne_sale.flag AS sf


[Thought, non-blocking] In contrast to my above comment, this hardcoded schema reference is necessary in order for the build to work, for two reasons:

sale.flag is a dbt source (i.e. it's managed outside of the dbt DAG) so {{ source('sale', 'flag') }} will always resolve to the sale schema, even when dbt runs in a dev or CI environment

z_dev_miwagne_sale is a separate dev environment, so even if sale.flag were defined as a dbt model and managed in the dbt DAG, dbt would resolve {{ ref('sale.flag') }} to the z_ci_add_additional_mydec_sales_to_vw_pin_sale_sale schema, not z_dev_miwagne_sale

You may know this already, but I figured I'd drop a comment to make it extra clear!

2 is very clarifying - that the managed DAG that build-and-test-dbt uses for staging vs prod assets is a separate thing from our hardwired sales val dev environment setup. Is that roughly correct?

Yup, that's right!

jeancochrane · 2026-06-16T16:39:13Z

            'has_characteristic_change' VALUE has_characteristic_change
        ) AS review_json -- noqa: enable=CP02,RF02
-    FROM {{ ref('sale.vw_flag') }} AS flag
+    FROM z_ci_add_additional_mydec_sales_to_vw_pin_sale_sale.vw_flag AS flag


[Question, non-blocking] This ref is the same as the one I commented on in default.vw_pin_sale: In theory, ref() should resolve correctly here, so my question stands about why we need to hardcode the schema name.

Follow up question - will the ref resolve correctly pointing to the dev table even when there is no diff to the file itself?

I believe so!

wrridgeway · 2026-06-16T19:41:38Z

Just to be clear - I approved the original changes to default.vw_pin_sale, but I'm not qualified to approve any of the sales val changes that need to be made here, that's gonna be up to @jeancochrane.

wagnerlmichael added 5 commits June 12, 2026 11:02

Upload etl script

4374da2

Edit comments and path

48cbb0f

Remove distinct

cb25535

Add first draft

cb8c8e7

Add schema change

811c6a9

wagnerlmichael commented Jun 12, 2026

View reviewed changes

Remove alias

27d897a

wagnerlmichael commented Jun 15, 2026

View reviewed changes

wrridgeway requested changes Jun 15, 2026

View reviewed changes

Comment thread dbt/models/default/default.vw_pin_sale.sql Outdated

wrridgeway marked this pull request as ready for review June 15, 2026 17:22

wrridgeway requested a review from a team as a code owner June 15, 2026 17:22

wagnerlmichael and others added 4 commits June 15, 2026 12:27

Update dbt/models/default/default.vw_pin_sale.sql

173e0dd

Co-authored-by: William Ridgeway <10358980+wrridgeway@users.noreply.github.com>

Add back escape symbol

c5622ad

Rework D filter

2262897

Update filters

e96971c

wrridgeway approved these changes Jun 15, 2026

View reviewed changes

wagnerlmichael added 5 commits June 16, 2026 08:39

Switch ref for flag table

75bd952

Drop source since tables are not in dag

beec2bf

Switch ref in sale vw_outlier

87197ab

Adjust pin sale edit to dev outlier view

28cb665

Adjust tests

b4a0387

jeancochrane reviewed Jun 16, 2026

View reviewed changes

wagnerlmichael added 2 commits June 16, 2026 12:20

Switch to reg

1a5fd76

Toggle ref back to master

2ce9630

wrridgeway self-requested a review June 16, 2026 19:39

#	line_5_instrument_type	n_all_mydec	n_in_injection_list
1	Warranty Deed	706392	8304
2	(NULL)	396265	2
3	Trustee Deed	127690	1548
4	Special Warranty Deed	118141	1668
5	Quit Claim Deed	16292	137
6	Judicial Sale	12217	89
7	Executor Deed	8613	84
8	Administrator's Deed	6637	112
9	Deed in Trust	5330	53
10	Other	4240	44
11	Beneficial interest	1523	15
12	Guardian's Deed	932	9
13	Limited Warranty Deed	811	9
14	Judge's Deed	551	4
15	Sheriff's Deed	361	5
. . .

Uh oh!

Conversation

wagnerlmichael commented Jun 12, 2026

Uh oh!

wagnerlmichael commented Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wagnerlmichael Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wagnerlmichael Jun 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wagnerlmichael Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wrridgeway left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wrridgeway left a comment

Choose a reason for hiding this comment

Uh oh!

jeancochrane left a comment

Choose a reason for hiding this comment

Uh oh!

jeancochrane Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wrridgeway commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wagnerlmichael commented Jun 12, 2026 •

edited

Loading

wagnerlmichael Jun 12, 2026 •

edited

Loading

wagnerlmichael Jun 12, 2026 •

edited

Loading

wagnerlmichael Jun 15, 2026 •

edited

Loading

wrridgeway left a comment •

edited

Loading

jeancochrane Jun 16, 2026 •

edited

Loading

wrridgeway commented Jun 16, 2026 •

edited

Loading