feat: standardize silver/gold field names, fix dashboard KPIs, add total_orders#34
Merged
Merged
Conversation
andre-salvati
added a commit
that referenced
this pull request
Jun 5, 2026
…tal_orders
Drop ds_kpi dataset from the dashboard — KPI counters (Total Value, Total Orders,
Number of Customers) now bind to ds_orders so all five filters update them.
Number of Orders removed (no order_id in gold layer).
Standardize field names across silver (curated.order_enriched) and gold
(report.order_agg) per three rules: {entity}_id suffix, entity-qualified names,
item_* prefix for item-level fields, no abbreviations. date cast to DateType in
silver so the type coercion lives in the right layer.
Add total_orders (COUNT DISTINCT order_id) to report.order_agg and a third KPI
tile on the dashboard.
commonSchemas.py now defines order_enriched_schema and order_agg_schema as the
canonical schemas for silver and gold; tests import from there instead of
inlining StructType definitions.
Also fix pre-existing unit_test_sdp.py bug where df_order tuples were missing
product_id and prod_category_id (4 values vs 6-field schema).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Qualify df_order["date"] in enrich_order() joins (was unqualified F.col("date"),
inconsistent with all other explicit DataFrame refs in the same select)
- Remove unused pyspark.sql.types import block in unit_test_sdp.py
- Add total_orders=40 assertion to _validate_load_test; fix stale error message
that still referenced old column name total_qty
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
andre-salvati
force-pushed
the
feat/field-standardization-and-dashboard-kpi
branch
from
4a3b244 to
48b4bb3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ds_kpidashboard dataset; KPI counters (Total Value, Total Orders, Number of Customers) now bind tods_ordersso all five filters update them. Added a third KPI tile for Total Orders.curated.order_enriched) and gold (report.order_agg):{entity}_idsuffix, entity-qualified names,item_*prefix for item-level fields, no abbreviations;datecast toDateTypein silver. Addedorder_enriched_schemaandorder_agg_schematocommonSchemas.pyas the canonical schemas for those layers.total_orders(COUNT DISTINCT order_id) toreport.order_aggand the dashboard.Test plan
make test)make run env=dev)prod.report.order_aggandprod.report.order_agg_sdpdropped and recreated viamake deploy env=prod+ prod integration rundf_order["date"]in join select, removed unused imports, addedtotal_ordersload-test assertion, fixed stale error message)🤖 Generated with Claude Code