Skip to content

Conversation

@prjh2-nhs
Copy link
Contributor

Description

1. State Table Lookup Logic (processor.py)

  • Modified verify_state_record_exist() to return DataMigrationState | None instead of bool
  • Returns state object when record exists (update scenario)
  • Returns None when no state exists (insert scenario)
  • Uses ConsistentRead=True to prevent race conditions
  • Logs DM_ETL_019 when state record found

2. Pipeline Workflow (processor.py)

  • _process_service() now checks state before saving
  • Insert path: No state → continues to save records
  • Update path: State exists → exits early (update logic not yet implemented)

3. DynamoDB Deserialization (data_migration_state.py)

  • Added from_dynamodb_item() class method to deserialize DynamoDB responses
  • Added make_source_record_id() helper to standardize key format (services#{id})
  • Added error handling with logging for deserialization failures

4. Test Coverage (test_processor.py)

  • Updated all existing tests to expect None instead of False
  • Added test_verify_state_record_exist_returns_none_for_new_service() - validates insert scenario
  • Added test_verify_state_record_exist_returns_object_for_existing_service() - validates update scenario
  • Added test_update_operation_exits_early_with_state() - validates early exit behavior

Context

This PR implements the foundation for differentiating between insert and update operations in the data migration pipeline by checking a state table. Currently implements insert behaviour and early exit for updates (update logic deferred to future work).

Sensitive Information Declaration

To ensure the utmost confidentiality and protect your and others privacy, we kindly ask you to NOT including PII (Personal Identifiable Information) / PID (Personal Identifiable Data) or any other sensitive data in this PR (Pull Request) and the codebase changes. We will remove any PR that do contain any sensitive information. We really appreciate your cooperation in this matter.

  • I confirm that neither PII/PID nor sensitive data are included in this PR and the codebase changes.

Copy link
Contributor

@nhs-shruthi-gowda nhs-shruthi-gowda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work some small comment

@prjh2-nhs prjh2-nhs force-pushed the task/FTRS-1595_lookup_last_migration_state_dynamo branch from 9d83d19 to 248cf72 Compare January 5, 2026 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants