e2e bigquerymultitable scenarios by itsmekumari · Pull Request #6 · cloudsufi/google-cloud

itsmekumari · 2023-08-08T07:43:36Z

e2e bigquerymultitable scenarios

Vipinofficial11 · 2023-08-10T06:14:59Z

src/e2e-test/features/bigquerymultitablesink/BiqQueryMultiTableSinkError.feature

+# License for the specific language governing permissions and limitations under
+# the License.
+
+@BigQueryMultiTable_Sink


We can just make the tag as BQMT_SINK.

Change done.

Vipinofficial11 · 2023-08-10T06:21:17Z

...tures/bigquerymultitablesink/MultipleDatabaseTableToBigQueryMultiTableSink_WithMacro.feature

+Feature: BigQueryMultiTable sink -Verification of BigQuery to BigQueryMultiTable successful data transfer using macros
+
+  @MULTIPLEDATABASETABLE_SOURCE_TEST
+  Scenario:Verify data is getting transferred from BigQuery to BQMT sink with all datatypes using macros


In the scenario it says we are running a pipeline from BQ to BQMT but we are selecting source as Multiple Database Tables which one is it ?

Changed to Multiple Database Tables

Vipinofficial11 · 2023-08-10T06:22:20Z

...tures/bigquerymultitablesink/MultipleDatabaseTableToBigQueryMultiTableSink_WithMacro.feature

+
+  @MULTIPLEDATABASETABLE_SOURCE_TEST
+  Scenario:Verify data is getting transferred from BigQuery to BQMT sink with split field using macros
+    Given Open Datafusion Project to configure pipeline


Same comment here related to BigQuery

Changed to Multiple Database Tables

Vipinofficial11 · 2023-08-10T06:22:58Z

...e-test/features/bigquerymultitablesink/MultipleDatabaseTableToBiqQueryMultiTableSink.feature

+    Then Validate the values of records transferred to BQMT sink is equal to the value from source MultiDatabase table
+
+  @MULTIPLEDATABASETABLE_SOURCE_DATATYPES_TEST
+  Scenario:Verify data is getting transferred from BigQuery to BQMT sink with split field


BigQuery -> Multiple Database Tables

Vipinofficial11 · 2023-08-10T06:39:35Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/locators/BigQueryMultiTableLocators.java

+  public static WebElement datasetProject;
+
+  @FindBy(how = How.XPATH, using = "//input[@data-cy='location']")
+  public static WebElement location;


Where are we using these 2 locators I don't see them in actions.

Vipinofficial11 · 2023-08-10T06:40:15Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/locators/BigQueryMultiTableLocators.java

+  @FindBy(how = How.XPATH, using = "//input[@data-cy='location']")
+  public static WebElement location;
+
+  @FindBy(how = How.XPATH, using = "//*[@data-cy='sqlStatements']//*[@data-cy='key']/input")


If this is a new property can we make a common locator out of this and add this in the framework ?

@priyabhatnagar25 A generic step is added similar type of locator for BQ framework PR, I believe we can use it.

Vipinofficial11 · 2023-08-10T06:42:36Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/BQMultiTableValidation.java

+   * @throws ClassNotFoundException If the class specified is not found.
+   */
+
+  public static boolean validateMySqlToBQRecordValues(String sourceTable) throws IOException, InterruptedException,


So we are using Multiple Database table plugin here but we are using a SQL Database/Table. It would be good if we can add few comments and context here describing this in this class.

Vipinofficial11 · 2023-08-10T06:43:39Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/BQMultiTableValidation.java

+   * @throws InterruptedException If the current thread is interrupted while waiting.
+   */
+
+  public static List<String> getTableByName() throws IOException, InterruptedException {


change the method name to getTableNameFromMySQL()

change done.

Vipinofficial11 · 2023-08-10T06:44:33Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/BQMultiTableValidation.java

+  public static List<String> getTableByName() throws IOException, InterruptedException {
+    List<String> tableNames = new ArrayList<>();
+    List<String> targetTableNames = Arrays.asList(tabA, tabB);
+    TableResult tableResult = getTableNamesFromDataSet();


Name it as getTableNamesFromBQDataSet()

Change done

bharatgulati · 2023-08-10T16:08:07Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/BQMultiTableValidation.java

+          case Types.VARCHAR:
+            String sourceString = rsSource.getString(currentColumnCount);
+//                        String sour = bigQueryData.get(jsonObjectIdx).get(columnTypeName).getAsString();
+//                        String sou2 = bigQueryData.get(jsonObjectIdx).get(columnType)


Remove commented line no. 155 & 156.

poojantcs · 2023-08-11T10:27:51Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/actions/BigQueryMultiTableActions.java

+    SeleniumHelper.getPropertiesLocators(BigQueryMultiTableLocators.class);
+  }
+
+    public static void enterSqlStatement(String value) {


Move the locator and the corresponding action to the framework repo, if it's not already there.

poojantcs · 2023-08-11T10:38:52Z

src/e2e-test/java/io/cdap/plugin/bigquerymultitable/stepdesign/MultiTableClient.java

+  public static void createSourceDatatypesTable(String sourceTable) throws SQLException, ClassNotFoundException {
+    try (Connection connect = getMysqlConnection(); Statement statement = connect.createStatement()) {
+      String createTableQuery = "CREATE TABLE " + sourceTable +
+        "(EmployeeID int, LastName varchar(255), City varchar(255), tablename varchar(255))";


Can we make this parameterized?

updated correct conn string, removed space in yml, removed verify goal in pom. changed sql connector java version to run in github actions and bqmt secret updated

itsmekumari requested a review from Vipinofficial11 August 8, 2023 07:43

Vipinofficial11 requested review from bharatgulati and removed request for bharatgulati August 8, 2023 11:13

Vipinofficial11 reviewed Aug 10, 2023

View reviewed changes

bharatgulati reviewed Aug 10, 2023

View reviewed changes

poojantcs reviewed Aug 11, 2023

View reviewed changes

itsmekumari assigned priyabhatnagar25 Aug 11, 2023

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 7 times, most recently from 79d82d2 to cd2286e Compare August 24, 2023 11:46

Vipinofficial11 force-pushed the e2e_bigquerymultitable branch from cd2286e to c5e3721 Compare August 24, 2023 17:43

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch from c5e3721 to e538b69 Compare August 28, 2023 07:25

bharatgulati force-pushed the e2e_bigquerymultitable branch 3 times, most recently from 89e0b0b to f0c2b16 Compare September 1, 2023 05:23

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 2 times, most recently from bc7b626 to d2c87d4 Compare September 1, 2023 06:44

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch from d2c87d4 to ba19070 Compare September 1, 2023 10:59

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch from ba19070 to 75665bd Compare October 6, 2023 16:35

neerajsinghal05 force-pushed the e2e_bigquerymultitable branch from f9e4123 to 8b2ad66 Compare November 16, 2023 11:34

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 3 times, most recently from 4c9db2e to 74e2152 Compare November 22, 2023 15:38

priyabhatnagar25 force-pushed the e2e_bigquerymultitable branch 8 times, most recently from 170af4a to 568fff4 Compare December 5, 2023 10:43

rahuldash171 force-pushed the e2e_bigquerymultitable branch 6 times, most recently from d808e6f to 0d56779 Compare December 21, 2023 08:59

priyabhatnagar25 added 2 commits January 17, 2024 09:30

BQMT e2e Tests Scenarios

3579401

Removed Issue tag Plugin-1669 in Runtime scenarios and test runner file,

9740cc8

updated correct conn string, removed space in yml, removed verify goal in pom. changed sql connector java version to run in github actions and bqmt secret updated

Vipinofficial11 force-pushed the e2e_bigquerymultitable branch from 54d7577 to ae12e20 Compare January 17, 2024 04:02

Debug runner path failures for BQMT

f8611c0

Vipinofficial11 force-pushed the e2e_bigquerymultitable branch from ae12e20 to f8611c0 Compare January 17, 2024 04:11

NavdeepCS force-pushed the develop branch 4 times, most recently from fc2efdf to 5f4f137 Compare March 1, 2024 06:20

vikasrathee-cs force-pushed the develop branch from 5f4f137 to 008f3a8 Compare March 1, 2024 06:27

Conversation

itsmekumari commented Aug 8, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants