[PDI-20034] - PDI Logging for Sqoop missing "File System Counters and Job Counters info

[joao-p-pereira]

Depends on pentaho/pentaho-hadoop-shims#1909

Fixes missing File System Counters and Job Counters in Sqoop job logs. Sqoop now resolves HadoopClientServices before attaching logging and registers the existing filtered Kettle appenders in the selected shim’s LoggerContext

The most important changes are:

Logging integration and class loader support:

• Refactored HadoopExecutionLogging and AbstractSqoopJobEntry to support attaching log appenders to logger contexts associated with a specific ClassLoader, ensuring logging works correctly with Hadoop shims and other plugin class loaders. This includes new overloads and logic to select the correct logger context. [1] [2] [3] [4]

• Updated the logic in executeSqoop to defer attaching log appenders until after the Hadoop client service is loaded, so the correct class loader is used for logger context attachment. [1] [2]

Dependency updates:

• Upgraded the commons-cli dependency from version 1.2 to 1.5.0 in pmr-libraries/pom.xml for improved compatibility and bug fixes, The PMR runtime now packages Commons CLI 1.5.0, providing the Option.builder(String) API required by Apache Vanilla Sqoop

Testing enhancements:

• Added a new unit test to verify that logging appenders are correctly attached to the logger context of a custom (shim) class loader, ensuring the new behavior is covered.

These changes improve the reliability and flexibility of logging when running Sqoop jobs in environments with custom or multiple class loaders.

[Copilot]

Pull request overview

Improves Sqoop job logging integration in PDI when running under Hadoop shims by attaching Kettle log appenders to the Log4j2 LoggerContext associated with a specific shim ClassLoader, ensuring Hadoop/Sqoop counters (e.g., File System Counters / Job Counters) appear in job logs.

Changes:

• Add classloader-aware logger-context selection in HadoopExecutionLogging and AbstractSqoopJobEntry when attaching log appenders.
• Defer Sqoop logging attachment until after HadoopClientServices is resolved to use the shim’s classloader.
• Upgrade commons-cli in the PMR libraries assembly and add a unit test covering shim logger-context attachment.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
kettle-plugins/sqoop/core/src/main/java/org/pentaho/big/data/kettle/plugins/sqoop/AbstractSqoopJobEntry.java	Adds classloader-aware log attachment and defers attachment until the shim service is loaded.
kettle-plugins/common/ui/src/main/java/org/pentaho/big/data/kettle/plugins/logging/HadoopExecutionLogging.java	Adds an overload to attach appenders using a LoggerContext resolved from a given ClassLoader.
kettle-plugins/sqoop/core/src/test/java/org/pentaho/big/data/kettle/plugins/sqoop/AbstractSqoopJobEntryTest.java	Adds a test ensuring appenders are attached to a shim classloader’s logger context.
assemblies/pmr-libraries/pom.xml	Bumps commons-cli from 1.2 to 1.5.0 for runtime compatibility.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[buildguy]

Note:

---

Frogbot also supports Contextual Analysis, Secret Detection, IaC and SAST Vulnerabilities Scanning. This features are included as part of the JFrog Advanced Security package, which isn't enabled on your system.

---

🐸 JFrog Frogbot

[buildguy]

✅ Build finished in 9m 55s

Build command:

mvn clean verify -B -e -Daudit -Djs.no.sandbox

---

👌 All tests passed!

Tests run: 1122, Failures: 0, Skipped: 17    Test Results

---

ℹ️ This is an automatic message