Skip to content

Conversation

@vojtabiberle
Copy link
Contributor

Release Notes

Add optional table data sampling to kbc llm export command with new flags --with-samples, --without-samples, --sample-limit, and --max-samples.

Plans for customer communication

None.

Impact analysis

This PR extends the kbc llm export command with:

  • Table data sampling via Storage API's data-preview endpoint
  • New CLI flags for controlling sample behavior
  • Security module scaffolding (public repo detection, secret encryption) for future use
  • CSV writer for sample output

The security module is implemented but not yet wired into the main export flow - it provides infrastructure for future PRs.

Change type

Feature

Justification

Part of the LLM twin format export feature (PR 6 of 7). Implements sample data export capability per RFC requirements.

Linear issue: DMD-922

Human Review Checklist

  • Verify JSON key casing inconsistency is acceptable: samples/index.json uses camelCase (tableId) while samples/{table}/metadata.json uses snake_case (table_id)
  • Confirm security module not being integrated into operation.go is expected (scaffolding for future)
  • Review sample limit defaults (100 rows per table, max 1000; 50 tables max)

Deployment

Merge & automatic deploy.

Rollback plan

Revert of this PR.

Post release support plan

None.


Link to Devin run: https://app.devin.ai/sessions/eecace21ad294aaabf3504f33fff40dc
Requested by: [email protected] (@vojtabiberle)

@linear
Copy link

linear bot commented Dec 19, 2025

@devin-ai-integration
Copy link
Contributor

devin-ai-integration bot commented Dec 19, 2025

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 8fd0a2a to 60a1ec7 Compare January 5, 2026 21:33
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from f8a3858 to 0e11410 Compare January 5, 2026 21:34
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 60a1ec7 to 6c8da70 Compare January 6, 2026 13:15
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 0e11410 to fb8b1b3 Compare January 6, 2026 13:15
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 6c8da70 to 2ca7d49 Compare January 6, 2026 17:18
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from fb8b1b3 to abdbe8f Compare January 6, 2026 17:19
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 2ca7d49 to 09f18af Compare January 6, 2026 19:00
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from abdbe8f to 4dc2192 Compare January 6, 2026 19:01
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 09f18af to ff88f73 Compare January 6, 2026 23:04
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch 3 times, most recently from 0eb3386 to cac5bcf Compare January 6, 2026 23:59
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 67c271f to 8dc0355 Compare January 7, 2026 00:14
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from cac5bcf to faaad27 Compare January 7, 2026 00:16
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 8dc0355 to 8a3a7a2 Compare January 7, 2026 00:21
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from faaad27 to 4b6ce89 Compare January 7, 2026 00:22
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 8a3a7a2 to 0ed951b Compare January 7, 2026 00:24
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch 2 times, most recently from 0a8aa40 to 07790e0 Compare January 7, 2026 00:31
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 0a816df to a98c3ae Compare January 7, 2026 00:46
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from c9c4bd4 to 7a8c89b Compare January 7, 2026 00:46
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from a98c3ae to 3c2a2e9 Compare January 7, 2026 00:56
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 7a8c89b to 0f454ba Compare January 7, 2026 00:56
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 3c2a2e9 to 99af6b7 Compare January 7, 2026 01:02
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 0f454ba to bd8a583 Compare January 7, 2026 01:02
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 99af6b7 to 1cdd31a Compare January 7, 2026 01:09
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from bd8a583 to f1f2c61 Compare January 7, 2026 01:09
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from ddb7978 to 029e073 Compare January 8, 2026 19:50
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 94c04b2 to e2c2296 Compare January 8, 2026 19:50
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 029e073 to 1dcec8e Compare January 8, 2026 20:06
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from e2c2296 to a3ee0b0 Compare January 8, 2026 20:06
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 1dcec8e to 9309d99 Compare January 8, 2026 20:10
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from a3ee0b0 to 390a686 Compare January 8, 2026 20:10
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 9309d99 to ab1e155 Compare January 8, 2026 20:14
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 390a686 to de5829b Compare January 8, 2026 20:14
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from ab1e155 to 7653daf Compare January 8, 2026 20:26
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from de5829b to d356a4c Compare January 8, 2026 20:26
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 7653daf to fea7e7a Compare January 8, 2026 20:41
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from d356a4c to e78dda4 Compare January 8, 2026 20:41
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from fea7e7a to a4bafec Compare January 8, 2026 20:55
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from e78dda4 to 3160b11 Compare January 8, 2026 20:55
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from a4bafec to 3984869 Compare January 8, 2026 23:56
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 3160b11 to f24dcf7 Compare January 8, 2026 23:59
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 3984869 to efe95bb Compare January 9, 2026 00:37
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from f24dcf7 to e585952 Compare January 9, 2026 00:37
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from efe95bb to 3a65c44 Compare January 9, 2026 01:17
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from e585952 to 77f1db7 Compare January 9, 2026 01:17
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from 3a65c44 to a2418fa Compare January 9, 2026 01:44
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 77f1db7 to 7ae1037 Compare January 9, 2026 01:44
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from a2418fa to f56e998 Compare January 9, 2026 01:50
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from 7ae1037 to d4e416b Compare January 9, 2026 01:50
devin-ai-integration bot and others added 4 commits January 9, 2026 09:34
- Add security.go with public repo detection and secret encryption
- Add writer/csv.go for CSV sample output
- Extend fetcher.go with FetchTableSample and FetchTableSamples methods
- Extend generator.go with GenerateSamples, generateSamplesIndex, generateSampleFile methods
- Add sample flags to export command (--with-samples, --without-samples, --sample-limit, --max-samples)
- Update operation.go to integrate sample fetching and generation
- Update options.go with sample-related options and helper methods

Co-Authored-By: [email protected] <[email protected]>
- Add TestEncryptSecrets_SimpleMap for basic secret detection
- Add TestEncryptSecrets_NestedMap for nested structure handling
- Add TestEncryptSecrets_SliceWithMaps for array processing
- Add TestEncryptSecrets_DeeplyNested for recursive encryption
- Add TestEncryptSecrets_EmptyMap and TestEncryptSecrets_NoSecrets edge cases
- Add TestDefaultSecurityOptions for configuration defaults

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
The default behavior is already no samples, so --without-samples
is redundant. Simplified to just --with-samples flag.
- Added DataQuality type with completeness, null counts, and distinct counts
- Added computeDataQuality function to analyze sample data
- Included data quality indicators in sample metadata output

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@vojtabiberle vojtabiberle force-pushed the vb/DMD-921/llm-export-generator branch from f56e998 to 3c18b07 Compare January 9, 2026 08:34
@vojtabiberle vojtabiberle force-pushed the vb/DMD-922/llm-export-samples-security branch from d4e416b to 13270ab Compare January 9, 2026 08:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant