Skip to content

Add Kroxylicious lab: shadow linking through a Kafka protocol proxy - #308

Open
pmw-rp wants to merge 1 commit into
redpanda-data:mainfrom
pmw-rp:kroxylicious-shadow-linking
Open

pmw-rp wants to merge 1 commit into
redpanda-data:mainfrom
pmw-rp:kroxylicious-shadow-linking

Conversation

@pmw-rp

@pmw-rp pmw-rp commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Adds a Kubernetes lab that routes a Redpanda shadow link through a Kroxylicious virtual cluster instead of reading the source cluster directly.

A small custom Kroxylicious filter presents every topic and consumer group under a p_ prefix, without renaming anything on the source cluster. A topic created as basic on the source arrives on the shadow cluster as p_basic; a consumer group created as consumer-group-foo arrives as p_consumer-group-foo. Because the shadow cluster ends up healthy and consistent with those renamed topics and groups intact — and fails over correctly — the lab doubles as proof that shadowing works end to end through a Kafka protocol proxy.

Based on https://github.com/pmw-rp/kroxy-linking-demo.

Where it lives

kubernetes/kroxylicious-shadow-linking/, next to kubernetes/shadow-linking, which this is a variant of. It deploys via kind + Helm + the Redpanda operator, so it belongs under kubernetes/ rather than docker-compose/.

Conventions

  • README.adoc at the lab root with the required header attributes (:page-layout: lab, :env-kubernetes: true, :page-categories:, :description:) and the same section shape as the sibling lab, plus a Limitations section covering the filter's deliberate gaps.
  • Categories are Disaster Recovery, Data Replication, Integration — all three already in use elsewhere in the repo.
  • The rpk commands are inline in the README rather than wrapper scripts, following the simplification made to kubernetes/shadow-linking in Add shadow linking lab for Kubernetes disaster recovery #284. The source demo's numbered demos/*.sh scripts are not carried over.
  • setup.sh follows the sibling lab's style (set -e, progress echoes, quiet Helm), with the proxy image build and kind load steps added.
  • The filter's own docs are converted from Markdown to AsciiDoc.
  • docs/modules/kubernetes/pages/kroxylicious-shadow-linking.adoc symlink added so Antora publishes the page.
  • No test-docs.yml wiring: the Kubernetes labs aren't covered there, since they need a cluster.

All four AsciiDoc files render clean under asciidoctor --failure-level=WARN.

Points for review

  • Sample data reused from the sibling lab. resources/syslog.avsc and resources/syslogs.json are copied from kubernetes/shadow-linking rather than from the source demo, which ships 1000 generated records with real-looking public IPs. The repo's version is 3 readable records using RFC 5737 documentation addresses.
  • Two fixes to the demo content. The shadow-side Schema Registry URL had a stray b. in its hostname (redpanda.shadow.b.svc.cluster.local). And resources/kind.yaml now names the cluster redpanda-kroxy, with kind load docker-image --name and the clean-up step targeting it — previously kind load would have silently targeted a cluster named kind.
  • kubernetes/README.adoc was stale, listing only gitops-helm. This adds entries for iceberg and shadow-linking alongside the new lab, matching how docker-compose/README.adoc indexes everything. Happy to drop those two lines if you'd rather keep the diff to the new lab only.
  • Versions. The Redpanda image and operator are pinned at v26.2.2, which is what the demo was validated against, versus v25.3.4 / v25.3.1 in kubernetes/shadow-linking.
  • The filter is demo-quality, documented as such in both the lab's Limitations section and the filter's own README: producing through the proxy, transactional producers, DeleteGroups and the KIP-848 consumer group protocol are all out of scope. kroxylicious/topic-prefixer/consumer-group-renaming.adoc records the wire-level tracing behind the consumer group half, including the finding that shadow linking correlates topics by ID rather than by name.
  • resources/reverse.yaml from the demo is deliberately left out: nothing references it, it isn't documented, and its comments are mislabeled.

🤖 Generated with Claude Code

Add a Kubernetes lab that routes a Redpanda shadow link through a
Kroxylicious virtual cluster instead of reading the source cluster
directly. A small custom filter presents every topic and consumer group
under a "p_" prefix, so the rename is visible end to end on the shadow
cluster while the source cluster keeps its real names. A healthy shadow
cluster is also proof that shadowing works through a Kafka proxy.

Structure follows kubernetes/shadow-linking, which this extends:

- README.adoc with the lab attributes, the rpk commands inline rather
  than as wrapper scripts, and Limitations covering the filter's
  deliberate gaps
- setup.sh in the same style, plus the image build and kind load steps
  the proxy needs
- The filter's own docs converted from Markdown to AsciiDoc
- resources/syslog.avsc and syslogs.json reused from the sibling lab, so
  the sample records stay readable and use documentation IP ranges
- docs/modules symlink so Antora publishes the page
- kubernetes/README.adoc entries for this lab and for the two existing
  labs the index had not yet picked up

Two fixes carried over from the source demo: the shadow-side Schema
Registry URL had a stray "b." in its hostname, and the kind cluster is
now named, so `kind load docker-image` and the clean-up step can target
it unambiguously.

Based on https://github.com/pmw-rp/kroxy-linking-demo

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@netlify

netlify Bot commented Sep 25, 2026

Copy link
Copy Markdown

👷 Deploy request for redpanda-labs-preview pending review.

Visit the deploys page to approve it

Name Link
🔨 Latest commit df43949

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant