Fix pandas_to_eland append to an aliased index#829
Open
vismaytiwari wants to merge 2 commits into
Open
Conversation
When es_dest_index is an alias, indices.get_mapping keys its response by the concrete index name, so the previous response[es_dest_index] lookup raised KeyError and made appends to an aliased index fail. Verify mapping compatibility against every index the name resolves to instead, which also covers an alias spanning multiple indices. Closes elastic#747
|
💚 CLA has been signed |
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
Author
|
Gentle nudge - CLA is signed and CI/Snyk are green. Happy to change anything you'd like; otherwise it's ready for a review whenever someone has time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
pandas_to_eland(..., es_if_exists="append")raisedKeyErrorwhenes_dest_indexis an alias rather than a concrete index name.The append path verifies mapping compatibility via:
indices.get_mappingkeys its response by the concrete index name, which differs from the alias passed in. So for an alias (e.g. an ILM rollover write alias) the subscript[es_dest_index]misses and raisesKeyError: '<alias>', exactly as reported in #747.Fix
Iterate the returned mappings instead of assuming the response is keyed by
es_dest_index:This keeps the existing behavior for a concrete index (a single entry whose key equals the index), fixes the alias case, and also handles an alias that resolves to more than one index (each resolved index is verified).
Tests
Added
test_es_if_exists_append_to_aliasintests/etl/test_pandas_to_eland.py: it creates an index, points an alias at it, and appends through the alias. The test fails onmainwithKeyErrorand passes with this change.Closes #747