Update data from 2024-01-16 (json, links, misc, schemas, txt, table_of_contents)

[herzberg]

Note: This does not include any updates to cltk-flat or cltk-full.

In order to make the upload smoother, I created a short script to help commit/upload piece by piece.

topCategories = ['Chasidut', 'Halakhah', 'Jewish Thought', 'Kabbalah',
                 'Liturgy', 'Midrash', 'Mishnah', 'Musar', 'Reference',
                 'Responsa', 'Second Temple', 'Talmud', 'Tanakh', 'Tosefta']

categoryDirs = ['json',
                # 'cltk-full', 'cltk-flat',
                'txt',]

otherDirs = ['misc', 'schemas', 'links', '.']

def gitLine(name: str):
    print('git add "%s" && git commit -m "update %s" && git push' % (name, name))

def main():
    for categoryDir in categoryDirs:
        for category in topCategories:
            name = '%s/%s' % (categoryDir, category)
            gitLine(name)
    for d in otherDirs:
        gitLine(d)

main()

Which provided these commands:

git add "json/Chasidut" && git commit -m "update json/Chasidut" && git push
git add "json/Halakhah" && git commit -m "update json/Halakhah" && git push
git add "json/Jewish Thought" && git commit -m "update json/Jewish Thought" && git push
git add "json/Kabbalah" && git commit -m "update json/Kabbalah" && git push
git add "json/Liturgy" && git commit -m "update json/Liturgy" && git push
git add "json/Midrash" && git commit -m "update json/Midrash" && git push
git add "json/Mishnah" && git commit -m "update json/Mishnah" && git push
git add "json/Musar" && git commit -m "update json/Musar" && git push
git add "json/Reference" && git commit -m "update json/Reference" && git push
git add "json/Responsa" && git commit -m "update json/Responsa" && git push
git add "json/Second Temple" && git commit -m "update json/Second Temple" && git push
git add "json/Talmud" && git commit -m "update json/Talmud" && git push
git add "json/Tanakh" && git commit -m "update json/Tanakh" && git push
git add "json/Tosefta" && git commit -m "update json/Tosefta" && git push
git add "txt/Chasidut" && git commit -m "update txt/Chasidut" && git push
git add "txt/Halakhah" && git commit -m "update txt/Halakhah" && git push
git add "txt/Jewish Thought" && git commit -m "update txt/Jewish Thought" && git push
git add "txt/Kabbalah" && git commit -m "update txt/Kabbalah" && git push
git add "txt/Liturgy" && git commit -m "update txt/Liturgy" && git push
git add "txt/Midrash" && git commit -m "update txt/Midrash" && git push
git add "txt/Mishnah" && git commit -m "update txt/Mishnah" && git push
git add "txt/Musar" && git commit -m "update txt/Musar" && git push
git add "txt/Reference" && git commit -m "update txt/Reference" && git push
git add "txt/Responsa" && git commit -m "update txt/Responsa" && git push
git add "txt/Second Temple" && git commit -m "update txt/Second Temple" && git push
git add "txt/Talmud" && git commit -m "update txt/Talmud" && git push
git add "txt/Tanakh" && git commit -m "update txt/Tanakh" && git push
git add "txt/Tosefta" && git commit -m "update txt/Tosefta" && git push
git add "misc" && git commit -m "update misc" && git push
git add "schemas" && git commit -m "update schemas" && git push
git add "links" && git commit -m "update links" && git push
git add "." && git commit -m "update ." && git push

[herzberg]

Please double check my work

[herzberg]

I've spot checked it, and I think it should be safe to merge

[herzberg]

Derekh HaShem vs. Derekh Hashem

Small issue that is not a big deal, but I figured its worth mention since I saw it.
The table_of_contents.json has Derekh Hashem which seems to be what Sefaria's website uses as the primary name.
while the path uses: json/Jewish Thought/Acharonim/Derekh HaShem (capital S)

[yodem]

Automated code review

This is an automated review (Claude), run by a Sefaria maintainer as part of a sweep through our long-open pull requests. A human is reading the results.

---

@herzberg — thank you for this, and I want to acknowledge plainly that no maintainer ever replied to you. Every comment on this thread is your own, including "Please double check my work." That shouldn't have happened, and I'm sorry.

The work itself was careful. You re-exported the corpus as of 2024-01-16, split it into reviewable per-category commits rather than one unreviewable blob, spot-checked the result, and flagged a genuine naming inconsistency (Derekh HaShem vs Derekh Hashem). We checked — that inconsistency is resolved in the current export.

Why it can no longer be merged. Two things happened while this was open:

1. In March 2026 the repository stopped holding bulk text data. The json/, txt/, links/, misc/ and schemas/ directories were removed. The texts now live in a public Google Cloud bucket (gs://sefaria-export/) regenerated monthly from our production database.
2. The manual export you were doing by hand is now an automated monthly job, and the January 2024 data has been superseded many times over.

The PR is also already in conflict with master on its own terms, independent of anything we're doing now.

So this is genuinely obsolete — but through a change of architecture on our side, not through any fault in your work.

We'd like to hear what you'd prefer rather than simply closing a PR that we left unanswered for two and a half years. Please join our developer Discord if you'd like to pick this up with us:

👉 https://sefaria.formstack.com/forms/sefaria_developer_discord_community

Thanks for the effort, and again — sorry for the silence.

[yodem]

Follow-up: this PR was closed automatically, not by a decision

Shortly after the review above, we reset this repository's git history — moving ~14 GB of accumulated history into a read-only archive so a clone drops from ~14 GB to under 1 MB.

That force-push auto-closed this PR. We did not expect it to: we believed GitHub only auto-closes a pull request when its base branch is deleted, and we rewrote master rather than deleting it. That was wrong — GitHub also closes a PR when the base branch's history is rewritten so the merge base no longer exists.

I'm flagging it because the comment above said we wanted to hear from you "rather than close this on you after so long" — and then a migration closed it minutes later. That's a bad look and it wasn't the intent. We got the mechanics wrong; we didn't change our minds.

Nothing of yours is lost. Your fork is untouched and your commits are intact, and the complete pre-reset history is preserved at Sefaria-Export-Archive.

If you'd like this reopened to continue the conversation, say so and we'll reopen it — the diff view will be unusable (no common ancestor with the new root commit), but the thread still works.

👉 https://sefaria.formstack.com/forms/sefaria_developer_discord_community

[yodem]

Hi — a note on where this stands.

We had to make some changes to this repository (moving ~14 GB of git history into a read-only archive), and that's what closed your PR. Reopening isn't possible — GitHub blocks it because the merge base no longer exists.

We're not asking you to re-submit this one, and that's not a judgement on the work: as noted in the review above, the issue you reported is already resolved in our current published data, so a new PR would have nothing left to fix. You were right; it just got fixed by another route while this sat unanswered.

Please do join our developers Discord, though — we'd rather hear about the next thing you spot somewhere it will actually get seen:

👉 https://sefaria.formstack.com/forms/sefaria_developer_discord_community

Thank you for the contribution, and sorry for how long it went without a reply.