DataJoint 2.0 Release Plan

[dimitri-yatsenko]

As we approach the final stages of development, we are excited to announce DataJoint 2.0, our first major release designed to align with the official DataJoint Specs 2.0

Target release date: Jan 2026

Tested Python Versions: Python 3.10-3.13

---

The centerpiece of this release is the introduction of Object-Augmented Schemas. We are also delivering a fully modernized framework for project configuration, data handling, and new scientific datatype support.

Compatibility Note: While we aim for data compatibility (meaning no data migration should be required), this release will introduce several backward-incompatible API features. Please plan to update your code to leverage the new functionalities.

---

Object-Augmented Schemas

DataJoint 2.0 fundamentally revises how relational tables are combined with large-scale scientific data objects. This new paradigm allows for more flexible and efficient handling of complex data types directly within the data pipeline.

• Core Implementation: Foundational support for Object-Augmented Schemas. FEAT: Object-augmented schemas -- File Type #1259
• Custom Data Type Plugins: An extensible architecture for user-defined data types. FEAT: Custom Types 2.0 #1260
  • <dj_blob>: A plugin for backward-compatible DataJoint blobs. FEAT: dj_blob type plugin for DataJoint 2.0 #1262
  • <dj_zarr>: A plugin for general Zarr 3.0 array support FEAT: Implement <dj_zarr> Custom Type Adaptor #1263
  • <multiphoton>: plugin for handling compressed, photon-limited movies (based on zarr)

Data Export & Import

• Diagram-Based Restriction: Use data diagrams to visually define and restrict data for queries (FEAT: Applying Restrictions to a Diagram #865).
• Workflow Publishing: Streamlined, diagram-based tools for exporting and publishing datasets.
• Database Restore: Functionality to restore a database state from an export.

AutoPopulate 2.0. #1258

This is a redesigned, scalable job management engine to replace the existing autopopulate framework.

• Per-Table Job Queues: Each computed table gets its own dedicated jobs table, accessible via Table.jobs.
• Direct Key Addressing: Jobs are identified by the primary key of the main table, not by hashes.
• Improved Job Lifecycle Management: A job queue with statuses like pending, reserved, and error, and an explicit refresh() method to update the queue.
• Job Prioritization: A new priority field allows for controlling the execution order of jobs.
• Enhanced Performance: Reduces database strain by eliminating frequent and expensive key_source queries from multiple workers.

Project Configuration & Extensibility

• Unified Management: A single system for handling secrets, config files, and environment variables. (IMPR: Modernize Project Configuration Management #1257)
• Modern Plugin Architecture: A new framework for enhancing and extending core DataJoint functionality. (IMPR: Upgrade plugin architecture #1251)

Schema Definition & Data Types

• Scientific Datatype Aliases: Support for common types like float32, float64, uint8, int8, uint16, int16, uint32, int32, uint64, and int64. FEAT: Core attributes types 2.0 #1261

Version 2.1 (Post 2.0 Release)

The following features are planned for a subsequent release shortly after 2.0, with a possibility of being included in the main release.

• PostgreSQL Support
• Native JSON Datatype Support (2.0) -- Postgres compatible

[dimitri-yatsenko]

Preparing for release in early January.

Branch Summary: claude/spec-issue-1243-YvqmF vs master

Total Changes: 111 files changed, ~17,200 insertions, ~3,300 deletions across 95 commits

---

1. Autopopulate 2.0 Jobs System (Major Feature)

Files: src/datajoint/jobs.py (502 lines new), docs/src/design/autopopulate-2.0-spec.md, tests/test_jobs.py

A complete redesign of the job handling system:

• Per-table job tables instead of schema-wide ~jobs table
• Rich status tracking: pending, reserved, success, error, ignore (vs old reserved/error/ignore)
• Full lifecycle visibility: created_time, scheduled_time, reserved_time, completed_time, duration
• No FK constraints on jobs tables for performance; stale jobs cleaned by refresh()
• API: MyTable.jobs.refresh(), MyTable.jobs.reserve(), MyTable.jobs.complete()
• Jobs table auto-dropped when parent table is dropped

---

2. Object Column Type (Major Feature)

Files: src/datajoint/objectref.py, src/datajoint/storage.py (775 lines), src/datajoint/staged_insert.py, docs/src/design/tables/object-type-spec.md (1,293 lines), tests/test_object.py (852 lines)

New unified storage backend for managed file/folder storage:

• Augmented Schema (AUS) paradigm: DataJoint fully owns object lifecycle
• fsspec-based backends: local, S3, GCS, Azure
• Two insert modes: Copy (small files) and Staged (large objects, Zarr, TileDB)
• Immutable after finalization; deletion only via record delete
• Partition patterns: Configurable path structure with {attribute} placeholders
• Store metadata (datajoint_store.json) for client verification

---

3. Settings System Modernization

Files: src/datajoint/settings.py (633 lines new), docs/src/client/settings.md

Complete refactoring using pydantic-settings:

• Recursive config search: Walks up directory tree to find datajoint.json
• Secrets separation: .secrets/ directory for sensitive values
• Environment variables: DJ_DATABASE_HOST, DJ_OBJECT_STORAGE_*, etc.
• Removed: dj.config.save_local(), dj.config.save_global(), set_password() functions

---

4. AttributeType System (Replaces AttributeAdapter)

Files: src/datajoint/attribute_type.py (531 lines), src/datajoint/migrate.py (250 lines)

• New <djblob> type for serialized Python objects (vs longblob for raw bytes)
• DJBlobType and migration utilities for blob columns
• Migration path from legacy blob types to new system

---

5. Type Aliases

Files: src/datajoint/declare.py, tests/test_type_aliases.py (184 lines)

New SQL type aliases for convenience:

Category	Aliases
Numeric	int8, int16, int32, int64, uint8, uint16, uint32, uint64, float32, float64
Boolean	bool → tinyint(1)

---

6. Code Modernization

Change	Details
Src layout	Code moved from datajoint/ to src/datajoint/
Minimum versions	Python 3.10+, MySQL 8.0+
Linting	Ruff formatter/linter replaces older tools
Dependencies	Added pydantic-settings, fsspec; updated pixi.lock
Pre-commit	Modernized configuration

---

7. Testing Infrastructure

Files: tests/conftest.py (major refactor), new tests/schema_object.py, tests/schema_type_aliases.py

• Pytest-managed Docker containers for database startup
• Refactored fixtures for cleaner test isolation
• New test files: test_object.py, test_type_aliases.py, test_attribute_type.py

---

Documentation Updates

Type	Files
New specs	autopopulate-2.0-spec.md, object-type-spec.md
Updated	credentials.md, settings.md, customtype.md, attributes.md
New guides	docs/src/design/tables/object.md, docs/src/manipulation/insert.md additions

---

Summary: This branch represents a major evolution of DataJoint Python, introducing a modern jobs system for distributed computing, a unified object storage layer, modernized settings management, and significant code quality improvements with Python 3.10+ baseline.