Synq: High-Performance Database Migration Tool

Synq is a blazing-fast, single-binary database migration tool engineered for zero-downtime workflows. Written in Go, it supports PostgreSQL, MySQL, SQLite, and MongoDB with automatic dependency resolution, resilient failure handling, and zero intermediate disk usage.

Latest Performance: Benchmarked at 311,234 rows/second (MySQL → PostgreSQL)

Why Synq?

Traditional tools like pg_dump or mysqldump rely on slow, disk-heavy intermediate files and often fail completely on a single error. Synq is different:

Streaming Pipeline: Data flows through fixed-size memory buffers. No files = no I/O bottlenecks
Intelligent Ordering: Automatically resolves Foreign Key dependencies (Topological Sort) to prevent constraint violations
Resilient: A "Dead Letter Queue" (DLQ) captures failed rows without stopping the migration. 99.9% of your data arrives even if 0.1% is bad
Security Audit: Built-in PII scanner to detect sensitive data (Emails, Credit Cards, SSNs) before it moves
Zero Dependencies: A single, statically compiled binary that runs anywhere
Multi-Database Support: PostgreSQL, MySQL, SQLite, and MongoDB

Installation

Linux / macOS

# Download (Adjust version as needed)
curl -L -o synq https://github.com/anubhav-auth/synq/releases/download/v1.0.0/synq-linux-amd64

# Make executable
chmod +x synq

# Move to PATH (optional)
sudo mv synq /usr/local/bin/

# Verify installation
synq version

Windows (PowerShell)

Invoke-WebRequest -Uri https://github.com/anubhav-auth/synq/releases/download/v1.0.0/synq-windows-amd64.exe -OutFile synq.exe
.\synq.exe version

From Source

# Clone the repository
git clone https://github.com/anubhav-auth/synq.git
cd synq

# Build binary
make build

# Binary will be in bin/synq
./bin/synq version

Supported Databases

Database	Source	Destination	Notes
PostgreSQL	✅	✅	Full support including schema introspection and FK dependencies
MySQL	✅	✅	Full support including transactions and bulk insert
SQLite	✅	✅	Limited to `workers: 1` to avoid locking issues
MongoDB	✅	✅	Schema-less; collections mapped to tables

CLI Commands Reference

migrate

Run a complete database migration from source to destination.

Syntax

synq migrate [flags]

Flags

Flag	Type	Default	Description
`--config`	string	`""`	Path to YAML configuration file
`--source`	string	`""`	Source database DSN (required)
`--dest`	string	`""`	Destination database DSN (required)
`--batch-size`	int	`5000`	Number of rows per batch
`--workers`	int	`4`	Number of parallel workers
`--dlq-path`	string	`"failures.jsonl"`	Path to Dead Letter Queue file
`--threshold`	float64	`0.05`	Error threshold (0.0 - 1.0)
`--output`	string	`"migration_report.json"`	Summary report output path
`--quiet`	bool	`false`	Suppress progress output
`--log-level`	string	`"info"`	Log level: `debug`, `info`, `warn`, `error`
`--allow-insecure`	bool	`false`	Allow non-TLS database connections
`--strategy`	string	`"auto"`	Dependency resolution strategy

Examples

Basic Migration:

synq migrate \
  --source "postgres://user:pass@localhost:5432/srcdb?sslmode=disable" \
  --dest "mysql://user:pass@localhost:3306/destdb"

With Configuration File:

synq migrate --config config.yaml

Custom Batch Size and Workers:

synq migrate \
  --source "postgres://user:pass@localhost:5432/srcdb" \
  --dest "mysql://user:pass@localhost:3306/destdb" \
  --batch-size 10000 \
  --workers 8

Debug Mode:

synq migrate --config config.yaml --log-level debug

With Error Threshold:

synq migrate \
  --source "postgres://user:pass@localhost:5432/srcdb" \
  --dest "mysql://user:pass@localhost:3306/destdb" \
  --threshold 0.01  # Abort if >1% of rows fail

validate

Run pre-flight validation checks without migrating data. Validates connectivity, schema availability, and compatibility between source and destination.

Syntax

synq validate [flags]

Flags

Flag	Type	Default	Description
`--source`	string	`""`	Source database DSN (required)
`--dest`	string	`""`	Destination database DSN (required)
`--allow-insecure`	bool	`false`	Allow non-TLS connections
`--strategy`	string	`"auto"`	Dependency resolution strategy

Example

synq validate \
  --source "postgres://user:pass@localhost:5432/srcdb" \
  --dest "mongodb://user:pass@localhost:27017/destdb"

Output:

Running Validation...
---------------------
Source: Connecting... OK
Source: Discovering resources... OK (15 resources found)
Destination: Connecting... OK

Compatibility Analysis:
  - Dependency Strategy: topological
  - No compatibility issues detected.

Validation Complete. Ready to migrate.

dry-run

Simulate a migration without writing data. Currently an alias for validate.

Syntax

synq dry-run [flags]

Example

synq dry-run --source "postgres://..." --dest "mysql://..."

audit

Scan source database for Personally Identifiable Information (PII) without migrating data. Detects emails, credit cards, and SSNs.

Syntax

synq audit [flags]

Flags

Flag	Type	Default	Description
`--source`	string	`""`	Source database DSN (required)
`--allow-insecure`	bool	`false`	Allow non-TLS connections

Example

synq audit --source "postgres://user:pass@localhost:5432/srcdb"

Output:

INFO Starting Audit Scan adapter=*postgres.Adapter
INFO Scanning table table=users
INFO Scanning table table=orders
INFO Scanning table table=payments
INFO Audit report generated successfully path=audit_report.json

Audit Report Format (audit_report.json):

{
  "timestamp": "2025-12-18T00:00:00Z",
  "source": "[REDACTED]",
  "tables": [
    {
      "name": "users",
      "pii_detected": true,
      "patterns": {
        "email": 15000,
        "credit_card": 0,
        "ssn": 0
      }
    },
    {
      "name": "orders",
      "pii_detected": false,
      "patterns": {}
    }
  ]
}

adapters

List all available database adapters and their capabilities.

Syntax

synq adapters [subcommand]

Subcommands

List All Adapters:

synq adapters

Output:

SCHEME         NAME         ROLES
------         ----         -----
postgres://    PostgreSQL   source, destination
mysql://       MySQL        source, destination
sqlite://      SQLite       source, destination
mongodb://     MongoDB      source, destination

Get Adapter Details:

synq adapters info postgres

Output:

PostgreSQL Adapter
------------------
Roles: source, destination

Source Capabilities:
  - Schema Discovery: Yes
  - Dependency Detection: Yes
  - Cursor Support: Yes
  - Parallel Read: Yes

Destination Capabilities:
  - Requires Schema: Yes
  - Transactions: Yes
  - Bulk Insert: Yes
  - Upsert: Yes
  - Schema Creation: Yes

version

Print the version number of Synq.

Syntax

synq version

Output:

Synq v0.5.0 (Phase 5: CLI & Config)

Configuration

YAML Configuration File

Create a config.yaml file to define your migration settings:

# Database Connection Strings
source: "postgres://user:password@localhost:5432/source_db?sslmode=disable"
dest: "mysql://user:password@localhost:3306/dest_db"

# Performance Settings
batch_size: 5000
workers: 4

# Error Handling
error_threshold: 0.05  # Abort if >5% of rows fail
dlq_path: "failures.jsonl"

# Output
output: "migration_report.json"
quiet: false
log_level: "info"

# Security
allow_insecure: false

# Table Filtering (Optional)
tables:
  include:
    - "users"
    - "orders"
    - "payments"
  exclude:
    - "temp_*"
    - "staging_*"

# Strategies
strategies:
  dependency: "topological"  # auto, topological, alphabetical, explicit, none
  type_mapping: "auto"       # auto, strict, permissive

# Column Transformations (Optional)
mappings:
  - table: "customers"
    transformations:
      - column: "email"
        action: "lowercase"
      - column: "legacy_id"
        action: "drop"
      - column: "status"
        action: "default"
        params:
          value: "active"
      - column: "phone"
        action: "mask"
        params:
          pattern: "XXX-XXX-####"

# Type Overrides (Advanced)
type_overrides:
  - source_system: "postgres"
    target_system: "mysql"
    mappings:
      - source_type: "JSONB"
        target_type: "TEXT"
      - source_type: "UUID"
        target_type: "VARCHAR(36)"

Connection Strings (DSN)

PostgreSQL

postgres://user:password@host:port/database?sslmode=disable
postgresql://user:password@host:port/database?pool_max_conns=20

Parameters:

sslmode: disable, require, verify-ca, verify-full
pool_max_conns: Maximum connection pool size (default: auto)

MySQL

user:password@tcp(host:port)/database?parseTime=true
user:password@tcp(host:port)/database?maxConns=25

Parameters:

parseTime: Parse DATE and DATETIME to time.Time (recommended: true)
maxConns: Maximum connection pool size (default: 25)

SQLite

sqlite:///path/to/database.db
sqlite://./local.db

Note: SQLite requires workers: 1 to avoid "database is locked" errors.

MongoDB

mongodb://user:password@host:port/database?authSource=admin
mongodb+srv://user:password@cluster.mongodb.net/database

Parameters:

authSource: Authentication database (default: admin)
retryWrites: Enable retry writes (default: true)

Configuration Options

Core Settings

Option	Type	Default	Description
`source`	string	required	Source database DSN
`dest`	string	required	Destination database DSN
`batch_size`	int	`5000`	Rows per batch. Increase for small rows, decrease for large BLOBs
`workers`	int	`4`	Parallel workers. Set to 1 for SQLite
`error_threshold`	float64	`0.05`	Abort if error rate exceeds this (0.0-1.0)
`dlq_path`	string	`"failures.jsonl"`	Failed rows written here with error messages
`output`	string	`"migration_report.json"`	Final summary report path
`quiet`	bool	`false`	Suppress console progress output
`log_level`	string	`"info"`	Logging verbosity: `debug`, `info`, `warn`, `error`
`allow_insecure`	bool	`false`	Allow non-TLS connections (not recommended for production)

Table Filtering

tables:
  include:  # Only migrate these tables (glob patterns supported)
    - "users"
    - "orders_*"
  exclude:  # Skip these tables (glob patterns supported)
    - "temp_*"
    - "staging_*"

Notes:

If include is specified, only those tables are migrated
exclude is applied after include
Supports glob patterns (*, ?)

Transformations

Apply in-flight data transformations during migration.

Available Actions

Action	Description	Parameters	Example
`lowercase`	Convert string to lowercase	None	`"John@Email.COM"` → `"john@email.com"`
`uppercase`	Convert string to uppercase	None	`"active"` → `"ACTIVE"`
`drop`	Remove column from destination	None	Column not written
`default`	Replace NULL with default value	`value`	NULL → `"active"`
`mask`	Mask sensitive data	`pattern`	`"555-1234"` → `"XXX-1234"`
`hash`	One-way hash (SHA256)	None	`"password"` → `"5e884..."`
`trim`	Remove leading/trailing whitespace	None	`" text "` → `"text"`

Example Configuration

mappings:
  - table: "users"
    transformations:
      - column: "email"
        action: "lowercase"
      
      - column: "ssn"
        action: "mask"
        params:
          pattern: "XXX-XX-####"
      
      - column: "legacy_field"
        action: "drop"
      
      - column: "account_status"
        action: "default"
        params:
          value: "pending"

Strategies

Dependency Resolution

Controls the order in which tables are migrated.

Strategy	Description	Use Case
`auto`	Auto-detect (topological if FKs exist, else alphabetical)	Recommended
`topological`	Respects foreign key dependencies	Relational databases with FK constraints
`alphabetical`	Migrate tables in alphabetical order	Simple migrations without FKs
`explicit`	User-defined order (via config)	Custom ordering requirements
`none`	Migrate all tables in parallel	NoSQL or when FK constraints disabled

Example:

strategies:
  dependency: "topological"

Type Mapping

Controls how data types are converted between databases.

Strategy	Description	Behavior
`auto`	Automatic conversion with sensible defaults	Recommended
`strict`	Fail on any type mismatch	Maximum safety
`permissive`	Best-effort conversion, may lose precision	Maximum compatibility

Example:

strategies:
  type_mapping: "auto"

Performance Benchmarks

Test Environment:

Dataset: 200,000 rows
Hardware: Standard development workstation
Date: December 2025

Memory Benchmarks

Performance metrics for a full migration cycle measuring memory allocation and execution time.

Source	Destination	Time/Op	Memory/Op	Allocations/Op	Notes
MySQL	Postgres	0.83s	140.14 MB	3,834,633	🏆 Fastest
Postgres	MongoDB	1.12s	356.37 MB	5,868,135	Memory intensive
MongoDB	Postgres	1.21s	272.11 MB	7,965,791	High allocations
MySQL	MongoDB	1.26s	358.65 MB	5,724,876	NoSQL target
SQLite	Postgres	1.67s	129.65 MB	2,971,031	Lightweight source
SQLite	MongoDB	1.62s	310.17 MB	5,305,351	Embedded to NoSQL
Postgres	MySQL	2.19s	219.34 MB	3,839,867	Balanced
MongoDB	MySQL	2.21s	352.84 MB	7,902,835	NoSQL to SQL
MongoDB	SQLite	3.73s	269.95 MB	7,369,234	Embedded target
Postgres	SQLite	4.09s	137.19 MB	3,063,285	SQLite write limits
SQLite	MySQL	4.16s	195.06 MB	2,866,134	Embedded source
MySQL	SQLite	4.89s	141.93 MB	2,751,549	SQLite bottleneck

Throughput Benchmarks

Processing speed measured in rows per second (Latest V2 Optimizations).

Source	Destination	Throughput	Change
MySQL	Postgres	311,234 rows/sec ⚡	+61%
MySQL	MongoDB	260,376 rows/sec	+78%
Postgres	MongoDB	229,786 rows/sec	+34%
Postgres	MySQL	139,608 rows/sec	+30%
SQLite	Postgres	115,400 rows/sec	-

Performance Tuning

Low Memory Environment (<512MB RAM)

batch_size: 1000
workers: 2

Set environment variable: GOGC=50 (force frequent garbage collection)

High Throughput / Large Dataset (>100M rows)

batch_size: 10000
workers: 16

Ensure database max_connections can handle workers + overhead.

Large Objects (BLOBs/JSON)

For tables with longtext or bytea columns >1MB:

batch_size: 100  # Drastically reduce batch size
workers: 4

Advanced Features

Checkpoint & Resume

Synq automatically tracks migration progress in .synq_state.json. If a migration is interrupted, simply run it again—Synq will skip completed tables and resume partially finished ones from the last successful primary key.

Asynchronous Fallback

When a bulk batch fails (e.g., due to a unique constraint violation), Synq automatically queues it for a background worker. The background worker retries the batch row-by-row while the main pipeline continues at full speed for valid data.

Global Worker Pool

Fixed-resource concurrency model that maintains constant memory and CPU overhead regardless of migration size, preventing context-switching bottlenecks.

Parallel Pre-flight

Schema discovery and destination table creation are parallelized using errgroup, ensuring that data starts flowing almost instantly even for databases with thousands of tables.

Dead Letter Queue (DLQ)

Failed rows are automatically saved to a JSONL file for later inspection and manual retry.

Migration Report

After migration completes, a JSON summary is generated.

Example Report:

{
  "start_time": "2025-12-18T00:00:00Z",
  "end_time": "2025-12-18T00:05:30Z",
  "duration_seconds": 330,
  "tables_migrated": 15,
  "total_rows": 1000000,
  "successful_rows": 998500,
  "failed_rows": 1500,
  "error_rate": 0.0015,
  "throughput_rows_per_sec": 3030.3,
  "dlq_path": "failures.jsonl"
}

Graceful Shutdown

Synq handles SIGINT (Ctrl+C) and SIGTERM gracefully:

synq migrate --config config.yaml
# Press Ctrl+C to stop
^C
INFO Migration cancelled by user/signal, exiting cleanly

Development

Prerequisites

Go 1.21+
Docker & Docker Compose (for integration tests)

Commands

# Clone repository
git clone https://github.com/anubhav-auth/synq.git
cd synq

# Install dependencies
go mod download

# Build binary
make build

# Run unit tests
make test-unit

# Start local databases
make up

# Run integration tests
make test-integration

# Stop databases
make down

# Clean build artifacts
make clean

Docker Compose Services

The included docker-compose.yml provides:

PostgreSQL (port 5432)
MySQL (port 3306)
MongoDB (port 27017)

docker compose up -d

License

This project is licensed under the MIT License.

Support & Contributing

Issues: https://github.com/anubhav-auth/synq/issues
Discussions: https://github.com/anubhav-auth/synq/discussions
Pull Requests: Welcome!

Quick Reference Card

Common Commands

# Basic migration
synq migrate --source "postgres://..." --dest "mysql://..."

# With config file
synq migrate --config config.yaml

# Validate before migration
synq validate --source "postgres://..." --dest "mysql://..."

# PII audit
synq audit --source "postgres://..."

# List adapters
synq adapters

# Get adapter capabilities
synq adapters info postgres

# Check version
synq version

Common DSN Formats

# PostgreSQL
postgres://user:pass@host:5432/db?sslmode=disable

# MySQL
user:pass@tcp(host:3306)/db?parseTime=true

# SQLite
sqlite:///path/to/db.db

# MongoDB
mongodb://user:pass@host:27017/db

Environment Variables

# Force frequent GC for low memory
export GOGC=50

# Increase Go max processes
export GOMAXPROCS=8

Built with ❤️ by the Synq team. Star us on GitHub!

Name		Name	Last commit message	Last commit date
Latest commit History 462 Commits
cmd		cmd
docs		docs
internal		internal
tests		tests
.gitignore		.gitignore
BENCHMARKS.md		BENCHMARKS.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
docker-compose.yml		docker-compose.yml
go.mod		go.mod
go.sum		go.sum
synq		synq
synq-exec-summary.md		synq-exec-summary.md
synq.yaml		synq.yaml

Folders and files

Latest commit

History

Repository files navigation

Synq: High-Performance Database Migration Tool

Table of Contents

Why Synq?

Installation

Linux / macOS

Windows (PowerShell)

From Source

Supported Databases

CLI Commands Reference

migrate

Syntax

Flags

Examples

validate

Syntax

Flags

Example

dry-run

Syntax

Example

audit

Syntax

Flags

Example

adapters

Syntax

Subcommands

version

Syntax

Configuration

YAML Configuration File

Connection Strings (DSN)

PostgreSQL

MySQL

SQLite

MongoDB

Configuration Options

Core Settings

Table Filtering

Transformations

Available Actions

Example Configuration

Strategies

Dependency Resolution

Type Mapping

Performance Benchmarks

Memory Benchmarks

Throughput Benchmarks

Performance Tuning

Low Memory Environment (<512MB RAM)

High Throughput / Large Dataset (>100M rows)

Large Objects (BLOBs/JSON)

Advanced Features

Checkpoint & Resume

Asynchronous Fallback

Global Worker Pool

Parallel Pre-flight

Dead Letter Queue (DLQ)

Migration Report

Graceful Shutdown

Development

Prerequisites

Commands

Docker Compose Services

License

Support & Contributing

Quick Reference Card

Common Commands

Common DSN Formats

Environment Variables

About

Resources

License

Uh oh!

Stars

Watchers

Packages