Skip to content

Improve training performance on large datasets like MGnify - #35

Open
idavi-bcs wants to merge 3 commits into
Kuhlman-Lab:mainfrom
idavi-bcs:bigdata-perf
Open

Improve training performance on large datasets like MGnify#35
idavi-bcs wants to merge 3 commits into
Kuhlman-Lab:mainfrom
idavi-bcs:bigdata-perf

Conversation

@idavi-bcs

Copy link
Copy Markdown

Fixes #34 .

Addresses three bottlenecks that appear when training with datasets that have many wildtype domains, with a few mutations each:

  1. Reading PDB models now access the disk only once, and does not make passes for chains that don't exist.
  2. For Pandas operations in MegaScaleDataset, slow explicit loops have been replaced with fast vectorized operations.
  3. Per-step metric logging has been converted to per-epoch, because Spearman CC slows down massively when the number of steps gets very large.

I've also added support for batches > 1. The original code had one wildtype PDB per batch, with hundreds of mutations. With datasets like MGnify where each wildtype has only a few mutations, this is insufficient to calculate meaningful gradients. Larger batch sizes also reduce the number of steps per epoch, which diminishes the importance of the Spearman performance issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Performance issues uncovered when training on MGnify (larger dataset)

1 participant