[suggest] early_stop imlementation

[thangckt]

hi @YutackPark

It would helpful if Sevenn can support an early stopper?

Do you think that?

Thanks

[YutackPark]

I agree with that. Actually, it was in TODO list. I'll keep open this issue. As we already have mechanism that writes 'best' checkpoint, it won't be difficult.

[thangckt]

hi @YutackPark
I tried to train some small models and realize that target quantities (energy/force/stress) converge in different speeds, which are sensitive to the below parameters:

    force_loss_weight: 0.1                     
    stress_loss_weight: 1e-06  

Ask my understand, by setting these parameters we manually scale the loss terms separately.
My question is:

• Automatically, is it possible to have a built-in strategy to balance these loss terms on-the-fly?
• Or manually, is there any way to determine the best values for these parameters?

Thank you so much for your help

[YutackPark]

To the best of my knowledge, there is no standard, systematic way to determine these. They are empirically determined for model complexity and dataset. One thing I know is different model has different optimal loss weights (for some models can be found in the literature, they use force loss weight bigger than 100. However it doesn't mean force is 100 times more important than the energy.)

[thangckt]

@YutackPark

Thank you for your guide.
Whether any of these strategies work in Sevenn?

My models as below:

I used weights for force and stress losses as above. And seemly that stress converges on a different scale compare to other two.
Do you have any advice to tweak the loss?

Thank you.

[YutackPark]

Thanks for sharing the article. I haven't seen such strategies. I'll have a look.
The loss weights are not multiplied to individual loss. You may want to multiply 0.1 and 1e-6 to force and stress, respectively, to see how they actually contribute to the total loss. However, in my case, I was not able to interpret them for actual model's accuracy and I had to determine them empirically by seeing validation set errors.

Note that they (loss) are ill-defined: for example, while energy and forces are normalized by the number of atoms, stress is normalized by volume (by definition). There exist multiple choices for these losses (may someone don't want to normalize the energy by the number of atoms), and interpreting them and choosing effective loss definition or weights are open question.

[thangckt]

@YutackPark
Thank you for your explaining.

I haven't seen such strategies. I'll have a look.

I am looking forward if you have any update.
Thanks.