Slurm nodes installation under Ubuntu16.04 by jianlianggao · Pull Request #2 · BioMedIA/slurm-formula

jianlianggao · 2018-04-30T14:13:52Z

The files can be used for installing Slurm nodes with Ubuntu 16.04. The Slurm master node is biomedia03

jopasserat

Couple of changes, most critical are:

use Jinja templates
use Pillar

jopasserat · 2018-04-30T15:48:43Z

-{% set rocsList = [] %}
-{% for node, values in pillar['slurm']['nodes']['batch']['cpus'].items() %} {% if node.startswith('roc') %} {% set rocsListTrash = rocsList.append(node) %} {% endif %} {% endfor %}
+NodeName=biomedia01 RealMemory=64000 CPUs=24 State=UNKNOWN
+


this should be part of the Jinja template

jopasserat · 2018-04-30T15:49:22Z

+NodeName=monal01 RealMemory=80000 CPUs=12 Gres=gpu:8 State=UNKNOWN
+
+# Partitions
+PartitionName=long Nodes=biomedia01,biomedia02,biomedia05,biomedia06,biomedia07,biomedia08,biomedia09,biomedia10,roc01,roc02,roc03 Default=YES MaxTime=43200


all roc machines should be in long as well. it's fine to have two partitions overlapping

jopasserat · 2018-04-30T15:49:43Z

 #AuthInfo=/var/run/munge/munge.socket.2
 AuthType=auth/munge
-DbdHost={{ pillar['slurm']['controller'] }}
+DbdHost=biomedia03


no hardcoded values please => use Pillar

jopasserat · 2018-04-30T15:50:16Z

-{% endif %}
+install slurm packages from local repo:
+  pkg.installed:
+    - sources:


jopasserat · 2018-04-30T15:51:32Z

 # disable SSH for anybody but root
 +:root:ALL
-:ALL EXCEPT (csg) dr jpassera bglocker:ALL
+-:ALL EXCEPT (csg) (biomedia) dr jpassera bglocker jgao:ALL


regular users shouldn't have SSH access to the cluster nodes, hence the previous config

jopasserat · 2018-04-30T15:52:09Z

+ConstrainSwapSpace=yes
+AllowedSwapSpace=10.0
+# Not well supported until Slurm v14.11.4 https://groups.google.com/d/msg/slurm-devel/oKAUed7AETs/Eb6thh9Lc0YJ
+#ConstrainDevices=yes


Should that be enabled and not commented out then?

jopasserat · 2018-04-30T15:52:20Z

 #
 # Workaround because Slurm does not recognize full hostname...
-ControlMachine={{ pillar['slurm']['controller'] }}
+ControlMachine=biomedia03


jopasserat · 2018-04-30T16:01:01Z

@@ -1,3 +0,0 @@
-#!/bin/bash


This script is useful to add new members. it should stay there

jopasserat · 2018-04-30T16:01:19Z

@@ -1,13 +0,0 @@
-#!/bin/bash


Script used to bootstrap new minions. Any reason to remove it?

jopasserat · 2018-04-30T16:02:01Z

-  * Slurm
-  * Slurm Database
-  * SSH
+To install Slurm nodes, you need to copy (on Slurm mater node)


Maybe move that lower in an "Instructions" subsection instead of replacing what the formula contains?

jianlianggao · 2018-04-30T18:06:02Z

Hi Jonathan, Thank you very much for the comments. I will modify and test on predicts cluster. Best wishes, Jianliang

________________________________ From: jopasserat <notifications@github.com> Sent: 30 April 2018 17:02:34 To: BioMedIA/slurm-formula Cc: Gao, Jianliang; Author Subject: Re: [BioMedIA/slurm-formula] Slurm nodes installation under Ubuntu16.04 (#2) @jopasserat requested changes on this pull request. Couple of changes, most critical are: * use Jinja templates * use Pillar

________________________________ In files/etc/slurm-llnl/slurm.conf<#2 (comment)>:

-{% set rocsList = [] %} -{% for node, values in pillar['slurm']['nodes']['batch']['cpus'].items() %} {% if node.startswith('roc') %} {% set rocsListTrash = rocsList.append(node) %} {% endif %} {% endfor %} +NodeName=biomedia01 RealMemory=64000 CPUs=24 State=UNKNOWN + this should be part of the Jinja template

________________________________ In files/etc/slurm-llnl/slurm.conf<#2 (comment)>:

+NodeName=roc11 RealMemory=257869 CPUs=32 State=UNKNOWN

+ +NodeName=roc12 RealMemory=257869 CPUs=32 State=UNKNOWN + +NodeName=roc13 RealMemory=257869 CPUs=32 State=UNKNOWN + +NodeName=roc14 RealMemory=257869 CPUs=32 State=UNKNOWN + +NodeName=roc15 RealMemory=257869 CPUs=32 State=UNKNOWN + +NodeName=roc16 RealMemory=257869 CPUs=32 State=UNKNOWN + +NodeName=monal01 RealMemory=80000 CPUs=12 Gres=gpu:8 State=UNKNOWN + +# Partitions +PartitionName=long Nodes=biomedia01,biomedia02,biomedia05,biomedia06,biomedia07,biomedia08,biomedia09,biomedia10,roc01,roc02,roc03 Default=YES MaxTime=43200 all roc machines should be in long as well. it's fine to have two partitions overlapping

________________________________ In files/etc/slurm-llnl/slurmdbd.conf<#2 (comment)>:

@@ -6,7 +6,7 @@ ArchiveSuspend=no

#ArchiveScript=/usr/sbin/slurm.dbd.archive #AuthInfo=/var/run/munge/munge.socket.2 AuthType=auth/munge -DbdHost={{ pillar['slurm']['controller'] }} +DbdHost=biomedia03 no hardcoded values please => use Pillar

________________________________ In init.sls<#2 (comment)>:

-/var/log/slurm-llnl/sched.log: - file.managed: - - group: slurm - - user: slurm - - require: - - user: slurm -{% endif %} +install slurm packages from local repo: + pkg.installed: + - sources: nice 👍

________________________________ In files/etc/security/access.conf<#2 (comment)>:

@@ -9,4 +9,4 @@

# who wants to be able to SSH in as root via public-key on Biomedia servers. # disable SSH for anybody but root +:root:ALL --:ALL EXCEPT (csg) dr jpassera bglocker:ALL +-:ALL EXCEPT (csg) (biomedia) dr jpassera bglocker jgao:ALL regular users shouldn't have SSH access to the cluster nodes, hence the previous config

________________________________ In files/etc/slurm-llnl/cgroup.conf<#2 (comment)>:

@@ -16,7 +16,8 @@ CgroupReleaseAgentDir=/var/spool/slurm-llnl/cgroup

ConstrainCores=yes TaskAffinity=yes #ConstrainRAMSpace=no -### not used yet -#ConstrainDevices=no -#AllowedDevicesFile=/etc/slurm-llnl/cgroup_allowed_devices_file.conf - +ConstrainSwapSpace=yes +AllowedSwapSpace=10.0 +# Not well supported until Slurm v14.11.4 https://groups.google.com/d/msg/slurm-devel/oKAUed7AETs/Eb6thh9Lc0YJ +#ConstrainDevices=yes Should that be enabled and not commented out then?

________________________________ In files/etc/slurm-llnl/slurm.conf<#2 (comment)>:

@@ -3,7 +3,7 @@

# See the slurm.conf man page for more information. # # Workaround because Slurm does not recognize full hostname... -ControlMachine={{ pillar['slurm']['controller'] }} +ControlMachine=biomedia03 Pillar

________________________________ In tools/add_users_to_slurm_groups.sh<#2 (comment)>:

@@ -1,3 +0,0 @@

-#!/bin/bash This script is useful to add new members. it should stay there

________________________________ In tools/install_salt-minion.sh<#2 (comment)>:

@@ -1,13 +0,0 @@

-#!/bin/bash Script used to bootstrap new minions. Any reason to remove it?

________________________________ In README.md<#2 (comment)>:

@@ -3,9 +3,9 @@

Salt formula provisioning a Slurm cluster

…

-Availables states: - * Munge - * Screen - * Slurm - * Slurm Database - * SSH +To install Slurm nodes, you need to copy (on Slurm mater node) Maybe move that lower in an "Instructions" subsection instead of replacing what the formula contains? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2 (review)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ARkEoAC125GdeIqtpcT2yBx_p7wBdqzbks5ttzWagaJpZM4TsphH>.

jopasserat · 2018-05-01T20:27:13Z

-PartitionName=rocsShort  Nodes={{ ','.join(rocsList) }} Default=NO MaxTime=60 Priority=5000
+#PartitionName=long Nodes=biomedia01,biomedia02,biomedia03,biomedia05 Default=YES MaxTime=43200
+#PartitionName=short Nodes=biomedia01,biomedia03,biomedia05 Default=NO MaxTime=60 Priority=5000
+PartitionName=gpus  Nodes=monal01 Default=NO MaxTime=10080 MaxCPUsPerNode=4 MaxMemPerNode=30720


You can remove the MaxCPUsPerNode=4 MaxMemPerNode=30720 settings. Just legacy code here

jianlianggao · 2018-05-02T04:13:11Z

Hi Jonathan, OK, thank you very much. I haven't done the modification yet. I'm now testing a shell script to submit docker run jobs via slurm. Best wishes, Jianliang

Sent from my Mobile

-------- Original Message -------- Subject: Re: [BioMedIA/slurm-formula] Slurm nodes installation under Ubuntu16.04 (#2) From: jopasserat To: BioMedIA/slurm-formula CC: "Gao, Jianliang" ,Author @jopasserat commented on this pull request.

________________________________ In files/etc/slurm-llnl/slurm.conf<#2 (comment)>:

-PartitionName=rocsLong Nodes={{ ','.join(rocsList) }} Default=NO MaxTime=43200 -PartitionName=rocsShort Nodes={{ ','.join(rocsList) }} Default=NO MaxTime=60 Priority=5000 +#PartitionName=long Nodes=biomedia01,biomedia02,biomedia03,biomedia05 Default=YES MaxTime=43200 +#PartitionName=short Nodes=biomedia01,biomedia03,biomedia05 Default=NO MaxTime=60 Priority=5000 +PartitionName=gpus Nodes=monal01 Default=NO MaxTime=10080 MaxCPUsPerNode=4 MaxMemPerNode=30720 You can remove the MaxCPUsPerNode=4 MaxMemPerNode=30720 settings. Just legacy code here — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#2 (review)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ARkEoHSzssmEsrvy-L9Dohsn3mR345GYks5tuMUhgaJpZM4TsphH>.

Updated slurm.conf & pillar.example using jinja template

…m state

jopasserat · 2018-05-16T14:40:46Z

-StoragePass={{ pillar['slurm']['db']['password'] }}
+StorageLoc=slurmdb
+StorageUser=slurm
+StoragePass=1BUy4eVv7X


password in cleartext in the commit history...

Slurm nodes installation under Ubuntu16.04

a48702a

The files can be used for installing Slurm nodes with Ubuntu 16.04. The Slurm master node is biomedia03

jopasserat requested changes Apr 30, 2018

View reviewed changes

jopasserat reviewed May 1, 2018

View reviewed changes

Jianliang and others added 3 commits May 3, 2018 10:34

Updated slurm.conf & pillar.example using jinja template

6632b26

Updated slurm.conf & pillar.example using jinja template

check Salt minions munge key ans Slurm version and conf

f2c7ef1

fixed bug of querying slurm version on minions by separating the slur…

b5893f8

…m state

jopasserat reviewed May 16, 2018

View reviewed changes

Jianliang Gao and others added 5 commits May 18, 2018 10:35

update Salt minion nodes by reconfiguration

0e8c1f2

aggregated node names

8e54231

updated init.sls and related config files for GPU nodes

0b8945e

updated init.sls for installing Slurm with dependencies

3510b4e

added conditional statement for GPU servers gres.conf

7e84382

Conversation

jianlianggao commented Apr 30, 2018

Uh oh!

jopasserat left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jianlianggao commented Apr 30, 2018 via email

Uh oh!

jopasserat May 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jianlianggao commented May 2, 2018 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jopasserat May 1, 2018 •

edited

Loading