LSF: Limiting number of cores per node on with LSF?

[raminammour]

Hello,

How do i limit the number of cores per node with the LSFManager?

I would normally use -R "span[ptile=...]" but the option seems to be ignored if passed as an option to lsf.

I am assuming it is because of the job array launching each core separately? @bjarthur would probably know :)

Thank you!

[bjarthur]

details like this are usually very dependent on how your sysadmins have configured your cluster.

would help if you posted your code, as well as logs showing why you think it is ignored.

[raminammour]

Sorry for the late response @bjarthur , I had no access to lsf system till now.

The jobs are launched on the same node for example below:

julia> addprocs_lsf(4,bsub_flags=`-R "span[ptile=2]"`)
[ Info: `bsub -R 'span[ptile=2]' -cwd ... -J 'julia-649580[1-4]' .../julia --worker=HD6PDHgBP`

julia> pmap(i->gethostname(),1:nworkers()) |> unique
1-element Vector{String}:
 "r310n11"

Without job arrays two nodes as expected:

bsub -Is -n 4 -R "span[ptile=2]" mpirun hostname
Job <512551> is submitted to default queue ...
<<Waiting for dispatch ...>>
<<Starting on r309n05>>
r309n07
r309n07
r309n05
r309n05

I appreciate the help!

[bjarthur]

i have never used the R flag, so i'm not sure i can help. sorry.

[DilumAluthge]

@DrChainsaw Do you have any familiarity with the -R flag?

[DrChainsaw]

Yes, a little bit but I'm not sure if my experience with it is helpful here. Fwiw -R is a bit of a swiss army knife that allows you to put requirements on the machines running the jobs. I have not used the span requirement myself, but from the above post and the manual it should put a requirement that more than span different machines shall be used. As the issue appears to demonstrate, all workers end up on the same machine even then span is 2.

Assuming the problem lies with this package, my prime suspect is that the resource string is somehow dropped through addprocs_lsf, for example because something is wrong with the quoting.

Just evaluating the command it on my computer gives this:

julia> `-R "span[ptile=2]"`
`-R 'span[ptile=2]'`

I see the double quotes are changed to single quotes. Perhaps LSF requires double quotes here? @DilumAluthge: Do you happen to know if there is a way to keep the double quotes?

@raminammour: There should be some output from LSF, either in an email or in some logfile if you provided that which mirrors the job submission command received by the LSF scheduler. If you happen to have access to that and post it here that would be helpful.

I guess you have moved on now though so I will try the -R flag on my system when I find the time.

[DilumAluthge]

I see the double quotes are changed to single quotes. Perhaps LSF requires double quotes here? @DilumAluthge: Do you happen to know if there is a way to keep the double quotes?

Hmmm. Not that I know off the top of my head.

[raminammour]

I lost access to the machine with the LSF manager, so I won't be able to help/debug much anymore. Sorry!