Fri, 23 Mar 2007
While computers mostly handle the Daylight Savings Time changeover automatically, sometimes work is required. In our case at ICHEC, arranging for some jobs to run at set times, means changing the scheduler at the changeover.
We use PBS and Maui to schedule what jobs run when on our cluster. Mostly jobs get run according to priorities: the scheduler (software) decides which what the priority jobs are and when; it collects free nodes until it has sufficient free CPUs to run the jobs at hand. It can do backfilling: when it's e.g. trying to run a large 128 node job but has only 10 nodes free, it will reserve nodes as they become free until it has all the resources needed to run the job. If by looking at the remaining time left on running jobs it realises that it won't get nodes free for the large job for say 2 hours, it can run small jobs with runtimes of less than two hours on the nodes it had reserved. Clever.
The fun issue comes when you need to set reservations: say you need to run a job at 2am every day on 32 nodes, as we do: the scheduler needs to plan accordingly. The scheduler needs to really have 32 nodes free at that time.
Thats 2am local time, of course: but next Sunday, at 2am, the clock changes...
So, working out the consequences, and preparing for the change, I sympathise with our American friends who've moved three weeks early to save energy. Thus causing lots of midnight oil to be burned around the world by sysadmins and programmers updating software. Hmmm.
Note to Self:
When Posting a blog entry asking for help on getting blog comments to work with openid, try leaving the comments setup on the blog working, so people can leave replies. Fixed now.
Thanks for supporting OpenID. I needed to test the WordPress OpenID plugin. :-)
Posted by Matthew P. McGuire at Mon Mar 26 14:41:13 2007
It case it wasn't clear, the problem we had to address is that, if at 1am it has to schedule jobs on a machine that needs to be free at 4am for a particular task, its not smart enough to realise that tonight in particular, there are only 2 hours between 1 and 4am. If it schedules a 3 hr job to run in that slot, the job wont complete in time for the priority job to run at 4am.
Posted by Alastair McKinstry at Thu Mar 29 16:53:14 2007
A popular move, but in the end, a complete flop.
Posted by Alastair McKinstry at Mon Apr 16 09:02:01 2007