Today's guest post comes from Ralph Castain, a principle engineer at Intel.  The bulk of this post is an email he sent explaining the concept of a "slot" in typical HPC schedulers. This is a little departure from the normal fare on this blog, but is still a critical concept to understand for running HPC applications efficiently.  With his permission, I re-publish Ralph's email here because it's a great analogy to explain the "slot" concept, which is broadly applicable to HPC users.

The question of "what is a [scheduler] slot" when discussing schedulers came up yesterday and was an obvious source of confusion, so let me try to explain the concept using a simple model.

Suppose I own a fleet of cars at several locations around the country. I am in the business of providing rides for people. Each car has 5 seats in it.

In one location, my clientele doesn't have much sense of personal space and is willing to be a little crowded. In that location, I sell tickets to share a car, and allow up to 10 people who are going in roughly the same direction to share a single vehicle (hey, they are willing to sit on each other's lap!).

In another location, my clients aren1t quite as "friendly" and really prefer to have their own seat. However, they are willing to share the car with others headed in the same direction, so I sell only as many tickets as I have seats -in this case, up to 5 tickets for a given car.

In a third location, my clients tend to be a little on the large side -when I have a large passenger, I find that everyone is happier if I don't fill the backseat. So when a customer flags that they are a little larger than average, I only sell 4 tickets for that car -i.e., I require that the middle seat in the back be empty so the passengers can spread out a bit. This may require that I schedule that larger client on a different (perhaps larger) car if I already have 4 people for one that would otherwise be available.

In all of the above locations, I will sell another ticket and allow a passenger to enter a car once someone is dropped off. So I try to keep my cars as full as possible by constantly adding a replacement customer when one leaves. However, I never allow more passengers in the car then what that location will tolerate -if someone tries to give a "free" lift to a person at the side of the road, I block them from doing so. In addition, if someone calls and asks for 8 tickets, I will schedule them across multiple cars according to the local policy.

In yet another location, I have very exclusive customers -they don't want anyone in the car with them. In this case, I simply lease them the entire car for the requested duration. They are free to do whatever they want with the vehicle (including picking up as many passengers as they like), so long as they return it clean and in good working condition.

The concept of the "slot" in schedulers is based on that max payload I define for each location. The scheduler is selling "tickets" to the servers/nodes based on some limit set by the system admin, which is usually based on the needs and policies of the local installation. As the above illustration shows, the definition of the "max payload" for a node can vary by site and node, and the scheduler takes into account a variety of requirements when allocating slots to a user.

Once we have an allocation, we then have to assign seats to individual customers. This is the "mapping" policy. When I map processes "by slot," what I mean is that I start with the first seat in the first car, and assign customers to seats in a sequential fashion, filling all the allocated seats in the first car before starting to fill the second one. This is best for a "chatty" group of customers, but can lead to one car being more heavily loaded than the others.

When I map processes "by node," I assign the first customer to the first seat in the first car, I assign the next customer to the first allocated seat in the second car, continuing round-robin until all the customers have been assigned an allocated seat. This balances the load in the cars, but is very inefficient if the customers needed to have a conversation.

Obviously, there are lots and lots of ways for allocating and assigning seatswithinthose cars... that's a whole separate topic.