You can set your nodes into Maintenance Mode by clicking Platform \ Computers and checking the maintenance mode option. Once maintenance mode is enabled, the /api/v1/status endpoint will begin returning 503 . This should be configured to disable traffic being routed to the node while maintenance is performed.
I have two nodes talking to the same sql instance and i set one of the nodes Maintenance mode to true. So I executed an script on any computer I would assume that the job would pick the machine that isnt in maintenance mode.
The only thing maintenance mode currently does is change that status. The idea was more for load balancers to understand that the node was down for maintenance and not to send users to it.
We certainly should enhance this to prevent it from running jobs as well.
We are still looking at this. The “relatively small” change was not the correct assumption and have had a heck of time getting hangfire to behave correctly when removing nodes dynamically while the service is running.