Package org.apache.storm.scheduler
Interface ISchedulingState
-
- All Known Implementing Classes:
Cluster,SingleTopologyCluster
public interface ISchedulingStateAn interface that provides access to the current scheduling state. The scheduling state is not guaranteed to be thread safe.
-
-
Method Summary
All Methods Instance Methods Abstract Methods Default Methods Modifier and Type Method Description NormalizedResourceRequestgetAllScheduledResourcesForNode(String nodeId)Get all scheduled resources for node.Set<Integer>getAssignablePorts(SupervisorDetails supervisor)Get the ports that are not blacklisted.List<WorkerSlot>getAssignableSlots()Get all non-blacklisted slots in the cluster.List<WorkerSlot>getAssignableSlots(SupervisorDetails supervisor)Return all non-blacklisted slots on this supervisor.intgetAssignedNumWorkers(TopologyDetails topology)Get the number of workers assigned to a topology.default Set<String>getAssignedRacks(String... topologyIds)Determine the list of racks on which topologyIds have been assigned.SchedulerAssignmentgetAssignmentById(String topologyId)get the current assignment for the topology.Map<String,SchedulerAssignment>getAssignments()Get all the assignments.Set<Integer>getAvailablePorts(SupervisorDetails supervisor)Return the available ports of this supervisor.NormalizedResourceOffergetAvailableResources(SupervisorDetails sd)Get the resources on the supervisor that are available to be scheduled.List<WorkerSlot>getAvailableSlots()Get all the available worker slots in the cluster.List<WorkerSlot>getAvailableSlots(SupervisorDetails supervisor)Return all the available slots on this supervisor.Set<String>getBlacklistedHosts()Get all of the hosts that are blacklisted.doublegetClusterTotalCpuResource()Get the total amount of CPU resources in cluster.Map<String,Double>getClusterTotalGenericResources()Get the total amount of generic resources (excluding CPU and memory) in cluster.doublegetClusterTotalMemoryResource()Get the total amount of memory resources in cluster.Map<String,Object>getConf()Get the nimbus configuration.StringgetHost(String supervisorId)Map a supervisor to a given host.default Map<String,String>getHostToRack()Get host -> rack map - the inverse of networkTopography.Map<String,List<ExecutorDetails>>getNeedsSchedulingComponentToExecutors(TopologyDetails topology)Get the component name to executor list for executors that need to be scheduled.Map<ExecutorDetails,String>getNeedsSchedulingExecutorToComponents(TopologyDetails topology)Get the executor to component name map for executors that need to be scheduled.Map<String,List<String>>getNetworkTopography()Get the network topography (rackId -> nodes in the rack).List<WorkerSlot>getNonBlacklistedAvailableSlots(List<String> blacklistedSupervisorIds)Get all the available worker slots in the cluster, that are not blacklisted.NormalizedResourceOffergetNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds)Get the resources in the cluster that are available for scheduling.doublegetScheduledCpuForNode(String nodeId)Get the total cpu currently scheduled on a node.doublegetScheduledMemoryForNode(String nodeId)Get the total memory currently scheduled on a node.Map<String,String>getStatusMap()Get all topology scheduler statuses.SupervisorDetailsgetSupervisorById(String nodeId)Get a specific supervisor with thenodeId.Map<String,SupervisorDetails>getSupervisors()Get all the supervisors.List<SupervisorDetails>getSupervisorsByHost(String host)Get all the supervisors on the specifiedhost.Map<String,SupervisorResources>getSupervisorsResourcesMap()Get the amount of used and free resources on a supervisor.TopologiesgetTopologies()Get all of the topologies.Map<String,TopologyResources>getTopologyResourcesMap()Get the amount of resources used by topologies.Collection<ExecutorDetails>getUnassignedExecutors(TopologyDetails topology)get the unassigned executors of the topology.Set<Integer>getUsedPorts(SupervisorDetails supervisor)Get all the used ports of this supervisor.Collection<WorkerSlot>getUsedSlots()Get all currently occupied slots.Collection<WorkerSlot>getUsedSlotsByTopologyId(String topologyId)get slots used by a topology.WorkerResourcesgetWorkerResources(WorkerSlot ws)Get the resources for a given slot.Map<String,Map<WorkerSlot,WorkerResources>>getWorkerResourcesMap()Gets the reference to the full topology->worker resource map.booleanisBlackListed(String supervisorId)Check is a given supervisor is on a blacklisted host.booleanisBlacklistedHost(String host)Check if a given host is blacklisted.booleanisSlotOccupied(WorkerSlot slot)Check if a slot is occupied or not.booleanneedsScheduling(TopologyDetails topology)Does the topology need scheduling.booleanneedsSchedulingRas(TopologyDetails topology)LikeneedsScheduling(TopologyDetails)but does not take into account the number of workers requested.List<TopologyDetails>needsSchedulingTopologies()Get all of the topologies that need scheduling.booleanwouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap)Would scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.
-
-
-
Method Detail
-
getTopologies
Topologies getTopologies()
Get all of the topologies.- Returns:
- all of the topologies that are a part of the cluster.
-
needsSchedulingTopologies
List<TopologyDetails> needsSchedulingTopologies()
Get all of the topologies that need scheduling.- Returns:
- all of the topologies that are not fully scheduled.
-
needsScheduling
boolean needsScheduling(TopologyDetails topology)
Does the topology need scheduling.A topology needs scheduling if one of the following conditions holds:
- Although the topology is assigned slots, but is squeezed. i.e. the topology is assigned less slots than desired.
- There are unassigned executors in this topology
-
needsSchedulingRas
boolean needsSchedulingRas(TopologyDetails topology)
LikeneedsScheduling(TopologyDetails)but does not take into account the number of workers requested. This is because the number of workers is ignored in RAS- Parameters:
topology- the topology to check- Returns:
- true if the topology needs scheduling else false.
-
getBlacklistedHosts
Set<String> getBlacklistedHosts()
Get all of the hosts that are blacklisted.- Returns:
- all of the hosts that are blacklisted
-
isBlackListed
boolean isBlackListed(String supervisorId)
Check is a given supervisor is on a blacklisted host.- Parameters:
supervisorId- the id of the supervisor- Returns:
- true if it is else false
-
isBlacklistedHost
boolean isBlacklistedHost(String host)
Check if a given host is blacklisted.- Parameters:
host- the name of the host- Returns:
- true if it is else false.
-
getHost
String getHost(String supervisorId)
Map a supervisor to a given host.- Parameters:
supervisorId- the id of the supervisor- Returns:
- the actual host name the supervisor is on
-
getUnassignedExecutors
Collection<ExecutorDetails> getUnassignedExecutors(TopologyDetails topology)
get the unassigned executors of the topology.- Parameters:
topology- the topology to check- Returns:
- the unassigned executors of the topology.
-
getNeedsSchedulingExecutorToComponents
Map<ExecutorDetails,String> getNeedsSchedulingExecutorToComponents(TopologyDetails topology)
Get the executor to component name map for executors that need to be scheduled.- Parameters:
topology- the topology this is for- Returns:
- a executor -> component-id map which needs scheduling in this topology.
-
getNeedsSchedulingComponentToExecutors
Map<String,List<ExecutorDetails>> getNeedsSchedulingComponentToExecutors(TopologyDetails topology)
Get the component name to executor list for executors that need to be scheduled.- Parameters:
topology- the topology this is for- Returns:
- a component-id -> executors map which needs scheduling in this topology.
-
getUsedPorts
Set<Integer> getUsedPorts(SupervisorDetails supervisor)
Get all the used ports of this supervisor.
-
getAvailablePorts
Set<Integer> getAvailablePorts(SupervisorDetails supervisor)
Return the available ports of this supervisor.
-
getAssignablePorts
Set<Integer> getAssignablePorts(SupervisorDetails supervisor)
Get the ports that are not blacklisted.- Parameters:
supervisor- the supervisor- Returns:
- the ports that are not blacklisted
-
getAvailableSlots
List<WorkerSlot> getAvailableSlots(SupervisorDetails supervisor)
Return all the available slots on this supervisor.
-
getAvailableSlots
List<WorkerSlot> getAvailableSlots()
Get all the available worker slots in the cluster.
-
getNonBlacklistedAvailableSlots
List<WorkerSlot> getNonBlacklistedAvailableSlots(List<String> blacklistedSupervisorIds)
Get all the available worker slots in the cluster, that are not blacklisted.- Parameters:
blacklistedSupervisorIds- list of supervisor ids that should also be considered blacklisted.
-
getAssignableSlots
List<WorkerSlot> getAssignableSlots(SupervisorDetails supervisor)
Return all non-blacklisted slots on this supervisor.- Parameters:
supervisor- the supervisor- Returns:
- the non-blacklisted slots
-
getAssignableSlots
List<WorkerSlot> getAssignableSlots()
Get all non-blacklisted slots in the cluster.
-
getUsedSlots
Collection<WorkerSlot> getUsedSlots()
Get all currently occupied slots.
-
isSlotOccupied
boolean isSlotOccupied(WorkerSlot slot)
Check if a slot is occupied or not.- Parameters:
slot- the slot be to checked.- Returns:
- true if the specified slot is occupied.
-
getAssignedNumWorkers
int getAssignedNumWorkers(TopologyDetails topology)
Get the number of workers assigned to a topology.- Parameters:
topology- the topology this is for- Returns:
- the number of workers assigned to this topology.
-
getAvailableResources
NormalizedResourceOffer getAvailableResources(SupervisorDetails sd)
Get the resources on the supervisor that are available to be scheduled.- Parameters:
sd- the supervisor.- Returns:
- the resources available to be scheduled.
-
wouldFit
boolean wouldFit(WorkerSlot ws, ExecutorDetails exec, TopologyDetails td, NormalizedResourceOffer resourcesAvailable, double maxHeap)
Would scheduling exec on ws fit? With a heap <= maxHeap total memory added <= memoryAvailable and cpu added <= cpuAvailable.- Parameters:
ws- the slot to put it inexec- the executor to investigatetd- the topology detains for this executorresourcesAvailable- all the available resourcesmaxHeap- the maximum heap size for ws- Returns:
- true it fits else false
-
getAssignmentById
SchedulerAssignment getAssignmentById(String topologyId)
get the current assignment for the topology.
-
getUsedSlotsByTopologyId
Collection<WorkerSlot> getUsedSlotsByTopologyId(String topologyId)
get slots used by a topology.
-
getSupervisorById
SupervisorDetails getSupervisorById(String nodeId)
Get a specific supervisor with thenodeId.
-
getSupervisorsByHost
List<SupervisorDetails> getSupervisorsByHost(String host)
Get all the supervisors on the specifiedhost.- Parameters:
host- hostname of the supervisor- Returns:
- the
SupervisorDetailsobject.
-
getAssignments
Map<String,SchedulerAssignment> getAssignments()
Get all the assignments.
-
getSupervisors
Map<String,SupervisorDetails> getSupervisors()
Get all the supervisors.
-
getAllScheduledResourcesForNode
NormalizedResourceRequest getAllScheduledResourcesForNode(String nodeId)
Get all scheduled resources for node.
-
getNonBlacklistedClusterAvailableResources
NormalizedResourceOffer getNonBlacklistedClusterAvailableResources(Collection<String> blacklistedSupervisorIds)
Get the resources in the cluster that are available for scheduling.- Parameters:
blacklistedSupervisorIds- other ids that are tentatively blacklisted.
-
getClusterTotalCpuResource
double getClusterTotalCpuResource()
Get the total amount of CPU resources in cluster.
-
getClusterTotalMemoryResource
double getClusterTotalMemoryResource()
Get the total amount of memory resources in cluster.
-
getClusterTotalGenericResources
Map<String,Double> getClusterTotalGenericResources()
Get the total amount of generic resources (excluding CPU and memory) in cluster.
-
getNetworkTopography
Map<String,List<String>> getNetworkTopography()
Get the network topography (rackId -> nodes in the rack).
-
getHostToRack
default Map<String,String> getHostToRack()
Get host -> rack map - the inverse of networkTopography.
-
getTopologyResourcesMap
Map<String,TopologyResources> getTopologyResourcesMap()
Get the amount of resources used by topologies. Used for displaying resource information on the UI.- Returns:
- a map that contains multiple topologies and the resources the topology requested and assigned. Key: topology id Value: an array that describes the resources the topology requested and assigned in the following format: {requestedMemOnHeap, requestedMemOffHeap, requestedCpu, assignedMemOnHeap, assignedMemOffHeap, assignedCpu}
-
getSupervisorsResourcesMap
Map<String,SupervisorResources> getSupervisorsResourcesMap()
Get the amount of used and free resources on a supervisor. Used for displaying resource information on the UI- Returns:
- a map where the key is the supervisor id and the value is a map that represents resource usage for a supervisor in the following format: {totalMem, totalCpu, usedMem, usedCpu}
-
getWorkerResourcesMap
Map<String,Map<WorkerSlot,WorkerResources>> getWorkerResourcesMap()
Gets the reference to the full topology->worker resource map.- Returns:
- map of topology -> map of worker slot ->resources for that worker
-
getWorkerResources
WorkerResources getWorkerResources(WorkerSlot ws)
Get the resources for a given slot.- Parameters:
ws- the slot- Returns:
- the resources currently assigned
-
getScheduledMemoryForNode
double getScheduledMemoryForNode(String nodeId)
Get the total memory currently scheduled on a node.- Parameters:
nodeId- the id of the node- Returns:
- the total memory currently scheduled on the node
-
getScheduledCpuForNode
double getScheduledCpuForNode(String nodeId)
Get the total cpu currently scheduled on a node.- Parameters:
nodeId- the id of the node- Returns:
- the total cpu currently scheduled on the node
-
getAssignedRacks
default Set<String> getAssignedRacks(String... topologyIds)
Determine the list of racks on which topologyIds have been assigned. Note that the returned set may containDNSToSwitchMapping.DEFAULT_RACKifgetHostToRack()is null or does not contain the assigned host.- Parameters:
topologyIds- for which assignments are examined.- Returns:
- set of racks on which assignments have been made.
-
-