`clap.cluster_manager`¶

Module Contents¶

Classes¶

`ClusterConfig`	A full cluster configuration
`ClusterConfigDatabase`	This class stores information about clusters configurations in CLAP system.
`ClusterConfigFile`	A dataclass that represents a cluster configuration file
`ClusterDescriptor`	Dataclass that describes an created CLAP cluster
`ClusterManager`	This class is used to start, stop, pause, resume and perform actions
`ClusterOptions`	Dataclass that stores information about optional cluster options in a
`ClusterRepositoryController`	This class is used manipulate clusters in a repository. It performs
`CommandActionType`	Dataclass that stores information about a command to perform from a
`NodeConfig`	Dataclass that stores information about a cluster’s node at cluster
`PlaybookActionType`	Dataclass that stores information about a playboook to execute from a
`RoleActionType`	Dataclass that stores information about a role’s action from a cluster
`RoleAdd`	Dataclass that stores information about a role that mst be added to
`SetupConfig`	Dataclass that stores information about a Setup configuration in a
`_ClusterConfig`	Temporary class about cluster config (used to parse cluster configuration
`_NodeConfig`	Temporary class about node config (used to parse cluster configuration

Attributes¶

`ActionType`	An setup action can be a role action, a command or a playbook
`logger`

clap.cluster_manager.ActionType¶: An setup action can be a role action, a command or a playbook

class clap.cluster_manager.ClusterConfig¶

A full cluster configuration

after :Optional[List[SetupConfig]]¶: List of SetupConfig to perform in after phase

after_all :Optional[List[SetupConfig]]¶: List of SetupConfig to perform in after_all phase

before :Optional[List[SetupConfig]]¶: List of SetupConfig to perform in before phase

before_all :Optional[List[SetupConfig]]¶: List of SetupConfig to perform in before_all phase

cluster_config_id :str¶: Name of the cluster configuration

nodes :Optional[Dict[str, NodeConfig]]¶: Dictionary with cluster’s node type as name and NodeConfig as value

options :Optional[ClusterOptions]¶: Optional cluster options (ClusterOptions)

class clap.cluster_manager.ClusterConfigDatabase(cluster_files, discard_invalids=True, load=True)¶

This class stores information about clusters configurations in CLAP system.

Parameters

cluster_files – List of cluster files to parse.
discard_invalids – If true, discard invalid configurations without raising any exception. Otherwise, raises a ClusterError exception.
load – If true, load all files when this class is created. Otherwise, use load() method.

clusters :Dict[str, ClusterConfig]¶: Dictionary with cluster configurations

_load_cluster_and_setups(self)¶

load(self)¶: Load all cluster configurations from cluster files. Configurations will be stored in cluster attribute.

class clap.cluster_manager.ClusterConfigFile¶

A dataclass that represents a cluster configuration file

clusters :Optional[Dict[str, _ClusterConfig]]¶: Dictionary with cluster name as key and _ClusterConfig as value

setups :Optional[Dict[str, SetupConfig]]¶: Dictionary with setup name as key and SetupConfig as value

exception clap.cluster_manager.ClusterConfigurationError¶

Bases: Exception

Common base class for all non-exit exceptions.

Initialize self. See help(type(self)) for accurate signature.

class clap.cluster_manager.ClusterDescriptor¶

Dataclass that describes an created CLAP cluster

cluster_config :ClusterConfig¶: Cluster configuration used

cluster_id :str¶: ID of the cluster

cluster_name :str¶: Name of the cluster configuration used

creation_time :float¶: Date that this cluster was created

is_setup :bool = False¶: Boolean indicating if cluster was already setup

update_time :float¶: Last time of information of this cluster was updated

exception clap.cluster_manager.ClusterError¶

Bases: Exception

Common base class for all non-exit exceptions.

Initialize self. See help(type(self)) for accurate signature.

class clap.cluster_manager.ClusterManager(node_manager, role_manager, config_db, cluster_repository_controller, private_dir, cluster_tag_prefix='.cluster:')¶

This class is used to start, stop, pause, resume and perform actions: in clusters. It is responsible to manage clusters (creating and removing them from the repository).

Parameters

node_manager – Class used to manage nodes.
role_manager – Class used to manage roles.
config_db – Class used to obtain cluster configurations.
cluster_repository_controller – Class used to manage clusters at a repository.
private_dir – Path to the private directory (where private keys are stored).
cluster_tag_prefix – Optional prefix to tag nodes that belongs to a cluster.

_run_setup_list(self, setups, node_ids)¶

Parameters

setups (List[SetupConfig]) –
node_ids (List[str]) –

Return type

bool

add_cluster_tag(self, node_ids, cluster_id, node_type)¶

Given a list of node ids, a cluster id and a cluster’s node type,: add the cluster tag to the nodes. Once the tag is added to a node, the node belongs to a cluster. So, a cluster is a set of nodes with tagged with an specified tag.

Parameters

node_ids (List[str]) – List node ids to add the cluster tag.
cluster_id (str) – ID of the cluster that this node will belong.
node_type (str) – Cluster’s node type.

Returns

The node ids of nodes that the tag was added.

Return type

List[str]

add_existing_nodes_to_cluster(self, cluster_id, node_types, max_workers=1)¶

Add already created nodes to a cluster as a desired cluster’s node: type. The cluster will be setup up after adding these nodes to the cluster.

Parameters

cluster_id (str) – ID of the cluster to add the nodes.
node_types (Dict[str, List[str]]) – Dictionary with cluster’s node type as key and list of node ids as values.
max_workers (int) – Number of threads to perform setup actions.

get_all_cluster_nodes(self, cluster_id)¶

Get all nodes that belong to a cluster.

Parameters: cluster_id (str) – ID of the cluster to retrieve the nodes.
Returns: A list of node ids of nodes that belongs to the cluster.
Return type: List[str]

get_all_clusters(self)¶

Get all clusters information from the cluster repository.

Returns: A list of clusters information.
Return type: List[ClusterDescriptor]

get_cluster_by_id(self, cluster_id)¶

Get a cluster information ClusterDescriptor from the: cluster repository.

Parameters: cluster_id (str) – ID of the cluster to retrieve.
Returns: A cluster information.
Return type: ClusterDescriptor

get_cluster_nodes_types(self, cluster_id)¶

Get all nodes and the nodes’ types from nodes that belong to a: cluster.

Parameters: cluster_id (str) – ID of the cluster to retrieve the nodes.
Returns: A dictionary where keys are the cluster’s node types and values are lists of nodes ids from nodes of this cluster’s node type.
Return type: Dict[str, List[str]]

grow(self, cluster_id, node_type, count=1, min_count=0, start_timeout=600)¶

Starts new nodes from a cluster, based on its cluster’s node type.: The nodes will be started and tagged to belong to the cluster.

Parameters

cluster_id (str) – ID of the cluster to add more nodes.
node_type (str) – Cluster’s node type to start.
count (int) – Number of nodes to start.
min_count (int) – Minimum number of nodes that must be started. If this number is not reached, all nodes are terminated.
start_timeout (int) – Timeout to start nodes. If nodes are not started within this timeout, it will be terminated.

Returns

A list of node ids of the nodes that were started.

Return type

List[str]

is_alive(self, cluster_id, retries=5, wait_timeout=30, update_timeout=30, max_workers=1, test_command='echo "OK"')¶

Check if a cluster is alive, checking the aliveness of all nodes: that belongs the cluster.

Parameters

cluster_id (str) – ID of the cluster to check for aliveness.
retries (int) – Number of check retries.
wait_timeout (int) – Timeout to perform another check if previous fails.
update_timeout (int) – Timeout to update node information.
max_workers (int) – Number of threads to check for aliveness.
test_command (str) – Command to be executed in nodes to test for aliveness.

Returns

A dictionary where keys are the IDs of nodes and values are booleans indicating if node is alive or not.

Return type

Dict[str, bool]

pause_cluster(self, cluster_id, timeout=180, max_workers=1)¶

Pause the cluster, pausing all nodes that belongs to it.

Parameters

cluster_id (str) – ID of the cluster to pause nodes
timeout (int) – Pause timeout
max_workers (int) – Number of threads to perform pause process.

Returns

ID of the nodes that were sucessfuly paused

Return type

List[str]

resume_cluster(self, cluster_id, timeout=180, max_workers=1)¶

Resumes a cluster, resuming all nodes that belongs to it

Parameters

cluster_id (str) – ID of the cluster to resume.
timeout (int) – Timeout to resume nodes.
max_workers (int) – Number of threads in the resume process.

Returns

ID of the nodes that were successfully resumed.

Return type

List[str]

run_action(self, action, node_ids)¶

Run a cluster’s action in a set of nodes.

Parameters

action (ActionType) – Cluster’s action to be performed
node_ids (List[str]) – ID of the nodes to perform this action

Returns

True if action as sucessfully performed and false otherwise

Return type

bool

run_role_add(self, role, node_ids)¶

Add nodes to a role

Parameters

role (RoleAdd) – Role to add to nodes
node_ids (List[str]) – ID of the nodes to add the role

Returns

True if nodes were added to the role and false otherwise

Return type

bool

run_setup(self, setup, node_ids)¶

Runs a cluster’s setup configuration at a list of nodes

Parameters

setup (SetupConfig) – Setup to perform in nodes
node_ids (List[str]) – ID of the nodes to perform this setup

Returns

True if the setup was successfully executed and false otherwise

Return type

bool

setup_cluster(self, cluster_id, nodes_being_added=None, max_workers=1, start_at_stage='before_all')¶

Setups a cluster. It will run all setups in order.

Parameters

cluster_id (str) – ID of the cluster to perform setup
nodes_being_added (Dict[str, List[str]]) – List of nodes that is being added now to the cluster. It affects before, node and after stages. If None, it supposes that all nodes are being added to the cluster now.
max_workers (int) – NUmber of threads to run setup configs.
start_at_stage (str) – Stage to start the configuration. It can be: ‘before_all’, ‘before’, ‘node’, ‘after’ or ‘after_all’

start_cluster(self, cluster_config, start_timeout=600, max_workers=1, destroy_on_min_count=True)¶

Create a cluster, based on a ClusterConfig. It will start: the desired nodes and tag them to belong to the cluster. After, a new cluster will be created at cluster’s repository.

Parameters

cluster_config (ClusterConfig) – Cluster configuration used to start a cluster
start_timeout (int) – Timeout to start nodes. If nodes are not started within this timeout, it will be terminated.
max_workers (int) – Number of threads to start nodes in parallel
destroy_on_min_count (bool) – If True, the cluster will be destroyed (all nodes will be terminated and cluster is not created) if any cluster’s node type min_count is not reached.

Returns

The cluster ID of the newly created cluster.

Return type

str

start_cluster_node(self, cluster_id, node_type, instance_info, count, start_timeout=600)¶

Parameters

cluster_id (str) –
node_type (str) –
instance_info (clap.configs.InstanceInfo) –
count (int) –
start_timeout (int) –

Return type

List[str]

stop_cluster(self, cluster_id, timeout=180, max_workers=1, remove_cluster=True)¶

Stop a cluster, stopping all nodes that belongs to it.

Parameters

cluster_id (str) – ID of the cluster to stop;
timeout (int) – Timeout to stop nodes.
max_workers (int) – Number of threads in the stop process.
remove_cluster (bool) – If True, also removes the cluster from the repository.

Returns

ID of the nodes that were successfully stopped.

Return type

List[str]

upsert_cluster(self, cluster)¶

Create or update a cluster in cluster’s repository.

Parameters: cluster (ClusterDescriptor) – Cluster to upsert.

class clap.cluster_manager.ClusterOptions¶

Dataclass that stores information about optional cluster options in a cluster configuration file

ssh_to :Optional[str]¶: Name od the node type to perform ssh

class clap.cluster_manager.ClusterRepositoryController(repository)¶

This class is used manipulate clusters in a repository. It performs all loads and stores of ClusterDescriptor in a repository.

Parameters: repository – Cluster repository used to store ClusterDescriptor.

get_all_clusters(self)¶

Retrieve all clusters from the repository.

Returns: A list of clusters in the repository.
Return type: List[ClusterDescriptor]

get_cluster_by_id(self, cluster_id)¶

Retrieve a cluster from the repository.

Parameters: cluster_id (str) – ID of the cluster to retrieve.
Returns: The cluster.
Return type: ClusterDescriptor

remove_cluster(self, cluster_id)¶

Remove a cluster from the repository based on its cluster ID.

Parameters: cluster_id (str) – ID of the cluster to remove.

upsert_cluster(self, cluster)¶

Upsert (create or update) a cluster in repository.

Parameters: cluster (ClusterDescriptor) – A cluster to be stored. The cluster ID will be used to identify the cluster in the repository.

exception clap.cluster_manager.ClusterSetupError¶

Bases: ClusterError

Common base class for all non-exit exceptions.

Initialize self. See help(type(self)) for accurate signature.

class clap.cluster_manager.CommandActionType¶

Dataclass that stores information about a command to perform from a cluster setup

command :str¶: The command to execute

exception clap.cluster_manager.InvalidClusterError(cluster_id)¶

Bases: Exception

Common base class for all non-exit exceptions.

Initialize self. See help(type(self)) for accurate signature.

exception clap.cluster_manager.InvalidSetupError(cluster_name, setup_name)¶

Bases: ClusterConfigurationError

Common base class for all non-exit exceptions.

Initialize self. See help(type(self)) for accurate signature.

class clap.cluster_manager.NodeConfig¶

Dataclass that stores information about a cluster’s node at cluster configuration

count :int¶: Number of nodes to start

min_count :Optional[int]¶: Minimum number of nodes that must sucessfully start

setups :List[SetupConfig]¶: List of SetupConfig that must be performed in this node

type :str¶: Type of the instance used (refers to instances.yaml file names)

exception clap.cluster_manager.NodeSizeError¶

Bases: ClusterConfigurationError

Common base class for all non-exit exceptions.

Initialize self. See help(type(self)) for accurate signature.

class clap.cluster_manager.PlaybookActionType¶

Dataclass that stores information about a playboook to execute from a cluster setup

extra :Optional[Dict[str, str]]¶: Optional extra arguments

playbook :str¶: Playbook to execute

class clap.cluster_manager.RoleActionType¶

Dataclass that stores information about a role’s action from a cluster setup

action :str¶: Role’s action name

extra :Optional[Dict[str, str]]¶: Optional extra arguments

role :str¶: Name of the role

class clap.cluster_manager.RoleAdd¶

Dataclass that stores information about a role that mst be added to nodes from a cluster setup

extra :Optional[Dict[str, str]]¶: Optional extra arguments from role’s setup action (if any)

name :str¶: Name of the role to be added

class clap.cluster_manager.SetupConfig¶

Dataclass that stores information about a Setup configuration in a cluster configuration file

actions :Optional[List[ActionType]]¶: Optional list of actions to perform

roles :Optional[List[RoleAdd]]¶: Optional list of roles to add

class clap.cluster_manager._ClusterConfig¶

Temporary class about cluster config (used to parse cluster configuration file only)

after :Optional[List[str]]¶

after_all :Optional[List[str]]¶

before :Optional[List[str]]¶

before_all :Optional[List[str]]¶

nodes :Optional[Dict[str, _NodeConfig]]¶

options :Optional[ClusterOptions]¶

class clap.cluster_manager._NodeConfig¶

Temporary class about node config (used to parse cluster configuration file only)

count :int¶

min_count :Optional[int]¶

setups :List[str]¶

type :str¶

clap.cluster_manager.logger¶

clap.cluster_manager¶

Module Contents¶

Classes¶

Attributes¶

`clap.cluster_manager`¶