remote — Spicerack 0.1.dev2+gcef5c6c7d documentation

remote — Spicerack 0.1.dev2+gcef5c6c7d documentation
Python API
remote
View page source
remote
Remote module to execute commands on hosts via Cumin.
exception
spicerack.remote.
RemoteCheckError
[source]
Bases:
SpicerackCheckError
Custom exception class for check errors of this module.
exception
spicerack.remote.
RemoteClusterExecutionError
results
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
failures
list
spicerack.remote.RemoteExecutionError
[source]
Bases:
RemoteError
Custom exception class for collecting multiple execution errors on a cluster.
Override the parent constructor to add failures and results as attributes.
Parameters
results
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
) -- the remote results.
failures
list
spicerack.remote.RemoteExecutionError
) -- the list of exceptions raised in the cluster execution.
exception
spicerack.remote.
RemoteError
[source]
Bases:
SpicerackError
Custom exception class for errors of this module.
exception
spicerack.remote.
RemoteExecutionError
retcode
int
message
str
results
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
None
[source]
Bases:
RemoteError
Custom exception class for remote execution errors.
Override parent constructor to add the return code attribute.
Parameters
retcode
int
) -- the return code of the remote execution.
message
str
) -- the exception message.
results
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
) -- the cumin results.
class
spicerack.remote.
LBRemoteCluster
config
cumin.Config
remote_hosts
spicerack.remote.RemoteHosts
conftool
spicerack.confctl.ConftoolEntity
None
[source]
Bases:
RemoteHostsAdapter
Class usable to operate on a cluster of servers with pooling/depooling logic in conftool.
Initialize the instance.
Parameters
config
cumin.Config
) -- cumin configuration.
remote_hosts
spicerack.remote.RemoteHosts
) -- the instance to act on the remote hosts.
conftool
spicerack.confctl.ConftoolEntity
) -- the conftool entity to operate on.
reload_services
services
list
str
svc_to_depool
list
str
batch_size
int
batch_sleep
float
None
None
verbose
bool
True
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
[source]
Reload services in batches, removing the host from all the affected services first.
Parameters
services
list
str
) -- A list of services to act upon.
svc_to_depool
list
str
) -- A list of services (in conftool) to depool.
batch_size
int
, default:
) -- the batch size for cumin, as an integer.Defaults to 1
batch_sleep
typing.Optional
float
, default:
None
) -- the batch sleep between groups of runs.
verbose
bool
, default:
True
) -- whether to print Cumin's output and progress bars to stdout/stderr.
Return type
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
Returns
What
cumin.transports.BaseWorker.get_results()
returns to allow to iterate over the results.
Raises
spicerack.remote.RemoteExecutionError
spicerack.remote.RemoteClusterExecutionError
-- if the Cumin execution
returns a non-zero exit code.
restart_services
services
list
str
svc_to_depool
list
str
batch_size
int
batch_sleep
float
None
None
verbose
bool
True
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
[source]
Restart services in batches, removing the host from all the affected services first.
Parameters
services
list
str
) -- A list of services to act upon.
svc_to_depool
list
str
) -- A list of services (in conftool) to depool.
batch_size
int
, default:
) -- the batch size for cumin, as an integer. Defaults to 1.
batch_sleep
typing.Optional
float
, default:
None
) -- the batch sleep between groups of runs.
verbose
bool
, default:
True
) -- whether to print Cumin's output and progress bars to stdout/stderr.
Return type
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
Returns
What
cumin.transports.BaseWorker.get_results()
returns to allow to iterate over the results.
Raises
spicerack.remote.RemoteExecutionError
spicerack.remote.RemoteClusterExecutionError
-- if the Cumin execution
returns a non-zero exit code.
run
commands
str
cumin.transports.Command
svc_to_depool
list
str
None
None
batch_size
int
batch_sleep
float
None
None
is_safe
bool
False
max_failed_batches
int
print_output
bool
True
print_progress_bars
bool
True
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
[source]
Run commands while depooling servers in groups of batch_size.
For clusters behind a load balancer, we typically want to be able to
depool a server from a specific service, then run any number of commands
on it, and finally repool it.
We also want to ensure we can only have at max N hosts depooled at any
time. Given cumin doesn't have pre- and post- execution hooks, we break
the remote run in smaller groups and execute on one group at a time,
in parallel on all the servers.
Note this works a bit differently than how the cumin moving window works,
as here we'll have to wait for the execution on all servers in a group
before moving on to the next.
Parameters
*commands
typing.Union
str
cumin.transports.Command
) -- Arbitrary number of commands to execute.
svc_to_depool
typing.Optional
list
str
]]
, default:
None
) -- A list of services (in conftool) to depool.
batch_size
int
, default:
) -- the batch size for cumin, as an integer. Defaults to 1.
batch_sleep
typing.Optional
float
, default:
None
) -- the batch sleep in seconds to use before scheduling the next batch of hosts.
is_safe
bool
, default:
False
) -- whether the command is safe to run also in dry-run mode because it's a read-only
max_failed_batches
int
, default:
) -- Maximum number of batches that can fail. Defaults to 0.
print_output
bool
, default:
True
) -- whether to print Cumin's output to stdout.
print_progress_bars
bool
, default:
True
) -- whether to print Cumin's progress bars to stderr.
Return type
list
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
Returns
What
cumin.transports.BaseWorker.get_results()
returns to allow to iterate over the results.
Raises
spicerack.remote.RemoteExecutionError
spicerack.remote.RemoteClusterExecutionError
-- if the Cumin execution
returns a non-zero exit code.
class
spicerack.remote.
Remote
config
str
dry_run
bool
True
None
[source]
Bases:
object
Remote class to interact with Cumin.
Initialize the instance.
Parameters
config
str
) -- the path of Cumin's configuration file.
dry_run
bool
, default:
True
) -- whether this is a DRY-RUN.
query
query_string
str
use_sudo
bool
False
spicerack.remote.RemoteHosts
[source]
Execute a Cumin query and return the matching hosts.
Parameters
query_string
str
) -- the Cumin query string to execute.
use_sudo
bool
, default:
False
) -- If True will prepend 'sudo -i' to every command.
Return type
spicerack.remote.RemoteHosts
query_confctl
conftool
spicerack.confctl.ConftoolEntity
**
tags
str
spicerack.remote.LBRemoteCluster
[source]
Execute a conftool node query and return the matching hosts.
Parameters
conftool
spicerack.confctl.ConftoolEntity
) -- the conftool instance for the node type objects.
tags
str
) -- Conftool tags for node type objects as keyword arguments.
Raises
spicerack.remote.RemoteError
-- if unable to query the hosts.
Return type
spicerack.remote.LBRemoteCluster
class
spicerack.remote.
RemoteHosts
config
cumin.Config
hosts
ClusterShell.NodeSet.NodeSet
dry_run
bool
True
use_sudo
bool
False
None
[source]
Bases:
object
Class to execute remote commands on hosts. The instances are also iterable.
Initialize the instance.
Parameters
config
cumin.Config
) -- the configuration for Cumin.
hosts
ClusterShell.NodeSet.NodeSet
) -- the hosts to target for the remote execution.
dry_run
bool
, default:
True
) -- whether this is a DRY-RUN.
use_sudo
bool
, default:
False
) -- if True will prepend
sudo
-i
to every command.
Raises
spicerack.remote.RemoteError
-- if no hosts were provided.
static
results_to_list
results
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
callback
collections.abc.Callable
None
None
list
tuple
ClusterShell.NodeSet.NodeSet
Any
[source]
Extract execution results into a list converting them with an optional callback.
Todo
move it directly into Cumin.
Parameters
results
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
) -- generator returned by run_sync() and run_async() to iterate over the results.
callback
typing.Optional
collections.abc.Callable
, default:
None
) -- an optional callable to apply to each result output (it can be multiline).
The callback will be called with a the string output as the only parameter and must return the
extracted value. The return type can be chosen freely.
Return type
list
tuple
ClusterShell.NodeSet.NodeSet
typing.Any
]]
Returns
A list of 2-element tuples with hosts
ClusterShell.NodeSet.NodeSet
as first item and the
extracted outputs
str
as second. This is because NodeSet are not hashable.
Raises
spicerack.remote.RemoteError
-- if unable to run the callback.
__iter__
collections.abc.Iterator
spicerack.remote.RemoteHosts
[source]
Iterate over all remote hosts.
Yields
spicerack.remote.RemoteHosts
-- an instance for each host.
Return type
collections.abc.Iterator
spicerack.remote.RemoteHosts
__len__
int
[source]
Returns the number of hosts targeted.
Return type
int
__str__
str
[source]
String representation of the instance.
Return type
str
Returns
the hosts of the current instance in
NodeSet
notation.
get_subset
subset
ClusterShell.NodeSet.NodeSet
spicerack.remote.RemoteHosts
[source]
Return a new RemoteHosts instance with a subset of the existing set of hosts.
Parameters
subset
ClusterShell.NodeSet.NodeSet
) -- the subset of hosts to select. They must all be part of the current set.
Raises
spicerack.remote.RemoteError
-- if any host in the subset is not part of the instance hosts.
Return type
spicerack.remote.RemoteHosts
Returns
a new instance with only the subset hosts.
reboot
batch_size
int
batch_sleep
float
None
180.0
None
[source]
Reboot hosts.
Parameters
batch_size
int
, default:
) -- how many hosts to reboot in parallel.
batch_sleep
typing.Optional
float
, default:
180.0
) -- how long to sleep between one reboot and the next.
Return type
None
run_async
commands
str
cumin.transports.Command
success_threshold
float
1.0
batch_size
int
str
None
None
batch_sleep
float
None
None
is_safe
bool
False
print_output
bool
True
print_progress_bars
bool
True
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
[source]
Execute commands on hosts matching a query via Cumin in async mode and return its results.
Parameters
*commands
typing.Union
str
cumin.transports.Command
) -- arbitrary number of commands to execute on the target hosts.
success_threshold
float
, default:
1.0
) -- to consider the execution successful, must be between 0.0 and 1.0.
batch_size
typing.Union
int
str
None
, default:
None
) -- the batch size for cumin, either as percentage (e.g.
25%
) or absolute number (e.g.
).
batch_sleep
typing.Optional
float
, default:
None
) -- the batch sleep in seconds to use in Cumin before scheduling the next host.
is_safe
bool
, default:
False
) -- whether the command is safe to run also in dry-run mode because it's a read-only command that
doesn't modify the state.
print_output
bool
, default:
True
) -- whether to print Cumin's output to stdout.
print_progress_bars
bool
, default:
True
) -- whether to print Cumin's progress bars to stderr.
Raises
spicerack.remote.RemoteExecutionError
-- if the Cumin execution returns a non-zero exit code.
Return type
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
run_sync
commands
str
cumin.transports.Command
success_threshold
float
1.0
batch_size
int
str
None
None
batch_sleep
float
None
None
is_safe
bool
False
print_output
bool
True
print_progress_bars
bool
True
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
[source]
Execute commands on hosts matching a query via Cumin in sync mode and returns its results.
Parameters
*commands
typing.Union
str
cumin.transports.Command
) -- arbitrary number of commands to execute on the target hosts.
success_threshold
float
, default:
1.0
) -- to consider the execution successful, must be between 0.0 and 1.0.
batch_size
typing.Union
int
str
None
, default:
None
) -- the batch size for cumin, either as percentage (e.g.
25%
) or absolute number (e.g.
).
batch_sleep
typing.Optional
float
, default:
None
) -- the batch sleep in seconds to use in Cumin before scheduling the next host.
is_safe
bool
, default:
False
) -- whether the command is safe to run also in dry-run mode because it's a read-only command that
doesn't modify the state.
print_output
bool
, default:
True
) -- whether to print Cumin's output to stdout.
print_progress_bars
bool
, default:
True
) -- whether to print Cumin's progress bars to stderr.
Raises
spicerack.remote.RemoteExecutionError
-- if the Cumin execution returns a non-zero exit code.
Return type
collections.abc.Iterator
tuple
ClusterShell.NodeSet.NodeSet
ClusterShell.MsgTree.MsgTreeElem
]]
split
n_slices
int
collections.abc.Iterator
spicerack.remote.RemoteHosts
[source]
Split the current remote in n_slices RemoteHosts instances.
Parameters
n_slices
int
) -- the number of slices to slice the remote in.
Yields
spicerack.remote.RemoteHosts
-- the instances for the subset of nodes.
Return type
collections.abc.Iterator
spicerack.remote.RemoteHosts
uptime
print_progress_bars
bool
True
list
tuple
ClusterShell.NodeSet.NodeSet
float
[source]
Get current uptime.
Parameters
print_progress_bars
bool
, default:
True
) -- whether to print Cumin's progress bars to stderr.
Return type
list
tuple
ClusterShell.NodeSet.NodeSet
float
]]
Returns
A list of 2-element
tuple
instances with hosts
ClusterShell.NodeSet.NodeSet
as first item and
float
uptime as second item.
Raises
spicerack.remote.RemoteError
-- if unable to parse the output as an uptime.
wait_reboot_since
since
datetime.datetime
print_progress_bars
bool
True
None
[source]
Poll the host until is reachable and has an uptime lower than the provided datetime.
Parameters
since
datetime.datetime
) -- the time after which the host should have booted.
print_progress_bars
bool
, default:
True
) -- whether to print Cumin's progress bars to stderr.
Raises
spicerack.remote.RemoteCheckError
-- if unable to connect to the host or the uptime is higher than expected.
When in DRY-RUN mode, it will raise only if unable to connect.
Return type
None
property
dry_run
bool
Getter for know if it was instantiated with DRY-RUN or not. Useful for RemoteHostsAdapter instances.
property
hosts
NodeSet
Returns a copy of the current hosts targeted.
class
spicerack.remote.
RemoteHostsAdapter
remote_hosts
spicerack.remote.RemoteHosts
None
[source]
Bases:
object
Base adapter to write classes that expand the capabilities of RemoteHosts.
This adapter class is a helper class to reduce duplication when writing classes that needs to add capabilities to a
RemoteHosts instance. The goal is to not extend the RemoteHosts but instead delegate to its instances.
This class fits when a single RemoteHosts instance is enough, but for more complex cases, in which multiple
RemoteHosts instances should be orchestrated, it's ok to not extend this class and create a standalone one.
Initialize the instance.
Parameters
remote_hosts
spicerack.remote.RemoteHosts
) -- the instance to act on the remote hosts.
__len__
int
[source]
Get the number of target hosts in this instance.
Return type
int
__str__
str
[source]
String representation of the instance.
Return type
str
Returns
the string representation of the
remote_hosts
of the instance in
NodeSet
notation.
property
remote_hosts
RemoteHosts
Getter for the remote_hosts property.
Returns
The instance to run commands on the underlying remote hosts.