Overwatch Base Package (`overwatch.base`)¶

General information from the README is available immediately below. Further module specific documentation is available further below in the package reference, with the modules listed below.

Base package details

Package reference

Package reference ¶

overwatch.base.config module ¶

Handles configuration of overwatch via yaml.

Configurations are built in a hierarchy, with the base configuration providing the first layer, and building up further until the specified module.

YAML parsing plugins are also specified here. This breaks the abstraction a little bit, but it makes things much simpler, so it’s worth the trade-off.

overwatch.base.config.bcrypt(loader, node)[source]¶

Hash any given passwords according to the provided number of rounds. Specified by !bcrypt (defined on registration below).

Block should look like:

bcryptExampleBlock: !bcrypt
    # Could be defined elsewhere and referenced using an anchor here
    bcryptLogRounds: 12
    user1: "password1"
    user2: "password2"

The block will result in two users, user1 and user2 being added to the database. Their passwords will be the hash of the given strings using 12 bcrypt log rounds.

Note

bcryptLogRounds can also be omitted. In that case, it will default to 12.

Parameters:	loader (yaml.SafeLoader) – YAML loader which is parsing the configuration. node (MappingNode) – Node containing something like the block above.
Returns:	Keys are usernames, while values are the corresponding hashed passwords.
Return type:	dict

overwatch.base.config.bcryptLogRounds = 12¶: Number of rounds of hashing when using bcrypt. Define the default value here so it can be accessed outside of the module. This is a hack, but I think it is worth the simplification in everything else.

class overwatch.base.config.configurationType(*args, **kwds)[source]¶

Bases: aenum.OrderedEnum

Specifies the module ordering for loading of configurations.

It is also used to specify the maximum level for which a config should be loaded. For example, if webApp is specified, it should load all configurations, while for processing, everything but processing should be loaded.

The numerical values of this enum basically specify the dependencies of the package.

Note

The names of these values must match the names of their corresponding modules!

api = <configurationType.api: 2>¶

base = <configurationType.base: 0>¶

processing = <configurationType.processing: 3>¶

receiver = <configurationType.receiver: 1>¶

webApp = <configurationType.webApp: 4>¶

overwatch.base.config.determineRunPageTemplates(loader, node)[source]¶

Determine which subsystems have run page templates on startup by determining the filenames of each run pages templates. We will later check for a subsystem specific filename is this list. Specified by !findRunPageTemplates (defined on registration below) and should be defined with the path to the templates directory.

Since this is run from the root directory, we need to go into the “webApp” directory to find the templates!

Parameters:	loader (yaml.SafeLoader) – YAML loader which is parsing the configuration. node (SequenceNode) – Node containing the list of the paths to join together.
Returns:	Subsystems which have a run page template.
Return type:	list

overwatch.base.config.joinPaths(loader, node)[source]¶

Join elements of a list into a path using os.path.join. Specified by !joinPaths (defined on registration below).

Inspired by here.

Could similarly use !!python/object/apply:os.path.join, with the downside of allowing lots of arbitrary code execution since you cannot use safe_load. Instead, we write this simple function and then explicitly make only this function available via the SafeLoader.

Parameters:	loader (yaml.Loader) – YAML loader which is parsing the configuration. node (SequenceNode) – Node containing the list of the paths to join together.
Returns:	The list elements joined together into a valid path.
Return type:	str

overwatch.base.config.readConfig(configType)[source]¶

Main function to read the Overwatch configuration.

It looks for values in a set of configuration files according to the module that is specified.

The configuration file must be named config.yaml. The files are read in such an order that values specified in later packages will override earlier ones. For example, webApp depends on processing, so if we specify different values for the same key in both module configurations, the value in the webApp config will be used.

In additional to looking in Overwatch modules, it also looks for a configuration in the current working directory, as well as in the user home directory.

The current (as of August 2018) order of override priority from highest to lowest is:

Current working directory
User home directory
overwatch.webApp
overwatch.processing
overwatch.receiver
overwatch.api
overwatch.base

(this basically follows the dependency tree).

Parameters:	configType (configurationType or str) – Type of the module for which we are loading the configuration.
Returns:	(Fully merged configuration, list of configuration filenames which were read)
Return type:	tuple

overwatch.base.config.readConfigFiles(fileList)[source]¶

Read the configurations from the given list of files.

Parameters:	fileList (list) – List of paths to configuration files.
Returns:	(list of configurations read from the files, list of configuration filenames which were read)
Return type:	tuple

overwatch.base.config.secretKey(loader, node)[source]¶

Determine the secret key for signing cookies in the webApp. Specified by !secretKey (defined on registration below).

If a value is already specified, that value is simply used. However, if an invalid value is passed (for example, None), a value is generated according to the best practices recommendation of the flask developers.

Parameters:	loader (yaml.SafeLoader) – YAML loader which is parsing the configuration. node (ScalarNode) – Node containing the secret key.
Returns:	The secret key for signing cookies.
Return type:	str

overwatch.base.dataTransfer module ¶

overwatch.base.deploy module ¶

Handles deployment and starting of Overwatch and related processes.

It can handle the configuration and execution of:

Environment setup
autossh for SSH tunnels.
ZODB for the Overwatch Database
Overwatch ZMQ receiver
Overwatch receiver data transfer
Overwatch DQM receiver
- Via uswgi, uwsgi behind nginx or directly.
Overwatch processing
Overwatch web app
- Via uswgi, uwsgi behind nginx or directly.
Overwatch data transfer from receivers to other Overwatch sites and EOS.
Overwatch data replay for data transfer via the data transfer module.
Overwatch data replay to simulate receiving new data using existing data.
Overwatch ZMQ receiver monitoring to watch for dead ZMQ receivers.

It can also handle receiving SSH Keys, grid certificates, and grid keys passed in via environment variables.

Various classes of files are stored in specific locations. In particular,

socket files are stored in “exec/sockets”.
config files are stored in “exec/config” (except for those which must be in the current folder, such as the supervisor config, or the overwatch custom config).
log files are in “exec/logs” when the process is executed directly. In the case of supervisor, we redirect the logs to the main supervisor process to allow for all logs to be captured by docker.

Usually, this module is executed directly in docker containers. All options are configured via a YAML file.

class overwatch.base.deploy.autossh(receiver, localPort, config)[source]¶

Bases: overwatch.base.deploy.executable

Start autossh to create a SSH tunnel.

Note

Arguments after config are values which will be used for formatting and are required in the config.

Parameters:

receiver (str) – Three letter subsystem name. For example, EMC.
localPort (int) – Port where the HLT data should be made available on the local system.
config (dict) – Configuration for the executable.
enabled (bool) – True if the task should actually be executed.
runInBackground (bool) – True if the process should be run in the background. This means that process will not be blocking, but it shouldn’t be used in conjunction with supervisor. Default: False.
forceRestart (bool) – True if the process should kill any previous processes before starting. If not and the process already exists, then nothing will be done.
hltPort (int) – Port where the HLT data is available on the remote system.
address (str) – SSH connection address.
port (int) – SSH connection port.
username (str) – SSH connection username.

knownHosts¶

Keep track of the known hosts executable. Default: None.

Type:	executable

setup()[source]¶: Setup the autossh tunnel by additionally setting up the known_hosts file.

overwatch.base.deploy.enableExecutablesFromEnvironment(config)[source]¶

Allow enabling executables in the environmental config.

This allows us to set a base config, but then enable specific executables based on the environment. This is particularly useful for docker containers, where we may want to split up the services into multiple containers while still maintaining the same base configuration.

The executables should be provided as a comma separated list in the environment variable OVERWATCH_EXECUTABLES.

Parameters:	config (dict) – The base configuration. It must contain an “executables” key.
Returns:	The configuration updated to execute the requested executables.
Return type:	dict

class overwatch.base.deploy.environment(config)[source]¶

Bases: object

Setup and create the necessary environment for execution.

It has similar structure to executable, but it is a fundamentally different object, so it doesn’t inherit from it.

Note

Arguments after config are values which are specified in the config.

Parameters:

config (dict) – Configuration for the executable.
root (dict) – Specify setup related to ROOT. Accepts a path key with the value as the location of the thisroot.sh script for ROOT.
vars (dict) –
zmqReceiver (dict) – Specify setup related to the ZMQ receiver. Accepts a path with the value as the directory which contains the receiver. Must be an absolute path. Default: “${PWD}/receiver/bin”.

name¶

Name of the process that we are starting. It doesn’t need to be the executable name, as it’s just used for informational purposes.

Type:	str

description¶

Description of the process for clearer display, etc.

Type:	str

config¶

Configuration for the environment.

Type:	dict

static retrieveSensitiveVariable(name, config)[source]¶

Retrieve the sensitive variable from an environment variable.

It retrieves it under the name specified under variableName in the config, or if not specified, it uses the name of the sensitive variable. It also checks that the variable is not empty.

Parameters:	name (str) – Name of the sensitive variable to be written to a file. config (dict) – Configuration related to the sensitive parameter. In particular, the relevant key is `variableName`, whose value specifies the name of the environment variable.
Returns:	Contents of the sensitive environment variable.
Return type:	str
Raises:	`ValueError` – The requested environment variable is empty.

setup()[source]¶

Setup for creating the execution environment.

In particular, we write our sensitive environment variables, configure ROOT, configure the ZMQ receiver, and set general environment variables.

setupEnvironmentVars()[source]¶

Setup execution environment.

We generically add any environment variables specified in “vars”.

setupReceiverPath()[source]¶

Set the PATH to include the ZMQ receiver executable.

Only configure if the ZMQ receiver path setup is enabled.

setupRoot()[source]¶

Setup ROOT in our execution environment.

This is done by executing thisroot.sh, capturing the output, and then assigning the updated environment to our current environment variables.

Parameters:	None –
Returns:	True if the ROOT environment was setup. Note that it may already be setup, so returning `False` is not necessarily a problem.
Return type:	bool

writeGridCertFromVariableToFile()[source]¶

Write grid certificate from an environment variable to file.

Used primarily for setting up the certificate in a docker container. It looks for a config dictionary stored in the environment dict under the name gridCert.

Note

Both the grid cert and the grid key are necessary for using the grid, EOS, etc!

Parameters:	None. –
Returns:	True if the var was written to file.
Return type:	bool

writeGridKeyFromVariableToFile()[source]¶

Write grid key from an environment variable to file.

Used primarily for setting up the key in a docker container. It looks for a config dictionary stored in the environment dict under the name gridKey.

Note

Both the grid cert and the grid key are necessary for using the grid, EOS, etc!

Parameters:	None. –
Returns:	True if the var was written to file.
Return type:	bool

static writeInfoToSensitiveFile(sensitiveVariable, defaultWriteLocation, config)[source]¶

Write the sensitive information to a sensitive file.

Parameters:	sensitiveVariable (str) – Value to be written to the sensitive file. defaultWriteLocation (str) – Default location for the file to be written (in case it isn’t specified). config (dict) – Configuration related to the sensitive parameter. In particular, the relevant key is `writeLocation`, whose value specifies the file into which the environment variable should be written.
Returns:	Filename where the sensitive variable was written.
Return type:	str
Raises:	`IOError` – If the file already exists and therefore will not be created.

writeSSHKeyFromVariableToFile()[source]¶

Write SSH key from an environment variable to file.

Used primarily for setting up the key in a docker container. It looks for a config dictionary stored in the environment dict under the name sshKey.

Parameters:	None. –
Returns:	True if the var was written to file.
Return type:	bool

writeSensitiveVariableToFile(name, defaultWriteLocation)[source]¶

Write sensitive variable stored in an environment variable to file.

This function drives must of the functionality. The configuration for the variable may have:

name:
    enabled: true
    variableName: "myVarName"
    writeLocation: "path/to/write/location"

Only the enabled field is required. variableName defaults to name if not specified, while writeLocation defaults to``defaultWriteLocation`` if not specified.

Parameters:	name (str) – Name of the sensitive variable to be written to a file. Used to retrieve the configuration in the YAML config.
Returns:	(variable, writeLocation) where variable (str) is the sensitive variable, and writeLocation (str) is the path to the file where the variable was written.
Return type:	tuple
Raises:	`RuntimeError` – If the task is disabled. This should be caught and execution continued.

class overwatch.base.deploy.executable(name, description, args, config)[source]¶

Bases: object

Base executable class.

Note

Arguments after config are values which are set via the config.

Parameters:

name (str) – Name of the process that we are starting. It doesn’t need to be the executable name, as it’s just used for informational purposes.
description (str) – Description of the process for clearer display, etc.
args (list) – List of arguments used to start the process.
config (dict) – Configuration for the executable.
enabled (bool) – True if the executable should actually be executed.
runInBackground (bool) – True if the process should be run in the background. This means that process will not be blocking, but it shouldn’t be used in conjunction with supervisor. Default: False.
forceRestart (bool) – True if the process should kill any previous processes before starting. If not and the process already exists, then nothing will be done.

name¶

Name of the process that we are starting. It doesn’t need to be the executable name, as it’s just used for informational purposes.

Type:	str

description¶

Description of the process for clearer display, etc.

Type:	str

args¶

List of arguments to be executed.

Type:	list

config¶

Configuration for the executable.

Type:	dict

processIdentifier¶

A unique string which identifies the process (to be used to check if it already exists). It may need to include arguments to be unique, which will then depend on the order in which the process arguments are defined. It is determined by the fully formatted arguments.

Type:	str

supervisor¶

True if the process launching should be configured for supervisor. This means that the process won’t be started immediately. Note that this is a class variable, so we only need to set it once when starting the deployment.

Type:	bool

shortExecutionTime¶

True if the executable executes and completes quickly. In this case, supervisor need special options to ensure that it doesn’t think that the executable failed immediately and should be restarted.

Type:	bool

logFilename¶

Filename for the log file. Default: exec/logs/{name}.log.

Type:	str

configFilename¶

Location where the generated configuration file should be stored. Default: None.

Type:	str

runInBackground¶

True if the process should be run in the background. This means that process will not be blocking, but it shouldn’t be used in conjunction with supervisor. Default: False.

Type:	bool

executeTask¶

Whether the executable should actually be executed. Set via the “enabled” field of the config. Default: False.

Type:	bool

static createFilenameDirectory(filename)[source]¶

Create the directory which contains the specified filename.

Note

If the directory already exists, nothing will be done.

Parameters:	filename (str) – Filename whose directory might need to be created.
Returns:	None.

formatMembers()[source]¶

Determine the description and arguments for the executable based on the given configuration.

This formatting is done by generically providing all arguments in the config to the format each string.

Note

This formatting will fail if a particular key doesn’t exist!

Parameters:	None –
Returns:	None. Modifies the `name`, `description`, `args`, and `logFilename` member variables.

getProcessPID()[source]¶

Retrieve the process PID via pgrep for a process identifier by a given identifier.

Note

Since the arguments should be unique, it is expected that only 0 or 1 PID can be returned. In the case of more than that, an exception is raised.

Parameters:	None. –
Returns:	PID(s) from `pgrep`.
Return type:	list
Raises:	`subprocess.CalledProcessError` – If the process called returns a error code other than 0 (which indicates success) or 1 (which indicates that the process was not found). `ValueError` – If we return more than one PID for the given process identifier.

killExistingProcess(sig=2)[source]¶

Kill processes by PID. The kill signal will be sent to the entire process group.

Parameters:	sig (signal.Signals) – Signal to be sent to the processes. Default: signal.SIGINT
Returns:	Number of processes killed.
Return type:	int
Raises:	`RuntimeError` – If the process identifier is found to still have an associated PID after attempting to kill the process.

run()[source]¶

Driver function for running executables.

It sets up the executable, determines if it should be executed, kills existing processes if necessary, allows them to run in the background, executes the process, and then checks if it started successfully.

Parameters:	None. –
Returns:	True if the process was started, or False if the process was not start for some expected reason (such as the task execution was not enabled, the process already exists, etc). If it failed in a manner that is not acceptable, it will raise an exception.
Return type:	bool
Raises:	`RuntimeError` – If the started process doesn’t appear to have launched successfully.

setup()[source]¶

Prepare for calling the executable.

Here we setup the arguments for the executable using values in the config and we determine the processIdentifier.

Parameters:	None –
Returns:	None

startProcessWithLog()[source]¶

Start (or otherwise setup) the process with the given arguments and log the output.

For a normal process, we configure it to log to the given filename and start it immediately. In the case that the process should be launched with supervisor, the process won’t be launched immediately. Instead, the process and log information will be appended to the existing supervisor configuration.

Parameters:	None. –
Returns:	If the process is started immediately, then we return the `Popen` class associated with the started process. Otherwise, we return None.
Return type:	subprocess.Popen or None

supervisor = False

overwatch.base.deploy.expandEnvironmentalVars(loader, node)[source]¶

Expand the environment variables in a scalar (str) value while reading the YAML config.

This plugs into the YAML loader that is responsible for loading the deployment information from the YAML configuration file.

Parameters:

loader (yaml.Loader) – YAML loader which is parsing the configuration.
node (SequenceNode) – Node containing the list of the paths to join together.

Returns:

The scalar value (str) with any recognized environment variables expanded. The expansion: is performed using os.path.expandvars.

Return type:

str

class overwatch.base.deploy.gridTokenProxy(*args, **kwargs)[source]¶

Bases: overwatch.base.deploy.executable

Initialize a grid token proxy.

We could nearly create this task with just functools.partial, but we decided to write out the entire class because we need to specify it as a short execution time task.

Parameters:	args (list) – Absorb ignored arguments from retrieveExecutable(). kwargs (dict) – Absorb ignored arguments from retrieveExecutable().

class overwatch.base.deploy.nginx(config)[source]¶

Bases: overwatch.base.deploy.executable

Start nginx to serve a uwsgi based web app.

Note

Arguments after config are values which will be used for formatting and are required in the config.

Note

It is generally recommended to run nginx in a separate container, but this option is maintained for situations where that is not possible. When run separately, we connect to the web app via an http socket, while When run together, we connect to the uwsgi web app via a socket.

Parameters:

config (dict) – Configuration for the executable.
webAppName (str) – Name of web app (especially the socket) which will be behind the nginx server.
basePath (str) – Path to the nginx settings and configuration directory. Default: “/etc/nginx”.
configPath (str) – Path to the main nginx configuration directory. Default: “${basePath}/conf.d”.
sitesPath (str) – Path to the nginx sites directory. Default: “${basePath}/sites-enabled”.

setup()[source]¶

Setup required for the nginx executable.

In particular, we need to write out the main configuration (which directs to the socket to which traffic should be passed), as well as the gzip configuration.

class overwatch.base.deploy.overwatchDataTransfer(config)[source]¶

Bases: overwatch.base.deploy.overwatchExecutable

Starts the overwatch data transfer executable.

This needs a separate executable because it may transfer data over rsync or xrd, both of which require their own initialization steps. rsync (really, ssh) requires sshKnownHosts(), while xrd requires gridTokenProxy().

setup()[source]¶

Setup required for Overwatch data transfer.

In particular, we write any passed custom configuration options out to an Overwatch YAML config file.

class overwatch.base.deploy.overwatchExecutable(*args, **kwargs)[source]¶

Bases: overwatch.base.deploy.executable

Starts an Overwatch (ie python-based) based executable.

In the config, it looks for:

additionalConfig (dict): Additional options to added to the YAML configuration.

Attributes:

setup()[source]¶

Setup required for Overwatch executables.

In particular, we write any passed custom configuration options out to an Overwatch YAML config file.

writeCustomConfig()[source]¶

Write out a custom Overwatch configuration file.

First, we read in any existing configuration, and then we update that configuration with the newly provided one, rewriting the entire config file.

As an example, for a config as

option1: true
myAdditionalOptions:
    opt2: true
    opt3: 3

we would pass in the key name myAdditionalOptions, and it would write opt2 and opt3 to self.configFilename.

Args: :returns: None.

class overwatch.base.deploy.overwatchFlaskExecutable(*args, **kwargs)[source]¶

Bases: overwatch.base.deploy.overwatchExecutable

Start an Overwatch Flask executable.

Used for starting the web app and the DQM receiver - both of these are flask based.

Note

All args are specified in the config.

Parameters:	additionalConfig (dict) – Additional options to added to the YAML configuration. uwsgi (dict) – Additional options for `uwsgi`. See the `uwsgi` executable class for more details. nginx (dict) – Additional options for `nginx`. See the `nginx` executable class for more details.

nginx¶

Contains the nginx executable if it was requested. This way, we don’t lose reference to the object. Default: None.

Type:	executable

setup()[source]¶

Setup required for an Overwatch flask executable.

In particular, we write any passed custom configuration options out to an Overwatch YAML config file, as well as potentially setup uwsgi and/or setup and run nginx.

overwatch.base.deploy.retrieveExecutable(name, config)[source]¶

Retrieve an expected by name.

This is an extremely minimal helper function to allow for flexibility in the future.

The available executables are:

supervisor
zodb
autossh
zmqReceiver
dqmReceiver
dataTransfer
dataReplayDataTransfer
dataReplay
receiverMonitor
processing
webApp

Parameters:	name (str) – Name of the executable “type”. For example, “processing” for Overwatch processing. An extensive list is in this docstring. config (dict) – Configuration to be used to initialize the object.
Returns:	The requested executable.
Return type:	executable
Raises:	`ValueError` – If the requested executable doesn’t exist.

overwatch.base.deploy.run()[source]¶: Setup and run Overwatch deployment from the terminal.

overwatch.base.deploy.runExecutables(executables)[source]¶

Run a given set of executables.

Parameters:	executables (dict) – Executable configurations to execute. Keys are the executable names (up to a “_{tag}”), and the values are their configurations.
Returns:	Executables which were run. Key is the name in the config, while the value is the executable.
Return type:	dict

class overwatch.base.deploy.sshKnownHosts(config)[source]¶

Bases: overwatch.base.deploy.executable

Create a SSH known_hosts file with the SSH address in the configuration.

Note

If the known_hosts file exists, it is assumed to have the address already and is not created! If this is problematic in the future, we could instead amend to the file. However, in that case, it will be important to check for duplicates.

Note

Arguments after config are values which will be used for formatting and are required in the config.

Parameters:

config (dict) – Configuration for the executable.
enabled (bool) – True if the task should actually be executed.
runInBackground (bool) – True if the process should be run in the background. This means that process will not be blocking, but it shouldn’t be used in conjunction with supervisor. Default: False.
forceRestart (bool) – True if the process should kill any previous processes before starting. If not and the process already exists, then nothing will be done.
address (str) – SSH connection address.
port (int) – SSH connection port.

knownHostsPath¶

Path to the known hosts file. Assumed to be at $HOME/.ssh/known_hosts.

Type:	str

setup()[source]¶

Setup creating the known_hosts file.

In particular, the executable should only be run if the known_hosts file doesn’t exist.

overwatch.base.deploy.startOverwatch(configFilename, configEnvironmentVariable)[source]¶

Start the various parts of Overwatch.

Components are only started if they are enabled in the configuration. Note that setup() is performed for all executables!

Parameters:	configFilename (str) – Filename of the configuration. configEnvironmentVariable (str) – Name of the environment variable which contains the configuration as a string. This is usually created by reading a config file into the variable with `var=$(cat deployConfig.yaml)`.
Returns:	None.
Raises:	`ValueError` – If both a configuration filename and a configuration environment variable are specified. Only specify one.

class overwatch.base.deploy.supervisor(*args, **kwargs)[source]¶

Bases: overwatch.base.deploy.executable

Start supervisor (through supervisord) to manage processes.

We don’t need options for this executable. It is either going to be launched or it isn’t.

Note

Control of supervisor is made available on port 9001.

Note

The overall program is called supervisor, while the daemon is known as supervisord and the config is stored in supervisord.conf.

Note

Don’t use run() for this executable. Instead, the setup and execution steps should be performed separately because the basic config is needed at the beginning, while the final execution is needed at the end.

Parameters:	args (list) – Absorb ignored arguments from retrieveExecutable(). kwargs (dict) – Absorb ignored arguments from retrieveExecutable().

run()[source]¶: We want to run in separate steps, so this function shouldn’t be used.

setup()[source]¶

Setup required for the supervisor executable.

In particular, we need to write out the main configuration.

class overwatch.base.deploy.uwsgi(*args, **kwargs)[source]¶

Bases: overwatch.base.deploy.executable

Start a uwsgi executable.

Additional options for uwsgi can be specified in the additionalOptions dict of the config. Primary options can be specified in the main uwsgi dict. (Functionality, values in either one will work fine, although http-socket or wsgi-socket and module are required to be in the main config).

Note

Arguments after config are values which are specified in the config.

Note

uwsgi stats are made available by default on port 9002 (while 9001 is occupied by the = supervisor controls). This value can be overridden by specifying the stats value in the config (or in the additional options).

Parameters:

name (str) – Name of the uwsgi web app. It should be unique, but without spaces!
description (str) – Description of the process for clearer display, etc.
args (list) – List of arguments to be executed.
config (dict) – Configuration for the executable.
enabled (bool) – True if the task should actually be executed.
runInBackground (bool) – True if the process should be run in the background. This means that process will not be blocking, but it shouldn’t be used in conjunction with supervisor. Default: False.
forceRestart (bool) – True if the process should kill any previous processes before starting. If not and the process already exists, then nothing will be done.
module (str) – Module (ie import) path for the web app. For example, overwatch.webApp.run. All Overwatch packages should have a run module. Note that this will not run the development server.
http-socket (str) – IP address and port under which the web app will be available via http.
wsgi-socket (str) – IP address and port or unix socket under which the web app will be available via the uwsgi protocol. Not used by default. Either it or http-socket can be specified - not both.
additionalOptions (dict) – Additional options beyond those specified above which should be added to the uwsgi config. It will override default values.

classmethod createObject(obj)[source]¶

Create the underlying uwsgi object if requested in the given executable config.

Note that this expects a uwsgi configuration block inside of the executable config.

Parameters:	cls (uwsgi) – uwsgi class which will be used to create a uwsgi executable. obj (executable) – Executable which may be run via uwsgi. Its configuration determines where it is utilized.
Returns:	Updated executable object.
Return type:	executable
Raises:	`KeyError` – If the executable config doesn’t contain an `uwsgi` configuration.

run()[source]¶: This should only be used to help configure another executable.

setup()[source]¶

Setup required for executing a web app via uwsgi.

Raises:	`ValueError` – If there is a space in the name. This is not allowed. `ValueError` – If both `http-socket` and `uwsgi-socket` are specified, which is invalid. `ValueError` – If neither `http-socket` and `uwsgi-socket` are specified. One must be specified. `KeyError` – If `module` is not specific in the configuration, such that the `uwsgi` config cannot not be defined.

class overwatch.base.deploy.zmqReceiver(config)[source]¶

Bases: overwatch.base.deploy.executable

Start the ZMQ receiver.

Note

Arguments after config are values which will be used for formatting and are required in the config.

Parameters:

config (dict) – Configuration for the executable.
enabled (bool) – True if the task should actually be executed.
runInBackground (bool) – True if the process should be run in the background. This means that process will not be blocking, but it shouldn’t be used in conjunction with supervisor. Default: False.
forceRestart (bool) – True if the process should kill any previous processes before starting. If not and the process already exists, then nothing will be done.
receiver (str) – Three letter subsystem name. For example, EMC.
localPort (int) – Port where the HLT data should be made available on the local system.
dataPath (str) – Path to where the data should be stored.
select (str) – Selection string for the receiver.
additionalOptions (list) – Additional options to be passed to the receiver.
tunnel (dict) – Configuration for autossh. See the autossh executable for a comprehensive set of options.

tunnel¶

Keep track of the autossh executable. Default: None.

Type:	executable

setup()[source]¶

Setup required for the ZMQ receiver.

In particular, we add any additionally requested receiver options from the configuration, as well as setting up the SSH tunnel.

class overwatch.base.deploy.zodb(config)[source]¶

Bases: overwatch.base.deploy.executable

Start the ZODB database.

Note

Arguments after config are values which will be used for formatting and are required in the config.

Parameters:

config (dict) – Configuration for the executable.
enabled (bool) – True if the task should actually be executed.
runInBackground (bool) – True if the process should be run in the background. This means that process will not be blocking, but it shouldn’t be used in conjunction with supervisor. Default: False.
forceRestart (bool) – True if the process should kill any previous processes before starting. If not and the process already exists, then nothing will be done.
address (str) – IP address for the database.
port (int) – Port for the database.
databasePath (str) – Path to where the database file should be stored.

setup()[source]¶: Setup required for the ZODB database.

overwatch.base.replay module ¶

overwatch.base.utilities module ¶

Shared utility functions used for organizing file structure, merging histograms, and transferring data.

overwatch.base.utilities.createFileDictionary(currentDir, runDir, subsystem)[source]¶

Creates dictionary of files and their unix timestamps for a given run directory.

This function effectively characterizes the files available for a subsystem in a given run, providing all files, as well as the length of the run.

Note

The filenames that are returned are of the form Run123456/SYS/file.root.

Parameters:	currentDir (str) – Path to the directory containing run directories. runDir (str) – Run directory to be considered. subsystem (str) – Subsystem to be considered.
Returns:	[Dictionary from time stamp to filename, time in minutes spanned by the run]
Return type:	list

overwatch.base.utilities.enumerateFiles(dirPrefix, subsystem)[source]¶

Determine the ROOT files which have been received from the HLT and need to be moved into the Overwatch run file structure for processing.

Parameters:	dirPrefix (str) – Path to the root directory where the data is stored. subsystem (str) – Subsystem to be considered.
Returns:	Files in provided directory that need to be moved.
Return type:	list

overwatch.base.utilities.extractTimeStampFromFilename(filename)[source]¶

Extracts unix time stamp from a given filename.

This works for combined, time slice, and files received from the HLT filenames, although the meaning varies depending on the type of filename passed. The type of the filename will be automatically detected based on substrings of the filename. This will determine the format of the timestamp to be extracted.

For possibles filenames are as follows:

For combined files, the format is prefix/combined.unixTimeStamp.root, so we can just extract it directly.
For time slices, the format is prefix/timeSlices.unixStartTime.unixEndTime.root, so we extract the two times, subtract them, and return the difference. Note that this makes it a different format than the other two timestamps.
For unprocessed files, the format is prefix/SYShistos_runNumber_hltMode_YYYY_MM_DD_HH_mm_ss.root`, so we can extract it by selecting ".root" files which do not have "combined" or "timeSlice" in their name, and also only contain one ``.. The time stamp is assumed to be in the CERN time zone, so we convert it to UTC and return the timestamp.
For other files processed into subsystems, the format is prefix/SYShists.YYYY_MM_DD_HH_mm_ss.root. We extract the time stamp and convert it to unix time. The time stamp is assumed to be in the CERN time zone, so we convert it to UTC and return the timestamp.

Note

The prefix/ can be anything (or non-existent), as long as it doesn’t contain any ..

Parameters:	filename (str) – Filename which contains the desired timestamp. The precise format of the timestamp depends on the type filename passed into the function.
Returns:	Timestamp extracted from the filename in number of seconds (unix time, except for time slices, where it is the length of the time stamp).
Return type:	int

overwatch.base.utilities.findCurrentRunDirs(dirPrefix='')[source]¶

Finds all of the dirs in the specified directory dir with “Run” in the name.

Parameters:	dirPrefix (str) – Path to where all of the run directories are stored. Default: current working directory.
Returns:	List of runs.
Return type:	list

overwatch.base.utilities.getDB(databaseLocation)[source]¶

Setup and retrieve the database available at the given location.

Parameters:	databaseLocation (str) – Path to the database. Must be a valid zodburi URI, which could be a local file, a socket, a network path, or another type.
Returns:	(ZODB db root PersistentMapping, ZODB.Connection.Connection object). The connection object should be closed work with the database is completed.
Return type:	tuple

class overwatch.base.utilities.handleSignals[source]¶

Bases: object

Helper class to gracefully handle a kill signal.

We handle SIGINT (for example, sent by ctrl-c) and SIGTERM (for example, sent by docker).

This class is adapted from the solution described here, and improved with the information here.

In the run module, it is expected to have some kind of code similar to below:

handler = handleSignals()
while not handler.exit.is_set():
    # Do something
    handler.exit.wait(parameters["dataTransferTimeToSleep"])

Parameters:	None. –

exit¶

Event to manage when we’ve received a signal. exit.set() is called when a signal is received, and can be checked via is_set().

Type:	threading.Event

exit = <threading._Event object>

exitGracefully(signum, frame)[source]¶: Handle the signal by storing that it was sent, allowing the run function to exit.

overwatch.base.utilities.moveFiles(dirPrefix, subsystemDict)[source]¶

For each subsystem, moves ROOT files received from the HLT into appropriate file structure for processing.

In particular, files from the HLT have the general form of SYShists_runNumber_hltMode_%Y_%m_%d_%H_%M_%S.root. For a particular example, files start like this:

EMChistos_300005_B_2015_11_24_18_05_10.root
EMChistos_300005_B_2015_11_24_18_09_12.root
...

and are then moved to

Run300005/
    EMC/
        EMChists.2015_11_24_18_05_10.root
        EMChists.2015_11_24_18_09_12.root

The new filenames are then returned in a nested dictionary of lists which has the following structure (assuming the example specified above):

>>> runsDict["Run300005"]["EMC"]
["EMChists.2015_11_24_18_05_10.root", "EMChists.2015_11_24_18_09_12.root"]
>>> runsDict["Run300005"]["hltMode"]
"B"

The run and subsystem directories are created as needed. The filenames are determined by convention to make it possible to extract the timestamps later. The HLT mode is also stored, moving the information from the input filename to the output dict. Some care is required here - otherwise the HLT information will be lost (although it can always be reconstructed via the logbook).

Note

HLT mode “E” corresponds to be replayed data which can be disregarded. Here, it is moved to the ReplayData directory so it doesn’t get processed, but it also isn’t entirely lost.

Note

The filenames returned here are different than those from createFileDictionary(). Here, we return just the filename in the nested dict, while there we return the full path to the file (not including the dirPrefix).

Parameters:	dirPrefix (str) – Path to the root directory where the data is stored. subsystemDict (dict) – Dictionary of subsystems (keys) and lists of files that need to be moved (values) for each subsystem.
Returns:	Nested dict which contains the new filenames and the HLT mode. For the precise structure, see above.
Return type:	dict

overwatch.base.utilities.moveRootFiles(dirPrefix, subsystemList)[source]¶

Simple driver function to move files received from the HLT into the appropriate directory structure for processing.

Parameters:	dirPrefix (str) – Path to the root directory where the data is stored. subsystemList (list) – List of subsystems to be considered.
Returns:	Nested dict which contains the new filenames and the HLT mode. For the precise structure, `moveFiles()`.
Return type:	dict

overwatch.base.utilities.removeOldestValueAndInsert(arr, value)[source]¶

Removes the oldest value from the start of a numpy array and appends a new value at the end.

Parameters:	arr (numpy.ndarray) – Array containing the values to modify. value (int or float) – Value to be appended.
Returns:	The modified array.
Return type:	numpy.ndarray

overwatch.base.utilities.retrieveHLTModeFromStoredRunInfo(runDirectory)[source]¶

Retrieve the HLT mode from a stored runInfo.yaml file.

The file is read from runDirectory/runInfo.yaml. Use the mode from the file if it exists, or otherwise set it as undefined = “U”.

Parameters:	runDirectory (str) – Path to the run directory where the `runInfo.yaml` file is stored.
Returns:	The HLT mode (if possible to extract), or “U” for unknown.
Return type:	str

overwatch.base.utilities.setupLogging(logger, logLevel, debug)[source]¶

General function to setup the proper logging outputs for an executable.

Creates loggers for logging to stdout, rotating file, and email (for warning or above logs). They are enabled depending on the configuration.

Parameters:

logger (logging.Logger) – Logger to be configured. This should be the logger of the executable.
logLevel (int) – Logging level. Select from any of the options defined in the logging module.
debug (bool) – Overall debug mode for the executable. True logs to the console while False logs to a rotating file handler and sets up the possibility of sending logs via email. Default: True

Returns:

None. The logger is fully configured.

overwatch.base.utilities.updateDBSensitiveParameters(db, overwriteSecretKey=True)[source]¶

Update sensitive parameters which are stored in the database. Those parameters include the users dictionary, as well as the secret key used for cookie signing.

These values are stored under the “config” key in the database and will be created if they do not yet exist.

Parameters:	db (PersistentMapping) – The root database persistent mapping. overwriteSecretKey (bool) – If true, the secret key in the database should be overwritten with a new value read from the configuration.
Returns:	None.

overwatch.base.utilities.writeRunInfoToFile(runDirectory, hltMode, forceWrite=False)[source]¶

Write run information to YAML file for storage.

The file is written to runDirectory/runInfo.yaml. Use the mode from the file if it exists, or otherwise set it as undefined = “U”.

Parameters:	runDirectory (str) – Path to the run directory where the `runInfo.yaml` file is stored. hltMode (str) – The HLT mode if set, or “U” for unknown. forceWrite (bool) – Write the information to the file regardless of whether the file already exists.
Returns:	None.

Overwatch Base Package (`overwatch.base`)¶

Package reference ¶

overwatch.base.config module ¶

overwatch.base.dataTransfer module ¶

overwatch.base.deploy module ¶

overwatch.base.replay module ¶

overwatch.base.utilities module ¶

Navigation

Related Topics

Overwatch Base Package (overwatch.base)¶

Package reference¶

overwatch.base.config module¶

overwatch.base.dataTransfer module¶

overwatch.base.deploy module¶

overwatch.base.replay module¶

overwatch.base.utilities module¶

Overwatch Base Package (`overwatch.base`)¶

Package reference ¶

overwatch.base.config module ¶

overwatch.base.dataTransfer module ¶

overwatch.base.deploy module ¶

overwatch.base.replay module ¶

overwatch.base.utilities module ¶