Overwatch Web App Package (overwatch.webApp)

General information from the README is available immediately below. Further module specific documentation is available further below in the package reference, with the modules listed below.

• Web App package details
  • Page template layout
  • AJAX and JSRoot options
  • Flask
    • Requests and URL routing
    • Error Format
    • flask-webassets, webassets filters, and debugging
  • Monitoring for errors

Package reference

overwatch.webApp.auth module

Contains auth functions.

For user authentication, https://exploreflask.com/users.html was extensively used as a guide.

class overwatch.webApp.auth.User(username, password)

Bases: flask_login.mixins.UserMixin

A basic user class to manage authentication.

Inherits from UserMixin, which implements a basic class for use with login_manger().

New users should be added into the external config file. This class provides no ability to store new users dynamically and assumes that passwords passed to it are already hashed by bcrypt.generate_password_hash(password, BCRYPT_LOG_ROUNDS).

The login_manager stores this class to manage users.

Note

There are also a few attributes inherited from UserMixin

Parameters:

• username (str) – Username of new user
• password (str) – Hashed password of new user. Password should be hashed with bcrypt.generate_password_hash(desiredPassword, BCRYPT_LOG_ROUNDS)

users

Contains all valid users with hashed passwords. Loaded from an extra config file.

Type:dict

id

The username of the instance of the object

Type:str

password

The password of the instance of the object. Note: This must be hashed by the user before passing to this object!

Type:str

checkPassword(plainTextPassword)

Check a plain text password against a hashed password.

Parameters:plainTextPassword (str) – The plain text password to test. Returns:True if the password matches the instance of the user. Return type:bool

static getUser(username, db)

Retrieve the username and password of a user.

Used by load_user() to maintain a logged in user session.

Parameters:username (str) – Username to retrieve Returns:

Returns an instance of the User class if the user exists. Otherwise, it

returns None.

Return type:User

overwatch.webApp.auth.authenticateUser(username, password, db)

Checks whether the user credentials are correct.

Parameters:

• username (str) – username of the attempted user.
• password (str) – plain text password of the attempted user.

Returns:

If the credentials were valid, an instance of the User class is returned so that the login_manager

can store that object and track which user is logged in. Otherwise, it returns None.

Return type:

User

overwatch.webApp.routing module

Contains routing functions.

Contains functions to ensure safe routing and redirection of the next URL. These functions are from http://flask.pocoo.org/snippets/62/, and were written by the author of Flask.

Slight modifications were made to redirectBack() to ensure that a login-logout loop was avoided under particular circumstances.

overwatch.webApp.routing.getRedirectTarget()

Extracts the Next target and checks its safety.

Note

Relies on the flask.request object.

Parameters:None – Returns:URL if the target is safe. Return type:str

overwatch.webApp.routing.isSafeUrl(target)

Checks URL for safety to ensure that it does not redirect unexpectedly.

Note

Relies on the flask.request object.

Parameters:target (str) – URL for the target to test. Returns:True if the URL is safe. Return type:bool

overwatch.webApp.routing.redirectBack(endpoint, **values)

Handles safe redirection.

It extracts the value of Next from flask.request. If the target is not safe, then redirect back to endpoint instead.

Note

Relies on the request.form dict.

Parameters:

• endpoint (str) – Where to redirect in case the Next url is not safe
• **values (dict) – Arguments to pass to url_for() in case of needing to redirect to endpoint instead.

Returns:

Redirect is called on the next URL if it is safe. Redirects to the

given endpoint if the URL is not safe.

Return type:

redirect to NextUrl

overwatch.webApp.run module

overwatch.webApp.utilities module

Web app specific utilities.

In particular, it handles tasks related to deployment and minimization which are not relevant to other Overwatch packages.

class overwatch.webApp.utilities.PolymerBundler(**kwargs)

Bases: webassets.filter.ExternalTool

Filter to bundle Polymer html imports into a single file for deployment.

Best practices dictate that the Polymer html imports should be combined into a single file to reduce the number of individual http requests. Polymer provides a tool to do so, called polymer-bundler. By taking advantage of webassets, we can automatically combine and minimize these files when starting a web app deployment.

To successfully define the filter, the following details must be addressed:

• polymer-bundler must only be executed with relative paths, so we cannot use ExternalTool.subprocess, since that gives absolute paths.
• To ensure that the polymer components also work when not bundled, the filter must be executed in a directory above the static dir.

These issues causes quite some complications! See the input(...) function for how to deal with these issues.

When webassets is run in debug mode, this filter will not be run! Instead, the standard (un-minified) version will be included. For information on forcing this filter to be run, see the web app README.

input(_in, out, **kwargs)

Plugin function for adding an external filter to webassets.

As of August 2018, the kwargs options available include:

kwargs = {'output': 'gen/polymerBundle.html',
          'output_path': '/pathToOverwatch/overwatch/webApp/static/gen/polymerBundle.html',
          'source_path': '/pathToOverwatch/overwatch/webApp/static/polymerComponents.html',
          'source': 'polymerComponents.html'}

Note

polymer-bundler parses arguments a bit strangely - values such as paths still need to be in a separate argument. Thus, the arguments looks more split than would usually be expected.

Parameters:

• _in (StringIO) – Input for the filter. Not used here.
• out (StringIO) – Output for the filter. The output for polymer-bundler is written here. This will eventually be written out to a file.
• **kwargs (dict) – Additional options required to run the filter properly. See the function description for the available options.

Returns:

None

method = False

name = 'PolymerBundler'

open = None

output = None

overwatch.webApp.validation module

Contains validation functions.

These functions are important to ensure that only valid values are passed to the processing functions. Validation could likely be improved by moving WTForms, which Overwatch already depends upon for CSRF protection.

overwatch.webApp.validation.convertRequestToPositiveInteger(paramName, source)

Converts a requested parameter into a positive integer.

This function is somewhat similar to the other conversion and validation functions, although it is a bit simpler.

Parameters:

• paramName (str) – Name of the parameter in which we are interested in.
• source (dict) – Source of the information. Usually request.args or request.form.

Returns:

The requested int or 0 if it was somehow invalid.

Return type:

int

overwatch.webApp.validation.convertRequestToPythonBool(paramName, source)

Converts a requested parameter to a python bool.

The validation is particularly useful for jsRoot and ajaxRequest. Note that this function is fairly similar to convertRequestToStringWhichMayBeEmpty.

Parameters:

• paramName (str) – Name of the parameter in which we are interested in.
• source (dict) – Source of the information. Usually request.args or request.form.

Returns:

True if the retrieved value was True.

Return type:

bool

overwatch.webApp.validation.convertRequestToStringWhichMayBeEmpty(paramName, source)

Handle strings which may be empty or contain “None”.

This validation is particularly useful for validating hist names and hist groups request strings to ensure that they are valid strings before doing further validation. Empty strings should be treated as None. The None strings are from the timeSlicesValues div on the runPage. Note that this function is fairly similar to convertRequestToPythonBool.

Parameters:

• paramName (str) – Name of the parameter in which we are interested in.
• source (dict) – Source of the information. Usually request.args or request.form.

Returns:

Validated string or None if the string is empty or “None”.

Return type:

str or None

overwatch.webApp.validation.extractValueFromNextOrRequest(paramName, source)

Extract the selected parameter from the next parameter or directly from the request.

First attempt to extract the named parameter from the next parameter in the args of the request. If it isn’t available, then attempt to extract it directly from the request args parameters. This is particularly useful for logging the user back in the case of a default username.

Parameters:

• paramName (str) – Name of the parameter to extract.
• source (dict) – Source of the information. Usually request.args or request.form.

Returns:

Value of the extracted parameter.

Return type:

str

overwatch.webApp.validation.retrieveAndValidateTimeSlice(subsystem, error)

Retrieves the time slice key and then returns the corresponding time slice (it is exists).

This function safely retrieves a timeSliceContainer. In the case of a valid time slice key, the corresponding object will be retrieved. However, in the case of “fullProcessing”, the object will be None so we can immediately return the full object. Errors will be appended under the timeSliceKey key.

Note

For the error format in error, see the web app README.

Parameters:

• subsystem (subsystemContainer) – Subsystem for which the time slices request was made.
• error (dict) – Contains any possible errors following the defined error format. We will append any new errors to it.

Returns:

(timeSliceKey, timeSlice) where timeSliceKey (str) is the key under which the time slice

is stored or “fullProcessing” (which indicates full processing), and timeSlice (timeSliceContainer) is the corresponding time slice retrieved from the subsystem, or None if for any reason it could not be retrieved.

Return type:

tuple

overwatch.webApp.validation.validateHistGroupAndHistName(histGroup, histName, subsystem, run, error)

Check that the given hist group or hist name exists in the subsystem.

Look for the requested hist group or hist name within a given subsystem. It requires that the hist group and hist name have already been validated to ensure that they are valid strings or None. Note that it could be perfectly valid for both to be None!

Note

As of Sept 2016, this check is not performed on the run page because it seems unnecessary to check every single value and there could be a substantial performance cost. This should be revisited in the future if it becomes a problem.

Note

For the error format in error, see the web app README.

Parameters:

• histGroup (str or None) – Requested hist group.
• histName (str or None) – Requested hist name.
• subsystem (subsystemContainer) – Subsystem which should contain the hist group and hist name.
• run (runContainer) – Run for which the hist group and hist name should exist.
• error (dict) – Contains any possible errors following the defined error format. We will append any new errors to it.

Returns:

It will append an error to the error dict if there is a problem with the given hist

group or hist nine. The error dict should be checked by the returning function to determine the result and decide how to proceed.

Return type:

None

overwatch.webApp.validation.validateLoginPostRequest(request)

Validates the login POST request.

Note

The error format is different here. Instead of a list in a dict, we simply have a string.

Parameters:request (Flask.request) – The request object from Flask.

Return

tuple: (errorValue, username, password), where errorValue (str) contains the error that

may have occurred, username (str) is the username extracted from POST request, and password (str) is the password extracted from POST request.

overwatch.webApp.validation.validateRunPage(runDir, subsystemName, requestedFileType, runs)

Validates requests to the various run page types (handling individual run pages and root files).

The return tuple contains the validated values. The error value should always be checked first before using the other return values (they will be safe, but may not be meaningful).

Note

For the error format in error, see the web app README.

Note

The listed args (after the first four) are provided through the flask request.args dictionary.

Parameters:

• runDir (str) – String containing the run number. For an example run 123456, it should be formatted as Run123456
• subsystemName (str) – The current subsystem in the form of a three letter, all capital name (ex. EMC).
• requestedFileType (str) – Either “runPage”, which corresponds to a standard run page or “rootFiles”, which corresponds to the page displaying the available root files.
• runs (BTree) – Dict-like object which stores all run, subsystem, and hist information. Keys are the in the runDir format (“Run123456”), while the values are runContainer objects. This should be retrieved from the database.
• jsRoot (bool) – True if the response should use jsRoot instead of images.
• ajaxRequest (bool) – True if the response should be via AJAX.
• requestedHistGroup (str) – Name of the requested hist group. It is fine for it to be an empty string.
• requestedHist (str) – Name of the requested histogram. It is fine for it to be an empty string.

Returns:

(error, run, subsystem, requestedFileType, jsRoot, ajaxRequest, requestedHistGroup, requestedHist, timeSliceKey, timeSlice)

where error (dict) contains any possible errors, run (runContainer) corresponds to the current run, subsystem (subsystemContainer) corresponds to the current subsystem, requestedFileType (str) is the type of run page (“runPage” or “rootFiles”), jsRoot (bool) is True if the response should use jsRoot, ajaxRequest (bool) is true if the response should be as AJAX, requestedHistGroup (str) is the name of the requested hist group, requestedHist (str) is the name of the requested histogram, timeSliceKey (str) is the time slice key, and timeSlice (timeSliceContainer) is the time slice object. For more on the last two arguments, see retrieveAndValidateTimeSlice(...).

Return type:

tuple

overwatch.webApp.validation.validateTimeSlicePostRequest(request, runs)

Validates the time slice POST request.

The return tuple contains the validated values. The error value should always be checked first before using the other return values (they will be safe, but may not be meaningful).

Warning

If an error occurs in determining the run or subsystem, we cannot retrieve the rest of the information necessary to validate the request, so the rest of the values in the return tuple are set to None.

Note

For the error format in errorValue, see the web app README.

Note

The listed args (after the first two) are provided through the flask request.form dictionary.

Parameters:

• request (Flask.request) – The request object from Flask.
• runs (BTree) – Dict-like object which stores all run, subsystem, and hist information. Keys are the in the runDir format (“Run123456”), while the values are runContainer objects.
• minTime (float) – Minimum time for the time slice.
• maxTime (float) – Maximum time for the time slice.
• runDir (str) – String containing the run number. For an example run 123456, it should be formatted as Run123456.
• subsystemName (str) – The current subsystem in the form of a three letter, all capital name (ex. EMC).
• scaleHists (str) – True if the hists should be scaled by the number of events. Converted from string to bool.
• hotChannelThreshold (int) – Value of the hot channel threshold.
• histGroup (str) – Name of the requested hist group. It is fine for it to be an empty string.
• histName (str) – Name of the requested histogram. It is fine for it to be an empty string.

Returns:

(errorValue, minTime, maxTime, runDir, subsystemName, scrollAmount) where errorValue (dict)

containers any possible errors, minTime (float) is the minimum time for the time slice, maxTime (float) is the maximum time for the time slice, runDir (str) is the run dir formatted string for which the time slice should be performed, subsystemName (str) is the current subsystem in the form of a three letter, all capital name (ex. EMC), and scrollAmount (float) is the amount to scroll down the page to return to precisely where the user was previously.

Return type:

tuple

overwatch.webApp.validation.validateTrending(request)

Validate requests to the trending page.

The return tuple contains the validated values. The error value should always be checked first before using the other return values (they will be safe, but may not be meaningful).

Note

For the error format in error, see the web app README.

Note

Function args are provided through the flask request.args dictionary.

Parameters:

• request (Flask.request) – The request object from Flask.
• jsRoot (bool) – True if the response should use jsRoot instead of images.
• ajaxRequest (bool) – True if the response should be via AJAX.
• subsystemName (str) – Name of the requested subsystem. It is fine for it to be an empty string. Provided via the histGroup field since it is treated identically, allowing us to avoid the need to define another field for this one case.
• histName (str) – Name of the requested histogram. It is fine for it to be an empty string.

Returns:

(error, subsystemName, requestedHist, jsRoot, ajaxRequest), where where error (dict) contains

any possible errors, subsystemName (str) corresponds to the current subsystem, subsystemName (str) is the requested subsystem in the form of a three letter, all capital name (ex. EMC). jsRoot (bool) is True if the response should use jsRoot, ajaxRequest (bool) is true if the response should be as AJAX.

Return type:

tuple

overwatch.webApp.webApp module

Web App for serving Overwatch results, as well as access to user defined reprocessing and times slices.

This is the main web app executable, so it contains quite some functionality, especially that which is not so obvious how to refactor when using flask. Routing is divided up into authenticated and unauthenticated views.

overwatch.webApp.webApp.contact()

Simple contact page so we can provide general information and support to users.

Also exposes useful links for development (for test data), and system status information to administrators (which must authenticate as such).

Note

Function args are provided through the flask request object.

Parameters:ajaxRequest (bool) – True if the response should be via AJAX. Returns:Contact page populated via template. Return type:Response

overwatch.webApp.webApp.handleCSRFError(error)

Handle CSRF error.

Takes advantage of the property of the CSRFError class which will return a string description when called with str().

Note

The only requests that could fail due to a CSRF token issue are those made with AJAX, so it is reasonable to return an AJAX formatted response.

Note

For the error format in errors, see the web app README.

Parameters:error (CSRFError) – Error object raised during as CSRF validation failure. Returns:json encoded response containing the error. Return type:str

overwatch.webApp.webApp.index(*args, **kwargs)

This is run list, which is the main page for logged in users.

The run list shows all available runs, which links to available subsystem histograms, as well as the underlying root files. The current status of data taking, as extracted from when the last file was received, is displayed in the drawer, as well as some links down the page to allow runs to be moved through quickly. The main content is paginated in a fairly rudimentary manner (it should be sufficient for our purposes). We have selected to show 50 runs per page, which seems to be a reasonable balance between showing too much or too little information. This can be tuned further if necessary.

Note

Function args are provided through the flask request object.

Parameters:

• ajaxRequest (bool) – True if the response should be via AJAX.
• runOffset (int) – Number of runs to offset into the run list. Default: 0.

Returns:

The main index page populated via template.

Return type:

Response

overwatch.webApp.webApp.load_user(user)

Used to retrieve a remembered user so that they don’t need to login again each time they visit the site.

Parameters:user (str) – Username to retrieve. Returns:

The user stored in the database which corresponds to the given username, or

None if it doesn’t exist.

Return type:auth.User

overwatch.webApp.webApp.login()

Login function. This is is the first page the user sees.

Unauthenticated users are also redirected here if they try to access something restricted. After logging in, it should then forward them to resource they requested.

Note

Function args are provided through the flask request object.

Parameters:

• ajaxRequest (bool) – True if the response should be via AJAX.
• previousUsername (str) – The username that was previously used to login. Used to check when automatic login should be performed (if it’s enabled).

Returns:

Response based on the provided request. Possible responses included validating

and logging in the user, rejecting invalid user credentials, or redirecting unauthenticated users from a page which requires authentication (it will redirect back after login).

Return type:

response

overwatch.webApp.webApp.logout(*args, **kwargs)

Logs out an authenticated user.

Once completed, it will always redirect to back to login(). If the user then logs in, they will be redirected back to index. Some care is required to handle all of the edge cases - these are handled via careful redirection in login() and the routing module.

Warning

Careful in making changes to the routing related to function, as it is hard coded in `routing.redirectBack()!

Note

previousUsername is provided to the next request so we can do the right thing on automatic login. In that case, we want to provide automatic login, but also allow the opportunity to logout and then explicitly login with different credentials.

Parameters:None – Returns:Redirect back to the login page. Return type:Response

overwatch.webApp.webApp.overwatchStatus(*args, **kwargs)

Query and determine the status of some parts of Overwatch.

This function takes advantage of the status functionality of the web app to determine the state of any deployed web apps that are specified in the web app config. This is achieved by sending requests to all other sites and then aggregating the results. Each request is allowed a 0.5 second timeout.

It will also provide information on when the last files were received from other sites.

This functionality will only work if the web app is accessible from the site where this is run. This may not always be the case.

Note

Since the GET requests are blocking, it can appear that the web app is hanging. However, this is just due to the time that the requests take.

Warning

This can behave somewhat strangely using the flask development server, especially if there is reloading. If possible, it is best to run with uwsgi for testing of this function.

Note

Function args are provided through the flask request object.

Parameters:ajaxRequest (bool) – True if the response should be via AJAX. Returns:Status template populated with the status of Overwatch sites specified in the configuration. Return type:Response

overwatch.webApp.webApp.protected(*args, **kwargs)

Serves the underlying files.

This function is response for actually making files available. Ideally, these would be served via the production web server, but since access to the data requires authentication, we instead have to provide access via this function. To provide this function, we utilized the approach described here.

Note

This function ignores GET parameters. This is done intentionally to allow for avoiding problematic caching by a browser. To avoid this caching, simply pass an additional get parameter after the filename which varies when we need to avoid the cache. This is particularly useful for time slices, where the name could be the same, but the information has changed since last being served.

Parameters:filename (str) – Path to the file to be served. Returns:File with the proper headers. Return type:Response

overwatch.webApp.webApp.runPage(*args, **kwargs)

Serves the run pages and root files for a request run.

This is really the main function for serving information in Overwatch. The run page provides subsystem specific histograms and information to the user. Time slices and user directed reprocessing is also made available through this page. If a subsystem has made a customized run page, this will automatically be served. If they haven’t, then a default page will be provided.

This function serves both run pages, which display histograms, as well as root files pages, which provide direct access to the underlying root files. Since they require similar information, it is convenient to provide access to both of them from one function.

Note

Some function args (after the first 3) are provided through the flask request object.

Warning

Careful if changing the routing for this function, as the display swtich for the time slices button in the web app depends on “runPage” being in this route. If this is changed, then the js also needs to be changed.

Parameters:

• runNumber (int) – Run number of interest.
• subsystemName (str) – Name of the subsystem of interest.
• requestedFileType (str) – Type of file in which we are interested. Can be either runPage (corresponding to a run page) or rootFiles (corresponding to access to the underlying root files).
• jsRoot (bool) – True if the response should use jsRoot instead of images.
• ajaxRequest (bool) – True if the response should be via AJAX.
• requestedHistGroup (str) – Name of the requested hist group. It is fine for it to be an empty string.
• requestedHist (str) – Name of the requested histogram. It is fine for it to be an empty string.

Returns:

A run page or root files page populated via template.

Return type:

Response

overwatch.webApp.webApp.statusQuery()

Returns the status of the Overwatch server instance.

This can be accessed by a GET request. If the request is successful, it will response with “Alive” and the response code 200. If it didn’t work properly, then the response won’t come through properly, indicating that an action must be taken to restart the web app.

Note

It doesn’t require authentication to simply the process of querying it. This should be fine because the information that the web app is up isn’t sensitive.

Parameters:None – Returns:

Contains a string, “Alive”, and a 200 response code to indicate that the web app is still up.

If the database is somehow not available, it will return “DB failed” and a 500 response code. A response timeout indicates that the web app is somehow down.

Return type:Response

overwatch.webApp.webApp.testingDataArchive(*args, **kwargs)

Provides a zip archive of test data for Overwatch development.

This function will look through at most the 5 most recent runs, looking for the minimum number of files necessary for running Overwatch successfully. These files will be zipped up and provided to the user. The minimum files are the combined file, and the most recent file received for the subsystem (they are usually the same, but it is easier to include both). If possible, an additional file is included for testing the time slice and trending functionality. It may not always be available if runs are extremely short. The zip archive will include all subsystems which are available. It will skip runs where any subsystem is unavailable to ensure that the data provided is of more utility.

Warning

Careful in changing the routing for this function, as the name of it is hard coded in webApp.routing.redirectBack(). This hard coding is to avoid a loop where the user is stuck accessing this file after logging in.

Parameters:None – Returns:Redirect to the newly created file. Return type:redirect

overwatch.webApp.webApp.timeSlice(*args, **kwargs)

Handles time slice and user reprocessing requests.

This is the main function for serving user requests. This function calls out directly to the processing module to perform the actual time slice or reprocessing request. It provides access to this functionality through the interface built into the header of the run page. In the case of a POST request, it handles, validates, and processes the timing request, rendering the result template and returning the user to the same spot as in the previous page. A GET request is invalid and will return an error (but the route itself is allowed to check that it is handled correctly).

This request should always be submitted via AJAX.

Note

Function args are provided through the flask request object.

Parameters:

• jsRoot (bool) – True if the response should use jsRoot instead of images.
• minTime (float) – Minimum time for the time slice.
• maxTime (float) – Maximum time for the time slice.
• runDir (str) – String containing the run number. For an example run 123456, it should be formatted as Run123456.
• subsystemName (str) – The current subsystem in the form of a three letter, all capital name (ex. EMC).
• scaleHists (str) – True if the hists should be scaled by the number of events. Converted from string to bool.
• hotChannelThreshold (int) – Value of the hot channel threshold.
• histGroup (str) – Name of the requested hist group. It is fine for it to be an empty string.
• histName (str) – Name of the requested histogram. It is fine for it to be an empty string.

Returns:

A run page template populated with information from a newly processed time slice (via a redirect

to runPage()). In case of error(s), returns the error message(s).

Return type:

Response

overwatch.webApp.webApp.upgradeDocker(*args, **kwargs)

Kill supervisord in the docker image so the docker image will stop.

On the ALICE offline infrastructure, a stopped docker image will cause the image to be upgraded and restarted. Thus, we can intentionally stop the image indirectly by killing the supervisor process to force an upgrade.

Note

This operation requires administrative rights (as the emcalAdmin user).

Warning

This is untested and is unsupported in any other infrastructure! It is experimental as of August 2018.

Note

Function args are provided through the flask request object.

Parameters:ajaxRequest (bool) – True if the response should be via AJAX. Returns:

Likely a timeout if the function was successful, or a response with an error message

if the function was unsuccessful.

Return type:Response or None