4.1   Architecture and Standard Components¶

In this design, all information of interest to metrics will be stored in a task’s metadata. The persisted metadata will be extracted by a dedicated “afterburner” task (named, for example, ComputeMetricsTask) that will find the appropriate keys and create Measurement objects. High-level handling of the measurements can be done by a single Job object. A non-Task prototype of this approach is used in the lsst.ap.verify.measurements package to handle running times, but in the interest of portability to other pipelines the final code should be in a dependency of ap_verify rather than a subpackage.

To minimize coupling with the task itself, the code that performs the measurements can be placed in decorators analogous to lsst.pipe.base.timeMethod. This approach also avoids code duplication for metrics that apply to more than one task class. However, as the number of metrics grows, so will the number of decorators attached to a class’s run method. Related metrics can be grouped in one decorator; for example, timeMethod measures not only timing, but also memory usage and other forms of profiling.

While tasks or their decorators are necessarily coupled to verify_metrics, ComputeMetricsTask need not know about most defined metrics if the metadata keys follow a particular format that allows discovery of measurements by iterating over the metadata (e.g., "<task-prefix>.verify.measurements.foo" for a metric named "package.foo"). Since the correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches), a standardized key (perhaps "<task-prefix>.verify.combiners.foo") can be used to specify the algorithm to combine the data. The use of a string to indicate the combiner only scales well if the majority of metrics share a small number of combiners, such as sum or average.

Figure 1 Illustration of how measurement data are passed up from tasks in the metadata-based architecture. anInstance and anotherInstance are ConcreteCmdLineTask objects run on different data.

Standardized metadata keys cannot handle metrics that depend on the results of multiple tasks (such as the DIASource fraction). In this case, information can still be passed up through metadata, but tasks should avoid using the verify.measurement prefix so that generic Measurement-making code does not mistakenly process them. Instead, each cross-task metric will need its own function in ComputeMetricsTask to search across all task classes for the relevant information and make a Measurement. Handling of cross-task metrics must therefore be coordinated across at least three packages – verify_metrics, the task package(s), and ComputeMetricsTask.

Standardized metadata keys can be used to record supplementary information about a measurement, for example by using verify.extras and verify.notes PropertySets.

4.2   Requirements for Task Creators and Maintainers¶

The main requirement imposed on authors of new tasks is the use of measurement decorators. It may be necessary to ensure decorators are applied in a particular order (for example, timeMethod should not include measurement overhead, so it should be listed last). If the decorators make assumptions about a task’s fields, they may constrain the implementation of the task itself. Implementation constraints go away if measurement metadata are written directly by a task’s methods, but then the task author is responsible for following all the conventions described above, including specifying a combiner and any other auxiliary metadata keys.

Custom task runners that call run multiple times per Task object must store the object’s metadata after each run, to keep it from getting lost. (This is not a problem for TaskRunner, which creates a new Task for each run.)

If all verification-related work is done by decorators, than maintaining instrumented tasks is easy; Task code can be changed and decorators added or removed as desired. The only risk is if decorators constrain task implementations in some way; such details must be clearly marked as unchangeable. If decorators depend on particular metadata keys being available, the lines that write those keys must be kept in sync with the key names passed to decorators. If tasks write measurement metadata directly, then maintainers must know not to touch those lines in any way.

Authors of new metrics must implement a decorator that measures them, most likely in pipe_base or a specific task’s package, and add it to all relevant task classes. The decorator must conform to all conventions regarding metadata keys. If the metric requires a new way to combine units of work, the new combiner must be implemented and registered under a unique name in ComputeMetricsTask.

4.3   Advantages and Disadvantages¶

A metadata-driven architecture limits changes to the task framework to imposing a convention for metadata keys; tasks need not depend on verify at all. However, it does require a centralized ComputeMetricsTask that frameworks like ap_verify or validate_drp must call after all other tasks have been run.

Adding most metrics requires changes to two packages (the minimum allowed by the verify framework), but cross-task metrics require three. Metrics cannot be added to or removed from a task without modifying code. Configs could be used to disable them, although keeping task- and metrics-related options separated would require a new config base class or a similarly far-reaching change to current configs.

Dividing a dataset into multiple units of work is poorly supported by a metadata-based architecture, because each metric may require a different way to synthesize a full-dataset measurement from the individual measurements, yet metadata does not allow code to be attached to measurements. On the other hand, it is very easy to support tracking of subtask measurements by both class and role, because the metadata naturally provide by-role information.

The biggest weakness of this architecture may well be its dependence on convention: metadata keys that don’t conform to the expected format must, in many cases, be silently ignored.

4.4   Example Metric Implementations¶

Note: in practice, all the metadata keys seen by ComputeMetricsTask would be prefixed by the chain of subtasks that produced them, requiring more complex handling than a lookup by a fixed name. This extra complexity is ignored in the examples, but is fairly easy to implement.

obj.metadata.add(name = "verify.measurements.%s_RunTime" % className,
                 value = deltaT)
obj.metadata.add(name = "verify.combiners.%s_RunTime" % className,
                 value = "sum")

def numAstroSources(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        # Any substitute for AstrometryTask must share its return value spec
        nSources = len(result.matches)
        self.metadata.add(name = "verify.measurements.NumAstroSources",
                         value = nSources)
        self.metadata.add(name = "verify.combiners.NumAstroSources",
                         value = "sum")
        return result
    return wrapper

class AstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, sourceCat, exposure):
        ...

class BetterAstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, sourceCat, exposure):
        ...

def numScienceSources(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        nSources = len(result.sourceCat)
        self.metadata.add(name = "verify.fragments.NumScienceSources",
                         value = nSources)
        return result
    return wrapper

class CalibrateTask(RefMatchTask):
    ...
    @numScienceSources
    @pipeBase.timeMethod
    def run(self, dataRef, exposure=None, background=None, icSourceCat=None,
        doUnpersist=True):
        ...

def numDiaSources(func):
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        nSources = len(result.sources)
        self.metadata.add(name = "verify.fragments.NumDiaSources",
                         value = nSources)
        return result
    return wrapper

class ImageDifferenceTask(RefMatchTask):
    ...
    @numDiaSources
    @pipeBase.timeMethod
    def run(self, sensorRef, templateIdList=None):
        ...

def measureDiaSourceFraction(self, allVerifyMetadata):
    SCIENCE_KEY = "fragments.NumScienceSources"
    DIA_KEY = "fragments.NumDiaSources"
    scienceSources = 0
    diaSources = 0
    for oneRunMetadata in allVerifyMetadata:
        if oneRunMetadata.exists(SCIENCE_KEY):
            scienceSources += oneRunMetadata.get(SCIENCE_KEY)
        if oneRunMetadata.exists(DIA_KEY):
            diaSources += oneRunMetadata.get(DIA_KEY)

    # Generic Measurements are not created if code not run, be consistent
    if scienceSources > 0:
        return lsst.verify.Measurement(
            "Fraction_DiaSource_ScienceSource",
            (diaSources / scienceSources) * u.dimensionless_unscaled))
    else:
        return None

def makeSpecializedMeasurements(self, job, allVerifyMetadata):
    ...
    measurement = self.measureDiaSourceFraction(allVerifyMetadata)
    if measurement is not None:
        job.measurements.insert(measurement)
    ...

5.1   Architecture and Standard Components¶

In this design, Measurement objects will be made by tasks. Tasks will be passed a Job object for collecting their Measurements, which can then be persisted by a top-level task. High-level handling of all Measurements would be handled by a Job living in an afterburner task (called, for example, ComputeMetricsTask), which consolidates the task-specific Job objects.

To minimize coupling with the task itself, the code that creates the Measurements can be placed in decorators similar to lsst.pipe.base.timeMethod, except that the decorators would update the job rather than Task.metadata. This approach also avoids code duplication for metrics that apply to more than one task class. However, as the number of metrics grows, so will the number of decorators attached to a class’s run method. Related metrics can be grouped in one decorator; for example, timeMethod measures not only timing, but also memory usage and other forms of profiling.

Measurements may depend on information that is internal to run or a task’s other methods. If this is the case, the Measurement may be created by an ordinary function called from within run, instead of by a decorator, or the internal information may be stored in metadata and then extracted by the decorator.

Directly constructed Measurements cannot handle metrics that depend on the results of multiple tasks (such as the DIASource fraction); such metrics must be measured in ComputeMetricsTask itself. There are two ways to get information on cross-task measurements to ComputeMetricsTask:

1. The necessary information can be stored in metadata.
2. We can impose a requirement that all cross-task metrics be expressible in terms of single-task metrics. In the DIASource fraction example such a requirement is a small burden, since both “Number of detected sources” and “Number of DIASources” are interesting metrics in their own right, but this may not be the case in general.

The correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches). This information can be provided by requiring that Measurement objects include a merging function, which can be invoked by ComputeMetricsTask.

Figure 2 Illustration of how measurements are handled in the direct-measurement and observer-based architectures. anInstance and anotherInstance are ConcreteCmdLineTask objects run on different data. The subtask of anotherInstance and the Measurement it produces are omitted for clarity.

5.2   Requirements for Task Creators and Maintainers¶

The main requirement imposed on authors of new tasks is the use of measurement decorators or functions. It may be necessary to ensure measurements are made in a particular order (for example, timing should not include measurement overhead). If measurement decorators make assumptions about a task’s fields, they may constrain the implementation of the task itself. Functions called from within run do not impose implementation constraints, but may be less visible to maintainers if they are buried in the rest of the task code.

If all verification-related work is done by decorators, than maintaining instrumented tasks is easy; task code can be changed and decorators added or removed as desired. The only major risk is if decorators constrain task implementations in some way; such details must be clearly marked as unchangeable. If measurements are made by functions called from within run, then the maintainability of the task depends on how well organized the code is – if measurement-related calls are segregated into their own block, maintainers can easily work around them.

Authors of new metrics must implement a decorator or function that measures them, most likely in pipe_base or a specific task’s package, and add it to all relevant task classes. The decorator or function must ensure the resulting Measurement has a combining functor. Standard combiners may be made available through a support package to reduce code duplication.

5.3   Advantages and Disadvantages¶

A direct-measurement architecture minimizes changes needed to the verify framework, which already assumes each task has an associated Job.

Adding most metrics requires changes to two packages (the minimum allowed by the verify framework), but cross-task metrics require three. Metrics cannot be added to or removed from a task without modifying code. Configs could be used to disable them, although keeping task- and metrics-related options separated would require a new config base class or a similarly far-reaching change to current configs.

Because of its decentralization, a direct-measurement architecture has trouble supporting cross-task metrics; in effect, one needs one framework for single-task metrics and a dedicated “afterburner” for cross-task metrics. This duality makes the system both harder to maintain and harder to develop new metrics for.

5.4   Example Metric Implementations¶

def timeMeasurement(func):
    @wraps(func)
    def wrapper(self, job, *args, **kwargs):
        try:
            return func(self, job, *args, **kwargs)
        finally:
            try:
                start = self.metadata.get("runStartCpuTime")
                end = self.metadata.get("runEndCpuTime")
            except pexExceptions.NotFoundError as e:
                raise AttributeError(
                    "@timeMethod must be listed after @timeMeasurement"
                ) from e
            metricName = "%s_RunTime" % type(self).__name__
            measurement = lsst.verify.Measurement(metricName,
                                                  (end - start) * u.seconds))
            measurement.combiner = verify.measSum
            job.measurements.insert(measurement)
    return wrapper

class AFancyTask(Task):
    ...
    @timeMeasurement
    @pipeBase.timeMethod
    def run(self, job, data):
        ...

def numAstroSources(func):
    @wraps(func)
    def wrapper(self, job, *args, **kwargs):
        result = func(self, job, *args, **kwargs)
        # Any substitute for AstrometryTask must share its return value spec
        nSources = len(result.matches)
        measurement = lsst.verify.Measurement(
            "NumAstroSources",
            nSources * u.dimensionless_unscaled))
        measurement.combiner = verify.measSum
        job.measurements.insert(measurement)
        return result
    return wrapper

class AstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, job, sourceCat, exposure):
        ...

class BetterAstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, job, sourceCat, exposure):
        ...

def numScienceSources(func):
    @wraps(func)
    def wrapper(self, job, *args, **kwargs):
        result = func(self, job, *args, **kwargs)
        nSources = len(result.sourceCat)
        measurement = lsst.verify.Measurement(
            "NumScienceSources",
            nSources * u.dimensionless_unscaled))
        measurement.combiner = verify.measSum
        job.measurements.insert(measurement)
        return result
    return wrapper

class CalibrateTask(RefMatchTask):
    ...
    @numScienceSources
    @pipeBase.timeMethod
    def run(self, job, dataRef, exposure=None, background=None, icSourceCat=None,
        doUnpersist=True):
        ...

def numDiaSources(func):
    @wraps(func)
    def wrapper(self, job, *args, **kwargs):
        result = func(self, job, *args, **kwargs)
        nSources = len(result.sources)
        measurement = lsst.verify.Measurement(
            "NumDiaSources",
            nSources * u.dimensionless_unscaled))
        measurement.combiner = verify.measSum
        job.measurements.insert(measurement)
        return result
    return wrapper

class ImageDifferenceTask(RefMatchTask):
    ...
    @numDiaSources
    @pipeBase.timeMethod
    def run(self, job, sensorRef, templateIdList=None):
        ...

def measureFraction(job, metric, numeratorName, denominatorName):
    try:
        numerator = job.measurements[numeratorName]
        denominator = job.measurements[denominatorName]
    except KeyError:
        # Measurements not made, fraction not applicable
        return

    fraction = numerator.quantity / denominator.quantity
    measurement = lsst.verify.Measurement(metric, fraction)
    # TODO: how to handle extras and notes?
    job.measurements.insert(measurement)

def makeSupplementaryMeasurements(masterJob):
    ...
    measureFraction(masterJob,
                    "Fraction_DiaSource_ScienceSource",
                    "NumDiaSources",
                    "NumScienceSources")
    ...

6.1   Architecture and Standard Components¶

In this design, Measurement objects will be made by an afterburner task (called, for example, ComputeMetricsTask) based on data produced by the pipeline. The measurements can be handled by a single Job living in ComputeMetricsTask.

To improve maintainability, the code that creates the Measurements can be segregated into multiple afterburner tasks. However, multiple tasks add considerable implementation overhead (custom task runners) and can make pipeline drivers more complicated. Since it is not clear along which lines, if any, it would be best to do the segregation, this note assumes a single ComputeMetricsTask containing the implementations of (almost) all metrics.

Measurements may depend on information that is not present in the processed data. If this is the case, tasks can be passed a Job object for collecting measurements (assumed to be created as in the direct-measurement architecture), or the information can be placed in the task metadata. In either approach, the data would be persisted by a top-level task, then handled by ComputeMetricsTask as part of the output data.

Supplementary context about a measurement can be extracted from persisted metadata, but may require dedicated code associated with individual tasks.

Figure 3 Illustration of how measurements are handled in the dataset-based architecture. anInstance and anotherInstance are ConcreteCmdLineTask objects run on different data.

6.2   Requirements for Task Creators and Maintainers¶

Tasks have very few new requirements in this framework. Most of the measurements are extracted from a task’s natural output data, whose format needs to be specified for other tasks’ use anyway. However, metrics that cannot be inferred from the data will need code added to applicable tasks, imposing requirements similar to those for the direct measurement architecture.

Authors of new metrics must implement a function in ComputeMetricsTask‘s package that measures them (a method in ComputeMetricsTask itself would lead to a single massive class, which would be hard to maintain). The function must enumerate and load applicable data from the repository. Tools for frequently used subsets may be provided by ComputeMetricsTask to reduce code duplication, where those subsets are not supported directly by the butler.

If a new metric must be measured directly by the task, the author will need to write both task-specific code, and code associated with ComputeMetricsTask for combining multiple units of work. It may be possible to standardize the latter (as assumed for the direct measurement architecture), so that non-dataset metrics only need updates to the task package. However, this in turn will make it more difficult to find the code implementing a particular metric.

6.3   Advantages and Disadvantages¶

A dataset-based architecture minimizes changes to individual tasks’ code, since it primarily interacts with them through their data products. Adding dataset-based metrics requires changes to two packages (the minimum allowed by the verify framework), but other metrics require three.

Because it avoids interacting with Task objects, this design is the best at dealing with cross-task metrics, and is (almost) immune to the problem of multiple units of work. However, it has trouble supporting metrics dealing with particular algorithms; in effect, one needs one framework for data-driven metrics and a separate system for internal metrics. This duality makes the system both harder to maintain and harder to develop new metrics for.

Attaching contextual information to a measurement can be difficult in a dataset-based design, because that information is often internal to the task even when the measurement itself can be computed from the data. However, data provenance and the verification environment can be easily attached.

Dataset-based metrics can be enabled or disabled with configs. Internal metrics are harder to control, but I expect that these metrics will be relatively cheap compared to those requiring statistical image analysis.

6.4   Example Metric Implementations¶

The examples assume that measurements are computed for all dataIds in a particular run (for example, timing measures the total time a task spent on all CCDs, not on a chip-by-chip basis). A hypothetical getAll function is provided for Butler retrieval of all datasets matching a possibly incomplete Butler dataId.

def measureRunningTimes(job, butler, dataId, topLevelTasks):
    timingMeasurements = defaultdict(list)
    for task in topLevelTasks:
        metadataType = task()._getMetadataName()
        allMetadata = getAll(butler, metadataType, dataId)

        for metadata in allMetadata:
            for subtaskId in getStoredTasksWith(metadata, "runEndCpuTime"):
                try:
                    start = self.metadata.get(subtaskId + ".runStartCpuTime")
                    end = self.metadata.get(subtaskId + ".runEndCpuTime")
                except pexExceptions.NotFoundError as e:
                    raise InvalidMeasurementError("Task %s has runEndCpuTime but no "
                                                  "runStartCpuTime." % subtaskId) from e

                # Multiple subtaskIds (with different parent tasks) may
                #    map to same task/metric
                metricName = "%s_RunTime" % getTaskClass(butler, dataId, subtaskId).__name__
                timingMeasurements[metricName].append((end - start) * u.seconds)

    for metric, times in timingMeasurements.items():
            totalTime = sum(times, 0.0 * u.seconds)
            measurement = lsst.verify.Measurement(metric, totalTime)
            job.measurements.insert(measurement)

def measureAstroMatches(job, butler, dataId):
    matchCatalogs = getAll(butler, "srcMatch", dataId)
    nMatches = 0 * u.dimensionless_unscaled
    for catalog in matchCatalogs:
        nMatches += len(catalog)
    measurement = lsst.verify.Measurement("NumAstroSources", nMatches)
    job.measurements.insert(measurement)

def measureDiaSourceFraction(job, butler, dataId, allConfig):
    matchCatalogs = getAll(butler, "src", dataId)
    nMatches = 0.0 * u.dimensionless_unscaled
    for catalog in matchCatalogs:
        nMatches += len(catalog)

    catalogType = allConfig.imageDifference.coaddName + "Diff_diaSrc"
    diaCatalogs = getAll(butler, catalogType, dataId)
    nDiaSources = 0.0 * u.dimensionless_unscaled
    for catalog in diaCatalogs:
        nDiaSources += len(catalog)

    measurement = lsst.verify.Measurement("Fraction_DiaSource_ScienceSource",
                                          nMatches / nDiaSources)
    job.measurements.insert(measurement)

7.1   Architecture and Standard Components¶

In this design, Measurement objects will be made by factory objects separate from the task itself. Tasks will be passed a Job object for collecting their Measurements, which can then be persisted by a top-level task. High-level handling of all Measurements would be handled by a Job living in an afterburner task (called, for example ComputeMetricsTask), which consolidates the task-specific Job objects.

The factories for the appropriate metrics will be registered with a task at construction time, using a new method (called Task.addListener, to allow for future applications other than metrics). The registration can be made configurable, although if each metric has its own factory, the config file will be an extra place that must be kept in sync with metrics definitions in verify_metrics. If one class measures multiple related metrics, then config changes are needed less often.

A task has a method (Task.notify) that triggers its registered factories on one of several standardized events (the examples assume there are three: Begin, Abort, and Finish); the events applicable to a given factory are specified at registration. Factories query the task for information they need, make the appropriate Measurement object(s), and pass them to the current run’s Job.

Measurements may depend on information that is internal to run or a task’s other methods. If this is the case, internal information may be stored in metadata and then extracted by the factory.

If metrics depend on the results of multiple tasks (such as the DIASource fraction), they can be worked around using the same techniques as for direct measurements. It is also possible to handle cross-task metrics by registering the same factory object with two tasks. However, supporting such a capability would require that factories be created and attached to tasks from above, which would take away this framework’s chief advantage – that it does not require centralized coordination, but is instead largely self-operating. See the visitor pattern for a design that does handle cross-task metrics this way.

Figure 4 Illustration of how measurements are created in the observer-based architecture, assuming all measurement information is available through metadata. Handling of measurements once they have been created works the same as for the direct measurement architecture.

The correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches). This information can be provided by requiring that Measurement objects include a merging function.

7.2   Requirements for Task Creators and Maintainers¶

Authors of new tasks must include in the task configuration information indicating which factories are to be attached to a task. The convention for defaults may be to register either all applicable factories, or a subset that is deemed to have little runtime overhead. The registration process itself can be handled by Task.__init__ with no direct developer intervention.

In general, maintaining instrumented tasks is easy. The only risk is if factories constrain task implementations in some way; such details must be clearly marked as unchangeable. If factories depend on particular metadata keys being available, the lines that write those keys must be kept in sync with the key names assumed by factories.

Authors of new metrics must implement a factory that measures them, most likely in pipe_base or a specific task’s package, and add it to all relevant configs. The factory must ensure the resulting Measurement has a combining functor, as for direct construction of Measurements.

7.3   Advantages and Disadvantages¶

An observer-based architecture provides maximum decentralization of responsibility: not only is each task responsible for handling its own measurements, but little to no task code needs to be aware of the specific metrics defined for each task. While the observer architecture is not the only one that allows run-time configuration of metrics, it is the one where such configuration fits most naturally by far. However, the high decentralization also gives it the worst support for cross-task metrics.

Adding single-task metrics requires changes to two packages, the minimum allowed by the verify framework. Metrics can be enabled and disabled at will.

Extracting measurements from a task may require that a task write metadata it normally would not, duplicating information and forcing a task to have some knowledge of its metrics despite the lack of explicit references in the code.

It would be more difficult to retrofit notify calls into the existing tasks framework than to only retrofit the use of Job objects. If task implementors are responsible for calling notify correctly, the requirement is difficult to enforce. If Task is responsible, then tasks would need one run method that serves as the API point of entry (for example, for use by TaskRunner), and a second workhorse method to be implemented by subclasses. Either approach involves significant changes to existing code.

7.4   Example Metric Implementations¶

These examples assume that InvalidMeasurementError is handled by notify to prevent metrics-related errors from leaking into primary task code.

class RunningTimeMeasurer:
    def __init__(self, task):
        self.task = task

    def update(job, event):
        if (event == "Begin"):
            self._start = time.clock()
        elif (event == "Abort" || event == "Finish"):
            try:
                deltaT = time.clock() - self._start
            catch AttributeError as e:
                raise InvalidMeasurementError("No Begin event detected") from e
            metricName = "%s_RunTime" % type(self.task).__name__
            measurement = lsst.verify.Measurement(metricName,
                                                  deltaT * u.seconds))
            measurement.combiner = verify.measSum
            job.measurements.insert(measurement)

config.listeners['RunningTimeMeasurer'] = EventListenerConfig()
config.listeners['RunningTimeMeasurer'].events = ['Begin', 'Abort', 'Finish']

class SourceCounter:
    def __init__(self, task, metric):
        self.task = task
        self.metricName = metric

    def update(job, event):
        if (event == "Finish"):
            try:
                nSources = self.task.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword"
                    ) from e
            measurement = lsst.verify.Measurement(
                self.metricName,
                nSources * u.dimensionless_unscaled))
            measurement.combiner = verify.measSum
            job.measurements.insert(measurement)

astrometer.listeners['SourceCounter'] = EventListenerConfig()
astrometer.listeners['SourceCounter'].args = ['NumAstroSources']  # Metric name
astrometer.listeners['SourceCounter'].events = ['Finish']

8.1   Architecture and Standard Components¶

In this design, Measurement objects will be made by factory objects separate from the task itself. The factory objects are created at a high level and can be applied to the task hierarchy – or even an entire pipeline – as a whole, so managing the resulting measurements can be done by a single Job object.

Measurement factories will be passed to a top-level task using a new method (Task.accept) after the task has completed its processing. Each task is responsible for calling a factory’s actOn method (named thus to allow for future applications other than metrics) with itself as an argument, as well as calling accept on its subtasks recursively. The actOn method is responsible for constructing a Measurement from the information available in the completed task. The Measurements can be stored in the factories that make them, and collected by the code that called the original accept method.

Each factory’s actOn method must accept any Task. Factories for metrics that apply only to certain tasks can check the type of the argument, and do nothing if it doesn’t match. This leads to a brittle design (an unknown number of factories must be updated if an alternative to an existing task is added), but it makes adding new tasks far less difficult than a conventional visitor pattern would.

Measurements may depend on information that is internal to run or a task’s other methods. If this is the case, internal information may be stored in metadata and then extracted by the factory.

Factories can handle metrics that depend on multiple tasks (such as the DIASource fraction) by collecting the necessary information in actOn, but delaying construction of a Measurement until it is requested. Constructing the Measurement outside of actOn is necessary because factories cannot, in general, assume that subtasks will be traversed in the order that’s most convenient for them.

The correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches). Factory classes can provide a merging function appropriate for the metric(s) they compute. The merging can even be internal to the factory, so long as it can keep straight which measurements belong to the same task. See the figure below for an example of a factory that creates measurements for both multiple tasks and multiple units of work for the same task.

Figure 5 Illustration of how measurements are handled in the visitor-based architecture. anInstance and anotherInstance are ConcreteCmdLineTask objects run on different data. The subtask of anotherInstance is omitted for clarity, as are aFactory‘s calls to task methods.

8.2   Requirements for Task Creators and Maintainers¶

Authors of new tasks must be aware of any metrics that apply to the new task but not to all tasks, and modify the code of applicable factories to handle the new task. If the factories make assumptions about a task’s fields, they may constrain the implementation of the task itself.

Custom task runners that call run multiple times per Task object must call accept after each run, to ensure no information is lost. (This is not a problem for TaskRunner, which creates a new Task object for each run.)

In general, maintaining instrumented tasks is easy. The only risk is if factories constrain task implementations in some way; such details must be clearly marked as unchangeable. If factories depend on particular metadata keys being available, the lines that write those keys must be kept in sync with the key names assumed by factories.

Authors of new metrics must implement a factory that measures them, most likely in a central verification package, and register it in a central list of metrics to be applied to tasks. The factory implementation must consider the consequences of being passed any Task, including classes that have not yet been developed.

8.3   Advantages and Disadvantages¶

Because it is so highly centralized, the visitor-based architecture is good at dealing with cross-task metrics – each visitor accesses all tasks run on a particular unit of work, whether it needs to or not.

The difficulty of adding new tasks is this architecture’s greatest weakness. Neither task code nor task configurations are aware of what metrics are being applied, making it difficult for authors of new tasks to know which measurers need to know about them. Metrics that apply to a broad category of tasks (e.g., “any task implementation that handles matching”) are the most vulnerable; neither universal metrics nor implementation-specific metrics are likely to need code changes in response to new tasks.

Adding metrics always requires changes to two packages, the minimum allowed by the verify framework. Metrics cannot be associated or disconnected from a specific task without modifying code, although the top-level registry makes it easy to globally disable a metric.

Extracting measurements from a task may require that a task write metadata it normally would not, duplicating information and forcing a task to have some knowledge of its metrics despite the lack of explicit references in the code.

8.4   Example Metric Implementations¶

class RunningTimeMeasurer(Measurer):
    def __init__(self):
        self.measurements = defaultdict(list)
        self.combiner = verify.measSum

    def actOn(task):
        try:
            start = task.metadata.get("runStartCpuTime")
            end = task.metadata.get("runEndCpuTime")
        except pexExceptions.NotFoundError:
            return
        metricName = "%s_RunTime" % type(task).__name__
        measurement = lsst.verify.Measurement(metricName,
                                              (end - start) * u.seconds))
        self.measurements[type(task)].append(measurement)

class SourceCounter(Measurer):
    def __init__(self, metric):
        self.measurements = defaultdict(list)
        self.combiner = verify.measSum
        self.metricName = metric

    def actOn(task):
        if isinstance(task, AstrometryTask) or isinstance(task, BetterAstrometryTask):
            try:
                nSources = self.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword"
                    ) from e
            measurement = lsst.verify.Measurement(
                self.metricName,
                nSources * u.dimensionless_unscaled))
            self.measurements[type(task)].append(measurement)

class DiaFractionMeasurer(Measurer):
    def __init__(self):
        self._scienceSources = 0
        self._diaSources = 0

    def actOn(task):
        if isinstance(task, CalibrateTask):
            try:
                self._scienceSources += task.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword in %s" % task
                    ) from e
        elif isinstance(task, ImageDifferenceTask):
            try:
                self._diaSources += task.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword in %s" % task
                    ) from e

    # override Measurer.getMergedMeasurements()
    def getMergedMeasurements():
        # Most Measurements are not created if code not run, be consistent
        if self._scienceSources > 0:
            measurement = lsst.verify.Measurement(
                "Fraction_DiaSource_ScienceSource",
                (self._diaSources / self._scienceSources) * u.dimensionless_unscaled)
            return [measurement]
        else:
            return []

9.1   Scalability to many metrics¶

• The metadata-based architecture requires a new decorator, per task, for each metric or group of metrics. In addition, the ComputeMetricsTask package needed to merge results from multiple units of work may bloat as new kinds of metrics are introduced.
• The direct measurement architecture requires a new decorator or function call, per task, for each metric or group of metrics.
• The dataset-based architecture requires a new config entry in ComputeMetricsConfig for each metric or group of metrics.
• The observer-based architecture requires a new config entry, per task, for each metric or group of metrics.
• The visitor-based architecture requires a new config entry in a central location for each metric or group of metrics.

The metadata-based architecture will scale the most poorly to large numbers of metrics, largely because of the need for a potentially large catalog of functions for processing the metadata. The dataset- and visitor-based architectures are the best at avoiding lengthy code or configuration information.

9.2   Supporting metrics that apply to any task¶

All five designs handle this case well. For all cases except the dataset-based architecture , the measurement code could live in pipe_base or a dependency. In the dataset-based architecture, such code lives in the package of ComputeMetricsTask.

9.3   Supporting metrics for groups of related tasks (such as alternate implementations)¶

Architectures may impose API restrictions on a task that are not required by its parent task, such as producing the same metadata or sharing object attributes.

• The metadata-based and direct measurement architectures require that all tasks in a group have the same run decorator.
• The dataset-based architecture treats all tasks that produce the same output identically.
• The observer-based architecture requires that all tasks in a group have the same measurement factory in their configs.
• The visitor-based architecture requires that the measurement factory know of all tasks in a group.

While all architectures except the dataset-based one require that a metric be explicitly associated with each member of the group, the visitor-based architecture handles group metrics worse than the others because task authors need to dig through all metrics to find out which ones they need to support.

9.4   Supporting task-specific metrics¶

• The metadata-based, direct measurement, and observer-based architectures handle this case naturally. The measurement code could live in the task package.
• The dataset-based architecture handles this case well if the desired information can be extracted from the task’s output data, but poorly if the metric refers to status information associated with the task itself.
• The visitor-based architecture allows factories to ignore all but the task of interest. The measurement code must be in a centralized location.

9.5   Supporting cross-task metrics¶

• The metadata-based architecture requires a special channel for each task’s information, and requires that ComputeMetricsTask have some custom code for assembling the final measurement.
• The dataset-based architecture has no special requirements, provided all the required information is present in the data.
• The direct measurement and observer-based architectures require either passing measurement information through metadata, or imposing restrictions on how metrics can be defined. ComputeMetricsTask must have some custom code for assembling the final measurement.
• The visitor-based architecture requires a nonstandard measurement factory.

The dataset-based architecture is by far the best at cross-task metrics; the direct measurement and observer-based architectures are the worst.

9.6   Associating measurements with a task class¶

• The metadata-based, direct measurement, observer-based, and visitor-based architectures interact with a task object, so the measurement can easily be made specific to the class if need be (the <class>_RunTime metric in the examples illustrates one way to do this).
• The dataset-based architecture would need to reconstruct the task class from config information.

The dataset-based architecture requires more complex code to support measurement tracking by implementation class, although most of this can be abstracted by ComputeMetricsTask.

9.7   Associating measurements with a subtask slot in a parent task¶

• The metadata-based architecture provides this information as part of the metadata key.
• The direct measurement and observer-based architectures can extract information about the task’s relationship with its parent from the task object directly. In the observer-based architecture, the functionality can be hidden in a base class for factories.
• The dataset-based architecture can have this information hard-coded in a measurement function, or use config information to provide it.
• The visitor-based architecture architecture can extract information about the task’s relationship with its parent from the task object, like an observer, or it can use config information to do so as part of a post-processing step.

The metadata-based architecture handles by-subtask metrics most naturally, but all five designs can easily provide this information.

9.8   Adding new metrics¶

• The metadata-based, direct measurement, and observer-based architectures require writing the appropriate measurement code, then registering it with each task of interest. All three designs provide workarounds to minimize the workload for widely-applicable metrics.
• The dataset-based architecture requires writing the appropriate measurement code and identifying the data it requires.
• The visitor-based architecture requires writing the appropriate measurement code, and having it test whether tasks apply to it.

Adding a universally applicable metric requires less work in the visitor-based architecture but more work in the others, while for task-specific metrics the situation is reversed.

9.9   Adding new tasks¶

• The metadata-based and direct measurement architectures require new tasks to have the appropriate decorators for their metrics. In the direct measurement architecture, some metrics may require internal function calls rather than decorators, which are more difficult to spot in old tasks’ code.
• The dataset-based architecture does not require extra work unless the new task produces new dataset types that existing measurers must be aware of. If such updates are necessary, however, the set of metrics to update is difficult to determine.
• The observer-based architecture requires new tasks to have the appropriate entries in their config.
• The visitor-based architecture may require changes to measurement code when new tasks are added. The set of metrics to update cannot be determined by looking at old tasks’ code.

The dataset-based architecture minimizes the work needed to implement new tasks. The visitor-based architecture is considerably worse at handling new tasks than the other four.

9.10   Allowing pipeline users to ignore metrics¶

None of the five designs require user setup or force the user to handle measurements. At worst, a Job object might be persisted unexpectedly, and persisted Jobs will become invisible once verify uses Butler persistence.

9.11   Remaining agnostic to units of work¶

• The metadata-based architecture has a lot of difficulty reporting measurements as if all the data were processed in a single task invocation. Because the combining code cannot be provided by the task package, it requires cross-package coordination in a way that is bug-prone and scales poorly to large numbers of metrics.
• The direct measurement and observer-based architectures give Measurements the code needed to combine them. This code must be called either from CmdLineTask.parseAndRun, or from a ComputeMetricsTask.
• The dataset-based architecture handles multiple units of work as part of the measurement process, although it must still be aware of them in order to make well-posed butler queries.
• The visitor-based architecture give Measurement factories the code needed to combine measurements. This code must be called from CmdLineTask.parseAndRun.

The dataset-based architecture is the best at handling multiple units of work. The metadata-based architecture is considerably worse than the others.

9.12   Supporting families of similar measurements¶

All five architectures can handle families of metrics (e.g., running time for different task classes, or astrometric quality for different CCDs) by treating them as independent measurements. However, in all cases except the dataset-based architecture some care would need to be taken to keep the measurements straight, particularly when combining measurements of the same metric for multiple units of work.

9.13   Enabling/disabling expensive metrics¶

• The metadata-based and direct measurement architectures incorporate measurements directly into code, making it difficult to remove them completely. They can still check a config flag before running, however.
• The observer-based architecture uses configs to attach measurement factories to tasks, so they can be easily added or removed. However, disabling calculation of a metric for all tasks requires touching many configs.
• The dataset-based and visitor-based architectures use a central config to enable measurement computation, so they can easily be added or removed. However, a measurement cannot be disabled for specific tasks without modifying code.

Given that we most likely wish to disable expensive metrics globally, the dataset- and visitor-based architectures provide the best support for this feature, and the observer-based architecture the worst.

9.14   Forward-compatibility with SuperTask¶

The design described in DMTN-055 makes a number of significant changes to the task framework, including requiring that tasks be immutable (a requirement currently violated by Task.metadata), defining pipelines via a new class rather than a high-level CmdLineTask, and introducing an ExecutionFramework for pre- and post-processing pipelines.

• The metadata-based architecture can be translated to SuperTask easily, once the metadata system itself is fixed to allow immutable tasks. The proposed ComputeMetricsTask could be partially or wholly replaced by ExecutionFramework.
• The direct measurement and dataset-based architectures would not require special adaptation, although the latter may need code changes to make use of the third-generation Butler.
• The observer-based architecture as presented would struggle with the effective statelessness of tasks, because run would need to be responsible for providing information to factories.
• The visitor-based architecture would require that pipelines ensure visitors are passed to each high-level task in the pipeline. It’s not clear how this would affect the Pipeline framework’s flexibility. The architecture would not be able to handle stateless tasks, however, as there is no other way to pass task information to a visitor.

The observer- and visitor-based architecture will adapt the worst to the SuperTask framework, while the other three will have relatively little difficulty.