4.1.1   Architecture and Standard Components¶

In this design, all information of interest to metrics will be stored in a task’s metadata. The metadata will be passed up to a dedicated package (named, for example, verify_measurements) that will find the appropriate keys and create Measurement objects. High-level handling of the measurements can be done by a single Job object. A prototype of this approach is used in the lsst.ap.verify.measurements package to handle running times, but in the interests of portability to other pipelines the final code should be a dependency of ap_verify rather than a subpackage.

To minimize coupling with the task itself, the code that performs the measurements can be placed in decorators analogous to lsst.pipe.base.timeMethod. This approach also avoids code duplication for metrics that apply to more than one task class. However, as the number of metrics grows, so will the number of decorators attached to a class’s run method. Related metrics can be grouped in one decorator; for example, timeMethod measures not only timing, but also memory usage and other forms of profiling.

While tasks or their decorators are necessarily coupled to verify_metrics, verify_measurements need not know about most defined metrics if the metadata keys follow a particular format that allows discovery of measurements by iterating over the metadata (e.g., "<task-prefix>.verify.measurements.foo" for a metric named "package.foo"). Since the correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches), a standardized key (perhaps "<task-prefix>.verify.combiners.foo") can be used to specify the algorithm to combine the data. The use of a string to indicate the combiner only scales well if the majority of metrics share a small number of combiners, such as sum or average.

Figure 1 Illustration of how measurement data are passed up from tasks in the metadata-based architecture. anInstance and anotherInstance are ConcreteCmdLineTask objects run on different data.

Standardized metadata keys cannot handle metrics that depend on the results of multiple tasks (such as the DIASource fraction). In this case, information can still be passed up through metadata, but tasks should avoid using the verify.measurement prefix so that generic Measurement-making code does not mistakenly process them. Instead, each cross-task metric will need its own function in verify_measurements to search across all task classes for the relevant information and make a Measurement. Handling of cross-task metrics must therefore be coordinated across at least three packages – verify_metrics, the task package(s), and verify_measurements.

Standardized metadata keys can be used to record supplementary information about a measurement, for example by using verify.extras and verify.notes PropertySets.

4.1.2   Requirements for Task Creators and Maintainers¶

The main requirement imposed on authors of new tasks is the use of measurement decorators. It may be necessary to ensure decorators are applied in a particular order (for example, timeMethod should not include measurement overhead, so it should be listed last). If the decorators make assumptions about a task’s fields, they may constrain the implementation of the task itself. Implementation constraints go away if measurement metadata are written directly by a task’s methods, but then the task author is responsible for following all the conventions described above, including specifying a combiner and any other auxiliary metadata keys.

Custom task runners that call run multiple times per Task object must copy the object’s metadata after each run, to keep it from getting lost. (This is not a problem for TaskRunner, which creates a new Task for each run.)

If all verification-related work is done by decorators, than maintaining instrumented tasks is easy; Task code can be changed and decorators added or removed as desired. The only risk is if decorators constrain task implementations in some way; such details must be clearly marked as unchangeable. If decorators depend on particular metadata keys being available, the lines that write those keys must be kept in sync with the key names passed to decorators (see DCR goodness of fit). If tasks write measurement metadata directly, then maintainers must know not to touch those lines in any way.

Authors of new metrics must implement a decorator that measures them, most likely in pipe_base or a specific task’s package, and add it to all relevant task classes. The decorator must conform to all conventions regarding metadata keys. If the metric requires a new way to combine units of work, the new combiner must be implemented and registered under a unique name in verify_measurements.

4.1.3   Advantages and Disadvantages¶

A metadata-driven architecture limits changes to the task framework to imposing a convention for metadata keys; tasks need not depend on verify at all. However, it does require a centralized Measurement-making package that frameworks like ap_verify or validate_drp must call after all tasks have been run.

Adding most metrics requires changes to two packages (the minimum allowed by the verify framework), but cross-task metrics require three. Metrics cannot be added to or removed from a task without modifying code. Configs could be used to disable them, although this breaks the separation of task and instrumentation code somewhat.

Dividing a dataset into multiple units of work is poorly supported by a metadata-based architecture, because each metric may require a different way to synthesize a full-dataset measurement from the individual measurements, yet metadata does not allow code to be attached to measurements. On the other hand, it is very easy to support tracking of subtask measurements by both class and role, because the metadata naturally provide by-role information.

The biggest weakness of this architecture may well be its dependence on convention: metadata keys that don’t conform to the expected format must, in many cases, be silently ignored.

4.1.4   Example Metric Implementations¶

Note: in practice, all the metadata keys seen by verify_measurements would be prefixed by the chain of subtasks that produced them, requiring more complex handling than a lookup by a fixed name. This extra complexity is ignored in the examples, but is fairly easy to implement.

obj.metadata.add(name = "verify.measurements.%s_RunTime" % className,
                 value = deltaT)
obj.metadata.add(name = "verify.combiners.%s_RunTime" % className,
                 value = "sum")

def numAstroSources():
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        # Any substitute for AstrometryTask must share its return value spec
        nSources = len(result.matches)
        self.metadata.add(name = "verify.measurements.NumAstroSources",
                         value = nSources)
        self.metadata.add(name = "verify.combiners.NumAstroSources",
                         value = "sum")
        return result
    return wrapper

class AstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, sourceCat, exposure):
        ...

class BetterAstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, sourceCat, exposure):
        ...

def numScienceSources():
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        nSources = len(result.sourceCat)
        self.metadata.add(name = "verify.fragments.NumScienceSources",
                         value = nSources)
        return result
    return wrapper

class CalibrateTask(RefMatchTask):
    ...
    @numScienceSources
    @pipeBase.timeMethod
    def run(self, dataRef, exposure=None, background=None, icSourceCat=None,
        doUnpersist=True):
        ...

def numDiaSources():
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        nSources = len(result.sources)
        self.metadata.add(name = "verify.fragments.NumDiaSources",
                         value = nSources)
        return result
    return wrapper

class ImageDifferenceTask(RefMatchTask):
    ...
    @numDiaSources
    @pipeBase.timeMethod
    def run(self, sensorRef, templateIdList=None):
        ...

def measureDiaSourceFraction(allVerifyMetadata):
    SCIENCE_KEY = "fragments.NumScienceSources"
    DIA_KEY = "fragments.NumDiaSources"
    scienceSources = 0
    diaSources = 0
    for oneRunMetadata in allVerifyMetadata:
        if oneRunMetadata.exists(SCIENCE_KEY):
            scienceSources += oneRunMetadata.get(SCIENCE_KEY)
        if oneRunMetadata.exists(DIA_KEY):
            diaSources += oneRunMetadata.get(DIA_KEY)

    # Generic Measurements are not created if code not run, be consistent
    if scienceSources > 0:
        return lsst.verify.Measurement(
            "Fraction_DiaSource_ScienceSource",
            (diaSources / scienceSources) * u.dimensionless_unscaled))
    else:
        return None

def makeSpecializedMeasurements(allVerifyMetadata):
    ...
    measurement = measureDiaSourceFraction(allVerifyMetadata)
    if measurement is not None:
        job.measurements.insert(measurement)
    ...

def dcrGoodnessOfFit(valueKey, typeKey):
    def customWrapper(func):
        @wraps(func)
        def wrapper(self, *args, **kwargs):
            try:
                return func(self, *args, **kwargs)
            finally:
                if self.metadata.exists(valueKey) and self.metadata.exists(typeKey):
                    gofValue = self.metadata.get(valueKey)
                    gofType = self.metadata.get(typeKey)
                    self.metadata.add(name = "verify.measurements.DcrGof",
                                     value = gofValue)
                    self.metadata.add(name = "verify.combiners.DcrGof",
                                     value = "dcrStatCombine")
                    # added to Measurement's `notes` member, AND needed by dcrStatCombine
                    self.metadata.add(name = "verify.notes.DcrGof.gofStatistic",
                                     value = gofType)
        return wrapper
    return customWrapper

class DcrMatchTemplateTask(CmdLineTask):
    ...
    @dcrGoodnessOfFit("gof", "gofType")
    @pipeBase.timeMethod
    def run(self, dataRef, selectDataList=[]):
        ...

def dcrStatCombine(allVerifyDcrMetadata):
    try:
        statisticType = allVerifyDcrMetadata[0].get(
            "notes.DcrGof.gofStatistic")
        if statisticType == "Chi-Squared":
            chisqCombine(allVerifyDcrMetadata)
        elif ...

4.2.1   Architecture and Standard Components¶

In this design, Measurement objects will be made by tasks. Tasks will have a Job object (Task.job) for collecting their Measurements, which can be either persisted or passed upward as part of a task’s return value. High-level handling of all Measurements would be handled by a Job living in a verification package (such as ap_verify), which consolidates the task-specific Job objects.

To minimize coupling with the task itself, the code that creates the Measurements can be placed in decorators similar to lsst.pipe.base.timeMethod, except that the decorators would update Task.job rather than Task.metadata. This approach also avoids code duplication for metrics that apply to more than one task class. However, as the number of metrics grows, so will the number of decorators attached to a class’s run method. Related metrics can be grouped in one decorator; for example, timeMethod measures not only timing, but also memory usage and other forms of profiling.

Measurements may depend on information that is internal to run or a task’s other methods. If this is the case, the Measurement may be created by an ordinary function called from within run, instead of by a decorator, or the internal information may be stored in metadata and then extracted by the decorator.

Directly constructed Measurements cannot handle metrics that depend on the results of multiple tasks (such as the DIASource fraction); such metrics must be measured in a centralized location. There are two ways to handle cross-task measurements:

1. The necessary information can be stored in metadata, and computed as a special step after all tasks have been run.
2. We can impose a requirement that all cross-task metrics be expressible in terms of single-task metrics. In the DIASource fraction example such a requirement is a small burden, since both “Number of detected sources” and “Number of DIASources” are interesting metrics in their own right, but this may not be the case in general.

The correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches). This information can be provided by requiring that Measurement objects include a merging function, which can be invoked either as part of the task parallelization framework (as shown in the figure), or as part of a measurement-handling package (as required by the metadata-based architecture).

Figure 2 Illustration of how measurements are handled in the direct-measurement and observer-based architectures, assuming Job persistance is not used and multiple units of work are combined as part of the existing parallelism framework. anInstance and anotherInstance are ConcreteCmdLineTask objects run on different data. The subtask of anotherInstance and the Measurement it produces are omitted for clarity.

4.2.2   Requirements for Task Creators and Maintainers¶

The main requirement imposed on authors of new tasks is the use of measurement decorators or functions. It may be necessary to ensure measurements are made in a particular order (for example, timing should not include measurement overhead). If measurement decorators make assumptions about a task’s fields, they may constrain the implementation of the task itself. Functions called from within run do not impose implementation constraints, but may be less visible to maintainers if they are buried in the rest of the task code.

If verify does not support multiple measurements of the same metric, then any task runner that calls run multiple times per Task object must extract the object’s job after each run, to prevent information from being lost. (This is not a problem for TaskRunner, which creates a new Task object for each run.)

If all verification-related work is done by decorators, than maintaining instrumented tasks is easy; task code can be changed and decorators added or removed as desired. The only major risk is if decorators constrain task implementations in some way; such details must be clearly marked as unchangeable. If measurements are made by functions called from within run, then the maintainability of the task depends on how well organized the code is – if measurement-related calls are segregated into their own block, maintainers can easily work around them.

Authors of new metrics must implement a decorator or function that measures them, most likely in pipe_base or a specific task’s package, and add it to all relevant task classes. The decorator or function must ensure the resulting Measurement has a combining functor. Standard combiners may be made available through a support package to reduce code duplication.

4.2.3   Advantages and Disadvantages¶

A direct-measurement architecture minimizes changes needed to the verify framework, which already assumes each task is responsible for persisting Job information.

Adding most metrics requires changes to two packages (the minimum allowed by the verify framework), but cross-task metrics require either two (if all single-task components are themselves metrics) or three (if the implementation is kept as general as possible). Metrics cannot be added to or removed from a task without modifying code. Configs could be used to disable them, although this breaks the separation of task and instrumentation code somewhat.

Because of its decentralization, a direct-measurement architecture has trouble supporting cross-task metrics; in effect, one needs one framework for single-task metrics and another for cross-task metrics.

4.2.4   Example Metric Implementations¶

def timeMeasurement():
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        try:
            return func(self, *args, **kwargs)
        finally:
            try:
                start = self.metadata.get("runStartCpuTime")
                end = self.metadata.get("runEndCpuTime")
            except pexExceptions.NotFoundError as e:
                raise AttributeError(
                    "@timeMethod must be listed after @timeMeasurement"
                ) from e
            metricName = "%s_RunTime" % type(self).__name__
            measurement = lsst.verify.Measurement(metricName,
                                                  (end - start) * u.seconds))
            measurement.combiner = verify.measSum
            self.job.measurements.insert(measurement)
    return wrapper

class AFancyTask(Task):
    ...
    @timeMeasurement
    @pipeBase.timeMethod
    def run(self, data):
        ...

def numAstroSources():
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        # Any substitute for AstrometryTask must share its return value spec
        nSources = len(result.matches)
        measurement = lsst.verify.Measurement(
            "NumAstroSources",
            nSources * u.dimensionless_unscaled))
        measurement.combiner = verify.measSum
        self.job.measurements.insert(measurement)
        return result
    return wrapper

class AstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, sourceCat, exposure):
        ...

class BetterAstrometryTask(RefMatchTask):
    ...
    @numAstroSources
    @pipeBase.timeMethod
    def run(self, sourceCat, exposure):
        ...

def numScienceSources():
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        nSources = len(result.sourceCat)
        measurement = lsst.verify.Measurement(
            "NumScienceSources",
            nSources * u.dimensionless_unscaled))
        measurement.combiner = verify.measSum
        self.job.measurements.insert(measurement)
        return result
    return wrapper

class CalibrateTask(RefMatchTask):
    ...
    @numScienceSources
    @pipeBase.timeMethod
    def run(self, dataRef, exposure=None, background=None, icSourceCat=None,
        doUnpersist=True):
        ...

def numDiaSources():
    @wraps(func)
    def wrapper(self, *args, **kwargs):
        result = func(self, *args, **kwargs)
        nSources = len(result.sources)
        measurement = lsst.verify.Measurement(
            "NumDiaSources",
            nSources * u.dimensionless_unscaled))
        measurement.combiner = verify.measSum
        self.job.measurements.insert(measurement)
        return result
    return wrapper

class ImageDifferenceTask(RefMatchTask):
    ...
    @numDiaSources
    @pipeBase.timeMethod
    def run(self, sensorRef, templateIdList=None):
        ...

def measureFraction(job, metric, numeratorName, denominatorName):
    try:
        numerator = job.measurements[numeratorName]
        denominator = job.measurements[denominatorName]
    except KeyError:
        # Measurements not made, fraction not applicable
        return

    fraction = numerator.quantity / denominator.quantity
    measurement = lsst.verify.Measurement(metric, fraction)
    # TODO: how to handle extras and notes?
    job.measurements.insert(measurement)

def makeSupplementaryMeasurements(masterJob):
    ...
    measureFraction(masterJob,
                    "Fraction_DiaSource_ScienceSource",
                    "NumDiaSources",
                    "NumScienceSources")
    ...

def chisqCombine(measurements):
    """Compute a chi-squared Measurement for a data set from values for subsets."""
    ...

def dcrGoodnessOfFit(valueKey, typeKey):
    def customWrapper(func):
        @wraps(func)
        def wrapper(self, *args, **kwargs):
            try:
                return func(self, *args, **kwargs)
            finally:
                if self.metadata.exists(valueKey) and self.metadata.exists(typeKey):
                    gofValue = self.metadata.get(valueKey)
                    gofType = self.metadata.get(typeKey)
                    measurement = lsst.verify.Measurement(
                        "DcrGof",
                        gofValue * getUnits(gofType))
                    measurement.combiner = getCombiner(gofType)
                    measurement.notes['gofStatistic', gofType]
                    self.job.measurements.insert(measurement)
        return wrapper
    return customWrapper

class DcrMatchTemplateTask(CmdLineTask):
    ...
    @dcrGoodnessOfFit("gof", "gofType")
    @pipeBase.timeMethod
    def run(self, dataRef, selectDataList=[]):
        ...

4.3.1   Architecture and Standard Components¶

In this design, Measurement objects will be made by factory objects separate from the task itself. Tasks will have a Job object for collecting their measurements, which can be either persisted or passed upward as part of a task’s return value. High-level handling of all measurements can be handled by a Job living in a verification package (such as ap_verify), which consolidates the task-specific Job objects.

The factories for the appropriate metrics will be registered with a task at construction time, using a new method (called Task.addListener, to allow for future applications other than metrics). The registration can be made configurable, although if each metric has its own factory, the config file will be an extra place that must be kept in sync with metrics definitions in verify_metrics. If one class measures multiple related metrics, then config changes are needed less often.

A task has a method (Task.notify) that triggers its registered factories on one of several standardized events (the examples assume there are three: Begin, Abort, and Finish); the events applicable to a given factory are specified at registration. Factories query the task’s metadata for information they need, make the appropriate Measurement object(s), and pass them back to the task’s Job.

Measurements may depend on information that is internal to run or a task’s other methods. If this is the case, internal information may be stored in metadata and then extracted by the factory.

If metrics depend on the results of multiple tasks (such as the DIASource fraction), they can be worked around using the same techniques as for direct measurements. It is also possible to handle cross-task metrics by registering the same factory object with two tasks. However, supporting such a capability would require that factories be created and attached to tasks from above, which would take away this framework’s chief advantage – that it does not require centralized coordination, but is instead largely self-operating. See the visitor pattern for a design that does handle cross-task metrics this way.

Figure 3 Illustration of how measurements are created in the observer-based architecture, assuming all measurement information is available through metadata. Handling of measurements once they have been created works the same as for the direct measurement architecture.

The correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches). This information can be provided by requiring that Measurement objects include a merging function.

4.3.2   Requirements for Task Creators and Maintainers¶

Authors of new tasks must include in the task configuration information indicating which factories are to be attached to a task. The convention for defaults may be to register either all applicable factories, or a subset that is deemed to have little runtime overhead. The registration process itself can be handled by Task.__init__ with no direct developer intervention.

If verify does not support multiple measurements of the same metric, then any task runner that calls run multiple times per Task object must extract the object’s job after each run, to prevent information from being lost. (This is not a problem for TaskRunner, which creates a new Task object for each run.)

In general, maintaining instrumented tasks is easy. The only risk is if factories constrain task implementations in some way; such details must be clearly marked as unchangeable. If factories depend on particular metadata keys being available, the lines that write those keys must be kept in sync with the key names assumed by factories.

Authors of new metrics must implement a factory that measures them, most likely in pipe_base or a specific task’s package, and add it to all relevant configs. The factory must ensure the resulting Measurement has a combining functor, as for direct construction of Measurements.

4.3.3   Advantages and Disadvantages¶

An observer-based architecture provides maximum decentralization of responsibility: not only is each task responsible for handling its own measurements, but little to no task code needs to be aware of the specific metrics defined for each task. While the observer architecture is not the only one that allows run-time configuration of metrics, it is the one where such configuration fits most naturally by far. However, the high decentralization also gives it the worst support for cross-task metrics.

Adding single-task metrics requires changes to two packages, the minimum allowed by the verify framework. Metrics can be enabled and disabled at will.

Extracting measurements from a task may require that a task write metadata it normally would not, duplicating information and forcing a task to have some knowledge of its metrics despite the lack of explicit references in the code.

It would be difficult to retrofit notify calls into the existing tasks framework. If task implementors are responsible for calling notify correctly, the requirement is difficult to enforce. If Task is responsible, then tasks would need one run method that serves as the API point of entry (for example, for use by TaskRunner), and a second workhorse method to be implemented by subclasses. Either approach involves significant changes to existing code.

4.3.4   Example Metric Implementations¶

These examples assume that InvalidMeasurementError is handled by notify to prevent metrics-related errors from leaking into primary task code.

class RunningTimeMeasurer:
    def __init__(self, task):
        self.task = task

    def update(event):
        if (event == "Begin"):
            self._start = time.clock()
        elif (event == "Abort" || event == "Finish"):
            try:
                deltaT = time.clock() - self._start
            catch AttributeError as e:
                raise InvalidMeasurementError("No Begin event detected") from e
            metricName = "%s_RunTime" % type(self.task).__name__
            measurement = lsst.verify.Measurement(metricName,
                                                  deltaT * u.seconds))
            measurement.combiner = verify.measSum
            self.task.job.measurements.insert(measurement)

config.listeners['RunningTimeMeasurer'] = EventListenerConfig()
config.listeners['RunningTimeMeasurer'].events = ['Begin', 'Abort', 'Finish']

class SourceCounter:
    def __init__(self, task, metric):
        self.task = task
        self.metricName = metric

    def update(event):
        if (event == "Finish"):
            try:
                nSources = self.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword"
                    ) from e
            measurement = lsst.verify.Measurement(
                self.metricName,
                nSources * u.dimensionless_unscaled))
            measurement.combiner = verify.measSum
            self.task.job.measurements.insert(measurement)

astrometer.listeners['SourceCounter'] = EventListenerConfig()
astrometer.listeners['SourceCounter'].args = ['NumAstroSources']  # Metric name
astrometer.listeners['SourceCounter'].events = ['Finish']

class DcrGoodnessOfFitMeasurer:
    def __init__(self, task):
        self.task = task

    def update(event):
        if (event == "Finish"):
            try:
                gofValue = self.metadata.get('gof')
                gofType = self.metadata.get('gofType')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `gof` and `gofType` metadata keywords"
                    ) from e
            measurement = lsst.verify.Measurement(
                "DcrGof",
                gofValue * getUnits(gofType))
            measurement.combiner = getCombiner(gofType)
            measurement.notes['gofStatistic', gofType]
            self.task.job.measurements.insert(measurement)

config.listeners['DcrGoodnessOfFitMeasurer'] = EventListenerConfig()
config.listeners['DcrGoodnessOfFitMeasurer'].events = ['Finish']

4.4.1   Architecture and Standard Components¶

In this design, Measurement objects will be made by factory objects separate from the task itself. The factory objects are created at a high level and applied to the task hierarchy as a whole, so managing the resulting measurements can be done by a single Job object.

Measurement factories will be passed to a top-level task using a new method (Task.accept) after the task has completed its processing. Each task is responsible for calling a factory’s actOn method (named thus to allow for future applications other than metrics) with itself as an argument, as well as calling accept on its subtasks recursively. The actOn method is responsible for constructing a Measurement from the information available in the completed task. The Measurements can be stored in the factories that make them, and collected by the code that called the original accept method.

Each factory’s actOn method must accept any Task. Factories for metrics that apply only to certain tasks can check the type of the argument, and do nothing if it doesn’t match. This leads to a brittle design (an unknown number of factories must be updated if an alternative to an existing task is added), but it makes adding new tasks far less difficult than a conventional visitor pattern would.

Measurements may depend on information that is internal to run or a task’s other methods. If this is the case, internal information may be stored in metadata and then extracted by the factory.

Factories can handle metrics that depend on multiple tasks (such as the DIASource fraction) by collecting the necessary information in actOn, but delaying construction of a Measurement until it is requested. Constructing the Measurement outside of actOn is necessary because factories cannot, in general, assume that subtasks will be traversed in the order that’s most convenient for them.

The correct way to merge measurements from multiple units of work depends on the metric (for example, the four use cases described above require three different approaches). Factory classes can provide a merging function appropriate for the metric(s) they compute. The merging can even be internal to the factory, so long as it can keep straight which measurements belong to the same task. See the figure below for an example of a factory that creates measurements for both multiple tasks and multiple units of work for the same task.

Figure 4 Illustration of how measurements are handled in the visitor-based architecture. anInstance and anotherInstance are ConcreteCmdLineTask objects run on different data. The subtask of anotherInstance is omitted for clarity, as are aFactory‘s calls to task methods.

4.4.2   Requirements for Task Creators and Maintainers¶

Authors of new tasks must be aware of any metrics that apply to the new task but not to all tasks, and modify the code of applicable factories to handle the new task. If the factories make assumptions about a task’s fields, they may constrain the implementation of the task itself.

Custom task runners that call run multiple times per Task object must call accept after each run, to ensure no information is lost. (This is not a problem for TaskRunner, which creates a new Task object for each run.)

In general, maintaining instrumented tasks is easy. The only risk is if factories constrain task implementations in some way; such details must be clearly marked as unchangeable. If factories depend on particular metadata keys being available, the lines that write those keys must be kept in sync with the key names assumed by factories.

Authors of new metrics must implement a factory that measures them, most likely in a central verification package, and register it in a central list of metrics to be applied to tasks. The factory implementation must consider the consequences of being passed any Task, including classes that have not yet been developed.

4.4.3   Advantages and Disadvantages¶

Because it is so highly centralized, the visitor-based architecture is the best at dealing with cross-task metrics – each visitor accesses all tasks run on a particular unit of work, whether it needs to or not.

The difficulty of adding new tasks is this architecture’s greatest weakness. Neither task code nor task configurations are aware of what metrics are being applied, making it difficult for authors of new tasks to know which measurers need to know about them. Metrics that apply to a broad category of tasks (e.g., “any task implementation that handles matching”) are the most vulnerable; neither universal metrics nor implementation-specific metrics are likely to need code changes in response to new tasks.

Adding metrics always requires changes to two packages, the minimum allowed by the verify framework. Metrics cannot be associated or disconnected from a specific task without modifying code, although the top-level registry makes it easy to globally disable a metric.

Extracting measurements from a task may require that a task write metadata it normally would not, duplicating information and forcing a task to have some knowledge of its metrics despite the lack of explicit references in the code.

4.4.4   Example Metric Implementations¶

class RunningTimeMeasurer(Measurer):
    def __init__(self):
        self.measurements = defaultdict(list)
        self.combiner = verify.measSum

    def actOn(task):
        try:
            start = task.metadata.get("runStartCpuTime")
            end = task.metadata.get("runEndCpuTime")
        except pexExceptions.NotFoundError:
            return
        metricName = "%s_RunTime" % type(task).__name__
        measurement = lsst.verify.Measurement(metricName,
                                              (end - start) * u.seconds))
        self.measurements[type(task)].append(measurement)

class SourceCounter(Measurer):
    def __init__(self, metric):
        self.measurements = defaultdict(list)
        self.combiner = verify.measSum
        self.metricName = metric

    def actOn(task):
        if isinstance(task, AstrometryTask) or isinstance(task, BetterAstrometryTask):
            try:
                nSources = self.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword"
                    ) from e
            measurement = lsst.verify.Measurement(
                self.metricName,
                nSources * u.dimensionless_unscaled))
            self.measurements[type(task)].append(measurement)

class DiaFractionMeasurer(Measurer):
    def __init__(self):
        self._scienceSources = 0
        self._diaSources = 0

    def actOn(task):
        if isinstance(task, CalibrateTask):
            try:
                self._scienceSources += task.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword in %s" % task
                    ) from e
        elif isinstance(task, ImageDifferenceTask):
            try:
                self._diaSources += task.metadata.get('sources')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `sources` metadata keyword in %s" % task
                    ) from e

    # override Measurer.getMergedMeasurements()
    def getMergedMeasurements():
        # Most Measurements are not created if code not run, be consistent
        if self._scienceSources > 0:
            measurement = lsst.verify.Measurement(
                "Fraction_DiaSource_ScienceSource",
                (self._diaSources / self._scienceSources) * u.dimensionless_unscaled)
            return [measurement]
        else:
            return []

class DcrGoodnessOfFitMeasurer(Measurer):
    def __init__(self):
        self.measurements = defaultdict(list)
        self.combiner = None

    def actOn(task):
        if isinstance(task, DcrMatchTemplateTask):
            try:
                gofValue = self.metadata.get('gof')
                gofType = self.metadata.get('gofType')
            except KeyError as e:
                raise InvalidMeasurementError(
                    "Expected `gof` and `gofType` metadata keywords"
                    ) from e
            measurement = lsst.verify.Measurement(
                "DcrGof",
                gofValue * getUnits(gofType)))
            measurement.notes['gofStatistic', gofType]
            self.combiner = getCombiner(gofType)  # assumed same for all runs
            self.measurements[type(task)].append(measurement)