Deblender Sprint Demo

Introduction  - Integrating “New Scarlet” with the stack 


A replacement deblending algorithm has been in research and development since S17.  We need to integrate it with the stack to make use of command line and batch processing tools to be able to run it at scale on real data.  Integration exercises require expertise from all the different components including: Stack architecture, New Scarlet algorithm, batch processing protocols, DRP Validation (e.g. fake source injection, QA, etc…) .  For this reason, we sought to accomplish this integration in the form of an all-hands sprint with DRP participants Fred Moolekamp, Nate Lust, Sophie Reed, Dan Taranu, Yusra AlSayyad, Lauren MacArthur (for the week without an Ops Rehearsal) and with Jim Bosch serving as product owner. 

Sprint Goals

Get New Scarlet running on HSC data with Stack command line tools (Put the wheels on the skateboard) so we can answer some questions about it. 

Out of scope: making Scarlet faster, better, and more robust.
In scope: List of tasks necessary to make Scarlet faster, better, and more robust later. 
  • A CommandlineTask that can be run without crashing. 
  • Ability to run multibandDriver without crashing
Questions:

  • How long does it take?
  • What do the failures look like (images)?
  • Fraction of exceptions per blend
  • Fraction of flagged failures per blend
  • Runtime per blend as a function of number of peaks and footprint area. 
  • Where is the time going? (Call stack)

Results

Analyses live in this repo:


  • What changes were needed to get it running (Fred)
  • Description of data run (Yusra)
  • OUTPUTS: 404 500x500 patches
# final demo repo in:
/datasets/hsc/repo/rerun/private/yusra/deblender-sprint/scarlet/w_2019_18_05k_noSym_demo
[yusra@lsst-dev01 9813]$ ls */forced_src-* | wc -l
404
[yusra@lsst-dev01 9813]$ ls */deblendedModel* | wc -l
405
  • Demo of running it with commandline and multiband. Instructions on how to run deblender-sprint branch of scarlet. (Yusra)


Testing (3 categories)

  • Failures:
  • Exception fractions (before and after) (Yusra)
  • Failures on images: “red boxes turn to  green boxes” (before and after) (Lauren)