​​Gen3 Middleware Command-Line Tools
​​THIS IS A WORK IN PROGRESS.  Not all commands are present, some present may need to be merged or redefined, and none of them have the full suite of options they’ll need.
​​
​​We envision a command-line tooling environment centered around a very small number of top-level commands (just two are planned so far: butler​ and pipetask​), with extensible subcommands.
​​Subcommands would share code and interfaces for common arguments, such as those that specify data repository locations or collections.
​​The extension mechanism is not specified by this document, but it must allow us to add a subcommand in a package downstream of the one that defines the parent tool - while the butler​ tool and some of its subcommands may be defined in daf_butler​, for example, its ingest​ subcommand should be defined in obs_base​.

​​Common Arguments

​​REPO​
  • ​​Path to an existing data repository root or configuration file. Implies <REPOCFG>​.
​​--sync​
  • ​​Instead of bulk-inserting dimension values, use Database.sync​ to compare them to existing ones and inset only those that are missing.  This may go slower for bulk operations and will result in incremental commits rather than a single atomic change to the data repository.
​​--create-repo​
  • ​​Synonym for running butler create​ before invoking this command.
​​--register instrument​
  • ​​Synonym for running butler register-instrument​ before invoking this command.
​​<REPOCFG>​
  • ​​Shorthand for the (unspecified) optional arguments that provide overrides to butler configuration.
​​<PEXCFG>​
  • ​​Shorthand for the -c/--config​ and -C/--configfile​ arguments that provide overrides for lsst.pex.config​ objects.
​​<DIMXPR>​
  • ​​Shorthand for the (unspecified) optional arguments that specify a dimension query expression.
​​<INCOLXPR>​
  • ​​Shorthand for the (unspecified) optional arguments that specify a CollectionSearch​ or CollectionQuery​.
​​<DSTQRY>​
  • ​​Shorthand for the arguments used to query for dataset types, i.e.
  • ​​[--regex PATTERN] [DST...]​

​​Dataset and collection management: butler​ subcommands

​​query-collections

​​butler query-collections [COL...] [--regex PATTERN] [--with-dataset DST] [--yaml] REPO
​​Query for the collections that match the given criteria.

​​query-dataset-types

​​butler query-dataset-types [--in-collection COL] [--with-storage-class SC] [--with-dimension DIM] [--names] [--yaml] REPO <DSTQRY>
​​Query for the dataset types that match the given critera.

​​query-datasets

​​butler query-datasets [--yaml] REPO <DSTQRY> <DIMXPR> <INCOLXPR>
​​Query for the datasets that match the given criteria.

​​query-dimensions

​​butler query-dimensions <DIMXPR> [--with-dataset DST <INCOLXPR>] [--yaml] [--records] REPO [DIM...]
​​Query for the data IDs and/or dimension records that match the given criteria.

​​associate

​​butler associate [--calib BEGIN..END] [--from-file FILE] REPO COLLECTION <DSTQRY> <DIMXPR> <INCOLXPR>
​​Associate datasets with a TAGGED or CALIBRATION collection.  This includes certifying master calibration datasets.

​​disassociate

​​butler disassociate [--from-file FILE] REPO COLLECTION <DSTQRY> <DIMXPR>
​​Disassociate datasets from a TAGGED or CALIBRATION collection.

​​chain

​​butler chain REPO COLLECTION <INCOLXPR>​
​​Define or update a CHAINED collection.

​​export

​​butler export REPO ???​