Partitions

Design Decisions

  • How are partitions represented to solids?
  • config / environment dict as configured by user
  • context resource
  • How do we group runs for the same partition?
  • tags
  • How do we group runs across partitions for the same pipeline?
  • tags
  • Where do we need to specify the set of possible partitions?
  • pipeline definition vs schedule definition vs standalone
  • Should shoot for standalone‚Ķ should get the schedule definition right, and then work on hacking around the dagit UI
  • How do we specify execution of a partition?
  • tags?
  • presets?
  • Where do we do partition selection?
  • explicit selector on schedule definitions
  • presets

TODO

  • Figure out how to resolve config/environment dict based on partition
  • Look at presets?
  • Figure out execution API for partition
  • for now, using tags
  • Figure out if we should allow non-partition jobs for partitioned pipelines, where should the partition selection happen

Requirements

Partition requirements:
  • Support time-based partitions (variable-sized)
  • Support fixed partitions (ML-style)

Execution requirements:
  • Support execution of a partition through dagit
  • Support execution of a partition through a scheduler
  • Support execution of a batched set of partitions (backfill)

Run requirements:
  • Take in config to designate partition

UI requirements:
  • View job/run status by partition

Approaches

Pipeline partitions

Add partition definition function on a PipelineDefinition
  • Pros:
  • Natural for pipeline author to already know how to partition data
  • Cons:
  • Potentially overfitting by making Partition too prominent

ScheduleType

Put partition definition function on a ScheduleDefinition
  • Pros: