Rust 2018 All Hands — Custom Test Frameworks
Index page: +Rust All Hands Berlin

  • Idea: Ability to have your own test/bench runner which are more fancy/special than the built-in ones
  • tests, benches
  • specify crate is a test/bench framework in Cargo.toml
  • dependent of these crates should just get to use the new stuff
  • cargo handles the loading
  • no lib for output handling, will probably write a helper crate and put it in the nursery
  • probably also write a helper crate for writing test frameworks
  • examples for custom test frameworks
  • stable #[bench]
  • quickcheck
  • compiletest
  • In general: on the roadmap but not in a rush to finish for edition release

Procedural Macro Design

  • concrete mechanism on how the test framework handles the generation of test code is not decided, options:
  1. proc macro annotations on functions (#[foo] fn bar() {}) + registry to generate a main function (“main() function generation with test collector” in the RFC)
  • can you register foo attr for a test framework and then have users do #![foo] in their code? → Manish: No, RFC says attr can only be applied to a function
  • generated main() code can only access public code.
  • just make every item public? → probably fine.
  • restrictions? API gives token streams only
  • helper crate?
  1. proc macro #![foo] processes all the code and outputs a new crate basically (“Whole-crate procedural macro” in the RFC)
  1. proc macro + cargo config, cargo tells rustc to apply proc macro to whole crate (“Alternative procedural macro with minimal compiler changes” in the RFC)
  • currently “whole crate proc macro” only kinda exists
  • addition to rustc
  • super dangerous powerful
  • doc tests?
  • cargo has 3 notions of targets (tests, examples, doc tests)
  • not now, possible in the future
  • impl?
  • cargo part depends on rustc one
  • matklad volunteered for cargo part
  • might be good for mentoring issues
  • → detailed impl plan

Cargo integration

  • cargo {test, bench}
  • runtime deps
  • extern crate test currently works because compiler built-in
  • custom test framework needs to be a dev-dependency
  • everyone will need a runtime crate
  • same problem as proc-macro users have (need to include both serde, serde_derive)

Common Output Format/API

Previously, in “Machine Readable Test Output”