Give this doc a home

Keep your docs nice and tidy by moving this doc to a folder.

Mypy strict-optional issues

This doc is shared publicly — see https://gist.github.com/gnprice/6d0413f022ca2a17d3d2bb1d4b96fbae .

Catalog of issues

In mypy as type-checker

Can’t easily work around

~~None / Generic / subclassing #1956~~

Callable rejected at own type #1957

if/else becomes object #1820

yield value expected #1958

Error-message quality

cryptic message on attribute use

really need to print whole types -- tuple edition

aspects of “Callable rejected at own type”

Can easily work around (including ugly workarounds, if small and few)

Tuple and empty container

conditional type in comprehension #1734

type update due to assignment

subtype on Callable neglecting optionality of args

list slices should be effectively covariant

Unsoundness

Attributes declared on class allowed to get away with None — this seems like it could be a real problem for users when relying on the types to understand something.

In mypy as type-checked codebase, but where the code is fine and it might be nice to support it

handing over a `Dict`, wishing it were covariant in value type

inferring an Optional local

conditional type binder doesn’t apply to dict items

In stubs

subprocess.Popen

In mypy as type-checked codebase

… 334 error messages …

Counts of error messages

482 total (including a small handful I duplicated for multiple issues)

334 in mypy as type-checked codebase

146 in mypy as type-checker

109 in None / Generic / subclassing

16 in Callable rejected at own type

13 in if/else becomes object

8 in other 8 issues

2 in stubs

Issues in type-checker

Pseudo-instance attributes declared on class

We have this really common pattern:

class C:

x = None # type: str

def __init__(self, x: str = None) -> None:

self.x = x

As written, this gets an error under --strict-optional at the self.x = x assignment. As it should.

But that’s not the only type error here. It’d be just as bad if it looked like this:

class C:

x = None # type: str

def __init__(self) -> None:

pass

def run(self) -> None:

print('x: ' + self.x)

And we don’t give an error there!

NodeVisitor — None / Generic / subclassing

E.g., Return type of "visit_mypy_file" incompatible with supertype "NodeVisitor"

We have

class NodeVisitor(Generic[T]):

...

def visit_mypy_file(self, o: 'mypy.nodes.MypyFile') -> T:

pass

...

class TraverserVisitor(NodeVisitor[None]):

...

def visit_mypy_file(self, o: MypyFile) -> None:

for d in o.defs:

d.accept(self)

Reproduces like so:

$ cat /tmp/foo.py

from typing import Generic, TypeVar

T = TypeVar('T')

class Base(Generic[T]):

def f(self) -> T:

pass

class SubNone(Base[None]):

def f(self) -> None:

pass

class SubInt(Base[int]):

def f(self) -> int:

return 1

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py: note: In class "SubNone":

/tmp/foo.py:10: error: Return type of "f" incompatible with supertype "Base"

parse_signature — Tuple and empty container

mypy/stubutil.py: note: In function "parse_signature":

mypy/stubutil.py:21: error: Incompatible return value type (got tuple(length 3), expected "Optional[Tuple[str, List[str], List[str]]]")

def parse_signature(sig: str) -> Optional[Tuple[str,

List[str],

List[str]]]:

...

if not arg_string.strip():

return (name, [], [])

Reproduces like so:

$ cat /tmp/foo.py

from typing import Optional, Tuple, List

def f1() -> Optional[Tuple[List[str]]]:

return ([],)

def f2() -> Optional[Tuple[List[str]]]:

return ([''],)

def f3() -> Optional[List[str]]:

return []

$ mypy /tmp/foo.py

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py: note: In function "f1":

/tmp/foo.py:4: error: Incompatible return value type (got "Tuple[List[object]]", expected "Optional[Tuple[List[str]]]")

So both the tuple and the empty container appear to be essential.

Lexer and many others — Callable rejected at own type

mypy/lex.py: note: In member "__init__" of class "Lexer":

mypy/lex.py:328: error: Incompatible types in assignment (expression has type Callable[[], None], target has type Callable[[], None])

Reproduces like so:

$ cat /tmp/foo.py

from typing import Callable

def f() -> None:

...

x = f # type: Callable[[], None]

reveal_type(f)

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:6: error: Incompatible types in assignment (expression has type Callable[[], None], variable has type Callable[[], None])

/tmp/foo.py:7: error: Revealed type is 'def ()'

$ mypy /tmp/foo.py

/tmp/foo.py:7: error: Revealed type is 'def ()'

At least two or three issues here:

That reveal_type output is goofy.

The types should match just fine.

Given that we’re giving an error saying they don’t match, we should never print things that actually look the same.

Also, what I suspect is the same issue (as #2 just above):

mypy/lex.py: note: In member "lex" of class "Lexer":

mypy/lex.py:378: error: Argument 2 to "get" of "dict" has incompatible type Callable[[], None]; expected "Optional[Callable[[], None]]"

and reproducer:

$ cat /tmp/foo.py

from typing import Callable

def f() -> None:

...

d1 = {} # type: Dict[int, Callable[[], None]]

d1.get(1, lambda: None)

d1.get(1, f)

d2 = {} # type: Dict[int, str]

d2.get(1, 'a')

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:8: error: Argument 2 to "get" of "dict" has incompatible type Callable[[], None]; expected "Optional[Callable[[], None]]"

So it works fine with things that aren’t Callable .

Many other places have similar issues.

One variation with a bit of a twist, where it seems to be foiling conditional type binding:

mypy/checkexpr.py: note: In member "check_argument_types" of class "ExpressionChecker":

mypy/checkexpr.py:658: error: Incompatible types in assignment (expression has type union type (2 items), variable has type "Optional[Callable[[Type, Type, int, Type, int, int, CallableType, Context, MessageBuilder], None]]")

mypy/checkexpr.py:677: error: None not callable

mypy/checkexpr.py:691: error: None not callable

...

check_arg: ArgChecker = None) -> None:

...

check_arg = check_arg or self.check_arg

...

check_arg(actual_type, arg_type, arg_kinds[actual],

callee.arg_types[i],

actual + 1, i + 1, callee, context, messages)

deserialize — if/else becomes object and fails to match Optional[Thing]

mypy/nodes.py: note: In member "deserialize" of class "Argument":

mypy/nodes.py:402: error: Argument 2 to "Argument" has incompatible type "object"; expected "Optional[Type]"

return Argument(Var.deserialize(data['variable']),

(None if data.get('type_annotation') is None

else mypy.types.Type.deserialize(data['type_annotation'])),

Reproducer:

$ cat /tmp/foo.py

from typing import Optional

x = None if 3 else 3 # type: Optional[int]

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:3: error: Incompatible types in assignment (expression has type "object", variable has type "Optional[int]")

My guess is we’re inappropriately doing what we call a join when we should really be inferring a Union .

A variation:

mypy/types.py: note: In member "deserialize" of class "CallableType":

mypy/types.py:656: error: List comprehension has incompatible type List[object]

@classmethod

def deserialize(cls, data: JsonDict) -> 'CallableType':

assert data['.class'] == 'CallableType'

# TODO: Set definition to the containing SymbolNode?

return CallableType([(None if t is None else Type.deserialize(t))

for t in data['arg_types']],

cryptic error message on attribute use

The error here is absolutely right, but the message is less helpful than our preferred standard:

mypy/nodes.py: note: In member "serialize" of class "SymbolTableNode":

mypy/nodes.py:2086: error: Some element of union has no attribute "fullname"

data['cross_ref'] = self.node.fullname()

The issue is that self.node has type Optional[SymbolNode] . We should at a minimum print that type.

Short repro:

$ cat /tmp/foo.py

from typing import Optional

x = 'abc' # type: Optional[str]

x.index('b')

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:4: error: Some element of union has no attribute "index"

any_constraints — lacking conditional type binder in comprehension

This is https://github.com/python/mypy/issues/1734.

mypy/constraints.py: note: In function "any_constraints":

mypy/constraints.py:187: error: Incompatible return value type (got "Optional[List[Constraint]]", expected List[Constraint])

def any_constraints(options: List[Optional[List[Constraint]]]) -> List[Constraint]:

...

valid_options = [option for option in options if option is not None]

if len(valid_options) == 1:

return valid_options[0]

...

$ cat /tmp/foo.py

from typing import Optional, List

l = [1, None] # type: List[Optional[int]]

ll = [x for x in l if x is not None] # type: List[int]

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:4: error: List comprehension has incompatible type List[Optional[int]]

Also interesting because of course we could be trying to pass somewhere that does want the Optional, and that should work too. Effectively a comprehension or a list literal needs to behave covariantly, even though in general List is invariant. Currently we implement that sort of logic with the context concept.

as_block — type update due to assignment

Looks a bit like #1825 but I think that’s actually a totally unrelated issue.

Guess we need to track updates inside a conditional block?

mypy/fastparse.py: note: In member "as_block" of class "ASTConverter":

mypy/fastparse.py:185: error: Some element of union has no attribute "set_line"

def as_block(self, stmts: List[ast35.stmt], lineno: int) -> Block:

b = None

if stmts:

b = Block(self.fix_function_overloads(self.visit_list(stmts)))

b.set_line(lineno)

return b

Short reproducer:

$ cat /tmp/foo.py

x = None

if 32:

x = 'abc'

x + x

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:4: error: Unsupported left operand type for + (some union)

Workaround:

$ cat /tmp/foo.py

x = None

if 32:

x = 'abc'

if x is not None:

x + x

$ mypy /tmp/foo.py --strict-optional

Really this calls for something like a dataflow analysis.

The same issue arises without an if :

$ cat /tmp/foo.py

x = None

x = 'abc'

x + x

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:3: error: Unsupported left operand type for + (some union)

though it’s less clear why one would want to write the code that way, so it’s not a problem to fail it.

really need to print whole types — tuple edition

mypy/parse.py: note: In member "parse_function_header" of class "Parser":

mypy/parse.py:550: error: Incompatible return value type (got tuple(length 5), expected tuple(length 5))

return (name, [], None, True, [])

subtype on Callable neglecting optionality of args

mypy/semanal.py: note: In member "anal_type" of class "SemanticAnalyzer":

mypy/semanal.py:1048: error: Argument 3 to "TypeAnalyser" has incompatible type Callable[[str, Context, bool, bool], None]; expected Callable[[str, Context], None]

a = TypeAnalyser(self.lookup_qualified,

self.lookup_fully_qualified,

self.fail)

...

def fail(self, msg: str, ctx: Context, serious: bool = False, *,

blocker: bool = False) -> None:

Short reproducer:

$ cat /tmp/foo.py

from typing import Callable

def f(x: str, y: bool = False, *, z: bool = False) -> None:

...

f('a')

ff = f # type: Callable[[str], None]

$ mypy /tmp/foo.py

$ mypy /tmp/foo.py --strict-optional

/tmp/foo.py:7: error: Incompatible types in assignment (expression has type Callable[[str, bool, bool], None], variable has type Callable[[str], None])

list slices should be effectively covariant

mypy/semanal.py: note: In member "process_typevar_declaration" of class "SemanticAnalyzer":

mypy/semanal.py:1314: error: Argument 2 to "process_typevar_parameters" of "SemanticAnalyzer" has incompatible type List[str]; expected List[Optional[str]]

res = self.process_typevar_parameters(call.args[1 + n_values:],

call.arg_names[1 + n_values:],

This is perfectly legitimate code. It’s possible to work around the issue, but a pain and makes the code worse. Basically this slice is very much like a list literal or comprehension — it’s something we know by construction doesn’t share mutability with anything else, so it’s provably just fine to treat the list covariantly.

wrap_context — yield value expected

mypy/build.py: note: In member "wrap_context" of class "State":

mypy/build.py:1170: error: Yield value expected

@contextlib.contextmanager

def wrap_context(self) -> Iterator[None]:

save_import_context = self.manager.errors.import_context()

self.manager.errors.set_import_context(self.import_context)

try:

yield

except CompileError:

raise

except Exception as err:

report_internal_error(err, self.path, 0)

self.manager.errors.set_import_context(save_import_context)

self.check_blockers()

Issues in mypy as annotated codebase

…

parse_docstring — handing over a Dict, wishing it were covariant in value type

mypy/docstring.py: note: In function "parse_docstring":

mypy/docstring.py:189: error: Incompatible types in assignment (expression has type Dict[str, str], variable has type Dict[str, Optional[str]])

waiter — inferring an Optional local

mypy/waiter.py: note: In member "_wait_next" of class "Waiter":

mypy/waiter.py:209: error: Incompatible types in assignment (expression has type None, variable has type "str")

if rc != 0:

if name not in self.xfail:

fail_type = 'FAILURE'

else:

fail_type = 'XFAIL'

else:

if name not in self.xfail:

fail_type = None

else:

fail_type = 'UPASS'

infer_constraints_for_callable — conditional type binder doesn’t apply to dict items

Perhaps actually for the best, because something else could mutate in between.

mypy/constraints.py: note: In function "infer_constraints_for_callable":

mypy/constraints.py:57: error: Argument 1 to "get_actual_type" has incompatible type "Optional[Type]"; expected "Type"

if arg_types[actual] is None:

continue

actual_type = get_actual_type(arg_types[actual], arg_kinds[actual],

tuple_counter)

Issues in typeshed stubs

mypy/waiter.py: note: In member "start" of class "LazySubprocess":

mypy/waiter.py:36: error: Argument 2 to "Popen" has incompatible type "Optional[str]"; expected "str"

mypy/waiter.py:36: error: Argument 3 to "Popen" has incompatible type "Optional[Dict[str, str]]"; expected Mapping[str, str]

self.process = Popen(self.args, cwd=self.cwd, env=self.env,

stdout=self.outfile, stderr=STDOUT)

Data

Ran mypy -p mypy --strict-optional , at commit 67d89b9e7 (master on the afternoon of 2016–07-27.)

See https://gist.github.com/gnprice/6d0413f022ca2a17d3d2bb1d4b96fbae#file-output.

David Fisher 9 years ago

This is probably a confluence of PartialTypes and context. Yikes.

Mypy strict-optional issues
Catalog of issues
Issues in type-checker
Pseudo-instance attributes declared on class
NodeVisitor — None / Generic / subclassing
parse_signature — Tuple and empty container
Lexer and many others — Callable rejected at own type
deserialize — if/else becomes object and fails to match Optional[Thing]
cryptic error message on attribute use
any_constraints — lacking conditional type binder in comprehension
as_block — type update due to assignment
really need to print whole types — tuple edition
subtype on Callable neglecting optionality of args
list slices should be effectively covariant
wrap_context — yield value expected
Issues in mypy as annotated codebase
parse_docstring — handing over a Dict, wishing it were covariant in value type
waiter — inferring an Optional local
infer_constraints_for_callable — conditional type binder doesn’t apply to dict items
Issues in typeshed stubs
Data

Give this doc a home

​​Catalog of issues

​​Issues in type-checker

​​Pseudo-instance attributes declared on class

​​NodeVisitor — None / Generic / subclassing

​​parse_signature — Tuple and empty container

​​Lexer and many others — Callable rejected at own type

​​deserialize — if/else becomes object​ and fails to match Optional[Thing]​

​​cryptic error message on attribute use

​​any_constraints — lacking conditional type binder in comprehension

​​as_block — type update due to assignment

​​really need to print whole types — tuple edition

​​subtype on Callable neglecting optionality of args

​​list slices should be effectively covariant

​​wrap_context — yield value expected

​​Issues in mypy as annotated codebase

​​parse_docstring — handing over a Dict, wishing it were covariant in value type

​​waiter — inferring an Optional local

​​infer_constraints_for_callable — conditional type binder doesn’t apply to dict items

​​Issues in typeshed stubs

​​Data

Catalog of issues

Issues in type-checker

Pseudo-instance attributes declared on class

NodeVisitor — None / Generic / subclassing

parse_signature — Tuple and empty container

Lexer and many others — Callable rejected at own type

deserialize — if/else becomes object and fails to match Optional[Thing]

cryptic error message on attribute use

any_constraints — lacking conditional type binder in comprehension

as_block — type update due to assignment

really need to print whole types — tuple edition

subtype on Callable neglecting optionality of args

list slices should be effectively covariant

wrap_context — yield value expected

Issues in mypy as annotated codebase

parse_docstring — handing over a Dict, wishing it were covariant in value type

waiter — inferring an Optional local

infer_constraints_for_callable — conditional type binder doesn’t apply to dict items

Issues in typeshed stubs

Data