Add type stubs

[ruancomelli]

Stub files can be added to the project to provide static type information that can be used by tools like mypy.

I believe that the simplest way to add this functionality to funcy is to provide stub files along with their respective modules. That is, for example, a stub file colls.pyi corresponding to the contents of colls.py; a funcs.pyi corresponding to funcs.py and so on. Corresponding snippets from funcs.py and funcs.pyi would look like:

funcs.py

def identity(x):
    return x

def constantly(x):
    return lambda *a, **kw: x

def caller(*a, **kw):
    return lambda f: f(*a, **kw)

funcs.pyi

from typing import Any, Callable, TypeVar

T = TypeVar('T')

def identity(x: T) -> T: ...
def constantly(x: T) -> Callable[..., T]: ...
def caller(*a: Any, **kw: Any) -> Callable[[Callable[..., T]], T]: ...

Take a look at more-itertools to see how this looks like when fully implemented.

If you are interested in this functionality, I could perhaps submit a draft PR by the end of the week.

[Suor]

I am not a big fan of adding types to funcy.

[cardoso-neto]

Python's type hints can be lacking when it comes to higher-order functions.
However, it does make some things much, much easier.
For example, having decent code completion can only be attained with decent type hints.
It is very helpful. You should experiment with it, so you can feel its value for yourself.

[ruancomelli]

Besides code completion, as mentioned by @cardoso-neto, type hints help to ensure code correctness even before execution. Moreover, when programming in typed contexts, funcy does not integrate well and often demands some kind of reworking: I either type-annotate everything manually or ask mypy to skip certain lines or (what I am getting more inclined to do) write stubs for myself.

However, I'm pretty sure that you are already aware of the benefits of adding type annotations everywhere, and the cons of not having them. Now, what are the annoyances of adding them to funcy? I can think of:

1. possible compatibility issues: We have to make sure that we will not break older Python versions. There are many workarounds for this, one of them being including stub files (.pyi) together with the regular files (.py) for the sake of pleasing type-checkers. Those files are ignored at runtime, so no problems here. This is the approach followed by more-itertools, for instance.
2. increased workload for further developments: In addition to implementing the desired functionality, contributors have to think about typing as well. While I agree that it is indeed a bit more work, I believe this is acceptable.
3. possible mismatch between implementation and stubs: If a contributor changes the signature of a function in the implementation files, but forgets to update the stub files, there will be a mismatch. However, AFAIK type-checkers can be used here to make sure everything is still fine.
4. writing all of those stub files will take a lot of time right now: Yes, I strongly agree with you here. But then we can always work on a separate branch and only add it to main once we complete this task.

Is there any other reason why you are not a fan of adding types to funcy?

[Suor]

Other issues with funcy is optional first arguments and passing different stuff instead of callables, see extended function semantics.

It's some extra code that's not really a code, which I will need to support if it goes here. I don't use type annotations so I won't spot something going out of sync naturally. Not sure whether there are automatic ways to do that though.

Anyway I would prefer someone else, who actually care about this, support type annotations. Probably outside of funcy repo, i.e. in typeshed.

[cardoso-neto]

I believe the extended function semantics could be supported with @overload annotations.
Something along the lines of

from __future__ import annotations

from typing import Callable, Iterable, Mapping, TypeVar, overload

X = TypeVar("X")
Y = TypeVar("Y")


@overload
def funcy_map(f: None, iter_: Iterable[X]) -> Iterable[X]: ...

@overload
def funcy_map(f: Mapping[X, Y], iter_: Iterable[X]) -> Iterable[Y]: ...

@overload
def funcy_map(f: Callable[[X], Y], iter_: Iterable[X]) -> Iterable[Y]: ...


def funcy_map(f, iter_):
    if isinstance(f, type(None)):
        return iter_
    if isinstance(f, Mapping):
        return (f[x] for x in iter_)
    return (f(p) for p in iter_)

That being said, thanks for considering it and thank you for the awesome lib.
It would seem that a typeshed types-funcy package or maybe even a typed fork would indeed be the best course of action to avoid burdening you with even more code to maintain.

[Suor]

Can't one use unions instead of overload? Should be less verbose. Overloading might still be needed for optional first args.

[cardoso-neto]

Sometimes, yes. But in cases where the returned type is conditioned on the type of the input, overloading is a must. Otherwise invalid types would be inferred:

def funcy_map(
    f: Callable[[X], Y] | Mapping[X, Y] | None,
    iter_: Iterable[X],
) -> Iterable[X] | Iterable[Y]:
    ...

result = funcy_map({1:'d'}, [1, 2, 3])
for x in result:
    x  # int | str

Here all the types I had above overloaded are "unionized", but mypy has no idea if it returned identities or what and evaluates the type of x as the union.
So in this case, the verbosity is worth it.

[Suor]

It's not -> Iterable[X] | Iterable[Y] it's just -> Iterable[Y] for the most part. Only None is the issue.

[cardoso-neto]

There's also sets, which would return Iterable[bool]. And slices which would return Iterable[tuple[Y, ...]] if I'm not mistaken.

[ruancomelli]

Can't one use unions instead of overload? Should be less verbose.
It's not -> Iterable[X] | Iterable[Y] it's just -> Iterable[Y] for the most part. Only None is the issue.

@Suor perfect; that is why overloads are necessary here, but for sure things can be less verbose. Taking only Nones, Callables and Mappings into account, we can simplify @cardoso-neto's implementation a bit:

@overload
def funcy_map(f: None, iter_: Iterable[X]) -> Iterable[X]: ...

@overload
def funcy_map(
    f: Callable[[X], Y] | Mapping[X, Y],
    iter_: Iterable[X]
) -> Iterable[Y]: ...

Unfortunately, it can't get much simpler than that because there is no way to annotate the return value as being something like Iterable[X] if f is None else Iterable[Y]. Whenever the return type changes depending on the inputs, we have to add a new overload.

I wrote a draft implementation for the basic type stubs for the extended function semantics, you can find it in my gist repository and add comments/suggestions at will. Note that it is an untested draft written for Python 3.9+, an actual implementation should target older Python versions (and be tested, of course!). The main problem I see here is that there are indeed lots of overloads - and I've only written stubs for two functions! The bright side is that new functions will probably be added as copy-paste-tune from those basic versions. We could also modularize things by creating custom generic types, but I'll leave that as an exercise for the reader :)

Anyway I would prefer someone else, who actually cares about this, support type annotations. Probably outside of funcy repo, i.e. in typeshed.

That is (almost) exactly what I was thinking. I'm not sure about the best way to publish this outside of funcy - I need to do some research first - but typeshed seems to be the way to go indeed. I am someone else who actually cares about this, and I would gladly write and maintain such a thing if you accept it; I'm just very busy at the moment, but perhaps in the next few weeks something could be done. What are your thoughts regarding this?

[Suor]

A devoted maintainer is always better. I will link from README and docs once this is a thing.

Also it looks like a lot of repetition in stubs, multiplied by 2 if you'll want to distinguish iterator and list versions, i.e. -> List[X] and Iterator[X] for many seqs utils. May consider generating them.

[julianfortune]

@ruancomelli Have you had a chance to start a project for typehints? Thanks!

[ruancomelli]

@julianfortune I have implemented some type-stubs locally and was thinking of eventually uploading them to typeshed once I got a complete set of annotations. Unfortunately, I've been in a lack of free time in the past months, so I didn't manage to make this ready for publishing.

Funnily enough, I happened to tweak those annotations yesterday after all of those months. I'm willing to get back to this project in the next few days, but don't refrain from doing it yourself if you wish 😁

[julianfortune]

@ruancomelli I don't think it makes sense to duplicate work, but if you want to make a repo with your work in progress, I would be happy to help out!

[ruancomelli]

Hey @julianfortune! It seems that I will not have enough time for this in the next few months, so please feel free to take on this project 😁

[ahmed-moubtahij]

@julianfortune @ruancomelli Any of you can point me to the repo of the WIP on this? I could contribute here and there.

[ruancomelli]

Hi @Ayenem! After your message, I finally took the time to clean up my draft type-annotations and assemble them in a repository: https://github.com/ruancomelli/funcy-stubs.

Feel free to contribute, I would be glad to review and approve PRs!

@julianfortune you may be interested in taking a look at it as well.

Note that a good portion of it is still a draft, unsuitable for release. You can see the modules where I deliberately skipped some functions: I defined a __getattr__ function in them. In addition, many types can be further narrowed to provide a better type inference for the end users.

Coming soon is a better development setup: I'm planning on writing stubtests and setting up GitHub workflows.

[ahmed-moubtahij]

Perfect :) Thanks @ruancomelli I'll give it a look

[bersbersbers]

I am not a big fan of adding types to funcy.

I believe they should be added where possible, as adding funcy to typed code will break existing type checks. One example:

import numpy as np
from funcy import retry

@retry(3)
def load_data() -> np.typing.NDArray:
    return np.array([0])

# initialize
data: np.typing.NDArray | None = None

# do some work
data = load_data()
np.pad(data, 1)

mypy says

Argument 1 to "pad" has incompatible type "Optional[ndarray[Any, dtype[Any]]]"

The example works perfectly when commenting out @retry.

I believe typing the decorator aspect of retry might be as easy as

from typing import Callable, TypeVar
F = TypeVar("F", bound=Callable)
retry: Callable[..., Callable[[F], F]]

(but I may be missing something).

[Suor]

Did you try funcy-stubs linked above?

[bersbersbers]

Did you try funcy-stubs linked above?

Only briefly, to be honest. It forces funcy==1.17,and even using that (pip install funcy-stubs), I still get

error: Call to untyped function "fallback" in typed context [no-untyped-call]

or

error: Untyped decorator makes function "load_data" untyped [misc]

[bersbersbers]

A related problem that I have is that I cannot seem to type-annotate the imports myself, see python/mypy#15170

[ruancomelli]

Hello, @bersbersbers!
Thanks for the feedback, I'll look into both issues and see if I can solve them in funcy-stubs. Also feel free to open any other typing-related issues in that repository.
Note that the type stubs in funcy-stubs is still incomplete, so a few issues are expected. If you have already correctly type-annotated a funcy function, please open a pull request in funcy-stubs 😉