Fix false alarm in E203 rule (issue #373, black incompatibility)

[DesbyP]

Issue #373

Rule E203 checks for extraneous whitespace before a colon.
Yet there is a corner case when the colon is used as a "slice" operator. PEP8 says:

However, in a slice the colon acts like a binary operator, and should have equal amounts on either side (treating it as the operator with the lowest priority). In an extended slice, both colons must have the same amount of spacing applied.

This behaviour isn't implemented and leads to conflict between pycodestyle and black.

Solution

This PR adds to pycodestyle the tolerance for the slice situation, without enforcing the "same amount of space on both sides of the colon" rule.
This way the current Python that code was pycodestyle compliant stays compliant.

Performance

On a Python code base of over 15k lines, a test with hyperfine shows no performance loss.

[asottile]

does this handle: x[lambda : foo]

[DesbyP]

@asottile good point.
I just added support for this case + tests.

[asottile]

does this handle: [lambda x={tuple([1]):2}: x]

[DesbyP]

@asottile yup

[asottile]

?

$ git rev-parse HEAD
c88d21ecce0dd0a9d77b7bd2c88024809f8b6f13
$ python3 -m pycodestyle t.py
$ echo $?
0

I expect the inner : to be an error

[DesbyP]

the PR modifies only the portion of code that treats extra whitespaces, it shouldn't change current behaviour about missing whitespaces.

This seems to be a different issue in pycodestyle, indeed the current version on branch master doesn't detect anything in this case either :

$ git checkout master
$ python3 -m pycodestyle t.py
$ echo $?
0

[DesbyP]

Hi
is it ok to merge ?

[DesbyP]

@asottile is it ok for you ?

[asottile]

sorry I haven't had a chance to look at the updated code

[DesbyP]

Hi @asottile, I hope you're ok in these troubled times.
It would be great if you could take a look at the code, this issue is a blocker for flake8 and black adoption in my company.
Thanks

[asottile]

I don't see how this is a blocker, you can use the suggested flake8 configuration from black

this change is quite complicated (adds a lot of edges and magic numbers) and I have not had time to look at every fine detail of it

[DesbyP]

I understand that this is not trivial to deploy.

We have a lot of projects dispersed in many teams, I can't promote a seamless integration of black if I ask to tweak flake8 configuration on every one of them.

[asottile]

We have a lot of projects dispersed in many teams, I can't promote a seamless integration of black if I ask to tweak flake8 configuration on every one of them.

why not? you'll have to modify all of the code in the repository and add some configuration for black already

[DesbyP]

not necessarily, black can be used out of the box without config
the goal is to integrate it as a background formatter in PyCharm

[pranasziaukas]

In my opinion, the fact of how the current E203 rule deals with slices is an unintended consequence.
Even worse, it makes PEP8-compliant code raise warnings, so it's not just about black.

So I agree that the fix is needed.

Edit: to illustrate, our pipeline fails at

for combo in combos[i : i + batch_size]:

which is very obviously PEP8-compliant, even listed as a correct usage example here.

[kislyuk]

It seems there is consensus that E203 is not a PEP8 style compliance error and should be disabled by default. The presence of the rule for E203 in the default configuration prevents flake8 and black, among other tools, from interoperating smoothly. Could the maintainer please consider merging this PR?

[hoylemd]

Looks good to me! I had one very minor suggestion re: formatting of multi-line conditions, but it's not a big deal at all.

I'm also not quite sure it's for me to approve the PR, so I'm commenting instead of Approving so I don't step on anyone's toes.

[hoylemd]

Suggested change

	elif (line[found - 1] != ',' and
	not (char == ':' and is_a_slice(line, found))):
	elif (line[found - 1] != ','
	and not (char == ':' and is_a_slice(line, found))):

I was under the impression that 'pythonic' style puts the binary operator after a line break, not before.

--- Personal preference / opinion only below this line. Disregard at your pleasure ---

I personally also prefer to format multi-line conditions like this:

        elif (
            line[found - 1] != ','
            and not (char == ':' and is_a_slice(line, found))
        ):

because it puts co-ordinate conditions in a visually co-ordinate structure, and also has dedicated lines to mark the bounds of the compound condition. (i.e. the elif ( and ): lines, visually positioned to encapsulate the conditions and indicate that they're subordinate to the elif construct itself. It does take up a lot more vertical space though, so I can understand why some might dislike that.

[s-weigand]

Guess this is the right pep for this:
https://www.python.org/dev/peps/pep-0008/#should-a-line-break-before-or-after-a-binary-operator

[DesbyP]

@hoylemd I agree and prefer this coding style as well.
Yet W503 in TravisCI prevents me of doing so :

pycodestyle.py:489:13: W503 line break before binary operator
pycodestyle.py:512:13: W503 line break before binary operator

If @s-weigand suggestion to ignore W503 was accepted then we could proceed :)

[FichteFoll]

I was under the impression that 'pythonic' style puts the binary operator after a line break, not before.

This was changed only a few years back (roughly when W504 was introduced and both were added to the default ignore list). pycodestyle itself does not use this style of line breaks before binary operators and you should always adhere to the project's style.

[DesbyP]

thank you for the precision
Then I guess the PR is ok for review

[s-weigand]

Since W503 (which lets the travis tests fail) was added to ignore default:

pycodestyle/CHANGES.txt

Line 185 in 203041b

* Added W503 to the list of codes ignored by default ignore list; #498

IMHO it should also be added to the tests config

pycodestyle/setup.cfg

Line 9 in 6a56d22

ignore = E226,E24,W504

Or maybe just use the default ignore.

[marctorsoc]

What's the status of this?

[DesbyP]

Hi @asottile,
The activity on this PR shows that a lot of people are waiting for this fix and are forced to use the workaround. Would you consider reviewing my commits ?
Thanks