Argument Clinic's [python input] blocks do not support empty lines in comments

[picnixz]

Bug report

Bug description:

The following is not a valid Python input clinic but should be:

/*[python input]
# Line 1
#
# Line 3
[python start generated code]*/

This results in an assertion error:

$ python ./Tools/clinic/clinic.py --force --make --exclude Lib/test/clinic.test.c --srcdir .
Traceback (most recent call last):
  File "/lib/python/cpython/./Tools/clinic/clinic.py", line 11, in <module>
    main()
  File "/lib/python/cpython/Tools/clinic/libclinic/cli.py", line 226, in main
    run_clinic(parser, args)
  File "/lib/python/cpython/Tools/clinic/libclinic/cli.py", line 205, in run_clinic
    parse_file(path,
  File "/lib/python/cpython/Tools/clinic/libclinic/cli.py", line 84, in parse_file
    cooked = clinic.parse(raw)
             ^^^^^^^^^^^^^^^^^
  File "/lib/python/cpython/Tools/clinic/libclinic/app.py", line 186, in parse
    for block in self.block_parser:
                 ^^^^^^^^^^^^^^^^^
  File "/lib/python/cpython/Tools/clinic/libclinic/block_parser.py", line 132, in __next__
    return_value = self.parse_clinic_block(self.dsl_name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/lib/python/cpython/Tools/clinic/libclinic/block_parser.py", line 194, in parse_clinic_block
    line = self._line()
           ^^^^^^^^^^^^
  File "/lib/python/cpython/Tools/clinic/libclinic/block_parser.py", line 155, in _line
    self.language.parse_line(line)
  File "/lib/python/cpython/Tools/clinic/libclinic/clanguage.py", line 70, in parse_line
    self.cpp.writeline(line)
  File "/lib/python/cpython/Tools/clinic/libclinic/cpp.py", line 139, in writeline
    assert line
           ^^^^
AssertionError
make: *** [Makefile:1055: clinic] Error 1

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-128152: Update Argument Clinic to support empty lines #128158

[picnixz]

I've had a quick look at the failure and this happens before we've detected the [python input] block. Similarly, #128153 faces the same issue, namely Clinic assumes that we're still processing a C file and doesn't check for the desired DSL.

@erlend-aasland I don't really know how to fix this and it's not really an issue since I can just write some fake characters to simulate an empty line (e.g., # ::) but I'd like to know if first detecting the DSL before attempting to parse it would be a good alternative or if the current design of AC is just too intricate for this feature.

[StanFromIreland]

I've opened a PR for this issue. I think the problem was that empty lines were breaking the logic so if they don't reach it there should be no more error.

[picnixz]

As I said on the PR, I'm not sure this is the best approach. The "empty lines" that are detected at this stage are actually lines of the form <space>#</space>, so it's not entirely clear whether they should be ignored at this point. The reason is that we first parse a file before deciding whether this new line is part of a python input block. If this is the case, we should switch to another parsing method (for instance, I haven't actually checked what happens if I write #if defined(...) inside the Python input clinic block).