2.20 where are we?

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 (was: 2.20 where are we?)

Matthew Peveler
You are right, that was added in my original debug effort. Attached is the
second patch without the added print. It looks like the files are getting
stored as `.bin` files, though are just plain text when you open them. If
someone could explain to me the proper way to attach patches here (or maybe
it's just gmail being funny?), I would appreciate it.

I'll have another email later on with patches for having this branch run
under both python2.7 & 3 as the necessary backport efforts were not really
all that extravagant with just a handful of shims around the changes you
noted in long vs int, unicode vs str, StringIO vs io, iter.next vs
iter.__next__, reload, xrange vs range. The primary thing missing is making
the stepmake/stepmake/python-module-*.make files work for the range of
versions, but I'm afraid that is out of my wheel house of ability.

Matt

On Thu, Sep 26, 2019 at 6:03 AM Jonas Hahnfeld <[hidden email]> wrote:

> Am Montag, den 23.09.2019, 12:26 -0300 schrieb Matthew Peveler:
> > On Sat, Sep 21, 2019 at 5:52 AM Jonas Hahnfeld via lilypond-devel <
> [hidden email]> wrote:
> > > On top of that I've worked on the attached patches which brings the
> make targets check / test and doc back to life with Python 3.7.4.
> >
> > In applying your patches and running "make check", I encountered a
> couple of errors in scripts/build/output-distance.py, which would be summed
> up as:
> >
> > 1. Python 2 old style classes had a default __cmp__ which would compare
> the id() of the classes, and used implicitly for sort. Python 3 removed
> this, but need to define a __lt__ method for sort compat.
> > 2. attempting to open midi files as str files, though they are filled
> with binary data (and python/midi.py seems to expect binary in interacting
> with the contents).
> >
> > Please see the attached for small patches for these two things, and got
> "make check" to run for me using Python 3.7.4 as well.
> >
> > Regards,
> > Matt
>
> Ha, I already wondered why there are two targets check and test that
> both ran into the same issues (when I started). Turns out, the target
> "check" runs "test" but not the other way around, and I only verified
> that "test" works in the end...
>
> The patches make sense to me and also work on my system. You should
> probably remove the print() statement in the second patch, though.
>
> Thanks,
> Jonas
>

_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel

0002-Fix-py3-compat-issues-in-output_distance.py.patch (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 (was: 2.20 where are we?)

Dan Eble
On Sep 27, 2019, at 16:34, Matthew Peveler <[hidden email]> wrote:
>
> I'll have another email later on with patches for having this branch run
> under both python2.7 & 3 as the necessary backport efforts were not really
> all that extravagant with just a handful of shims around the changes you
> noted in long vs int, unicode vs str, StringIO vs io, iter.next vs
> iter.__next__, reload, xrange vs range.

Are these complications desirable?  A clean and obvious implementation requiring Python 3 will be easier to maintain.

Dan


_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3

David Kastrup
Dan Eble <[hidden email]> writes:

> On Sep 27, 2019, at 16:34, Matthew Peveler <[hidden email]> wrote:
>>
>> I'll have another email later on with patches for having this branch run
>> under both python2.7 & 3 as the necessary backport efforts were not really
>> all that extravagant with just a handful of shims around the changes you
>> noted in long vs int, unicode vs str, StringIO vs io, iter.next vs
>> iter.__next__, reload, xrange vs range.
>
> Are these complications desirable?  A clean and obvious implementation
> requiring Python 3 will be easier to maintain.

And that's what we are going to end up with ultimately.  But I don't see
that a clean cut from Python2 to Python3 in a single step is easily
feasible and testable for all targets at once including all GUB targets.

If we are talking about "just a handful of shims", that certainly sounds
like causing the least trouble.  The plan would not be to maintain them
for extended amounts of time but rather to help getting the Python3
transition over with with the least amount of pain.

--
David Kastrup

_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3

Dan Eble
On Sep 30, 2019, at 14:22, David Kastrup <[hidden email]> wrote:
>
> But I don't see
> that a clean cut from Python2 to Python3 in a single step is easily
> feasible and testable for all targets at once including all GUB targets.

OK.

On Sep 27, 2019, at 16:34, Matthew Peveler <[hidden email]> wrote:
> The primary thing missing is making
> the stepmake/stepmake/python-module-*.make files work for the range of
> versions, but I'm afraid that is out of my wheel house of ability.

Matt, I might be able to help with this, but can you describe more specifically what is lacking?  When your py changes are done, will the current makefiles continue to work in environments with Python 2, but require updates to work in environments with Python 3?

Regards,

Dan


_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 (was: 2.20 where are we?)

Joram Berger
In reply to this post by Matthew Peveler
Am 27.09.19 um 22:34 schrieb Matthew Peveler:
> long vs int, unicode vs str, StringIO vs io, iter.next vs
> iter.__next__, reload, xrange vs range.

It is very well feasible to support both version. I hope by shims you
mean something like¹

    from __future__ import division, print_function
    from builtins import range

because there is no need to customly write those. I consider the link
quite helpful.

Cheers


¹ https://python-future.org/compatible_idioms.html

_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3

Matthew Peveler
In reply to this post by Dan Eble
On Sep 30, 2019, at 14:22, David Kastrup <[hidden email]> wrote:
>
> But I don't see
> that a clean cut from Python2 to Python3 in a single step is easily
> feasible and testable for all targets at once including all GUB targets.

Yeah, I did this for my own edification, and with the thought that it might
make it (based on my prior experience with porting some python2 libraries)
that going at a step process with a window of maintaining both as it allows
for upstreaming the python3 stuff onto master as the same time as other
work goes on, and there isn't a problem of diverging branches.

This does assume however that GUB is updated to at least target at least
Python 2.6 (for __future__.print_function).

On Mon, Sep 30, 2019 at 4:12 PM Dan Eble <[hidden email]> wrote:

> Matt, I might be able to help with this, but can you describe more
> specifically what is lacking?  When your py changes are done, will the
> current makefiles continue to work in environments with Python 2, but
> require updates to work in environments with Python 3?
>
> Regards,
> —
> Dan

Please see attached for the patches, which given the work done prior by
Jonas, allows for running the various make targets under both python2&3
(assuming application of my other two patches for py3), though my principle
test was that the targets finished without error, as I am still not as well
acquainted with the build process to fully verify build results.

The python2 explicit stuff is in the final patch, with at least the shebang
stuff being applicable for being replaced with #!@TARGET_PYTHON@ like all
the other python files (which I plan to do in the next day or so). The
stepmake stuff though is were I have no idea how to handle as I am not sure
exactly what its purpose/doing is.

_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel

0001-explicitly-use-new-style-classes.patch (28K) Download Attachment
0002-add-__future__.print_function-for-files-that-print.patch (20K) Download Attachment
0003-add-shim-for-py2-py3-compat-for-reload.patch (1K) Download Attachment
0004-shim-for-py2-py3-compat-for-xrange.patch (2K) Download Attachment
0005-add-back-in-explicit-unicode-handling.patch (13K) Download Attachment
0006-handle-py2-py3-compat-for-iter-next.patch (3K) Download Attachment
0007-add-long-py2-py3-compat.patch (7K) Download Attachment
0008-add-shim-for-raw_print-for-py2-py3-compat.patch (3K) Download Attachment
0010-python2-specific-commits.patch (3K) Download Attachment
0009-add-StringIO-shim-for-py2-py3-compat.patch (984 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Python 3 (was: 2.20 where are we?)

Matthew Peveler
In reply to this post by Joram Berger
On Mon, Sep 30, 2019 at 5:46 PM Joram <[hidden email]> wrote:

> Am 27.09.19 um 22:34 schrieb Matthew Peveler:
> > long vs int, unicode vs str, StringIO vs io, iter.next vs
> > iter.__next__, reload, xrange vs range.
>
> It is very well feasible to support both version. I hope by shims you
> mean something like¹
>
>     from __future__ import division, print_function
>     from builtins import range
>
> because there is no need to customly write those. I consider the link
> quite helpful.
>
> Cheers
>
>
> ¹ https://python-future.org/compatible_idioms.html


Yes for using __future__, but no for builtins  while it would make things
easier, it is not something that is actually built into python and would be
installable for PyPI via the future package (
https://pypi.org/project/future/). So I had to hand define the shims,
though I mostly went for

python2 -> python3 shim such:
if sys.version_info > (3,):
    xrange = range
    raw_input = input

as I dislike overwriting builtin functions/variables if I can avoid it as
it leads to potentially surprising behavior, though reversing that would
not be very difficult if so desired.

Matt
_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3

Werner LEMBERG
In reply to this post by Matthew Peveler

> This does assume however that GUB is updated to at least target at
> least Python 2.6 (for __future__.print_function).

GUB already contains python 2.6 support.  However, lilypond isn't set
up to use it.  This should be rather simple to do, I think.  However,
I haven't gub used since months and right now I lack time to re-learn
it...


    Werner

_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3

Dan Eble
In reply to this post by Matthew Peveler
On Sep 30, 2019, at 17:05, Matthew Peveler <[hidden email]> wrote:
>
> Please see attached for [ten] patches, which given the work done prior by Jonas, allows for running the various make targets under both python2&3 (assuming application of my other two patches for py3)

The thought that gave rise to my question was that I would be more comfortable collaborating if these changes were in git rather than a dozen patches in multiple messages in the mailing list archive.  (I don’t intend that to sound mean.)

> my principle test was that the targets finished without error
> . . .
> The stepmake stuff though is were I have no idea how to handle as I am not sure exactly what its purpose/doing is.

Is that pessimism?

If you want to discover whether the targets you are testing exercise the rules in python-module-rules.make, you can insert a command that will fail when you get there, e.g.

$(outdir)/%.pyo: $(outdir)/%.py
        false # DO NOT COMMIT
        $(PYTHON) -O -c 'import py_compile; py_compile.compile ("$<")'

If your tests are exercising those rules and nothing else seems to go wrong, that should reduce your concern.

Regards,

Dan


_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3

Matthew Peveler
On Mon, Sep 30, 2019 at 7:08 PM Dan Eble <[hidden email]> wrote:

> The thought that gave rise to my question was that I would be more
> comfortable collaborating if these changes were in git rather than a dozen
> patches in multiple messages in the mailing list archive.  (I don’t intend
> that to sound mean.)
>

I've been maintaining my work in
https://github.com/MasterOdin/lilypond/tree/py3_future2, though I fear I
squashed Jonas' patches into a singular commit between Knut's and my
commits. However, I think before going further, it is important that the
maintainers finalize what they would like to see in terms of python
compatibility for LilyPond and GUB, though a fair number of the changes can
be landed in upstream master and work on Python 2.4 as well (move to new
style classes, change the string raise to the LilyError class, etc.),
though some require a minimum of Python 2.6 (changes to exception handlers,
__future__.print_function).

I'd be happy to start putting together some patches and submitting them for
review if that would be helpful in this effort, or if this effort is better
directed somewhere else within the python infrastructure of LilyPond/GUB.

> my principle test was that the targets finished without error
> > . . .
> > The stepmake stuff though is were I have no idea how to handle as I am
> not sure exactly what its purpose/doing is.
>
> Is that pessimism?
>

Somewhat, though more at the bigger picture of that this makefile is far
far more complex than the simple ones I've ever dealt with and trying to
figure out the pieces of how it all fits together, plus the necessary
modifications to get it cross version compatible.


> If you want to discover whether the targets you are testing exercise the
> rules in python-module-rules.make, you can insert a command that will fail
> when you get there, e.g.
>
> $(outdir)/%.pyo: $(outdir)/%.py
>         false # DO NOT COMMIT
>         $(PYTHON) -O -c 'import py_compile; py_compile.compile ("$<")'
>
> If your tests are exercising those rules and nothing else seems to go
> wrong, that should reduce your concern.
>
Sorry, I was unclear, should not have rushed to put that last email
together as I was walking out the door. It's more accurate to say that I
understand conceptually what that particular snipped does and how one could
make it work on just Python 2 or Python 3.7, but I do not know one would
approach making it work on both Python 2 and Python3.7 (and potentially
other Python3 targets) and that I think it would be something in if
statements in python-module-vars.make to be in pseudo-code:

if python3:
  py_version = python -c "import sys; print(str(sys.version_info[0]) +
str(sys.version_info[1]))"
  build_dir = $(outdir)/__pycache__
  extension = .cpython-$(py_version).pyc
else:
  build_dir = $(outdir)
  extension = .pyc

with the build target being:

$(build_dir)/%$(extension): $(outdir)/%.py

-

I briefly tried to convert that to actual make syntax, but was getting
nowhere slowly, and have given up on that bit for now.

Matt
_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
Reply | Threaded
Open this post in threaded view
|

Re: Python 3

Dan Eble
On Sep 30, 2019, at 19:46, Matthew Peveler <[hidden email]> wrote:
>  
> I've been maintaining my work in https://github.com/MasterOdin/lilypond/tree/py3_future2
> . . .
> … but I do not know one would approach making it work on both Python 2 and Python3.7 (and potentially other Python3 targets) and that I think it would be something in if statements in python-module-vars.make to be in pseudo-code:

That’s what I needed to know.  I’ll investigate.

Thanks,

Dan


_______________________________________________
lilypond-devel mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-devel
12