packaging lilypond as a docker container?

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

packaging lilypond as a docker container?

Han-Wen Nienhuys-3
What is the state of our binary releases? We currently have to
maintain GUB, and GUB builds are quite slow. Our apple story is even more
complicated, because of the Apple hardware requirement.

Wouldn't it be much more simple to build lilypond as a Docker application?

Then we could just offer a single binary to download, which windows/mac
users can run. We don't have to cross-compile the app which further reduces
build times. The containerized app is still hermetic, so we can be in full
control of the dependency versions

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen
Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Carl Sorensen-3


On 1/20/20, 2:38 PM, "lilypond-devel on behalf of Han-Wen Nienhuys" <lilypond-devel-bounces+c_sorensen=[hidden email] on behalf of [hidden email]> wrote:

    What is the state of our binary releases? We currently have to
    maintain GUB, and GUB builds are quite slow. Our apple story is even more
    complicated, because of the Apple hardware requirement.

We currently have a user who has figured out how to take a MacPorts build and turn it into an application bundle.  I think that we are in the final stages of having that build ready to go.
   
    Wouldn't it be much more simple to build lilypond as a Docker application?

I don't know anything about building lilypond as a Docker application.  If it were possible to execute a docker application from the command line in MacOS, then I think that would meet my requirements.  I need to be able to have multiple binaries installed so that I can run multiple versions from Frescobaldi.
   
    Then we could just offer a single binary to download, which windows/mac
    users can run. We don't have to cross-compile the app which further reduces
    build times. The containerized app is still hermetic, so we can be in full
    control of the dependency versions

As far as I can see right now, the time it takes to complete a GUB build is not that important.  But the complexity of the GUB build system is hugely important.  It's a big obstacle to getting contributors going.

How difficult would it be to set up a build environment for making the Docker application?  A second major obstacle to developing is the difficulty of setting up a build environment for lilypond, especially in Windows and MacOS.  The recommended way to build now is via a virtual machine, with the extra challenges of trying to maintain the VM image.  If the process of making the Docker application would also allow a simple set up for a build environment in non-Linux machines, I think that would be a huge win.

Thanks,

Carl
 

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

David Kastrup
Carl Sorensen <[hidden email]> writes:

> On 1/20/20, 2:38 PM, "lilypond-devel on behalf of Han-Wen Nienhuys"
> <lilypond-devel-bounces+c_sorensen=[hidden email] on behalf of
> [hidden email]> wrote:
>
>     What is the state of our binary releases? We currently have to
>     maintain GUB, and GUB builds are quite slow. Our apple story is even more
>     complicated, because of the Apple hardware requirement.
>
> We currently have a user who has figured out how to take a MacPorts
> build and turn it into an application bundle.  I think that we are in
> the final stages of having that build ready to go.
>    
>     Wouldn't it be much more simple to build lilypond as a Docker application?
>
> I don't know anything about building lilypond as a Docker application.
> If it were possible to execute a docker application from the command
> line in MacOS, then I think that would meet my requirements.  I need
> to be able to have multiple binaries installed so that I can run
> multiple versions from Frescobaldi.
>    
>     Then we could just offer a single binary to download, which windows/mac
>     users can run. We don't have to cross-compile the app which further reduces
>     build times. The containerized app is still hermetic, so we can be in full
>     control of the dependency versions
>
> As far as I can see right now, the time it takes to complete a GUB
> build is not that important.  But the complexity of the GUB build
> system is hugely important.  It's a big obstacle to getting
> contributors going.

Contributors don't need to bother with GUB.  GUB is just used internally
for cranking out releases.  Nobody uses GUB when testing: instead
LilyDev is being used.  No crosscompilation, instead cross execution.

> How difficult would it be to set up a build environment for making the
> Docker application?  A second major obstacle to developing is the
> difficulty of setting up a build environment for lilypond, especially
> in Windows and MacOS.  The recommended way to build now is via a
> virtual machine, with the extra challenges of trying to maintain the
> VM image.  If the process of making the Docker application would also
> allow a simple set up for a build environment in non-Linux machines, I
> think that would be a huge win.

Not sure where this is getting, but it might just be a case of beer.
Actually, more like a bottle of beer.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Karlin High
In reply to this post by Carl Sorensen-3
On 1/20/2020 4:05 PM, Carl Sorensen wrote:
> If the process of making the Docker application would also allow a simple set up for a build environment in non-Linux machines, I think that would be a huge win.

There already is a LilyDev Docker image.

<https://github.com/fedelibre/LilyDev/tree/master/docker>

I tried it once, but was unable to get it working. It was my first and
only experience with Docker, so that's probably my fault. I was also
unmotivated to persevere due to having LilyDev environments running on
various other VM things. (VirtualBox, Microsoft Hyper-V, Windows
Subsystem for Linux, etc)
--
Karlin High
Missouri, USA

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Han-Wen Nienhuys-3
In reply to this post by David Kastrup
On Mon, Jan 20, 2020 at 11:57 PM David Kastrup <[hidden email]> wrote:

> > As far as I can see right now, the time it takes to complete a GUB
> > build is not that important.  But the complexity of the GUB build
> > system is hugely important.  It's a big obstacle to getting
> > contributors going.
>
> Contributors don't need to bother with GUB.  GUB is just used internally
> for cranking out releases.  Nobody uses GUB when testing: instead
> LilyDev is being used.  No crosscompilation, instead cross execution
>

if GUB is used, who is maintaining and/or working on it?

I enjoy working on LilyPond a lot, but GUB is different. I want GUB to
disappear so I never have to look at it again.

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen
Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Kevin Barry
In reply to this post by Carl Sorensen-3
On Mon, Jan 20, 2020 at 10:05:23PM +0000, Carl Sorensen wrote:
>    
>     Wouldn't it be much more simple to build lilypond as a Docker application?
>
> I don't know anything about building lilypond as a Docker application.  If it were possible to execute a docker application from the command line in MacOS, then I think that would meet my requirements.  I need to be able to have multiple binaries installed so that I can run multiple versions from Frescobaldi.

Running docker on MacOS requires a virtual machine running Linux. The
Docker for desktop app for MacOS hides that detail away from the user
for the most part.

It is technically possible to do what you want (make it so Frescobaldi
can call multiple versions of LilyPond) with docker, but there would be
some work involved: you'd probably need to have one-liner shell scripts
for each version of LilyPond that execute the relevant container.

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Karlin High
In reply to this post by Han-Wen Nienhuys-3
On 1/21/2020 1:49 AM, Han-Wen Nienhuys wrote:
> if GUB is used, who is maintaining and/or working on it?

Almost exactly a year ago, there was a sizable "Please test gub" effort
initiated by Knut Petersen.

<https://lists.gnu.org/archive/html/lilypond-devel/2019-01/msg00221.html>

Clear instructions were given for exactly how to test GUB, and just that
thread seems to have involved about 16 people.

More recently, it looks like Jonas Hahnfeld's work with Python 3
migration involved quite a number of changes to GUB.

 From the notes I've kept, Jan Nieuwenhuizen proposed replacing GUB with
something based on Guix.

<https://lists.gnu.org/archive/html/lilypond-user/2016-09/msg00690.html>

That idea was mentioned a few times since, but I can't recall if
GUB-replacement discussions have moved beyond that or not.
--
Karlin High
Missouri, USA

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Dev mailing list
Am Dienstag, den 21.01.2020, 02:38 -0600 schrieb Karlin High:

> On 1/21/2020 1:49 AM, Han-Wen Nienhuys wrote:
> > if GUB is used, who is maintaining and/or working on it?
>
> Almost exactly a year ago, there was a sizable "Please test gub" effort
> initiated by Knut Petersen.
>
> <
> https://lists.gnu.org/archive/html/lilypond-devel/2019-01/msg00221.html
> >
>
> Clear instructions were given for exactly how to test GUB, and just that
> thread seems to have involved about 16 people.
>
> More recently, it looks like Jonas Hahnfeld's work with Python 3
> migration involved quite a number of changes to GUB.
Yep, and it's not been a very pleasant experience. To be precise here:
It's not about porting GUB to Python 3, it's just that porting LilyPond
will introduce a dependency on an updated package for the next release.
So while David is mostly correct that contributors don't need to bother
with GUB, that doesn't hold true once you want to bump one of its
dependencies...

Based on this experience, I've thought about how to improve the process
of building binary releases for LilyPond. What I have been
experimenting with is a set of portable shell scripts that build mostly
static executables. I've a prototype ready at
https://github.com/hahnjo/lilypond-binaries which works for Linux and
FreeBSD. I don't see a reason it cannot work on macOS, but I don't have
the possibility to test...

>  From the notes I've kept, Jan Nieuwenhuizen proposed replacing GUB with
> something based on Guix.
>
> <
> https://lists.gnu.org/archive/html/lilypond-user/2016-09/msg00690.html
> >
>
> That idea was mentioned a few times since, but I can't recall if
> GUB-replacement discussions have moved beyond that or not.

As far as I understand, Guix is a package manager in the first place.
In the thread Jan proposes to use it for cross-compilation, which I
think is the primary reason for the complexity in GUB. So why go that
route once more?
Instead I designed the mentioned scripts such that they produce native
executables. That's also how TeXlive builds their packages, for
example, and it works quite well if you compile on the oldest OS that
you intend to support (CentOS 7, FreeBSD 11).

I haven't decided yet how to compile for Windows. Maybe that's still a
valid use case for cross-compilation (but only with a very limited
scope). I'm not going to tackle this until we've switched to Python:
Then we can just use the embeddable zip file provided by the Python
project without bothering with cross-compilation for this beast.
Everything else (Guile etc.) worked without problems in my early tests.

If this attempt sounds interesting to the community, I would be happy
to submit the scripts for inclusion into the LilyPond repository
itself. That would also solve another issue with GUB: Currently it's a
separate repository with no way to keep it in sync with changing
dependencies for LilyPond...

Regards,
Jonas

P.S.: to come back to the whole container idea: after you have the
binaries, you can of course build a docker container from them...

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

David Kastrup
Jonas Hahnfeld <[hidden email]> writes:

> Am Dienstag, den 21.01.2020, 02:38 -0600 schrieb Karlin High:
>> On 1/21/2020 1:49 AM, Han-Wen Nienhuys wrote:
>> > if GUB is used, who is maintaining and/or working on it?
>>
>> Almost exactly a year ago, there was a sizable "Please test gub" effort
>> initiated by Knut Petersen.
>>
>> <
>> https://lists.gnu.org/archive/html/lilypond-devel/2019-01/msg00221.html
>> >
>>
>> Clear instructions were given for exactly how to test GUB, and just that
>> thread seems to have involved about 16 people.
>>
>> More recently, it looks like Jonas Hahnfeld's work with Python 3
>> migration involved quite a number of changes to GUB.
>
> Yep, and it's not been a very pleasant experience. To be precise here:
> It's not about porting GUB to Python 3, it's just that porting LilyPond
> will introduce a dependency on an updated package for the next release.
> So while David is mostly correct that contributors don't need to bother
> with GUB, that doesn't hold true once you want to bump one of its
> dependencies...
>
> Based on this experience, I've thought about how to improve the process
> of building binary releases for LilyPond. What I have been
> experimenting with is a set of portable shell scripts that build mostly
> static executables. I've a prototype ready at
> https://github.com/hahnjo/lilypond-binaries which works for Linux and
> FreeBSD. I don't see a reason it cannot work on macOS, but I don't have
> the possibility to test...

The problem is "on macOS" since current macOSX library/toolbox licenses
all prohibit use in cross-building environments.  It looks like a given
that we will not be able to offer macOSX libraries in future as the
result of a unified release process.  Our sources build reasonably well
that macOSX integrators using some of the macOSX typical packaging
systems are able to provide reasonable replacements.

But the elephant in the room is Windows.  Han-Wen says he never wants to
touch GUB again (and he actually didn't as far as I remember) but I
don't want to touch Windows again, and GUB provides them.  The
dependency nightmare on a non-UNIX like platform was what brought GUB
into being in the first place: native builds are much more of an ongoing
problem.  Just look at the continuous effort the Git project has in
keeping a Windows version available.

>>  From the notes I've kept, Jan Nieuwenhuizen proposed replacing GUB with
>> something based on Guix.
>>
>> <
>> https://lists.gnu.org/archive/html/lilypond-user/2016-09/msg00690.html
>> >
>>
>> That idea was mentioned a few times since, but I can't recall if
>> GUB-replacement discussions have moved beyond that or not.
>
> As far as I understand, Guix is a package manager in the first place.
> In the thread Jan proposes to use it for cross-compilation, which I
> think is the primary reason for the complexity in GUB. So why go that
> route once more?
> Instead I designed the mentioned scripts such that they produce native
> executables. That's also how TeXlive builds their packages, for
> example, and it works quite well if you compile on the oldest OS that
> you intend to support (CentOS 7, FreeBSD 11).
>
> I haven't decided yet how to compile for Windows. Maybe that's still a
> valid use case for cross-compilation (but only with a very limited
> scope).

Windows really is the elephant in the room.  MacOSX will cater with
native port systems like MacPorts etc and other UNIX-like systems also
have working packagers and package systems.

> If this attempt sounds interesting to the community, I would be happy
> to submit the scripts for inclusion into the LilyPond repository
> itself. That would also solve another issue with GUB: Currently it's a
> separate repository with no way to keep it in sync with changing
> dependencies for LilyPond...

Maybe we should entertain two branches of GUB matching current stable
and unstable release tracks?  Or otherwise deal with a difference in
dependencies for stable/unstable?

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Dev mailing list
Am Dienstag, den 21.01.2020, 11:28 +0100 schrieb David Kastrup:

> Jonas Hahnfeld <
> [hidden email]
> > writes:
>
> > Am Dienstag, den 21.01.2020, 02:38 -0600 schrieb Karlin High:
> > > On 1/21/2020 1:49 AM, Han-Wen Nienhuys wrote:
> > > > if GUB is used, who is maintaining and/or working on it?
> > >
> > > Almost exactly a year ago, there was a sizable "Please test gub" effort
> > > initiated by Knut Petersen.
> > >
> > > <
> > > https://lists.gnu.org/archive/html/lilypond-devel/2019-01/msg00221.html
> > >
> > >
> > > Clear instructions were given for exactly how to test GUB, and just that
> > > thread seems to have involved about 16 people.
> > >
> > > More recently, it looks like Jonas Hahnfeld's work with Python 3
> > > migration involved quite a number of changes to GUB.
> >
> > Yep, and it's not been a very pleasant experience. To be precise here:
> > It's not about porting GUB to Python 3, it's just that porting LilyPond
> > will introduce a dependency on an updated package for the next release.
> > So while David is mostly correct that contributors don't need to bother
> > with GUB, that doesn't hold true once you want to bump one of its
> > dependencies...
> >
> > Based on this experience, I've thought about how to improve the process
> > of building binary releases for LilyPond. What I have been
> > experimenting with is a set of portable shell scripts that build mostly
> > static executables. I've a prototype ready at
> > https://github.com/hahnjo/lilypond-binaries
> >  which works for Linux and
> > FreeBSD. I don't see a reason it cannot work on macOS, but I don't have
> > the possibility to test...
>
> The problem is "on macOS" since current macOSX library/toolbox licenses
> all prohibit use in cross-building environments.  It looks like a given
> that we will not be able to offer macOSX libraries in future as the
> result of a unified release process.  Our sources build reasonably well
> that macOSX integrators using some of the macOSX typical packaging
> systems are able to provide reasonable replacements.
>
> But the elephant in the room is Windows.  Han-Wen says he never wants to
> touch GUB again (and he actually didn't as far as I remember) but I
> don't want to touch Windows again, and GUB provides them.  The
> dependency nightmare on a non-UNIX like platform was what brought GUB
> into being in the first place: native builds are much more of an ongoing
> problem.  Just look at the continuous effort the Git project has in
> keeping a Windows version available.
>
> > >  From the notes I've kept, Jan Nieuwenhuizen proposed replacing GUB with
> > > something based on Guix.
> > >
> > > <
> > > https://lists.gnu.org/archive/html/lilypond-user/2016-09/msg00690.html
> > >
> > >
> > > That idea was mentioned a few times since, but I can't recall if
> > > GUB-replacement discussions have moved beyond that or not.
> >
> > As far as I understand, Guix is a package manager in the first place.
> > In the thread Jan proposes to use it for cross-compilation, which I
> > think is the primary reason for the complexity in GUB. So why go that
> > route once more?
> > Instead I designed the mentioned scripts such that they produce native
> > executables. That's also how TeXlive builds their packages, for
> > example, and it works quite well if you compile on the oldest OS that
> > you intend to support (CentOS 7, FreeBSD 11).
> >
> > I haven't decided yet how to compile for Windows. Maybe that's still a
> > valid use case for cross-compilation (but only with a very limited
> > scope).
>
> Windows really is the elephant in the room.  MacOSX will cater with
> native port systems like MacPorts etc and other UNIX-like systems also
> have working packagers and package systems.
So if I provided a working script to build a zip file for Windows,
would you be ok with switching away from GUB?

In that case I just noticed that I missed one character in my message:
> I'm not going to tackle this until we've switched to Python *3*:
> Then we can just use the embeddable zip file provided by the Python
> project without bothering with cross-compilation for this beast.
> Everything else (Guile etc.) worked without problems in my early tests.

Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

David Kastrup
Jonas Hahnfeld <[hidden email]> writes:

> Am Dienstag, den 21.01.2020, 11:28 +0100 schrieb David Kastrup:
>>
>> Windows really is the elephant in the room.  MacOSX will cater with
>> native port systems like MacPorts etc and other UNIX-like systems
>> also have working packagers and package systems.
>
> So if I provided a working script to build a zip file for Windows,
> would you be ok with switching away from GUB?

We'd want an actual installer.  That's what the average user can work
with.  Alternatively, get the Frescobaldi people to take over
installation from a zip file and make Frescobaldi our "installer" (and
ask them to make the result useable also without Frescobaldi).  That
introduces a dependency I personally would consider a nuisance, but the
installer is intended for people with little comfort navigating command
lines and picking installation places and options, and most will end up
using Frescobaldi anyway.

> In that case I just noticed that I missed one character in my message:
>> I'm not going to tackle this until we've switched to Python *3*:
>> Then we can just use the embeddable zip file provided by the Python
>> project without bothering with cross-compilation for this beast.
>> Everything else (Guile etc.) worked without problems in my early tests.

If it ends up as "just install Python according to instructions", it's
sort of a real obstacle compared to an all-in-one installer.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Werner LEMBERG
In reply to this post by Karlin High

> Almost exactly a year ago, there was a sizable "Please test gub"
> effort initiated by Knut Petersen.

By the way: Any idea what has happened with Knut?  He doesn't respond
even to private e-mails since a few months...


    Werner

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Dan Eble
In reply to this post by Karlin High
On Jan 20, 2020, at 18:21, Karlin High <[hidden email]> wrote:
>
> There already is a LilyDev Docker image.
>
> <https://github.com/fedelibre/LilyDev/tree/master/docker>
>
> I tried it once, but was unable to get it working. It was my first and only experience with Docker, so that's probably my fault. I was also unmotivated to persevere . . .

I'm the original author and de-facto maintainer of that Dockerfile.  I assume I'm also the sole user, based on the lack of questions about it.

My experience is that it's more stable than VirtualBox, and I can more easily keep my source backed up (because it's in my host filesystem, not the VM filesystem), but the performance of accessing the host filesystem is mightily disappointing (unlike on Linux hosts).

If anyone is reading this and thinking of trying it, you might be better off encouraging me to publish some of the private changes I've been testing, rather than trying what's in Federico's repo.  I tend to slack off in that regard because no one is asking.

Dan


Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Michael Käppler-2
Would be happy to try the Dockerfile, since I've never used Docker
before and find
the concept interesting.
Could you push your changes to your LilyDev fork and/or file a pull
request against
Federico's repo?

Cheers,
Michael

Am 21.01.2020 um 16:47 schrieb Dan Eble:

> On Jan 20, 2020, at 18:21, Karlin High <[hidden email]> wrote:
>> There already is a LilyDev Docker image.
>>
>> <https://github.com/fedelibre/LilyDev/tree/master/docker>
>>
>> I tried it once, but was unable to get it working. It was my first and only experience with Docker, so that's probably my fault. I was also unmotivated to persevere . . .
> I'm the original author and de-facto maintainer of that Dockerfile.  I assume I'm also the sole user, based on the lack of questions about it.
>
> My experience is that it's more stable than VirtualBox, and I can more easily keep my source backed up (because it's in my host filesystem, not the VM filesystem), but the performance of accessing the host filesystem is mightily disappointing (unlike on Linux hosts).
>
> If anyone is reading this and thinking of trying it, you might be better off encouraging me to publish some of the private changes I've been testing, rather than trying what's in Federico's repo.  I tend to slack off in that regard because no one is asking.
> —
> Dan
>
>


Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Han-Wen Nienhuys-3
In reply to this post by David Kastrup
On Tue, Jan 21, 2020 at 11:28 AM David Kastrup <[hidden email]> wrote:

> But the elephant in the room is Windows.  Han-Wen says he never wants to
> touch GUB again (and he actually didn't as far as I remember) but I
>

excuse me?

$ git log  |grep ^Author | sort | uniq -c|sort -n |tail
     41 Author: PhilHolmes <[hidden email]>
    108 Author: Masamichi Hosoda <[hidden email]>
    109 Author: lilytest <[hidden email]>
    160 Author: Masamichi Hosoda <[hidden email]>
    169 Author: Han-Wen Nienhuys <[hidden email]>
    482 Author: janneke <[hidden email]>
    569 Author: Han-Wen Nienhuys <[hidden email]>
    899 Author: hanwen <[hidden email]>
    912 Author: hanwen <[hidden email]>
   2801 Author: Jan Nieuwenhuizen <[hidden email]>

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen
Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Han-Wen Nienhuys-3
In reply to this post by David Kastrup
Let me explain my thinking more in detail. (I've CC'd Jan in case he wants
to give further historical insights)

The objective of GUB was to provide installers for LilyPond with the
following requirements:

1) build for all platforms (OSX, Windows, Linux) with minimal effort,
without having to manage windows and OSX machines.
2) all platforms use exactly the same versions of all software, so everyone
has the same experience, and problems always reproduce across platforms.

1) implied setting up a cross compiling environment. This is a pain in the
ass, because GCC must be compiled for each (host, target) pair separately.
This is very slow and takes up a lot of space.

However, the major reason why this is a PITA is that many pieces of
software don't support cross-compiling well. In particular, some software
runs itself as part of the build (lilypond is an example, but so do Python
and Fontconfig IIRC). Making them cross-compile is something that Jan did
in his professional career (embedded software!), so he knew how to do it
for LilyPond. It is however very intricate, and time consuming to test. One
has to pore over compilation logs, understand what all these packages are
trying to do, and then come up with ways to do the same in a
cross-compiling manner. This is why the GUB specifications are patching
compilation steps in creative ways.

I found it demoralizing to work on, because I want to work on music
typesetting, and not on patching up someone else's wonky compilation script.

Because we now have a very expensive and complicated setup (GCC !), it
makes sense to utilize this setup across builds. For example, if we upgrade
the lilypond version, we'd rather not have to rebuild GCC from scratch. So,
GUB has a packaging system, where you can upgrade LilyPond (or any
intermediate dependency) without starting from scratch.

We started out with a set of shell scripts, but they were hard to maintain,
so we built GUB. So, to me, Jonas' proposal to have a set of simple
compilation scripts is a little funny.

Cross compiling has these problems above, but the "native" compile also has
problems. One has ensure that the native compile doesn't pick up
dependencies from the system it is compiled from, as that could cause
unpredictable behavior. You'd run GUB on a machine that is used for
LilyPond, so all of the dependencies can be consumed from both GUB and the
host machine, and it is easy for something to go wrong here.

In the intervening 15 years, many things have changed.

A negative: apple licensing restrictions say that we can't cross-compile to
OSX (although to be honest, I never read the terms & conditions for the
XCode stuff I downloaded from apple back in the day. Maybe we've always
been in violation of some T&C)

A posiitive: all major platforms run on x86 64-bit CPUs, and there are Open
Source VM solutions for OSX and Windows. So, a VM running Linux on OSX or
Windows can be almost as fast as running Linux directly. This makes GUB's
cross compilation to other architectures superfluous.

A positive: Linux now supports namespaces, and in particular: file system
namespaces. This makes it possible to build a native Linux package fully
isolated from its host system, thus skipping GUB's laborious GCC setup.
This also lets you run a regular package manager on the namespaced file
system, and this is essentially what Docker gives you: a way to install
arbitrary set of dependencies and use them. This makes GUB's package
manager superfluous.

A positive: Docker is open source, and extremely popular technology. Docker
does the hard work of packaging up VM + Docker for windows and OSX as
installer, and providing a management/distribution mechanism for images.

All in all, I think we could use Docker for distributing LilyPond. Then we
would have a distribution mechanism that works for OSX  and Windows, and it
has no overhead to compiling LilyPond as normal Linux package. Because we
can control the base image, we can precisely control the dependencies, and
be sure that the binary behaves the same across all OSes. The best is that
building releases won't require any arcane knowledge of cross-compilation
quirks.

GUIX is Jan's current project. I think it has some similarities to GUB, but
it is focused on providing an environment where all binaries are
reproducibly built from source. This is much more ambitious than GUB, and
seems overkill compared to what we need for LilyPond. I think using it also
entails many more compilation steps, which would makes the release process
slow again.


On Tue, Jan 21, 2020 at 11:28 AM David Kastrup <[hidden email]> wrote:

> Jonas Hahnfeld <[hidden email]> writes:
>
> > Am Dienstag, den 21.01.2020, 02:38 -0600 schrieb Karlin High:
> >> On 1/21/2020 1:49 AM, Han-Wen Nienhuys wrote:
> >> > if GUB is used, who is maintaining and/or working on it?
> >>
> >> Almost exactly a year ago, there was a sizable "Please test gub" effort
> >> initiated by Knut Petersen.
> >>
> >> <
> >> https://lists.gnu.org/archive/html/lilypond-devel/2019-01/msg00221.html
> >> >
> >>
> >> Clear instructions were given for exactly how to test GUB, and just
> that
> >> thread seems to have involved about 16 people.
> >>
> >> More recently, it looks like Jonas Hahnfeld's work with Python 3
> >> migration involved quite a number of changes to GUB.
> >
> > Yep, and it's not been a very pleasant experience. To be precise here:
> > It's not about porting GUB to Python 3, it's just that porting LilyPond
> > will introduce a dependency on an updated package for the next release.
> > So while David is mostly correct that contributors don't need to bother
> > with GUB, that doesn't hold true once you want to bump one of its
> > dependencies...
> >
> > Based on this experience, I've thought about how to improve the process
> > of building binary releases for LilyPond. What I have been
> > experimenting with is a set of portable shell scripts that build mostly
> > static executables. I've a prototype ready at
> > https://github.com/hahnjo/lilypond-binaries which works for Linux and
> > FreeBSD. I don't see a reason it cannot work on macOS, but I don't have
> > the possibility to test...
>
> The problem is "on macOS" since current macOSX library/toolbox licenses
> all prohibit use in cross-building environments.  It looks like a given
> that we will not be able to offer macOSX libraries in future as the
> result of a unified release process.  Our sources build reasonably well
> that macOSX integrators using some of the macOSX typical packaging
> systems are able to provide reasonable replacements.
>
> But the elephant in the room is Windows.  Han-Wen says he never wants to
> touch GUB again (and he actually didn't as far as I remember) but I
> don't want to touch Windows again, and GUB provides them.  The
> dependency nightmare on a non-UNIX like platform was what brought GUB
> into being in the first place: native builds are much more of an ongoing
> problem.  Just look at the continuous effort the Git project has in
> keeping a Windows version available.
>
> >>  From the notes I've kept, Jan Nieuwenhuizen proposed replacing GUB
> with
> >> something based on Guix.
> >>
> >> <
> >> https://lists.gnu.org/archive/html/lilypond-user/2016-09/msg00690.html
> >> >
> >>
> >> That idea was mentioned a few times since, but I can't recall if
> >> GUB-replacement discussions have moved beyond that or not.
> >
> > As far as I understand, Guix is a package manager in the first place.
> > In the thread Jan proposes to use it for cross-compilation, which I
> > think is the primary reason for the complexity in GUB. So why go that
> > route once more?
> > Instead I designed the mentioned scripts such that they produce native
> > executables. That's also how TeXlive builds their packages, for
> > example, and it works quite well if you compile on the oldest OS that
> > you intend to support (CentOS 7, FreeBSD 11).
> >
> > I haven't decided yet how to compile for Windows. Maybe that's still a
> > valid use case for cross-compilation (but only with a very limited
> > scope).
>
> Windows really is the elephant in the room.  MacOSX will cater with
> native port systems like MacPorts etc and other UNIX-like systems also
> have working packagers and package systems.
>
> > If this attempt sounds interesting to the community, I would be happy
> > to submit the scripts for inclusion into the LilyPond repository
> > itself. That would also solve another issue with GUB: Currently it's a
> > separate repository with no way to keep it in sync with changing
> > dependencies for LilyPond...
>
> Maybe we should entertain two branches of GUB matching current stable
> and unstable release tracks?  Or otherwise deal with a difference in
> dependencies for stable/unstable?
>
> --
> David Kastrup
>


--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen
Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

David Kastrup
In reply to this post by Han-Wen Nienhuys-3
Han-Wen Nienhuys <[hidden email]> writes:

> On Tue, Jan 21, 2020 at 11:28 AM David Kastrup <[hidden email]> wrote:
>
>> But the elephant in the room is Windows.  Han-Wen says he never wants to
>> touch GUB again (and he actually didn't as far as I remember) but I

Something happened between brain and keyboard.  What I meant to write
was "and he actually didn't as far as I remember during the time of my
active tenure on LilyPond".  I agree that this does come off quite wrong
and will be misleading at the very least to everyone who hasn't stuck
around long enough to be versed in LilyPond's long-term history.

Sorry for that.

>>
>>
>
> excuse me?
>
> $ git log  |grep ^Author | sort | uniq -c|sort -n |tail
>      41 Author: PhilHolmes <[hidden email]>
>     108 Author: Masamichi Hosoda <[hidden email]>
>     109 Author: lilytest <[hidden email]>
>     160 Author: Masamichi Hosoda <[hidden email]>
>     169 Author: Han-Wen Nienhuys <[hidden email]>
>     482 Author: janneke <[hidden email]>
>     569 Author: Han-Wen Nienhuys <[hidden email]>
>     899 Author: hanwen <[hidden email]>
>     912 Author: hanwen <[hidden email]>
>    2801 Author: Jan Nieuwenhuizen <[hidden email]>

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

Dev mailing list
In reply to this post by Han-Wen Nienhuys-3
Let me respond to some of your points:

Am Mittwoch, den 22.01.2020, 11:00 +0100 schrieb Han-Wen Nienhuys:

> Let me explain my thinking more in detail. (I've CC'd Jan in case he wants to give further historical insights)
>
>
> The objective of GUB was to provide installers for LilyPond with the following requirements:
>
> 1) build for all platforms (OSX, Windows, Linux) with minimal effort, without having to manage windows and OSX machines.
>
> 2) all platforms use exactly the same versions of all software, so everyone has the same experience, and problems always reproduce across platforms.
>
> 1) implied setting up a cross compiling environment. This is a pain in the ass, because GCC must be compiled for each (host, target) pair separately. This is very slow and takes up a lot of space.
And doesn't even work. Did you recently try to run the binaries
produced for FreeBSD? They're dynamically linked to GUB's glibc which
isn't packaged up. Not surprisingly, FreeBSD has its own libc with
different so-versions and so on.

> However, the major reason why this is a PITA is that many pieces of software don't support cross-compiling well. In particular, some software runs itself as part of the build (lilypond is an example, but so do Python and Fontconfig IIRC). Making them cross-compile is something that Jan did in his professional career (embedded software!), so he knew how to do it for LilyPond. It is however very intricate, and time consuming to test. One has to pore over compilation logs, understand what all these packages are trying to do, and then come up with ways to do the same in a cross-compiling manner. This is why the GUB specifications are patching compilation steps in creative ways.

The patches are horrible to maintain once you want to bump more than
just to a new patch version.

> I found it demoralizing to work on, because I want to work on music typesetting, and not on patching up someone else's wonky compilation script.
>
> Because we now have a very expensive and complicated setup (GCC !), it makes sense to utilize this setup across builds. For example, if we upgrade the lilypond version, we'd rather not have to rebuild GCC from scratch. So, GUB has a packaging system, where you can upgrade LilyPond (or any intermediate dependency) without starting from scratch.

I do the same for the dependencies of LilyPond, except for all that
complicated packaging stuff. If you screw something up, just rebuild
it. This takes around ~20 minutes on my Laptop from 2016.

> We started out with a set of shell scripts, but they were hard to maintain, so we built GUB. So, to me, Jonas' proposal to have a set of simple compilation scripts is a little funny.

As a predecessor of GUB, this probably involved cross-compilation? I'm
not doing this, it's just native compilation all way long.

> Cross compiling has these problems above, but the "native" compile also has problems. One has ensure that the native compile doesn't pick up dependencies from the system it is compiled from, as that could cause unpredictable behavior. You'd run GUB on a machine that is used for LilyPond, so all of the dependencies can be consumed from both GUB and the host machine, and it is easy for something to go wrong here.

I think this got much better, and I looks like I finally convinced pkg-
config not to use anything from the system. This only leaves the
compiler headers and libc. All of the rest is built as static libraries
which end up in the LilyPond executable.

> In the intervening 15 years, many things have changed.
>
> A negative: apple licensing restrictions say that we can't cross-compile to OSX (although to be honest, I never read the terms & conditions for the XCode stuff I downloaded from apple back in the day. Maybe we've always been in violation of some T&C)
>
> A posiitive: all major platforms run on x86 64-bit CPUs, and there are Open Source VM solutions for OSX and Windows. So, a VM running Linux on OSX or Windows can be almost as fast as running Linux directly. This makes GUB's cross compilation to other architectures superfluous.
>
> A positive: Linux now supports namespaces, and in particular: file system namespaces. This makes it possible to build a native Linux package fully isolated from its host system, thus skipping GUB's laborious GCC setup. This also lets you run a regular package manager on the namespaced file system, and this is essentially what Docker gives you: a way to install arbitrary set of dependencies and use them. This makes GUB's package manager superfluous.
>
> A positive: Docker is open source, and extremely popular technology. Docker does the hard work of packaging up VM + Docker for windows and OSX as installer, and providing a management/distribution mechanism for images.
>
> All in all, I think we could use Docker for distributing LilyPond. Then we would have a distribution mechanism that works for OSX  and Windows, and it has no overhead to compiling LilyPond as normal Linux package. Because we can control the base image, we can precisely control the dependencies, and be sure that the binary behaves the same across all OSes. The best is that building releases won't require any arcane knowledge of cross-compilation quirks.
So you want to require users to setup Docker on Windows to run
LilyPond? Not the lowest barrier we could possibly have IMO.

Jonas

>
> GUIX is Jan's current project. I think it has some similarities to GUB, but it is focused on providing an environment where all binaries are reproducibly built from source. This is much more ambitious than GUB, and seems overkill compared to what we need for LilyPond. I think using it also entails many more compilation steps, which would makes the release process slow again.
>
>
>
> On Tue, Jan 21, 2020 at 11:28 AM David Kastrup <[hidden email]> wrote:
> > Jonas Hahnfeld <[hidden email]> writes:
> >
> > > Am Dienstag, den 21.01.2020, 02:38 -0600 schrieb Karlin High:
> > >> On 1/21/2020 1:49 AM, Han-Wen Nienhuys wrote:
> > >> > if GUB is used, who is maintaining and/or working on it?
> > >>
> > >> Almost exactly a year ago, there was a sizable "Please test gub" effort
> > >> initiated by Knut Petersen.
> > >>
> > >> <
> > >> https://lists.gnu.org/archive/html/lilypond-devel/2019-01/msg00221.html
> > >> >
> > >>
> > >> Clear instructions were given for exactly how to test GUB, and just that
> > >> thread seems to have involved about 16 people.
> > >>
> > >> More recently, it looks like Jonas Hahnfeld's work with Python 3
> > >> migration involved quite a number of changes to GUB.
> > >
> > > Yep, and it's not been a very pleasant experience. To be precise here:
> > > It's not about porting GUB to Python 3, it's just that porting LilyPond
> > > will introduce a dependency on an updated package for the next release.
> > > So while David is mostly correct that contributors don't need to bother
> > > with GUB, that doesn't hold true once you want to bump one of its
> > > dependencies...
> > >
> > > Based on this experience, I've thought about how to improve the process
> > > of building binary releases for LilyPond. What I have been
> > > experimenting with is a set of portable shell scripts that build mostly
> > > static executables. I've a prototype ready at
> > > https://github.com/hahnjo/lilypond-binaries which works for Linux and
> > > FreeBSD. I don't see a reason it cannot work on macOS, but I don't have
> > > the possibility to test...
> >
> > The problem is "on macOS" since current macOSX library/toolbox licenses
> > all prohibit use in cross-building environments.  It looks like a given
> > that we will not be able to offer macOSX libraries in future as the
> > result of a unified release process.  Our sources build reasonably well
> > that macOSX integrators using some of the macOSX typical packaging
> > systems are able to provide reasonable replacements.
> >
> > But the elephant in the room is Windows.  Han-Wen says he never wants to
> > touch GUB again (and he actually didn't as far as I remember) but I
> > don't want to touch Windows again, and GUB provides them.  The
> > dependency nightmare on a non-UNIX like platform was what brought GUB
> > into being in the first place: native builds are much more of an ongoing
> > problem.  Just look at the continuous effort the Git project has in
> > keeping a Windows version available.
> >
> > >>  From the notes I've kept, Jan Nieuwenhuizen proposed replacing GUB with
> > >> something based on Guix.
> > >>
> > >> <
> > >> https://lists.gnu.org/archive/html/lilypond-user/2016-09/msg00690.html
> > >> >
> > >>
> > >> That idea was mentioned a few times since, but I can't recall if
> > >> GUB-replacement discussions have moved beyond that or not.
> > >
> > > As far as I understand, Guix is a package manager in the first place.
> > > In the thread Jan proposes to use it for cross-compilation, which I
> > > think is the primary reason for the complexity in GUB. So why go that
> > > route once more?
> > > Instead I designed the mentioned scripts such that they produce native
> > > executables. That's also how TeXlive builds their packages, for
> > > example, and it works quite well if you compile on the oldest OS that
> > > you intend to support (CentOS 7, FreeBSD 11).
> > >
> > > I haven't decided yet how to compile for Windows. Maybe that's still a
> > > valid use case for cross-compilation (but only with a very limited
> > > scope).
> >
> > Windows really is the elephant in the room.  MacOSX will cater with
> > native port systems like MacPorts etc and other UNIX-like systems also
> > have working packagers and package systems.
> >
> > > If this attempt sounds interesting to the community, I would be happy
> > > to submit the scripts for inclusion into the LilyPond repository
> > > itself. That would also solve another issue with GUB: Currently it's a
> > > separate repository with no way to keep it in sync with changing
> > > dependencies for LilyPond...
> >
> > Maybe we should entertain two branches of GUB matching current stable
> > and unstable release tracks?  Or otherwise deal with a difference in
> > dependencies for stable/unstable?
> >
> > --
> > David Kastrup
>
>
>

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: packaging lilypond as a docker container?

David Kastrup
In reply to this post by Han-Wen Nienhuys-3
Han-Wen Nienhuys <[hidden email]> writes:

> Let me explain my thinking more in detail. (I've CC'd Jan in case he wants
> to give further historical insights)
>
> The objective of GUB was to provide installers for LilyPond with the
> following requirements:
>
> 1) build for all platforms (OSX, Windows, Linux) with minimal effort,
> without having to manage windows and OSX machines.
> 2) all platforms use exactly the same versions of all software, so everyone
> has the same experience, and problems always reproduce across platforms.
>
> 1) implied setting up a cross compiling environment. This is a pain in the
> ass, because GCC must be compiled for each (host, target) pair separately.
> This is very slow and takes up a lot of space.

[Skipping over a number of details]

> In the intervening 15 years, many things have changed.
>
> A negative: apple licensing restrictions say that we can't cross-compile to
> OSX (although to be honest, I never read the terms & conditions for the
> XCode stuff I downloaded from apple back in the day. Maybe we've always
> been in violation of some T&C)

I think the "no crosscompilation" thing was some Jobs vendetta against
something specific he didn't like, and was introduced to thwart one
thing in particular.  I don't remember the details, but it's possible
that the ancient XCode is from before that event.  We have tried a bit
of finding out but unsuccessfully so.

A possibility would be to go to whatever became of OpenDarwin and use
that, but as far as I remember some of the font handling libraries in
use were actually tied to MacOSX in particular (Cocoa by now?).

> A posiitive: all major platforms run on x86 64-bit CPUs, and there are
> Open Source VM solutions for OSX and Windows. So, a VM running Linux
> on OSX or Windows can be almost as fast as running Linux
> directly. This makes GUB's cross compilation to other architectures
> superfluous.

In ancient times, I've had to be the guinea pig for VMware in a large
company and the experience was not thrilling: two of the typical
problems were that not the full power of a 2-CPU computer was made
available to Linux, only part of the RAM (and the basic RAM was getting
managed by Windows) and mapping the file system through a Windows file
system was really bad since Linux is so much faster in that regard.
That particularly concerned Git use which really depends on fast file
system operations.  Creating a virtual disk helped somewhat, creating an
actual partition for Linux helped quite more.  At some point of time I
started booting that partition, and, well, the experiment did not
deliver a lot more relevant data afterwards...

I have no experience with Docker and containers.  That was a full VM at
the time.

> A positive: Linux now supports namespaces, and in particular: file
> system namespaces. This makes it possible to build a native Linux
> package fully isolated from its host system, thus skipping GUB's
> laborious GCC setup.  This also lets you run a regular package manager
> on the namespaced file system, and this is essentially what Docker
> gives you: a way to install arbitrary set of dependencies and use
> them. This makes GUB's package manager superfluous.
>
> A positive: Docker is open source, and extremely popular
> technology. Docker does the hard work of packaging up VM + Docker for
> windows and OSX as installer, and providing a management/distribution
> mechanism for images.
>
> All in all, I think we could use Docker for distributing
> LilyPond. Then we would have a distribution mechanism that works for
> OSX and Windows, and it has no overhead to compiling LilyPond as
> normal Linux package. Because we can control the base image, we can
> precisely control the dependencies, and be sure that the binary
> behaves the same across all OSes. The best is that building releases
> won't require any arcane knowledge of cross-compilation quirks.

It suffers from the "someone needs to do it" syndrome and I have
absolutely no idea how smooth such executables will install and behave
compared to native ones on MacOSX and Windows.  I'd have to rely on
evaluation by the respective OS users to have an idea about that.

> GUIX is Jan's current project. I think it has some similarities to
> GUB, but it is focused on providing an environment where all binaries
> are reproducibly built from source. This is much more ambitious than
> GUB, and seems overkill compared to what we need for LilyPond. I think
> using it also entails many more compilation steps, which would makes
> the release process slow again.

I don't think it has an answer for the elephant in the room: Windows.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: Re: packaging lilypond as a docker container?

Mats Bengtsson-4

On 2020-01-22 11:36, David Kastrup wrote:

> Han-Wen Nienhuys <[hidden email]> writes:
> A posiitive: all major platforms run on x86 64-bit CPUs, and there are
> Open Source VM solutions for OSX and Windows. So, a VM running Linux
> on OSX or Windows can be almost as fast as running Linux
> directly. This makes GUB's cross compilation to other architectures
> superfluous.
> In ancient times, I've had to be the guinea pig for VMware in a large
> company and the experience was not thrilling: two of the typical
> problems were that not the full power of a 2-CPU computer was made
> available to Linux, only part of the RAM (and the basic RAM was getting
> managed by Windows) and mapping the file system through a Windows file
> system was really bad since Linux is so much faster in that regard.
> That particularly concerned Git use which really depends on fast file
> system operations.  Creating a virtual disk helped somewhat, creating an
> actual partition for Linux helped quite more.  At some point of time I
> started booting that partition, and, well, the experiment did not
> deliver a lot more relevant data afterwards...
>
> I have no experience with Docker and containers.  That was a full VM at
> the time.

I recently installed Ubuntu within the Windows Subsystem for Linux in
Windows 10 and installed the standard Linux package of Lilypond therein,
which runs very smoothly. Our sys. admins don't give us admin rights for
our Win 10 installation, so this is the first time for many years that
I've been able to run Lilypond on that computer. For the same reason, I
haven't been able to benchmark the processing speed against the Win
version of Lilypond, but at least it runs significantly faster than on
my old Linux machine, so it cannot be too bad.

However, just as any other solution based on Docker or other VMs, it's
not aimed at the average Windows user, especially if you want to run
Frescobaldi or any other GUI based editor, since Windows WSL doesn't
support X applications natively.

    /Mats


12