Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

classic Classic list List threaded Threaded
46 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Dev mailing list
this looks like a good idea, thanks!


https://codereview.appspot.com/553700043/diff/547740044/mf/invoke-mf2pt.sh
File mf/invoke-mf2pt.sh (right):

https://codereview.appspot.com/553700043/diff/547740044/mf/invoke-mf2pt.sh#newcode4
mf/invoke-mf2pt.sh:4: mf2pt1=$(realpath $1)
I only know `realpath` as a GNU make command.  Ditto for other commands
below in this script.

Is this script work in progress?

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

David Kastrup
Why would we want to move away from GNU make as a build system?

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Han-Wen Nienhuys-3
In reply to this post by Dev mailing list
On 2020/03/14 23:46:17, dak wrote:
> Why would we want to move away from GNU make as a build system?

The current build system is a per directory system, which is
fundamentally flawed. (See http://aegis.sourceforge.net/auug97.pdf).
This has several implications:

* we can't get enough parallelism in the doc build

* there are several underspecified dependencies (eg. if you change
something scripts/build/ , a build in mf/ will not pick up the change)

then there are problems with stepmake

* because each subdirectory has its own output directory  (eg. mf/out/
), a significant part of our build is involved with copying/symlinking
files so we don't have out-www/ in our HTML links.

* our build system relies heavily on GNU make magic. As a programming
language, GNU Make just isn't very good. It's not expressive, and it has
no typesystem. This makes it hard for people to work on the build
system.

* stepmake tried to be generic Make templating system (but did a bad job
at it). At the same time, the genericity is causing complexity

I have been experimenting with Ninja (https://ninja-build.org/), but
Ninja has a very restricted model. Make functions, wildcards, etc. are
not possible. In order to advance my experiments, I'm trying to clean up
the build so I can port more of the build over.

But even if we don't migrate to something else, cleaning up the build
makes it easier to maintain.

See https://github.com/hanwen/lilypond/tree/ninja if you want to try it
out.








Moving to something el

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Dev mailing list
In reply to this post by Dev mailing list

https://codereview.appspot.com/553700043/diff/545710043/mf/invoke-mf2pt.sh
File mf/invoke-mf2pt.sh (right):

https://codereview.appspot.com/553700043/diff/545710043/mf/invoke-mf2pt.sh#newcode4
mf/invoke-mf2pt.sh:4: mf2pt1=$(realpath $1)
Second try.  According to

 
https://unix.stackexchange.com/questions/136494/whats-the-difference-between-realpath-and-readlink-f
 
https://unix.stackexchange.com/questions/101080/realpath-command-not-found

it seems that today a `realpath` binary is available on most systems.
However, it is not mentioned in the suite of programs that autoconf uses
universally.  I thus recommend that we add a `configure` test for it –
or at least mention it in the list of LilyPond build requisites.

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

jonas.hahnfeld
In reply to this post by Dev mailing list
On 2020/03/14 23:57:48, hanwenn wrote:
> I have been experimenting with Ninja (https://ninja-build.org/), but
Ninja has a
> very restricted model. Make functions, wildcards, etc. are not
possible. In
> order to advance my experiments, I'm trying to clean up the build so I
can port
> more of the build over.

From the website: "it [Ninja] is designed to have its input files
generated by a higher-level build system". And that's to be taken
literally, the syntax is so low-level that it's no fun to read it, let
alone write. So big NACK from me for switching to Ninja.

Both Meson and CMake are able to generate files for Ninja, but Meson is
pretty new and I'm not sure yet whether it will become one of those
projects that goes away after a few years (see Scons). CMake would
certainly work, but has its own problems and corner cases. In sum we
should definitely cleanup the current build system and see where it gets
us.

What I'm not really happy with is replacing all of this with custom
Python scripts, however small they are. Skimming over scan-mf-deps.py,
that looks totally doable in around half the lines with a sh script.
What's the motivation to use Python for all of this?

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Han-Wen Nienhuys-3
On Sun, Mar 15, 2020 at 12:05 PM <[hidden email]> wrote:

>
> On 2020/03/14 23:57:48, hanwenn wrote:
> > I have been experimenting with Ninja (https://ninja-build.org/), but
> Ninja has a
> > very restricted model. Make functions, wildcards, etc. are not
> possible. In
> > order to advance my experiments, I'm trying to clean up the build so I
> can port
> > more of the build over.
>
> From the website: "it [Ninja] is designed to have its input files
> generated by a higher-level build system". And that's to be taken
> literally, the syntax is so low-level that it's no fun to read it, let
> alone write. So big NACK from me for switching to Ninja.

Have you looked at the code I wrote?   I know about Ninja, and in fact
I personally know its creator.

Using Ninja means generating a Ninja file. I looked for a bit at CMake
and Meson, but I'm not convinced they'll be a win; for one I'd have to
learn yet another poorly conceived and limited language. The LilyPond
build is highly custom (metafont, lilypond-book, texi2html), so it's
also not obvious that either they'll bring much to the table.

So I'm simply writing the build.ninja file from a Python script
directly, without any attempt to make it generic. We'll see how far
that brings me.

> Both Meson and CMake are able to generate files for Ninja, but Meson is
> pretty new and I'm not sure yet whether it will become one of those
> projects that goes away after a few years (see Scons). CMake would
> certainly work, but has its own problems and corner cases. In sum we
> should definitely cleanup the current build system and see where it gets
> us.
>
> What I'm not really happy with is replacing all of this with custom
> Python scripts, however small they are. Skimming over scan-mf-deps.py,
> that looks totally doable in around half the lines with a sh script.
> What's the motivation to use Python for all of this?

What's the motivation to use sh for all this?

You can certainly make it shorter with shell, but will it be
significantly simpler, easier to understand or easier to maintain?

I decided for Python in this case, because recursion seems like
something I don't want to do in a sh script.

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Carl Sorensen-3


On 3/15/20, 8:39 AM, "lilypond-devel on behalf of Han-Wen Nienhuys" <lilypond-devel-bounces+c_sorensen=[hidden email] on behalf of [hidden email]> wrote:

    On Sun, Mar 15, 2020 at 12:05 PM <[hidden email]> wrote:
    >
    >
    > What I'm not really happy with is replacing all of this with custom
    > Python scripts, however small they are. Skimming over scan-mf-deps.py,
    > that looks totally doable in around half the lines with a sh script.
    > What's the motivation to use Python for all of this?
   
    What's the motivation to use sh for all this?
   
    You can certainly make it shorter with shell, but will it be
    significantly simpler, easier to understand or easier to maintain?
   
    I decided for Python in this case, because recursion seems like
    something I don't want to do in a sh script.

Personally, I'm totally in favor of using python for creating ninja files.  We have a well-documented, well-maintained, powerful language.  No different flavors (bash, sh, dash, csh, ksh...).

Reading the docs,  ninja looks promising as a build system.  Seeing that it's used to build Chrome gives me some warm and fuzzy feelings.  Years ago I hoped we could use Scons to smooth things out, but it got swamped in performance issues.  We have some hope that Ninja won't suffer the same fate.

I'm excited to see what comes out of this.

Thanks,

Carl
 

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Werner LEMBERG
In reply to this post by Han-Wen Nienhuys-3
>> What's the motivation to use Python for all of this?
>
> What's the motivation to use sh for all this?
>
> You can certainly make it shorter with shell, but will it be
> significantly simpler, easier to understand or easier to maintain?
>
> I decided for Python in this case, because recursion seems like
> something I don't want to do in a sh script.

I agree with Han-Wen.

Jonas, the rule is very simple: Use whatever you think is best.  A
contributor invests time and has to make sure that the code works and
is well-documented.  But why should we enforce a certain programming
language (of the ones we already use, that is)?

For example, if I contributed a more complicated script, it would be
written in Perl...


    Werner

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

David Kastrup
Werner LEMBERG <[hidden email]> writes:

>>> What's the motivation to use Python for all of this?
>>
>> What's the motivation to use sh for all this?
>>
>> You can certainly make it shorter with shell, but will it be
>> significantly simpler, easier to understand or easier to maintain?
>>
>> I decided for Python in this case, because recursion seems like
>> something I don't want to do in a sh script.
>
> I agree with Han-Wen.
>
> Jonas, the rule is very simple: Use whatever you think is best.  A
> contributor invests time and has to make sure that the code works and
> is well-documented.  But why should we enforce a certain programming
> language (of the ones we already use, that is)?
>
> For example, if I contributed a more complicated script, it would be
> written in Perl...

I don't think "Use whatever you think is best." makes a lot of sense for
a community-maintained project.  It ends up leaving the project in an
incoherent state with lots of different approaches that made personal
sense to the person having started them.

Makefiles are an established technique well-known to most contributors.
Recursive python scripts for writing human-unreadable controlling files
in some build system that is the current fad of the year incompatible
with anything else: who is going to maintain that five years from now?

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

jonas.hahnfeld
In reply to this post by Dev mailing list
On 2020/03/15 14:39:06, hanwenn wrote:
> On Sun, Mar 15, 2020 at 12:05 PM <mailto:[hidden email]>
wrote:
> > On 2020/03/14 23:57:48, hanwenn wrote:
> > > I have been experimenting with Ninja (https://ninja-build.org/),
but
> > Ninja has a
> > > very restricted model. Make functions, wildcards, etc. are not
> > possible. In
> > > order to advance my experiments, I'm trying to clean up the build
so I
> > can port
> > > more of the build over.
> >
> > From the website: "it [Ninja] is designed to have its input files
> > generated by a higher-level build system". And that's to be taken
> > literally, the syntax is so low-level that it's no fun to read it,
let

> > alone write. So big NACK from me for switching to Ninja.
>
> Have you looked at the code I wrote?   I know about Ninja, and in fact
> I personally know its creator.
>
> Using Ninja means generating a Ninja file. I looked for a bit at CMake
> and Meson, but I'm not convinced they'll be a win; for one I'd have to
> learn yet another poorly conceived and limited language. The LilyPond
> build is highly custom (metafont, lilypond-book, texi2html), so it's
> also not obvious that either they'll bring much to the table.
>
> So I'm simply writing the build.ninja file from a Python script
> directly, without any attempt to make it generic. We'll see how far
> that brings me.

I did so now and ... does this mean you want to write yet another meta
build system? That sounds horrible to maintain.

> > What's the motivation to use Python for all of this?
>
> What's the motivation to use sh for all this?

Let me explain with my background of porting to Python 3: It was a
nightmare which leads me to the conclusion that Python scripts are not
easy to maintain over an extended period of time. I recently read
https://gregoryszorc.com/blog/2020/01/13/mercurial's-journey-to-and-reflections-on-python-3/
and this very much confirms this point.
sh instead is pretty much the same since decades and *very* portable.

On 2020/03/15 15:10:05, wl_gnu.org wrote:
> Jonas, the rule is very simple: Use whatever you think is best.  A
> contributor invests time and has to make sure that the code works and
> is well-documented.  But why should we enforce a certain programming
> language (of the ones we already use, that is)?

Yes, and others might disagree with a particular choice. For build
scripts, I think it's the wrong language (see above).

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

David Kastrup
In reply to this post by Carl Sorensen-3
Carl Sorensen <[hidden email]> writes:

> On 3/15/20, 8:39 AM, "lilypond-devel on behalf of Han-Wen Nienhuys"
> <lilypond-devel-bounces+c_sorensen=[hidden email] on behalf of
> [hidden email]> wrote:
>
>     On Sun, Mar 15, 2020 at 12:05 PM <[hidden email]> wrote:
>     >
>     >
>     > What I'm not really happy with is replacing all of this with custom
>     > Python scripts, however small they are. Skimming over scan-mf-deps.py,
>     > that looks totally doable in around half the lines with a sh script.
>     > What's the motivation to use Python for all of this?
>    
>     What's the motivation to use sh for all this?
>    
>     You can certainly make it shorter with shell, but will it be
>     significantly simpler, easier to understand or easier to maintain?
>    
>     I decided for Python in this case, because recursion seems like
>     something I don't want to do in a sh script.
>
> Personally, I'm totally in favor of using python for creating ninja
> files.  We have a well-documented, well-maintained, powerful language.
> No different flavors (bash, sh, dash, csh, ksh...).

Regarding the "no different flavors" mantra, you might want to speak
with the people who were (and still are) involved with the Python2 to
Python3 porting effort.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Han-Wen Nienhuys-3
In reply to this post by jonas.hahnfeld
On Sun, Mar 15, 2020 at 5:04 PM <[hidden email]> wrote:

>
> On 2020/03/15 14:39:06, hanwenn wrote:
> > On Sun, Mar 15, 2020 at 12:05 PM <mailto:[hidden email]>
> wrote:
> > > On 2020/03/14 23:57:48, hanwenn wrote:
> > > > I have been experimenting with Ninja (https://ninja-build.org/),
> but
> > > Ninja has a
> > > > very restricted model. Make functions, wildcards, etc. are not
> > > possible. In
> > > > order to advance my experiments, I'm trying to clean up the build
> so I
> > > can port
> > > > more of the build over.
> > >
> > > From the website: "it [Ninja] is designed to have its input files
> > > generated by a higher-level build system". And that's to be taken
> > > literally, the syntax is so low-level that it's no fun to read it,
> let
> > > alone write. So big NACK from me for switching to Ninja.
> >
> > Have you looked at the code I wrote?   I know about Ninja, and in fact
> > I personally know its creator.
> >
> > Using Ninja means generating a Ninja file. I looked for a bit at CMake
> > and Meson, but I'm not convinced they'll be a win; for one I'd have to
> > learn yet another poorly conceived and limited language. The LilyPond
> > build is highly custom (metafont, lilypond-book, texi2html), so it's
> > also not obvious that either they'll bring much to the table.
> >
> > So I'm simply writing the build.ninja file from a Python script
> > directly, without any attempt to make it generic. We'll see how far
> > that brings me.
>
> I did so now and ... does this mean you want to write yet another meta
> build system? That sounds horrible to maintain.

It doesn't have to be a *meta* build system, because it doesn't have
to cater to anything but LilyPond. That makes me confident that it
will be much simpler than any of the existing systems.

> > > What's the motivation to use Python for all of this?
> >
> > What's the motivation to use sh for all this?
>
> Let me explain with my background of porting to Python 3: It was a
> nightmare which leads me to the conclusion that Python scripts are not
> easy to maintain over an extended period of time. I recently read
> https://gregoryszorc.com/blog/2020/01/13/mercurial's-journey-to-and-reflections-on-python-3/
> and this very much confirms this point.
> sh instead is pretty much the same since decades and *very* portable.

I very much agree with you, but out of the existing language within
our tree (Perl, Python, Shell, C++, Make), it is the least worst
option. Make is too limited (see the mess that our makefiles are
today), Shell has all the problems of Make. We can't use C++ in the
build system, and Perl is a godawful mess of line-noise.

What do you suggest instead?

For all the code where I have a choice of language, I nowadays use Go,
and I'd be happy to do a (meta)build system in Go.

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Han-Wen Nienhuys-3
In reply to this post by David Kastrup
On Sun, Mar 15, 2020 at 5:03 PM David Kastrup <[hidden email]> wrote:
> Makefiles are an established technique well-known to most contributors.
> Recursive python scripts for writing human-unreadable controlling files

What recursive script are you talking about ?

> in some build system that is the current fad of the year incompatible
> with anything else

Are you talking about Ninja? You realize that it is the standard
infrastructure of virtually all modern build systems, including
Meson/CMake. The chrome browser is built with Ninja, as is the Android
platform, and the Clang compiler. Ninja was introduced in 2012, so
it's not exactly the fad of the year.

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

jonas.hahnfeld
In reply to this post by Dev mailing list
On 2020/03/15 16:16:06, hanwenn wrote:
> On Sun, Mar 15, 2020 at 5:04 PM <mailto:[hidden email]>
wrote:
> > I did so now and ... does this mean you want to write yet another
meta
> > build system? That sounds horrible to maintain.
>
> It doesn't have to be a *meta* build system, because it doesn't have
> to cater to anything but LilyPond. That makes me confident that it
> will be much simpler than any of the existing systems.

It's a (build) system that generates files for a build system. I'd call
this "meta" but this is likely a matter of definition.

> I very much agree with you, but out of the existing language within
> our tree (Perl, Python, Shell, C++, Make), it is the least worst
> option. Make is too limited (see the mess that our makefiles are
> today), Shell has all the problems of Make. We can't use C++ in the
> build system, and Perl is a godawful mess of line-noise.
>
> What do you suggest instead?

Not doing this at all.
In cases like this, I begin to question the very fundamental
assumptions. In this case: Do we really need to dynamically generate the
dependencies if there's really no tool for it? My answer is "no" after
seeing that the last change of 'include' statements was in 2013 (if you
count a2f44bbf0e; otherwise maybe earlier). As such, how about just
making them static? See https://codereview.appspot.com/555440062 to get
the idea.

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

jonas.hahnfeld
In reply to this post by Dev mailing list
On 2020/03/15 18:11:44, hanwenn wrote:
> On Sun, Mar 15, 2020 at 5:03 PM David Kastrup <mailto:[hidden email]>
wrote:
> > Makefiles are an established technique well-known to most
contributors.
> > Recursive python scripts for writing human-unreadable controlling
files
>
> What recursive script are you talking about ?
>
> > in some build system that is the current fad of the year
incompatible
> > with anything else
>
> Are you talking about Ninja? You realize that it is the standard
> infrastructure of virtually all modern build systems, including
> Meson/CMake. The chrome browser is built with Ninja, as is the Android
> platform, and the Clang compiler.

Begin a LLVM committer, let me disagree with this one: LLVM/Clang uses
CMake which is *able* to generate Ninja. Or Makefiles. Or files for
VisualStudio. (Did I forget one "generator" target?)

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Han-Wen Nienhuys-3
In reply to this post by jonas.hahnfeld
On Sun, Mar 15, 2020 at 7:14 PM <[hidden email]> wrote:

> > On Sun, Mar 15, 2020 at 5:04 PM <mailto:[hidden email]>
> wrote:
> > > I did so now and ... does this mean you want to write yet another
> meta
> > > build system? That sounds horrible to maintain.
> >
> > It doesn't have to be a *meta* build system, because it doesn't have
> > to cater to anything but LilyPond. That makes me confident that it
> > will be much simpler than any of the existing systems.
>
> It's a (build) system that generates files for a build system. I'd call
> this "meta" but this is likely a matter of definition.

Python + Ninja seems like its a system with less surprises, and
therefore, better maintainability than autoconf + make.

> > I very much agree with you, but out of the existing language within
> > our tree (Perl, Python, Shell, C++, Make), it is the least worst
> > option. Make is too limited (see the mess that our makefiles are
> > today), Shell has all the problems of Make. We can't use C++ in the
> > build system, and Perl is a godawful mess of line-noise.
> >
> > What do you suggest instead?
>
> Not doing this at all.
> In cases like this, I begin to question the very fundamental
> assumptions. In this case: Do we really need to dynamically generate the
> dependencies if there's really no tool for it? My answer is "no" after
> seeing that the last change of 'include' statements was in 2013 (if you
> count a2f44bbf0e; otherwise maybe earlier). As such, how about just
> making them static? See https://codereview.appspot.com/555440062 to get
> the idea.

FWIW, at some point the dependency generation was part of
mf-to-table.py, which generated from the log file (using Python). I
don't know why it was moved to be a make macro.

If we think we're not going to change the mf files, why not just check
in the OTF files?

(tongue in cheek, but you get the idea).

In general leaving functionality intact leads to less discussion,
which lets me move faster.

In this case in particular, though, I think that as long as we're
building the fonts from mf using make, we should generate correct
dependencies. The build system is kind of a mess, but I think
hard-coding hardcoding dependencies is strictly worse.

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Han-Wen Nienhuys-3
In reply to this post by jonas.hahnfeld
On Sun, Mar 15, 2020 at 7:14 PM <[hidden email]> wrote:

> > I very much agree with you, but out of the existing language within
> > our tree (Perl, Python, Shell, C++, Make), it is the least worst
> > option. Make is too limited (see the mess that our makefiles are
> > today), Shell has all the problems of Make. We can't use C++ in the
> > build system, and Perl is a godawful mess of line-noise.
> >
> > What do you suggest instead?
>
> Not doing this at all.
> In cases like this, I begin to question the very fundamental
> assumptions. In this case: Do we really need to dynamically generate the

You didn't answer my question, but I realize it was imperfectly stated.

My question is: what overall architecture do you propose for the build system?

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

David Kastrup
In reply to this post by Han-Wen Nienhuys-3
Han-Wen Nienhuys <[hidden email]> writes:

> On Sun, Mar 15, 2020 at 5:03 PM David Kastrup <[hidden email]> wrote:
>> Makefiles are an established technique well-known to most contributors.
>> Recursive python scripts for writing human-unreadable controlling files
>
> What recursive script are you talking about ?

It is you who stated the following:

> I decided for Python in this case, because recursion seems like
> something I don't want to do in a sh script.

So apparently you want to use Python for recursive scripting, or why
bring it up?

>> in some build system that is the current fad of the year incompatible
>> with anything else
>
> Are you talking about Ninja? You realize that it is the standard
> infrastructure of virtually all modern build systems, including
> Meson/CMake.

It is one backend of several for those, and the more common CMake
predates Ninja.

> The chrome browser is built with Ninja, as is the Android platform,

The Ninja website itself states "parts of Android".  It also states
"Where other build systems are high-level languages Ninja aims to be an
assembler." and "it is designed to have its input files generated by a
higher-level build system,".

So your Python scripts are intended to eventually constitute another
higher-level build system instead of reusing an existing one, with Ninja
operating at the assembler level?

This just screams like being slated to become a very person-bound
dependency in the form of an idiosyncratic build system designed from
scratch in essential parts, and eventually a liability.

The problems with Make you stated you wanted to address with that are
more problems with the _current_ Makefile architecture and dependency
setup rather than inherent with Make.  While a complete switch of tools
has the advantage of making it easier to abandon parts of the design
that are equal parts legacy and liability, the principal problems of
incomplete dependencies particularly cross-directory, incomplete job
control, and somewhat incrutable setup of what rules belong where in the
stepmake system are not an inherent problem of Make but of what we have
built around it.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

jonas.hahnfeld
In reply to this post by Dev mailing list
On 2020/03/15 18:52:27, hanwenn wrote:
> > Not doing this at all.
> > In cases like this, I begin to question the very fundamental
> > assumptions. In this case: Do we really need to dynamically generate
the
> > dependencies if there's really no tool for it? My answer is "no"
after
> > seeing that the last change of 'include' statements was in 2013 (if
you
> > count a2f44bbf0e; otherwise maybe earlier). As such, how about just
> > making them static? See https://codereview.appspot.com/555440062 to
get

> > the idea.
>
> FWIW, at some point the dependency generation was part of
> mf-to-table.py, which generated from the log file (using Python). I
> don't know why it was moved to be a make macro.
>
> If we think we're not going to change the mf files, why not just check
> in the OTF files?
>
> (tongue in cheek, but you get the idea).

IMHO you need to differentiate between changes to mf files (which I
think we should enable) and changes to the *structure* of the mf files.
The latter gets slightly more complicated with static dependencies. But
I don't expect many of them, much to the contrary of '#include's in C++.
_If_ dependencies become a problem in the future _and_ we're still using
make that point, we can of course revisit this issues.

On 2020/03/15 18:54:44, hanwenn wrote:
> On Sun, Mar 15, 2020 at 7:14 PM <mailto:[hidden email]>
wrote:
> > > I very much agree with you, but out of the existing language
within
> > > our tree (Perl, Python, Shell, C++, Make), it is the least worst
> > > option. Make is too limited (see the mess that our makefiles are
> > > today), Shell has all the problems of Make. We can't use C++ in
the
> > > build system, and Perl is a godawful mess of line-noise.
> > >
> > > What do you suggest instead?
> >
> > Not doing this at all.
> > In cases like this, I begin to question the very fundamental
> > assumptions. In this case: Do we really need to dynamically generate
the
>
> You didn't answer my question, but I realize it was imperfectly
stated.
>
> My question is: what overall architecture do you propose for the build
system?

Using whatever makes sense from the GNU build system and reduce the
maintenance burden of the build system as much as possible. FWIW I fully
agree with David that the current shortcomings originate from what
LilyPond builds around Autoconf, GNU make et al. Using standard
solutions (aclocal, possibly Automake for dependencies of C++ files)
should enable us to reuse what others have already solved.

https://codereview.appspot.com/553700043/

Reply | Threaded
Open this post in threaded view
|

Re: scripts/build/scan-mf-deps: script to generate MF dependencies (issue 553700043 by hanwenn@gmail.com)

Han-Wen Nienhuys-3
On Sun, Mar 15, 2020 at 8:44 PM <[hidden email]> wrote:

> On 2020/03/15 18:54:44, hanwenn wrote:
> > On Sun, Mar 15, 2020 at 7:14 PM <mailto:[hidden email]>
> wrote:
> > > > I very much agree with you, but out of the existing language
> within
> > > > our tree (Perl, Python, Shell, C++, Make), it is the least worst
> > > > option. Make is too limited (see the mess that our makefiles are
> > > > today), Shell has all the problems of Make. We can't use C++ in
> the
> > > > build system, and Perl is a godawful mess of line-noise.
> > > >
> > > > What do you suggest instead?
> > >
> > > Not doing this at all.
> > > In cases like this, I begin to question the very fundamental
> > > assumptions. In this case: Do we really need to dynamically generate
> the
> >
> > You didn't answer my question, but I realize it was imperfectly
> stated.
> >
> > My question is: what overall architecture do you propose for the build
> system?
>
> Using whatever makes sense from the GNU build system and reduce the
> maintenance burden of the build system as much as possible. FWIW I fully
> agree with David that the current shortcomings originate from what
> LilyPond builds around Autoconf, GNU make et al. Using standard
> solutions (aclocal, possibly Automake for dependencies of C++ files)
> should enable us to reuse what others have already solved.

I don't disagree with the analysis of the root cause, but getting rid
of the recursive make for a non-recursive one means a complete
rewrite, and not one that can be done incrementally. When you do a
complete rewrite, you might as well reconsider the fundamental
assumptions, one of which is "use the GNU build stack".

I have plenty of experience with Automake, GNU make, building
lilypond, and build systems in general. My conclusion of this
experience is that it is a pretty poor stack, and for that reason I am
experimenting with something that is completely different.

I have simplfied the change under review. I'm sure there is a sed
incantation that could achieve the same, but I don't want to bother
learning it. If you can suggest one, I'd be happy to use it.

--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

123