Identify included files

classic Classic list List threaded Threaded
56 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 18 May, 2020, at 11:30 AM, David Wright <[hidden email]> wrote:
>
> If you were compiling a C program foo.c into an executable, what you
> would be trying to avoid is recompiling fnbar.c into fnbar.o over
> and over again (and all of its similar companions).
>
> Can you explain which of your files are the equivalents of fnbar.o.
> (In python, for another example, they'd be your *.pyc files, and
> python looks after all this housekeeping itself.)

As I see it, the filetype equivalencies roughly map as this:

.ily -> .h
.ly -> .c
.pdf -> .o

To this I would also add one more file type for storing dependency information
.dly -> .d


> On 18 May, 2020, at 11:35 AM, David Wright <[hidden email]> wrote:
>
> If you follow some simple rules in your source layout, constructing
> such a list might be most straightforward to do in the shell, using
> grep. For example,
>
> $ grep -e '^[[:space:]]*\\include ' foo-top-level.ly
>
> will find all the includes in the source, restricting it to the start
> of lines, so that you can avoid false hits from, say, internal
> documentation or test includes.
>
> You would insert the resulting filenames into two lists, files visited
> and all files. Iterate on the files list, skipping those visited,
> until the files to visit reaches zero (lists are of equal length).
>
> One complication is applying the search of libraries in the same
> manner as LP, particularly if you rely on the order of searching.
> Another is if you're using external libraries where you have no
> control over their include format.
>
> If you're not comfortable with shell programming, obviously it would
> be preferable to use a language like Python or Perl for coding.
>
> The resulting list could be used for, say, making a tarball of source
> files for distribution, or for driving make to produce PDFs that are
> assembled by, say, TeX programs into final documents.

My original thoughts were to write something in Python or bash and it probably would have looked something like what you propose.  Being a firm believer in not reinventing the wheel whenever possible, I decided to ask the list to see if this had been done already first.


>
> On 18 May, 2020, at 12:22 PM, antlists <[hidden email]> wrote:
>
> Okay, does that mean you want a list of all the lilypond include files, or just your own?
>
> If you're digging into the internals of lilypond, then it'll be a big job, and makes it dangerous to upgrade lilypond. Because I assume that means you're actually modifying lilypond's internal structure. What I do, anything of lilypond's that I modify, I split it out into my directory structure, and then over-ride lilypond's version from within my source. That way, what happens to lilypond during an upgrade is no concern of mine (other than changes to syntax which mess up my source :-)
>
> If it's just your own files you're concerned about, then grep will probably do the trick. I don't know how to recurse directories, but making it search all the files in a directory for "include", and tell you which file it found that line in, is pretty basic.
>
> You can then read up on make for how to create those virtual objects, and use an editor to put them in your makefile. Not totally automated, I admit, but fairly simple grunt-work. The sort of thing I could have written a BASIC program to do in 10 minutes, except I don't have access to a BASIC I understand how to use any more - ironic - as computers have got easier to use, so the programming languages have got harder...
>
I’m not playing with the lilypond internals (and don’t need them in my list of prerequisites), but I do occasionally have files which make use of oll packages (I’ve been playing with the Edition Engraver recently).  Those packages have their own hierarchy outside my project hierarchy.  Further, the location of their hierarchy may be different on different machines (I always try to set things up with collaboration in mind, even when I’m not currently collaborating with anyone).  As a result, scripts which don’t understand how LilyPond finds files and simply look for “\include” statements are likely to come up short and not find all prerequisites.



> On 18 May, 2020, at 11:47 AM, Urs Liska <[hidden email]> wrote:
>
> I'm pretty sure that the only two ways to reasonably go forward is to
> either invoke LilyPond to produce this list or to use a library that
> already knows how LilyPond works. As I mentioned somewhere at the top
> of this thread, Frescobaldi knows pretty well how to recurse into the
> include files.
>
> I would write a tool in Python that calls python-ly (well, probably
> quickly, it's designated successor (
> https://github.com/frescobaldi/quickly)) to produce the list of
> dependencies and feed that into Make.
I’ve reached a similar conclusion.  Right now I’m going forward with the lilypond route simply because David K.’s parse-only.ly means I’m closer to that solution than I am with python-ly or quickly (as I’ve never really worked with these packages before and would be starting from scratch).

In fact, I’ve attached my first go at it.  I’ve modified parse-only.ly to solve the context mod problem and to change the output format.  First off, parse-only.ly itself is removed from the list (as it’s not actually a prerequisite for building the pdf).  Second, the output is formatted in a make-friendly way with the files separated by a space (instead of a new line) and the name of the expected pdf prepended (and separated from the file list by a colon)

The makefile then contains a rule for creating .dly files from .ly files; an include statement for reading in all the dly files; and a rule for creating pdf files from .ly files.  My preliminary testing has everything working, but I’m fairly new to writing makefiles myself, so I’d appreciate anyone with more experience giving this a once over and critique.


✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ

makefile (390 bytes) Download Attachment
parse-only.ly (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
As an exercise in Scheme programming (and because I’m something of a perfectionist), I’m working on improving the output when using parse-only.ly by having it account for `--format` flags and the various backends when constructing the target portion of the make rule (the version I previously posted always assumes the target is pdf).  To that end I’ve been exploring some of the scheme functions and have come across the ly:output-formats and ly:parser-output-name as welll as (ly:get-option 'backend).  However, I have a few questions:

ly:output-formats is a list whose contents are determined by the `--format` flags.  However, when eps format is requested, ps, not eps is what’s added to the list.  To determine that the output is eps I then also need to look at (ly:get-option 'backend), which will be eps or ps (or svg or scm, but they only produce a single file format, regardless of --format flags).  I’ve also noticed that if both ps and eps files are requested with flags, then apparently only the eps file is produced, though ly:output-formats will have two ps entries.  So, my question is can lilypond ever produce both ps and eps file from a single run, or does the use of the eps backend preclude the production of ps files?

ly:parser-output-name stores the base name of the output file, but doesn’t contain directory information about where the output file is supposed to go (unlike the list of files in ly:source-files, which do contain the directory information about the location of the source files).  This make sense because the output defaults to the current working directory when lilypond is called.  However, if I use the --output option, I can force LilyPond to switch working directories before processing the file (and thus change the destination of the output).  However, I cannot find where that directory information is stored.  Is it stored somewhere (if so, where)?

Use of the -dbackend=eps (or setting the backend to eps in the file) also causes the production of several other files (*-systems.tex, *-systems.texi, *-systems.count, and the eps and pdf files for each system) while -E (or --eps) does not.  So far, however, I haven’t been to determine the difference between these two situations.  Is there some variable (or scheme function) that can be used to figure out if these files are being produced (and how many systems there are)?

Finally, what about midi?  I have yet to find a scheme function that will tell me if the file being parsed will produce a midi file when typeset.  Is there anything that would do that?

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 20 May, 2020, at 12:05 PM, Fr. Samuel Springuel <[hidden email]> wrote:
>
> Use of the -dbackend=eps (or setting the backend to eps in the file) also causes the production of several other files (*-systems.tex, *-systems.texi, *-systems.count, and the eps and pdf files for each system) while -E (or --eps) does not.  So far, however, I haven’t been to determine the difference between these two situations.  Is there some variable (or scheme function) that can be used to figure out if these files are being produced (and how many systems there are)?

Okay, I found the answer to this one.  The aux-files option (accessed with (ly:get-option 'aux-files)) indicates that these files are being created.  

It’s also occurred to me that the only way to determine the number of systems is to actually do the typesetting.  Any attempt, therefore, to determine the targets and prerequisites automatically without typesetting the document cannot know exactly how many there are and has to be written from that perspective.  This part of the question was akin to asking someone how many apples are in a bag while keeping the bag hidden from them.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
In reply to this post by Fr. Samuel Springuel
On Mon 18 May 2020 at 14:32:31 (-0400), Fr. Samuel Springuel wrote:

> > On 18 May, 2020, at 11:30 AM, David Wright <[hidden email]> wrote:
> >
> > If you were compiling a C program foo.c into an executable, what you
> > would be trying to avoid is recompiling fnbar.c into fnbar.o over
> > and over again (and all of its similar companions).
> >
> > Can you explain which of your files are the equivalents of fnbar.o.
> > (In python, for another example, they'd be your *.pyc files, and
> > python looks after all this housekeeping itself.)
>
> As I see it, the filetype equivalencies roughly map as this:
>
> .ily -> .h
> .ly -> .c
> .pdf -> .o

So the final output is constructed¹ from several PDF files, and these
are what you don't want to recompile each time the constructing takes
place. So what program(s) are you using to construct the final output?

I don't understand your equivalence between .ily and .h files. The
.ily file(s) can contain just as much code as the .ly file(s), whereas
.h files don't contain any code at all (in the sense of producing
executable code for the next stage.

So I'm unable to see which files you avoid compiling, and in what form
their compiled derivatives are stored between runs.

> To this I would also add one more file type for storing dependency information
> .dly -> .d

¹ I've avoided the word assemble for obvious reasons.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 21/05/2020 01:49, David Wright wrote:
> I don't understand your equivalence between .ily and .h files. The
> .ily file(s) can contain just as much code as the .ly file(s), whereas
> .h files don't contain any code at all (in the sense of producing
> executable code for the next stage.

a) just like .h files, .ily files can't be passed directly to a compiler
for compilation.

b) who says .h files don't produce executable code? They often contain
code, sometimes entire functions.

I understand the equivalence exactly - it's what you DO to the file.

You're thinking "what you *should* put in the file". Since when have
people done what they *should*?

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Kastrup
antlists <[hidden email]> writes:

> On 21/05/2020 01:49, David Wright wrote:
>> I don't understand your equivalence between .ily and .h files. The
>> .ily file(s) can contain just as much code as the .ly file(s), whereas
>> .h files don't contain any code at all (in the sense of producing
>> executable code for the next stage.
>
> a) just like .h files, .ily files can't be passed directly to a
> compiler for compilation.
>
> b) who says .h files don't produce executable code? They often contain
> code, sometimes entire functions.

The point of .h files is that they are intended to be included by more
than one compilation unit resulting in an executable.

As such, they are limited with regard to the code they can contain.
They can contain

a) inline function definitions: those can occur in multiple compilation
units as long as they are identical.
b) static inline: do not even need to be identical, are per-unit.
c) static data definitions, are per-unit.
d) template function definitions

I don't think that .ly/.ily is all that comparable here...

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 21/05/2020 14:36, David Kastrup wrote:

> antlists <[hidden email]> writes:
>
>> On 21/05/2020 01:49, David Wright wrote:
>>> I don't understand your equivalence between .ily and .h files. The
>>> .ily file(s) can contain just as much code as the .ly file(s), whereas
>>> .h files don't contain any code at all (in the sense of producing
>>> executable code for the next stage.
>>
>> a) just like .h files, .ily files can't be passed directly to a
>> compiler for compilation.
>>
>> b) who says .h files don't produce executable code? They often contain
>> code, sometimes entire functions.
>
> The point of .h files is that they are intended to be included by more
> than one compilation unit resulting in an executable.

Which is exactly how I use .ily files ...
>
> As such, they are limited with regard to the code they can contain.
> They can contain
>
> a) inline function definitions: those can occur in multiple compilation
> units as long as they are identical.

You're describing the way I define variables that contain all my notes.
You're also describing how I define the musical structure.

> b) static inline: do not even need to be identical, are per-unit.
> c) static data definitions, are per-unit.
> d) template function definitions

I redefine the header function - I've NEVER seen a header laid out the
way lilypond does it, apart from parts produced by lilypond ...

Plus I have my own dynamics definition file that extends lilypond's ones
>
> I don't think that .ly/.ily is all that comparable here...
>
The way I use .ily files EXACTLY matches your definition of a .h file...
how on earth do you use them?

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
In reply to this post by antlists
On Thu 21 May 2020 at 13:57:00 (+0100), antlists wrote:
> On 21/05/2020 01:49, David Wright wrote:
> > I don't understand your equivalence between .ily and .h files. The
> > .ily file(s) can contain just as much code as the .ly file(s), whereas
> > .h files don't contain any code at all (in the sense of producing
> > executable code for the next stage.
>
> a) just like .h files, .ily files can't be passed directly to a
> compiler for compilation.

Why do you say that?

> b) who says .h files don't produce executable code? They often contain
> code, sometimes entire functions.
>
> I understand the equivalence exactly - it's what you DO to the file.

Good, that means I have a second source of edification as to what the
intent of this analysis is.

> You're thinking "what you *should* put in the file". Since when have
> people done what they *should*?

Letting that pass, let's say we have a source file foo.c and it needs
an extraordinary quantity of functionality that's contained in bar.c.
We put the latter's function declarations into bar.h and compile
bar.{c,h}¹ into bar.o.

Now we can compile foo.c (which #includes bar.h) over and over while
debugging it, all without ever recompiling bar.c. foo.c knows all it
needs to know about bar.c because of bar.h's declarations.

I now have a suite.ly that typesets my Suite for flute and piano.
It \includes nine files: flute{I,II,III}.ily, piano{I,II,III}.ily
(the notes), and flute-part.ily, piano-part.ily and complete-score.ily,
(my \score templates). It also \includes {marks,dyns,fingers}.ily that
contain LP code for my preferred styles of markings, dynamics and fingering.

I have a file, suite.dep, which contains the information that suite.ly
depends on these files to produce three targets, {flute,piano,suite}.pdf
The same file might also indicate that I need just flute{I,II,III}.ily,
flute-part.ily and {marks,dyns,fingers}.ily for just a flute.pdf.

What do I write in a top-level theFlute.ly file to produce this
flute.pdf and how does it benefit from the suite.sep information?
How do I use this information profitably, in the way that bar.h did?

¹ I'm abbreviating with shell's "brace expansion" notation.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 21/05/2020 16:36, David Wright wrote:

> On Thu 21 May 2020 at 13:57:00 (+0100), antlists wrote:
>> On 21/05/2020 01:49, David Wright wrote:
>>> I don't understand your equivalence between .ily and .h files. The
>>> .ily file(s) can contain just as much code as the .ly file(s), whereas
>>> .h files don't contain any code at all (in the sense of producing
>>> executable code for the next stage.
>>
>> a) just like .h files, .ily files can't be passed directly to a
>> compiler for compilation.
>
> Why do you say that?

Because while it may not be globally true, I have tried accidentally to
do things like "lilypond voiceTrombone.ily" and it blew up quite
spectacularly. I would expect most of mine, and in truth most other
peoples', .ily files to blow up if fed straight to lilypond.

And if they do compile, the output probably isn't much use ... :-)

>
>> b) who says .h files don't produce executable code? They often contain
>> code, sometimes entire functions.
>>
>> I understand the equivalence exactly - it's what you DO to the file.
>
> Good, that means I have a second source of edification as to what the
> intent of this analysis is.
>
>> You're thinking "what you *should* put in the file". Since when have
>> people done what they *should*?
>
> Letting that pass, let's say we have a source file foo.c and it needs
> an extraordinary quantity of functionality that's contained in bar.c.
> We put the latter's function declarations into bar.h and compile
> bar.{c,h}¹ into bar.o.

Does bar.c contain a main{} definition?
>
> Now we can compile foo.c (which #includes bar.h) over and over while
> debugging it, all without ever recompiling bar.c. foo.c knows all it
> needs to know about bar.c because of bar.h's declarations.

Can you now link bar.o into foo? I honestly don't know - I don't know
how a linker will behave if fed two main{} functions ... ?

>
> I now have a suite.ly that typesets my Suite for flute and piano.
> It \includes nine files: flute{I,II,III}.ily, piano{I,II,III}.ily
> (the notes), and flute-part.ily, piano-part.ily and complete-score.ily,
> (my \score templates). It also \includes {marks,dyns,fingers}.ily that
> contain LP code for my preferred styles of markings, dynamics and fingering.
>
> I have a file, suite.dep, which contains the information that suite.ly
> depends on these files to produce three targets, {flute,piano,suite}.pdf
> The same file might also indicate that I need just flute{I,II,III}.ily,
> flute-part.ily and {marks,dyns,fingers}.ily for just a flute.pdf.
>
> What do I write in a top-level theFlute.ly file to produce this
> flute.pdf and how does it benefit from the suite.sep information?
> How do I use this information profitably, in the way that bar.h did?
>
> ¹ I'm abbreviating with shell's "brace expansion" notation.
>
> Cheers,
> David.
>
Now we get to the meat of the matter. You've put bar.c's *declarations*
into bar.h. Does lilypond even have declarations? Also, it's perfectly
normal practice to put *definitions* in .h files - static variables,
macros, inline functions, etc etc.

I put most of my lilypond variable *definitions* into .ily files. Yup,
.h files *tend* to contain declarations not definitions, but as I say, I
don't think lilypond syntax has  declarations?

So yes, am I right in thinking you view .ily files as closer to .o
files? With lilypond closer to a linker than a compiler?

At the end of the day, I see it as .h files can contain declarations,
.ily files contain declarations. .h files are included, .ily files are
included. Some languages (like lilypond) are monolithic and don't really
have the concept of libraries. So I find it extremely easy to view
.ly/.ily as being similar to .c/.h. Not the same, because lilypond is
not gcc ... :-)

There's no one absolute view - you see things different to me, doesn't
mean you (or me) is right.

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
In reply to this post by David Kastrup
> On 21 May, 2020, at 9:36 AM, David Kastrup <[hidden email]> wrote:
>
> antlists <[hidden email]> writes:
>
>> On 21/05/2020 01:49, David Wright wrote:
>>> I don't understand your equivalence between .ily and .h files. The
>>> .ily file(s) can contain just as much code as the .ly file(s), whereas
>>> .h files don't contain any code at all (in the sense of producing
>>> executable code for the next stage.
>>
>> a) just like .h files, .ily files can't be passed directly to a
>> compiler for compilation.
>>
>> b) who says .h files don't produce executable code? They often contain
>> code, sometimes entire functions.
>
> The point of .h files is that they are intended to be included by more
> than one compilation unit resulting in an executable.
>
> As such, they are limited with regard to the code they can contain.
> They can contain
>
> a) inline function definitions: those can occur in multiple compilation
> units as long as they are identical.
> b) static inline: do not even need to be identical, are per-unit.
> c) static data definitions, are per-unit.
> d) template function definitions
>
> I don't think that .ly/.ily is all that comparable here...
>
> --
> David Kastrup
>

The analogy is not perfect, but then no analogy is (otherwise it wouldn’t be an analogy).  Lilypond doesn’t enforce a distinction between .ly and .ily files in the same way that C enforces the distinction between .h and .c files.  Further, .pdf files are stand alone in a way that .o files are not.  The linker level which combines .o files into an executable would have as its closest approximation in the LilyPond world assembling several pdfs into some kind of portfolio.  And this is another point where the analogy breaks in that an executable cannot contain incompatible .o files but a portfolio can contain pdfs of scores with vastly different styles.  The result may not be as professional looking, but it’s possible.

Here are the parts of the analogy that do work:

1) .ily files cannot produce scores.  Sure, I can call LilyPond on them directly, but regardless of the options I use, no score will ever result.  This may be purely by convention (as in, LilyPond itself doesn’t enforce this, it really couldn’t care what the extension on its input files is), but it’s a convention which I rigorously apply in the files I write.  In this respect, .ily files are like .h files which also cannot, by themselves, ever produce a .o file. For .h files, at least a dummy.c file is needed to include the .h file.  My .ily files likewise need a .ly file to include them to produce a score.

2) .ly files do produce scores.  If I call LilyPond on an .ly file, then I expect a score to come out (in pdf format by default, but other formats are possible too if I use the right options or flags).  Again, this is primarily a convention, as LilyPond won’t throw an error on a .ly file which doesn’t produce a score (but has no other problems).  However, it is a convention that I strictly apply to my files.  In this respect .ly files are like .c files, which can be compiled to create .o files.

3) Further the relation between the .ly and .ily files is similar to the relation between .c and .h files.  .ly (.c) files do not, themselves, depend on the .ily (.h) files that they include.  There is no build process which creates a .ly (.c) file from the included .ily (.h) files.  However, if I make a change to a .ily (.h) file, then the .pdf (.o) file which is produced when LilyPond (GCC) compiles a .ly (.c) file which includes that changed file is out of date and needs to be recompiled. In this regard the directed acyclic graph (DAG) which illustrates these relationships is the same, just with one less level (unless I make use of a pdf merge tool to replace the linker level).


My goal is to inform make of these relationships so that when I tweak a LilyPond source, only the scores affected by that source are recompiled.  This may be several pdfs, but it’s not all of them, even when I tweak an ily file because not all ily files are included in all scores.  Furthermore, the more I work on this project, the more I tend to split up the “formatting” file so that I actually have several, each of which is responsible for a single aspect of the formatting.  This makes it easier for me to find code when I need to make adjustments to a particular aspect of the formatting.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
In reply to this post by antlists
On Thu 21 May 2020 at 17:54:38 (+0100), antlists wrote:

> On 21/05/2020 16:36, David Wright wrote:
> > On Thu 21 May 2020 at 13:57:00 (+0100), antlists wrote:
> > > On 21/05/2020 01:49, David Wright wrote:
> > > > I don't understand your equivalence between .ily and .h files. The
> > > > .ily file(s) can contain just as much code as the .ly file(s), whereas
> > > > .h files don't contain any code at all (in the sense of producing
> > > > executable code for the next stage.
> > >
> > > a) just like .h files, .ily files can't be passed directly to a
> > > compiler for compilation.
> >
> > Why do you say that?
>
> Because while it may not be globally true,

It's not true at all.

> I have tried accidentally
> to do things like "lilypond voiceTrombone.ily" and it blew up quite
> spectacularly. I would expect most of mine, and in truth most other
> peoples', .ily files to blow up if fed straight to lilypond.

Big deal. I won't ask what's in voiceTrombone.ily.

> And if they do compile, the output probably isn't much use ... :-)

That depends on what output you're expecting, and whether the file
conforms to the expectations of the person compiling it. By that
I mean that it's possible for anybody to name any file quz.h or
quz.ily and then say "Gosh, look what happens when I compile it!".

Here I am compiling clockchips.h from the linux kernel:

$ gcc clockchips.h
$ ls -lgG clockchips*
-rw-r--r-- 1    7445 Apr 23 03:30 clockchips.h
-rw-r----- 1 1790384 May 21 20:01 clockchips.h.gch

No output on the terminal is good news. I see that a pre-compiled
header has been generated.
Here I am compiling an .ily file pasted from the LM:

$ ~/lilypond-2.21.0-1.linux-64/bin/lilypond definitions.ily
GNU LilyPond 2.21.0
Processing `definitions.ily'
Parsing...
definitions.ily:1: warning: no \version statement found, please add

\version "2.21.0"

for future compatibility
Success: compilation successfully completed
$ ls -lgG definitions*
-rw-r----- 1 575 May 21 19:50 definitions.ily
$

> > > b) who says .h files don't produce executable code? They often contain
> > > code, sometimes entire functions.
> > >
> > > I understand the equivalence exactly - it's what you DO to the file.
> >
> > Good, that means I have a second source of edification as to what the
> > intent of this analysis is.
> >
> > > You're thinking "what you *should* put in the file". Since when have
> > > people done what they *should*?
> >
> > Letting that pass, let's say we have a source file foo.c and it needs
> > an extraordinary quantity of functionality that's contained in bar.c.
> > We put the latter's function declarations into bar.h and compile
> > bar.{c,h}¹ into bar.o.
>
> Does bar.c contain a main{} definition?

Why would it?

> > Now we can compile foo.c (which #includes bar.h) over and over while
> > debugging it, all without ever recompiling bar.c. foo.c knows all it
> > needs to know about bar.c because of bar.h's declarations.
>
> Can you now link bar.o into foo? I honestly don't know - I don't know
> how a linker will behave if fed two main{} functions ... ?

This isn't too hard. As I wrote, bar.c has functions, and bar.h has
declarations for those functions so that compiling foo.c can check
the calls of those functions without ever seeing bar.c (ie without
recompiling bar.c over and over).

> > I now have a suite.ly that typesets my Suite for flute and piano.
> > It \includes nine files: flute{I,II,III}.ily, piano{I,II,III}.ily
> > (the notes), and flute-part.ily, piano-part.ily and complete-score.ily,
> > (my \score templates). It also \includes {marks,dyns,fingers}.ily that
> > contain LP code for my preferred styles of markings, dynamics and fingering.
> >
> > I have a file, suite.dep, which contains the information that suite.ly
> > depends on these files to produce three targets, {flute,piano,suite}.pdf
> > The same file might also indicate that I need just flute{I,II,III}.ily,
> > flute-part.ily and {marks,dyns,fingers}.ily for just a flute.pdf.
> >
> > What do I write in a top-level theFlute.ly file to produce this
> > flute.pdf and how does it benefit from the suite.sep information?
> > How do I use this information profitably, in the way that bar.h did?
> >
> > ¹ I'm abbreviating with shell's "brace expansion" notation.
> >
> Now we get to the meat of the matter. You've put bar.c's
> *declarations* into bar.h. Does lilypond even have declarations? Also,
> it's perfectly normal practice to put *definitions* in .h files -
> static variables, macros, inline functions, etc etc.
>
> I put most of my lilypond variable *definitions* into .ily files. Yup,
> .h files *tend* to contain declarations not definitions, but as I say,
> I don't think lilypond syntax has  declarations?
>
> So yes, am I right in thinking you view .ily files as closer to .o
> files? With lilypond closer to a linker than a compiler?

No.

> At the end of the day, I see it as .h files can contain declarations,
> .ily files contain declarations. .h files are included, .ily files are
> included. Some languages (like lilypond) are monolithic and don't
> really have the concept of libraries. So I find it extremely easy to
> view .ly/.ily as being similar to .c/.h. Not the same, because
> lilypond is not gcc ... :-)
>
> There's no one absolute view - you see things different to me, doesn't
> mean you (or me) is right.

Look¹, I'm not the one making this analogy. I'm trying to make any
sense of it, and why this "dependency file" is being built.

How do I use .ily files—almost all of them are definitions of
layouts, score structures, paper formats, variables' default values,
my abbreviations, and scheme code where it's wrapped in LP syntax.
They reside in a library directory that also has three subdirectories:
a collection of systematically named paper-margin files, a load of
boilerplate for exhaustively constructing MIDI file voice combinations,
and raw .scm files (like swing.scm).

One or two .ily files consist of source that's been factorised out of
the .ly file(s) that calls them. So, for example, there are festal
hymns that have words for the different seasons, so most of the
seasonal versions can be factorised out into an .ily file and then
inserted at the appropriate points in the main .ly file. There's
no saving in volume of compilation, only duplication.

I haven't used any equivalent of .o files in LP and don't know how
you would (which is why I'm asking). The nearest is with LaTeX
documents that construct PDFs from .tex files and small .pdf files,
so LaTeX (and sometimes an enclosing shell script) is acting as
the "link-loader". Even here, I don't save a lot of compilation.
For example, when I typeset a psalm for the choir, the (Anglican)
chants are automatically recompiled by LP so that they can be
transposed on the fly, to make the changes in chant work. (I often
repoint the psalm to the chants, too.)

¹ insert "at the thread" here.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 22/05/20 02:08, David Wright wrote:
> On Thu 21 May 2020 at 17:54:38 (+0100), antlists wrote:
>> On 21/05/2020 16:36, David Wright wrote:
>>> On Thu 21 May 2020 at 13:57:00 (+0100), antlists wrote:

>
>> I have tried accidentally
>> to do things like "lilypond voiceTrombone.ily" and it blew up quite
>> spectacularly. I would expect most of mine, and in truth most other
>> peoples', .ily files to blow up if fed straight to lilypond.
>
> Big deal. I won't ask what's in voiceTrombone.ily.

A variable declaration. You know - EXACTLY THE SAME as C programmers
often stick in .h files ...
>
>> And if they do compile, the output probably isn't much use ... :-)
>

>>
>> Does bar.c contain a main{} definition?
>
> Why would it?

Why WOULDN'T it?

You said that bar.c contained a load of functionality that you wanted in
foo.c. How am I supposed to know whether bar.c is a program or a library?

>
>>> Now we can compile foo.c (which #includes bar.h) over and over while
>>> debugging it, all without ever recompiling bar.c. foo.c knows all it
>>> needs to know about bar.c because of bar.h's declarations.
>>
>> Can you now link bar.o into foo? I honestly don't know - I don't know
>> how a linker will behave if fed two main{} functions ... ?
>
> This isn't too hard. As I wrote, bar.c has functions, and bar.h has
> declarations for those functions so that compiling foo.c can check
> the calls of those functions without ever seeing bar.c (ie without
> recompiling bar.c over and over).
>
>>> I now have a suite.ly that typesets my Suite for flute and piano.
>>> It \includes nine files: flute{I,II,III}.ily, piano{I,II,III}.ily
>>> (the notes), and flute-part.ily, piano-part.ily and complete-score.ily,
>>> (my \score templates). It also \includes {marks,dyns,fingers}.ily that
>>> contain LP code for my preferred styles of markings, dynamics and fingering.
>>>
>>> I have a file, suite.dep, which contains the information that suite.ly
>>> depends on these files to produce three targets, {flute,piano,suite}.pdf
>>> The same file might also indicate that I need just flute{I,II,III}.ily,
>>> flute-part.ily and {marks,dyns,fingers}.ily for just a flute.pdf.
>>>
>>> What do I write in a top-level theFlute.ly file to produce this
>>> flute.pdf and how does it benefit from the suite.sep information?
>>> How do I use this information profitably, in the way that bar.h did?

I don't have a clue. Your workflow sounds completely weird and totally
incomprehensible to me. You know what? I don't give a monkeys - your
world view is what you choose to see. You see things different to me!

>>>
>>> ¹ I'm abbreviating with shell's "brace expansion" notation.
>>>
>> Now we get to the meat of the matter. You've put bar.c's
>> *declarations* into bar.h. Does lilypond even have declarations? Also,
>> it's perfectly normal practice to put *definitions* in .h files -
>> static variables, macros, inline functions, etc etc.
>>
>> I put most of my lilypond variable *definitions* into .ily files. Yup,
>> .h files *tend* to contain declarations not definitions, but as I say,
>> I don't think lilypond syntax has  declarations?
>>
>> So yes, am I right in thinking you view .ily files as closer to .o
>> files? With lilypond closer to a linker than a compiler?
>
> No.

Okay.

>
>> At the end of the day, I see it as .h files can contain declarations,
>> .ily files contain declarations. .h files are included, .ily files are
>> included. Some languages (like lilypond) are monolithic and don't
>> really have the concept of libraries. So I find it extremely easy to
>> view .ly/.ily as being similar to .c/.h. Not the same, because
>> lilypond is not gcc ... :-)
>>
>> There's no one absolute view - you see things different to me, doesn't
>> mean you (or me) is right.
>
> Look¹, I'm not the one making this analogy. I'm trying to make any
> sense of it, and why this "dependency file" is being built.

I didn't make it either! I just happen to totally agree with it.

>
> How do I use .ily files—almost all of them are definitions of
> layouts, score structures, paper formats, variables' default values,
> my abbreviations, and scheme code where it's wrapped in LP syntax.
> They reside in a library directory that also has three subdirectories:
> a collection of systematically named paper-margin files, a load of
> boilerplate for exhaustively constructing MIDI file voice combinations,
> and raw .scm files (like swing.scm).
>
> One or two .ily files consist of source that's been factorised out of
> the .ly file(s) that calls them. So, for example, there are festal
> hymns that have words for the different seasons, so most of the
> seasonal versions can be factorised out into an .ily file and then
> inserted at the appropriate points in the main .ly file. There's
> no saving in volume of compilation, only duplication.

WHICH IS EXACTLY A REASON FOR .h FILES! Removing duplication is a good
way of removing errors (or making them easier to fix :-)

>
> I haven't used any equivalent of .o files in LP and don't know how
> you would (which is why I'm asking). The nearest is with LaTeX
> documents that construct PDFs from .tex files and small .pdf files,
> so LaTeX (and sometimes an enclosing shell script) is acting as
> the "link-loader". Even here, I don't save a lot of compilation.
> For example, when I typeset a psalm for the choir, the (Anglican)
> chants are automatically recompiled by LP so that they can be
> transposed on the fly, to make the changes in chant work. (I often
> repoint the psalm to the chants, too.)
>
Well, the way *I* see *ME* using .o files in lilypond is exactly the way
I use make in lilypond - they are virtual targets that don't have a
physical existence. The fact that they happen to have the same name as a
physical file that does exist is neither here nor there. If a .ily has a
bunch of include statements, I have a virtual target (a .o if you like)
with the parent name that depends on the sub-names.

In other words, I am using make to manage my .ily files in EXACTLY THE
SAME way I would use make to manage my .h files in a C project. It
sounds to me like you don't work that way. What on earth are your .dep
files? What are your .sep files (or was that a typo? :-) Your workflow
doesn't make sense to me!

I know I'm weird - I "don't do" visual, and guis, and all that sort of
stuff. I'm much happier with text. I know that makes me a "unix
greybeard" - it's only about 10% of the population that feels that way,
but I REALLY don't understand why most people are much happier with
pictures.

I know you say you want to understand what we're thinking, but you're
coming across as though you believe your POV is the "right" one. From
*my* POV yours doesn't make sense! Forget C, forget lilypond, look at
the broader picture, look at make, and as far as I'm concerned, .ily
files fit in my make worldview exactly the same way as C .h files fit
in. For me that analogy is perfect.

Cheers,
Wol


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
In reply to this post by David Wright
> On 21 May, 2020, at 9:08 PM, David Wright <[hidden email]> wrote:
>
> Look¹, I'm not the one making this analogy. I'm trying to make any
> sense of it, and why this "dependency file" is being built.

Well, perhaps it’s time to drop the analogy then and work with an example.  For reference sake, let’s use the makefile example in the documentation:

http://lilypond.org/doc/v2.21/Documentation/usage/make-and-makefiles

In that file you’ve got the following section which lists file dependencies:

# The dependencies of the movements.
$(piece)I.pdf: $(piece)I.ly $(notes)
$(piece)II.pdf: $(piece)II.ly $(notes)
$(piece)III.pdf: $(piece)III.ly $(notes)
$(piece)IV.pdf: $(piece)IV.ly $(notes)

# The dependencies of the full score.
$(piece).pdf: $(piece).ly $(notes)

# The dependencies of the parts.
$(piece)-cello.pdf: $(piece)-cello.ly cello.ily
$(piece)-horn.pdf: $(piece)-horn.ly horn.ily
$(piece)-oboes.pdf: $(piece)-oboes.ly oboe.ily
$(piece)-viola.pdf: $(piece)-viola.ly viola.ily
$(piece)-violinOne.pdf: $(piece)-violinOne.ly violinOne.ily
$(piece)-violinTwo.pdf: $(piece)-violinTwo.ly violinTwo.ily

As I see it, there are two problems with this section:

1) None of these dependencies mention symphonyDefs.ily.  If I make any changes to it (say I change the paper size from A4 to Letter), then make will **not** know the pdfs are now out of date.

2) These lists are manually maintained.  Each time I add another part I’ve got to add another line.  If I add more dependencies (perhaps by pulling from a library of commonly used tweaks that I reuse from project to project), I have to make sure those are listed.  If I decide to change the output format from pdf to eps, I’ve got to change everyone of these rules (as well as a couple of other things). Basically, any little change to the project structure requires me to edit the Makefile, sometimes in several places, and that introduces opportunities for error.

The point of the dependency files that I’m trying to build is to fix these problems and do so automatically (so that once it is fixed, its fixed for all time).  Under this idea (with parse-only.ly in the file structure alongside Makefile, latest version is attached) the above section of the makefile becomes:

LY_all := $(wildcard Parts/*.ly) $(wildcard Scores/*.ly)

include $(LY_all:.ly=.dly)

%.dly: %.ly parse-only.ly
        @set -e; rm -f $@; \
        $(LILY_CMD) --silent --init parse-only.ly $< > $@.$$$$; \
        sed '1s,^,$@ ,' < $@.$$$$ > $@; \
        rm -f $@.$$$$

You’ll note, this is fewer total lines and it doesn’t care how many parts or scores there are.  Add a part or a score and make will notice and create the dependency list for that part/score automatically.  Find a snippet that does something that you want to happen in one or more scores, simply add the necessary \include statements to the appropriate file(s) and make will notice and update accordingly.  Want to switch from pdf to eps (by adding the -E flag to LILY_CMD)?  Well, there are other parts of the Makefile which will need edits (because the Makefile was not written with this switch in mind), but as far as the dependencies are concerned no changes are necessary, you just need to clean out the existing dly files (which are written for the pdf targets) and make will create the new ones for the eps targets.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ

parse-only.ly (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Michael Gerdau
In reply to this post by antlists

>> Big deal. I won't ask what's in voiceTrombone.ily.
>
> A variable declaration. You know - EXACTLY THE SAME as C programmers
> often stick in .h files ...

I don’t think that’s correct.

For one according to my understanding LP has no variable declarations whatsoever, only definitions.

For two LP allows for an arbitrary number of definitions of the same variable, the latest (as of the parser) overriding any previous. In C there’s just one definition and an arbitrary number of repetitions of the same declaration.

Kind regards,
Michael

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 23/05/2020 06:06, Michael Gerdau wrote:
>
>>> Big deal. I won't ask what's in voiceTrombone.ily.
>>
>> A variable declaration. You know - EXACTLY THE SAME as C programmers
>> often stick in .h files ...
>
> I don’t think that’s correct.
>
> For one according to my understanding LP has no variable declarations whatsoever, only definitions.

I think I'm getting confused :-) no change there :-)
>
> For two LP allows for an arbitrary number of definitions of the same variable, the latest (as of the parser) overriding any previous. In C there’s just one definition and an arbitrary number of repetitions of the same declaration.
>
int i = 1;
enum something = {
   a = 1;
   b = 2;
   c = 3;
}

voiceTromboneI = \relative c { c d e f g a b c }

Forgive my rusted solid C syntax - I can't remember it properly :-)

The two languages may be different, but my mental view of the Computer
Science behind them tells me to treat them the same. I declare them in
an include file, and use them where I need them.

(Whether the language treats them as such, the theory says they are
global constants...)

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
In reply to this post by antlists
On Fri 22 May 2020 at 12:12:17 (+0100), Wols Lists wrote:

> On 22/05/20 02:08, David Wright wrote:
> > On Thu 21 May 2020 at 17:54:38 (+0100), antlists wrote:
> >> On 21/05/2020 16:36, David Wright wrote:
> >>> On Thu 21 May 2020 at 13:57:00 (+0100), antlists wrote:
>
> >> I have tried accidentally
> >> to do things like "lilypond voiceTrombone.ily" and it blew up quite
> >> spectacularly. I would expect most of mine, and in truth most other
> >> peoples', .ily files to blow up if fed straight to lilypond.
> >
> > Big deal. I won't ask what's in voiceTrombone.ily.
>
> A variable declaration. You know - EXACTLY THE SAME as C programmers
> often stick in .h files ...

Sure, but am I supposed to divine what's wrong with it from your
statement that it "blows up".

$ cat scale.ily
scale = \relative { c' d e f g a b c }
$ ~/lilypond-2.21.0-1.linux-64/bin/lilypond scale.ily
GNU LilyPond 2.21.0
Processing `scale.ily'
Parsing...
scale.ily:1: warning: no \version statement found, please add

\version "2.21.0"

for future compatibility
Success: compilation successfully completed
$

> >> And if they do compile, the output probably isn't much use ... :-)
>
> >> Does bar.c contain a main{} definition?
> >
> > Why would it?
>
> Why WOULDN'T it?

Because, in trying to understand the equivalences that were given,

.ily -> .h
.ly -> .c
.pdf -> .o

I'm not deliberately generating corner cases.

> You said that bar.c contained a load of functionality that you wanted in
> foo.c. How am I supposed to know whether bar.c is a program or a library?

Because of the next paragraph.

> >>> Now we can compile foo.c (which #includes bar.h) over and over while
> >>> debugging it, all without ever recompiling bar.c. foo.c knows all it
> >>> needs to know about bar.c because of bar.h's declarations.

Main programs have to know about the libraries they call. That's what
header lines were invented for. Then the header lines were put in a
separate file so that the library source could include the header
lines too (just as a confirmatory check).

> >> Can you now link bar.o into foo? I honestly don't know - I don't know
> >> how a linker will behave if fed two main{} functions ... ?
> >
> > This isn't too hard. As I wrote, bar.c has functions, and bar.h has
> > declarations for those functions so that compiling foo.c can check
> > the calls of those functions without ever seeing bar.c (ie without
> > recompiling bar.c over and over).
> >
> >>> I now have a suite.ly that typesets my Suite for flute and piano.
> >>> It \includes nine files: flute{I,II,III}.ily, piano{I,II,III}.ily
> >>> (the notes), and flute-part.ily, piano-part.ily and complete-score.ily,
> >>> (my \score templates). It also \includes {marks,dyns,fingers}.ily that
> >>> contain LP code for my preferred styles of markings, dynamics and fingering.
> >>>
> >>> I have a file, suite.dep, which contains the information that suite.ly
> >>> depends on these files to produce three targets, {flute,piano,suite}.pdf
> >>> The same file might also indicate that I need just flute{I,II,III}.ily,
> >>> flute-part.ily and {marks,dyns,fingers}.ily for just a flute.pdf.
> >>>
> >>> What do I write in a top-level theFlute.ly file to produce this
> >>> flute.pdf and how does it benefit from the suite.sep information?
> >>> How do I use this information profitably, in the way that bar.h did?
>
> I don't have a clue. Your workflow sounds completely weird and totally
> incomprehensible to me. You know what? I don't give a monkeys - your
> world view is what you choose to see. You see things different to me!

It isn't my workflow. It's just an attempt to split a project into
hypothetical .ly and .ily files in some manner suitable for matching
the equivalences above.

My turn to write a list of equivalences, with flute≡cello, piano≡horn:

flute{I,II,III}.ily        cello.ily
piano{I,II,III}.ily        horn.ily
{marks,dyns,fingers}.ily   figures.ily
flute-part.ily             symphony-cello.ly
piano-part.ily             symphony-horn.ly
complete-score.ily         symphony.ly
theFlute.ly                symphony-cello.ly and cello.ily for cello part
      → flute.pdf                → symphony-cello.pdf
thePiano.ly                symphony-horn.ly and horn.ily for horn part
      → piano.pdf                → symphony-horn.pdf
theSuite.ly                symphony.ly and all the above for full symphony
      → suite.pdf                → symphony.pdf

That's not very different from the LP documentation (in the right hand
column). My *hypothetical* workflow allowed for movement II to become
outrageously popular, such that people would want to buy it separately :)

When theFlute.ly is run, it \includes the appropriate .ily files,
which LP finds without needing any suite.dep file to point out that
theFlute.ly depends on flute-part.ily. Hence my question about the
benefit it's meant to give me.

> >>> ¹ I'm abbreviating with shell's "brace expansion" notation.
> >>>
> >> Now we get to the meat of the matter. You've put bar.c's
> >> *declarations* into bar.h. Does lilypond even have declarations? Also,
> >> it's perfectly normal practice to put *definitions* in .h files -
> >> static variables, macros, inline functions, etc etc.
> >>
> >> I put most of my lilypond variable *definitions* into .ily files. Yup,
> >> .h files *tend* to contain declarations not definitions, but as I say,
> >> I don't think lilypond syntax has  declarations?
> >>
> >> So yes, am I right in thinking you view .ily files as closer to .o
> >> files? With lilypond closer to a linker than a compiler?
> >
> > No.
>
> Okay.

As I said, none of this is "my view". However, I'd already accepted
that one might see an equivalence between .o and .pdf files and had
asked what program (like, say, LaTeX) might be suggested as the
equivalent of a link-loader.

> >> At the end of the day, I see it as .h files can contain declarations,
> >> .ily files contain declarations. .h files are included, .ily files are
> >> included. Some languages (like lilypond) are monolithic and don't
> >> really have the concept of libraries. So I find it extremely easy to
> >> view .ly/.ily as being similar to .c/.h. Not the same, because
> >> lilypond is not gcc ... :-)
> >>
> >> There's no one absolute view - you see things different to me, doesn't
> >> mean you (or me) is right.
> >
> > Look¹, I'm not the one making this analogy. I'm trying to make any
> > sense of it, and why this "dependency file" is being built.
>
> I didn't make it either! I just happen to totally agree with it.
> >
> > How do I use .ily files—almost all of them are definitions of
> > layouts, score structures, paper formats, variables' default values,
> > my abbreviations, and scheme code where it's wrapped in LP syntax.
> > They reside in a library directory that also has three subdirectories:
> > a collection of systematically named paper-margin files, a load of
> > boilerplate for exhaustively constructing MIDI file voice combinations,
> > and raw .scm files (like swing.scm).
> >
> > One or two .ily files consist of source that's been factorised out of
> > the .ly file(s) that calls them. So, for example, there are festal
> > hymns that have words for the different seasons, so most of the
> > seasonal versions can be factorised out into an .ily file and then
> > inserted at the appropriate points in the main .ly file. There's
> > no saving in volume of compilation, only duplication.
>
> WHICH IS EXACTLY A REASON FOR .h FILES!

Astonishing idea. There's no documented interface between each pair
of files. I had a file like:

notes = {the notes}
words = {Easter words}
\score
words = {Pentecost words}
\score

so I split it into this pair which reside in the same directory:

a.ly:
notes = {the notes}
words = {Easter words}
\book{
\bookOutputSuffix "easter"
\include b.ily
}
words = {Pentecost words}
\book{
\bookOutputSuffix "pentecost"
\include b.ily
}

b.ily:
\score

I see no similarity here to a header file or a library file.

> Removing duplication is a good
> way of removing errors (or making them easier to fix :-)

Agreed. In one of these hymn pairs, the \score is actually a sequence
of three scores: the refrain, and two alternating verse structures.
So it helps prevent accidents.

> > I haven't used any equivalent of .o files in LP and don't know how
> > you would (which is why I'm asking). The nearest is with LaTeX
> > documents that construct PDFs from .tex files and small .pdf files,
> > so LaTeX (and sometimes an enclosing shell script) is acting as
> > the "link-loader". Even here, I don't save a lot of compilation.
> > For example, when I typeset a psalm for the choir, the (Anglican)
> > chants are automatically recompiled by LP so that they can be
> > transposed on the fly, to make the changes in chant work. (I often
> > repoint the psalm to the chants, too.)
> >
> Well, the way *I* see *ME* using .o files in lilypond is exactly the way
> I use make in lilypond - they are virtual targets that don't have a
> physical existence. The fact that they happen to have the same name as a
> physical file that does exist is neither here nor there. If a .ily has a
> bunch of include statements, I have a virtual target (a .o if you like)
> with the parent name that depends on the sub-names.
>
> In other words, I am using make to manage my .ily files in EXACTLY THE
> SAME way I would use make to manage my .h files in a C project. It
> sounds to me like you don't work that way.

I manage .ily files more like I used to manage my .py module library
when I had Python projects of a respectable size, where import is
roughly equivalent to \include. But the comparison is extremely
superficial. About the only similarity is that .ily&.py files are
included&imported by disparate programs, which is why they reside in
~/LilyLib&~/PythonLib rather than in ~/scores/&~/bin/.

But again, there were .py modules which were specific to one or two
main programs, and these resided with their caller rather than in
the library. They would be developed and debugged together, and
have no separate existence.

> What on earth are your .dep
> files? What are your .sep files (or was that a typo? :-)

[Yes, a typo.]

I didn't quote the two lines following the list of equivalents,
mainly because I didn't know what it meant:

   "To this I would also add one more file type for storing dependency information
    .dly -> .d"

So my brain misremembered dependencies as .dep, probably from years of
generating modules.dep files in the linux kernel.

> Your workflow
> doesn't make sense to me!

Well, for the sort of scores I produce, I have enough idle CPU time
for battling with dependencies and make to be a waste of time.
When I run a score, I generally let LP generate all the versions.
If I'm typing in new material, I make sure that the first format
that's typeset is the one that's producing a copy that matches my
source material. (I put temporary line and page breaks in the
\global so that my PDF output matches the source manuscript.)
LP carries on running any other formats while I'm looking at the
PDF and editing the source file. It's finished by the time I next
want to run the source. (The formats might be two and four staves,
± any accompaniment, ± accompaniment alone, ± soloists, descants,
conductor's copy, separate 𝄞/𝄢 copies for tenors, anything …)

I don't run LP directly, but from a script, and that looks after
filenames, directory locations, library inclusion, MIDI renaming
(for sensible collation), cropping, and so on.

Where things are more complicated, like the transposed Anglican
chants that I mentioned, I use Python, so I can write in LaTeX
\newpagesnippet{../chants/d-hawes-E@f}{}
and a double chant in E by Hawes will be printed in F at the
top of the next page.

> I know I'm weird - I "don't do" visual, and guis, and all that sort of
> stuff. I'm much happier with text. I know that makes me a "unix
> greybeard" - it's only about 10% of the population that feels that way,
> but I REALLY don't understand why most people are much happier with
> pictures.

Ditto.

> I know you say you want to understand what we're thinking, but you're
> coming across as though you believe your POV is the "right" one. From
> *my* POV yours doesn't make sense! Forget C, forget lilypond, look at
> the broader picture, look at make, and as far as I'm concerned, .ily
> files fit in my make worldview exactly the same way as C .h files fit
> in. For me that analogy is perfect.

Not intended. Having suggested one method (a simple, unix way) of
searching for \includes, I was asking for clarification of why this
dependency information was required, and how it would reduce the
amount of compilation (seeing that LP has no equivalent of .pyc
(compiled byte code) or .o (compiled object code) that can be stored).

*My* POV (or my workflow) is irrelevant and, as you can see, bears no
resemblance to the workflow you were apparently attributing to me and
criticising.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
In reply to this post by Fr. Samuel Springuel
On Thu 21 May 2020 at 13:10:40 (-0400), Fr. Samuel Springuel wrote:

> > On 21 May, 2020, at 9:36 AM, David Kastrup <[hidden email]> wrote:
> > antlists <[hidden email]> writes:
> >> On 21/05/2020 01:49, David Wright wrote:
> >>> I don't understand your equivalence between .ily and .h files. The
> >>> .ily file(s) can contain just as much code as the .ly file(s), whereas
> >>> .h files don't contain any code at all (in the sense of producing
> >>> executable code for the next stage.
> >>
> >> a) just like .h files, .ily files can't be passed directly to a
> >> compiler for compilation.
> >>
> >> b) who says .h files don't produce executable code? They often contain
> >> code, sometimes entire functions.
> >
> > The point of .h files is that they are intended to be included by more
> > than one compilation unit resulting in an executable.
> >
> > As such, they are limited with regard to the code they can contain.
> > They can contain
> >
> > a) inline function definitions: those can occur in multiple compilation
> > units as long as they are identical.
> > b) static inline: do not even need to be identical, are per-unit.
> > c) static data definitions, are per-unit.
> > d) template function definitions
> >
> > I don't think that .ly/.ily is all that comparable here...
>
> The analogy is not perfect, but then no analogy is (otherwise it wouldn’t be an analogy).  Lilypond doesn’t enforce a distinction between .ly and .ily files in the same way that C enforces the distinction between .h and .c files.  Further, .pdf files are stand alone in a way that .o files are not.  The linker level which combines .o files into an executable would have as its closest approximation in the LilyPond world assembling several pdfs into some kind of portfolio.  And this is another point where the analogy breaks in that an executable cannot contain incompatible .o files but a portfolio can contain pdfs of scores with vastly different styles.  The result may not be as professional looking, but it’s possible.
>
> Here are the parts of the analogy that do work:
>
> 1) .ily files cannot produce scores.  Sure, I can call LilyPond on them directly, but regardless of the options I use, no score will ever result.  This may be purely by convention (as in, LilyPond itself doesn’t enforce this, it really couldn’t care what the extension on its input files is), but it’s a convention which I rigorously apply in the files I write.  In this respect, .ily files are like .h files which also cannot, by themselves, ever produce a .o file. For .h files, at least a dummy.c file is needed to include the .h file.  My .ily files likewise need a .ly file to include them to produce a score.

As you say, these file extensions are just conventions.

$ cat scale.ily
\relative { c' d e f g \include "included.pdf" a b c }
$ cat included.pdf
{
  f e f g
}
$ ~/lilypond-2.21.0-1.linux-64/bin/lilypond scale.ily
GNU LilyPond 2.21.0
Processing `scale.ily'
Parsing...
scale.ily:1: warning: no \version statement found, please add

\version "2.21.0"

for future compatibility
Interpreting music...
Preprocessing graphical objects...
Finding the ideal number of pages...
Fitting music on 1 page...
Drawing systems...
Layout output to `/tmp/lilypond-50Ds6o'...
Converting to `scale.pdf'...
Deleting `/tmp/lilypond-50Ds6o'...
Success: compilation successfully completed
$

Only the final .pdf (in scale.pdf) is unavoidable.

> 2) .ly files do produce scores.  If I call LilyPond on an .ly file, then I expect a score to come out (in pdf format by default, but other formats are possible too if I use the right options or flags).  Again, this is primarily a convention, as LilyPond won’t throw an error on a .ly file which doesn’t produce a score (but has no other problems).  However, it is a convention that I strictly apply to my files.  In this respect .ly files are like .c files, which can be compiled to create .o files.
>
> 3) Further the relation between the .ly and .ily files is similar to the relation between .c and .h files.  .ly (.c) files do not, themselves, depend on the .ily (.h) files that they include.  There is no build process which creates a .ly (.c) file from the included .ily (.h) files.  However, if I make a change to a .ily (.h) file, then the .pdf (.o) file which is produced when LilyPond (GCC) compiles a .ly (.c) file which includes that changed file is out of date and needs to be recompiled. In this regard the directed acyclic graph (DAG) which illustrates these relationships is the same, just with one less level (unless I make use of a pdf merge tool to replace the linker level).
>
>
> My goal is to inform make of these relationships so that when I tweak a LilyPond source, only the scores affected by that source are recompiled.  This may be several pdfs, but it’s not all of them, even when I tweak an ily file because not all ily files are included in all scores.  Furthermore, the more I work on this project, the more I tend to split up the “formatting” file so that I actually have several, each of which is responsible for a single aspect of the formatting.  This makes it easier for me to find code when I need to make adjustments to a particular aspect of the formatting.

If you follow some simple rules in your source layout, constructing
such a list might be most straightforward to do in the shell, using
grep. For example,

$ grep -r -l -e '^[[:space:]]*\\include[[:space:]]*"tweaked-file.¹ly"' top-of-tree/ … … ²

will find all the source files under all the tops-of-trees/ that pull
in the tweaked file. The files that are found will either be .ly files
that need recompiling, or .ily files whose names now need to be
searched for themselves:

$ grep -r -l -e '^[[:space:]]*\\include[[:space:]]*"found-file1.ily"' \
             -e '^[[:space:]]*\\include[[:space:]]*"found-file2.ily"' top-of-tree/ … …

(for all found-file*.ily files)

Repeat (while accumulating the .ly filenames) until no more .ily
files are being found. Now recompile all the .ly files that have
been accumulated.

If you've tweaked several files, the first grep will also have
multiple -e strings, one for each tweaked filename.

¹ insert "i" as appropriate.

² some people may need to add
  | grep -v '~$'
  to filter out their editor's backup files. I also added another [[:space:]]*
  to allow for variations in whitespace after the \include.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 23 May, 2020, at 7:34 PM, David Wright <[hidden email]> wrote:
>
> If you follow some simple rules in your source layout, constructing
> such a list might be most straightforward to do in the shell, using
> grep.

It’s not quite as straight-forward as you seem to think:

1) You haven’t accounted for the possibility of multiple folders with varying levels of hierarchy.  The changed file might be in ../ relative to one file, ../../ relative to another, ../some/dir/levels relative to a third and other possible variations.  It might also be specified /with/an/absolute/path in some files.  All of that complicates the search string for grep.

2) You haven’t accounted for -I options when lilypond is called.  Some of the files included might not be in the same directory tree at all.  You’ll need access to the list of -I options with which lilypond is called in order to properly identify all the possible top-of-tree/ folders.  You might be able to extract what -I flags a particular project has that are specific to it by examining some file in the project folders, but what do you do if that information is stored in a personal shell script or alias setup that catches all lilypond calls and makes sure certain -I flags are added (as might be common for someone using openLilyLib or with a personal library of formatting snippets)?  Now your shell script has to go analyze your personal shell script or shell profile.

3) What about the relative-includes option?  If an included file includes other files, where are these files located?  Is the given path relative to the current file or to the top-level file?  Our grep search string is getting even more complicated to account for this.

Sure, your ideas would probably catch most cases, and it would be possible to write a shell script that accounted for these complications.  But you’re duplicating a whole lot of work that’s already been done.  LilyPond itself already knows how to deal with these factors to find the included files (either because it’s built-in or because you’ve already told it what to do when you configured your system).  Why reinvent this wheel in a shell script if LilyPond can be made to output what it already knows for you?


Another issue: How are you keeping track of which files have changed (and thus should seed your script)?  If you recompile frequently after only changing a file or two, it’s fairly easy, but what if the change you’re working on takes several days to (either due to complexity or restricted time) or touches a whole bunch of files?  Are you supposed to manually remember all those changes?  Or maybe you’re planning on querying your version control system (you are using one, right?) for that information?  As a last resort, you might also look at modified times on the pdfs (if they exist) and the source files.

Of course, given a proper list of file dependencies, make will do all this work for you.  That’s what it is for.  Pair make with a LilyPond based tool which can determine dependencies automatically and it becomes even simpler because the system is self-updating.

Now, I will grant you, that if LilyPond could not be convinced to output what it knows about file dependencies (and the Frescobaldi project hadn’t written python-ly which has the include_args() command in the ly.slexer module), then I probably would have written a shell script like you describe to get the dependencies information (or, more likely, to just do the recompiling itself and not bothered with make at all).  However, once David K. showed that it was possible for LilyPond to output that information (and even better, without going through all the work necessary to produce a score at the same time) it makes the most logical sense to me to use that rather than start from scratch with a shell script (or start from include_args() in python-ly to write a Python script).

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 24/05/20 21:28, Fr. Samuel Springuel wrote:
> It’s not quite as straight-forward as you seem to think:
>
> 1) You haven’t accounted for the possibility of multiple folders with varying levels of hierarchy.  The changed file might be in ../ relative to one file, ../../ relative to another, ../some/dir/levels relative to a third and other possible variations.  It might also be specified /with/an/absolute/path in some files.  All of that complicates the search string for grep.

This is a real can of worms ...

I THINK all paths are relative to where lilypond is running. So any
include file that itself contains relative includes is asking for
trouble - it's "random" whether those paths will be valid or not...

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel


On May 25, 2020, at 2:58 AM, Wols Lists <[hidden email]> wrote:

On 24/05/20 21:28, Fr. Samuel Springuel wrote:
It’s not quite as straight-forward as you seem to think:

1) You haven’t accounted for the possibility of multiple folders with varying levels of hierarchy.  The changed file might be in ../ relative to one file, ../../ relative to another, ../some/dir/levels relative to a third and other possible variations.  It might also be specified /with/an/absolute/path in some files.  All of that complicates the search string for grep.

This is a real can of worms ...

I THINK all paths are relative to where lilypond is running. So any
include file that itself contains relative includes is asking for
trouble - it's "random" whether those paths will be valid or not...

Cheers,
Wol


It depends on the state of the relative-includes variable.  If true, then paths are relative to the file in which the include statement appears.  If false, the paths are relative to the main input file.  Default behavior is false.  It’s explained here: http://lilypond.org/doc/v2.21/Documentation/notation/including-lilypond-files

123