Identify included files

classic Classic list List threaded Threaded
56 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Valentin Villenave-3
On 5/25/20, R. Padraic Springuel <[hidden email]> wrote:
> It depends on the state of the relative-includes variable.  If true, then
> paths are relative to the file in which the include statement appears.  If
> false, the paths are relative to the main input file.  Default behavior is
> false.

Careful, the default will be set to true from version 2.21.2 on:
https://git.savannah.gnu.org/cgit/lilypond.git/commit/?id=16b5212f7c1745e14e29792fdcf7cb307bc23ad5

Cheers,
-- V.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
In reply to this post by Fr. Samuel Springuel
On Sun 24 May 2020 at 16:28:53 (-0400), Fr. Samuel Springuel wrote:
> > On 23 May, 2020, at 7:34 PM, David Wright <[hidden email]> wrote:
> >
> > If you follow some simple rules in your source layout, constructing
> > such a list might be most straightforward to do in the shell, using
> > grep.
>
> It’s not quite as straight-forward as you seem to think:

That's why I pasted the sentence above from my previous suggestion.
I didn't think it was worth your generalising your script to cover
every possibility that LP offers, but that's in your hands.

> 1) You haven’t accounted for the possibility of multiple folders with varying levels of hierarchy.  The changed file might be in ../ relative to one file, ../../ relative to another, ../some/dir/levels relative to a third and other possible variations.  It might also be specified /with/an/absolute/path in some files.  All of that complicates the search string for grep.

That's right. You can also flip back and forth between
relative-to-top-level-file's-directory and
relative-to-calling-file's-directory in the same file,
which my grep will know nothing about.

You can make it yet tougher: place different files named foo.ily in
several directories, and then select different versions from different
locations, guided entirely by the directories from which each \include
originates. Again, grep will be blind to such tricks.

With constructions baroque enough to match the music, a shell script
would obviously be more work than it's worth, particularly compared
with DK's method. You'd probably be better off concentrating on
writing a script to deal with automating the DK method and using its
results (as well as keeping track of all this complexity in the
source, if it hasn't already caused insanity).

> 2) You haven’t accounted for -I options when lilypond is called.  Some of the files included might not be in the same directory tree at all.  You’ll need access to the list of -I options with which lilypond is called in order to properly identify all the possible top-of-tree/ folders.  You might be able to extract what -I flags a particular project has that are specific to it by examining some file in the project folders, but what do you do if that information is stored in a personal shell script or alias setup that catches all lilypond calls and makes sure certain -I flags are added (as might be common for someone using openLilyLib or with a personal library of formatting snippets)?  Now your shell script has to go analyze your personal shell script or shell profile.

Oh, come on, that's the easy bit. Whatever options you call LP with,
you can make the same information available to the script when you
use it as they're both at the level of the shell.

As for catching all LP calls, do people actually type their LP
commands, with all its options, directly into the shell each time they
run it, rather than using scripts?

BTW, bear in mind that you may need a fresh run of parse-only.ly
to generate a new .dep file/Makefile ("DEP" to save typos) whenever
you change your set of includes, else they're stale.

> 3) What about the relative-includes option?  If an included file includes other files, where are these files located?  Is the given path relative to the current file or to the top-level file?  Our grep search string is getting even more complicated to account for this.

Already treated under (1).

> Sure, your ideas would probably catch most cases, and it would be possible to write a shell script that accounted for these complications.  But you’re duplicating a whole lot of work that’s already been done.  LilyPond itself already knows how to deal with these factors to find the included files (either because it’s built-in or because you’ve already told it what to do when you configured your system).  Why reinvent this wheel in a shell script if LilyPond can be made to output what it already knows for you?

I agree. there's no magic here. If you want to add complexity in one
place, you have to match it in another. Just so long as you need it.
(We haven't seen your use case.)

> Another issue: How are you keeping track of which files have changed (and thus should seed your script)?  If you recompile frequently after only changing a file or two, it’s fairly easy, but what if the change you’re working on takes several days to (either due to complexity or restricted time) or touches a whole bunch of files?  Are you supposed to manually remember all those changes?  Or maybe you’re planning on querying your version control system (you are using one, right?) for that information?  As a last resort, you might also look at modified times on the pdfs (if they exist) and the source files.

That's a question I might ask you: these are your files, your edits,
your working methods, your problem area.

> Of course, given a proper list of file dependencies, make will do all this work for you.  That’s what it is for.  Pair make with a LilyPond based tool which can determine dependencies automatically and it becomes even simpler because the system is self-updating.

I've yet to see how make helps you at this level, ie within the LP
run. As I wrote earlier, I can see how it helps in constructing a
number of LP's *outputs* (like PDFs), and some may use it in the
way I use a script: to select the LP version, assemble the commandline
options, postprocess the output files, and suchlike.

But all LP needs to do its job is the name of the top-level file
and a search-list of libraries. It can find the \includes for itself
in the libraries given and, having found them, it compiles them all
regardless. Completely unlike, say, a C program.

But it seems to me that your OP had the makings of an A/B problem.
You originally asked for a script that worked in the forward
direction: a list of top-level file's dependencies, for constructing
DEP. Having got LP to perform that with DK's method, it turns out
your goal is something different: to discover all the top-level files
that need to be recompiled when you edit an arbitrary \include file.

That means that, having edited a mid- to low-level source file in your
library, you have to search every DEP to see whether that file is
mentioned. This includes those of the "someone" who uses your library.
And all those DEPs must be kept up-to-date if the search is to be
performed correctly.

The idea behind my later script was: given "some simple rules in your
source layout" and the constituency of your LP sources, the backwards
search for reverse dependencies can be carried out any time any source
file is altered, without the need to keep DEP files up-to-date,
because it greps the active set of files.

> Now, I will grant you, that if LilyPond could not be convinced to output what it knows about file dependencies (and the Frescobaldi project hadn’t written python-ly which has the include_args() command in the ly.slexer module), then I probably would have written a shell script like you describe to get the dependencies information (or, more likely, to just do the recompiling itself and not bothered with make at all).  However, once David K. showed that it was possible for LilyPond to output that information (and even better, without going through all the work necessary to produce a score at the same time) it makes the most logical sense to me to use that rather than start from scratch with a shell script (or start from include_args() in python-ly to write a Python script).

Given a top-level .ly file and DK's code, I don't see how you
would avoid any of the recompilng required when you use DK's
output. The last line of the script says

    #(format #t "~{~a\n~^~}" (ly:source-files))

How do you use source files without recompiling them—that's what 's
perplexed me all along in this thread. All I've tried to do is give
an idea for finding a list of current top-level files that \include an
arbitrary, specific file. Anyone, with assistance from LP, can make
this as hard as they like for themselves.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
On 25 May, 2020, at 12:46 PM, David Wright <[hidden email]> wrote:

> But it seems to me that your OP had the makings of an A/B problem.
> You originally asked for a script that worked in the forward
> direction: a list of top-level file's dependencies, for constructing
> DEP. Having got LP to perform that with DK's method, it turns out
> your goal is something different: to discover all the top-level files
> that need to be recompiled when you edit an arbitrary \include file.

Not really an A/B problem in that I’m not changing the end goal.  I needed (and asked) about A (constructing the list of dependencies) because solving A was the prerequisite for getting make to do B (figuring out which scores needed to be recompiled when I changed a source file).  While I briefly toyed with the idea of doing B myself (and indeed, had a proof of concept bash script), Jacques pointing to make just made it clear to me that B was already solved by make, provided I solved A in a make-friendly way.

> That means that, having edited a mid- to low-level source file in your
> library, you have to search every DEP to see whether that file is
> mentioned. This includes those of the "someone" who uses your library.
> And all those DEPs must be kept up-to-date if the search is to be
> performed correctly.

That’s true, but this is something make can do automatically given a rule which defines how to create the DEPs.  Since such a rule already exists for doing this with CC -M (and is in the gnumake manual: https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html#Automatic-Prerequisites), it was fairly trivial for me to adapt that.

> The idea behind my later script was: given "some simple rules in your
> source layout" and the constituency of your LP sources, the backwards
> search for reverse dependencies can be carried out any time any source
> file is altered, without the need to keep DEP files up-to-date,
> because it greps the active set of files.

If I’m understanding you correctly, your point is that A doesn’t need to be solved at all, if you’re willing to solve B in a totally different fashion (i.e. not use make).  And I would say that that is true.  But it means ignoring the tools that are already at my disposal.  While I’m all for learning new things, I hate spending time doing things that someone else has already done if I can use their work.

>
> Given a top-level .ly file and DK's code, I don't see how you
> would avoid any of the recompilng required when you use DK's
> output. The last line of the script says
>
>    #(format #t "~{~a\n~^~}" (ly:source-files))
>
> How do you use source files without recompiling them—that's what 's
> perplexed me all along in this thread. All I've tried to do is give
> an idea for finding a list of current top-level files that \include an
> arbitrary, specific file. Anyone, with assistance from LP, can make
> this as hard as they like for themselves.

I think this is where you’ve missed something.  When using DK’s code as the init file (or my later version, which make the output more make-friendly), LilyPond **does not actually typeset the music.**  All it does is read through the files to construct ly:source-files and then output that list to stdout.  It is the equivalent, in some sense, of the -M option for a CC (which is where the whole analogy got started).  cc -M sample.c does not create sample.o, it outputs the list of files that sample.o depends on.  Likewise lilypond --init parse-only.ly sample.ly does not create sample.pdf, it outputs the list of files that sample.pdf depends on.  In both cases we’re using the compiler’s (cc or lilypond) already existing knowledge of how to read its source and find the mentioned includes, but instead of telling it to use that knowledge to create the target (the object file or pdf), we’re telling it to use that knowledge to create the dependency list.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 25/05/20 19:25, Fr. Samuel Springuel wrote:
> I think this is where you’ve missed something.  When using DK’s code as the init file (or my later version, which make the output more make-friendly), LilyPond **does not actually typeset the music.**  All it does is read through the files to construct ly:source-files and then output that list to stdout.  It is the equivalent, in some sense, of the -M option for a CC (which is where the whole analogy got started).  cc -M sample.c does not create sample.o, it outputs the list of files that sample.o depends on.  Likewise lilypond --init parse-only.ly sample.ly does not create sample.pdf, it outputs the list of files that sample.pdf depends on.  In both cases we’re using the compiler’s (cc or lilypond) already existing knowledge of how to read its source and find the mentioned includes, but instead of telling it to use that knowledge to create the target (the object file or pdf), we’re telling it to use that knowledge to create the dependency list.

So. Am I correct in thinking that, if you change one .ily file, you need
to rebuild the entire makefile? WHY?

I don't know the exact make syntax but my makefile would be something like

partTrombone.pdf partTrombone.ly
   lilypond partTrombone.ly

partTrombone.ly voiceTrombone.ily
   :

voiceTrombone.ily dynamics.ily
   :

Here I'm assuming that ":" tells make that this is a virtual object - it
can't be made but it's affected by changes to the file(s) that it
depends on.

So if I edit dynamics.ily, then do "make partTrombone.pdf", make will
cascade that virtual dependency up, realise that partTrombone.pdf is out
of date, and rebuild it. And it means if put a new include into
dynamics.ily, I just need to create/update the line for dynamics.ily,
and everything that depends on it will rebuild if I run the "make" command.

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 25 May, 2020, at 3:12 PM, Wols Lists <[hidden email]> wrote:
>
> So. Am I correct in thinking that, if you change one .ily file, you need
> to rebuild the entire makefile? WHY?
<snip>
> And it means if put a new include into
> dynamics.ily, I just need to create/update the line for dynamics.ily,
> and everything that depends on it will rebuild if I run the "make" command.

Not the entire makefile, just the dly files (which are only partial makefiles) which depend on that ily file.  This is necessary to take care of your later issue automatically.  By making the dly files depend on the same files which the associate pdf file depends on make will automatically check for changes to the number of included files.

> partTrombone.ly voiceTrombone.ily
>   :


This is not a valid statement for make (even if coerced into the correct syntax).  While partTrombone.ly may include voiceTrombone.ily, it does not actually depend on it.  Dependency in make implies one thing is required to *create* the other, not that they cannot be used independently (which is what you appear to mean by “virtual object”).

Each dly file looks like this:

target.dly target.pdf : target.ly include_1.ily include_2.ily path/to/include_3.ily

The separator (:) splits the targets (the files which can be built) from their prerequisites (the files needed to do the building).  If make determines that any target doesn’t exist or is out-of-date, then it will rebuild the target.  That’s all that’s in a single dly file, so it does not constitute a complete makefile.  Instead it is read into the master makefile, which also contains the rules for building both dly files and pdf files.

The master makefile looks like the one I posted earlier in this thread.  It co ntains the rules for building dly files and pdf files.  Both of these are pattern rules.  This means that one rule tells make how to build all dly files from a ly (and parse-init.ly) file and another tells it how to build all pdf files from a ly file.

Perhaps a working example will be more productive than a purely theoretical discussion.  Here’s a repository that I built using code from the documentation:

https://github.com/rpspringuel/lilypond_make

Right now it just tries to actualize the example from http://lilypond.org/doc/v2.21/Documentation/usage/make-and-makefiles (using content taken from http://lilypond.org/doc/v2.21/Documentation/snippets/staff-notation#staff-notation-orchestra-choir-and-piano-template).  I had to make a few changes to make it work on my system, but I tried to make those changes be as system-agnostic as possible.  If you’ll accept that this is a working representation of the makefile example from the documentation, then I’ll go about showing how it gets modified to use the system that I’ve worked out.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Matt Wallis-3
 > Le 15 mai 2020 à 18:03, Fr. Samuel Springuel <address@hidden> a écrit :
 >
 >> On 15 May, 2020, at 3:43 AM, Valentin Villenave <address@hidden> wrote:
 >>
 >> On 5/15/20, Fr. Samuel Springuel <address@hidden> wrote:
 >>> Before I start writing a script to do this, is there an existing
tool which
 >>> will identify all the `\include` files that a LilyPond file depends on?
 >>> Even better, one that will work in a recursive fashion?
 >>
 >> Since 2.19.39, you can simply use this:
 >>
 >> #(display (ly:source-files))

That's good! I didn't know you could do that.

 >>
 >
 > I’m hoping for something that doesn’t require actually typesetting the
 > document.  My end goal is a script which will somewhat intelligently
 > determine which files in a project have changed and re-typeset them as
 > needed.  Having to typeset the document in order to determine which
files it
 > depends on means I’ve already done the typesetting.  Is there a way
to pass
 > that and \set Score.skipTypesetting = ##t at the command line?
Elsewhere in this thread, there are useful comparisons between C/C++
source and Lilypond source. Also there's the suggestion of using 'make'
to express dependencies.

For C/C++, there is an excellent discussion of the issues about
automatically producing dependencies and dealing with them in make:

http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/

I think much of this is relevant to lilypond too. There's even a section
entitled "Dependencies for non C files".

With regard to wanting to avoid typesetting when you want to produce
only the dependencies, this section discusses how to set up a makefile
to use dependency information that is created as a side-effect of
compilation:

http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#combine

I know I would find it useful if some of the dependency generation flags
(-MMD, -MF, -MP, -MF , see
https://gcc.gnu.org/onlinedocs/gcc/Preprocessor-Options.html) available
on C/C++ preprocessor were implemented in lilypond. The work discussed
in this thread might be paving a path towards that.


Lilypond can produce multiple output files:

One difference between C/C++ compilers and lilypond is that where a C
compiler produces exactly one .o file for each .c file, lilypond can
generate  several output files e.g. .pdf and .midi, but there are other
examples too - I sometimes produce a separate .png for each page of a
score. And I expect that the set of output files may depend on the
lilypond command line options, and not just on the .ly file on which it
is being run.

Is there a way to get lilypond to output this list of targets, analogous
to the ly:source-files capability?

I think in most cases, the issues around multiple targets can be ignored
but there may be some cases where a full solution would need to handle
them ... more thought required.

Matt

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 26 May, 2020, at 6:18 AM, Matt Wallis <[hidden email]> wrote:
>
> For C/C++, there is an excellent discussion of the issues about automatically producing dependencies and dealing with them in make:
>
> http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/
>
> I think much of this is relevant to lilypond too. There's even a section entitled "Dependencies for non C files".
>
> With regard to wanting to avoid typesetting when you want to produce only the dependencies, this section discusses how to set up a makefile to use dependency information that is created as a side-effect of compilation:
>
> http://make.mad-scientist.net/papers/advanced-auto-dependency-generation/#combine
>
> I know I would find it useful if some of the dependency generation flags (-MMD, -MF, -MP, -MF , see https://gcc.gnu.org/onlinedocs/gcc/Preprocessor-Options.html) available on C/C++ preprocessor were implemented in lilypond. The work discussed in this thread might be paving a path towards that.

I hadn’t seen that article before, but I’ve started reading it and can definitely see that there are some possibilities there.  I’m going to take the time to pursue this and see how I can adapt my process.


> Lilypond can produce multiple output files:
>
> One difference between C/C++ compilers and lilypond is that where a C compiler produces exactly one .o file for each .c file, lilypond can generate  several output files e.g. .pdf and .midi, but there are other examples too - I sometimes produce a separate .png for each page of a score. And I expect that the set of output files may depend on the lilypond command line options, and not just on the .ly file on which it is being run.
>
> Is there a way to get lilypond to output this list of targets, analogous to the ly:source-files capability?
>
> I think in most cases, the issues around multiple targets can be ignored but there may be some cases where a full solution would need to handle them ... more thought required.

I’ve already accounted for this in my most recent version of parse-only.ly.  It’s not as straight forward as ly:source-files but by looking at (ly:get-option 'backend), (ly:get-option 'aux-files), (ly:parser-output-name) and (ly:output-formats), it is possible to determine which output files lilypond is going to produce (except for the midi, I have yet to find any option or function which will indicate that a midi is being produced).  The applicable code is:

% Construct list of target extensions
% We look at which backend is being used and what formats have been requested to determine this list
% Since some backends cause the list of formats to be ignored, we check that first, only looking at the
% formats if the backend would allow that.
#(define target-extensions (cond ((equal? (ly:get-option 'backend) 'svg) (list "svg"))
                                 ((equal? (ly:get-option 'backend) 'scm) (list "scm"))
                                 ((equal? (ly:get-option 'backend) 'ps) (uniq-list (sort-list (ly:output-formats) string<?)))
                                 ((equal? (ly:get-option 'backend) 'eps) (map (lambda (str) (if (equal? str "ps") "eps" str)) (uniq-list (sort-list (ly:output-formats) string<?))))
                                 (else '())))

% add the "." to the beginning of each extension
#(define target-extensions (map (lambda (str) (string-append "." str)) target-extensions))

% For the eps backend, we have to add some additional files which are created if the aux-files option is true
#(if (and (equal? (ly:get-option 'backend) 'eps) (ly:get-option 'aux-files))
     (if (member "pdf" target-extensions)
         (append! target-extensions '("-systems.tex" "-systems.texi" "-systems.count" "-1.eps" "-1.pdf"))
         (append! target-extensions '("-systems.tex" "-systems.texi" "-systems.count" "-1.eps"))))

% Construct list of target files.
% First grab the target-basename: either the same as the basename of the input file, or from --output=FILE
% (or --output=DIR/FILE).
% Note: If you use --output=DIR (or --output=DIR/FILE) when calling lilypond, then this output directory information
%       is used to change the working directory and discarded.  As a result, the make rule output by parsing a
%       lilypond file with this file will not contain this information.
%       If you use -dno-strip-output-dir but not --output=..., then the directory information of the *input* file
%       will be part of target-basename.
%       If both -dno-strip-output-dir and --output=FILE (or --output=DIR/FILE) then the basename will be taken from
%       --output and all directory information will be lost.
#(define target-basename (ly:parser-output-name))
% The target list is then assembled by appending the target-extensions.
#(define target-files
     (map
      (lambda (extension) (string-append target-basename extension))
      target-extensions))



✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Timothy Lanfear
On 26/05/2020 16:15, Fr. Samuel Springuel wrote:
> I’ve already accounted for this in my most recent version of
> parse-only.ly. It’s not as straight forward as ly:source-files but by
> looking at (ly:get-option 'backend), (ly:get-option 'aux-files),
> (ly:parser-output-name) and (ly:output-formats), it is possible to
> determine which output files lilypond is going to produce (except for
> the midi, I have yet to find any option or function which will
> indicate that a midi is being produced). The applicable code is:

Maybe this code can give some hints on how to decide if midi is being
produced.

\version "2.20.0"

#(define (outputdeftype def)
   (cond
     ((ly:output-def-lookup def 'is-layout #f) (display "\nlayout"))
     ((ly:output-def-lookup def 'is-midi #f)   (display "\nmidi"))))

\book {
   \score {
     { c'1 }
     \layout{}
     \midi{}
   }

#(for-each
    (lambda (score) (for-each outputdeftype (ly:score-output-defs score)))
    (ly:book-scores $current-book))

}

--
Timothy Lanfear, Bristol, UK.


Timothy Lanfear, Bristol, UK.
Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 26 May, 2020, at 12:09 PM, Timothy Lanfear <[hidden email]> wrote:
>
> Maybe this code can give some hints on how to decide if midi is being produced.

That’s the first code I’ve seen that’s been able to definitively show when a midi file is being produced.  Thank you.

Now, there is the issue of it being inside the book block, which makes it harder to incorporate into something like parse-only.ly (which generally operates either before or after all input is read), but at least it gives me a starting point for something to play with.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
On Tue 26 May 2020 at 13:06:50 (-0400), Fr. Samuel Springuel wrote:
> > On 26 May, 2020, at 12:09 PM, Timothy Lanfear <[hidden email]> wrote:
> >
> > Maybe this code can give some hints on how to decide if midi is being produced.
>
> That’s the first code I’ve seen that’s been able to definitively show when a midi file is being produced.  Thank you.
>
> Now, there is the issue of it being inside the book block, which makes it harder to incorporate into something like parse-only.ly (which generally operates either before or after all input is read), but at least it gives me a starting point for something to play with.

Because of the inability to collate multiple MIDI files, I've always
renamed them. I tee the log into a nonce file, which I grep for their
names in a postprocessing script. (I then trash the log unless there
was an error code, in which case I leave it, renaming it foo.error.)

End of main script:

    "$Youngestlily" "$Pointclick" --include="$HOME"/LilyLib/… … "$Sourcefilename" 2>&1 | tee "$Unique";
    [ $PIPESTATUS -ne 0 ] && Lilyerror="ERROR";
    midirename "$Unique";
    if [ -z "$Lilyerror" ]; then
        rm -f "${Sourcefilename/%.ly/.error}";
    else
        printf '\n%s\n\n' "$Lilyerror";
        mv "$Unique" "${Sourcefilename/%.ly/.error}";
        return 2;
    fi )
}

Fragment of midirename script:

    grep -e '^MIDI output to `' "$1" | sed -e 's/^[^`]*.//;s/.\.\.\.$//' >> "$Unique"

PDFs show up in the same way, but not PNGs (which aren't in my own workflow).

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 26 May, 2020, at 6:13 PM, David Wright <[hidden email]> wrote:
>
> Because of the inability to collate multiple MIDI files, I've always
> renamed them. I tee the log into a nonce file, which I grep for their
> names in a postprocessing script. (I then trash the log unless there
> was an error code, in which case I leave it, renaming it foo.error.)

I’m beginning to think that this may be the way to go.  The article that Matt linked makes some good points about when dependency files actually need to be updated.  I hadn’t considered those points before and they lead to something different from what the GNUMake manual suggests as far as building the dependency files.  Implementing those ideas will require me to rework some of what I’ve been doing, but also would open up the possibility to do things like analyze the stderr from the compilation for messages which clearly identify the output.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
I have now updated the repository I posted earlier (https://github.com/rpspringuel/lilypond_make) to contain two additional branches:

GNUmake_dependencies: Implements dependency generation as described in the GNUmake manual

mad-scientist_dependencies: Implements dependency generation as described in the article that Matt linked.  Right now I’ve used a simple sed script in the recipe to add the midi file to the targets of the dly file generated by LilyPond, but I haven’t given up on a more general solution.

Comments from anyone willing to look these over and play around with them are greatly appreciated.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Matt Wallis-3
On 06/06/2020 16:34, Fr. Samuel Springuel wrote:
> I have now updated the repository I posted earlier (https://github.com/rpspringuel/lilypond_make) to contain two additional branches:
>
> GNUmake_dependencies: Implements dependency generation as described in the GNUmake manual
>
> mad-scientist_dependencies: Implements dependency generation as described in the article that Matt linked.  Right now I’ve used a simple sed script in the recipe to add the midi file to the targets of the dly file generated by LilyPond, but I haven’t given up on a more general solution.
>
> Comments from anyone willing to look these over and play around with them are greatly appreciated.

Hi Samuel,
Great work! I have not had a play, but I have had a quick look through
https://github.com/rpspringuel/lilypond_make/blob/mad-scientist_dependencies/Makefile

Just a few comments, mostly general things about makefiles, practices
that I usually follow.

Be clear about the difference between = and :=. You are using = when :=
would work fine.

Put everything that moves (!) into a variable. For example, I'd have

PDF_DIR := PDF/

Include the backslash when you define a directory (as above). This is a
habit I've got into and now rely on. It helps me.

Use `mkdir -p` to create dirs because it doesn't fail if the directory
exists. Having said that, this goes against another principle I often
try to follow: Write software according to Dijkstra's weakest
preconditions, by making it as fragile as possible (but no more
fragile!). That way the program will break if the implementation does
not follow the design, making it easier for bugs to become apparent.
With the `mkdir -p` example, it could be argued that because these
directory targets only come into play via order-only prerequisites, they
should never be called upon unless the directory does not exist...

I found
https://lists.gnu.org/archive/html/lilypond-user/2020-05/msg00148.html 
very interesting.
It describes how to get lilypond to print out dependencies. Have you
looked into this?

Best regards,
Matt

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Matt Wallis-3
On 09/06/2020 10:54, Matt Wallis wrote:
> Include the backslash when you define a directory (as above).
s/backslash/slash/

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
In reply to this post by Matt Wallis-3
> On 9 Jun, 2020, at 5:54 AM, Matt Wallis <[hidden email]> wrote:
>
> I found https://lists.gnu.org/archive/html/lilypond-user/2020-05/msg00148.html very interesting.
> It describes how to get lilypond to print out dependencies. Have you looked into this?

That’s from the start of this thread and I’m making use of it in the with-deps.ly file (and in parse-only.ly on the GNUmake_dependencies branch).  I don’t use it exactly as it was originally presented in that message because the format wasn’t make-freindly, but ly:source-files is the starting point for generating the list of prerequisites.

> Be clear about the difference between = and :=. You are using = when := would work fine.

That’s a good point.  I’m carrying over some bad practices from the original Makefile example in the documentation and should clean that up as much as possible.

> Put everything that moves (!) into a variable. For example, I'd have
>
> PDF_DIR := PDF/
>
> Include the backslash when you define a directory (as above). This is a habit I've got into and now rely on. It helps me.

Can you elaborate on this?  Why would this be better than what I’ve got?



✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Matt Wallis-3
On 09/06/2020 15:57, Fr. Samuel Springuel wrote:
>> On 9 Jun, 2020, at 5:54 AM, Matt Wallis <[hidden email]> wrote:
>>
>> I found https://lists.gnu.org/archive/html/lilypond-user/2020-05/msg00148.html very interesting.
>> It describes how to get lilypond to print out dependencies. Have you looked into this?
>
> That’s from the start of this thread and I’m making use of it in the with-deps.ly file (and in parse-only.ly on the GNUmake_dependencies branch).  I don’t use it exactly as it was originally presented in that message because the format wasn’t make-freindly, but ly:source-files is the starting point for generating the list of prerequisites.

Sorry - I should have read more carefully!

>> Be clear about the difference between = and :=. You are using = when := would work fine.
>
> That’s a good point.  I’m carrying over some bad practices from the original Makefile example in the documentation and should clean that up as much as possible.
>
>> Put everything that moves (!) into a variable. For example, I'd have
>>
>> PDF_DIR := PDF/
>>
>> Include the backslash when you define a directory (as above). This is a habit I've got into and now rely on. It helps me.
>
> Can you elaborate on this?  Why would this be better than what I’ve got?

Good question. I've been doing this out of habit for so long, I had to
think hard to remember why! I've come up with two reasons:

1. The trailing slash in foo/ forces it to be a directory. For example,
even when foo does not exist you can't treat foo/ as a regular file:

     $ cat > foo/
     bash: foo/: Is a directory

2. An accidental extra slash causes no problems, but an accidental
missing slash changes the meaning:

    foo//bar is the same as foo/bar, but
    foobar is not the same as foo/bar


Matt

123