Identify included files

classic Classic list List threaded Threaded
51 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Identify included files

Fr. Samuel Springuel
Before I start writing a script to do this, is there an existing tool which will identify all the `\include` files that a LilyPond file depends on?  Even better, one that will work in a recursive fashion?

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Urs Liska-3


Am 15. Mai 2020 03:21:55 MESZ schrieb "Fr. Samuel Springuel" <[hidden email]>:
>Before I start writing a script to do this, is there an existing tool
>which will identify all the `\include` files that a LilyPond file
>depends on?  Even better, one that will work in a recursive fashion?

I know of Frescobaldi and lyluatex that have implemented thus recursive search.

Urs

>
>✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
>Fr. Samuel, OSB
>(R. Padraic Springuel)
>St. Anselm’s Abbey
>4501 South Dakota Ave, NE
>Washington, DC, 20017
>202-269-2300
>(c) 202-853-7036
>
>PAX ☧ ΧΡΙΣΤΟΣ

--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Valentin Villenave-3
In reply to this post by Fr. Samuel Springuel
On 5/15/20, Fr. Samuel Springuel <[hidden email]> wrote:
> Before I start writing a script to do this, is there an existing tool which
> will identify all the `\include` files that a LilyPond file depends on?
> Even better, one that will work in a recursive fashion?

Since 2.19.39, you can simply use this:

#(display (ly:source-files))

And you can even embed all the required source files into the PDF with:

#(ly:set-option 'embed-source-code #t)

Cheers,
V.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 15 May, 2020, at 3:43 AM, Valentin Villenave <[hidden email]> wrote:
>
> On 5/15/20, Fr. Samuel Springuel <[hidden email]> wrote:
>> Before I start writing a script to do this, is there an existing tool which
>> will identify all the `\include` files that a LilyPond file depends on?
>> Even better, one that will work in a recursive fashion?
>
> Since 2.19.39, you can simply use this:
>
> #(display (ly:source-files))
>

I’m hoping for something that doesn’t require actually typesetting the document.  My end goal is a script which will somewhat intelligently determine which files in a project have changed and re-typeset them as needed.  Having to typeset the document in order to determine which files it depends on means I’ve already done the typesetting.  Is there a way to pass that and \set Score.skipTypesetting = ##t at the command line?

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Jacques Menu Muzhic
Hello Samuel,

The tool you’re looking for to track changed files and re-handle them is ‘make’.

JM

> Le 15 mai 2020 à 18:03, Fr. Samuel Springuel <[hidden email]> a écrit :
>
>> On 15 May, 2020, at 3:43 AM, Valentin Villenave <[hidden email]> wrote:
>>
>> On 5/15/20, Fr. Samuel Springuel <[hidden email]> wrote:
>>> Before I start writing a script to do this, is there an existing tool which
>>> will identify all the `\include` files that a LilyPond file depends on?
>>> Even better, one that will work in a recursive fashion?
>>
>> Since 2.19.39, you can simply use this:
>>
>> #(display (ly:source-files))
>>
>
> I’m hoping for something that doesn’t require actually typesetting the document.  My end goal is a script which will somewhat intelligently determine which files in a project have changed and re-typeset them as needed.  Having to typeset the document in order to determine which files it depends on means I’ve already done the typesetting.  Is there a way to pass that and \set Score.skipTypesetting = ##t at the command line?
>
> ✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
> Fr. Samuel, OSB
> (R. Padraic Springuel)
> St. Anselm’s Abbey
> 4501 South Dakota Ave, NE
> Washington, DC, 20017
> 202-269-2300
> (c) 202-853-7036
>
> PAX ☧ ΧΡΙΣΤΟΣ
>
>


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
In reply to this post by Fr. Samuel Springuel
> On 15 May, 2020, at 12:03 PM, Fr. Samuel Springuel <[hidden email]> wrote:
>
>> On 15 May, 2020, at 3:43 AM, Valentin Villenave <[hidden email]> wrote:
>>
>> On 5/15/20, Fr. Samuel Springuel <[hidden email]> wrote:
>>> Before I start writing a script to do this, is there an existing tool which
>>> will identify all the `\include` files that a LilyPond file depends on?
>>> Even better, one that will work in a recursive fashion?
>>
>> Since 2.19.39, you can simply use this:
>>
>> #(display (ly:source-files))
>>
>
> I’m hoping for something that doesn’t require actually typesetting the document.  My end goal is a script which will somewhat intelligently determine which files in a project have changed and re-typeset them as needed.  Having to typeset the document in order to determine which files it depends on means I’ve already done the typesetting.  Is there a way to pass that and \set Score.skipTypesetting = ##t at the command line?
>

So, I’ve done some experimenting on my own and here’s what I’ve discovered:

`-dno-print-pages` will prevent lilypond from producing typeset output.  My preliminary testing seems to indicate that this is indeed faster than actually typesetting, but I haven’t rigorously tested this to determine what kind of speed up is produced.

`#(display (ly:source-files))` must be after any `\include` statements in the LilyPond file in order for the list to be complete.  As such, it is best positioned at the end of the file.

`lilypond -` will read in from stdin, allowing piping input into the command and thus using pipes to modify a file “on-the-fly” without actually changing the file on the disk.

Combining all this, I get the following:

echo "#(display (time ly:source-files))" | cat <file> - | lilypond -dno-print-pages -

This will send a list of the included files to stdout (there’s also some stuff on stderr which you’ll see if testing on the console).  The first two entries will be `init.ly` (with full path) and `-`, which can be thrown away, but the remaining entries are what I’m looking for (and it’s fully recursive).

One difficulty: if reading input from stdin, `\include` statements which navigate through the directory tree may experience problems if there is a mismatch between the current working directory and the location of the file being pushed through the pipes to stdin.  One way to get around this is to make use of the --include option to add in the directory of the file.  One easy way to do this is to define a variable `filename` with the name of the file, then the full command is:

echo "#(display (time ly:source-files))" | cat “$filename" - | lilypond --include="$(dirname $filename)" -dno-print-pages -

Now I just need to turn this list into something that can be used to figure out if the target needs to be recompiled.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Kastrup
In reply to this post by Fr. Samuel Springuel
"Fr. Samuel Springuel" <[hidden email]> writes:

> Before I start writing a script to do this, is there an existing tool
> which will identify all the `\include` files that a LilyPond file
> depends on?  Even better, one that will work in a recursive fashion?

You can use a file like


%% Toplevel initialisation file.

\version "2.19.22"

#(if (guile-v2)
  (begin
   (use-modules (ice-9 curried-definitions))
   (setlocale LC_ALL "")
   (setlocale LC_NUMERIC "C")))

#(session-initialize
  (lambda ()
   ;; we can't use ly:parser-include-string here since that does not
   ;; actually do any parsing but merely switches inputs, so the
   ;; session saved by the session initializer after calling this
   ;; function has not actually started.  A parser clone, in contrast,
   ;; can run and complete synchronously and shares the module with
   ;; the current parser.
   (ly:parser-parse-string (ly:parser-clone)
    "\\include \"declarations-init.ly\"")))


#(define (do-nothing . x) #f)

#(define toplevel-book-handler do-nothing)
#(define toplevel-bookpart-handler do-nothing)
#(define toplevel-music-handler do-nothing)
#(define toplevel-score-handler do-nothing)
#(define toplevel-text-handler do-nothing)

#(define book-bookpart-handler do-nothing)
#(define book-music-handler do-nothing)
#(define book-score-handler do-nothing)
#(define book-text-handler do-nothing)

#(define bookpart-score-handler do-nothing)
#(define bookpart-text-handler do-nothing)
#(define bookpart-music-handler do-nothing)
#(define output-def-music-handler do-nothing)
#(define context-mod-music-handler do-nothing)

#(note-names-language default-language)

#(define toplevel-scores (list))
#(define toplevel-bookparts (list))
#(define $defaultheader #f)
#(define $current-book #f)
#(define $current-bookpart #f)
#(define version-seen #f)
#(define expect-error #f)
#(define output-empty-score-list #f)
#(define output-suffix #f)

#(use-modules (scm clip-region))
#(use-modules (srfi srfi-1))
#(use-modules (ice-9 pretty-print))

\maininput

#(format #t "~{~a\n~^~}" (ly:source-files))


and call it with

lilypond --init parse-only.ly actual-file-name.ly 2> /dev/null

and it should be pretty fast.  Of course the file list includes
parse-only.ly...


--
David Kastrup
Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 15 May, 2020, at 5:04 PM, David Kastrup <[hidden email]> wrote:
>
> "Fr. Samuel Springuel" <[hidden email]> writes:
>
>> Before I start writing a script to do this, is there an existing tool
>> which will identify all the `\include` files that a LilyPond file
>> depends on?  Even better, one that will work in a recursive fashion?
>
> You can use a file like
>
> ...
> and call it with
>
> lilypond --init parse-only.ly actual-file-name.ly 2> /dev/null
>
> and it should be pretty fast.  Of course the file list includes
> parse-only.ly...
>
>
> --
> David Kastrup

Very nice.  I’ve tested the various options and this seems to be the best option.  While I haven’t tested with a really big file, this option appeared to be independent of file size while the option I came up with isn’t.

Is there any chance that something like this could be introduced into LilyPond core?

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Kastrup
"Fr. Samuel Springuel" <[hidden email]> writes:

>> On 15 May, 2020, at 5:04 PM, David Kastrup <[hidden email]> wrote:
>>
>> "Fr. Samuel Springuel" <[hidden email]> writes:
>>
>>> Before I start writing a script to do this, is there an existing tool
>>> which will identify all the `\include` files that a LilyPond file
>>> depends on?  Even better, one that will work in a recursive fashion?
>>
>> You can use a file like
>>
>> ...
>> and call it with
>>
>> lilypond --init parse-only.ly actual-file-name.ly 2> /dev/null
>>
>> and it should be pretty fast.  Of course the file list includes
>> parse-only.ly...
>>
>>
>> --
>> David Kastrup
>
> Very nice.  I’ve tested the various options and this seems to be the
> best option.  While I haven’t tested with a really big file, this
> option appeared to be independent of file size while the option I came
> up with isn’t.
>
> Is there any chance that something like this could be introduced into LilyPond core?

The problem is that there are quite a few things in there that make no
overwhelming sense for the task you want to solve.  In order to allow
for "snappy" init files doing specific tasks like this version of
parse-only.ly does, one should likely move some of its content to
declarations-init.ly or elsewhere.  At the current point of time, the
ability to set up different init files is not really something one can
cleanly put to good use.

So generally I'd be wary expecting this to continue working forever, but
hopefully if it stops working, the necessary changes should make for a
cleaner solution then.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
> On 16 May, 2020, at 5:31 PM, David Kastrup <[hidden email]> wrote:

>
> The problem is that there are quite a few things in there that make no
> overwhelming sense for the task you want to solve.  In order to allow
> for "snappy" init files doing specific tasks like this version of
> parse-only.ly does, one should likely move some of its content to
> declarations-init.ly or elsewhere.  At the current point of time, the
> ability to set up different init files is not really something one can
> cleanly put to good use.
>
> So generally I'd be wary expecting this to continue working forever, but
> hopefully if it stops working, the necessary changes should make for a
> cleaner solution then.
>
> --
> David Kastrup
That’s really too bad as this is something that seems like it would really come in handy.  Certainly, as I’m learning make having something which is equivalent to the -M flag for a C compiler would be very handy for generating dependencies.

Though, I did find one problem thus far: \omit seems to create problems when placed as a context mod.  In the attached file I’ve got the two methods for getting rid of bar numbers (from the manuals).  If I use the \omit version, then processing with parse-only.ly generates "error: not a context mod”.  If I use the \remove version, then everything is fine.

Of course, I still get the list of files I want on the stdout, so I can totally live with this.  The only difficulty is that when this occurs lilypond exits with a non-zero exit code, so any scripts I write based on this can’t use the error code to determine if the command completed successfully.


✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ

test.ly (199 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Kastrup
"Fr. Samuel Springuel" <[hidden email]> writes:

>> On 16 May, 2020, at 5:31 PM, David Kastrup <[hidden email]> wrote:
>>
>> The problem is that there are quite a few things in there that make no
>> overwhelming sense for the task you want to solve.  In order to allow
>> for "snappy" init files doing specific tasks like this version of
>> parse-only.ly does, one should likely move some of its content to
>> declarations-init.ly or elsewhere.  At the current point of time, the
>> ability to set up different init files is not really something one can
>> cleanly put to good use.
>>
>> So generally I'd be wary expecting this to continue working forever, but
>> hopefully if it stops working, the necessary changes should make for a
>> cleaner solution then.
>>
>> --
>> David Kastrup
>
> That’s really too bad as this is something that seems like it would
> really come in handy.  Certainly, as I’m learning make having
> something which is equivalent to the -M flag for a C compiler would be
> very handy for generating dependencies.
>
> Though, I did find one problem thus far: \omit seems to create
> problems when placed as a context mod.  In the attached file I’ve got
> the two methods for getting rid of bar numbers (from the manuals).  If
> I use the \omit version, then processing with parse-only.ly generates
> "error: not a context mod”.  If I use the \remove version, then
> everything is fine.

Uh.  I might be clearing out too many hooks here.  The ones for creating
output mods and context mods would likely be fine.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
In reply to this post by Fr. Samuel Springuel
On 15/05/2020 21:17, Fr. Samuel Springuel wrote:
> Now I just need to turn this list into something that can be used to figure out if the target needs to be recompiled.

As Jacques said, "make".

At the top of your directory structure you can have a makefile, and it
just contains a list of all your targets, and the files they depend on.
Okay, every time you add a new include, you need to update it, but...

Then when you change something, you just go to your top directory, type
"make", and watch everything affected by your recent changes recompile.

The really nice thing about it, is it will handle recursive includes by
itself. Can't remember the syntax, but if an include file pulls in other
includes, you can define that include file as a target, which then flags
any other that targets that use it. So each target in your makefile only
needs to include the includes that it depends *directly* upon.

Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Kastrup
antlists <[hidden email]> writes:

> On 15/05/2020 21:17, Fr. Samuel Springuel wrote:
>> Now I just need to turn this list into something that can be used to
>> figure out if the target needs to be recompiled.
>
> As Jacques said, "make".
>
> At the top of your directory structure you can have a makefile, and it
> just contains a list of all your targets, and the files they depend
> on. Okay, every time you add a new include, you need to update it,
> but...
>
> Then when you change something, you just go to your top directory,
> type "make", and watch everything affected by your recent changes
> recompile.
>
> The really nice thing about it, is it will handle recursive includes
> by itself. Can't remember the syntax, but if an include file pulls in
> other includes, you can define that include file as a target, which
> then flags any other that targets that use it. So each target in your
> makefile only needs to include the includes that it depends *directly*
> upon.

That's not how Makefile dependencies work.  Dependencies track the
relation of _output_ files to their _input_.  They do not track the
hierarchy of different input files that only refer to each other by
name: for the purpose of Make, those are independent and equivalent.
But the _output_ file(s) generated from them are dependent on all of the
respective input files in use, and tracking those and generating the
dependencies was what Fr. Samuel was asking about in the first place.

--
David Kastrup

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
On 18/05/2020 13:44, David Kastrup wrote:

> antlists <[hidden email]> writes:
>
>> On 15/05/2020 21:17, Fr. Samuel Springuel wrote:
>>> Now I just need to turn this list into something that can be used to
>>> figure out if the target needs to be recompiled.
>>
>> As Jacques said, "make".
>>
>> At the top of your directory structure you can have a makefile, and it
>> just contains a list of all your targets, and the files they depend
>> on. Okay, every time you add a new include, you need to update it,
>> but...
>>
>> Then when you change something, you just go to your top directory,
>> type "make", and watch everything affected by your recent changes
>> recompile.
>>
>> The really nice thing about it, is it will handle recursive includes
>> by itself. Can't remember the syntax, but if an include file pulls in
>> other includes, you can define that include file as a target, which
>> then flags any other that targets that use it. So each target in your
>> makefile only needs to include the includes that it depends *directly*
>> upon.
>
> That's not how Makefile dependencies work.  Dependencies track the
> relation of _output_ files to their _input_.  They do not track the
> hierarchy of different input files that only refer to each other by
> name: for the purpose of Make, those are independent and equivalent.
> But the _output_ file(s) generated from them are dependent on all of the
> respective input files in use, and tracking those and generating the
> dependencies was what Fr. Samuel was asking about in the first place.
>
Yup. But he wanted to know "which files do I need to recompile". Which
is exactly what makefiles do.

so if trombone.pdf is compiled from trombone.ly, which includes
trombone-notes.ly, which includes dynamics.ly, then I know there is some
sort of syntax which will:

define trombone-notes.ly as a target, define it as changed if its
dependencies change, and say "it doesn't exist as a file that can be
created with a command".

So if dynamics.ly changes, it cascades up and causes trombone.pdf to be
re-compiled, but this is the important bit - you don't need to
*ex*plicitly define trombone.pdf as relying on dynamics.ly


Just had a slight rethink - which affects things - namely what exactly
is Fr. Samuel trying to achieve. Does he want a list of dependencies as
a thing in its own right, or does he want to know which targets need to
be recompiled as a result of changing a file. Because "make" is all he
needs if the latter is what he's aiming for. If he really wants the
former, then yes he'll need all thse lily functions etc that people have
pointed him to.


Cheers,
Wol

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Urs Liska-3
Am Montag, den 18.05.2020, 14:37 +0100 schrieb antlists:

> On 18/05/2020 13:44, David Kastrup wrote:
> > antlists <[hidden email]> writes:
> >
> > > On 15/05/2020 21:17, Fr. Samuel Springuel wrote:
> > > > Now I just need to turn this list into something that can be
> > > > used to
> > > > figure out if the target needs to be recompiled.
> > >
> > > As Jacques said, "make".
> > >
> > > At the top of your directory structure you can have a makefile,
> > > and it
> > > just contains a list of all your targets, and the files they
> > > depend
> > > on. Okay, every time you add a new include, you need to update
> > > it,
> > > but...
> > >
> > > Then when you change something, you just go to your top
> > > directory,
> > > type "make", and watch everything affected by your recent changes
> > > recompile.
> > >
> > > The really nice thing about it, is it will handle recursive
> > > includes
> > > by itself. Can't remember the syntax, but if an include file
> > > pulls in
> > > other includes, you can define that include file as a target,
> > > which
> > > then flags any other that targets that use it. So each target in
> > > your
> > > makefile only needs to include the includes that it depends
> > > *directly*
> > > upon.
> >
> > That's not how Makefile dependencies work.  Dependencies track the
> > relation of _output_ files to their _input_.  They do not track the
> > hierarchy of different input files that only refer to each other by
> > name: for the purpose of Make, those are independent and
> > equivalent.
> > But the _output_ file(s) generated from them are dependent on all
> > of the
> > respective input files in use, and tracking those and generating
> > the
> > dependencies was what Fr. Samuel was asking about in the first
> > place.
> >
> Yup. But he wanted to know "which files do I need to recompile".
> Which
> is exactly what makefiles do.
>
> so if trombone.pdf is compiled from trombone.ly, which includes
> trombone-notes.ly, which includes dynamics.ly, then I know there is
> some
> sort of syntax which will:
>
> define trombone-notes.ly as a target, define it as changed if its
> dependencies change, and say "it doesn't exist as a file that can be
> created with a command".
>
> So if dynamics.ly changes, it cascades up and causes trombone.pdf to
> be
> re-compiled, but this is the important bit - you don't need to
> *ex*plicitly define trombone.pdf as relying on dynamics.ly
>
>
> Just had a slight rethink - which affects things - namely what
> exactly
> is Fr. Samuel trying to achieve. Does he want a list of dependencies
> as
> a thing in its own right, or does he want to know which targets need
> to
> be recompiled as a result of changing a file. Because "make" is all
> he
> needs if the latter is what he's aiming for. If he really wants the
> former, then yes he'll need all thse lily functions etc that people
> have
> pointed him to.

Sounds nice, but how can you tell Make what include files a given .ly
file depends on? Without inside knowledge into the LilyPond way of
recursing into the input files?

Urs

>
>
> Cheers,
> Wol
>


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Fr. Samuel Springuel
In reply to this post by antlists
> On 18 May, 2020, at 9:37 AM, antlists <[hidden email]> wrote:
>
> define trombone-notes.ly as a target, define it as changed if its dependencies change, and say "it doesn't exist as a file that can be created with a command".

Right, that’s the step I’m trying to automate.  Make itself cannot determine the list of prerequisites.  You either need to do that manually (a real pain) or provide a tool that can examine a file, determine what its prerequisites are and then write them out in a way that make understands.

From the manual for GNU Make: https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html#Automatic-Prerequisites

The example there uses the -M flag for a C compiler.  I’m looking to come up with an equivalent for LilyPond.

✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝✝
Fr. Samuel, OSB
(R. Padraic Springuel)
St. Anselm’s Abbey
4501 South Dakota Ave, NE
Washington, DC, 20017
202-269-2300
(c) 202-853-7036

PAX ☧ ΧΡΙΣΤΟΣ


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
On Mon 18 May 2020 at 10:29:22 (-0400), Fr. Samuel Springuel wrote:
> > On 18 May, 2020, at 9:37 AM, antlists <[hidden email]> wrote:
> >
> > define trombone-notes.ly as a target, define it as changed if its dependencies change, and say "it doesn't exist as a file that can be created with a command".
>
> Right, that’s the step I’m trying to automate.  Make itself cannot determine the list of prerequisites.  You either need to do that manually (a real pain) or provide a tool that can examine a file, determine what its prerequisites are and then write them out in a way that make understands.
>
> From the manual for GNU Make: https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html#Automatic-Prerequisites
>
> The example there uses the -M flag for a C compiler.  I’m looking to come up with an equivalent for LilyPond.

If you were compiling a C program foo.c into an executable, what you
would be trying to avoid is recompiling fnbar.c into fnbar.o over
and over again (and all of its similar companions).

Can you explain which of your files are the equivalents of fnbar.o.
(In python, for another example, they'd be your *.pyc files, and
python looks after all this housekeeping itself.)

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

David Wright
In reply to this post by Fr. Samuel Springuel
On Fri 15 May 2020 at 12:03:33 (-0400), Fr. Samuel Springuel wrote:
> > On 5/15/20, Fr. Samuel Springuel <[hidden email]> wrote:
> >> Before I start writing a script to do this, is there an existing tool which
> >> will identify all the `\include` files that a LilyPond file depends on?
> >> Even better, one that will work in a recursive fashion?
>
> I’m hoping for something that doesn’t require actually typesetting the document.  My end goal is a script which will somewhat intelligently determine which files in a project have changed and re-typeset them as needed.  Having to typeset the document in order to determine which files it depends on means I’ve already done the typesetting.  Is there a way to pass that and \set Score.skipTypesetting = ##t at the command line?

If you follow some simple rules in your source layout, constructing
such a list might be most straightforward to do in the shell, using
grep. For example,

$ grep -e '^[[:space:]]*\\include ' foo-top-level.ly

will find all the includes in the source, restricting it to the start
of lines, so that you can avoid false hits from, say, internal
documentation or test includes.

You would insert the resulting filenames into two lists, files visited
and all files. Iterate on the files list, skipping those visited,
until the files to visit reaches zero (lists are of equal length).

One complication is applying the search of libraries in the same
manner as LP, particularly if you rely on the order of searching.
Another is if you're using external libraries where you have no
control over their include format.

If you're not comfortable with shell programming, obviously it would
be preferable to use a language like Python or Perl for coding.

The resulting list could be used for, say, making a tarball of source
files for distribution, or for driving make to produce PDFs that are
assembled by, say, TeX programs into final documents.

Cheers,
David.

Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

Urs Liska-3
Am Montag, den 18.05.2020, 10:35 -0500 schrieb David Wright:

> On Fri 15 May 2020 at 12:03:33 (-0400), Fr. Samuel Springuel wrote:
> > > On 5/15/20, Fr. Samuel Springuel <[hidden email]> wrote:
> > > > Before I start writing a script to do this, is there an
> > > > existing tool which
> > > > will identify all the `\include` files that a LilyPond file
> > > > depends on?
> > > > Even better, one that will work in a recursive fashion?
> >
> > I’m hoping for something that doesn’t require actually typesetting
> > the document.  My end goal is a script which will somewhat
> > intelligently determine which files in a project have changed and
> > re-typeset them as needed.  Having to typeset the document in order
> > to determine which files it depends on means I’ve already done the
> > typesetting.  Is there a way to pass that and \set
> > Score.skipTypesetting = ##t at the command line?
>
> If you follow some simple rules in your source layout, constructing
> such a list might be most straightforward to do in the shell, using
> grep. For example,
>
> $ grep -e '^[[:space:]]*\\include ' foo-top-level.ly
>
> will find all the includes in the source, restricting it to the start
> of lines, so that you can avoid false hits from, say, internal
> documentation or test includes.
>

The include files may live in totally different directories, which are
essentially knows to LilyPond only.

I'm pretty sure that the only two ways to reasonably go forward is to
either invoke LilyPond to produce this list or to use a library that
already knows how LilyPond works. As I mentioned somewhere at the top
of this thread, Frescobaldi knows pretty well how to recurse into the
include files.

I would write a tool in Python that calls python-ly (well, probably
quickly, it's designated successor (
https://github.com/frescobaldi/quickly)) to produce the list of
dependencies and feed that into Make.

Urs

> You would insert the resulting filenames into two lists, files
> visited
> and all files. Iterate on the files list, skipping those visited,
> until the files to visit reaches zero (lists are of equal length).
>
> One complication is applying the search of libraries in the same
> manner as LP, particularly if you rely on the order of searching.
> Another is if you're using external libraries where you have no
> control over their include format.
>
> If you're not comfortable with shell programming, obviously it would
> be preferable to use a language like Python or Perl for coding.
>
> The resulting list could be used for, say, making a tarball of source
> files for distribution, or for driving make to produce PDFs that are
> assembled by, say, TeX programs into final documents.
>
> Cheers,
> David.
>


Reply | Threaded
Open this post in threaded view
|

Re: Identify included files

antlists
In reply to this post by Fr. Samuel Springuel


On 18/05/2020 15:29, Fr. Samuel Springuel wrote:

>> On 18 May, 2020, at 9:37 AM, antlists <[hidden email]> wrote:
>>
>> define trombone-notes.ly as a target, define it as changed if its dependencies change, and say "it doesn't exist as a file that can be created with a command".
>
> Right, that’s the step I’m trying to automate.  Make itself cannot determine the list of prerequisites.  You either need to do that manually (a real pain) or provide a tool that can examine a file, determine what its prerequisites are and then write them out in a way that make understands.
>
>  From the manual for GNU Make: https://www.gnu.org/software/make/manual/html_node/Automatic-Prerequisites.html#Automatic-Prerequisites
>
> The example there uses the -M flag for a C compiler.  I’m looking to come up with an equivalent for LilyPond.
>
Okay, does that mean you want a list of all the lilypond include files,
or just your own?

If you're digging into the internals of lilypond, then it'll be a big
job, and makes it dangerous to upgrade lilypond. Because I assume that
means you're actually modifying lilypond's internal structure. What I
do, anything of lilypond's that I modify, I split it out into my
directory structure, and then over-ride lilypond's version from within
my source. That way, what happens to lilypond during an upgrade is no
concern of mine (other than changes to syntax which mess up my source :-)

If it's just your own files you're concerned about, then grep will
probably do the trick. I don't know how to recurse directories, but
making it search all the files in a directory for "include", and tell
you which file it found that line in, is pretty basic.

You can then read up on make for how to create those virtual objects,
and use an editor to put them in your makefile. Not totally automated, I
admit, but fairly simple grunt-work. The sort of thing I could have
written a BASIC program to do in 10 minutes, except I don't have access
to a BASIC I understand how to use any more - ironic - as computers have
got easier to use, so the programming languages have got harder...

Cheers,
Wol

123