Long Compile time with version 2.19.44

classic Classic list List threaded Threaded
18 messages Options
BGM
Reply | Threaded
Open this post in threaded view
|

Long Compile time with version 2.19.44

BGM
Recently I updated my lilypond from 2.18 to 2.19.44.  I am using Frescobaldi for the GUI.
It used to take only a few seconds to compile a score, but now it takes at least 28 seconds every time.
This is the case even if I don't create a midi.

Am I missing something, or is this the case for everyone?

Even for this minimal example without any compile errors, it took 28.3 seconds - which is my new average time.

\version "2.19.44"
\header {
    title = "Minimal Example"
}

verses = \new Voice = "verses" \relative c'' {
    \time 4/4
    \key c \major
    a4 b4 c4 d4 | e4 d4 f2  \bar "||"
}

wordsOne = \new Lyrics  \lyricsto "verses" {
 Here are some wo -- rds to sing
}
wordsTwo = \new Lyrics  \lyricsto "verses" {
And line up wi -- th them -- selves
}
wordsThree = \new Lyrics \lyricsto "verses" {
See the stanz -- as line up in place
}

\score {
        <<
        \new Staff{   \verses }
       \wordsOne
        \wordsTwo
       \wordsThree
        >>
    \layout {

    }
}

Here is my output:

Starting lilypond-windows.exe 2.19.44 [Untitled]...

Processing `c:/users/brgabr~1/appdata/local/temp/frescobaldi-a08kr0/tmplcpzp0/document.ly'

Parsing...

Interpreting music...

Preprocessing graphical objects...

Finding the ideal number of pages...

Fitting music on 1 page...

Drawing systems...

Layout output to `./tmp-lilypond-TVi8uT'...

Converting to `document.pdf'...

Deleting `./tmp-lilypond-TVi8uT'...

Success: compilation successfully completed

Completed successfully in 28.3".





_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

tisimst


On Thu, Jul 7, 2016 at 2:25 PM, BGM [via Lilypond] <[hidden email]> wrote:
Recently I updated my lilypond from 2.18 to 2.19.44.  I am using Frescobaldi for the GUI.
It used to take only a few seconds to compile a score, but now it takes at least 28 seconds every time.
This is the case even if I don't create a midi.

Am I missing something, or is this the case for everyone?

Even for this minimal example without any compile errors, it took 28.3 seconds - which is my new average time.

\version "2.19.44"
\header {
    title = "Minimal Example"
}

verses = \new Voice = "verses" \relative c'' {
    \time 4/4
    \key c \major
    a4 b4 c4 d4 | e4 d4 f2  \bar "||"
}

wordsOne = \new Lyrics  \lyricsto "verses" {
 Here are some wo -- rds to sing
}
wordsTwo = \new Lyrics  \lyricsto "verses" {
And line up wi -- th them -- selves
}
wordsThree = \new Lyrics \lyricsto "verses" {
See the stanz -- as line up in place
}

\score {
        <<
        \new Staff{   \verses }
       \wordsOne
        \wordsTwo
       \wordsThree
        >>
    \layout {

    }
}

Here is my output:

Starting lilypond-windows.exe 2.19.44 [Untitled]...

Processing `c:/users/brgabr~1/appdata/local/temp/frescobaldi-a08kr0/tmplcpzp0/document.ly'

Parsing...

Interpreting music...

Preprocessing graphical objects...

Finding the ideal number of pages...

Fitting music on 1 page...

Drawing systems...

Layout output to `./tmp-lilypond-TVi8uT'...

Converting to `document.pdf'...

Deleting `./tmp-lilypond-TVi8uT'...

Success: compilation successfully completed

Completed successfully in 28.3".


I can confirm a very similar looooooong compilation time on 2.19.36 (over a minute for me, actually). Re-factoring the code to put all the \new statements inside the \score block then made the compilation time to 2 seconds:

%%%%%%%%%%%

\version "2.19.36"
\header {
    title = "Minimal Example"
}

verses = \relative c'' {
    \time 4/4
    \key c \major
    a4 b4 c4 d4 | e4 d4 f2  \bar "||"
}

wordsOne = \lyricmode {
 Here are some wo -- rds to sing
}
wordsTwo = \lyricmode {
And line up wi -- th them -- selves
}
wordsThree = \lyricmode {
See the stanz -- as line up in place
}

\score {
        <<
        \new Staff { \new Voice = "verses" \verses }
        \new Lyrics \lyricsto "verses" \wordsOne
        \new Lyrics  \lyricsto "verses" \wordsTwo
        \new Lyrics  \lyricsto "verses" \wordsThree
        >>
    \layout { }
}

%%%%%%%%%%%

So, no explanation from me, only an observation.

HTH,
Abraham
BGM
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

BGM
Okay, when I try exactly that, I still get 28.8 seconds for compile time.  Moving the \new statements does not change anything for me.
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

David Kastrup
In reply to this post by BGM
"Br. Gabriel-Marie | SSPX" <[hidden email]> writes:

> Recently I updated my lilypond from 2.18 to 2.19.44.  I am using
> Frescobaldi for the GUI.
> It used to take only a few seconds to compile a score, but now it
> takes at least 28 seconds every time.
> This is the case even if I don't create a midi.
>
> Am I missing something, or is this the case for everyone?
>
> Even for this minimal example without any compile errors, it took 28.3
> seconds - which is my new average time.

[...]

Nothing unusual here (64bit executable, native Linux compilation).  I
suspect some platform/font/conversion/path/Frescobaldi problem.

Does the Frescobaldi window make it possible to see just after which
console output LilyPond spends most of the time playing dead?

--
David Kastrup

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
BGM
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

BGM
Well, it seems like it hangs on just starting lilypond.  (and, as I've mentioned, I didn't have this problem with version 2.18)

As soon as it gets past this line...
So, this line takes 99% of the time in the progressbar:
Starting lilypond-windows.exe 2.19.44 [whatever.ly]...

then all the rest progresses nice and fast.

Processing `C:/hymns/whatever.ly'
Parsing...
Interpreting music...[8][16]
Preprocessing graphical objects...
Interpreting music...
MIDI output to `whatever.mid'...
Finding the ideal number of pages...
Fitting music on 1 page...
Drawing systems...
Layout output to `./tmp-lilypond-WYGD9b'...
Converting to `whatever.pdf'...
Deleting `./tmp-lilypond-WYGD9b'...
Success: compilation successfully completed
Completed successfully in 30.0".
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

David Wright
On Thu 07 Jul 2016 at 15:15:42 (-0700), BGM wrote:

> Well, it seems like it hangs on just starting lilypond.  (and, as I've
> mentioned, I didn't have this problem with version 2.18)
>
> As soon as it gets past this line...
> So, this line takes 99% of the time in the progressbar:
> Starting lilypond-windows.exe 2.19.44 [whatever.ly]...
>
> then all the rest progresses nice and fast.
>
> Processing `C:/hymns/whatever.ly'
> Parsing...
> Interpreting music...[8][16]
> Preprocessing graphical objects...
> Interpreting music...
> MIDI output to `whatever.mid'...
> Finding the ideal number of pages...
> Fitting music on 1 page...
> Drawing systems...
> Layout output to `./tmp-lilypond-WYGD9b'...
> Converting to `whatever.pdf'...
> Deleting `./tmp-lilypond-WYGD9b'...
> Success: compilation successfully completed
> Completed successfully in 30.0".

Here are some timings off a reasonably sized carol that generates
seven MIDI files and three PDFs. Old and new versions alternate
(2.18.2 and 2.19.42).

# echo 3 > /proc/sys/vm/drop_caches

real    0m33.921s
user    0m20.040s
sys     0m0.716s

real    0m28.471s
user    0m17.696s
sys     0m0.756s

real    0m20.586s
user    0m19.832s
sys     0m0.556s

real    0m18.838s
user    0m17.600s
sys     0m0.744s

real    0m20.535s
user    0m19.800s
sys     0m0.596s

real    0m18.677s
user    0m17.456s
sys     0m0.844s

So the new version wins by about two seconds.

Note the difference that caching the program makes. You've got to run
a program twice (at least) to get a realistic time.

I always put all the structure of my scores in \score;
ie I don't write   variable = \new ...
so I have no comparative data. However, if you try doing
this, remember again to run both versions twice or you may
only be measuring the efficiency of the operating system.

Cheers,
David.

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

Andrew Bernard
In reply to this post by BGM
Hi Gabriel-Marie,

What platform are you using? Linux, Windows, Mac? Can you tell us
that, and the OS version? Which Frescobaldi are you running - platform
distribution, or a build from source?

Can you obtain timings from the command line to compare with Frescobaldi?

I have observed the Frescobaldi 2.19 behaves strangely compared to
2.18 in all sorts of ways. Given that there is not a vast set of
improvements to be found in 2.19 when all is said and done. you may be
better off sticking with Frescobaldi 2.18.

You could also try posting to the Frescobaldi group, but it is very
low traffic and not very responsive - but you never know!

Andrew

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
BGM
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

BGM
Thanks for you interest, fellas.

I downgraded lilypond to version 2.19.36-1 and everything works as expected - that is, it compiles any of my scores normally in just a second or two.

My problems were all with the latest lilypond 2.19.44-1 version.

I am using Frescibaldo 2.19 on Windows 7 Pro x64.
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

David Wright
On Thu 07 Jul 2016 at 19:37:54 (-0700), BGM wrote:
> Thanks for you interest, fellas.
>
> I downgraded lilypond to version 2.19.36-1 and everything works as expected
> - that is, it compiles any of my scores normally in just a second or two.
>
> My problems were all with the latest lilypond 2.19.44-1 version.

Well, I downloaded 2.19.44-1 (I'm linux x86) and the first run
of the same carol was very slow; the disk was churning throughout.
However, this laptop does have a *very* slow disk, usually being
gently fried at about 65-70°C.

These runs are all the same version, 2.19.44-1. I dropped the
caches halfway through.

real    1m17.986s
user    0m31.128s
sys     0m1.176s

real    0m17.420s
user    0m16.948s
sys     0m0.380s

real    0m17.368s
user    0m16.884s
sys     0m0.400s

# echo 3 > /proc/sys/vm/drop_caches

real    0m30.748s
user    0m16.960s
sys     0m0.596s

real    0m17.565s
user    0m16.832s
sys     0m0.500s

So I've gained another second! However, it's always possible the
system load is lower (ie, fewer tabs open in the browser).
But I don't see a problem in this version unless some particular
construction of yours provokes it.

Cheers,
David.

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

Andrew Bernard
Hi David,

A long time UNIX user myself (pre 1970’s…), I follow what you are saying, but you may want to explain to ordinary folks what clearing the caches does and the command 'echo 3 > /proc/sys/vm/drop_caches ‘, especially since the OP is on Windows and does not have such nice things available.

OP - on the matter of downgrading, in my personal experience I have seen no speed difference between the versions you mention.


Andrew

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

David Wright
On Thu 07 Jul 2016 at 23:17:26 (-0700), Andrew Bernard wrote:
>
> A long time UNIX user myself (pre 1970’s…), I follow what you are saying,
> but you may want to explain to ordinary folks what clearing the caches does
> and the command 'echo 3 > /proc/sys/vm/drop_caches ‘, especially since the
> OP is on Windows and does not have such nice things available.

OK. The first time you run a program, it has to be loaded into memory
from the disk, along with the various library functions that the
program uses. Once it starts running, the LP source and all its
library calls are similarly loaded. As it runs, any new files created
will have their extents allocated on the disk as they're written.
All in all, a lot of work.

The second time around, much of this work can be avoided because
operating systems don't have to throw away the loaded version of
the program and the buffers containing the LP source files.
Overwriting the output files will also be faster because all the
pointers to the files' locations are already in memory.

How much difference this makes depends on the OS's capabilities.
Linux is very efficient at hanging on to all it can; a common
complaint of new users is that their computer is running out of
memory, which is usually answered with a pointer to
http://www.linuxatemyram.com/

Writing 3 to the pseudofile /proc/sys/vm/drop_caches is linux's
method of instructing the kernel to throw away all the information
that's been cached in memory. (The /proc (likewise /sys) filesystem
is a way of communicating with the OS kernel through reading and
writing what appear to be just ordinary files.)

In the case of the very first run of LP 2.19.44-1, which was
extremely slow, there's also the matter of disk-caching. During
that run, 73 files totalling 1.6MB were stored with names like
~/.lilypond-fonts.cache-2/94f...595-i686-linux.cache-7
(shortening the random part which is a GUID).
Because those files belong to me, the kernel can't throw them
away, so at least the work in generating this cache didn't have
to be repeated.

Windows probably does similar tricks, but less transparently.

Cheers,
David.

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
BGM
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

BGM
I tried this.  I took a Windows 7 computer that never before had Lilypond and I installed Lilypond 2.19.44 and Frescobaldi 2.19 and I had the same compile problem taking a really long time to compile - and that using the "choir hymn" template that comes with Frescobaldi's "New" menu.  

So, to continue the test, I downgraded Frescobaldi to 2.18 and had no problem at all with Lilypond 2.19.44!

So the problem, for me, at least, and on two separate Window 7 computers, was the combination of Frescobaldi 2.19 and Lilypond 2.19.44.

Thanks for all your attention with this.  I really do appreciate how active and helpful this list and its users are.
BGM
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

BGM
Alright, so that was on a fresh system.

So, I completely removed Lilypond and Frescobaldi from my workstation - preferences and all  - and reinstalled Lilypond 2.19.44 and Frescobaldi 2.18 from scratch.  But I have the same old problem.

So there is something in my computer that is messing up Lilypond 2.19.44.

And in response to David Wright's notions of the cache - when I installed Lilypond and Frescobaldi on the fresh system, the very first compile/run was as fast and normal as could be.  So it's not a matter of caching here, I believe.

And I don't think it's about particular scores either.  I've been running the speed tests with Frescobaldi's built-in templates, "Choir Hymn" and "Lead Sheet".  It's all the same.

BGM
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

BGM
So, on my workstation, I've tried different versions of Lilypond until I found the most recent version that works correctly.


That version is 2.19.42-1.

Therefore, I deduce that the problem I am having comes from the changes between version 2.19.42 and 2.19.43.

I'm fetching the Windows binaries from here: http://download.linuxaudio.org/lilypond/binaries/mingw/

And also for the record, I have no problems with either 2.18 nor 2.19 versions of Frescobaldi.  Both versions work perfectly with Lilypond 2.19.43.
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

m.tarenskeen
In reply to this post by BGM
Have you tried running different versions of lilypond on different systems, without using frescobaldi? I mean running lilypond directly from the commandline?

MT


-------- Oorspronkelijk bericht --------
Onderwerp: Re: Long Compile time with version 2.19.44
Van: BGM
Aan: [hidden email]
Cc:


I tried this. I took a Windows 7 computer that never before had Lilypond and
I installed Lilypond 2.19.44 and Frescobaldi 2.19 and I had the same compile
problem taking a really long time to compile - and that using the "choir
hymn" template that comes with Frescobaldi's "New" menu.

So, to continue the test, I downgraded Frescobaldi to 2.18 and had no
problem at all with Lilypond 2.19.44!

So the problem, for me, at least, and on two separate Window 7 computers,
was the combination of Frescobaldi 2.19 and Lilypond 2.19.44.

Thanks for all your attention with this. I really do appreciate how active
and helpful this list and its users are.



--
View this message in context: http://lilypond.1069038.n5.nabble.com/Long-Compile-time-with-version-2-19-44-tp192343p192400.html
Sent from the User mailing list archive at Nabble.com.

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user

_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

Johan Vromans
In reply to this post by BGM
On Fri, 8 Jul 2016 08:40:56 -0700 (MST)
BGM <[hidden email]> wrote:

> So there is something in my computer that is messing up Lilypond 2.19.44.

Looks like a job for strace to pinpoint the exact place where the process
stalls.

My intuition is that someone is making a network call that times out.
The delay is much too big for caching, even swapping.
Could be DNS, or maybe someone is trying to 'phone home', ...


_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

Andrew Bernard
Hi Johan,

Th OP is on Windows where there is no strace, I don’t think. However, I suspect you are on the right track. I attempted to build Frescobaldi 2.19.latest-whatever a while ago on several Linux distros and gave up after a few weeks of frustration due to the complex Qt code timing out in, sure enough., network calls to sockets. So something related to this has arisen in the 2.19 series. Not being particularly fond of Python programming and knowing nothing at Qt, I abandoned the work.

Andrew


On 9 July 2016 at 4:11:22 AM, Johan Vromans ([hidden email]) wrote:

On Fri, 8 Jul 2016 08:40:56 -0700 (MST)
BGM <[hidden email]> wrote:

> So there is something in my computer that is messing up Lilypond 2.19.44.

Looks like a job for strace to pinpoint the exact place where the process
stalls.

My intuition is that someone is making a network call that times out.
The delay is much too big for caching, even swapping.
Could be DNS, or maybe someone is trying to 'phone home', ...


_______________________________________________
lilypond-user mailing list
[hidden email]
https://lists.gnu.org/mailman/listinfo/lilypond-user
Reply | Threaded
Open this post in threaded view
|

Re: Long Compile time with version 2.19.44

francisperea
In reply to this post by BGM
Hi all,

I'm just writing to confirm that under macOS Sierra (10.12) Lilypond 2.19.48 is extremely slow.

I've downgraded to 2.19.47 and finally to 2.19.46.

2.19.47 was also terribly slow but 2.19.46 worked as usual.

As a sample, a simple file took 1.2 seconds to compile under 2.19.46 and 51.4 seconds under 2.19.48.

Just to provide some reference for those in the same situation and perhaps to help developers locate the problem.

Regards,

Francis Perea