regression tests for `lilypond-book`?

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

regression tests for `lilypond-book`?

Werner LEMBERG

It seems to me that not a single file in

  input/regression/lilypond-book

gets processed during `make doc`.  Has this ever worked?


    Werner

Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Jonas Hahnfeld
Am Mittwoch, den 02.09.2020, 18:37 +0200 schrieb Werner LEMBERG:
> It seems to me that not a single file in
>
>   input/regression/lilypond-book
>
> gets processed during `make doc`.  Has this ever worked?

Apparently 2.14 had some content:
http://lilypond.org/doc/v2.14/input/regression/lilypond-book/collated-files.pdf
(later versions are empty)

But to be honest, I don't see a benefit of having it. It's basically
unit-testing lilypond-book, there's no real scores coming out of it.

Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Werner LEMBERG

>> It seems to me that not a single file in
>>
>>   input/regression/lilypond-book
>>
>> gets processed during `make doc`.  Has this ever worked?
>
> Apparently 2.14 had some content:
> http://lilypond.org/doc/v2.14/input/regression/lilypond-book/collated-files.pdf
> (later versions are empty)
>
> But to be honest, I don't see a benefit of having it. It's basically
> unit-testing lilypond-book, there's no real scores coming out of it.

OK, but we don't have unit-testing for this directory either, right?


    Werner

Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Jonas Hahnfeld
Am Mittwoch, den 02.09.2020, 18:56 +0200 schrieb Werner LEMBERG:

> >> It seems to me that not a single file in
> >>
> >>   input/regression/lilypond-book
> >>
> >> gets processed during `make doc`.  Has this ever worked?
> >
> > Apparently 2.14 had some content:
> > http://lilypond.org/doc/v2.14/input/regression/lilypond-book/collated-files.pdf
> > (later versions are empty)
> >
> > But to be honest, I don't see a benefit of having it. It's basically
> > unit-testing lilypond-book, there's no real scores coming out of it.
>
> OK, but we don't have unit-testing for this directory either, right?
`make test` / `make check`? Why would you use `make doc` for that?
There's nothing to inspect visually.

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Werner LEMBERG

>> OK, but we don't have unit-testing for this directory either, right?
>
> `make test` / `make check`?

Yes.

> Why would you use `make doc` for that?

I wouldn't.


    Werner

Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Jonas Hahnfeld
Am Mittwoch, den 02.09.2020, 19:23 +0200 schrieb Werner LEMBERG:
> >> OK, but we don't have unit-testing for this directory either, right?
> >
> > `make test` / `make check`?
>
> Yes.

And that descends into input/regression/lilypond-book:
https://gitlab.com/lilypond/lilypond/-/jobs/715114070#L88

> > Why would you use `make doc` for that?
>
> I wouldn't.

So where's the problem?

Jonas


>     Werner

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Werner LEMBERG
>> >> OK, but we don't have unit-testing for this directory either,
>> >> right?
>
> And that descends into input/regression/lilypond-book:
> https://gitlab.com/lilypond/lilypond/-/jobs/715114070#L88

OK, thanks, missed that, but...

> So where's the problem?

... it doesn't do a comparison against correct input.


    Werner


PS: Do 'git grep lilypond-book/collate', and you will see some
    out-of-date information.

Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Han-Wen Nienhuys-3
On Wed, Sep 2, 2020 at 7:56 PM Werner LEMBERG <[hidden email]> wrote:
> > So where's the problem?
>
> ... it doesn't do a comparison against correct input.

The lilypond-book directory tests features of lilypond-book, but it
doesn't any real comparison. Meaningful comparison would be to compare
PDF files (after processing with LaTeX) across versions, but we don't
have support for that in the build system, and the code involved is
much simpler compared to lilypond itself, that there is little value
in doing these tests.

> PS: Do 'git grep lilypond-book/collate', and you will see some
>     out-of-date information.
>


--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

Reply | Threaded
Open this post in threaded view
|

Re: regression tests for `lilypond-book`?

Werner LEMBERG
>> ... it doesn't do a comparison against correct input.
>
> The lilypond-book directory tests features of lilypond-book, but it
> doesn't any real comparison. Meaningful comparison would be to compare
> PDF files (after processing with LaTeX) across versions,

I disagree.  There are two issues that should be tested.

* Check whether PDFs (and other output formats) can be actually
  generated.  AFAICS, this is what the current tests do.

* Process the test input files with the current `lilypond-book`
  version, check its output for correctness (including manual
  compilation with LilyPond so that the PDFs can be inspected *once*),
  then store the output files from lilypond-book – and only from
  lilypond-book, without calling LilyPond – as a baseline.

  Newer versions simply compare its output against this baseline.


    Werner