create code tests

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

create code tests

lilypond-5
Hello,

Currently I am writing code for musicxml output for lilypond.
As a part of this process I want to create code tests
For me it is possible to check a lot of code by first running a guile
command, and then a python script.
Another part of the tests can be done by running lilypond on a file, and
then run a python script.

Is het possible to do this kind of tests in the current lilypond regression
test environment?

Jaap de Wolff


Reply | Threaded
Open this post in threaded view
|

Re: create code tests

Dan Eble
On Nov 28, 2019, at 15:07, [hidden email] wrote:

> Is het possible to do this kind of tests in the current lilypond regression
> test environment?

Jaap,

I don't have a comprehensive knowledge of LilyPond's tests, but I have worked on some of the infrastructure recently, so maybe I can help you.

Please provide more detail.  What are the inputs and outputs of your tests?  What steps are involved?  Are expectations coded such that your script can detect whether the tests passed or failed, or is human review required? etc.

Dan


Reply | Threaded
Open this post in threaded view
|

RE: create code tests

lilypond-5
> -----Original Message-----
> From: Dan Eble <[hidden email]>
> Sent: Thursday, November 28, 2019 10:00 PM
> To: [hidden email]
> Cc: [hidden email]
> Subject: Re: create code tests
>
> On Nov 28, 2019, at 15:07, [hidden email] wrote:
>
> > Is het possible to do this kind of tests in the current lilypond
> > regression test environment?
>
> Jaap,
>
> I don't have a comprehensive knowledge of LilyPond's tests, but I have worked
> on some of the infrastructure recently, so maybe I can help you.
>
> Please provide more detail.  What are the inputs and outputs of your tests?
> What steps are involved?  Are expectations coded such that your script can
> detect whether the tests passed or failed, or is human review required? etc.
> —
> Dan

Dan,

I did not write a test yet, I only did it by hand, but there are a lot of tests which I want to do.
When I have the right test environment / tools it should be possible to write scripts in such a way that no human intervention is required.
The first bunch of tests will be a script running in guile, that generates xml files, to test code I have written.
Next step is to run a test to check whether those xml files are according to the expectation.
The next bunch will be several lilypond files that must be processed by lilypond.
As a result musicxml files will be created, of which also must be checked if they met the expectation.

So what I want:
run guile script1 check result with a script (possible python, but shell scripts or other programs are also possible)
Run guile script 2 check result
.. script n check result
Run lilypond on file1 check result with script
Run lilypond on file2 check result with script
-- on file m check result with script.

And I want to know which possibilities I have to do this kind of things within the lilypond developer environment.

So that when I do
$make LILYPOND_BRANCH=mybranch lilypond

I can find the results of the tests I have written in the test log

Jaap


Reply | Threaded
Open this post in threaded view
|

Re: create code tests

James Lowe-3
In reply to this post by Dan Eble
Hello,

On 28/11/2019 21:00, Dan Eble wrote:
> On Nov 28, 2019, at 15:07, [hidden email] wrote:
>
>> Is het possible to do this kind of tests in the current lilypond regression
>> test environment?
> Jaap,
>
> I don't have a comprehensive knowledge of LilyPond's tests, but I have worked on some of the infrastructure recently, so maybe I can help you.

As far as I am aware the 'tests' that LilyPond runs are basically using
itself to compile the tests in the regression during test-baseline (and
of course when you make doc) and then doing the same thing again when
running make check except you get a visual difference by comparing png
outputs between the two.

This is why you need a human to check any obviously poor differences
rather than some minor adjustment because of, say, some spacing change.

With xml we don't such visual tests so I think that only the fact that
they get compiled without complaint during test-baseline and check is
enough.

At least that is how I understand it.

James


Reply | Threaded
Open this post in threaded view
|

Re: create code tests

James Lowe-3
In reply to this post by Dan Eble
hello

On 28/11/2019 21:00, Dan Eble wrote:
> On Nov 28, 2019, at 15:07, [hidden email] wrote:
>
>> Is het possible to do this kind of tests in the current lilypond regression
>> test environment?
> Jaap,
>
> I don't have a comprehensive knowledge of LilyPond's tests, but I have worked on some of the infrastructure recently, so maybe I can help you.

I also found this:

./input/regression/musicxml/book-musicxml-testsuite.py

I remembered https://sourceforge.net/p/testlilyissues/issues/3607/ and
it mentions it there.


James


Reply | Threaded
Open this post in threaded view
|

Re: create code tests

Dan Eble


> On Nov 28, 2019, at 18:57, James <[hidden email]> wrote:
>
> hello
>
> On 28/11/2019 21:00, Dan Eble wrote:
>> On Nov 28, 2019, at 15:07, [hidden email] wrote:
>>
>>> Is het possible to do this kind of tests in the current lilypond regression
>>> test environment?
>> Jaap,
>>
>> I don't have a comprehensive knowledge of LilyPond's tests, but I have worked on some of the infrastructure recently, so maybe I can help you.
>
> I also found this:
>
> ./input/regression/musicxml/book-musicxml-testsuite.py
>
> I remembered https://sourceforge.net/p/testlilyissues/issues/3607/ and it mentions it there.

Well, step one is to avoid intertwining the existing xml->ly test infrastructure with the new ly->xml test infrastructure.  I would put ly->xml test input in a different directory.  I would probably also plan to rename input/regression/musicxml to input/regression/musicxml2ly, but I don't see a good reason to spend time on that until the new tests are established.

(I have to go now.  I'll continue later.)

Dan


Reply | Threaded
Open this post in threaded view
|

Re: create code tests

Dan Eble
On Nov 28, 2019, at 19:22, Dan Eble <[hidden email]> wrote:
>
> Well, step one is to avoid intertwining the existing xml->ly test infrastructure with the new ly->xml test infrastructure.  I would put ly->xml test input in a different directory.  I would probably also plan to rename input/regression/musicxml to input/regression/musicxml2ly, but I don't see a good reason to spend time on that until the new tests are established.
>
> (I have to go now.  I'll continue later.)

How comfortable are you working with makefiles?  I could give you an outline of how I would approach adding ly->xml tests, but it would not be very detailed and would probably still require some problem solving.  It might be less work all around for me to set up makefile rules with placeholder commands that you can then modify.

Dan

Reply | Threaded
Open this post in threaded view
|

RE: create code tests

lilypond-5
From: Dan Eble <[hidden email]>
Sent: Friday, November 29, 2019 4:51 PM
To: [hidden email]
Cc: lilypond-devel <[hidden email]>
Subject: Re: create code tests

 

On Nov 28, 2019, at 19:22, Dan Eble <[hidden email] <mailto:[hidden email]> > wrote:

 

Well, step one is to avoid intertwining the existing xml->ly test infrastructure with the new ly->xml test infrastructure.  I would put ly->xml test input in a different directory.  I would probably also plan to rename input/regression/musicxml to input/regression/musicxml2ly, but I don't see a good reason to spend time on that until the new tests are established.

(I have to go now.  I'll continue later.)

 

How comfortable are you working with makefiles?  I could give you an outline of how I would approach adding ly->xml tests, but it would not be very detailed and would probably still require some problem solving.  It might be less work all around for me to set up makefile rules with placeholder commands that you can then modify.



Dan

[>]

 

I am familiar with the use of makefile. So if you can give me directions with some placeholder commands, or comment at the right place in the makefile (s)

However, I am less familiar with the use of autoconf etc.

 

Another question:

Is it possible for me to create my own branch on the git-repository?

I have a already have a developer account, but that is an sourceforge account, and git is hosted on savannah?

At this moment I do not want to put it in staging yet, but it is better for the community that my effort is on the community git repository,

And it is easier to do a full build test.

I did try to push my own branch, but I did get a ‘not allowed’ answer, after logging in.

 

 

Jaap

 

Reply | Threaded
Open this post in threaded view
|

Re: create code tests

Dan Eble
On Nov 29, 2019, at 16:20, <[hidden email]> <[hidden email]> wrote:
>  
> I am familiar with the use of makefile. So if you can give me directions with some placeholder commands, or comment at the right place in the makefile (s)

Well, start at the "test" target in the top-level GNUmakefile.in.  It runs make in a number of subdirectories.  I would start by adding something like this there:

        $(MAKE) -C input/regression/ly2musicxml out=test local-test

You're going to need to create that ly2musicxml directory (or another name, if you have a better idea) and put a GNUmakefile there.  I would copy a GNUmakefile from a nearby directory and figure out what needs to be changed and what unnecessary things can be removed.

The target "local-test" should run your scripts.  Ideally, this should involve rules that regenerate the files only when they are out of date.  You've said that expectations can be tested automatically, so I think that, for now, you should do that immediately when the output files are generated, and cause make to fail if the expectations are not met.  (Later, we might want to allow all tests to run and summarize the failures at the end, but this seems simpler.)  Arrange the makefile so that if make is run again immediately without fixing anything, it will fail again in the same way; test this.

Once a test has passed, no subsequent "make test" should rerun it until either its prerequisites are modified or "make test-clean" has been run to remove the test results.

The "test-clean" target is defined in the top-level GNUmakefile.in.  It runs "make clean" in input/regression, and input/regression/GNUmakefile defines SUBDIRS.  If you add your new subdirectory to SUBDIRS there and have placed all your output files in $(outdir), the existing infrastructure should then remove them automatically.

Dan


Reply | Threaded
Open this post in threaded view
|

Re: create code tests

Carl Sorensen-3
Dan,

This is a great explanation.

Can you add this to the CG?  Anywhere that it seems to make sense.

Thanks,

Carl


On 11/29/19, 3:18 PM, "lilypond-devel on behalf of Dan Eble" <lilypond-devel-bounces+c_sorensen=[hidden email] on behalf of [hidden email]> wrote:

    On Nov 29, 2019, at 16:20, <[hidden email]> <[hidden email]> wrote:
    >  
    > I am familiar with the use of makefile. So if you can give me directions with some placeholder commands, or comment at the right place in the makefile (s)
   
    Well, start at the "test" target in the top-level GNUmakefile.in.  It runs make in a number of subdirectories.  I would start by adding something like this there:
   
    $(MAKE) -C input/regression/ly2musicxml out=test local-test
   
    You're going to need to create that ly2musicxml directory (or another name, if you have a better idea) and put a GNUmakefile there.  I would copy a GNUmakefile from a nearby directory and figure out what needs to be changed and what unnecessary things can be removed.
   
    The target "local-test" should run your scripts.  Ideally, this should involve rules that regenerate the files only when they are out of date.  You've said that expectations can be tested automatically, so I think that, for now, you should do that immediately when the output files are generated, and cause make to fail if the expectations are not met.  (Later, we might want to allow all tests to run and summarize the failures at the end, but this seems simpler.)  Arrange the makefile so that if make is run again immediately without fixing anything, it will fail again in the same way; test this.
   
    Once a test has passed, no subsequent "make test" should rerun it until either its prerequisites are modified or "make test-clean" has been run to remove the test results.
   
    The "test-clean" target is defined in the top-level GNUmakefile.in.  It runs "make clean" in input/regression, and input/regression/GNUmakefile defines SUBDIRS.  If you add your new subdirectory to SUBDIRS there and have placed all your output files in $(outdir), the existing infrastructure should then remove them automatically.
    —
    Dan
   
   
   

Reply | Threaded
Open this post in threaded view
|

RE: create code tests

lilypond-5
I will start to work.
However I am first installing a new lilydev virtual machine, because the old one was not up to date.
That takes some more time then expected

> -----Oorspronkelijk bericht-----
> Van: Carl Sorensen <[hidden email]>
> Verzonden: Friday, November 29, 2019 11:20 PM
> Aan: Dan Eble <[hidden email]>; [hidden email]
> CC: lilypond-devel <[hidden email]>
> Onderwerp: Re: create code tests
>
> Dan,
>
> This is a great explanation.
>
> Can you add this to the CG?  Anywhere that it seems to make sense.
>
> Thanks,
>
> Carl
>
>
> On 11/29/19, 3:18 PM, "lilypond-devel on behalf of Dan Eble" <lilypond-
> devel-bounces+c_sorensen=[hidden email] on behalf of [hidden email]>
> wrote:
>
>     On Nov 29, 2019, at 16:20, <[hidden email]> <lilypond@de-
> wolff.org> wrote:
>     >
>     > I am familiar with the use of makefile. So if you can give me directions
> with some placeholder commands, or comment at the right place in the
> makefile (s)
>
>     Well, start at the "test" target in the top-level GNUmakefile.in.  It runs
> make in a number of subdirectories.  I would start by adding something like
> this there:
>
>     $(MAKE) -C input/regression/ly2musicxml out=test local-test
>
>     You're going to need to create that ly2musicxml directory (or another
> name, if you have a better idea) and put a GNUmakefile there.  I would copy
> a GNUmakefile from a nearby directory and figure out what needs to be
> changed and what unnecessary things can be removed.
>
>     The target "local-test" should run your scripts.  Ideally, this should involve
> rules that regenerate the files only when they are out of date.  You've said
> that expectations can be tested automatically, so I think that, for now, you
> should do that immediately when the output files are generated, and cause
> make to fail if the expectations are not met.  (Later, we might want to allow
> all tests to run and summarize the failures at the end, but this seems simpler.)
> Arrange the makefile so that if make is run again immediately without fixing
> anything, it will fail again in the same way; test this.
>
>     Once a test has passed, no subsequent "make test" should rerun it until
> either its prerequisites are modified or "make test-clean" has been run to
> remove the test results.
>
>     The "test-clean" target is defined in the top-level GNUmakefile.in.  It runs
> "make clean" in input/regression, and input/regression/GNUmakefile defines
> SUBDIRS.  If you add your new subdirectory to SUBDIRS there and have
> placed all your output files in $(outdir), the existing infrastructure should then
> remove them automatically.
>     —
>     Dan
>
>
>