issue verification

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

issue verification

Jonas Hahnfeld
Hi all,

unless I'm missing something, there has been no activity on this since
the community effort in May [1][2] and issues since version 2.21.2
don't yet have Status::Verified labels. As I'm currently updating CG to
mention our use of milestones (not sure why I forgot about this
previously), the verification procedure should probably also get
updated.
I haven't been active in this area (and don't plan to be), so it would
be great if somebody could look over the current description (assuming
the procedure should be continued).

To give a head start, here's again the list of all Status::Fixed:
https://gitlab.com/lilypond/lilypond/-/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=Status%3A%3AFixed

Furthermore you can directly get to the issues associated with a
milestone, for example for 2.21.2:
https://gitlab.com/lilypond/lilypond/-/milestones/320#tab-labels

Thanks,
Jonas

1: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00284.html
2: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00379.html

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Michael Käppler-2

Am 17.09.2020 um 09:47 schrieb Jonas Hahnfeld:

> Hi all,
>
> unless I'm missing something, there has been no activity on this since
> the community effort in May [1][2] and issues since version 2.21.2
> don't yet have Status::Verified labels. As I'm currently updating CG to
> mention our use of milestones (not sure why I forgot about this
> previously), the verification procedure should probably also get
> updated.
> I haven't been active in this area (and don't plan to be), so it would
> be great if somebody could look over the current description (assuming
> the procedure should be continued).
>
> To give a head start, here's again the list of all Status::Fixed:
> https://gitlab.com/lilypond/lilypond/-/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=Status%3A%3AFixed
>
> Furthermore you can directly get to the issues associated with a
> milestone, for example for 2.21.2:
> https://gitlab.com/lilypond/lilypond/-/milestones/320#tab-labels
>
> Thanks,
> Jonas
>
> 1: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00284.html
> 2: https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00379.html
Hi Jonas,
I just took a quick glance on it and have many questions... :)
Let me summarize them with an example:

Take
https://gitlab.com/lilypond/lilypond/-/issues/5999

1. Federico wrote in the thread you mentioned [1]:
"In the last comment you should find a commit id (if it's missing you'll
have to search it)."

I think this refers to the time before Gitlab, when issues had
a message like

Trim unused toplevel targets.
author    Han-Wen Nienhuys <[hidden email]>
     Sun, 22 Mar 2020 00:16:36 +0100 (00:16 +0100)
committer    Han-Wen Nienhuys <[hidden email]>
     Tue, 31 Mar 2020 21:18:36 +0100 (22:18 +0200)
commit    0755e2f3dbb99ac256fb447d6ed14d3fe87b8ab5

at the end of the thread, right?

2. So I thought I had to look at the corresponding MRs to find
the commits I shall verify.
For #5999, there are two MRs listed as 'Related', which are:

https://gitlab.com/lilypond/lilypond/-/merge_requests/145
Add PDF bookmark support & multi-level TOC outline ability

and

https://gitlab.com/lilypond/lilypond/-/merge_requests/147
Avoid scm_append_x for property values

I'm not sure how !145 is related to #5999. Maybe #5999 did show up
after !145, and !147 did fix it.

Anyway, am I right that I have to extract all commit ids from both MRs
and check if they got into 2.21.2? Or do I have some faulty thinking here?
This would be very complicated and time-consuming, especially when MRs
consist of several commits.

3. What I do not get: A milestone is simply a named period of time, to which
you can assign MRs and issues, right? So after each release, you (or
someone else)
start a new milestone with the next version number. If an issue has been
closed, you
set the issue's milestone to the current milestone.
An issue is closed automatically when the corresponding MR has "Closes
#..." in the description.
So if an issue is closed and the milestone of the next release is added,
what is there left
to verify? Can't we be sure that the commits of the corresponding,
closed MR have gone in?

Michael


Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Jean ABOU SAMRA
Hi,

Le 17/09/2020 à 13:27, Michael Käppler a écrit :

>
> Am 17.09.2020 um 09:47 schrieb Jonas Hahnfeld:
>
>> Hi all,
>>
>> unless I'm missing something, there has been no activity on this since
>> the community effort in May [1][2] and issues since version 2.21.2
>> don't yet have Status::Verified labels. As I'm currently updating CG to
>> mention our use of milestones (not sure why I forgot about this
>> previously), the verification procedure should probably also get
>> updated.
>> I haven't been active in this area (and don't plan to be), so it would
>> be great if somebody could look over the current description (assuming
>> the procedure should be continued).
>>
>> To give a head start, here's again the list of all Status::Fixed:
>> https://gitlab.com/lilypond/lilypond/-/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=Status%3A%3AFixed
>>
>>
>> Furthermore you can directly get to the issues associated with a
>> milestone, for example for 2.21.2:
>> https://gitlab.com/lilypond/lilypond/-/milestones/320#tab-labels
>>
>> Thanks,
>> Jonas
>>
>> 1:
>> https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00284.html
>> 2:
>> https://lists.gnu.org/archive/html/lilypond-devel/2020-05/msg00379.html
> Hi Jonas,
> I just took a quick glance on it and have many questions... :)
> Let me summarize them with an example:
>
> Take
> https://gitlab.com/lilypond/lilypond/-/issues/5999
>
> 1. Federico wrote in the thread you mentioned [1]:
> "In the last comment you should find a commit id (if it's missing you'll
> have to search it)."
>
> I think this refers to the time before Gitlab, when issues had
> a message like
>
> Trim unused toplevel targets.
> author    Han-Wen Nienhuys <[hidden email]>
>     Sun, 22 Mar 2020 00:16:36 +0100 (00:16 +0100)
> committer    Han-Wen Nienhuys <[hidden email]>
>     Tue, 31 Mar 2020 21:18:36 +0100 (22:18 +0200)
> commit    0755e2f3dbb99ac256fb447d6ed14d3fe87b8ab5
>
> at the end of the thread, right?

I think so.

>
> 2. So I thought I had to look at the corresponding MRs to find
> the commits I shall verify.
> For #5999, there are two MRs listed as 'Related', which are:
>
> https://gitlab.com/lilypond/lilypond/-/merge_requests/145
> Add PDF bookmark support & multi-level TOC outline ability
>
> and
>
> https://gitlab.com/lilypond/lilypond/-/merge_requests/147
> Avoid scm_append_x for property values
>
> I'm not sure how !145 is related to #5999. Maybe #5999 did show up
> after !145, and !147 did fix it.
>
> Anyway, am I right that I have to extract all commit ids from both MRs
> and check if they got into 2.21.2? Or do I have some faulty thinking
> here?
> This would be very complicated and time-consuming, especially when MRs
> consist of several commits.
>
> 3. What I do not get: A milestone is simply a named period of time, to
> which
> you can assign MRs and issues, right?


As far as I understand, that's it, roughly. Milestones are just a kind
of extended
label concept (I think they integrate with the release feature somehow,
too). At any
rate, the documentation is over there:
https://docs.gitlab.com/ee/user/project/milestones/

> So after each release, you (or
> someone else)
> start a new milestone with the next version number. If an issue has been
> closed, you
> set the issue's milestone to the current milestone.
> An issue is closed automatically when the corresponding MR has "Closes
> #..." in the description.
> So if an issue is closed and the milestone of the next release is added,
> what is there left
> to verify? Can't we be sure that the commits of the corresponding,
> closed MR have gone in?
>
> Michael

In my opinion, we no longer need to verify patches. That is, there is
no need to check that the commits are present in master using committishes.
This was done when branches were manually managed, meaning that a patch
could
have slipped through because of a wrong usage of Git; I hardly see how
something could go wrong on GitLab.

Basically, the new setup for me would mean installing the development
release (downloaded from lilypond.org, not self-compiled, because
build issues are a non-negligible source for problems -- the Contributor's
Guide prescribes this), and going through all Status::Fixed issues,
testing wether the bug is effectively gone. And if there is a need for a
regression test or more documentation, opening a new issue for this.

What I do not yet understand is when milestones should be assigned
to issues and MRs. So far Jonas has done this from time to time. That was
after each release, right? Should we switch to doing this as part of the
verification process?

By the way, verifying regression tests is also in order…

Thoughts? I can help a bit next week-end.

For reference, here's the relevant CG chapter, which is to be updated:

http://lilypond.org/doc/v2.21/Documentation/contributor/bug-squad-checklists

Thanks,
Jean


Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Jonas Hahnfeld
Am Donnerstag, den 17.09.2020, 20:58 +0200 schrieb Jean Abou Samra:
> What I do not yet understand is when milestones should be assigned
> to issues and MRs. So far Jonas has done this from time to time. That was
> after each release, right?

Yes, mostly so. I'm just continuing what CG previously said about the
Fixed_x.y.z labels: Closed issues should be labeled Status::Fixed and
assigned a label, now a milestone, so they can be found. Additionally,
I assign all merge requests without a milestone after Phil does a
release and I need to create a new milestone anyway. That has the
benefit that the milestone bundles all information about the changes
that went into a release, in a single place.

> Should we switch to doing this as part of the verification process?

The problem I'm seeing is that it would become harder to find the
issues to verify: Most of the time, there will be fixed issues for the
current version still in development, which cannot be verified. Making
everybody wanting to help with verification skip over the same set of
issues potentially wastes a lot of time...

Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re[2]: issue verification

Trevor Daniels
Jonas, you wrote 17/09/2020 20:32:31
Subject: Re: issue verification

>The problem I'm seeing is that it would become harder to find the
>issues to verify: Most of the time, there will be fixed issues for the
>current version still in development, which cannot be verified. Making
>everybody wanting to help with verification skip over the same set of
>issues potentially wastes a lot of time...
>
>Jonas
Maybe I'm misunderstanding the problem, but a list of issues to verify
for a particular release, say 2.21.4, can be found by

Issues -> Milestones -> Release LilyPond 2.21.4 -> Milestone 2.21.4
Page down to see a list of completed issues (closed).

These are the ones to verify for 2.21.4 I believe.

Trevor


Reply | Threaded
Open this post in threaded view
|

Re: Re[2]: issue verification

Jonas Hahnfeld
Am Donnerstag, den 17.09.2020, 20:30 +0000 schrieb Trevor:

> Jonas, you wrote 17/09/2020 20:32:31
> Subject: Re: issue verification
>
> > The problem I'm seeing is that it would become harder to find the
> > issues to verify: Most of the time, there will be fixed issues for the
> > current version still in development, which cannot be verified. Making
> > everybody wanting to help with verification skip over the same set of
> > issues potentially wastes a lot of time...
> >
> > Jonas
> Maybe I'm misunderstanding the problem, but a list of issues to verify
> for a particular release, say 2.21.4, can be found by
>
> Issues -> Milestones -> Release LilyPond 2.21.4 -> Milestone 2.21.4
> Page down to see a list of completed issues (closed).
>
> These are the ones to verify for 2.21.4 I believe.
Right, that's what we currently have. As far as I understood Jean
wondered if we could move the step of assigning the milestone to
verification? In that case, the described procedure would not work.

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Jean ABOU SAMRA

Le 17/09/2020 à 22:34, Jonas Hahnfeld a écrit :

> Am Donnerstag, den 17.09.2020, 20:30 +0000 schrieb Trevor:
>> Maybe I'm misunderstanding the problem, but a list of issues to verify
>> for a particular release, say 2.21.4, can be found by
>>
>> Issues -> Milestones -> Release LilyPond 2.21.4 -> Milestone 2.21.4
>> Page down to see a list of completed issues (closed).
>>
>> These are the ones to verify for 2.21.4 I believe.
> Right, that's what we currently have. As far as I understood Jean
> wondered if we could move the step of assigning the milestone to
> verification? In that case, the described procedure would not work.

Yes, this answers my question.

In the future I'll assign milestones when merging my MRs and closing issues.

Best,
Jean


Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Michael Käppler-2
In reply to this post by Jonas Hahnfeld
Am 17.09.2020 um 21:32 schrieb Jonas Hahnfeld:

> Am Donnerstag, den 17.09.2020, 20:58 +0200 schrieb Jean Abou Samra:
>> What I do not yet understand is when milestones should be assigned
>> to issues and MRs. So far Jonas has done this from time to time. That was
>> after each release, right?
> Yes, mostly so. I'm just continuing what CG previously said about the
> Fixed_x.y.z labels: Closed issues should be labeled Status::Fixed and
> assigned a label, now a milestone, so they can be found. Additionally,
> I assign all merge requests without a milestone after Phil does a
> release and I need to create a new milestone anyway. That has the
> benefit that the milestone bundles all information about the changes
> that went into a release, in a single place.´
I still do not understand why this verification could be necessary in the
current process:

1. Phil did a release, let's say 2.21.7. you open a new milestone for
the next release,
2.21.8. You assign all merge requests that do not have a milestone (that
is, all
that were merged between 2.21.6 and 2.21.7) to milestone 2.21.7.
2. Someone tackles an open issue with a merge request. He writes
'Closes #1234' in the description. After the MR has been accepted and
merged, issue #1234 will be automatically closed.
(But Status::Fixed must be set manually, right?)
3. Phil does 2.21.8, and you or someone else does assign the milestone
'2.21.8'
to closed and fixed issues.

The cycle starts from the beginning.

 From my point of view, if an issue is
1. closed
2. marked as "Status::Fixed"
3. assigned to milestone x.y.z

one can be sure that it is fixed in release x.y.z
What should we verify?

Another point is what Jean mentioned, 'verifying' in the sense of
checking, whether a particular issue is really fixed in the release
that corresponds to the issue's milestone.

Do you mean we should try to start a new community effort to do that?

Michael





Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Michael Käppler-2
Am 17.09.2020 um 23:01 schrieb Michael Käppler:
> From my point of view, if an issue is
> 1. closed
> 2. marked as "Status::Fixed"
> 3. assigned to milestone x.y.z
>
> one can be sure that it is fixed in release x.y.z
What I meant was: 'that the commit with the purpose of
fixing the issue has gone into release x.y.z'


Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Jonas Hahnfeld
In reply to this post by Michael Käppler-2
Am Donnerstag, den 17.09.2020, 23:01 +0200 schrieb Michael Käppler:
> From my point of view, if an issue is
> 1. closed
> 2. marked as "Status::Fixed"
> 3. assigned to milestone x.y.z
>
> one can be sure that it is fixed in release x.y.z
> What should we verify?

See below.

> Another point is what Jean mentioned, 'verifying' in the sense of
> checking, whether a particular issue is really fixed in the release
> that corresponds to the issue's milestone.

Yes, that is what CG mentions should be done, please refer there.

> Do you mean we should try to start a new community effort to do that?

I'm not implying anything. The question was that somebody who knows
about the procedure (not me) should go over the instructions given in
the documentation. Until that happens, it's pointless to discuss among
three people (you, Jean, me) who don't know about it.

Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Michael Käppler-2
Am 18.09.2020 um 08:57 schrieb Jonas Hahnfeld:

> Am Donnerstag, den 17.09.2020, 23:01 +0200 schrieb Michael Käppler:
>>  From my point of view, if an issue is
>> 1. closed
>> 2. marked as "Status::Fixed"
>> 3. assigned to milestone x.y.z
>>
>> one can be sure that it is fixed in release x.y.z
>> What should we verify?
> See below.
>
>> Another point is what Jean mentioned, 'verifying' in the sense of
>> checking, whether a particular issue is really fixed in the release
>> that corresponds to the issue's milestone.
> Yes, that is what CG mentions should be done, please refer there.
I found the following section ambiguous:

 >>>>

"If the issue refers to a bug, try to reproduce the bug with the latest
officially released version (not one you’ve built yourself from source);
if the bug is no longer there, mark the issue “Status::Verified” (i.e.,
“the fix has been verified to work”).

Quite a few of these will be issues tracking patches. *You do not have
to prove these patches work - simply that they have been pushed into the
code base.* The developer should have put information similar to “Pushed
as as d8fce1e1ea2aca1a82e25e47805aef0f70f511b9” in the tracker. The long
list of letters and numbers is called the “committish”."

<<<<

Please let me give another try:

1. If an issue refers to a bug and is marked as Status::Fixed, test if
the bug is really gone in the latest release.
2. If an issue refers to another change (Enhancement, Documentation,
...) and has a MR assigned, verify if the corresponding MR
has gone into the release. (This should not be necessary with the
current process, IMHO).

>> Do you mean we should try to start a new community effort to do that?
> I'm not implying anything. The question was that somebody who knows
> about the procedure (not me) should go over the instructions given in
> the documentation. Until that happens, it's pointless to discuss among
> three people (you, Jean, me) who don't know about it.
Ok. My starting point was to help with the verifying process, which is
only possible, if I
understand what we want to do and how we want it to be done.
I'm not sure if waiting for more insightful people is promising, given
that up to now
nobody else replied.
Maybe the discussion gives some insight where the CG could be clearer
for people
like me who do not know about the procedure.

Thanks for your time,
Michael
Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Phil Holmes
In reply to this post by Jonas Hahnfeld
I don't know if this isn't clear, but just to state the original point of
verifying issues.

If the change is claimed to fix a bug, we compile the previously buggy code
with the release in which the bug is claimed fixed and check that the bug is
no longer there.  It it has really disappeared, we change the status of the
issue from Fixed to Verified.  i.e. we are certain that the bug is no longer
there.

If the change is to provide updated functionality, then it can be really
quite hard to verify the new functionality, and in any case the patch review
system should do that.  So we simply check the patch was pushed into the
claimed build.  If it's clear that it was, we mark the status as Verified.

That was the original intention of verifying issues.

--
Phil Holmes


----- Original Message -----
From: "Jonas Hahnfeld" <[hidden email]>
To: "Michael Käppler" <[hidden email]>; "Jean Abou Samra"
<[hidden email]>; "lilypond-devel" <[hidden email]>
Sent: Friday, September 18, 2020 7:57 AM
Subject: Re: issue verification



Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Michael Käppler-2
Am 18.09.2020 um 11:15 schrieb Phil Holmes:

> I don't know if this isn't clear, but just to state the original point
> of verifying issues.
>
> If the change is claimed to fix a bug, we compile the previously buggy
> code with the release in which the bug is claimed fixed and check that
> the bug is no longer there.  It it has really disappeared, we change
> the status of the issue from Fixed to Verified.  i.e. we are certain
> that the bug is no longer there.
>
> If the change is to provide updated functionality, then it can be
> really quite hard to verify the new functionality, and in any case the
> patch review system should do that.  So we simply check the patch was
> pushed into the claimed build.  If it's clear that it was, we mark the
> status as Verified.
>
> That was the original intention of verifying issues.
Thank you, Phil,
that confirms my previous thoughts.

I can start with verifying issues for 2.21.2 today.

Cheers,
Michael

>
> --
> Phil Holmes
>
>
> ----- Original Message ----- From: "Jonas Hahnfeld" <[hidden email]>
> To: "Michael Käppler" <[hidden email]>; "Jean Abou Samra"
> <[hidden email]>; "lilypond-devel" <[hidden email]>
> Sent: Friday, September 18, 2020 7:57 AM
> Subject: Re: issue verification
>
>


Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Michael Käppler-2
Am 18.09.2020 um 13:07 schrieb Michael Käppler:

> Am 18.09.2020 um 11:15 schrieb Phil Holmes:
>> I don't know if this isn't clear, but just to state the original point
>> of verifying issues.
>>
>> If the change is claimed to fix a bug, we compile the previously buggy
>> code with the release in which the bug is claimed fixed and check that
>> the bug is no longer there.  It it has really disappeared, we change
>> the status of the issue from Fixed to Verified.  i.e. we are certain
>> that the bug is no longer there.
>>
>> If the change is to provide updated functionality, then it can be
>> really quite hard to verify the new functionality, and in any case the
>> patch review system should do that.  So we simply check the patch was
>> pushed into the claimed build.  If it's clear that it was, we mark the
>> status as Verified.
>>
>> That was the original intention of verifying issues.
> Thank you, Phil,
> that confirms my previous thoughts.
>
> I can start with verifying issues for 2.21.2 today.
Just started, some further thoughts:

1. Having looked at only four issues now, there were already two that
described bugs, but lacked a description how to reproduce the problem.
See #6001 with a link to a very long thread on lilypond-devel and
#5995 with no clear error description. ('clear' in the sense of
understandable
for people that want to help verifying fixes and are not necessarily
involved with
coding.

2. Some issues cannot be verified with the actual releases since they
refer to special functionality like the GS API, which is not active in
the releases.  (See #5996)

3. Some issues can only be verified with special tools
(See #5983)

Cheers,
Michael






Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Federico Bruni-2
Il giorno ven 18 set 2020 alle 14:48, Michael Käppler
<[hidden email]> ha scritto:
> Having looked at only four issues now, there were already two that
> described bugs, but lacked a description how to reproduce the problem

Yes, it's quite common: I would say >90% of cases.
In my experience verifying issues was mainly checking the presence of
the commit in a git tag.
Current Gitlab workflow makes this kind of verification obsolete.




Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Jean ABOU SAMRA
In reply to this post by Phil Holmes
Hi,

Le 18/09/2020 à 11:15, Phil Holmes a écrit :

> I don't know if this isn't clear, but just to state the original point
> of verifying issues.
>
> If the change is claimed to fix a bug, we compile the previously buggy
> code with the release in which the bug is claimed fixed and check that
> the bug is no longer there.  It it has really disappeared, we change
> the status of the issue from Fixed to Verified.  i.e. we are certain
> that the bug is no longer there.
>
> If the change is to provide updated functionality, then it can be
> really quite hard to verify the new functionality, and in any case the
> patch review system should do that.  So we simply check the patch was
> pushed into the claimed build.  If it's clear that it was, we mark the
> status as Verified.
>
> That was the original intention of verifying issues.

Thanks for the insight.

So, Michael and I verified nearly all issues in Status::Fixed.
The ones remaining are listed here:

https://gitlab.com/lilypond/lilypond/-/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=Status%3A%3AFixed&not[milestone_title]=2.21.7

Among these,

https://gitlab.com/lilypond/lilypond/-/issues/6027

which I don't know how to test (Jonas?).

The others (5 items) are Windows-related. Maybe Michael or someone
else with access to a Windows machine can go over them?

Otherwise, part of the work was extremely simple, namely check that
the commits had made it into some branch under release/, which is fast;
and part of it was much harder, when trying to reproduce memory-related
bugs, for example. I reopened a few of these issues.

This doesn't answer Jonas' initial question: what should be done,
in terms of policies, with regard to issue verification? I don't
know. Personally I think it's not really needed as long as every
bug fix contains a regression test.

Speaking of regression tests, I also went through comparison pages
from "2.21.6 vs. 2.21.5" to "2.21.2 vs. 2.21.1" included. Honestly,
I've had my share of this; could somebody jump in to verify the
last two pages as well? These are:

lilypond.org/test/v2.21.1-1/compare-v2.21.0-1/index.html

http://lilypond.org/test/v2.21.0-1/compare-v2.20.0-1/index.html

Depending on your time and energy available, you may even go
farther. Or simply (I mean simple, not costless) verify the
entire test suite:

http://lilypond.org/doc/v2.21/input/regression/collated-files.html

That'll be all for today.

Best,
Jean


Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Michael Käppler-2
Am 18.09.2020 um 23:56 schrieb Jean Abou Samra:

> Hi,
>
> Le 18/09/2020 à 11:15, Phil Holmes a écrit :
>
>> I don't know if this isn't clear, but just to state the original
>> point of verifying issues.
>>
>> If the change is claimed to fix a bug, we compile the previously
>> buggy code with the release in which the bug is claimed fixed and
>> check that the bug is no longer there.  It it has really disappeared,
>> we change the status of the issue from Fixed to Verified.  i.e. we
>> are certain that the bug is no longer there.
>>
>> If the change is to provide updated functionality, then it can be
>> really quite hard to verify the new functionality, and in any case
>> the patch review system should do that.  So we simply check the patch
>> was pushed into the claimed build.  If it's clear that it was, we
>> mark the status as Verified.
>>
>> That was the original intention of verifying issues.
>
> Thanks for the insight.
>
> So, Michael and I verified nearly all issues in Status::Fixed.
Actually, Jean did the vast majority, because I had to leave on Friday
afternoon for a rehearsal weekend.

> The ones remaining are listed here:
>
> https://gitlab.com/lilypond/lilypond/-/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=Status%3A%3AFixed&not[milestone_title]=2.21.7
>
>
> Among these,
>
> https://gitlab.com/lilypond/lilypond/-/issues/6027
>
> which I don't know how to test (Jonas?).
>
> The others (5 items) are Windows-related. Maybe Michael or someone
> else with access to a Windows machine can go over them?
I verified them and noticed a couple of another issues.
(#6044-#6046)

Lilypond-book on Windows seems broken to me, because
the correct way to run it is neither obvious nor documented.
The script lacks a file ending (2.20.0 had lilypond-book.py, 2.21.6 only
lilypond-book) and can only be run like 'python lilypond-book foo.lytex'

Jean, what would you propose as a good way to ship these python scripts
for the Windows releases?

>
> Otherwise, part of the work was extremely simple, namely check that
> the commits had made it into some branch under release/, which is fast;
> and part of it was much harder, when trying to reproduce memory-related
> bugs, for example. I reopened a few of these issues.
>
> This doesn't answer Jonas' initial question: what should be done,
> in terms of policies, with regard to issue verification? I don't
> know. Personally I think it's not really needed as long as every
> bug fix contains a regression test.
Do we agree that 'verification' in the sense of
'checking whether a particular commit made it into a release x.y.z'
is not necessary anymore, because attached merge requests show that
the commit has gone in?

>
> Speaking of regression tests, I also went through comparison pages
> from "2.21.6 vs. 2.21.5" to "2.21.2 vs. 2.21.1" included. Honestly,
> I've had my share of this; could somebody jump in to verify the
> last two pages as well? These are:
>
> lilypond.org/test/v2.21.1-1/compare-v2.21.0-1/index.html
>
> http://lilypond.org/test/v2.21.0-1/compare-v2.20.0-1/index.html
I find that very difficult, because some changes may be intended, others
not...
Also there are often slight spacing changes that are probably "Heisenbugs",
like James mentioned.
>
> Depending on your time and energy available, you may even go
> farther. Or simply (I mean simple, not costless) verify the
> entire test suite:
>
> http://lilypond.org/doc/v2.21/input/regression/collated-files.html
>
> That'll be all for today.
Thank you very much, Jean!
>
> Best,
> Jean
>


Reply | Threaded
Open this post in threaded view
|

Re: issue verification

Jean ABOU SAMRA
Hi,

Le 21/09/2020 à 14:30, Michael Käppler a écrit :

> Am 18.09.2020 um 23:56 schrieb Jean Abou Samra:
>> Hi,
>>
>> Le 18/09/2020 à 11:15, Phil Holmes a écrit :
>>
>>> I don't know if this isn't clear, but just to state the original
>>> point of verifying issues.
>>>
>>> If the change is claimed to fix a bug, we compile the previously
>>> buggy code with the release in which the bug is claimed fixed and
>>> check that the bug is no longer there.  It it has really disappeared,
>>> we change the status of the issue from Fixed to Verified.  i.e. we
>>> are certain that the bug is no longer there.
>>>
>>> If the change is to provide updated functionality, then it can be
>>> really quite hard to verify the new functionality, and in any case
>>> the patch review system should do that.  So we simply check the patch
>>> was pushed into the claimed build.  If it's clear that it was, we
>>> mark the status as Verified.
>>>
>>> That was the original intention of verifying issues.
>>
>> Thanks for the insight.
>>
>> So, Michael and I verified nearly all issues in Status::Fixed.
> Actually, Jean did the vast majority, because I had to leave on Friday
> afternoon for a rehearsal weekend.

Don't worry, the majority of them was actually tracking patches,
and those were pretty quickly verified.


>> The ones remaining are listed here:
>>
>> https://gitlab.com/lilypond/lilypond/-/issues?scope=all&utf8=%E2%9C%93&state=closed&label_name[]=Status%3A%3AFixed&not[milestone_title]=2.21.7
>>
>>
>>
>> Among these,
>>
>> https://gitlab.com/lilypond/lilypond/-/issues/6027
>>
>> which I don't know how to test (Jonas?).
>>
>> The others (5 items) are Windows-related. Maybe Michael or someone
>> else with access to a Windows machine can go over them?
> I verified them and noticed a couple of another issues.
> (#6044-#6046).

Thanks!


> Lilypond-book on Windows seems broken to me, because
> the correct way to run it is neither obvious nor documented.
> The script lacks a file ending (2.20.0 had lilypond-book.py, 2.21.6 only
> lilypond-book) and can only be run like 'python lilypond-book foo.lytex'
>
> Jean, what would you propose as a good way to ship these python scripts
> for the Windows releases?

I'm not knowledgeable about this, sorry.You'd have to ask Jonas for example.


>>
>> Otherwise, part of the work was extremely simple, namely check that
>> the commits had made it into some branch under release/, which is fast;
>> and part of it was much harder, when trying to reproduce memory-related
>> bugs, for example. I reopened a few of these issues.
>>
>> This doesn't answer Jonas' initial question: what should be done,
>> in terms of policies, with regard to issue verification? I don't
>> know. Personally I think it's not really needed as long as every
>> bug fix contains a regression test.
> Do we agree that 'verification' in the sense of
> 'checking whether a particular commit made it into a release x.y.z'
> is not necessary anymore, because attached merge requests show that
> the commit has gone in?

I think there is sufficient agreement on that. Do you want to prepare
a merge request to update the CG?

>> Speaking of regression tests, I also went through comparison pages
>> from "2.21.6 vs. 2.21.5" to "2.21.2 vs. 2.21.1" included. Honestly,
>> I've had my share of this; could somebody jump in to verify the
>> last two pages as well? These are:
>>
>> lilypond.org/test/v2.21.1-1/compare-v2.21.0-1/index.html
>>
>> http://lilypond.org/test/v2.21.0-1/compare-v2.20.0-1/index.html
> I find that very difficult, because some changes may be intended, others
> not...
> Also there are often slight spacing changes that are probably
> "Heisenbugs",
> like James mentioned.

Those are the tricky part, indeed.

Best,
Jean