[RFC] Enabling GitLab CI

classic Classic list List threaded Threaded
31 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[RFC] Enabling GitLab CI

Jonas Hahnfeld
I'd like to propose that we enable GitLab CI and use it for
automatically testing our merge requests. It would run 'make', 'make
test', and 'make doc' like the current manual testing process. Once we
have that, GitLab can be configured to enforce a successful pipeline
before merging. As we only allow fast-forward merges, this means each
MR has received testing in the form it hits master. This would
effectively replace the current setup of pushing to staging.

The system would use Docker containers which is similar in spirit to
Han-Wen's proposal in February [1]. However the system is much simpler
and the containers don't come with a full development environment.
Instead they have barely enough to compile and test LilyPond.
For the actual changes and to get a feel of the UX, see the MR:
https://gitlab.com/lilypond/lilypond/-/merge_requests/57


Advantages
==========

 - We get automatic testing, tightly integrated into GitLab.
 - It helps the Patch Meister (James); see below for one caveat.
 - No patchy required for pushing to master.


Disadvantages
=============

 - Using Docker images, we get a single standardized environment for
the tests. While being great for reproducibility, it looses the
diversity we had so far (please refer to earlier discussions).
 - Comparison of regression tests is not yet integrated, the main
problem being the need for a baseline. I already have an idea or two
how this could work, but for now I'm focusing on the initial setup.
This means James still needs to download the patches and run 'make
check' (make doc being run automatically).
 - There's no possibility to push without a merge request, and it needs
to be merged through the UI. I don't consider this a loss personally.
 - There's no possibility to push multiple changes at once: Merge
requests are rebased, tested and merged one by one. GitLab is planning
to implement a queuing mechanism, and if all else fails we can invent
our own scripts.


Where to Run
============

Now towards infrastructure: Being a public project and open-source
GitLab currently offers 50,000 CI minutes per month for free. [1] This
has the advantage that test results are available immediately.
50,000 minutes divided by ~60 minutes per build, this is ~800 builds
per month, or around 25 per day. Not sure what happens if we exceed
this, comments claim it's "enforced" in some way. However there's no
page to see the current usage (for a public group project)...

If this is too little, if GitLab reduces the amount of free minutes, or
if we want to get faster builds, we can always add our own machines.
That is a matter of installing Docker and the runner (packages provided
by GitLab). Configuration is as simple as running one command and
pasting the URL as well as a token.
We can even go hybrid and use the best of both worlds. My tests suggest
that specific runners are preferred, using the shared runners as
fallback.

Let me know what you think
Jonas


1: https://lists.gnu.org/archive/html/lilypond-devel/2020-02/msg00416.html
2: Our 'lilypond' group is older than https://about.gitlab.com/releases/2020/03/18/ci-minutes-for-free-users/

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Dan Eble
On May 17, 2020, at 15:27, Jonas Hahnfeld <[hidden email]> wrote:
> if we want to get faster builds, we can always add our own machines.
> That is a matter of installing Docker and the runner (packages provided
> by GitLab). Configuration is as simple as running one command and
> pasting the URL as well as a token.

How much routine network traffic will the runner generate?  Obviously, it will need to pull the Docker image when it is updated, and that will happen occasionally, but I'm more curious about the building & testing itself.  Will there be large logs or build artifacts sent back to GitLab or otherwise served to people browsing the project?

I might be willing to plug in a cheap, headless computer to crank through patches night and day, but probably not if it will upload GBs of results.

Dan


Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

James Lowe-9
In reply to this post by Jonas Hahnfeld
On 17/05/2020 20:27, Jonas Hahnfeld wrote:
>   - Comparison of regression tests is not yet integrated, the main
> problem being the need for a baseline. I already have an idea or two
> how this could work, but for now I'm focusing on the initial setup.
> This means James still needs to download the patches and run 'make
> check' (make doc being run automatically).

Yes that would save me time.

Something that strikes me immediately.

* If a CI run makes doc are we going to have an entry and or label in
the comment for a success/failure?

I don't want to bother with any test if make/make doc fails, so I assume
that it will change the label to take it off patch new. But at the same
time if a patch is 'new' and hasn't had the make/make doc test, do I
still run the make check anyway? If so then it passed the label would go
to review and I assume the CI run would still catch that and do the
make/make doc. Otherwise I am going to not only have to check for a
patch 'new' I am going to have to know that it is waiting on a make doc
first if you see what I mean? (will I have to second guess the CI so to
speak)

Thanks.

James


Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Jonas Hahnfeld
In reply to this post by Dan Eble
Am Sonntag, den 17.05.2020, 16:10 -0400 schrieb Dan Eble:
> On May 17, 2020, at 15:27, Jonas Hahnfeld <[hidden email]> wrote:
> > if we want to get faster builds, we can always add our own machines.
> > That is a matter of installing Docker and the runner (packages provided
> > by GitLab). Configuration is as simple as running one command and
> > pasting the URL as well as a token.
>
> How much routine network traffic will the runner generate?  Obviously, it will need to pull the Docker image when it is updated, and that will happen occasionally, but I'm more curious about the building & testing itself.  Will there be large logs or build artifacts sent back to GitLab or otherwise served to people browsing the project?
>
> I might be willing to plug in a cheap, headless computer to crank through patches night and day, but probably not if it will upload GBs of results.

Obviously the builder needs to git-fetch the repo. Additionally the
current configuration uploads all untracked files after the build step
for usage in both test and doc. By disabling debug information, this is
only 22 MB in size with at most 1x upload plus 2x download (not sure if
it would cache locally). If this is of concern, we can of course fuse
the three stages without artifacts in between.

All log files (pure text) are uploaded to gitlab.com and served from
there. After a build is finished, the builder serves no state. Also
note that the runner contacts the server, ie no trouble with firewalls
/ NAT / dynamic IPs et al.

Regards
Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Karlin High
In reply to this post by Dan Eble
On Sun, May 17, 2020, 3:11 PM Dan Eble <[hidden email]> wrote:

> I might be willing to plug in a cheap, headless computer to crank through
> patches night and day, but probably not if it will upload GBs of results.
>

I can also offer this, from a computer shop with lots of spare hardware and
an unmetered 30 mbps up & down Internet connection. I've done various
distributed computing things like Folding@Home and Storj, and this doesn't
sound much harder.
--
Karlin High
Missouri, USA

>
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Jonas Hahnfeld
In reply to this post by James Lowe-9
Am Sonntag, den 17.05.2020, 21:14 +0100 schrieb James Lowe:

> On 17/05/2020 20:27, Jonas Hahnfeld wrote:
> >   - Comparison of regression tests is not yet integrated, the main
> > problem being the need for a baseline. I already have an idea or two
> > how this could work, but for now I'm focusing on the initial setup.
> > This means James still needs to download the patches and run 'make
> > check' (make doc being run automatically).
>
> Yes that would save me time.
>
> Something that strikes me immediately.
>
> * If a CI run makes doc are we going to have an entry and or label in
> the comment for a success/failure?
>
> I don't want to bother with any test if make/make doc fails, so I assume
> that it will change the label to take it off patch new. But at the same
> time if a patch is 'new' and hasn't had the make/make doc test, do I
> still run the make check anyway? If so then it passed the label would go
> to review and I assume the CI run would still catch that and do the
> make/make doc. Otherwise I am going to not only have to check for a
> patch 'new' I am going to have to know that it is waiting on a make doc
> first if you see what I mean? (will I have to second guess the CI so to
> speak)
I'm not sure I fully understand the question(s).

First of all, the system tests every new commit added to a merge
request, no matter how small or if the diff is the same as before.
(That's the advantage of computers, they don't get bored at doing the
same over and over again.) If the tests pass, you get nice green ticks
everywhere. On error GitLab shows red crosses instead, in particular
the stage that failed (build, test, doc). This currently doesn't add
the Patch::needs_work label, but I can certainly make our bot do this.

As the pipelines start immediately after the push, they might already
be finished when you look at the merge request. If something didn't
pass, I think it's perfectly ok to skip manual tests.
In case the pipelines are still running or even pending, I think you
should ignore that merge request for the time being and maybe come back
later. We'll see how much of an issue this turns out to be.

Regards
Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

James Lowe-9
On 18/05/2020 10:29, Jonas Hahnfeld wrote:

> Am Sonntag, den 17.05.2020, 21:14 +0100 schrieb James Lowe:
>> On 17/05/2020 20:27, Jonas Hahnfeld wrote:
>>>    - Comparison of regression tests is not yet integrated, the main
>>> problem being the need for a baseline. I already have an idea or two
>>> how this could work, but for now I'm focusing on the initial setup.
>>> This means James still needs to download the patches and run 'make
>>> check' (make doc being run automatically).
>> Yes that would save me time.
>>
>> Something that strikes me immediately.
>>
>> * If a CI run makes doc are we going to have an entry and or label in
>> the comment for a success/failure?
>>
>> I don't want to bother with any test if make/make doc fails, so I assume
>> that it will change the label to take it off patch new. But at the same
>> time if a patch is 'new' and hasn't had the make/make doc test, do I
>> still run the make check anyway? If so then it passed the label would go
>> to review and I assume the CI run would still catch that and do the
>> make/make doc. Otherwise I am going to not only have to check for a
>> patch 'new' I am going to have to know that it is waiting on a make doc
>> first if you see what I mean? (will I have to second guess the CI so to
>> speak)
> I'm not sure I fully understand the question(s).
>
> First of all, the system tests every new commit added to a merge
> request, no matter how small or if the diff is the same as before.

Yes but make doc isn't instantaneous it takes time .. a good amount of
time. No matter how small the diff is.

> (That's the advantage of computers, they don't get bored at doing the
> same over and over again.) If the tests pass, you get nice green ticks
> everywhere. On error GitLab shows red crosses instead, in particular
> the stage that failed (build, test, doc). This currently doesn't add
> the Patch::needs_work label, but I can certainly make our bot do this.

Yes please, I need a simple way to know not even to bother testing a
make check because if make doc fails the entire patch is useless. In
some cases stuff that breaks make doc also would break 'make' (when
generating our 'info' pages, some doc edits if not done correctly
manifest themselves for just make rather than breaking only 'make doc).

>
> As the pipelines start immediately after the push, they might already
> be finished when you look at the merge request. If something didn't
> pass, I think it's perfectly ok to skip manual tests.
but how do I know? That is the nub of what I am asking. If a patch is
'new' how do I know that an automated make doc is 'in progress, has
completed with errors, has completed without errors' as I am not going
to bother to do any work on it if it is the first two cases. What is the
point?
> In case the pipelines are still running or even pending, I think you
> should ignore that merge request for the time being and maybe come back
> later. We'll see how much of an issue this turns out to be.

It will be an issue. Unless it is easy for me to check.


Imagine you sit down and see 20 patch-new merge requests to test.

Which one do you do first?

Which one has already had its make doc run?

Which one has failed/passed/is in progress of its make doc run?


I hope that is clearer.

James


Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Kevin Barry
On Mon, May 18, 2020 at 10:48:55AM +0100, James Lowe wrote:
> but how do I know? That is the nub of what I am asking. If a patch is 'new'
> how do I know that an automated make doc is 'in progress, has completed with
> errors, has completed without errors' as I am not going to bother to do any
> work on it if it is the first two cases. What is the point?

The merge request list has an icon next to every merge request showing
the current pipline status. There are different icons for passed,
failed and in-progress. Is that what you want? See the merge request
page for inkscape:
https://gitlab.com/groups/inkscape/-/merge_requests

Is the information you would like to be able to see visible there?

It will only show a pass if all stages pass, as far as I know.

Kevin

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Urs Liska-3
Am Montag, den 18.05.2020, 11:15 +0100 schrieb Kevin Barry:

> On Mon, May 18, 2020 at 10:48:55AM +0100, James Lowe wrote:
> > but how do I know? That is the nub of what I am asking. If a patch
> > is 'new'
> > how do I know that an automated make doc is 'in progress, has
> > completed with
> > errors, has completed without errors' as I am not going to bother
> > to do any
> > work on it if it is the first two cases. What is the point?
>
> The merge request list has an icon next to every merge request
> showing
> the current pipline status. There are different icons for passed,
> failed and in-progress. Is that what you want? See the merge request
> page for inkscape:
> https://gitlab.com/groups/inkscape/-/merge_requests
>
> Is the information you would like to be able to see visible there?

No, at least not at the time I looked.
What James needs is additionally an icon that states that MR is
*currently* being tested.

But that will also be visible in the interface (which Jonas addressed
with his comment that in this case you can simply skip the MR and have
a look later when it can be assumed finished.

HTH
Urs

>
> It will only show a pass if all stages pass, as far as I know.
>
> Kevin
>


Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Kevin Barry
On Mon, May 18, 2020 at 12:17:53PM +0200, Urs Liska wrote:
> No, at least not at the time I looked.
> What James needs is additionally an icon that states that MR is
> *currently* being tested.

There is an icon for that (it's blue and looks like a half-filled pie
chart) - I just couldn't find a merge request list that actually had any
in progress.

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

James Lowe-9
On 18/05/2020 11:21, Kevin Barry wrote:
> On Mon, May 18, 2020 at 12:17:53PM +0200, Urs Liska wrote:
>> No, at least not at the time I looked.
>> What James needs is additionally an icon that states that MR is
>> *currently* being tested.
> There is an icon for that (it's blue and looks like a half-filled pie
> chart) - I just couldn't find a merge request list that actually had any
> in progress.

OK I don't want to go hunting if I can avoid it, and not clicking 3 or 4
pages to get to that.

I have two basic methods at the moment.

Countdown.py (which is Jonas' great cli tool) it's what you see when I
do the countdown (that's literally cut/paste).

Or

*if* i really have to go into gitab's UI I live here

https://gitlab.com/lilypond/lilypond/-/labels

and all I do is click on the 'merge request' link and this gives me the
current MR with their label.

Apart from that the only time I edit or view and entire MR is to add my
text/label.

While it may not seem such a big deal, when you have to refresh / click
different pages or scroll here and there to see a 'button' or 'flashy
icon' in an already-too-busy interface, it adds a lot of friction. So
the less I have to click/do before I begin the testing/countdown process
the better.

James


Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Kevin Barry
On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote:
> Countdown.py (which is Jonas' great cli tool) it's what you see when I do
> the countdown (that's literally cut/paste).

I haven't seen that script, but the gitlab API exposes pipeline
information. It should be enough to correlate a merge request with a
pipeline status (running, pending, success, failed, etc).

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Jonas Hahnfeld
Am Montag, den 18.05.2020, 11:53 +0100 schrieb Kevin Barry:
> On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote:
> > Countdown.py (which is Jonas' great cli tool) it's what you see when I do
> > the countdown (that's literally cut/paste).
>
> I haven't seen that script, but the gitlab API exposes pipeline
> information. It should be enough to correlate a merge request with a
> pipeline status (running, pending, success, failed, etc).

For reference, it's here:
https://gitlab.com/lilypond/infrastructure/-/blob/master/countdown.py

The API does expose this information, but the script needs to query the
pipeline status per merge request as it's not included in the list of
MRs. That sounds expensive, I have to experiment with this.

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Jonas Hahnfeld
Am Montag, den 18.05.2020, 12:59 +0200 schrieb Jonas Hahnfeld:

> Am Montag, den 18.05.2020, 11:53 +0100 schrieb Kevin Barry:
> > On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote:
> > > Countdown.py (which is Jonas' great cli tool) it's what you see when I do
> > > the countdown (that's literally cut/paste).
> >
> > I haven't seen that script, but the gitlab API exposes pipeline
> > information. It should be enough to correlate a merge request with a
> > pipeline status (running, pending, success, failed, etc).
>
> For reference, it's here:
> https://gitlab.com/lilypond/infrastructure/-/blob/master/countdown.py
>
> The API does expose this information, but the script needs to query the
> pipeline status per merge request as it's not included in the list of
> MRs. That sounds expensive, I have to experiment with this.
See https://gitlab.com/lilypond/infrastructure/-/merge_requests/5

With a few tweaks I got this into an acceptable range, but it still
takes up to 10 seconds for ~20 merge requests. That's the cost you pay
for O(n) requests to the API and parsing all that response data...

Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Jonas Hahnfeld
Am Montag, den 18.05.2020, 17:50 +0200 schrieb Jonas Hahnfeld:

> Am Montag, den 18.05.2020, 12:59 +0200 schrieb Jonas Hahnfeld:
> > Am Montag, den 18.05.2020, 11:53 +0100 schrieb Kevin Barry:
> > > On Mon, May 18, 2020 at 11:29:35AM +0100, James Lowe wrote:
> > > > Countdown.py (which is Jonas' great cli tool) it's what you see when I do
> > > > the countdown (that's literally cut/paste).
> > >
> > > I haven't seen that script, but the gitlab API exposes pipeline
> > > information. It should be enough to correlate a merge request with a
> > > pipeline status (running, pending, success, failed, etc).
> >
> > For reference, it's here:
> > https://gitlab.com/lilypond/infrastructure/-/blob/master/countdown.py
> >
> >
> > The API does expose this information, but the script needs to query the
> > pipeline status per merge request as it's not included in the list of
> > MRs. That sounds expensive, I have to experiment with this.
>
> See
> https://gitlab.com/lilypond/infrastructure/-/merge_requests/5
>
>
> With a few tweaks I got this into an acceptable range, but it still
> takes up to 10 seconds for ~20 merge requests. That's the cost you pay
> for O(n) requests to the API and parsing all that response data...
To make this less ambiguous: The time scales with the number of merge
requests that have label Patch::new. Right now, this is zero (good job,
James!) so for testing I waived this "optimization".

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Dan Eble
In reply to this post by Jonas Hahnfeld
On May 17, 2020, at 15:27, Jonas Hahnfeld <[hidden email]> wrote:
>
> before merging. As we only allow fast-forward merges, this means each
> MR has received testing in the form it hits master. This would
> effectively replace the current setup of pushing to staging.

I'm for this.

Dan


Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Jonas Hahnfeld
Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble:
> On May 17, 2020, at 15:27, Jonas Hahnfeld <[hidden email]> wrote:
> > before merging. As we only allow fast-forward merges, this means each
> > MR has received testing in the form it hits master. This would
> > effectively replace the current setup of pushing to staging.
>
> I'm for this.

Thanks. What do others (David, Han-Wen, Valentin) think about this?
There's certainly room for improvement, but with an initial setup I can
start writing documentation.

Jonas

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

Han-Wen Nienhuys-3
On Thu, May 21, 2020 at 12:34 PM Jonas Hahnfeld <[hidden email]> wrote:

>
> Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble:
> > On May 17, 2020, at 15:27, Jonas Hahnfeld <[hidden email]> wrote:
> > > before merging. As we only allow fast-forward merges, this means each
> > > MR has received testing in the form it hits master. This would
> > > effectively replace the current setup of pushing to staging.
> >
> > I'm for this.
>
> Thanks. What do others (David, Han-Wen, Valentin) think about this?
> There's certainly room for improvement, but with an initial setup I can
> start writing documentation.

Let's try it.

I think the FF requirement might lead to traffic jams on countdown
days, so a next step might be making the countdown process more
continuous.  But we don't have to address that problem right now.

> Jonas



--
Han-Wen Nienhuys - [hidden email] - http://www.xs4all.nl/~hanwen

Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

James Lowe-9

On 21/05/2020 12:02, Han-Wen Nienhuys wrote:
> so a next step might be making the countdown process more
> continuous.

What does that mean - even conceptually?

The countdown is specifically to allow everyone some time to breathe and
digest patches submitted without the fear of having to be always-on and
missing anything. I understand the 'jam' on countdown days, but honestly
how is having a countdown every day (or continuously) going to change that?

I think this was touched on before, when you came back into the
development group didn't seem to get the point of what my role was for
in regard to countdowns.

I know that other developers have been frustrated with the countdown as
well (at least at first) and while I appreciate that not all patches
need that much of a review, it has been evident even in these these last
few weeks that some things you have felt trivial or self-evident has
lead to much discussion and debate.

Also note that I currently do a 2 day countdown now (mainly because I
have the scope working from home), the original idea of the countdown
was for 4 days as that was felt to give enough time for everyone to see
a submitted patch AND give it consideration AND continue to develop at
the same time. We have more devlopers and many more patches than then
and we're alreadyt working on a shorter countdown.

So please remember the point of countdown, it's a deliberate braking
system to stop rushing things through that some might want to care
about. That doesn't mean I need to do it, but neither does it mean we
need 'continuous' countdown (whatever that means).

Thanks.

James



Reply | Threaded
Open this post in threaded view
|

Re: [RFC] Enabling GitLab CI

David Kastrup
In reply to this post by Jonas Hahnfeld
Jonas Hahnfeld <[hidden email]> writes:

> Am Dienstag, den 19.05.2020, 08:08 -0400 schrieb Dan Eble:
>> On May 17, 2020, at 15:27, Jonas Hahnfeld <[hidden email]> wrote:
>> > before merging. As we only allow fast-forward merges, this means each
>> > MR has received testing in the form it hits master. This would
>> > effectively replace the current setup of pushing to staging.
>>
>> I'm for this.
>
> Thanks. What do others (David, Han-Wen, Valentin) think about this?
> There's certainly room for improvement, but with an initial setup I can
> start writing documentation.

The "traffic jam" problem could be avoided by retaining the option of
pushing to staging.  That would occur without CI, but one could
occasionally trigger the merge with CI on staging to have everything in
it migrate to master.  Since staging would be used by the more
experienced people desiring to bunch their work before testing, the
triggering could also happen manually by whoever thinks he has pushed
enough stuff to staging to give it a whirl.

--
David Kastrup

12