[plan] Remove blockers to the "minimum viable" state of DrupalCI to ship Drupal 8 RC1 [#2467925] | Drupal.org

[plan] Remove blockers to the "minimum viable" state of DrupalCI to ship Drupal 8 RC1 [#2467925] | Drupal.org
Skip to search
Can we use first and third party cookies and web beacons to
understand our audience, and to tailor promotions you see
[plan] Remove blockers to the "minimum viable" state of DrupalCI to ship Drupal 8 RC1
Closed (fixed)
Project:
DrupalCI: Drupal.org Testing Infrastructure
Component:
Documentation
Priority:
Critical
Category:
Plan
Assigned:
webchick
Issue tags:
alpha target
infrastructure blocker for Drupal 8.0.0
revisit July 1 2015
blocked by drupalci
Reporter:
webchick
Created:
8 Apr 2015 at 22:46 UTC
Updated:
11 Sep 2015 at 14:44 UTC
Jump to comment:
Most recent
Most recent file
Coming from the
DrupalCI sprint debrief call
today, it was raised that we really need firm answers to the following questions prior to deployment.
For the "simplest thing that can possibly be deployed" to make Drupal 8 shippable:
In general, we only need patch-level testing against
one
environment, generally aiming toward the most common setup users are likely to have (which in the case of Drupal 8 is PHP 5.5 and MySQL 5.5, based on stats below). So that's #1 in the following list of required environments:
PHP 5.5 & MySQL 5.5
PHP 5.6 & MySQL 5.5
PHP 7 & MySQL 5.5
PHP 5.5 & SQLite 3.6.8
PHP 5.5 & PostgreSQL 9.1
PHP 5.5 & PostgreSQL 9.4
Additionally, we need:
Nightly testing with emails upon failure including a
diff against the previous night's result
for environments that do not currently pass the test suite in HEAD. (probably #3-6.) The committers adopt a policy of reverting things that introduce a new failure.
Commit-level testing of combinations that do pass in HEAD (currently #1 and possibly #2).
Patch-level testing of #1 with an
option
of on-demand patch-level testing for other combinations. We need this so that we can test issues that are likely to regress alternate databases against those databases, and to confirm that fixes for a specific environment actually fix them.
All of this easily visible in the UI and issue queue.
Then we can decide whether to increase the test rate for #3-#6 once they pass, weighing the benefit against the extra resource requirements.
If any of those items would take long to implement though, please say so, because maybe we can get away with less than that, or substitute one of those capabilities for something easier to implement. Having even one of those bullet points up and running sooner would be better than additional weeks/months of not having any of them.
Resources
Drupal 8's minimum PHP version (5.5)
Drupal 8's minimum MySQL/Maria version (5.5.3)
- or will be once
#2473301: Raise MySQL requirement to 5.5.3
is in.
Drupal 8's minimum SQLite version (3.6.8)
Drupal 8's minimum PostgreSQL version (8.3)
W3tech PHP version usage stats
(PHP 5.3 = 42.6%; PHP 5.4 = 29.3%)
WordPress PHP/MySQL version usage stats
(PHP 5.3 = 36.3%; PHP 5.4 = 38.4%; MySQL 5.5 = 60.3%)
Note that PHP 5.3 is also very common so is probably what we want to test D6/D7 against (we can't test D8 against it because D8 requires 5.4.5).
Current State of DrupalCI In Production
DrupalCI testing is now enabled on production for Drupal Core and Contrib. The
automated testing tab
now shows the DrupalCI tests and test configuration matrix below the PIFT/PIFR tests. All of the MVP environments exist. The test summary 'bubbles' on this page link through to the
test history for that environment
Hitlist for DrupalCI MVP (Drupal 8 blockers)
DrupalCI MVP - Drupal 8 Blockers
Order
Issue
Assignment
Comments
#2524426: [meta] DrupalCI pgsql Random exceptions, fails, testbot issues
isntall
Needs support of pgsql core devs
#2508567: DrupalCI SQLite random failures
isntall
Needs support of sqlite core devs
#2539496: PHP 7 testing needs recent builds and needs to display the build number
isntall
#2538462: Contrib Testing Test discovery not working for d6/d7
jthorson
?,
mixologic
#2518026: Notify core maintainers (and anyone who wants to subscribe) of test failures (for a branch & environment) with a diff against the previous nights result
drumm
8.x release candidate 'very nice to haves':
DrupalCI MVP - Drupal 8 very nice to haves
Order
Issue
Assignment
Comments
n/a
#2495411: Make simpletest fail a test when it detects pages that need more than 64MB
Core devs
De-prioritized
#2552853: MySQL and MariaDB test results should be consistent
Unassigned
Official MVP spec says we support MySQL not MariaDB, this should be followup, not blocker
Complete
Order
Issue
Assignment
Comments
n/a
#2518008: Improve the XML output/parsing of the test runner
mixologic
lars-skjærlund
n/a
Re-test/Add-Test UI
drumm
n/a
PHP7 Environment
isntall
n/a
#2517980: [DrupalCI Integration] Enable patch testing
drumm
Issues blocking retiring the legacy testbots are being prioritized here:
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
Comment
File
Size
Author
#57
Screen Shot 2015-07-16 at 9.24.39 AM.png
57.15 KB
webchick
#49
d7-core-testing.png
73.93 KB
David_Rothstein
Comments
Comment
#1
drumm
he/him
NY, US
commented
8 April 2015 at 22:48
Issue summary:
View changes
or
to post comments
Comment
#2
alexpott
he/they
🇪🇺🌍
commented
8 April 2015 at 22:50
Related issues:
#2465221: Raise the minimun version requirement for SQLite to 3.6.8
or
to post comments
Comment
#3
alexpott
he/they
🇪🇺🌍
commented
8 April 2015 at 22:53
Thinking about:
What level of testing on each? (per-patch; per-commit; periodically, and if so, what period?)
It would be nice if all patches had PHPUnit tests run on all supported PHP minor versions - it only takes a minute to do a full PHPUnit run.
or
to post comments
Comment
#4
xjm
she/her
commented
9 April 2015 at 12:00
I do not think that it is a requirement to test the full matrix of all databases vs. PHP versions etc. as the chance of having a failure that only occurs on PHP 5.5. with SQLite is low. I think the minimum requirement is to test core's supported databases in the minimum supported version, and core's supported PHP versions. We should test combinations that make it easiest to quickly identify the potential cause of the failure:
PHP 5.4 & MySQL/Maria
PHP 5.5 & MySQL/Maria
PHP 5.6 & MySQL/Maria
PHP 7 & MySQL/Maria
PHP 5.4 & SQLite
PHP 5.4 & PostgreSQL
I do not consider different cache backends, PHP configurations, etc. a release blocker; all that should wait until much later.
In terms of timing, I think the minimum requirement that we have is:
Nightly testing with emails upon failure including a
diff against the previous night's result
for environments that do not currently pass the test suite in HEAD. (AFAIK in my list above this would be #3-6.) The committers adopt a policy of reverting things that introduce a new failure.
Commit-level testing of combinations that do pass in HEAD (currently #1 and I think #2).
Patch-level testing of #1 with an
option
of on-demand patch-level testing for other combinations. We need this so that we can test issues that are likely to regress alternate databases against those databases, and to confirm that fixes for a specific environment actually... fix.
All of this easily visible in the UI and issue queue.
Then we can decide whether to increase the test rate for #3-#6 once they pass, weighing the benefit against the extra resource requirements.
I like @alexpott's suggestion of running PHPunit tests on all of #1-6 on the patch level as well since the resource overhead is minimal. We could also test simply installation and the database component tests per patch as well to save resources while getting most of the benefit.
or
to post comments
Comment
#5
effulgentsia
commented
10 April 2015 at 18:23
+1 to the 6 combinations and 4 bullet points in
#4
as being a good viable state. If any of those items would take long to implement though, please say so, because maybe we can get away with less than that, or substitute one of those capabilities for something easier to implement. Having even one of those bullet points up and running sooner would be better than additional weeks/months of not having any of them.
or
to post comments
Comment
#6
webchick
she/they
Vancouver 🇨🇦
commented
10 April 2015 at 18:35
What version of MySQL/Maria, PostgreSQL, and SQLite? Minimum?
or
to post comments
Comment
#7
xjm
she/her
commented
13 April 2015 at 16:09
@webchick, yeah, I think it should be just the minimum supported version for each.
or
to post comments
Comment
#8
webchick
she/they
Vancouver 🇨🇦
commented
14 April 2015 at 19:41
Sitting with Daniel and Sascha at dev days... Their 2 cents:
- PHP 5.4 + Maria - patch-level testing. If it fails, return a fail. Test this FIRST before running.
- everything else - every 24 hours testing of HEAD to catch regressions, along with an indicator of +/- fails versus the last run. If we do these as patch-level tests, we will vastly increase testing time and/or amazon cost required. We can always revert any patches that introduce new fails in any environment.
IMO, we also need to add PHP 5.3 for Drupal 7. That's got the highest install base ATM. I don't have the URL but google "PHP usage stats" or something like that.
That's the MVP. In the "ASAP after that" phase:
PHPUnit testing against all PHP versions + Maria. Later, all kernel tests.
In the "would be very nice" phase:
- testbot goes to all needs review issues, and tries to apply the latest patch (not run tests necessarily, since most will not apply, and this way to could be done very fast).
Note: This is typed from my phone so might not be 100% :)
or
to post comments
Comment
#9
catch
he/him
commented
22 April 2015 at 19:43
+1 to #4, we might want to tweak things later but this would be a great point to start from.
or
to post comments
Comment
#10
webchick
she/they
Vancouver 🇨🇦
commented
22 April 2015 at 19:53
Assigned:
Unassigned
webchick
Ok great. Working on updating the issue summary.
or
to post comments
Comment
#11
webchick
she/they
Vancouver 🇨🇦
commented
22 April 2015 at 20:59
Issue summary:
View changes
Status:
Active
» Reviewed & tested by the community
Ok, think that covers it.
or
to post comments
Comment
#12
webchick
she/they
Vancouver 🇨🇦
commented
22 April 2015 at 21:00
Issue summary:
View changes
$HTML--;
or
to post comments
Comment
#13
webchick
she/they
Vancouver 🇨🇦
commented
22 April 2015 at 21:03
Issue summary:
View changes
Small clarification.
or
to post comments
Comment
#14
hestenet
He/Him
Portland, OR 🇺🇸
commented
23 April 2015 at 18:27
The association team met up to review these MVP requirements this morning and wanted make some clarifications and ask a couple questions.
With respect to MySQL/MariaDB - for now, wherever the requirements list above says "MySQL/MariaDB" we'll be opting for MySQL. At the same time we want to note that in the long run we may not want to treat these as interchangeable, as MariaDB may begin to diverge in some future versions.
Regarding Postgres - During the Portland sprint we came to conclusion that we'd prefer to support Postgres 9.1.2 as our minimum version. We know that 8.3 is still considered an official minimum, per:
but
PHP 8 is well past end of life
That decision isn't really ours to make of course - but this issue makes the case for 9.1.2 as well:
#2464481: PostgreSQL: deal with case insensitivity
So is it satisfactory to use MySQL where ever MySQL/MariaDB has been suggested above and to build Maria containers sometime post MVP?
And can we at least strongly consider 9.1.2 as minimum version for Postgres on the new testbots?
or
to post comments
Comment
#15
webchick
she/they
Vancouver 🇨🇦
commented
23 April 2015 at 21:14
Unless anyone else disagrees:
So is it satisfactory to use MySQL where ever MySQL/MariaDB has been suggested above and to build Maria containers sometime post MVP?
Yep. This is inline with using whatever would be the most common platform, and MariaDB definitely isn't it. My understanding is also that they're more or less inline until a later version of 5.5.
And can we at least strongly consider 9.1.2 as minimum version for Postgres on the new testbots?
I think having
any
version of PostgreSQL at all is preferable to having no version of PostgreSQL so tentatively +1 on 9.1.2. However, it'd feel better if core went the same route. ATM though there doesn't look like there's a strong consensus for a version move in
#2464481: PostgreSQL: deal with case insensitivity
yet. One of the objections was that moving to 9.1.2 would expose
new
test failures, and
seems to back that up.
So definitely yes to strongly consider. But it might be easier to move to 8.3 in the "MVP" version, unless other folks from the DrupalCI team want to chime in at the core issue with why 8.3 is bad mojo.
or
to post comments
Comment
#16
mradcliffe
he/him
commented
24 April 2015 at 01:55
PostgreSQL 8.3
is EOL on Centos 5 and 6 as well. 9.1.2 probably should be the earliest version to support.
On
Ubuntu 14.04 (Trusty)
, which the testbots run on, the earliest version is PostgreSQL 8.4 though earlier versions are still supported on older Ubuntu versions.
or
to post comments
Comment
#17
xjm
she/her
commented
24 April 2015 at 16:35
Thanks @hestenet and @mradcliffe. I filed
#2477413: Increase minimum version requirement for Postgres to 9.1.2
to discuss increasing the supported minimum version for core for 8.x. (I think one dedicated PHP 5.3/MySQL bot is sufficient for D7 testing, so we don't need to worry about supporting testing on the versions of Postgres supported by D7.)
or
to post comments
Comment
#18
catch
he/him
commented
27 April 2015 at 12:48
Doesn't the current infrastructure run MariaDB? Either way I think it's OK to treat them as interchangeable until we don't.
I'd personally tend towards testing later versions of postgres (i.e. not the minimum, but the current stable). At least this way if core raises its requirements, then no need to update the bot at the same time.
Also in terms of postgres/sqlite testing if we have one bot, I'd probably tend towards current PHP stable on those too (i.e. 5.5) - since again if we drop support for PHP 5.4 it would just mean stopping sending 8.x patches to that bot, not 'HEAD broken' on postgres/sqlite until an upgrade of PHP on those.
Don't have a strong preference, just seems like it could be less disruptive when core requirements change.
or
to post comments
Comment
#19
Crell
commented
27 April 2015 at 14:56
If PHP 5.5 is what we are going to list as the recommended version, I'm fine with having testbots run that. However, we should at least still run a linting pass on 5.4 to catch accidental use of newer language features as 5.5 does have a few convenience shortcuts that we don't want people to use accidentally (yet). Functionality-wise, the 5.4->5.5 upgrade is, as I understand it, smooth enough that we likely don't need the full test suite run on both.
or
to post comments
Comment
#20
catch
he/him
commented
27 April 2015 at 15:28
However, we should at least still run a linting pass on 5.4
That would be covered by the MySQL + PHP 5.4 bot. MySQL we'll have tests with every major PHP version.
For 5.5 testing I was specifically talking about the postgres and sqlite bots which at least to start with will only test on one PHP version each.
We'll get commit-level (not patch-level) testing on everything that's expected to pass if I understand things correctly. And a test run every 24 hours for environments with known failures.
or
to post comments
Comment
#21
drumm
he/him
NY, US
commented
5 May 2015 at 23:55
Issue summary:
View changes
Doesn't the current infrastructure run MariaDB? Either way I think it's OK to treat them as interchangeable until we don't.
Yes, for Drupal.org sites. The DrupalCI containers do not necessarily need to follow.
So is it satisfactory to use MySQL where ever MySQL/MariaDB has been suggested above and to build Maria containers sometime post MVP?
Yep. This is inline with using whatever would be the most common platform, and MariaDB definitely isn't it. My understanding is also that they're more or less inline until a later version of 5.5.
Updating the issue summary for this.
or
to post comments
Comment
#22
hestenet
He/Him
Portland, OR 🇺🇸
commented
6 May 2015 at 03:31
As we drill down into this a bit further we realized we could use a bit more clarification around this requirement:
Nightly testing with emails upon failure including a
diff against the previous night's result
for environments that do not currently pass the test suite in HEAD. (probably #3-6.) The committers adopt a policy of reverting things that introduce a new failure.
Could someone unpack what they'd like to see for the diff against the previous night's result a bit further?
Is this diff a diff of which tests were passing/failing the previous night vs. which passed/failed in the latest run? Or is there something more specific needed?
or
to post comments
Comment
#23
webchick
she/they
Vancouver 🇨🇦
commented
6 May 2015 at 15:02
I'll run this past the core committer team tomorrow for more specifics, but IMO it's a diff of which tests were passing/failing versus the previous night's run.
Basically, we want to be able to catch things like this:
do this currently by providing an aggregate +/- number and a link to the complete log of a test run for more details.
or
to post comments
Comment
#24
amateescu
commented
6 May 2015 at 15:35
For the "ideal state" side of things, I think that sometime in future when everything works properly, patch-level testing for core should be done on the fastest possible PHP / DB combination, which probably means PHP 7 & SQLite in memory/tmpfs.
or
to post comments
Comment
#25
webchick
she/they
Vancouver 🇨🇦
commented
28 May 2015 at 16:13
Issue tags:
revisit July 1 2015
We spoke about the various "fix it or nix it" critical issues on our Drupal core branch maintainer call today. While there's no way to know when we're going to hit the "~5 critical issue" threshold, the DA is in the process of planning a D8 Accelerate sprint from July 2 - 8. So July 1 is a logical date set as the deadline for things like PostgreSQL support.
or
to post comments
Comment
#26
Mixologic
He/Him/His
Portland, OR
commented
29 May 2015 at 01:09
Issue tags:
blocked by drupalci
or
to post comments
Comment
#27
hestenet
He/Him
Portland, OR 🇺🇸
commented
15 June 2015 at 19:36
Issue summary:
View changes
As further discussed in: [2477413] - For the DrupalCI MVP we'll be able to provide Postgres 9.1.18 and 9.4.4 (rather than Postgres 8.3). At this time that issue is awaiting RTBC, but there seems to be consensus. Updating the issue summary here accordingly.
or
to post comments
Comment
#28
hestenet
He/Him
Portland, OR 🇺🇸
commented
15 June 2015 at 19:41
Issue summary:
View changes
or
to post comments
Comment
#29
daffie
commented
18 June 2015 at 08:55
Issue summary:
View changes
Dries has approved
#2296557: [policy] Require PHP 5.5
. So the lowest PHP version we are going to use is 5.5. Updated the issue summary to reflect this.
or
to post comments
Comment
#30
hestenet
He/Him
Portland, OR 🇺🇸
commented
2 July 2015 at 05:43
Issue summary:
View changes
Adding some much needed detail into the DA's work on deploying the DrupalCI MVP. My apologies that more of these updates weren't included here in real-time!
DrupalCI MVP Deployment to Drupal.org
On June 11th the DA
demoed the DrupalCI integration with Drupal.org
with some of the core maintainers. We collected some initial feedback, asked some questions, and set out the path to bring DrupalCI into production over the course of the next two weeks. The main architectural shift with this MVP proposal was to remove the results server component of DrupalCI in favor of directly parsing test results on the Jenkins/Dispatcher instance and exposing those results on Drupal.org.
On June 17th the DA enabled DrupalCI in production for branch and commit testing for Drupal Core. The
automated testing tab
now shows the DrupalCI tests and test configuration matrix below the PIFT/PIFR tests. At the same time we enabled some initial jobs for the environments that were stable. (In particular, holding off on the Postgres testing as we worked to make the results we were getting from these bots more consistent). We emailed a number of the core maintainers to let them know that these tests were now available.
Shortly after these initial tests were enabled the
minimum version of php for Drupal 8 was increased to php 5.5
, so we updated the tests accordingly.
On June 23rd the DA enabled testing of Postgres on Drupal.org after better stablizing the Postgres bots. However - we are still seeing some inconsistencies in test results from one run to the next on the Postgres bots - and could still use the assistance of core devs working on Postgres support (particularly those who have stable containers running locally) in fixing up the stability of our Postgres environment.
@Drumm also created
test history pages on Drupal.org
which can be seen by clicking through any of the test result summaries on the Automated Testing tab of the project, (and will be available wherever else those summaries appear, such as next to a patch in an issue). The test history page provides a summary and meta data about the most current test, as well as a link to the results in Jenkins. It also provides a history of the past tests run in the same environment.
Since that time we have 1) begun rearchitecting the jobs in Jenkins to better allow us to provide a diff against the previous results for each environment - a precursor to enabling the email notifications to core maintainers, and 2) begun improving the Junit/XML output of the test runner to provide a better organization of the test results (by group) and expose additional detail, such as debug messages that are available in the qa.drupal.org version of the tests results.
I'll update the results summary with the current hit list, link in or create issues for those where appropriate, and also list some items that are out of scope for initial deployment, but that will need to be done to retire PIFT/PIFR.
(As a sidenote - we had some breakthroughs that enabled us to dramatically improve the stability of the old testbots, so that maintaining those bots has required much less labor, saving us from being pulled away from this work. Thank you @mixologic!)
or
to post comments
Comment
#31
hestenet
He/Him
Portland, OR 🇺🇸
commented
2 July 2015 at 06:17
Issue summary:
View changes
or
to post comments
Comment
#32
hestenet
He/Him
Portland, OR 🇺🇸
commented
2 July 2015 at 06:21
Issue summary:
View changes
or
to post comments
Comment
#33
webchick
she/they
Vancouver 🇨🇦
commented
2 July 2015 at 06:31
Thank you SO MUCH for that update!!
or
to post comments
Comment
#34
webchick
she/they
Vancouver 🇨🇦
commented
7 July 2015 at 22:36
So by way of an update on the Drupal 8 core side...
This past week we've had the
D8 Accelerate "Lemon" sprint
where
a butt-load of critical issues were fixed
, bringing our
current total down to 17
Of those 17 issues, two of them are blocked on testbots running PHP 5.5:
#2508231: Raise minimum required version of PHP to 5.5.9
(which makes this requirement change for D8) and
#2493911: Update guzzle, goutte and mink-goutte-driver to the latest release
(which depends on that one getting done).
The momentum around criticals right now makes it
possible
, though certainly not
certain
, that the next D8 beta might be our
last
D8 beta (!).
We would really like to put the PHP 5.5 requirement in the next beta, so we can get some wider testing and shake out any fallout bugs prior to RC1.
The release managers have been talking (nothing definitively announced yet) about releasing the next beta next week,
on July 15
. (We can be a bit flexible with that date by a few days / up to a week, but August 5 would essentially be our "drop-dead" date for RC1 if we want to have a prayer of releasing Drupal 8.0.0 prior to DrupalCon Barcelona, and we need at least a couple of weeks before that to allow for testing.)
So I know there is branch-level testing with the new testbot happening on
and it
seems to be passing pretty reliably
(yay!). But we're still not seeing patch-level testing for DrupalCI, and that's the level of testing we would need in order to raise the requirement since obviously without that,
all
of our tests would fail the old PIFT bots running PHP 5.4. :P
I'm not sure how close DrupalCI patch testing is to completion... we might need to update the PIFT bots instead as a short-term solution?
TL;DR: Any ETA when PHP 5.5 bots will be available for D8? :D
or
to post comments
Comment
#35
basic
commented
7 July 2015 at 23:30
Support for PHP 5.5 was mostly complete for the PIFR/PIFT bots already, and @isntall finished the pieces missing there. In short:
We have bots ready to run PHP 5.5 on pifr/pift
@drumm was waiting to update d.o to request the PHP 5.5 bots based core devs need (we NEED to have it on the pifr/pift ones because we need DCI patch testing along side pifr/pift patch testing to verify we are regression free)
Since we want to raise that minimum requirement to 5.5 on the 15th (middle of next week) we’re out of time to turn on DCI patch testing and have any reasonable length of time to do side by side testing
We need to schedule this change in a change record/change notice for next week and coordinate with core devs the time we switch pifr/pift to 5.5
or
to post comments
Comment
#36
chx
commented
7 July 2015 at 23:38
Let me ask (this is vital for one of my patches): will these bots have opcache on? I presume yes otherwise they'd be very slow but I need to ask.
or
to post comments
Comment
#37
isntall
commented
7 July 2015 at 23:49
chx: For DrupalCI Opcache has been enabled by default . So will APCu. And for the old testbots on qa.d.o we will also have Opcache and APCu enabled by default.
or
to post comments
Comment
#38
effulgentsia
commented
8 July 2015 at 01:05
We need to schedule this change in a change record/change notice for next week and coordinate with core devs the time we switch pifr/pift to 5.5
Sorry if this is a stupid question, but why not just do it? Like as soon as someone with the necessary creds is able to perform the switch? I'm assuming we're only talking about making this switch for testing 8.x patches, in which case, how would this impact core devs in a way that requires coordination? As far as I'm concerned, the sooner it happens the better, no need to wait until the 15th.
or
to post comments
Comment
#39
webchick
she/they
Vancouver 🇨🇦
commented
8 July 2015 at 05:10
Yeah, I would agree there. If the switch to PHP 5.5 wasn't made until the 15th, that means we'd have to postpone the beta until at least July 17th, because we'd be committing two big patches which would need at least a couple of days to check for regressions that were accidentally introduced and could only be caught with manual testing. And that assumes the bot switchover to PHP 5.5 introduces absolutely no fall-out problems testbot-wise, and it's only core's bugs we need to deal with (though I personally can't remember a change we've made to PIFT/PIFR that's ever gone 100% right the first time. :P)
It'd be awesome if it could instead happen either later this week or early next week.
or
to post comments
Comment
#40
webchick
she/they
Vancouver 🇨🇦
commented
8 July 2015 at 05:12
THAT SAID, I seem to recall that PIFR/PIFT isn't smart enough to run different PHP versions on different versions of Drupal. And if that's the case, we would need to check
very
carefully to make sure that we didn't kill automated testing in D7 and especially D6 in the process.
or
to post comments
Comment
#41
berdir
German
Switzerland
commented
8 July 2015 at 06:16
Drupal 8 is already running on 5.4 while 7 and 6 still uses 5.3. So switching 5.4 to 5.5 should not affect them?
or
to post comments
Comment
#42
fabianx
commented
8 July 2015 at 06:30
#38
: I am pretty happy they want to coordinate with core-devs.
i.e. if they had "just" done the testbot migration in the middle of "Lemon", that would have severely impacted productivity.
In that regard (and because there could be problems), I think it is good to warn in advance.
+1 to warn one or two days in advance (more is not necessary though).
or
to post comments
Comment
#43
webchick
she/they
Vancouver 🇨🇦
commented
8 July 2015 at 06:37
#41: Oh, whew! Carry on then. :)
#42: Yep, fully agreed. All I was saying is "please don't switch it the same day as the beta, we need a couple of days to shake out any issues." :)
or
to post comments
Comment
#44
xjm
she/her
commented
8 July 2015 at 09:41
Today is the last day of the sprint. If infra is ready, I'd recommend announcing the 5.5 change for perhaps Friday July 10, or if weekend deployment is an option, over the weekend? I agree on "sooner the better" in general.
or
to post comments
Comment
#45
Mixologic
He/Him/His
Portland, OR
commented
8 July 2015 at 18:16
The current plan is to build a php5.5.9 testbot today, and manually force it to grab the patch from
, though that currently doesnt appear to apply, may need a reroll? (
or
to post comments
Comment
#46
Mixologic
He/Him/His
Portland, OR
commented
8 July 2015 at 20:06
We're running into difficulties on getting the legacy testbots running 5.5.9 (they are on debian wheezy), so in the interests of time we're going to go ahead with legacy testbots on 5.5.26.
Currently, the DCI testbots are running on 5.5.23. We're working on downgrading the container to 5.5.9.
If our smoke test succeeds and we proceed with the legacy testbot upgrade, we will have 5.5.26 in parallel with 5.5.23.
During that time, green patches will be "greenish" - meaning that there is a slim chance that any patch going in could be impacted by whatever php fixes were put into place between 5.5.9 and 5.5.26, and that there may be one or more 'new fails' revealed once we start using the drupalCI containers at 5.5.9.
Hopefully thats acceptable risk, otherwise we can spend a lot more time trying to figure out how to get the legacy testbots on 5.5.9.
or
to post comments
Comment
#47
catch
he/him
commented
8 July 2015 at 20:09
I don't think 5.5.26/23 is a problem at all tbh.
or
to post comments
Comment
#48
webchick
she/they
Vancouver 🇨🇦
commented
8 July 2015 at 22:46
Yeah, agreed. Please don't spend
a lot of
any cycles on downgrading. If it turns out there are massive differences between 5.5.9 and 5.5.2[whatever] we can always spin up 5.5.9 as one of the alternate environments sometime in the next few months.
The blocker is 5.5.* not 5.5.9.
or
to post comments
Comment
#49
David_Rothstein
commented
11 July 2015 at 15:59
Issue summary:
View changes
Status
File
Size
new
d7-core-testing.png
73.93 KB
Wow, it's really cool the multi-environment testing is live!
Why isn't it turned on for Drupal 7, though? I seem to have the power to turn it on via the UI (and would like to) but don't want to if it will break something. Will it?
In the short term, this is actually more important for Drupal 7 than Drupal 8, since there are hundreds of thousands of live Drupal 7 sites actually running with these various combinations of PHP/database versions...
More specifically I'm asking about doing this:
or
to post comments
Comment
#50
drumm
he/him
NY, US
commented
11 July 2015 at 17:15
D7 has some differences we haven't been looking for yet.
runtests.sh
is in a different location, which we have set up:
. The best way to help is to run the test runner containers locally against D7 to see if tests pass, and if other environment variables need to be set.
Within 8.x, core maintainers should go ahead and re-configure as necessary.
By the way, if any of those select boxes don't make sense to run, maybe 8.x with PHP 5.4, some of those select boxes can be removed to declutter things. This should help out contrib maintainers, so the test combinations are helpful.
or
to post comments
Comment
#51
webchick
she/they
Vancouver 🇨🇦
commented
12 July 2015 at 20:18
Drupal 8 on PHP 5.4 (or below) definitely doesn't make sense anymore, so if removing that possibility is possible (heh), that sounds great.
or
to post comments
Comment
#52
drumm
he/him
NY, US
commented
14 July 2015 at 20:05
Drupal 8 on PHP 5.4 (or below) definitely doesn't make sense anymore, so if removing that possibility is possible (heh), that sounds great.
Done.
or
to post comments
Comment
#53
tvn
commented
15 July 2015 at 12:04
Category:
Task
» Plan
or
to post comments
Comment
#54
jibran
he/him
Urdu
Toronto, Canada
commented
16 July 2015 at 16:21
I don't know where to post this but can we test patches in
#2530640: PHP 7 testing issue
on
PHP 7 & MySQL 5.5
or
to post comments
Comment
#55
webchick
she/they
Vancouver 🇨🇦
commented
16 July 2015 at 16:26
Yeah, seems like we're still missing PHP 7.
shows only:
Test PHP 5.4 & MySQL 5.5
Retest PHP 5.5 & MySQL 5.5
Test PHP 5.6 & MySQL 5.5
Test PHP 5.4 & SQLite 3.8
Test PHP 5.5 & SQLite 3.8
Test PHP 5.5 & PostgreSQL 9.1
or
to post comments
Comment
#56
webchick
she/they
Vancouver 🇨🇦
commented
16 July 2015 at 16:26
And incidentally, we can drop all PHP 5.4 combos from this list for D8.
or
to post comments
Comment
#57
webchick
she/they
Vancouver 🇨🇦
commented
16 July 2015 at 16:29
Status
File
Size
new
Screen Shot 2015-07-16 at 9.24.39 AM.png
57.15 KB
However, when PHP 7 is added to that list, you get there by clicking the "Add tests" link on a test result:
This takes you to a
list of checkboxes
where you can queue different environments for test/re-test.
or
to post comments
Comment
#58
hestenet
He/Him
Portland, OR 🇺🇸
commented
16 July 2015 at 22:59
Issue summary:
View changes
or
to post comments
Comment
#59
catch
he/him
commented
20 July 2015 at 13:11
We may need to consider testing on both MariaDB and MySQL.
(via @alexpott) shows a local fail on MySQL (without long index support) whereas MariaDB passed.
or
to post comments
Comment
#60
alexpott
he/they
🇪🇺🌍
commented
20 July 2015 at 13:12
Also
Drupal\system\Tests\Entity\EntityDefinitionUpdateTest
fails locally for long indexes.
or
to post comments
Comment
#61
morgantocker
commented
20 July 2015 at 15:22
@catch, @alexpott: this is a false pass due to configuration (strict sql mode is now enabled by newer versions of MySQL; it is not in MariaDB, where this is a warning instead). The same issue is discussed in
Summary of discussion:
- Truncation is unsafe on unique indexes because a constraint can not safely be applied.
- It is mostly safe on selective non-unique indexes.
- Thus "not-safe" to rely on database truncation by disabling strict sql mode to create tables.
- A more complete fix is available in 5.7, where the barracuda file format + innodb_large_prefix is enabled by default. This allows indexes up to 3072 bytes.
or
to post comments
Comment
#62
morgantocker
commented
20 July 2015 at 15:33
There are a number of features which have been removed or deprecated in MySQL 5.7. May I suggest adding it to the test-matrix as well, so that no features are introduced which specifically make use of what is intended to be removed?
See:
or
to post comments
Comment
#63
hestenet
He/Him
Portland, OR 🇺🇸
commented
22 July 2015 at 18:42
Issue summary:
View changes
Updating the summary to reflect that:
Contrib testing is now enabled
PHP 7 testing is enabled
Also - this has been called out through some back channels, but
please
update:
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
if you see any regressions between old bots to new, issues with the test output, etc -- basically any feedback you have not covered by existing issues that would prevent turning off the old testbots.
or
to post comments
Comment
#64
David_Rothstein
commented
22 July 2015 at 20:19
Awesome!
It looks like PHP 7 testing is only an option for Drupal 8 though (not Drupal 7) - any reason?
I'm also wondering about the fact that PHP 5.3 is not an option, even though the legacy testbots run on that (but will bring that up in more detail at
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
). If a project decides to enable the new options now (and e.g. selects "PHP 5.4 & MySQL 5.5" as the one to run on "Issues & commit") would that mean they lose the ability to test patches on PHP 5.3 with the old testbot?
or
to post comments
Comment
#65
hestenet
He/Him
Portland, OR 🇺🇸
commented
22 July 2015 at 20:28
Getting 5.3 running in DrupalCI is definitely part of the plan (it's hiding in the issue summary up there under "Not in the scope of MVP, but required so that we can retire PIFT/PIFR" probably needs its own child issue at this point). Getting those bots working and getting CI working for D7 in general is definitely a next major priority.
or
to post comments
Comment
#66
David_Rothstein
commented
22 July 2015 at 20:31
Ah thanks, sorry I missed that. Well it's now mentioned explicitly at
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
too, which is probably a good thing :)
or
to post comments
Comment
#67
hestenet
He/Him
Portland, OR 🇺🇸
commented
22 July 2015 at 20:37
Issue summary:
View changes
Definitely a good thing. Added a new child issue(
#2538042: DrupalCI PHP 5.3 containers need to be stable and available for testing
) and updated the summary
or
to post comments
Comment
#68
drumm
he/him
NY, US
commented
22 July 2015 at 20:42
It looks like PHP 7 testing is only an option for Drupal 8 though (not Drupal 7) - any reason?
I assumed Drupal 7 wasn't going to support PHP 7. I can add that as an option if testing there is useful.
or
to post comments
Comment
#69
webchick
she/they
Vancouver 🇨🇦
commented
22 July 2015 at 20:46
In all likelihood, D7 will still be around for another 5+ years, so I think making PHP7 pass on it makes a lot of sense.
or
to post comments
Comment
#70
catch
he/him
commented
22 July 2015 at 20:49
Following up to #59 for both the short and long term, I think we should change the default test runner to a MySQL or MariaDB configuration that doesn't support long indexes - then patches added indexes that are too long will fail and we won't break HEAD. afaik that's the only issue we have where MySQL version/variant matters at the moment.
Multiple MySQL/MariaDB testing would be great to have down the line for other things that might come up, but even if we have that, the default should be as strict as we support.
or
to post comments
Comment
#71
xjm
she/her
commented
22 July 2015 at 20:52
On
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
@David_Rothstein pointed out that we probably need a 5.3 environment for D7 testing only.
or
to post comments
Comment
#72
berdir
German
Switzerland
commented
22 July 2015 at 20:53
Based on a test run I did, it looked like D7 actually had fewer issues with PHP 7, at least the sort that requires API changes (like renaming String/Null/... classes). So enabling certainly doesn't hurt.
Yay, PHP 7 testing.
Would be awesome to include a php --version into the build output so it's clear against what we are testing exactly. For example, a lot of those reported exceptions have been fixed a while back, see
If you look at my test results in
#2454439: [META] Support PHP 7
, there were new PHP 7 changes and bugs every other week that broke stuff again. So it would be pretty important to have an up to date PHP7 build that is updated from time to time, e.g. with the latest beta/alpha version or just build it again every few weeks until we have a stable release. Is that possible? That would allow us to quickly identify regressions/changes in PHP7.
or
to post comments
Comment
#73
morgantocker
commented
22 July 2015 at 21:03
@catch, what did you think about my comment re: 5.7 deprecation? (
) I ran the d8 test-suite some time back and it didn't appear to be using any features scheduled to be removed, but it's possible they could be introduced. This is a similar-ish problem to PHP7.
or
to post comments
Comment
#74
catch
he/him
commented
22 July 2015 at 21:12
@morgantocker that's good to add to 'ideal state', I don't think it's 'minimal state' at all. Changing the default to not support long indexes feels like 'minimal state' though.
or
to post comments
Comment
#75
David_Rothstein
commented
22 July 2015 at 21:30
I assumed Drupal 7 wasn't going to support PHP 7. I can add that as an option if testing there is useful.
Please do - thanks! It's not something that's badly needed right away, but as others pointed out above it will be needed eventually.
The initial tests Berdir did with PHP 7 on Drupal 7 are at
#2454439-31: [META] Support PHP 7
- so even before people are ready to use Drupal 7 on PHP 7 in practice, it's still useful to have a testbot keep track of any progress on that issue (rather than having to run the tests manually).
or
to post comments
Comment
#76
drumm
he/him
NY, US
commented
22 July 2015 at 22:33
7 for 7 is enabled now.
Some contrib projects have proactively enabled DrupalCI testing for Drupal 7.x modules, and you can see how it fails,
. I'm not the expert on this, I think the SQLite DB used by SimpleTest for results is set up a bit differently in D7.
For contrib, and even core, do go ahead and remove the legacy testing when DrupalCI is meeting your needs.
or
to post comments
Comment
#77
hestenet
He/Him
Portland, OR 🇺🇸
commented
23 July 2015 at 15:15
Issue summary:
View changes
or
to post comments
Comment
#78
webchick
she/they
Vancouver 🇨🇦
commented
24 July 2015 at 03:42
Tagged a few issues this evening as blocked by drupalci. I'm not 100% positive they are, but there's definitely some overlap:
#1713332: The SQLite database driver fails to drop simpletest tables
#2508567: DrupalCI SQLite random failures
#2524426: [meta] DrupalCI pgsql Random exceptions, fails, testbot issues
Suggestion from @catch (via @xjm) was that we might be able to work around these failures by at least temporarily using single concurrency for the SQLite and Postgres builds, which could be acceptable since these test suites mostly only run nightly. Is that an option to get more reliable results from DrupalCI until any other issues are resolved?
or
to post comments
Comment
#79
yesct
commented
30 July 2015 at 15:38
Related issues:
#2204253: Output PHP version of testbot environment in review log
@Berdir in
#72
include a php --version into the build output
I guess that would be
#2204253: Output PHP version of testbot environment in review log
or
to post comments
Comment
#80
webchick
she/they
Vancouver 🇨🇦
commented
14 August 2015 at 06:20
Title:
[policy, no patch] Determine both the "minimum viable" and "ideal state" of PHP/database/etc. versions
» [policy, no patch] Determine the "minimum viable" state of DrupalCI to ship Drupal 8
Status:
Reviewed & tested by the community
» Active
Turning this issue into more of a "grab bag" so we have one centralized place to discuss this.
We had a pow-wow today with most of the core committers (@xjm, @alexbronstein, @alexpott, @catch) as well as most of the DA tech team (@drumm, @basic, @isntall, @joshuami, @Mixologic, @hestenet, @japerry... hope I didn't forget anyone!) to go through the list at
#2267715: [meta] Drupal.org (websites/infra) blockers to a Drupal 8 RC1
DrupalCI currently supports the following environments:
PHP 5.4 & MySQL 5.5
PHP 5.5 & MySQL 5.5
PHP 5.6 & MySQL 5.5
PHP 7 & MySQL 5.5
PHP 5.4 & SQLite 3.8
PHP 5.5 & SQLite 3.8
PHP 5.5 & PostgreSQL 9.1
The one combination we're missing from the issue summary is PHP 5.5 & PostgreSQL 9.4 but I don't know why we'd need to test against two PostgreSQL versions as a "must-have." Testing against PostgreSQL
at all
is light years beyond our previous capabilities. This appears to have been added by @hestenet in
#27
, but I would suggest removing it and putting 9.4 in the nice-to-have bucket.
As a result, I closed out both
#697220: Run tests on all supported database servers
and
#1867192: Testbots need to run on 5.4, 5.5, 5.6 and 7
. Hooray!!
That said, there is still more work to do in terms of unblocking Drupal 8 RC1:
1. SQLite isn't passing:
(current stats 13,218 pass, 9 fail). There was some debate on whether SQLite ever
actually
passed, given the random fails in
#2508567: DrupalCI SQLite random failures
. One theory was that those random fails were caused by concurrency, but even turning concurrency off doesn't help.
2. PostgreSQL not only isn't passing, but seems to only be running ~100 tests atm:
Something appears to have gone awry August 13, 2015 - 17:21 when results bumped from 93,455 pass, 201 fail to 12,744 pass, 58 fail and then again on August 11, 2015 - 11:38 when they bumped down to 203 pass, 30 fail.
3. catch brought up in
#70
a weird environment issue we hit with MariaDB that MySQL testbots didn't catch. Rather than spin up a whole new MariaDB environment, suggestion was to configure the bots to turn off long index support on MySQL.
4. PHP 7 support doesn't really help us until
#2539496: PHP 7 testing needs recent builds and needs to display the build number
is done.
5. In the opposite direction, we are trying to get in
#2495411: Make simpletest fail a test when it detects pages that need more than 64MB
to help with the D8 critical issue
#2497185: [no patch] Create standardized core profiling scenarios and start tracking metrics for them
. That's passing on DrupalCI, but failing on PIFT.
I think that was everything, but please feel free to amend if I skipped something!
or
to post comments
Comment
#81
mradcliffe
he/him
commented
14 August 2015 at 18:34
#2 is my fault. I had success running a certain configuration and asked @isntall to try it to see if it would fix random exceptions, but it ended up not being successfully on AWS. The commit has been reverted, but the container may not have been rebuilt yet.
or
to post comments
Comment
#82
hestenet
He/Him
Portland, OR 🇺🇸
commented
17 August 2015 at 20:44
Issue summary:
View changes
or
to post comments
Comment
#83
hestenet
He/Him
Portland, OR 🇺🇸
commented
17 August 2015 at 20:52
Issue summary:
View changes
or
to post comments
Comment
#84
hestenet
He/Him
Portland, OR 🇺🇸
commented
17 August 2015 at 21:45
Issue summary:
View changes
Related issues:
#2552853: MySQL and MariaDB test results should be consistent
, +
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
or
to post comments
Comment
#85
hestenet
He/Him
Portland, OR 🇺🇸
commented
18 August 2015 at 15:07
Going to do some reorganization on the issue summary here so that we're only tracking the D8 blockers in this issue and that we use
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
for the disable legacy testbots hit list. Also going to add some more insight into our priority order and who on the DA team is working on these.
or
to post comments
Comment
#86
hestenet
He/Him
Portland, OR 🇺🇸
commented
18 August 2015 at 15:14
Issue summary:
View changes
Branched feature requests for the Drupal 8.1.x to a new issue
#2553289: [meta] Feature requests for DrupalCI to support Drupal 8.1.x
or
to post comments
Comment
#87
hestenet
He/Him
Portland, OR 🇺🇸
commented
18 August 2015 at 15:41
Issue summary:
View changes
or
to post comments
Comment
#88
hestenet
He/Him
Portland, OR 🇺🇸
commented
18 August 2015 at 15:42
Issue summary:
View changes
or
to post comments
Comment
#89
hestenet
He/Him
Portland, OR 🇺🇸
commented
19 August 2015 at 01:57
Issue summary:
View changes
or
to post comments
Comment
#90
catch
he/him
commented
26 August 2015 at 21:23
Issue summary:
View changes
or
to post comments
Comment
#91
webchick
she/they
Vancouver 🇨🇦
commented
26 August 2015 at 21:38
Was talking about this today with @catch and @effulgentsia.
We might actually be done here, because...
#2524426: [meta] DrupalCI pgsql Random exceptions, fails, testbot issues
appears to have been tracked down to a regression on the Drupal core side:
#2548725: Fix regression to menu serialization in upgrade path that causes thousands of errors in PostgreSQL
(need to confirm with @mradcliffe)
#2508567: DrupalCI SQLite random failures
and
#2539496: PHP 7 testing needs recent builds and needs to display the build number
are done.
#2538462: Contrib Testing Test discovery not working for d6/d7
and
#2518026: Notify core maintainers (and anyone who wants to subscribe) of test failures (for a branch & environment) with a diff against the previous nights result
seem like they're more appropriate blocking
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
than RC1.
#2495411: Make simpletest fail a test when it detects pages that need more than 64MB
and
#2552853: MySQL and MariaDB test results should be consistent
are (very) nice to haves.
The only issues left in the
blocked by drupalci
queue are self-referential to this issue and the pgsql issue.
So, once we get confirmation that core and not drupalci is the problem there, we may be able to mark this sucker
fixed
! :)
or
to post comments
Comment
#92
webchick
she/they
Vancouver 🇨🇦
commented
26 August 2015 at 21:45
Title:
[policy, no patch] Determine the "minimum viable" state of DrupalCI to ship Drupal 8
» [plan] Remove blockers to the "minimum viable" state of DrupalCI to ship Drupal 8 RC1
Status:
Active
» Reviewed & tested by the community
For better visibility...
or
to post comments
Comment
#93
webchick
she/they
Vancouver 🇨🇦
commented
28 August 2015 at 14:43
Status:
Reviewed & tested by the community
» Fixed
Discussed with most of the other core committers yesterday—@xjm, @catch, @alexpott, @effulgentsia—and they agreed that in terms of blocking D8, DrupalCI is in a "done enough" state. Hooray!
Since
#2538462: Contrib Testing Test discovery not working for d6/d7
and
#2518026: Notify core maintainers (and anyone who wants to subscribe) of test failures (for a branch & environment) with a diff against the previous nights result
are now children of
#2534132: Disable Legacy Testbots and use drupalCI as our testing infrastructure
, I think we can call this one fixed! WOOHOO! Thanks, DA!!! :D
or
to post comments
Comment
#94
11 September 2015 at 14:44
Status:
Fixed
» Closed (fixed)
Automatically closed - issue fixed for 2 weeks with no activity.
or
to post comments
Contribution record
Parent issue
Child issues
#2509222: Create a page for each test
#2517958: Improve consistency of Postgres environments
#2517980: [DrupalCI Integration] Enable patch testing
#2518008: Improve the XML output/parsing of the test runner
Add child issue
clone issue
Related issues
Referenced by
#2373451: DrupalCI: Sitewide job configuration
#2477413: Increase minimum version requirement for Postgres to 9.1.2
#2508231: Raise minimum required version of PHP to 5.5.9
Infrastructure management for Drupal.org provided by
Need a Drupal 7 extended support partner? Consider Tag1.
News items
News
Planet Drupal
Social media
Sign up for Drupal news
Security advisories
Jobs
Our community
Community
Services
Training
Hosting
Contributor guide
Groups & meetups
DrupalCon
Code of conduct
Documentation
Documentation
Drupal Guide
Drupal User Guide
Developer docs
API.Drupal.org
Drupal code base
Download & Extend
Drupal core
Modules
Themes
Distributions
Governance of community
About
Web accessibility
Drupal Association
About Drupal.org
Drupal is a
registered trademark
of
Dries Buytaert