The Drupal.org Complexity | Drupal.org

The Drupal.org Complexity | Drupal.org
Skip to search
Can we use first and third party cookies and web beacons to
understand our audience, and to tailor promotions you see
The Drupal.org Complexity
By
joshuami
on
19 May 2016
At DrupalCon New Orleans, during both
Dries's keynote
and at the
State of Drupal Core Conversation
, question of whether/when to move to Github came up again.
Interestingly, I was already deep into researching ways we could reduce the cost of our collaboration tools while bringing in new contributors. This post is meant to serve as a little history of how we got to where we are and to provide information about how we might choose to go forward.
It's complex
To say Drupal.org is complex is an understatement. There are few systems with more integration points than
Drupal.org and its related sites and services
The ecosystem is complex with lots of services that share integrations like login (Bakery single sign on) and cross-site code/themes.
It all starts with the code
The slogan "come for the code stay for the community" is accurate. The community would not exist without the unifying effort of collaborating to create the code. The growth of the community is primarily because of the utility the code provides and the (relative) ease of creating a wide range of websites and applications using Drupal core combined with contributed modules and themes that allow that framework to be extended.
Drupal.org was an extension of the development of Drupal for a very long time. Up until Drupal 6, Drupal.org was always upgraded the day of the release of a new version. When Drupal was smaller with a more limited scope, this made a lot of sense. With the release of Drupal 6 and the surge in usage of Drupal, more and more contributors started working on the Drupal.org infrastructure and creating new sites and services to speed the collaborative work of the community.
One of the biggest transitions during the Drupal 6 lifecycle and community surge was The Great Git Migration. Much of the complexity of Drupal.org and the related sites and services was created during this time period. Understanding that timeline will help in understanding just how much work went into Drupal.org and Drupal at that time.
The
Great Git Migration
In the
Great Git Migration
, all of the history of Drupal code was migrated to Git from CVS. The timeline for migrating to git was about what you would expect. Community conversation took time, getting volunteers to start the process took time, finally, there was a phase were dedicated (paid) resources were contracted to finish the work.
May 2009
- Google Summer of Code project proposed to
create tools to make the migration of the version control system possible
on Drupal.org.
February 2010
Discussion to select the next version control system
was kicked off; at the time, it was a choice between Git, Baazar and Mercurial.
July 2010
The Great Git Migration project
is created on Drupal.org to manage the related issues and planning.
February 2011
First commit to the Git repositories
by the migration bot.
July 2011
- The Great Git Migration project and discussions around version control selection are last updated. This represents the semi-official end of the migration project.
August 2011
- The
"let's move to Github" wiki/discussion
is kicked off.
Our repos are vast
We have over 35,000 projects that total over 50 GB on disk.
All of the Git repos on Drupal.org are associated with Projects.(e.g. modules, themes, distributions, etc.)
We have issues
At the end of 2015, there were nearly 900,000 issues on Drupal.org. Drupal core alone has over 74,000 issues—over 14,000 of those issues are open. Open issues is not an indicator of code quality, but it is an indicator of how many people have contributed to a project. In general, the more issues a project has, the more challenging it is for maintainers to continuously triage those bug reports, feature requests, plans, tasks and support requests.
The issue queues are part project management, part bug tracking. As such, they are organic and messy and have lots of rules that have been documented over years of community development. We have 23 pages of documentation dedicated to explaining how to use the issue queues. There is additional documentation dedicated to how to properly fill out an issue for core, for Drupal.org, and for numerous other contributed projects.
Issues are integrated into collaboration on Drupal.org
Those issues belong to projects and are connected to the Git repos through hooks that show a system comment when an issue is related by node ID (issue number) to a commit in Git.
Issues can have patches uploaded to them that are the primary means of suggesting a change to code hosted on Drupal.org. The patch-based workflow has extensive documentation, but it is not a simple task for a novice user to jump in and start contributing.
Most Git hosting solutions (Github, Gitlab, Bitbucket, etc.) either have some version of an issue or at least integrate with an issue tracking system (Jira, Pivotal Tracker, etc.) and provide pull request functionality (a.k.a. merge requests).
Having the same name is where the similarities and consistencies stop. Issues on Drupal.org have status, priority, category, component, tagging and more that are unique to Drupal project workflow. It would be a significant exercise to remap all of those categorizations to a new system.
Packaging
If the projects are what you can browse and find, and the issues are how you collaborate and change the code, the next most important service for Drupal is likely the packaging system.
Packaging is based on project maintainers creating a
release
of the code by associating a branch of the Git repository with the release. Every 5 minutes, our automation infrastructure checks for new releases and will package those releases into a downloadable file to represent the project.
Few developers actually access this directly from the project page anymore. They are much more likely to use Git, Drush, Console or Composer to automate that part of the workflow. Drush, and to some extent Composer, both use the packaged files when a command is issued. Also, the Drupal feature of just putting the code in the correct directory and it will run—with no compiling—is fundamental to the history of Drupal site building.
Updates
Another crucial Drupal service, updates is built into how Drupal core checks on itself to see if it is up to date.
The 1.3 million plus websites that call home to updates.drupal.org get back XML that is then parsed by that installation's update status module; that updates module has different names depending on the version of Drupal. Each month, about 12 terabytes of traffic to our CDN is requests for updates XML. Considering this is a bunch of text files, this is an amazing number to consider. Some sites call home once a week, some once a day, and some do it every few minutes. (Really people! Be nice to your free updates service. Telling your server to ask for updates daily is plenty frequent enough.)
Tallying the unique site keys that request this information is how we get our
usage statistics
. While this is probably not the most precise way to measure our usage, it is directionally accurate. There are plenty of development sites in those stats and plenty of production websites that don't call home. It should roughly balance out. To be anymore precise, users of Drupal would have to give up some privacy. We've balanced accuracy with privacy.
Because of our awesome CDN (thanks, Fastly!), we are able to deliver up to date packages and updates information in milliseconds after we update the underlying data.
Composer
On May 3rd, we launched the
alpha version of our Composer endpoints
on Drupal.org. If you don't know about
Composer
, you should
read up on it
. Composer is package management for PHP. (It's similar to what NPM does for Node.js or RubyGems does for Ruby.)
Core developers have been using Composer for some time as a means to manage the dependencies of PHP libraries that are now included in core.
The
Composer endpoints
allow any Drupal site developer to use
composer install
to build out their websites.
The new Composer service will also allow contrib project maintainers to using composer.json files to define the requirements for their modules and themes. The service even translates project release versions into semantic versioning. Semantic versioning was the biggest reason we could not "just" use
Packagist.org
like other projects in the PHP community.
This is all a huge benefit, but more importantly, we now have deep integration between a best practice approach to PHP dependency management and the Drupal.org code repos that can scale to our community needs.
Testing with DrupalCI
Speaking of needs, DrupalCI ran 67,000 test runs in January 2016. Each test run for Drupal core has 18,511 tests per run. That means over 100,000 assertions (steps) in the unit and functional tests that make sure Drupal's code is stable and that an accepted patch does not create a regression.
At the time of this post, we are using Amazon Web Services cc2.8xlarge EC2 spot instances for our testbots. These bots are powerful. They have 2 processors with 8 hardware cores. AWS claims they can provide 88 EC2 compute units per instance. They are packed with processing power because we have a lot of tests to run. While there are bigger instances, the combination of price and power allows us to keep Drupal core complete test runs right around 30 minutes. We autoscale up to 20 of these instances depending on demand, which keeps queue times low and allows maintainers to get quick feedback about whether a patch works or not.
I truly believe that getting DrupalCI up and stable is what allowed Drupal 8 to get to a full release last fall. Without it, we would have continued to struggle with test times that were well over an hour and a system that required surplus testbots to be manually spun up when a big sprint was happening. That was costly and a huge time waste.
If anyone asks me "what's the most important thing your team did in 2015", I can unequivocally say "unblocking core development to get Drupal 8 released."
Issue credits
The second most important service we built in 2015—but certainly the more visible—is a system for crediting users and organizations that contribute on Drupal.org.
Issue credits sprang forth from an
idea that Dries proposed
around DrupalCon Austin in June of 2014. At the time, his intent was a means of structuring commit messages or using commit notes to provide the credit. Eventually, we shifted the implementation to focus on participation in issues rather than code commits. This made it possible to credit collaboration that did not result in a code change.
I won't get into the specifics; I wrote a
A guide to issue credits and the Drupal.org marketplace
earlier this year. Issue credits have been extremely successful.
As there name implies, we store the data about credits as a relationship to closed issues. Issue credits touch issues, projects, users, organizations and the marketplace on Drupal.org.
Why not just migrate all of this complexity to Github?
Why can't we just move all this to Github?
— said
lots of people
, often
To be fair, this is a challenging discussion. Angie Byron (
webchick
) wrote an amazingly concise
summary of the Github issue
on Groups.drupal.org.
That wiki/discussion/bikeshed was heated. The conversation lasted over two years. I started as CTO about 6 months into the conversation. Along with a couple of other themes, the Github move has been a constant background conversation that has defined much of my time leading the Drupal.org team.
How are these services connected?
To truly understand the problem of a migration of this scale, we have to look at how all of the major Drupal.org services are connected.
Each block in this diagram is a service. Each line is a point of integration between the services. Some of these
services
are on Drupal.org or subsites with thousands of lines of custom code defining the interactions. Other
services
are not built in Drupal and represent projects in Java (Jenkins) or Python (our Git daemon) with varying degrees of customization and configuration.
As the diagram suggests, it is truly a
web
of integrations. Pull one or more services out of this ecosystem and you have to either refactor a ton of code or remove a critical component of how the community collaborates and how our users build sites with Drupal.
It's kinda like a great big game of
Jenga
What would a migration to Github require?
Please believe me when I say that if it were "easy" or "simple", we would have made either moved to Github or at least
upgraded our Git collaboration with nifty new tools
on our own infrastructure.
However, disrupting the development of Drupal 8 would have been devastating to the project. We were correct to collectively backlog this project.
So if we were to try this migration now, what would it take? First, you have to consider the services that Github would effectively replace.
Github replaces:
Git repositories
Issues
Patches (they would become pull requests)
Git viewing (and we'd get inline editing for quick fix and onboarding)
That's four (4!) services that we would not have to maintain anymore. Awesome! Cost savings everywhere! Buy a boat!
Wait a second. You have 16 integration points that you need to refactor. Some of them would come with the new system. Issues, pull requests, repos and the viewer would all just work with huge improvements. That leaves us with 12 integration points that would require a ton of discovery and refactoring.
Users
- we have 100,000 Drupal.org users that are pretty engaged. (We have over 1 million user accounts—but that number is likely a little inflated by spam accounts.) Do we make them all get Github accounts? Do we integrate Github login to Drupal.org? Do we just link the accounts like Symfony does?
Projects
- Github is not a project browsing experience. Drupal.org is a canonical repository where the "one true project" lives for packaging and updates. At the very least, we have to integrate our projects with Github. Does that mean we have to keep a Git repo associated to the project that has hooks to pull in changes from Github?
Testing
- One of the less complex integration refactors would be getting DrupalCI integrated with pull requests. That effort would still be a months long project.
And DrupalCI would be its own effort to migrate to another testing service because it is tailored to the issue queue workflow and tightly integrated with projects.
Those are just a few of the major integration points.
I have a personal goal to detail every single integration and get that documented somewhere on Drupal.org. I don't think that level of documentation will increase the ability for others to contribute to the Drupal.org infrastructure—though that would be a pleasant side effect. I do think it is necessary for us to continue to support and maintain our systems and ensure that all of the tribal knowledge from the Drupal.org team can be passed on.
What would it cost?
I have joked that it would take roughly 1 million dollars (USD) to complete a Github migration. (
Cue Dr. Evil.
) That is only partially meant in jest.
As anyone who has estimated a large project knows, there is a point of uncertainty that leads project owners to guess at what they are
willing
to pay for the project.
If we take the four biggest lifts in the Drupal project's history, what do we get?
Drupal.org redesign
- There were tens of people involved in the project, hundreds giving feedback. The timeline was about a year from start to implementation.
The Great Git Migration
- There were tens of people involved in the project. Far fewer users gave feedback, but the project took about two years from brainstorming to initial commit to the Git repos—with a few months of clean up after.
Drupal.org upgrades to Drupal 7
- The project took about two years with tens of people involved in about 8 months of clean up issues.
Drupal 8
- 5 years of development by over 3,000 contributors.
I don't think than anyone would argue that each of these projects would have been bid at well over $1 million. I would put a migration to Github at somewhere between the complexity of The Great Git Migration and Drupal 8.
In none of these cases did the Drupal Association actually spend $1 million USD in project dollars. However, in all of the projects, there was lengthy discussion followed by substantial volunteer contribution, and then a significant bit of paid work to finish the job. It's a pattern that makes sense and will likely repeat itself over and over.
Would it be worth it?
I'm going to go back to the
summary on the Github discussion
. There are reasons why both options seem to be the best possible option or the worst possible option.
Would a best practice workflow and toolset be worth the change? Absolutely. Github (or Gitlab) tools are easier for newcomers to learn. Further, because we are using PHP and Javascript libraries that are hosted on Github, we could get contributions from developers and designers that are involved in those projects and do not wish to have account on Drupal.org.
The drawbacks are considerable. We cannot afford a full migration right now. Dries put it well at DrupalCon Los Angeles during core conversations. The Drupal Association is not a bag of money. With significant growth of revenue, there is a long term possibility of more paid developer resources, but not in the short term. It is too much to ask volunteers to give up a year of their life to run the project as a community initiative. That leads to burn out and frustration.
We should also consider whether the disruption to the current collaboration workflow will be worth it. I don't think so. Not if that disruption meant stalling the update of contrib projects that are critical to solidifying Drupal 8 adoption. (Though I could argue that much of this upgrade to Drupal 8 work is being performed on Github as some—perhaps many—developers prefer those tools.)
Is there a middle ground?
Drupal spends a lot of time getting to the middle ground. Many of the best innovations in Drupal come from getting to the middle ground—from reaching a general consensus and then allowing someone who has support and time iron out the details.
So for the first step, we
should
add functionality to projects on Drupal.org that allow maintainers to shift their workflow to Github while still publishing their project on Drupal.org. This allows the canonical browsing of projects, the continued support of the security team, and most importantly the continued distribution of Drupal through Composer, release packaging and the updates system.
We have a solid way forward for these integrations as the requirements are narrow enough in scope to accomplish in a 4-6 month timeframe using dedicated resources. We would still need to figure out how to award an issue credit to someone that participated in an issue on Github. We might be able to institute commit credits that could be parsed into issue credits from the participation on Github, but it would not be as inclusive as the current model.
It would be important to phase in this new feature rather than make a wholesale change. Once that integration is in place, we could extend DrupalCI to test pull requests similar to how we currently test patches submitted to an issue.
Stay flexible
We need to be flexible. GitHub has a lot of potential as tool for open source distribution and collaboration—likely for the foreseeable future. However, not every major project is on GitHub. The Linux Kernel uses Git repositories with cli tools and a patch-based workflow that relies heavily on email. It works for them. Wordpress is still on Subversion—even though they've started to accept some pull requests on GitHub. These projects are poised to make the right decision rather than a rash decision.
The sky will not fall if we keep our current model, but we are losing opportunities to grow as a community of contributors. Rather than a wholesale migration, we must understand the value and history of this web of integration points. Targeting our efforts on specific integrations points can achieve our goal of opening our doors to the developers who live and breathe GitHub, without losing the character of our collaboration. And in the long run, this focus on services and integrations can make us more adaptable to the next change in the broader development landscape.
Special thanks to
Tim Lehnen
Ryan Aslett
Rudy Grigar
and
Neil Drumm
for helping me pull this data together and to the rest of the Drupal.org team for reviewing the post for typos.
Republished from joshuami.com
or
to post comments
Comments
This is really, really good.
catch
he/him
commented
19 May 2016 at 14:39
This is really, really good.
I have one nitpick about issues, but it's an important one for the conclusion I think:
Having the same name is where the similarities and consistencies stop. Issues on Drupal.org have status, priority, category, component, tagging and more that are unique to Drupal project workflow. It would be a significant exercise to remap all of those categorizations to a new system.
This is very important, and is one of my own main worries about trying to migrate issue anywhere:
But then:
Wait a second. You have 16 integration points that you need to refactor. Some of them would come with the new system. Issues, pull requests, repos and the viewer would all just work with huge improvements. That leaves us with 12 integration points that would require a ton of discovery and refactoring.
Pull requests, repos and the viewer should all just work, but this doesn't really count for issues.
or
to post comments
Good point!
joshuami
he/him
Portland, OR
commented
19 May 2016 at 15:22
Pull requests, repos and the viewer should all just work, but this doesn't really count for issues.
Good point! And I agree. Issues would "work", but they would work so differently that they would represent a learning curve for long time contributors. We would have to rethink process and workflow. That is really easy to bikeshed, so that process would likely take time and create friction for those setting the new direction.
or
to post comments
Great summary of something
kreynen
commented
19 May 2016 at 17:36
Great summary of something that is incredibly complicated!
Now my nitpick...
Semantic versioning was the biggest reason we could not "just" use Packagist.org like other projects in the PHP community.
Shouldn't the lack of licensing and security reviews in the code Packagist links to also have been big reasons not to "just" use Composer as well?
Can someone explain why we've been so quick to shift from being overly strict about what we've allowed to be distributed from Drupal.org to outsourcing package management? I really don't understand why some of the same enterprise organizations that have vocal about keeping everything coming from Drupal.org licensed as GPLv2 now are happy to move to a solution that allows them to end up with code on their servers that is licensed as Apache2 and AGPLv3.
or
to post comments
@kreyen, good point. We built
joshuami
he/him
Portland, OR
commented
19 May 2016 at 18:07
@kreyen, good point. We built our own
Composer endpoints
as a way to better control what we distribute. We have not outsourced package management to
Packagist.org
The biggest
technological hurdle
to using Packagist.org was semantic versioning and the difference between "modules" and "packages", but I wholeheartedly agree there are licensing issues to work out there as well.
The good news is that Composer support should make it easier for a developer using Drupal to review the resulting licensing of their installations. Composer has licensing information built into its structure if the developer chooses to use it. Technically, our Composer endpoints are
only
providing code that is licensed GPL version 2 or later. There is the larger question of what to do when a project has a dependency in its composer.json requiring a package/library/code not hosted on Drupal.org.
We are looking at legal review for to tackle this question for usage of our Composer endpoints. Open source on Github is heavily impacted by the issue of license compatibility. Developers building the packages may not care, but developers using the packages for an enterprise customer likely have to care deeply. That's a terrible mismatch in expectation.
On Drupal.org, the answer might be for us to display a warning when a licensing mismatch is detected in composer.json. How we chose to handle licensing based on legal feedback will likely impact the approach other projects take.
It is also worth noting that this is another layer of complexity that I did not get into for this post, but definitely factors into the larger discussion about what we *should* or *should not* do with our collaboration tools.
or
to post comments
Great article, especially the
mglaman
WI, USA
commented
19 May 2016 at 21:45
Great article, especially the follow up to workflow tools and the postponement of #2488266
or
to post comments
Nice write and great approach - also credit via git commits!
levelos
commented
26 May 2016 at 15:14
Great write up Josh, that's a lot of information to synthesize and convey! ThinkShout whole heartedly supports the move to integrate GitHub as middle ground, it would really simplify our internal workflows for contrib development.
Related, we've long pined to get contribution credit via commits rather than issues. I'm sure we're not alone in this, but most of our contrib work does not involve d.o. issues. We're working off internal roadmaps and client requirements and do not get recognized for our contributions.
Lev Tsypin
CEO and Co-Founder at ThinkShout
or
to post comments
Issue related to that here:
catch
he/him
commented
26 May 2016 at 15:53
Issue related to that here:
or
to post comments
Great article and overview.
fabianx
commented
26 May 2016 at 17:27
Great article and overview.
I am very much looking forward to the planned git hooks integration.
While seemingly not as important anymore, let me play Devils OSS advocate and say this:
"Proprietary code that is under control of just a single company can be problematic."
It is one of the reasons OSS was created. It is one of the reasons we have (some) free and open source operating systems, too.
If GH closes business tomorrow, what do we do then?
Yes, we can hopefully still find a copy on some developers machine of the whole GIT tree, but what about the issues, the workflows, the triggers, the code review comments, etc.?
Being dependent on one company led to the development of GIT. Ironically now most OSS developers use a subversion like workflow with one central server that is the master repository and under proprietary control.
I love GitHub personally and don't have anything against proprietary code - don't get me wrong.
However lets not forget the roots of open source, please. And drupal.org is mostly free of proprietary services that _hold_ data right now. It would be nice to keep it that way.
Yes one could argue that the Drupal Association is holding data, too, but that is not a proprietary service, but part of a OSS network IMHO.
Another thing is that collaboration on a patch still is quite difficult on GH and the issue workflow still works better when 10s of people work on a patch together.
I still hope once the first integration features are done that then the idea of just pushing a branch to a private namespace to create a DIFF easily will be on the roadmap again.
or
to post comments
I _really_ agree with you on
fuzzy76
commented
27 May 2016 at 07:22
I _really_ agree with you on not keeping vital data on a proprietary service. In the proposal, d.o would still have git repositories but they will be synced periodically (only when doing releases perhaps?). That will keep the code out in the open. The rest is perhaps a reason to keep the issue queue on d.o, atleast for the time being.
Another thing is that collaboration on a patch still is quite difficult on GH and the issue workflow still works better when 10s of people work on a patch together.
Yes, but an unscientifically gut-feeling tells me that 99% of issues on d.o are closed based on a single contributor. Let's not optimize for 1%-cases, but for the vast majority. :)
or
to post comments
GitHub GitHub GitHub
leducdubleuet
commented
27 May 2016 at 10:44
This is all very interesting, thank you very much! But the idea of moving to GitHub seems horrible to me, why not host our own GitLab? We already use Mailchimp for newsletters, while at it, why not use Disqus for the comments? :-)
or
to post comments
Thanks for the detailed
larruda
commented
1 June 2016 at 19:58
Thanks for the detailed explanation, Joshua. I was one of the guys that asked pretty much the "GitHub migration" question after your presentation. Most of the infos here you have briefly told me there, but it's nice to see all this in numbers, diagrams and arguments. It looks for me that, unfortunately (or not), the whole Drupal ecosystem has gone mostly in different ways than the rest of PHP community, and we have built so many things so far that, as you said and I agree, the costs and trouble to change part of it is tremendous complex, though the benefits are visible.
or
to post comments
Keep issues on d.o and link to GitLab/GitHub commits?
mlncn
commented
4 June 2016 at 16:27
Regarding the idea that "We would still need to figure out how to award an issue credit to someone that participated in an issue on Github." — have you considered integrating with GitLab or GitHub in such a way that all issues, at least where credit is desired,
must
be on Drupal.org?
Then the integration from a technical standpoint is accepting links to commits or pull requests on GitLab and/or GitHub in lieu of patches, but treating them like patches (even formatting them that way, for consistency for reviewers and ease for the testbot?
This is something of the opposite of
#2676042: Automatically assign issue credit for committing patches
(which would benefit myself and Agaric quite a lot), which i'm not opposed to but there's something nice about the equal weighting and process of this new approach to credited contributions: simply credit in an issue.
benjamin,
Agaric
or
to post comments
GitLab is open to helping Drupal
sytse
commented
15 June 2016 at 17:12
Thanks for the informative article.
I just want to add that if you want to run your own GitLab installation we would be very open to help with that.
If there is additional functionality needed in GitLab we're willing to build that or get it merged quickly.
Feel free to reach out to us anytime.
or
to post comments
wow :D
larruda
commented
15 June 2016 at 21:12
wow :D
or
to post comments
We use Gitlab a lot, and have
liam mcdermott
commented
21 June 2016 at 15:56
We use Gitlab a lot, and have done for several years, it's great software. I'm so happy there's a Free software alternative to Github, it's incredible that so many choose to sacrifice freedom for a little convenience.
I expect Gitlab would be difficult to fit it into Drupal.org's way of working though, for example: many people get involved in Drupal module development from the user community, they start working in the issue queues and fall into coding from there; this will happen a lot less if we erect barriers to entry, which includes a visual difference between the
user's area
and the
developer's area
. Therefore anything Drupal integrates with would should be themeable to look and behave like just another part of Drupal.org.
On a side-note: back in the day, Drupal found the reverse of this out from Drupal 6 usability testing. Because the default admin theme was the same as the theme for user-facing parts of the site, users struggled to comprehend the difference; on Drupal.org this blurring of the lines between parts of the site for users and developers is beneficial. Developers on Drupal.org do not alienate users, and that's a
GoodThing™
The modules insisting people use Github to contribute are erecting a barrier between developers and users who could have been developers in future. This is killing the Drupal community's future in the chase for developers who use Github, who may or may not choose Drupal.
or
to post comments
Do you actually believe that
fuzzy76
commented
24 June 2016 at 07:11
Do you actually believe that having a different interface in the code contributor pages will shun away a lot of users? Do you actually think there are any web application users that haven't at one point in time interacted with the issue tracker on Github to report a bug?
And even if there are users that are turned away, how do you think that compare to the number of skillful developers that are turned away by a patch-based workflow that the rest of the world left years ago? With the use of Symfony components we are more available to "outside" experienced developers than ever, and those are the ones we are scaring away.
or
to post comments
Self-hosted and open source
fuzzy76
commented
16 June 2016 at 11:55
Self-hosted and open source and modern tools in a single package. :)
or
to post comments
lol what
Anonymous (not verified)
commented
23 June 2016 at 15:50
complexity.. you say about it like it is advantage or something. Just LOL
With migration to github you will get TON, i say, TON of professionals who may just put commit or two just because.. just because they were bored and did a good thing.
Why? Because they already are on github. And maybe they just did commit via browser.
Now with your mega cool infrastructure :
- i must create account here
- i must do some fancy patch file
- i must fill some other crap
really? no way ppl, no way.
also, who cares about your satellite websites? want store them privately - who cares! store privately.
Make hooks to github and baby sit your 'infrastructure' in sync.
p.s.
i work with drupal for ~2 months, and i struggle my self to make a single patch, as i this patch policy just wastes my time
or
to post comments
Unfortunately I have to agree
larruda
commented
23 June 2016 at 16:07
Unfortunately I have to agree with you man. Just yesterday I was teaching about 10 colleagues on how to generate a patch, what is an interdiff and how to do that, and issue queue statuses and workflow, and you know it was so much information and people really got confused. It's too way complex and we are adding more complexity over time just because there's so much done till now that "it's too much work to change".
I'm afraid of how this is going to end. We're always going to the the island. Use packagist? No, create own own. Use TravicCI? No, DrupalCI is better...
or
to post comments
Better to be free on an
liam mcdermott
commented
23 June 2016 at 17:58
Better to be free on an island than force everyone to use proprietary software. If creating patches is too hard that's an argument for making Drupal's process better, not necessarily for adopting GitHub.
The stuff about these masses of developers who will magically appear to contribute drive-by pull requests (using their browsers) is also bogus:
1. The Drupal contribution process is based around a community of developers, testers, documenters, and users communally working on each change and not drive-by pull requests.
2. Is there any evidence of this happening on a large scale (with Symfony for example), how do you know the developers on GitHub are even interested in Drupal? My guess is most decent developers don't want to use PHP anyway.
3. Drupal is a complex, mature code base and what problems it has are usually deep, difficult to solve and will require changes to more than one source file; this does not fit the quick-fix in a browser use-case.
I'm thinking GitLab might be an okay compromise, but the problem with that is shutting out non-developers from the contribution process. I'm stunned that no-one seems to understand or care about this.
And we don't say 'It's complicated' like it's a good thing, that's the workflow required to support Drupal development. Unfortunately GitHub's ultra-simple issue tracker isn't capable of supporting that, again though: if the UX of Drupal's issue tracker is bad that's an argument to improve it, not necessarily switch everything to GitHub (and force proprietary software on everyone).
or
to post comments
Better to be free on an
fuzzy76
commented
24 June 2016 at 07:05
Better to be free on an island than force everyone to use proprietary software
Pull-requests aren't proprietary. Gitlab CE is open source. So is Packagist.
Drupal is a complex, mature code base and what problems it has are usually deep, difficult to solve and will require changes to more than one source file
But most contrib modules aren't. They're small, isolated problem spaces in a handful of files.
I'm thinking GitLab might be an okay compromise, but the problem with that is shutting out non-developers from the contribution process
How? Why is it harder for a non-developer to participate on a module in Gitlab than on Drupal's infrastructure? On the contrary, Gitlab uses conventions from every single other code collaboration site out there. In contrast to Drupal.org which invents its very own special workflows and conventions, making it much harder for both developers and non-developers to participate. I am stunned that you don't seem to understand or care about that.
In the end, the largest problem here is probably using a CMS hacked to emulate a project / code management tool. It's the PHP hammer all over again.
Unfortunately GitHub's ultra-simple issue tracker isn't capable of supporting that
You can use labels to emulate most of the functionality D.O's issue tracker has. On Gitlab, you have even more functionality for project management. And on gitlab.com (and the EE version) you even get task boards.
if the UX of Drupal's issue tracker is bad that's an argument to improve it
But there is not enough resources to actually accomplish anything, neither funding from DA nor voluntary manpower. Improvements are not happening for the foreseeable future.
or
to post comments
Except for core, most module
fabianx
commented
23 June 2016 at 17:50
Except for core, most module maintainers would also accept a PULL request, which can be easily downloaded as a patch - if you really struggle with it.
- Clone the project
- Push to (your own) GitHub
- PR away
Hint: whenever you have a PR you can just do /pull/42 => /pull/42.diff
and you get your diff file.
Having worked both with GitHub and Drupal Core / Contrib, it is pretty evident that a patch based workflow slows you down a little.
However that is usually a good thing as most patches are very well tested and formatted.
In the end however for any production work (using composer or just plain) you _will_ need to manage patches yourself.
If you only commit e.g. a patch to ctools for your own codebase, you'll soon land in upgrade hell ...
So you make a patch and put it in sites/all/patches/ctools_[issue-id].diff
Or in your composer patches section or ...
So ultimatively if you want to _use_ your work right now (and in most business scenarios that is what happens) and you don't want to wait for the maintainer to commit your patch, which can take years - GH or not - you need to maintain patches or repositories that you upgrade.
Once however you are at that level, GH is giving you almost nothing anymore.
However Drupal gives you one thing that does not happen in other OSS projects as much (and Symfonfy just created a bot for it to workaround GH restrictions):
Collaboration of people!
And this collaboration of real people is so much more useful than any technology, where you write a quick PR and ...
There are ways to make our tools better and they are embraced.
However GH is not the solution for collaboration of many people on issues - even though I very much like it.
And ultimatively everyone can manage their project on GH or d.org or ...
That is the beauty of OSS and the open web.
or
to post comments
Composer and Packagist link will help quick access
rajab natshah
he/him
commented
10 July 2016 at 07:20
Thanks for the info.
I hop that we add a
Packagist
link for each packaged project, in the drupal project page.
It could be in a new
tab
called
composer
like the
Version control
tab then in that tab we could see all the required or packaged versions for the composer.
packagist release
management info, like
Administer releases
At this point every time we do have to switch back and forth between
drupal.org
and the current
packagist.drupal-composer.org
then search for the project, and have a look at the available versions for the composer.
Thank you :)
or
to post comments
It seems to me that a Drupal
jdleonard
he/him
Austin, Texas
commented
9 October 2016 at 22:58
It seems to me that a Drupal.org-hosted install of GitLab CE or GitLab EE (GitLab has offered to provide a free GitLab EE license for the Drupal project) with some glue (e.g. commit hooks, LDAP configuration, contributions to GitLab CE) to integrate it with Drupal.org's issues queues, DrupalCI, and Drupal.org single-sign-on affords many benefits of GitHub and Drupal.org without many of the pitfalls.
I think it would be helpful if someone with knowledge of the integration that would be needed to enable and/or improve upon Drupal.org's existing workflows could explain what would need to be done. Without this insight, it's difficult to take next steps, including identifying the parties who would be responsible for implementing and funding such an integration.
Could someone who knows who has such knowledge please direct such people here or wherever else is appropriate?
JD Leonard
or
to post comments
pls implement pulls faster
Anonymous (not verified)
commented
11 October 2016 at 20:42
pls implement pulls faster and move to project management platform instead of frankenstein forum-git-place_to_talk-patch_fest
If you don't do it in near future - your project will be dead. You stuck in 2000s
D8 looks promising only because of Symfony core. This holds this project interesting to me, but if you continue with patching crap - you can close shop rather sooner than later
or
to post comments
TOTALLY AGREE
larruda
commented
11 October 2016 at 20:50
At the company I work for I'm teaching DOZENS of developers on how to contribute to Drupal and modules and it's really HARD to explain and learn such workflow. Feedbacks are that it's too way complicated. Not to mention when I'm asked "why it's not on github?" u.u
or
to post comments
What part is complicated?
fabianx
commented
12 October 2016 at 13:04
What part is complicated?
Is it the workflow itself?
Is it the making of a patch?
--
While I agree it would be really sweet to just push a branch and have a nice UI (TM) to create a patch out of it, the patch based approach (for non-typo related things) has several advantages compared to the usual GitHub workflow:
* History is clean and not lost
-> Especially in GitHub it gets frequently impossible to figure out what the end result of a discussion was (I failed twice both in Symfony and in VideoJS to make any sense).
-> Also in Drupal we often have lots of different technical approaches competing; it happened more than once that I did go back to find in the history a better approach to a problem that I am having, while another patch was committed. With GH force-pushes or reverts, that is impossible as history is not clean.
* Interdiffs are logical changesets instead of per-commit (too granular) or per patch (too large); they usually allow continuous reviews - which is not working nicely with GitHub - unless you commit each change to a new branch and have a PR for each change, but then you need a different branch for each technical apporach, whioch gets very confusing fast.
--
Instead of all the above I thought about:
# git push-issue-patch [issue-no] [base-branch like core/8.3.x] [previous-commit else automatically taken from last push]
which automatically would create and upload interdiff and patch file (but would let you review first and add a message, too).
Would that help?
or
to post comments
a bit of sense
Anonymous (not verified)
commented
12 October 2016 at 14:08
What part is complicated?
patch history
patch versioning
creating patch
huge amount of useless patch files.. it's just swamp
no tree comments
300+ comments and 5 years non fixes error is acceptable
I can continue. Also this process with modules.. 2 years lag between dev and release versions.. that's ridiculous
""
Especially in GitHub it gets frequently impossible to figure out what the end result of a discussion was (I failed twice both in Symfony and in VideoJS to make any sense).
""
You failed twice.. can you count how many times you have succeeded? With this patching stuff only crazy ppl continue to contribute, or those that are very close to project. This forum stuff is unfriendly.
regards
or
to post comments
It is insisting on a
fuzzy76
commented
13 October 2016 at 08:46
It is insisting on a different workflow from the rest of the universe. We are not that special, and the number of potential contributors we scare away are not worth the price.
If I am a maintainer, I'd like to receive pull requests that I can review, comment on and even edit in place. If I am a contributor, I'd like to cooperate with others on a pull request instead of juggling 10 different patches to achieve some kind of consensus. These are much larger productivity gains than interdiffs.
or
to post comments
Looks like we're stuck in
larruda
commented
12 October 2016 at 14:58
Looks like we're stuck in this system while practically all other softwares are moving ahead just because of a few "dinosaurs" (with all respect) from the project that can't accept changes and evolution. I already see people NOT involving because YES, it's too way complicated to deal with patches. Once you get used to it, you get along. But the problem is in this thick line between accepting that you want to learn and understand and those who give up.
or
to post comments
Not "stuck", but slow to change
kreynen
commented
12 October 2016 at 17:33
I don't think we're "stuck", but changes in a project and community of Drupal's size takes time. I'm hoping Composer driven Distributions gives us enough reasons and momentum to maintain an index of distributions that link to external repos (like WordPress, Backdrop, JQuery, CKEditor, etc) while still hosting packaged downloads of the code, tracking usage, and providing a unified system for issue tracking. See
for where distributions might be headed. The use case for this approach is much easier to make for distributions, but if it works at that level it will also be easier to make the case for a hybrid options for the rest of contrib.
or
to post comments
Drupal.org blog
News and updates about Drupal.org website. Posted by the
Drupal Association
technology team.
Infrastructure management for Drupal.org provided by
Need a Drupal 7 extended support partner? Consider Tag1.
News items
News
Planet Drupal
Social media
Sign up for Drupal news
Security advisories
Jobs
Our community
Community
Services
Training
Hosting
Contributor guide
Groups & meetups
DrupalCon
Code of conduct
Documentation
Documentation
Drupal Guide
Drupal User Guide
Developer docs
API.Drupal.org
Drupal code base
Download & Extend
Drupal core
Modules
Themes
Distributions
Governance of community
About
Web accessibility
Drupal Association
About Drupal.org
Drupal is a
registered trademark
of
Dries Buytaert