Hosted Service Outages May 21 & 22

On May 21st the hosted service began experiencing connectivity issues between parts of their application hosting tier, which ultimately was caused by a mix of timeouts and TLS handshake failures when our web2 frontend requested data from our API. This resulted in many requests seeing a 522 error from Cloudflare, an application error from the website, or a Slim error message while we were attempting to troubleshoot.

We’ve been in contact with (our host) since shortly after seeing the issue crop up. They confirmed that the region we’re currently hosting the application in was having some infrastructure performance issues shortly thereafter. This morning they reached out to us indicating that these issues have been resolved.

As part of this discussion, staff noticed that our account is currently provisioned in their older, less advanced US1 region. They’ve recommended that we migrate to the newer US2 region as soon as we can manage to do so. Our plan, now that the fires are out, is to perform that migration next week.


The unfortunate part of this site reliability issue was that it hit smack in the middle of at least one big event in the US. We don’t want this to happen again and are working to ensure upcoming events will have a smooth experience with our platform. The region move is part of this effort, but we’ve also implemented additional performance tweaks at the database level to ensure API responsiveness, as well as increased logging, monitoring, and alerting across the board to ensure that we’re (nearly) the first ones to notice another issue like this.


If you’d like to ask any questions or chat with the Joindin Leadership Team, feel free to join us in our Discord chat group: Thanks to the team, including our primary contact Larry Garfield, for advising us on measures to avoid another repeat of this issue.

How Events Get Approved

A lot of the work the Joindin Leadership team has been working on is understanding how the hosted site has been running so we can evaluate how to make things better. One of those first tasks was to document and ensure all of our event approvers are using the same criteria to approve events. To increase transparency we’d like to share our previously internally only guide for how events should be approved. Our guide isn’t perfect, but it’s the process that has been in place from some time now and before we go making broad changes we should be acutely aware of how things work today. The guide that all of our event approvers are using is outlined below:

  1. Has to be a community-focused event (so not just a for-profit event from an event planning corporation).
  2. Has to have an English description (can also have a second language, but must include an English description).
  3. Has to have speakers/talks (so not a bootcamp or online school course)
  4. Has to be getting listed for the purpose of collecting feedback for the speakers (can’t just be looking for advertising).

Number 4 is kind of subjective, but you kind of get a feel for events after going through a bunch. It’s usually pretty obvious, like JSDay versus an Annual Symposium on Some Scientific Thing (and then in the description, they talk about how all authors will be published in the Symposium book to be sold after the conference). The first would be accepted, and the second would be rejected.

Other than that, it’s up to the discretion of the person approving the events to determine if it’s really trying to collect feedback via our system or just use us or an advertising platform.

We’re going to be revisiting these guidelines over the next few months as the Joindin Leadership Team evaluates the direction we want to take the hosted service platform. If you ever have any questions about an event that was, or was not published on our site please feel free to reach out to info at joind dot in. Service Migration Maintenance Window

The Joindin Leadership team would like to announce this upcoming Tuesday, March 19, 2019 at 14:00:00 pm UTC we will be performing maintenance on the domains. While we anticipate a smooth transition we are planning for the worst. The goal of the maintenance window will be to switch our production environment currently hosted by the great team at Combell to The best case scenario will be a 2-3 hour window where the hosted services will be in a read only state. The worst case scenario would be all hosted services fail and DNS is reverted back to Combell as we triage what caused the issues.

Part of this move will also involve the decommissioning of This site will no longer be accessible after the migration is complete.

We will update this post once the maintenance has been completed to give the all clear. Thanks for your understanding!


The Joindin Foundation

Today the Leadership Team is excited to announce the formation of The Joindin Foundation. The Joindin Foundation is a nonprofit organization that has applied for recognition of exemption under Section 501(c)(3) of the Internal Revenue Code of The United States of America. We have the support of some really amazing people and I would like to introduce you to the founding board members: Adam Englander (Leadership Team), Beth Tucker Long (Longtime Administrator), and myself Joe Ferguson. The three of us will oversee the nonprofit organization which will operate the hosted service and oversee the open source project.

We would also like to announce that with the official organization comes an official address: PO Box 342457 Memphis, TN 38184. We would love for you to send us snail mail postcards and even stickers of your event!

You can also learn about what our leadership team has been working on by listening to the brand new episode of the Voices Of the Elephpant Podcast

February Update

The Leadership Team has been hard at work in recent weeks and we’re just getting our feet wet! A couple of highlights:

Eric Poe’s primary goal is to lower barriers-to-entry for devs to get started with working with helping out the project. This includes looking at how we build a dev environment and updating documentation.

We’ve also seen a flurry of activity in our Github repositories:

Ian Littman was on code review duty, providing feedback on the first phase of switching the API to use proper dependency injection, which allowed a bug fix to the spam check service to be more thoroughly tested, among other fixes/enhancements. He also pushed (and, with Eric Poe’s help, tested) the first release of the API to production since October 27th.

Jos Elstgeest has been diving deep into the code:

  • Implementing DI in the API, adding tests to both API and web2
  • Making preparations for refactoring API and web2 to psr-4.
  • Mostly detective work on the code and writing more tests.

We want to thank ALL of our contributors past and present for all their hard work. We’ve been going through all of the issues across the project and have been focusing on securing our production environment, merging low hanging fruit pull requests, and adding enhancements to our primary site including that pesky contact form bug. We also want to thank Beth Tucker Long and David Stockton for being awesome administrators of the live site. If you submitted an event or asked for help from the site these two have been AMAZING to the project. THANK YOU both very much! You might recognize David’s name from our new leadership team as he was one of the several who committed to keeping the open source project and hosted service going.

The rest of the leadership team has been hard at work responding to and updating pull requests, deploying updates, verifying functionality between the legacy repository and web2, building a security guidelines document, planning infrastructure updates, and even a couple of things we’re not quite ready to announce yet!

Thank you all to the people who have reached out to individuals on the leadership team to thank us for what we’re doing. Speaking for myself: it means a ton, thank you. We’ve got some exciting plans in the pipeline and looking forward to the future!

Leadership Meeting Feb 5th 2019

Below are the notes from the first leadership meeting.

Agenda & Notes

  • Welcome & Introductions
    • Who you are, why are you here
    • Primary Responsible Party
      • Joe Ferguson
      • Paul McNeely
    • Product Responsible Party
      • Andreas Heigl
      • Eric Poe
    • Development Responsible Party
      • Jos Elstgeest (zghosts)
      • Ian Littman
    • Security Responsible Party
      • Adam Englander
      • Mike Willbanks
    • General Responsible Party
      • Michael Cullum
      • David Stockton
  • Leadership structure and philosophy
    • No one is alone
    • Silence is acquiescence
    • Always teaching, always learning
  • Source of truth
    • Ditch Mailing lists (Marked Read Only)
      • Announce
      • Developers
      • Features
    • Blog:
    • Make Discord official and depreciate IRC?
      • Discord wins (For now)
  • Official Organization?
    • GoFundMe for $1,000 minimum goal.
      • ~$600 set up costs w/ ~120$ /year on going for basics
      • Balance of money raised would go to operating costs
      • Stretch Goal Ideas
        • Sending Conferences Stickers
        • New sticker designs? (Square), (no text, just wide logo)
    • More info from Joe forthcoming
  • Contributions
  • Infrastructure, Hosting and Deployments
    • Immediate: Jenkins, Server Security
      • Firewall rules
      • Disabling password auth
      • Jenkins has been set to require SSH proxy through the server for security
      • Joe, Michael and Ian currently have access. Others who need it, ask Joe.
    • Long term (March): Future hosting and deployments
      • Ian to develop a PoC
  • Next steps/priorities – Product & Development
    • Privacy policy – Andreas
    • GDPR – Andreas & Zawadi
    • Travis: Setup frisby tests
    • Outstanding PRs
    • Hosting / Deployments
    • Legacy functionality porting
      • What’s left
    • Automated testing
    • Disaster Recovery
      • Joe backed up configs to Keybase Storage
      • Need to back up and automate the process

New Leadership Team

Hello! Welcome to’s blog. You might have heard the hosted service was going to be discontinued by March 31st, 2019. When this news spread across the PHP community several members came together and submitted a proposal to the current and previous maintainers to build a new team of volunteers to take over the open source project and the hosted service. We have fantastic news to share with you: our proposal was accepted!

The leadership team was formed by volunteers responding to an open call for individuals to commit time and energy to taking over the large task of not only the open source project but the maintenance of the hosted service. The leadership team all signed and collectively presented the proposal to the maintainers. Meet the new Leadership Team:

Primary Responsible Party

  • Joe Ferguson
  • Paul McNeely

Product Responsible Party

  • Andreas Heigl
  • Eric Poe

Development Responsible Party

  • Jos Elstgeest (zghosts)
  • Ian Littman

Security Responsible Party

  • Adam Englander
  • Mike Willbanks

General Responsible Party

  • Michael Cullum
  • David Stockton

You might have already seen a few of us going through pull requests and issues. We have added a team to the GitHub organization so you can mention the entire group as @joindin/leadership-team in issues or pull requests. If you have pending issues or pull requests please note some of us will be ramping up on the project for the first time, however, we will address issues as we come across them.

What’s Next?

The leadership team will meet on February 5th to formalize the short term plan based on knowledge transfer from the previous maintainers to the new team. The product and development side of the team will be tasked with continuing the current plan to decommission the current hosted service while launching a replacement host in parallel without utilizing legacy applications. This will mean there may be a brief period of time where the hosted service has fewer features, we’re hopeful in our work to find these features and get them ready for a contributor to work on quickly will allow us to minimize any service issues.

The current target date to decommission the current live server is March 31st, 2019 and of course is subject to change. The team will also be creating a disaster recovery plan from scratch to use as a guide for not only backing up the current system and data but to plan the new infrastructure and workflows for a more modern and efficient system. The security side of the team will be working closely in these processes as well as being tasked with ensuring all services and systems are following industry standard protocols. If you are interested in contributing to these projects we’re using Github to plan and organize.

We Need YOU

A key part of our proposal was to ensure the longevity of the project by consistently onboarding new contributors and maintainers. This is why all of our teams have two individuals. We should constantly be cross-training and sharing knowledge not only to reduce bus factor but to teach the next generation of open source contributors.

The leadership team is committed to making participation in this project a harassment-free experience for everyone, regardless of the level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, or nationality via our Contributor Code of Conduct.

You are important. You are welcome here.