Ghost & Ember @ EmberCamp

On Thursday, I had the privilege of speaking at EmberCamp in London about the experience of switching Ghost to using Ember. I wasn't sure what to expect from EmberCamp, as I'm not really an Ember dev, but I found myself having a thoroughly enjoyable day. The atmosphere of the conference was noticeably positive, upbeat and open minded, it was a great place to be.

Jamie & Leah did a great job of organising this conference to be a small, friendly, community event and I think Yehuda's keynote worked perfectly to set up a positive mindset for the day. I'm really happy that I got to be a part of it :)

Below are the slides from my talk. They won't make much sense on their own (they probably didn't in context either, haha) - so in this post, I thought I'd take a little bit of time to write down the key points of my talk as a reference. Turns out it's a 1500 word essay, who knew? Enjoy...

Update 2015-12-01: The full talk is now available on youtube.

Ghost & Ember

Before Ember, the Ghost admin was a small but messy backbone.js application. In January last year, we hit a roadblock with it. The code had become incredibly fragile, and we had reached a point where we felt we just could not add anything else to it, for the fear that it would just collapse in on itself. We realised we needed a new toolset.

In order to figure out which tool to use, we needed to have a big public discussion, and gather as much input from as many members of the community as possible. This was done via a GitHub issue, which ended up being a very positive, productive discussion.

Ember was chosen for two main reasons:

1. It fit our need for a tool that made lots of decisions for us - making it easier to collaborate in an Open Source environment.
2. The Ember community got involved with the discussion and showed us that they were a friendly, passionate and helpful group - they gave us the confidence to choose Ember.

Migration

As we started looking at the work of switching, we quickly realised that we needed to improve our JSON API before we'd be able to support an Ember app in Ghost. At the time, our API wasn't clean, consistent or complete so there was much to do, particularly around the auth flows. Building an Ember app necessarily requires you to have a strong API, so this is something to keep in mind when choosing a framework.

Because the API was an area we were familiar with, and Ember was not, the Ghost team ended up focusing heavily on the API improvements. Meanwhile a collection of people from the Ember community and regular Ghost contributors set to work building the admin in Ember on a branch. This meant that the Ghost team barely got to touch Ember - the project was almost entirely completed by volunteers.

Part way through the conversion, we got cold feet about shipping a new version of Ghost that had no tangible improvements for our users. We decided that rather than just completely rewriting the existing admin, we'd write the new admin with multi user support (til then, Ghost had been a single author blog).

So, the project took us about 6 months in total. Longer than we expected, but as well as getting a shiny new Ember admin there were lots of other positive outcomes. For one, we now had a shiny new API.

In particular, we learned a great deal from working with the Ember community. We have always seen the Ember project as a model open source project. Soon after the migration was finished, we adopted the idea of having a 'Core Team' and invited some of our volunteer contributors to join.

One year later

It has now been over a year since we shipped the first version of Ghost with an Ember admin UI. This gives us an interesting viewpoint from which we can look back over the decision making process, the migration process and the challenges we've faced in the time since to really assess the impact that switching to Ember has had on our product.

Firstly, we're certain we made the right choice. If we made the decision over again today, we'd still choose Ember. We are now operating on a whole different level in terms of the difficulty of the problems we are able to solve easily with the tools we have.

However, we still feel we're not quite moving as fast as we could be, we believe there's more power & productivity in Ember that we haven't yet unlocked. To explain this thinking, let me explain my theory of Ember & complexity:

Ember & Complexity

In explanations about the intentions or goals of Ember, I've heard Yehuda & Tom talk a lot about  complexity. One thing I found very interesting is the idea that, in client-side JS apps, there is a tendency to end up with overly complex applications, largely because of the use of the global scope.

That is, if every object in your application has access to every other object in your application, then it becomes easy to make use of that and build tightly coupled systems. Because of the network effect, complexity can grow exponentially as you add new objects to the application.

If complexity grows exponentially, then there comes a 'point of no return' where you balk at the task of adding anything new because it is so hard. That's what happened with our backbone application.

Ember takes measures to prevent this from happening.  Every time you add a new object in an Ember app it starts off isolated. You have to choose to add access to other objects. This is just one of the things that can help to keep complexity to a minimum.

The theory here, is that Ember sort of flips the complexity graph on its head.

When you start with an Ember app, the boiler plate or foundation is already significantly complex - so there's a well-known steep learning curve to getting familiar with all of that 'stuff'.

However, once you overcome that learning curve, the intention is, you hit a 'sweet spot' where that complexity will grow linearly. So there's an upfront cost and you reap the benefits later. As several people said to me after my talk: if you're building multiple Ember apps, you also only ever have to pay this cost once.

For Ghost, I believe that 'sweet spot' is still in the not too distant future. There are still a number of challenges that we are in the process of overcoming that will eventually lead to us (hopefully) hitting that sweet spot.

Challenges

Overcoming the Ember learning-curve as an organisation, has been a challenge. We didn't do a great job of internalising Ember knowledge during the migration project, meaning we ended up without a full-time technical leader for the Ember app.

Just a few months ago, we took a major step towards fixing this by hiring our very own Ember magician. This means we now have clear leadership and direction for the Ember app.

Testing

One major thing that I believe has been holding us back, has been testing. Ember unit testing wasn't much of a thing when we first started out, so we relied on our existing end-to-end tests that were written in Casper.js. These still worked after we switched to Ember but they lost a lot of their value.

That test suite had been designed to test for the high level problems we were having with Backbone.js. Things like whether items render in the right place at the right time, and if saving still works, etc. However, with Ember.js, the issues that result in bugs and common regressions are a completely different class of problem. Usually a detail of a state or state change, perhaps an edge case, somewhere within a complex component.

These are problems that lend themselves much better to unit testing than to end-to-end testing.

Last year we looked into adding unit tests, and with Robert Jackson's help got our first unit tests in place. However, as there was no real leadership in that area at the time, there hasn't been a push to get good quality coverage.

More recently, we hit a critical point with our old end-to-end tests. They were no longer delivering anywhere near enough value in terms of the problems they detected vs how much work they required to maintain. Random failures meant we had to babysit PRs to get them to pass in Travis.

A few weeks ago, I had a realisation. I had been holding onto those tests because they made me feel confident that the admin panel was working. Yet, if the tests aren't really testing the right things, then this is an illusion - a false sense of confidence.

So I turned the tests off. Now I feel very uncomfortable, but at least that sense of discomfort reflects reality.

We're now working towards getting a full suite of Ember acceptance tests in place. As high quality tests are critical to the idea of 'moving fast and not breaking too many things' I am certain that this will catapult us further over that learning curve and into the sweet spot :)

Ember moves fast

When we started out, despite Ember's opinionatedness, there was still confusion around some aspects, particularly views, controllers and components. It's fantastic that the framework is evolving to solve those problems, however it does leave a bunch of legacy code in its wake that needs refactoring in order to keep up.

We've been blessed to have contributors and Ember community members do a great deal of the work to keep us up to date (and we pretty much are), but I think it begs the question:

What else could have been achieved with that time and effort?

Note: After my talk, it seemed this may have come across as a direct criticism. It really isn't intended that way. I value the fact that Ember evolves quickly and isn't afraid to remove features that don't work well - having to keep up is the trade-off. Perhaps there's a way to shift or improve the balance, but I certainly wouldn't want Ember to stop evolving.

Open Source is awesome

I worried that some of my talk came across as overly negative. It's easy to be critical in hindsight, so I felt it was worth highlighting some key positive aspects of the process.

1. The conversion was a triumph of the open source community.

Our app was built almost entirely by volunteers. The Ember community is amazing, full of people who will help you. It's a great group to be a part of.

2. We haven't gotten stuck

A fear I've heard many times is that because Ember is 'monolithic', sometime in the future you may find yourself in a situation where you want to do something the framework doesn't allow, and you may get stuck.

We're over a year down the line and that hasn't happened :)

3. Stand on the shoulders of giants

Ember is an incredibly powerful tool if you embrace it. When you build an app the way Ember intends, you're leveraging the knowledge, experience and learning of the incredibly smart people behind the project. When you build an app with a tool like Ember, you are standing on the shoulders of giants. This is the true power of Open Source Software.