From start-up to scale-up

8 min readMar 7, 2020

Lessons Learned

I’d like to share my experience at Football Index following up on what I have written about the subject before joining Football Index (See my previous posts here: Startup Software Delivery. Part I — Starting Up and Startup Software Delivery. Part II — Scaling Up). Football Index is an innovative fantasy football-like stock market.

Beyond MVP

When I joined Football Index in 2017, the founders had achieved what every entrepreneur dreams of: product-market fit. This is what I typically call MVP. You have a working product and paying customers.

The very first steps for me were to:

Set up a collaborative development environment (systems and process)
Automate delivery and QA as much as possible (CI/CD)

This is what I was recommending in Startup Software Delivery I. I think that from day one it is important to initiate those two initiatives. That also enabled people to work from anywhere, and help with talent sourcing. So we started working on new features and platform rearchitecting with one agile team. This is a summary of our tools and processes:

We used JIRA as a tool for product and sprint backlogs, for two reasons: as a change management tool in a compliant way (this is a regulated business), and to enable distributed/remote work.
We required a product manager to prioritise work (as user stories) and we work together to prepare the backlog head of planning meetings
Every two weeks we had sprint reviews and sprint planning meetings (video calls) where we would review what was done and we would plan for the next weeks.
We had a continuous information flow, mainly only on slack
We automated code verifications and tests and we integrated CI/CD tools with our SCM (Github). We followed the stable trunk rule with feature branches and pull requests. Pull requests trigger automated checks and tests (CI) but also require peer reviews — three amigos (QA, DEV, Product).

Hiring

I have successfully applied my ideas from Startup Software Delivery I. Given that the number of people to hire was limited, we haven’t hired a tech recruitment team. I mainly used my network, some online platforms (Angel.co is great for remote talent) and recruitment agencies.

Culture, Motivation and Teamwork

There has been a lot of academic research about team dynamics and motivation. Google has done an internal study which is now famous: The five keys to a successful Google team. The findings match previous research:

Psychological safety: Can we take risks on this team without feeling insecure or embarrassed?
Dependability: Can we count on each other to do high quality work on time?
Structure & clarity: Are goals, roles, and execution plans on our team clear?
Meaning of work: Are we working on something that is personally important for each of us?
Impact of work: Do we fundamentally believe that the work we’re doing matters?

If the answer yes, then it likely you have a high performing team. In fact these dynamics are related to culture and motivation. Culture in an organisation is the way people interact and do things when they don’t follow direct orders or a process. What organisations and HR departments often get wrong is that actual culture is different from publicly advertised values. Also culture cannot be ordered as this then becomes a process. I personally fine the following cultural aspects to be effective for product engineering teams:

Psychological safety
Open communication
Freedom and ownership
Collaboration and team empowerment
Problem solving rather task driven and shared objectives
Happiness and fun

And in terms of motivation:

alignment of interests between individuals and the organisation
Psychological safety
Honest feedback
Clear objectives and responsibilities
“Hygiene” factors: office, salary, equipment, etc.

In What Silicon Valley “Gets” about Software Engineers that Traditional Companies Do Not, Gergely Orosz outlines the engineering culture that made Silicon Valley a success:

Autonomy for software engineers
Curious problem solvers, not mindless resources
Internal data, code, and documentation transparency
Exposure to the business and to business metrics
Engineer-to-engineer comms over triangle-communication
Investing in a less frustrating developer experience
Higher leverage → higher {autonomy, pay}
Engineers are value generators, not resources

From One team to Many teams

Once the team grew beyond the ideal size of 6 to 9, I had to organise multiple teams. This is never easy and no solution is perfect. I tried to apply what I preached at Startup Software Delivery Part 2, which is based on the Spotify agile scaling approach and Transferwise engineering culture. We didn’t have the scale for multiple product teams. We had one product — Football Index UK. But we had to tackle many challenges and build new systems. We tried a matrix/Spotify like structure:

The idea is for squads to be as cross-functional as possible, but they would own some part of the system. In practice not all squads could be cross-functional, because we had to optimise for tight deadlines and a fast delivery pace. So some specialisation was needed. Also, 100% autonomy and 100% cross functional teams cannot exist, as Spotify themselves figured out (https://www.jeremiahlee.com/posts/failed-squad-goals/)

During my time there, we had the following teams:

Mobile team doing Android and IOS development
A full stack team, working on legacy and web
A backend team, building new micro services and systems
An infrastructure team
Data team

We also had the following chapters:

Node
React
Java
DevOps
QA
Product
Data

It worked generally well, with two main issues/challenges:

Confusion around team / chapter separation. For instance DevOps engineers tended to see themselves as a separate team. While they really belonged to a delivery team formally, but also formed a chapter
Initial difficulty to split the original agile team into two, as people talk to each other daily and have worked across every system previously.

Product and Strategy

One of the key roles of a CTO is to create or facilitate the IT strategy of a company. Here is my take on it.

Assuming we have an idea about the general business strategy, which of course is strongly dependent on the tech strategy, we can try to answer the same questions as in general strategy: why, how, what. In my experience, the most useful tool in an agile, innovative environment is the roadmap. The roadmap is a document shared between stakeholders which defines key objectives in the short term, near term (6 months) and longer term. The OKR (Objectives and Key Results) framework can also be used to define those objectives. This document is constantly evolving based on progress, feedback and new realities. It is typically updated every few months or more often when needed.

How do we define what is in the roadmap and how do we prioritise? Key objectives are driven by:

technical priorities
business strategy
customer requirements

Why not only customer requirements? That is the case for service companies who work exclusively for their customers. But when building a product, we must take into consideration what contributes to strategic objectives also what is technically feasible or what needs R&D in order to answer questions around feasibility and cost.

Scaling tech

Scaling the team came just in time because, as I mentioned in Startup Software Delivery Part 2:

Systems will be under heavy pressure
Everything will start breaking down
You will need to cater to different customer groups and different stakeholders

We were hitting the limits in terms of database and infrastructure. Also ,security and high availability requirements only go up. The bigger the business, the higher the impact and the cost of any outage, data issue or security issue. As expected, scaling data access in read mode was easy. The challenge is scaling transactions. This is why we started implementing a CQRS architecture and also using a third party matching engine developed by Nasdaq technologies.

Here is the outline of what we did to scale our tech:

Migration to AWS with private VPCs and infrastructure as code
EKS (AWS Hosted kubernetes)
RDS Aurora as hosted database. This allowed massive horizontal scaling in read queries.
Elastic Search for searches and time series data. ES Also offloaded a lot of load from the DB
Caching: Redis, CDNs
Queuing of operations with AWS SQS
All data queries in a separate OLAP database: BigQuery from Google

Compliance

As a regulated business we had to follow different compliance standards:

ISO 27001
PCI DSS
GDPR
UKGC regulations

The good news is that many of the requirements overlap, typically around data protection and security. Compliance didn’t hinder our ability to be agile. We just had to follow our quality and delivery processes and document everything. This where JIRA was useful as a self documenting tool for every software change.

Lessons learned

I’d like to share a few the lessons learned. First and foremost is stakeholder management. And this experience is shared with my peers. You may build a good product and have a lean, efficient and happy team. But if you don’t manage perceptions your efforts can be in vain. And this is very hard, as stakeholder management can take a lot of your precious time, that should be used to build teams and products.

Another take away is that quality always matters. And quality assurance is not just testing. It is product design, architectural trade-offs, team processes, delivery processes and tooling, monitoring, data analysis. Quality should be a concern for everyone, as issues can be costly for the business.

The last point is somewhat related to quality and risk management. Try to avoid breaking changes and “big bang” releases. This actually requires work on product level. Having the CI/CD machinery is great. But you also need to scope work items or user stories in a way that makes them releasable in a CD manner. You can use decoupling/unbundling, feature flags, and negotiations with product. There is a tendency for product people to want many new features in a bundled way. Our job as engineers is to try and make things releasable continuously in order to manage risks and breaking changes.