When bet365 CTO Martin Davies joined the online gaming company in 2001, it employed a single permanent programmer and six contract developers, having been founded in a Stoke-on-Trent Portakabin the year before.
Fast-forward 10 years and the company now has a 300-strong tech department consisting of 160 developers and 140 infrastructure support staff, which Davies has built up with the help of bet365 co-founder and CEO Denise Coates.
The company has a total of 1,400 staff still based in Stoke-on-Trent, turned over £5.4bn in 2010 and has a website that is live in 17 languages, with four million users across 240 countries.
Stoke native Davies has been involved in IT for 30 years. He was a software developer, helped build the tech infrastructure at a start-up company and worked as an IT consultant during Y2K.
Having joined bet365 when it was still a fledgling online gaming platform, Davies is now dealing with the challenges presented by rapid growth and an increasingly complex technology infrastructure.
Speaking to silicon.com recently, Davies discussed how he and his team are facing these tech challenges and how technology is core to bet365’s success.
A tech-driven organisation
As a web-based company, bet365’s business is inseparable from its technology. It invested £60m in tech in 2010 alone. However, most of the technology development work is carried out inhouse so the company has as much control over technology as possible.
The tech team is divided into 11 areas of expertise, including database administration, middleware systems, mobile, net architecture, product development and security.
“So we’re very keen on designing, building, managing and supporting everything ourselves. We don’t use managed services at all unless it’s absolutely unavoidable,” Davies told silicon.com.
He added that developing and running technology inhouse allows bet365 to achieve better service levels than if third-party technology was used, as it means the people who developed the technology – and have the best interests of the business at heart – can be easily reached when users have issues.
The company occasionally links up with other companies when it comes to technology in which it doesn’t have the necessary expertise, such as certain payments systems, but inhouse tech development remains a strong focus.
The role as CTO
Davies’ role is embedded in the technology side of the business. As a result…
…he doesn’t get too involved in the day-to-day management of the tech teams, preferring his head of software to take care of that. “I’m a CTO and I do the technology,” he told silicon.com.
Davies is involved in tackling the more difficult tech challenges and developing the strategy to progress the company’s technology infrastructure over the next few years.
“What I tend to be doing on a daily basis is getting involved in looking at the technologies we’re using, pinch points that we’ve got coming up and the strategy for how we’re going to deal with the continuing issues that we have with scalability, because we’re still growing quite rapidly and inevitably as you’re growing, new pain comes out of that and you’ve got to be planning for it,” he said.
Davies also has to make sure the technology supports bet365’s business objectives, so he works with the firm’s technology suppliers on improving their products to meet those goals.
The most important attributes for the CTO job are to have an open mind and to get involved with the people who are doing the nitty gritty of tech development, according to Davies.
“So I talk to software developers, I talk to the guys that are responsible for the network, I’m involved in the design of way the systems work at really quite a low level at times – I’ve even written code on the odd occasion,” he said.
As a web company, bet365 also enjoys the input of co-founder and CEO Denise Coates in the technology strategy. Davies works closely with Coates on the development of the website in particular.
Dealing with increasing complexity
“We are doing a lot of very complicated things – the business that we have is a very complex business and our infrastructure now is scattered across five datacentres in two countries: we have two in London, one in Manchester and two in Gibraltar.”
One of the major additions to the company’s gaming portfolio is in-play betting, which was added to the pure sports betting the company started out with about five years ago.
The in-play betting uses a variety of methods including data feeds and modelling techniques for changing the odds on bets, although Davies said there is usually a human involved in overseeing the changes that go through to make sure they’re sensible.
During the week, Davies said between 450,000 and 650,000 bets are available on bet365 per week. “That’s rather too many for people to maintain themselves so they tend to do some parts of it and the rest of it we do through modelling techniques,” Davies said.
Another relatively new service is live sport broadcasts, with Italian and Spanish league football contributing to more than 15,000 live events being broadcast on the website every year.
bet365 uses tech partner Perform to broadcast events, including the online-only England vs Ukraine football World Cup qualifier in October 2009, which was made available to bet365 subscribers by match rights holder Kentaro.
Although the business isn’t on the same scale as some other web companies, the complexity of the processes that…
…bet365 needs to carry out is significantly greater.
“There are many companies that have got a bigger scale than us. If you look at Amazon, it’s a huge, huge company [but] their prices aren’t changing on the books 2,000 times every second where our prices are changing all the time. For the systems to work efficiently they have to be got out to the customer. And I think it’s the fact that you’ve got the combination of high volume, low latency and complexity that makes it interesting,” Davies said.
Latency, scalability and security
Davies and his team are constantly working to improve bet365’s products and the reduction of latency is one of the main areas of focus. “The lower we can get the latency, the better the product is,” Davies said.
“When we’re making changes in our back-office systems, we’re looking to get those changes out to the customer within two seconds. So we’re very, very time-sensitive,” Davies added.
Scalability and availability of services are also a priority to enable the in-play gaming technology and advertising delivery to continue to expand. The cloud and virtualisation technology are helping in this particular area as well.
Security is also a big priority for bet365, which comes as no surprise considering the industry it operates in.
“The industry has a very high profile and is a very successful industry on the internet, so obviously it’s always an issue. We have an inhouse security team that’s carrying out continual testing on the website and… third-party external testing goes on on our website very frequently,” Davies said.
The use of innovative technology
The quantity of live data has required the company to deal with a huge amount of additional complexity that some of its systems have begun to struggle with.
To cope with these demands, the company has developed technology that is among the most advanced of any online business, while Davies said he has “no preconceptions” about new ways of doing things.
“My philosophy is we want the best solution – it doesn’t matter what technologies we use to achieve that, we’ll bring people in if we need to. You need to think outside of the box and think of different ways of doing things when you’re trying to solve difficult problems,” Davies said.
“We’re not a company that will say, ‘You can’t use that’. That is just not the way we do things, unless there’s a huge licensing cost associated with it. So we are very open-minded about things,” he added.
An example of the company thinking differently is its efforts to push more technology on to the internet through the use of…
…the Comet web application model, which allows web services to push data out to people’s browsers.
Essentially, the system uses permanently open connections to push sequential data – or deltas – out to customer browsers. The browser is constantly updated rather than the user requiring a complete web page to be downloaded to view changes.
This technology is used for the in-play betting service and the advertising banner system that bet365 runs on its own site as well as on those belonging to partners. These sites can have as many as one million concurrent users in total.
Since the company started to use this approach towards the end of 2009, it has been able to improve its services while keeping cost and bandwidth costs down.
The company is also looking at next-generation technologies to address other issues that more established technology is beginning to struggle with. “At the moment, we’re looking at lots and lots and lots of different technologies,” Davies told silicon.com.
Among the technologies that the company is starting to use or evaluate is NoSQL, which is essentially database-management technology for structured data that differs from classic relational database management systems. For example, it doesn’t always use fixed tables of data and is often easier to scale up.
“What we’re looking to achieve [by using NoSQL-type technology] is to free ourselves up from some of the pain barriers that you’ve got when you’re using traditional database technology because they only scale so far,” he said.
Traditional database tech would mean the company would need to look at techniques such as sharding, where different datacentres are used to serve different regions, or adding layers of technology, both of which Davies said would create further issues.
As you would expect of a technology-based business, bet365 has invested extensively in internal cloud technology where it can bring benefits.
The cloud infrastructure allows the technology team to deploy applications in a virtualised environment – for testing or production use – and move functionality between locations if there are issues with any of the datacentres.
The company uses VMware’s vFabric GemFire in-memory distributed data-management technology to pool and share resources, which Davies said improves the performance of applications, reduces latency and makes them more easily scalable.
As the systems become more complex, the need to find out how they are running becomes more important so that the services customers are using continue to run as efficiently as required.
“The ability to know what’s going on inside [systems] to a very detailed level becomes more important because it’s harder to find a problem. Having deep visibility into all aspects of what your systems are doing is really very important and we’re spending quite a lot of time developing tools to enable us to do that,” Davies said.
“The monitoring aspects of trying to find out things when things go wrong… you need to be able to see very deep into the data structures inside all the different layers of what you’ve been doing.”
The company has developed its own tech to monitor latency and performance across the various elements of the technology infrastructure, including a system that allows every packet of data that travels across the network to be analysed.
Allowing developers to experiment
bet365’s approach to using innovative technology extends to the way it allows its developers to work.
To encourage innovation, the company is…
…open to any new ideas that developers come up with. The developers know the business and are interested in technology so are in the best position to tackle the issues that arise.
“There is no reason why you have to impose how something should work on to [developers]. You should be listening to what they’re doing, listening to what they’re talking about and then, if you think that there’s value in it, adopting it and going down that route,” Davies said.
Another approach the company is keen on is to make sure the people who designed and built the technology are involved in its implementation across the business.
This policy means technology staff can try out various approaches rather than deal with fixed specifications from day one, which, Davies said, “if you’re any good at what you do, is a bit devaluing for you as an individual”.
“You want to be able to influence what you’re working on – it’s an important thing,” he added.
Despite this open-minded approach to technology innovation, Davies said his team doesn’t have time to follow the Google model of spending a certain proportion of their working week exploring new ideas.
However, there are many developers, including Davies, who build code in their spare time that can potentially become part of the bet365 portfolio.
Attracting IT talent
As the most prominent technology-based company in Stoke-on-Trent, bet365 has started to find IT skills are becoming a scarcer commodity. “It has started to prove more difficult because we’ve pretty much employed everybody that does IT in Stoke-on-Trent,” Davies said.
However, with the company located in the middle of the UK and its use of “interesting technology”, Davies said bet365 is still able to attract people from beyond the Potteries.
Although discussions have taken place about the possibility of expanding to other cities in the UK, Davies said spreading out the IT team could be counterproductive.
“There have been times when we’ve had an idea [and it’s] live three days later. One of the things we don