Data Centers

Decentralizing the cloud: Where do microgrids fit in?

Thoran Rodrigues discusses the benefits and drawbacks of centralized cloud computing power in large data centers and introduces the alternate idea of microgrids.

The cloud is an implementation of the concept of utility computing: computing resources - processing power, networking, storage and so on - being sold as service, through a distribution network, to be consumed by users as they need it. This concept is, in large part, already a reality today. It's possible to anyone with an Internet connection (the distribution network) to purchase computing power or storage on an hourly basis, creating and allocating servers and storage space as the need arises. Several large-scale systems already make use of this dynamic to increase processing capacity while at the same time reducing costs.

There is a direct analogy with electricity: Electricity is centrally generated in large power plants, and made available through a distribution network all the way to the outlets in users' homes. Most people don't have to worry about where the electricity is coming from, or if the electrical equipment is going to be compatible with the electricity that is available. While there are some variations - voltage levels, outlet formats - the electrical grid is, for the most part, a standardized system that everyone can easily access and use.

Centralization

Cloud computing follows the same lines: computing power is centrally generated in large data centers, and it is also made available through a distribution network (the Internet) to end-users everywhere. Concentrating computing power on large data centers brings with it several benefits: providers can take advantage of economies of scale to reduce costs and pass savings on to users; a centralized data center can be easier to manage; security may also become easier to manage, since there may be fewer points of attack to secure; finally, concentrating computing resource generation in the hands of a few companies can lead to specialization and to improved services and offerings.

On the other hand, it also creates a single point-of-failure. If all computing power came from a single data center, all it would take to bring our world to a halt would be to bring that one data center down. Even though most large cloud computing providers offer redundant systems on multiple data centers, the fact is that, as we rely more and more on the cloud, we become more vulnerable to failures. And, as I've mentioned several times, these data centers become much more valuable targets to anyone wanting to wreak havoc in the system.

How dangerous is centralization? Let's take, for instance, the hack on GitHub that was discovered and fixed recently. GitHub offers cloud-based version control repositories and software. A huge amount of software projects all over the world, from thousands of different companies use GitHub for their day-to-day needs. While no foul play has been detected yet, the potential repercussions of this event are much larger than they would be if each company had their own version control system, on their own servers, completely isolated.

Decentralization and microgrids

While centralization brings about a number of issues, the core ideas behind it - easy access to virtual resources, pay by usage, the ability to buy, or even sell, excess computing capacity - are solid. So it can be interesting to look at alternatives that can keep the advantages, but do away with most, or at least some, of the problems that I pointed out.

The obvious solution to centralization is de-centralization. To cloud computing this means that, instead of relying on large data centers to provide the computing power for end-users, to rely on the excess computing power available on PCs everywhere. In a sense, it is the same idea that has already been put into practice by the Seti@Home project: users have more computing capacity than they need, and they are willing to share this with others in exchange for credits, or even money. And this is the idea behind a very interesting new project called Cloud@Home, being developed by the same people. Through Cloud@Home, anyone with a PC can become a provider of "cloud" computing power through a virtualization platform. The project details can be found on a recent article on IEEE and on other sources.

The question is, can it work? There are several issues that need to be properly addressed by the project, starting with security and trust. With a completely decentralized system, it would be necessary to ensure the security of all data on each and every node, and on the overall platform, to avoid any breaches. It would also be much harder to build trust, since instead of hiring large companies, with established reputations, users would have to rely on other users. Ultimately, the idea of "microgrids" of computing can revolutionize the cloud landscape. Though large companies will always be necessary to provide large-scale power and highly reliable services, this idea has the potential to make the cloud much more accessible and widespread.

About

After working for a database company for 8 years, Thoran Rodrigues took the opportunity to open a cloud services company. For two years his company has been providing services for several of the largest e-commerce companies in Brazil, and over this t...

5 comments
Nick Hardiman
Nick Hardiman

I think you are onto something here. There could be a battle in the future between the centralizers vs. the de-centralizers. I know Greenpeace have done a lot of work on planning a distributed power network using CHP buildings, and the Freedombox people are working on distributing Internet services. Will you be covering those?

InfoStack
InfoStack

Thoran, you need to view the model horizontally. the distribution network is the real weakness here. Unlike smartgrids, computing/communication is 2-way. Only occasionally do consumers of electricity feed it back into the grid. But computing/communication users are constantly uploading. As we go to 2-way video and high-definition audio (Siri/voice recognition systems) the distribution network will become more critical. Because of remonopolization (particularly in the US, but generally elsewhere as well) bandwidth is 20x slower than it should be. Lastly, I would argue for something called centralized heirarchical networking. Consider akamai or similar solutions. Initially an application/service is highly centralized, but as demand grows (metcalfe's law) greater intelligence/processing gets pushed to the edge. Then a new (centralized) service supersedes the old. From the outside it appears to be a constant process of expansion and collapse, expansion and collapse, etc.... Therefore Centralized Hierarchical Networks (CHNs) to manage this constant state of growth.

brendondm
brendondm

First of all I like the microgrid concept. It makes the availability so powerful, but I see a problem with the administration of content as well as level of service it will provide. With the technology available it is so easy to do cloud based hosting which is part of the attraction for cloud computing, but as with mail hosting the quality has become poorer due to everyone hosting their own mail solutions. I would suggest companies and these huge datacentres taking a look at using something like geo clustering, available in many of the technologies used in the cloud. This would ensure the high availability but also offer the safety of decentralization. Like the electricity analogy by the way!!! With IPv6's multihoming as well as anycast addressing this is becoming more of possibility and because you will always connect to the nearest resource (geographically) this will make cloud computing more possible in bandwidth scarce areas, as you will essentially never need to go through a global backbone (ie seacom) to get to your resources and you will have all the availability you want. This is a particularly interesting topic as cloud computing is really dominating the market. Lastly although the decentralization of our networks will be fantastic in terms of the benefits it offers, not many admins and engineers would want to decentralize their networks after having just gone through ALL the effort of centralizing them.

thoran.rodrigues
thoran.rodrigues

I think the battle is, in many ways, already happening. For better or worse, our technology models move from decentralization to centralization and back: first we had isolated, custom-built computers, moving to centralized mainframes, moving to PCs, moving to datacenters, and now the cloud. Many of the ideas fueling the cloud space today came from distributed computing projects. I hadn't thought about covering this centralization/decentralization issue from that angle, but I think it's a great idea.

thoran.rodrigues
thoran.rodrigues

I agree with you that CHNs are an excellent idea. We can expand on it by seeing the computing grid in a similar fashion as the electric one: at first we'll have most consumers being simply consumers, contributing almost nothing back to the "grid", while most processing will be done by large datacenters. As you mentioned, as more processing power becomes available on the edges, end-users may be able to provide for more and more local needs of the grid. This localization would also reduce the problems with the distribution network: if I can get my processing power from my neighbor, I don't have to tax the global network by going to a server on the other side of the country, or even the world.