Storagebod Rotating Header Image

Data Centres

2014 – A Look Forward….

As as we come to the end of another year, it is worth looking forward to see what if anything is going to change in the storage world next year because this year has pretty much been a bust as to innovation and radical new products.

So what is going to change?

I get the feeling not a huge amount.

Storage growth is going to continue for the end-users but the vendors are going to continue to experience a plateau of revenues. As end-users, we will expect more for our money but it will be mostly more of the same.

More hype around Software-Defined-Everything will keep the marketeers and the marchitecture specialists well employed for the next twelve months but don’t expect anything radical. The only innovation is going to be around pricing and consumption models as vendors try to maintain margins.

Early conversations this year point to the fact that the vendors really have little idea how to price their products in this space; if your software+commodity-hardware=cost-of-enterprise-array, what is in it for me?  If vendors get their pricing right; this could be very disruptive but at what cost to their own market position?

We shall see more attempts to integrate storage into the whole-stacks and we’ll see more attempts to converge compute, network and storage at hardware and software levels. Most of these will be some kind of Frankenpliance and converged only in shrink-wrap.

Flash will continue to be hyped as the saviour of the data-centre but we’ll still struggle to find real value in the proposition in many places as will many investors. There is a reckoning coming. I think some of the hybrid manufacturers might do better than the All-Flash challengers.

Hopefully however the costs of commodity SSDs will keep coming down and it’ll finally allow everyone to enjoy better performance on their work-laptops!

Shingled Magnetic Recording will allow storage densities to increase and we’ll see larger capacity drives ship but don’t expect them to appear in mainstream arrays soon; the vibration issues and re-write process is going to require some clever software and hardware to fully commercialise these. Still for those of us who are interested in long-term archive disks, this is an area worth watching.

FCoE will continue to be a side-show and FC, like tape, will soldier on happily. NAS will continue to eat away at the block storage market and perhaps 2014 will be the year that Object storage finally takes off.

Feeling Lucky?

And here we go again, another IT systems failure at RBS; RBS appear to have been having a remarkable run of high-profile core-system failures but I suspect that they have been rather unlucky or at least everyone else has been lucky. Ross McEwan, the new Chief Executive of RBS has admitted that decades of under-investment in IT Systems is to blame.

Decades seems to be an awful long-time but may well be accurate; certainly when I started working in IT twenty-five years ago, the rot had already set in. For example, the retail-bank that I started at had it’s core standing order system written in pounds, shillings and pence with a translation routine sitting on top of it; yet many of these systems were supposed to have been re-written as part of the millenium-bug investigations. Most of this didn’t happen, whole-scale rewrites of systems decades old and with few people who understood how they worked, this was simply not a great investment; just patch it up and move on.

RBS are not going to be the only large company sitting on a huge liability in the form of legacy applications; pretty much all of the banks do and many others. Applications have been moved from one generation of mainframe to the next and they still generally work but the people who know how they really work are long gone.

Yet this is no longer constrained to mainframe operations; many of us can point at applications running on kit which is ten years or more old on long-deprecated operating-systems. Just talk to your friendly DBA about how many applications are still dependent on Oracle 8 and in cases even earlier. Every data-centre has an application sitting in the corner doing something but no-one knows what it is and no-one will turn-off just in case.

Faced with ever declining IT budgets; either a real decline or being expected to do more with the same amount; legacy applications are getting left behind. Yes, we come across attempts to encapsulate the application in a VM and run it on the latest hardware but it still does not fix the legacy issue.

If it ain’t broke, don’t fix it…but the thing is, most software is broken but you’ve just not yet come across the condition that breaks it. Now the condition that breaks it may well be the untrained operator who does not know the cunning work-around to keep an application running; work-arounds simply should not become standard operating procedure.

Question is as we chase the new world of dynamic operations with applications churning every day; who is brave enough to argue for budget to go back and fix those things which aren’t broken. Who is going to be brave enough to argue for budget to properly decommission legacy systems, you know those systems who only have one user who happens to have a C at the beginning of their job title?

Now it seems that Ross McEwan may be one who is actually being forced into taking action; is anyone else going take action without a major failure and serious reputational damage? Or do people just feel lucky?




In With The New

As vendors race to be better, faster and to differentiate themselves in an already busy marketplace, the real needs of the storage teams can be left un-met and also that of the storage consumer. At times it is as if the various vendors are building dragsters, calling them family saloons and hoping that nobody notices. The problems that I blogged about when I started out blogging seem still mostly unsolved.


Storage management at scale is still problematic; it is still extremely hard to find a toolset that will allow a busy team to be able to assess health, performance, supportability and capacity at a glance. Still too many teams are using spreadsheets and manually maintained records to manage their storage.

Tools which allow end-to-end management of an infrastructure from rust to silicon and all parts in-between still don’t exist or if they do, they come with large price-tags which invariably do not have a real ROI or a realistic implementation strategy.

As we build more silos in the storage-infrastructure; getting a view of the whole estate is harder now than ever. Multi-vendor management tools are in general lacking in capability with many vendors using subtle changes to inflict damage on the competing management tools.


Data mobility across tiers where those tiers are spread across multiple vendors is hard; applications are generally not currently architected to encapsulate this functionality in their non-functional specifications. And many vendors don’t want you to be able to move data between their devices and competitors for obvious reasons.

But surely the most blinkered flash start-up must realise that this needs to be addressed; it is going to be an unusual company who will put all of their data onto flash.

Of course this is not just a problem for the start-ups but it could be a major barrier for adoption and is one of the hardest hurdles to overcome.


Although we have scale-out and scale-up solutions; scaling is a problem. Yes, we can scale to what appears to be almost limitless size these days but the process of scaling brings problems. Adding additional capacity is relatively simple; rebalancing performance to effectively use that capacity is not so easy. If you don’t rebalance, you risk hotspots and even under-utilisation.

It requires careful planning and timing even with tools; it means understanding the underlying performance characteristics and requirements of your applications. And with some of the newer architectures that are storing metadata and de-duping; this appears to be a challenge to vendors. Ask questions of vendors as to why they are limited to a number of nodes; there will sheepish shuffling of feet and alternative methods of federating a number of arrays into one logical entity will quickly come into play.

And then mobility between arrays becomes an issue to be addressed.

Deterministic Performance

As arrays get larger; more workloads get consolidated onto a single array and without the ability to isolate workloads or guarantee performance; the risk of bad and noisy neighbours increases. Few vendors have yet grasped the nettle of QoS and yet fewer developers actually understand what their performance characteristics and requirements.

Data Growth

Despite all efforts to curtail this; we store ever larger amounts of data. We need an industry-wide initiative to look at how we better curate and manage data. And yet if we solve the problems above, the growth issue will simply get we reduce the friction and the management overhead, we’ll simply consume more and more.

Perhaps the vendors should be concentrating on making it harder and even more expensive to store data. It might be the only way to slow down the inexorable demand for ever more storage. Still, that’s not really in their interest.

All of the above is in their interest…makes you wonder why they are still problems.



Tape – the Death Watch..

Watching the Spectralogic announcements from a far and getting involved in a conversation about tape on Twitter has really brought home the ambivalent relationship I have with tape; it is a huge part of my professional life but if it could be removed my environment, I’d be more than happy.

Ragging on the tape vendors does at times feel like kicking a kitten but ultimately tape sucks as a medium; it’s fundamental problem is that it is a sequential medium in a random world.

If you are happy to write your data away and only ever access it in truly predictable fashions; it is potentially fantastic but unfortunately much of business is not like this. People talk about tape as being the best possible medium for cold storage and that is true, as long as you never want to thaw large quantities quickly. If you only ever want to thaw a small amount and in relatively predictable manner; you’ll be fine with tape. Well, in the short term anyway.

And getting IT to look at an horizon which more than one refresh generation away is extremely tough.

Of course, replacing tape with disk is not yet economic over the short-term views that we generally take; the cost of disk is still high when compared to tape; disk’s environmental footprint is still pretty poor when compared to tape and from a sheer density point of view, tape still has a huge way to go…even if we start factor in upcoming technologies such as shingled disks.

So for long-term archives; disk will continue to struggle against tape…however does that means we are doomed to live with tape for years to come? Well SSDs are going to take 5-7 years to hit parity with disk prices; which means that they are not going to hit parity with tape for some time.

Yet I think the logical long-term replacement for tape at present is SSDs in some form or another; I fully expect the Facebooks and the Googles of this world to start to look at the ways of building mass archives on SSD in an economic fashion. They have massive data requirements and as they grow to maturity as businesses; the age of that data is increasing…their users do very little in the way of curation, so that data is going to grow forever and it probably has fairly random access patterns.

You don’t know when someone is going to start going through someone’s pictures, videos and timelines; so that cold data could warm pretty quickly.  So having to recall it from tape is not going to be fun; the contention issues for starters and unless you come up with ways of colocating all of an individual’s data on a single tape; a simple trawl could send a tape-robot into melt down. Now perhaps you could do some big data analytics and start recalling data based on timelines; employ a bunch of actuaries to analyse the data and recall data based on actuarial analysis.

The various news organisations already do this to a certain extent and have obits prepared for most major world figures. But this would be at another scale entirely.

So funnily enough…tape, the medium that wouldn’t die could be kiboshed by death. And if the hyper-scale companies can come up with an economic model which replaces tape…I’ll raise a glass to good time and mourn it little..

And with that cheerful note…I’ll close..



Die Lun DIE!

I know people think that storagebods are often backward thinking and hidebound by tradition and there is some truth in that. But the reality is that we can’t afford to carry on like this; demands are such that we should grasp anything which makes our lives easier.

However we need some help both with tools and education; in fact we could do with some radical thinking as well; some moves which allow us to break with the past. In fact what I am going to suggest almost negates my previous blog entry here but not entirely.

The LUN must die, die, die….I cannot tell you how much I loathe the LUN as a abstraction now; the continued existence of the LUN offends mine eyes! Why?

Because it allows people to carry on asking for stupid things like multiple 9 gigabyte LUNs for databases and the likes. When we are dealing with terabyte+ databases; this is plain stupid. It also encourage people to believe that they can do a better job of laying out an array than an automated process.

We need to move to a more service oriented provisioning model; where we provision capacity and ask for a IOPS and latency profile appropriate to the service provision. Let the computers work it all out.

This has significant ease of management and removes what has become a fairly pointless abstraction from the world. It means it easier to configure replication, data-protection, snaps and clones and the like. It means that growing an environment becomes simpler as well.

It would make the block world feel at closer to the file world. Actually, it may even allow us to wrap a workload into something which feels like an object; a super-big-object but still an object.

We move to a world where applications can request space programmatically if required.

As we start to move away from an infrastructure which is dominated by the traditional RAID architectures; this makes more sense than the current LUN abstraction.

If I already had one of these forward-looking architectures, say XIV or 3PAR; I’d look at ways of baking this in now..this should be relatively easy for them, certainly a lot easier than some of the more legacy architectures out there. But even the long-in-the-tooth and tired architectures such as VMAX should be able to be provisioned like that.

And then what we need is vendors to push this as the standard for provisioning…yes, you can still do it the old way but it is slower and may well be less performant.

Once you’ve done that….perhaps we can have a serious look at Target Driven Zoning; if you want to move to a Software Defined Data-Centre; enhancements to the existing protocols like this are absolutely key.


Five Years On (part 3)

So all the changes referenced in part 2, what do they mean? Are we are at an inflection point?

The answer to the latter question is probably yes but we could be at a number of inflection points both localised vendor inflection points but also industry-wide ones as well. But we’ll probably not know for a couple more years and then with hindsight we can look back and see.

The most dramatic change that we have seen in the past five years is the coming of Flash-based storage devices; this is beginning to change our estates and what we thought was going to become the norm.

Five years ago; we were talking about general purpose, multi-tier arrays; automated tiering and provisioning but all coming together in a single monolithic device. The multi-protocol filer model was going to become the dominant model; this was going to allow us to break down silos in the data centre and to simply the estate.

Arrays were getting bigger as were disks; i/o density was a real problem and generally the slowest part of any system was the back-end storage.

And then SSDs began to happen; I know that flash-based/memory-based arrays have been around for a long time but they were very much specialist and a niche market. But the arrival of the SSD; flash in familar form-factor at a slightly less eye-watering price was a real change-bringer.

EMC and others scrambled to make use of this technology; treat them as a faster disk tier in the existing arrays was the order of the day. Automated Storage Tiering technology was the must have technology for many array manufacturers; few customers could afford to run all of their workloads on an entirely SSD-based infrastructure.

Yet if you talk to the early adopters of SSDs in these arrays; you will soon hear some horror stories; the legacy arrays simply were not architected to make best use of the SSDs in them. And arguably still aren’t; yes, they’ll run faster than your 15k spinning rust tier but you are not getting the full value from them.

I think that all the legacy array manufacturers knew that there were going to be bottle-necks and problems; the different approaches that the vendors take almost points to this and the different approaches taken by a single vendor..from using flash as a cache to utilising it simply as a faster disk…using it as extension of the read cache to using it as both a read and write cache.

Vendors claiming that they had the one true answer….none of them did.

This has enabled a bunch of start-ups to burgeon; where confusion reigns, there is opportunity for disruption. That and the open-sourcing of ZFS has built massive opportunity for smaller start-ups, the cost of entry into the market has dropped. Although if you examine many of the start-ups offerings; they are really  a familiar architecture but aimed at a different price point and market as opposed to the larger storage vendors.

And we have seen a veritable snow-storm of cash both in the form of VC-money but also acquisition as the traditional vendors realise that they simply cannot innovate quickly enough within their own confines.

Whilst all this was going on; there has been an incredible rise in the amount of data that is now being stored and captured. The more traditional architectures struggle; scale-up has it’s limits in many cases and techniques from the HPC market place began to become mainstream. Scale-out architectures had begun to appear; firstly in the HPC market, then into the media space and now with the massive data demands of the traditional enterprises…we see them across the board.

Throw SSDs, Scale-Out together with Virtualisation; you have created a perfect opportunity for all in the storage market to come up with new ways of fleecing providing value to their customers.

How do you get these newly siloed data-stores to work in harmonious and easy to manage way? How do we meet the demands of businesses that are growing ever faster. Of course we invent a new acronym that’s how….’SDS’ or ‘Software Defined Storage’

Funnily enough; the whole SDS movement takes me right back to the beginning; many of my early blogs were focused on the terribleness of ECC as a tool to manage storage. Much of it due to the frustration that it was both truly awful and was trying to do to much.

It needed to be simpler; the administration tools were getting better but the umbrella tools such as ECC just seemed to collapse under their own weight. Getting information out of them was hard work; EMC had teams devoted to writing custom reports for customers because it was so hard to get ECC to report anything useful. There was no real API and it was easier to interrogate that database directly.

But even then it struck me that it should have been simple to code something which sat on top of the various arrays (from all vendors); queried them and pulled back useful information. Most of them already had fully featured CLIs; it should have been not beyond the wit of man to code a layer that sat above the CLIs that took simple operations such as ‘allocate 10x10Gb LUNs to host ‘x’ ‘ and turn them into the appropriate array commands; no matter which array.

I think this is the promise of SDS. I hope the next five years will see the development of this; that we see storage with in a data-centre becoming more standardised from an programmatic point of view.

I have hopes but I’m sure we’ll see many of the vendors trying to push their standard and we’ll probably still be in a world of storage silos and ponds…not a unified Sea of Storage.



Five Years On (part 2)

Looking back over the last five years; what has changed in the storage industry?

Well, there have certainly been a few structural changes; the wannabes, the could-bes, theyve mostly disappeared through acquisition or general collapse. The big players are still the big players; EMC, HDS, HP, IBM and NetApp still pretty much dominate the industry.

And their core products are pretty much the same at present; there’s been little revolution and a bit of evolution but the array in the data-centre today doesn’t yet feel much different from the array from five years ago.

Five years ago I was banging on about how useless ECC was and how poor the storage management tools are in general. The most used storage management tool was Excel. That was five years ago and as it was then, so it is today. No-one has yet produced a great storage management tool to enable the management of these ever growing estates.

Yet, there has been a massive improvement in the storage administration tools; anyone with a modicum of storage knowledge should be able to configure almost any array these days. Yes, you will be working at the GUI but I can take an IBM storage admin and point them at an EMC array, they will be able to carve it up and present storage.

Utilisation figures for storage still tend to be challenging; there is a great deal of wastage as I have blogged about recently. Some of this is poor user behaviour and some is poor marketing behaviour in that there is no way way to use what has been sold effectively.

So pretty much nothing has changed then?


Apart from the impact of SSD and Flash on the market; the massive number of start-ups focused on this sector…

Oh…and scale-out; Scale-Out is the new Scale-Up…Go Wide or Go Home..

Oh..then there’s virtualisation; the impact of virtualisation on the storage estate has been huge…

And then there’s that thing called Cloud which no-one can grasp and means different things to everyone..

And then there’s the impact of Amazon and their storage technologies..

And Big Data and the ever exploding growth of data collected and the ever hyperbolic hype-cycle.

So nothing’s really changed whilst everything has.


Speed2Pb is Heavy Going…

EMC Marketing have done it again and managed to turn into what might be an interesting refresh of a product into something that just irritates me and many others.

It started off badly when they put the Sneak Preview video up and decided to have a dig at their lead competitor; then some tweets where they used the tag #NotAppy.

And then the hype cycle started to ramp up. So we have a ridiculously overblown launch with some tenuous link to F1. Tyre changing competitions and the likes which appear to be fun but just break up the presentations and destroy the flow.

EMC are just very bad at this sort of launch; speeds, feeds, marketing mumbo-jumbo linked in with videos/events which trash competitors, bore the audience and add little value. But all with the highest production values.

So what did the event feel like? It felt like an internal kick-off, an event where EMC high-five themselves and pretty ignore the customers. This felt more like an EMC event of eons ago along with a smattering of cheer-leading from Social Media.

There was little about the value, the use-case and what it will allow customers to do.

Death by PowerPoint; overly complex and busy slides.

And no humour…no humour! Make me laugh, make me smile!

Obviously I’m sure that it all played well to everyone else…and I’m not the target audience.

However, I think the technologies launched might be interesting; I think if the VNX2 code has undergone a rewrite, it’s long overdue and an achievement. It deserved better…

Such Fun…

With EMC allegedly putting the VMAX into the capacity tier and suggesting that performance cannot be met by the traditional SAN; are we finally beginning to look at the death of the storage array?

The storage array as a shared monolithic device came about almost directly as the result of distributed computing; the necessity for a one-to-many device was not really there when the data-centre was dominated by the mainframe. And yet as computing has become ever more distributed; the storage array has begun to struggle more and more to keep up.

Magnetic spinning platters of rust have hardly increased in speed in a decade or more; their capacity has got ever bigger tho’; storage arrays have got denser and denser from a capacity point of view, yet real-world performance has just not kept pace. More and more cache has helped to hide some of this; SSDs have helped but to what degree?

It also has not helped that the plumbing for most SANs is Fibre-channel; esoteric, expensive and ornery, the image of the storage array is not good.

Throw in the increased compute power and the ever incessant demands for more data processing, coupled with an attitude to data-hoarding at a corporate scale which would make even the most OCD amongst of us look relatively normal.

And add the potential for storage-arrays to become less reliable and more vulnerable to real data-loss as RAID becomes less and less of an viable data-protection methodology at scale.

Cost and complexity with a sense of unease about the future means that storage must change. So what are we seeing?

A rebirth in DAS? Or perhaps simply a new iteration of DAS?

From Pernix to ScaleIO to clustered-filesystems such as GPFS; the heart of the new DAS is Shared-Nothing-Clusters. ex-Fusion-IO’s David Flynn appears to be doing something to pool storage attached to servers; you can bet that there will be a Flash part to all this.

We are going to have a multitude of products; interoperability issues like never before, implementation and management headaches…do you implement one of these products or many? What happens if you have to move data around between these various implementations? Will they present as a file-system today? Are they looking to replace current file-systems; I know many sys-admins who will cry if you try to take VxFS away from them.

What does data protection look like? I must say that the XIV data-protection methods which were scorned by many (me included) look very prescient at the moment (still no software XIV tho’? What gives IBM…).

And then there is application specific nature of much of this storage; so many start-ups are focused on VMware and providing storage in clever ways to vSphere…when VMware’s storage roadmap looks so rich and so aimed taking that market, is this wise?

The noise and clamour from the small and often quite frankly under-funded start-ups is becoming deafening…and I’ve yet to see a compelling product which I’d back my business on. The whole thing feels very much like the early days of the storage-array; it’s kind of fun really.

From Servers to Service?

Should Enterprise Vendors consider becoming Service Providers? When Rich Rogers of HDS  tweeted this and my initial response was

This got me thinking, why does everyone think that Enterprise Vendors shouldn’t become Service Providers? Is this a reasonable response or just a knee-jerk, get out of my space and stick to doing what you are ‘good’ at.

It is often suggested that you should not compete with your customers; if Enterprise Vendors move into the Service Provider space, they compete with some of their largest customers, the Service Providers and potentially all of their customers; the Enterprise IT departments.

But the Service Providers are already beginning to compete with the Enterprise Vendors, more and more of them are looking at moving to a commodity model and not buying everything from the Enterprise Vendors; larger IT departments are thinking the same. Some of this is due to cost but much of it is that they feel that they can do a better job of meeting their business requirements by engineering solutions internally.

If the Enterprise Vendors find themselves squeezed by this; is it really fair that they should stay in their little box and watch their revenues dwindle away? They can compete in different ways, they can compete by moving their own products to more of a commodity model, many are already beginning to do so; they could compete by building a Service Provider model and move into that space.

Many of the Enterprise Vendors have substantial internal IT functions; some have large services organisations; some already play in the hosting/outsourcing space.  So why shouldn’t they move into the Service Provider space? Why not leverage the skills that they already have?

Yes, they change their business model; they will have to be careful that they ensure that they compete on a level playing field and look very carefully that they are not utilising their internal influence on pricing and development to drive an unfair competitive advantage. But if they feel that they can do a better job than the existing Service Providers; driving down costs and improving capability in this space….more power to them.

If an online bookstore can do it; why shouldn’t they? I don’t fear their entry into the market, history suggests that they have made a bit of a hash of it so far…but guys fill your boots.

And potentially, it improves things for us all; as the vendors try to manage their kit at scale, as they try to maintain service availability, as they try to deploy and develop an agile service; we all get to benefit from the improvements…Service Providers, Enterprise Vendors, End-Users…everyone.