Storagebod Rotating Header Image

Cloud

Viperidae – not that venomous?

May 9th, 2013
by Martin Glassborow.
3 comments

There’s a lot of discussion about what ViPR is and what it isn’t; how much of this confusion is deliberate and how much is simply the normal of fog of war which pervades the storage industry is debateable. Having had some more time to think about it; I have some more thoughts and questions.

Firstly, it is a messy announcement; there’s a hotch-potch of products here, utilising IP from acquisitions and from internal EMC initiatives. There’s also an attempt to build a new narrative which doesn’t seem to work; perhaps it worked better when put into the context of an EMC World event but not so much from the outside.

And quite simply, I don’t see anything breathtaking or awe-inspiring but perhaps I’m just hard to impress these days?

But I think there are some good ideas here.

ViPR as a tool to improve storage management and turn it into something which is automatable is a pretty good idea. But we’ve had the ability to script much of this for many years; the problem has always been that every vendor has some different way of doing this, syntax and tools are different and often not internally consistent between themselves.

Building pools of capability and service; calling it a virtual array…that’s a good idea but nothing special. If ViPR can have virtual arrays which federate and span multiple arrays; moving workloads around within the virtual array, maintaining consistency groups and the like across arrays from different vendors; now that’d be something special. But that would almost certainly put you into the data-path and you end up building a more traditional storage virtualisation device.

Taking an approach where the management of array is abstracted and presented in a consistent manner; this is not storage virtualisation, perhaps it is storage management virtualisation?

EMC have made a big deal about the API being open and that anyone will be able to implement plug-ins for it; any vendor should be able to produce a plug-in which will allow ViPR to ‘manage’ their array.

I really like the idea that this also presents a consistent API to the ‘user’; allowing the user to not care about what the storage vendor is at the other end; they just ask for disk from a particular pool and off it goes. This should be able to be done from an application, a web-front-end or anything else which interacts with an API.

So ViPR becomes basically a translation layer.

Now, I wonder how EMC will react to someone producing their own clean-room implementation of the ViPR API? If someone does a Eucalyptus to them? Will they welcome it? Will they start messing around with the API? I am not talking about plug-ins here, I am talking about a ViPR-compatible service-broker.

On more practical things, I am also interested on how ViPR will be licensed? A capacity based model? A service based model? Number of devices?

What I am not currently seeing is something which looks especially evil! People talk about lock-in? Okay, if you write a lot of ViPR based automation and provisioning, you are going to be kind of locked-in but I don’t see anything that stops your arrays working if you take ViPR out. As far as I can see, you could still administer your arrays in the normal fashion?

But that in itself could be a problem; how does ViPR keep itself up to date with the current state of a storage estate? What if your storage guys try to manage both via ViPR and the more traditional array management tools?

Do we again end up with the horrible situation where the actual state of an environment is not reflected in the centralised tool.

I know EMC will not thank me for trying to categorise ViPR as just another storage management tool ‘headache’ and I am sure there is more to it. I’m sure that there will be someone along to brief me soon.

And I am pretty positive about what they are trying to do. I think the vitriol and FUD being thrown at it is out of all proportion but then again, so was the announcement.

Yes, I know have ignored the Object on File or File on Object part of the announcement. I’ll get onto that in a later post.

 

 

Tiny Frozen Hand

Feb 26th, 2013
by Martin Glassborow.
6 comments

So EMC have finally announced VMAX Cloud Edition; a VMAX iteration that has little to do with technology and everything to do with the way that EMC want us to consume storage. I could bitch about the stupid branding but too many people are expecting that!

Firstly, and in many ways the most important part of the announcement is around the cost model; EMC have moved to a linear cost model; in the past, purchasing a storage array had a relatively large front-loaded cost in that you have to purchase the controllers etc; this meant that your cost per terabyte was high to start with, it then declined and the potentially rose again as you added more controllers and then declined again.

This led to a storage-hugging attitude; that’s my storage array and you can’t use it. A linear cost model allows IT to provide the Business with a fixed cost per terabyte whether you were the first to use it or last to use it.  This allows us to move to a consumption and charging model that is closer to that of Amazon and the Cloud providers.

It is fair to point out that actually EMC and other vendors already have various ways to doing this already but they could be complex and used financial tools to enable.

Secondly, EMC are utilising a RESTful API to allow storage to be allocated programmatically from a service catalogue. There are also methods of metering and charging back for storage utilisation. Along with an easy to use portal; the consumption model continues to move to an on-demand model. If you work in IT and are not comfortable with this, you are in for a rough ride for quite some time.

Thirdly, the cost models that I have seen are very aggressive; EMC want to push this model and this technology.  If you want to purchase 50Tbs and beyond and you want it on EMC, I can’t see why you would buy any other block storage from EMC. It is almost as if EMC are forcing VNX into a SMB niche. In fact, if EMC can hit some of the price-points I’ve had hinted at; everyone is in a race to the bottom. It could be a Google vs Amazon price-battle.

Fourthly and probably obviously; EMC are likely to be shipping more capacity than an end-user requires, allowing them to grow with minimal disruption. If I was EMC, I’d ship quite a lot of extra capacity and allow a customer to burst into at no charge for a fair proportion of the year. Burst capacity often turns into bought capacity; our storage requirements are rarely temporary and quickly temporary becomes permanent. Storage procurement is never zipless; it always has long term consequences but if EMC can make it look and feel zipless…

I’m expecting EMC also to move to a similar model for the Isilon storage as well; it is well suited to this sort of model. And yet again,  this leaves VNX in an interesting position.

Out in the cold, with a tiny frozen hand….dying of consumption.

 

Doctors in the Clouds

Jan 30th, 2013
by Martin Glassborow.
No comments yet

At the recent London Cloud Camp; there was a lot of discussion about DevOps on the UnPanel; as the discussion went on, I was expecting the stage to be stormed by some of the older members in the audience. Certainly some of the tweets and the back-channel conversations which were going on were expressing some incredulity at some of the statements from the panel.

Then over beer and pizza; there were a few conversations about the subject and I had a great chat with Florian Otel who for a man who tries to position HP as a Cloud Company is actually a reasonable and sane guy (although he does have the slightly morose Scandinavian thing down pat but that might just be because he works for HP). The conversation batted around the subject a bit until I hit an analogy for DevOps that I liked and over the past twenty-four hours, I have knocked it around a bit more in my head. And although it doesn’t quite work, I can use it as the basis for an illustration.

Firstly, I am not anti-DevOps at all; the whole DevOps movement reminds me of when I was fresh-faced mainframe developer; we were expected to know an awful lot about our environment and infrastructure. We also tended to interact and configure our infrastructure with code; EXITS of many forms were part of our life.

DevOps however is never going to kill the IT department (note: when did the IT department become exclusively linked with IT Operations?) and you are always going to have specialists who are required to make and fix things.

So here goes; it is a very simple process to instantiate a human being really. The inputs are well known and it’s a repeatable process. This rather simple process however instantiates a complicated thing which can go wrong in many ways.

When it goes wrong, often the first port of call is your GP; they will poke and prod, ask questions and the good GP will listen and treat the person as a person. They will fix many problems and you go away happy and cured. But most GPs actually have only a rather superficial knowledge of everything that can go wrong; this is fine, as many problems are rather trivial. It is important however that the GP knows the limits of their knowledge and knows when to send the patient to a specialist.

The specialist is a rather different beast; what they generally see is a component that needs fixing; they often have lousy bedside manners and will do drastic things to get things working again. They know their domain really well and you really wouldn’t want to be without them. However to be honest, are they a really good investment? If a GP can treat 80% of the cases that they are faced with, why bother with the specialists? Because having people drop dead for something that could be treated is not especially acceptable.

As Cloud and Dynamic Infrastructures make it easier to throw up new systems with complicated interactions with other systems; unforeseeable consequences may become more frequent, your General Practitioner might be able to fix 80% of the problems with a magic white-pill or tweak here or there….but when your system is about to collapse in a heap, you might be quite thankful that you still have your component specialists who make it work again. Yes, they might be grumpy and miserable; their bedside manner might suck but you will be grateful that they are there.

Yes, they might work for your service provider but the IT Ops guys aren’t going away; in fact, you DevOps have taken away a lot of the drudgery of the Ops role. And when the phone rings, we know it is going to be something interesting and not just an ingrown toe-nail.

Of course the really good specialist also knows when the problem presented is not their speciality and pass it on. And then there is the IT Diagnostician; they are grumpy Vicodin addicts and really not very nice!

Just How Much Storage?

Jan 7th, 2013
by Martin Glassborow.
1 comment

A good friend of mine recently got in contact to ask my professional opinion on something for a book he was writing; it always amazes me that anyone asks my professional opinion on anything…especially people who have known me for many years but as he’s a great friend, I thought I’d  try to help.

He asked me how much a petabyte of storage would cost today and when I thought it would affordable for an individual? Both parts of the question are interesting in their own way.

How would a petabyte of storage cost? Why, it very much depends; it’s not as much as it cost last year but not as a cheap as some people would think. Firstly, it depends on what you might want to do with it; capacity, throughput and I/O performance are just part of the equation.

Of course then you’ve got the cost of actually running it; 400-500 spindles of spinning stuff takes a reasonable amount of power, cooling and facilities. Even if you can pack it densely, it is still likely to fall through the average floor.

There are some very good deals to be had mind you but you are still looking at several hundred thousand pounds, especially if you look at a four year cost.

And when will the average individual be able to afford a petabyte of storage? Well without some significant changes in storage technology; we are some time away from this being feasible. Even with 10 Terabyte disks, we are talking over a hundred disks.

But will we ever need a petabyte of personal storage? That’s extremely hard to say; I wonder if we will we see the amount of personal storage peak in the next decade?

And as for on-premises personal storage?

That should start to go into decline, for me it is already beginning to do so; I carry less storage around than I used to…I’ve replaced my 120Gb iPod with a 32 Gb phone but if I’m out with my camera, I’ve probably got 32Gb+ of cards with me. Yet with connected cameras coming and 4G (once we get reasonable tariffs), this will probably start to fall off.

I also expect to see the use of spinning rust go into decline as PVRs are replaced with streaming devices; it seems madness to me that a decent proportion of the world’s storage is storing redundant copies of the same content. How many copies of EastEnders does the world need to be stored on a locally spinning drive?

So I am not sure that we will get to a petabyte of personal storage any time soon but we already have access to many petabytes of storage via the Interwebs.

Personally, I didn’t buy any spinning rust last year and although I expect to buy some this year; this will mostly be refreshing what I’ve got.

Professionally, looks like over a petabyte per month is going to be pretty much run-rate.

That is a trend I expect to see continue; the difference between commercial and personal consumption is going to grow. There will be scary amounts of data around about you and generated by you; you just won’t know it or access it.

More Musings on Consumers and Cloud

Oct 29th, 2012
by Martin Glassborow.
No comments yet

I must stop looking at Kickstarter, I will end up bankrupt but it does give some insight in what is going to drive the Cloud and Big Data from a consumer point of view.

From Ubi to Ouya to Memeto; these are all new consumer devices that are going to further drive our relience on the Cloud and Cloud Services. The latter being a life-logging device that could drive the amount of storage that we require through the roof; they believe that an individual could be generating 1.5Tb of data per annum, they really don’t need a huge amount of customers to be generating multiple petabytes of data per year. And if they are successful, more devices will appear, some offering higher resolution, some potentially starting to offer video….

Many people will look at such services and wonder why anyone would be interested but it really doesn’t matter…these services are going to make today’s services look frugal in their use of storage.

And then the growth of voice recognition services where the recognition has been driven to the Cloud and causing a massive increase in the growth of compute requirements.

Throw in non-linear viewing of media and services like Catch-Up TV and we have almost a perfect storm…

Amazon Goes Glacial

Aug 21st, 2012
by Martin Glassborow.
8 comments

Amazon have announced a pretty interesting low-cost archive solution called Amazon Glacier; certainly the pricing which works out at $120 per terabyte per annum with 11 9’s availability is competitive. For those of us working with large media archives could this be a competitor with tape and all the tape management headaches that tape brings. I look after a fairly large media archive and there are times when I would gladly see the back of tape for-ever.

So is the Amazon Glacier the solution we are looking for? Well for the type of media archives that I manage, unfortunately the answer is not yet or not for all use cases. The 3-4 hour latency introduced on a recall by the Glacier does not fit many media organisations, especially those who might have a news component. At times even the minutes that retrieving from tape take seems to be unacceptable, especially to news editors and the like. And even at $120 per terabyte; when you are growing at multiple petabytes a year, the costs fairly quickly add-up.

Yet, this is the first storage product which has made be sit up and think that we could replace tape. If the access times were reduced substantially and it looked more like a large tape library; this would be an extremely interesting service. I just need the Glacier to move a bit faster.

Enterprise and Cloud

Aug 2nd, 2012
by Martin Glassborow.
4 comments

Anybody working in storage cannot fail to come across the term ‘Enterprise Storage’; a term which is often used to justify the cost of what it is commodity item that is stuck together with some clever software; ask a sales-man from a vendor as to what makes their storage ‘Enterprise’ and you will get a huge amount of fluff but with little substance. ‘Enterprise Storage’ is a marketing term.

And now we are seeing the word ‘Enterprise’ being used by some Cloud Service Providers and Cloud Vendors to try to distinguish their Cloud server from their competitors, especially when trying to differentiate themselves from Amazon. Yet, is this just a marketing term again? I don’t think it is but not for entirely positive reasons.

If your application has been properly architected and designed to run in a Cloud based infrastructure; you almost certainly don’t need to be running in an ‘Enterprise Cloud’ and the extra expense that brings; if you have tried to shoe-horn an existing application into the Cloud, you might well need to consider an Enterprise Cloud. Because many Enterprise Clouds are simply hosted environments re-branded as Cloud, often utilising virtualisation sitting on top of highly resilient hardware; they remove many of the transition costs to the Cloud by not actually transitioning to a Cloud Model.

A properly designed Cloud application will meet all the availability and performance requirements of the most demanding Enterprise and users whether it runs in an commodity cloud or an Enterprise Cloud. Redeveloping your existing application portfolio may well feel prohibitively expensive and hence many will avoid doing this. Ultimately though, many of these existing applications which live in the Enterprise Cloud will transition to a SAAS environment; CRM, ERP and other common enterprise applications are the obvious candidates. This will leave the those applications which make your Business special and from which you derive some kind of competitive advantage; these are the applications and architectures that you should be thinking about re-architecting and re-developing, not just dumping them into an ‘Enterprise Cloud’.

Try not to buy into the whole ‘Enterprise Cloud’ thing apart from as a transitionary step; think about what you need to do to run your business on any ‘Commodity Cloud’; how you design applications which are scalable and resilient at the application layer as opposed to infrastructure, think about how you make those applications environmentally agnostic with the ability to take advantage of spot pricing and brokerage. Or if you really don’t believe in the Cloud, stop pretending to and stop using Enterprise as camouflage.

Razor – An Idiot’s Guide to Getting Started!

Jun 2nd, 2012
by Martin Glassborow.
1 comment

My role at work does not really allow me to play with tech very often and unless I have a real target in mind, I’m not as much as an inveterate hacker as I used to be; I generally find a problem, fix a problem and then leave it alone until solution is broken. So I don’t tend to go into any depth on anything but every now and then, I see something and think that’s cool, can I get it to work.

When I saw the new tool from Nicholas Weaver and EMC built on Puppet to configure ‘bare-metal’ machines; I decided that was something cool enough to have a play with. But there were a few problems, I knew what Puppet is but I’d never used it and I really didn’t have a clue what I was doing.

Still, I followed the links to the documentation and started to hack; after a couple of failed attempts due to missing prerequisites, I decided to be a bit more methodical and document what I was doing.

So this is what I did…more or less. And hopefully this might be helpful to someone else. I am assuming a certain amount of capability tho’! So more an Idiot Savant than just an Idiot.

I have a number of core services already running at home; I have my own internal DNS and DHCP server running on Ubuntu, I have a couple of NAS servers and a couple of machines running ESX5i.

All the documentation for Razor is based around Ubuntu Precise Pangolin 12.04; so first thing to do was to build a Precise VM on one of the ESX5i boxes. This was provisioned with 2 Gigs and 2 virtual cores.

1) Install Ubuntu 12.04 Server and I always do an OpenSSH Server build at installation; I leave everything else to after I’ve done a base install.

2) I added a static assignment for the server in my DHCP box and created a DNS entry for it.

3) After the server was installed, I did my normal ‘break all the security’ and used sudo to set a root password and allow myself to log directly on as root. I’m at home and can’t be bothered to use sudo for everything.

4) I started installing packages, I’m not sure whether the order matters but this the order I did things and all this was done as root

EDIT:According to Nick and he should know, the Razor module installs Node and Mongo-db automagically…I had some problems the first couple of times and decided to do it myself, this is possibly because I’m an extremely clever idiot and break idiot proof processes.

Node.JS

apt-get install nodejs npm

MongoDB

I didn’t use the standard packaged version and pulled down the package from mongodb.org

apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
vi /etc/apt/sources.list
Added deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
apt-get update
apt-get install mongodb-10gen

Puppet

Yet again I didn’t use the standard packaged version and pulled down from puppetlabs.com

wget http://apt.puppetlabs.com/puppetlabs-release_1.0-3_all.deb
dpkg -i puppetlabs-release_1.0-3_all.deb
apt-get update
apt-get install puppetmaster

Ruby 1.9.3

Note that the above install does install Ruby but appears to bring down Ruby 1.8; Razor wants a later version.

apt-get install ruby1.9.3

This seems to do what you want!

At this point you should be in the position to start installing Razor.

Razor

This is very much cribbed from the Puppet Labs page here

puppet module install puppetlabs-razor

chown -R puppet:puppet /etc/puppet/modules

puppet apply /etc/puppet/modules/razor/tests/init.pp --verbose

This should run cleanly and at this stage you should have some confidence that Razor will probably work.

/opt/razor/bin/razor

This shows that it does work.

/opt/razor/bin/razor_daemon.rb start

/opt/razor/bin/razor_daemon.rb status

This starts the razor daemon and confirms it is running. Our friends at PuppetLabs forgot to tell you start the daemon, it’s kind of obvious I guess but made this idiot pause for a few minutes.

Configuring Your DHCP Server

I run my own DHCP server based and this needed to be configured to point netbooting servers at the Puppet/Razor server.

I amended the /etc/dhcp/dhcp.conf file and added the following

filename "pxelinux.0";
next-server ${tftp_server_ipaddress};

in the subnet declaration.

At this point, you should be ready to really start motoring and it should be plain sailing I hope. Certainly this idiot managed to follow the rest of the instructions for Example Usage on Puppetlabs.

Of course now I’ve got lots of reading to do around Puppet and the likes but at the moment, it does appear to work.

So great work Nick and everyone at EMC.

The Last of the Dinosaurs?

May 22nd, 2012
by Martin Glassborow.
5 comments

Myself and Chris ‘The Storage Architect’ Evans were having a twitter conversation during the EMC keynote where they announced the VMAX 40K; Chris was watching the live-stream and I was watching the Chelsea Flower Show, from Chris’ comments, I think that I got the better deal.

But we got to talking about the relevance of the VMAX and the whole bigger is better thing. Every refresh, the VMAX just gets bigger and bigger, more spindles and more capacity. Of course EMC are not the only company guilty of the bigger is better hubris.

VMAX and the like are the ‘Big Iron’ of the storage world; they are the choice of the lazy architect, the infrastructure patterns that they support are incredibly well understood and text-book but do they really support Cloud-like infrastructures going forward?

Now, there is no doubt in my mind that you could implement something which resembles a cloud or let’s say a virtual data-centre based on VMAX and it’s competitors. Certainly if you were a Service Provider which aspirations to move into the space; it’s an accelerated on-ramp to a new business model.

Yet just because you can, does that mean you should? EMC have done a huge amount of work to make it attractive, an API to enable to you to programmatically deploy and manage storage allows portals to be built to encourage self-service model. Perhaps you believe that this will allow light-touch administration and the end of the storage administrator.

And then myself and Chris started to talk about some of the realities; change control on a box of this size is going to be horrendous; in your own data-centre, co-ordination is going to be horrible but as a service provider? Well, that’s going to be some interesting terms and conditions.

Migration, in your own environment,  to migrate a petabyte array in a year means migrating 20 terabytes a week more or less. Now, depending on your workload, year-ends, quarter-ends and known peaks, your window for migrations could be quite small. And depending how you do it, it is not necessarily non-service impacting; mirroring at the host level means significantly increasing your host workload.

As a service provider; you have to know a lot about the workloads that you don’t really influence and don’t necessarily understand. As a service provider customer, you have to have a lot of faith in your service provider. When you are talking about massively-shared pieces of infrastructure, this becomes yet more problematic. You are going to have to reserve capacity and capability to support migration; if you find yourself overcommitting on performance i.e you make assumptions that peaks don’t all happen at once, you have to understand the workload impact of migration.

I am just not convinced that these massively monolithic arrays are entirely sensible; you can certainly provide secure multi-tenancy but can you prevent behaviours impacting the availability and performance of your data? And can you do it in all circumstances, such as code-level changes and migrations.

And if you’ve ever seen the back-out plan for a failed Enginuity upgrade; well the last time I saw one, it was terrifying.

I guess the phrase ‘Eggs and Baskets’ comes to mind; yet we still believe that bigger is better when we talk about arrays.

I think we need to have some serious discussion about optimum array sizes to cope with exceptions and when things go wrong. And then some discussion about the migration conundrum. Currently I’m thinking that a petabyte is as large as I want to go and as for the number of hosts/virtual hosts attached, I’m not sure. Although it might be better to think about the number of services an array supports and what can co-exist, both performance-wise but also availability window-wise.

No, the role of the Storage Admin is far from dead; it’s just become about administering and managing services as opposed to LUNs. Yet, the long-term future of the Big Iron array is limited for most people.

If you as an architect continue to architect all your solutions around Big Iron storage…you could be limiting your own future and the future of your company.

And you know what? I think EMC know this…but they don’t want to scare the horses!

Death of the Home Directory

May 12th, 2012
by Martin Glassborow.
No comments yet

Well, when I say that the Home Directory is dying; I mean that it is probably moving and with it some problems are going to be caused.

As I wander round our offices, I often see a familiar logo in people’s system trays; that of a little blue open box. More and more people are moving their documents into the Cloud; they really don’t care about security, the just want the convenience of their data where ever they are. As the corporate teams enforce a regime of encryption on USB flash-disks; everyone has moved onto Cloud-based storage. So yes, we are looking at ways that we can build internal offerings which bring the convenience but feel more secure. Are they any more secure? And will people use them?

I suspect that unless there are very proscriptive rules which block access to sites such as Dropbox, Box, Google Drive and the likes; this initiative will completely fail. The convenience of having all your data in one place and being able to work on any device will over-ride security concerns. If your internal offering does not support every device that people want to use; you may well be doomed.

And then this brings me onto BYOD; if you go down this route and evidence suggests that many will do so..you have yet more problems. Your security perimeter is changing and you are allowing potential hostile systems onto your network; in fact, you always probably did and hadn’t really thought about it.

I have heard of companies who are trying to police this by endorsing a BYOD policy but insisting that all devices should be ‘certified’ prior to being attached to the corporate network. Good luck with that! Even if you manage to certify the multitude of devices that your staff could turn up with as secure and good to go; that certification is only valid at that point or as long as nothing changes, no new applications installed, no updates installed and probably no use made of the device at all.

Security will need to move to the application and this could mean all of the applications; even those familiar applications such as Word and Excel. Potentially, this could mean locking down data and never allowing it be stored in a non-encrypted format on a local device.

The responsibility for ensuring your systems are secure is moving; the IT security teams will need to deal with a shifting perimeter and an increasingly complicated threat model. Forget about updating anti-virus and patching operating systems; forget about maintaining your firewall; well don’t but if you think that is what security is all about, you are in for a horrible shock.

 

← Earlier Posts Newer Posts →