Storagebod Rotating Header Image

Me

Storage is Interesting…

May 21st, 2013
by Martin Glassborow.
No comments yet

A fellow blogger has a habit of referring to storage as snorage and I suspect that is the attitude of many. What’s so interesting about storage, it’s just that place that you keep your stuff? And many years ago as an entry level systems programmer; there were two teams that I was never going to join…one being the test team and the other being the storage team, because they were boring. Recently I have run both a test team and a storage team and enjoyed the experience immensely.

So why do I keep doing storage? Well, firstly I have little choice but to stick to infrastructure; I’m a pretty lousy programmer and it seems that I can do less damage in infrastructure. If you ever received more cheque-books in the post from a certain retail bank, I can only apologise.

But storage is cool; firstly it’s BIG and EXPENSIVE; who doesn’t like raising orders for millions? It is also so much more than that place where you store your stuff; you have to get it back for starters. I think that people are beginning to realise that storage might be a little more complex than first thought; a few years ago , the average home user only really worried about how much disk that they had but the introduction of SSDs into the consumer market has hammered home how the type of storage matters and the impact it can have on the user experience.

Spinning rust platters keep getting bigger but for many, this just means that the amount of free-disk keeps increasing, the increase in speed is what people really want. Instant On..it changes things.

So even in the consumer market; storage is taking on a multi-dimensional personality; it scales both in capacity but also in speed. In the Enterprise; things are more interesting.

Capacity is obvious; how much space do you need? Performance? Well, performance is more complex and has more facets than most realise. Are you interested in IOPs? Are you interested in throughput? Are you interested in aggregate throughput or single stream? Are you dealing with large or small files? Large or small blocks? Random or sequential?

Now for 80% of use-cases; you can probably get away with taking a balanced approach and just allocating storage from a general purpose pool. But 20% of your applications are going to need something different and that is where it gets interesting.

Most of the time when I have conversations with application teams or vendors; when I ask the question as to what type of storage that they require, the answer comes back is generally fast. There then follows a conversation as to what fast means and whether the budget meets their desire to be fast.

If we move to ‘Software Defined Storage’, this could be a lot more complex than people think. Application developers may well have to really understand how their applications store data and how they interact with the infrastructure that they live on. If you pick the wrong pool,  your application performance could drop through the floor or the wrong availability level, you experience a massive outage.

So if you thought storage was snorage; most developers and people still do, you might want to start taking an interest. If infrastructure becomes code; I may need to get better a coding but some of you are going to have to get better at infrastructure. Move beyond fast and large and understand the subtleties; it is interesting…I promise you!

A New Job – for you maybe?

May 8th, 2013
by Martin Glassborow.
1 comment

Although it is no great secret as to who my employer is; I’m sure you can all use Google and LinkedIn, I very rarely post about the specifics of my job. It does influence what I write about obviously but often less than you might think.

This post however is entirely linked to my employer. I have a vacancy in my team and perhaps someone reading this might be interested in coming to work for me. The job advert is here and very good it is too; well, I wrote most of it. However perhaps you might want some more detail.

My storage team provides support and delivery expertise for storage in our Broadcast Technology group. A mixture of delivery, support, some design and general madness will make up your day. Tasks vary from the very mundane; from bar-coding tapes and loading libraries to placating users to the planning of installation of new storage devices to scripting to sitting through vendor powerpoints and trying not to laugh too hard; with all points in between as well.

We use a mixture of traditional storage and some more esoteric stuff; so you may find yourself  upgrading a NetApp filer one-day; adding clients to a TSM server the next; working with the server and network teams to work out why GPFS is misbehaving the next.

At times, you will find yourself on the bleeding edge and doing things that few have done before. Answering big questions like how you build a grow-forever archive and little questions as to how to restore from said archive.

You will get frustrated at both the pace of change but also at times, why does it take so long to adopt some technologies? But you will get to do some cool stuff and when you turn the TV on, you can smile and take pleasure in the fact that you helped put those pictures on the screen.

You can see the technologies I’m interested in in the advert. But more important is attitude and aptitude. You need to want to learn and you must bring a sense of fun and enjoyment with you.

If you’ve got any questions…ask below or work out my email address.

And NO AGENCIES please!

And just in case you missed it…click here

 

 

 

 

Doctors in the Clouds

Jan 30th, 2013
by Martin Glassborow.
No comments yet

At the recent London Cloud Camp; there was a lot of discussion about DevOps on the UnPanel; as the discussion went on, I was expecting the stage to be stormed by some of the older members in the audience. Certainly some of the tweets and the back-channel conversations which were going on were expressing some incredulity at some of the statements from the panel.

Then over beer and pizza; there were a few conversations about the subject and I had a great chat with Florian Otel who for a man who tries to position HP as a Cloud Company is actually a reasonable and sane guy (although he does have the slightly morose Scandinavian thing down pat but that might just be because he works for HP). The conversation batted around the subject a bit until I hit an analogy for DevOps that I liked and over the past twenty-four hours, I have knocked it around a bit more in my head. And although it doesn’t quite work, I can use it as the basis for an illustration.

Firstly, I am not anti-DevOps at all; the whole DevOps movement reminds me of when I was fresh-faced mainframe developer; we were expected to know an awful lot about our environment and infrastructure. We also tended to interact and configure our infrastructure with code; EXITS of many forms were part of our life.

DevOps however is never going to kill the IT department (note: when did the IT department become exclusively linked with IT Operations?) and you are always going to have specialists who are required to make and fix things.

So here goes; it is a very simple process to instantiate a human being really. The inputs are well known and it’s a repeatable process. This rather simple process however instantiates a complicated thing which can go wrong in many ways.

When it goes wrong, often the first port of call is your GP; they will poke and prod, ask questions and the good GP will listen and treat the person as a person. They will fix many problems and you go away happy and cured. But most GPs actually have only a rather superficial knowledge of everything that can go wrong; this is fine, as many problems are rather trivial. It is important however that the GP knows the limits of their knowledge and knows when to send the patient to a specialist.

The specialist is a rather different beast; what they generally see is a component that needs fixing; they often have lousy bedside manners and will do drastic things to get things working again. They know their domain really well and you really wouldn’t want to be without them. However to be honest, are they a really good investment? If a GP can treat 80% of the cases that they are faced with, why bother with the specialists? Because having people drop dead for something that could be treated is not especially acceptable.

As Cloud and Dynamic Infrastructures make it easier to throw up new systems with complicated interactions with other systems; unforeseeable consequences may become more frequent, your General Practitioner might be able to fix 80% of the problems with a magic white-pill or tweak here or there….but when your system is about to collapse in a heap, you might be quite thankful that you still have your component specialists who make it work again. Yes, they might be grumpy and miserable; their bedside manner might suck but you will be grateful that they are there.

Yes, they might work for your service provider but the IT Ops guys aren’t going away; in fact, you DevOps have taken away a lot of the drudgery of the Ops role. And when the phone rings, we know it is going to be something interesting and not just an ingrown toe-nail.

Of course the really good specialist also knows when the problem presented is not their speciality and pass it on. And then there is the IT Diagnostician; they are grumpy Vicodin addicts and really not very nice!

The Right Stuff?

Jan 15th, 2013
by Martin Glassborow.
2 comments

I must be doing something right or perhaps very wrong but the last few months have seen this blog pick-up a couple of ‘accolades’ that have left me feeling pretty chuffed.

Firstly Chris Mellor asked whether El Reg could carry my blog; as a long-term reader of Chris’ and of The Register, this made my year. To be picked up by the scurrilous El Reg is pretty cool.

And yesterday I got an email from EMC telling me that I had been voted to EMCElect! Now, that’s a pretty good start to this year.

This doesn’t mean that I’m going to go easy on EMC; I don’t think that’s what they want from me and if I did, El Reg wouldn’t want me either.

So I guess I’ll keep doing what I’m doing and hope you continue to enjoy it.

Scale-Out Fun For Everyone?

Dec 6th, 2012
by Martin Glassborow.
11 comments

Recently I’ve been playing with a new virtual appliance; well new to me in that I’ve only just got my hands on it. It’s one of the many that our friends in EMC have built and it is one which could do with a wider audience.

A few years ago Chad Sakac managed to make the Celerra virtual appliance available to one and all; a little sub-culture built up around it and many VMware labs have been built around it; when the Celerra and Clariion morphed into the VNX range, the virtual appliance followed. Nicholas Weaver further enhanced it and made it better and easier to use. It’s a great way for the amateur to play with an Enterprise class NAS and get some experience; I suspect it is also a great way for EMC to get community feedback and input on the usability and features in the VNX. A win/win as we like to say.

But EMC have another NAS product, one that I suspect over the long term will become the foundation of their NAS offerings; it is certainly important to their Big Data aspirations; yes, the Isilon range of Scale-Out NAS. I’d always suspected that there must be an appliance version kicking around; I mean anyone who has ever played with an Isilon box will have realised that it really is just an Intel server. You can order the SuperMicro motherboard which it is built on and pretty build your own if you wanted.

At a recent meeting, I was talking about the need for a training/test system for some of my guys to play on and lamenting that I probably could not justify the cost; our Isilon TC said ‘Why don’t I send you links to the Virtual Appliances?’

I bit his hand off and now I have a little virtual Scale-Out NAS to play with. It’s pretty much as easy to set up as the real thing without all the hassles of racking and stacking; I’ve got it running with 5 virtual nodes with a small amount of disk and can mess around with to my heart’s content.

I wish that you guys could also have a play but perhaps the guys from the Isilon team are bit nervous that we might do some silly things like put it into a production environment. I guess some of you might be that stupid but it didn’t stop them putting out the Celerra/Clariion version. So EMC can you give the community an early Christmas present and get the Isilon appliance out there.

Scale-Out NAS is going to be a really important growth sector; OneFS is a great product and it takes away a lot of the pain in building them and helps to demystify the whole thing.

At worst, a few geeks like me get to have some fun and you get some interesting feedback but I suspect you might find some people doing some interesting things with it and build a decent community.

And IBM, perhaps you could do the same and build a SONAS appliance and get that out as well?

I’d love to see EMC make the Enginuity appliance generally available but that does have stupid memory and CPU requirements, so I’m not holding my breath for that….

New Lab Box: HP ML110 G7

Jun 12th, 2012
by Martin Glassborow.
6 comments

I keep meaning to do a blog post on the Bod home-office, something like the Lifehacker Workspace posts but I never quite get round to it. Still, suffice to say, my home workspace is pretty nice; it’s kind of the room I wanted when I was thirteen but cooler! The heart of the workspace though is the tech, I have tech for gaming, working, chilling and generally geeking; desktops of every flavour and a few servers for good measure.

Recently, as you will know I have been playing with Razor and Puppet and I found that my little HP Microserver was struggling a bit with the load I was putting on it and I started to think about getting something with a bit more oomph. I had decided that I was going to put something together based on Intel’s Xeon technology and began to put together a shopping list.

Building PCs is something I kind of enjoy but then as luck would have it, an email dropped into my mailbox from Servers Plus in the UK offering £150 cashback on the HP ProLiant ML110 G7 Tower Server with a E3-1220 Quad Core processor; this brought the price down to £240 including VAT. And I was sold..no PC building for me this time.

As well as the afore mentioned E3-1220, the G7 comes equipped with a 2Gb ECC UDIMM, 2x Gigabit Ethernet ports, iL03 sharing the first Ethernet, 250GB Hard Disk, 350W power supply and generally great build quality (although I could do a better job with the cable routing I reckon).

The motherboard can support up to six SATA devices and there are four non-hotswap caddies for no-screw hard-disk installation, one of which holds the 250 GB Hard Disk. Installing additional drives was a doddle and involved no cursing and hunting for SATA cables. I did not bother to install an optical disk as I intended to network boot and install from my Razor server.

Maximum supported memory is an odd 16GB; the chipset definitely supports 32GB but there are very mixed reports of running an ML110 G7 with 32GB; I just purchased a couple of generic 4 GB ECC DIMMS for about £50 to bring it up to 10 GB for the time being. I’d be interested in hearing if anyone has got a ML110 G7 running successfully with 32GB. There’s no technical reason for HP to limit the capability and it does seem strange. The DIMM slots are easily accessible and no contortions are required to install the additional memory.

There are 4 PCIe card slots available; 1×16, 2×4 and 1×1; this should be ample for most home servers as it already comes with two onboard ethernet ports.

After installing the additional memory and hard-disks, I powered the box up and let it register with my Razor server; added a policy to install ESXi on it and let it go.

A quick note about the iLO3, it is the basic license which allows you to power-up and power-off, do some basic health checking and monitoring but no remote terminal; this is not a huge problem for me as the server is in the same room and I can easily put a monitor on it if required.

The ML110 is pretty damn quiet considering the amount of fans in it but start putting under load and you will know it’s there but it’s no noisier than my desktop when I am gaming and all the fans are spinning. It is certainly noisier than the Microserver though.

Once ESXi was installed; bringing up the vSphere let me see that all the components are recognised as expected; all the temperature monitors and fans were also being seen. Power management is available and can be set to Low power if you want for your home lab.

So I would say that if you want a home lab box with a little more oomph than the HP Microserver; the ML110 G7, especially with the £150 cash-back takes some beating. If it could be upgraded all the way to 32GB, then it would be awesome.

 

 

 

Razor – An Idiot’s Guide to Getting Started!

Jun 2nd, 2012
by Martin Glassborow.
1 comment

My role at work does not really allow me to play with tech very often and unless I have a real target in mind, I’m not as much as an inveterate hacker as I used to be; I generally find a problem, fix a problem and then leave it alone until solution is broken. So I don’t tend to go into any depth on anything but every now and then, I see something and think that’s cool, can I get it to work.

When I saw the new tool from Nicholas Weaver and EMC built on Puppet to configure ‘bare-metal’ machines; I decided that was something cool enough to have a play with. But there were a few problems, I knew what Puppet is but I’d never used it and I really didn’t have a clue what I was doing.

Still, I followed the links to the documentation and started to hack; after a couple of failed attempts due to missing prerequisites, I decided to be a bit more methodical and document what I was doing.

So this is what I did…more or less. And hopefully this might be helpful to someone else. I am assuming a certain amount of capability tho’! So more an Idiot Savant than just an Idiot.

I have a number of core services already running at home; I have my own internal DNS and DHCP server running on Ubuntu, I have a couple of NAS servers and a couple of machines running ESX5i.

All the documentation for Razor is based around Ubuntu Precise Pangolin 12.04; so first thing to do was to build a Precise VM on one of the ESX5i boxes. This was provisioned with 2 Gigs and 2 virtual cores.

1) Install Ubuntu 12.04 Server and I always do an OpenSSH Server build at installation; I leave everything else to after I’ve done a base install.

2) I added a static assignment for the server in my DHCP box and created a DNS entry for it.

3) After the server was installed, I did my normal ‘break all the security’ and used sudo to set a root password and allow myself to log directly on as root. I’m at home and can’t be bothered to use sudo for everything.

4) I started installing packages, I’m not sure whether the order matters but this the order I did things and all this was done as root

EDIT:According to Nick and he should know, the Razor module installs Node and Mongo-db automagically…I had some problems the first couple of times and decided to do it myself, this is possibly because I’m an extremely clever idiot and break idiot proof processes.

Node.JS

apt-get install nodejs npm

MongoDB

I didn’t use the standard packaged version and pulled down the package from mongodb.org

apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
vi /etc/apt/sources.list
Added deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen
apt-get update
apt-get install mongodb-10gen

Puppet

Yet again I didn’t use the standard packaged version and pulled down from puppetlabs.com

wget http://apt.puppetlabs.com/puppetlabs-release_1.0-3_all.deb
dpkg -i puppetlabs-release_1.0-3_all.deb
apt-get update
apt-get install puppetmaster

Ruby 1.9.3

Note that the above install does install Ruby but appears to bring down Ruby 1.8; Razor wants a later version.

apt-get install ruby1.9.3

This seems to do what you want!

At this point you should be in the position to start installing Razor.

Razor

This is very much cribbed from the Puppet Labs page here

puppet module install puppetlabs-razor

chown -R puppet:puppet /etc/puppet/modules

puppet apply /etc/puppet/modules/razor/tests/init.pp --verbose

This should run cleanly and at this stage you should have some confidence that Razor will probably work.

/opt/razor/bin/razor

This shows that it does work.

/opt/razor/bin/razor_daemon.rb start

/opt/razor/bin/razor_daemon.rb status

This starts the razor daemon and confirms it is running. Our friends at PuppetLabs forgot to tell you start the daemon, it’s kind of obvious I guess but made this idiot pause for a few minutes.

Configuring Your DHCP Server

I run my own DHCP server based and this needed to be configured to point netbooting servers at the Puppet/Razor server.

I amended the /etc/dhcp/dhcp.conf file and added the following

filename "pxelinux.0";
next-server ${tftp_server_ipaddress};

in the subnet declaration.

At this point, you should be ready to really start motoring and it should be plain sailing I hope. Certainly this idiot managed to follow the rest of the instructions for Example Usage on Puppetlabs.

Of course now I’ve got lots of reading to do around Puppet and the likes but at the moment, it does appear to work.

So great work Nick and everyone at EMC.

Flash Changed My Life

May 18th, 2012
by Martin Glassborow.
2 comments

All the noise about all flash arrays and acquisitions set me thinking a bit about SSDs and flash; how it has changed things for me.

To be honest, the flash discussions haven’t yet really impinged on my reality in my day-to-day job, we do have the odd discussion about moving metadata onto flash but we don’t need it quite yet; most of the stuff we do is sequential large I/O and spinning rust is mostly adequate. Streaming rust i.e tape is actually adequate for a great proportion of our workload. But we keep a watching-eye on the market and where the various manufacturers are going with flash.

But flash has made a big difference to the way that I use my personal machines and if I was going to deploy flash in a way that would make the largest material difference to my user-base, I would probably put it in their desktops.

Firstly, I now turn my desktop off; I never used to unless I really had to but waiting for it to boot or even awake from sleep was at times painful. And sleep had a habit of not sleeping or flipping out on a restart; turning the damn thing off is much better. This has had the consequence that I now have my desktops on an eco-plug which turns off all the peripherals as well; good for the planet and good for my bills.

Secondly, the fact that the SSD is smaller means that I keep less crap on it and am a bit more sensible about what I install. Much of my data is now stored on the home NAS environment which means I am reducing the number of copies I hold; I find myself storing less data. There is another contributing factor; fast Internet access means that I tend to keep less stuff backed-up and can stream a lot from the Cloud.

Although the SSD is smaller and probably needs a little more disciplined house-keeping; running a full virus check which I do on occasion is a damn sight quicker and there are no more lengthy defrags to worry about.

Thirdly, applications load a lot faster; although my desktop has lots of ‘chuff’ and can cope with lots of applications open, I am more disciplined about not keeping applications open because their loading times are that much shorter. This helps keeping my system running snappily, as does shutting down nightly I guess.

I often find on my non-SSD work laptop that I have stupid numbers of documents open; some have been open for days and even weeks. This never happens on my desktop.

So all-in-all; I think if you really want bang-for-buck and to put smiles on many of your users’ faces; the first thing you’ll do is flash-enable the stuff that they do everyday.

 

 

Your Life, Their Product

Apr 10th, 2012
by Martin Glassborow.
1 comment

So whilst the UK was recovering from over-indulging in chocolate eggs; across the Atlantic, Facebook were splashing out $1 Billion on Instagram. And still the world continued to spin and orbit the Sun. So what does this mean to us all; there will be a lot of soul searching and discussion but ultimately this just continues to productise your life and your experience.

I watched the Google’s Project Glass video prior to the Facebook announcement and was thinking if Google were to buy someone like Instagram; the anonymity of the crowd has gone, the glasses could identify the person you were looking at immediately. Of course Facebook could do the same thing and create their own Project Social Glass. You will no longer be able to sit in coffee shop quietly unrecognised, you would be instantly identifiable. Would you be entirely comfortable with that? I know I won’t be.

There are times in our lives where we just want to want be alone and not identified; to remove that opportunity and to have that constant feeling that you are being watched will change our natures. In our allowing of our lives to productised, we may lose something which is essential to our well-being; Facebook is arguably already removing the right to make mistakes and the ability to forget.

Could it remove the right to be anonymous? Are we heading towards the perfect storm which shatters our illusions of privacy? For even if it is an illusion, it is an important one.

We have to be very careful as to where this road takes us.

Of course Facebook could have just spent $1 Billion on an app to make crap photos look like they were taken 40 years ago.

 

Do you need a desktop?

Apr 3rd, 2012
by Martin Glassborow.
3 comments

Work provide me with a laptop which spends most of its time locked to my desk. It’s quite a nice business laptop but really I can’t be bothered to carry it around. On occasion, when I’m working from home and realise that I am going to need access to some of corporate applications which require VPN access, it’ll come home with me but mostly not.

To be quite honest, even my MBA doesn’t travel that much, up and down the stairs is about as far as it goes. It is quite the nicest and most practical laptop that I’ve ever owned but I think we are getting close to the stage where a tablet can do almost everything that I need where-ever I am.

I was thinking as I was working today whether what I was doing required the traditional desktop experience and could I simply use my iPad as the access device instead. The answer is mostly yes, almost all the applications that I use are generic enough that there are good enough replacements on the iPad or they are accessed by a web interface anyway.

There are a few blockers tho’ at present

1) at present I can’t get my iPad onto the corporate wireless, this means that I can’t access a number of key applications due to ‘security’ restrictions but I can access email which appears to be our preferred file delivery/transfer mechanism.

2) I need a real keyboard to type on, there is a limit to how much I am prepared to type on a screen keyboard. I could overcome this relatively easily by bringing a bluetooth keyboard in.

3) Wired Ethernet is a necessity when working in some of our data centres or secure areas.

4) Unfortunately, I struggle without PowerPoint and Visio unfortunately; I can cope without Word, Excel is a little more problematic but it’s manageable. Keynote is nice but it makes a real mess of rendering PowerPoint in my experience.

5) Working on an external display is often a much nicer experience than using the tablet screen, even tho’ the retina display is the wonderful. But I have both the HDMI and VGA dongles which gets round this. But I wish that Apple could find a way to put a mini-DisplayPort on the iPad as using the adapters means that I loose any chance of using a USB device. Not important most of the time but very useful for transferring files from cameras and other devices.

But then I started thinking some more, perhaps I don’t really need a tablet either for work. Perhaps a smartphone which I dock would do? What we could do with is a standard dock for all mobile devices which charges, displays on an external screen and allows input from a standard keyboard/mouse.

Planes, trains, hotels and the like could simply provide a dock and you would end up carrying even less. At that point a device the size of a Samsung Note or Kindle Fire becomes a very interesting proposition.

And yet, I still expect to keep my PC desktop for some time….why? It’s still the best serious gaming platform out there. But for almost everything else I could probably manage with a mobile device.

← Earlier Posts