Storagebod Rotating Header Image

General IT Management

Service Power..

Getting IT departments to start thinking like service providers is an up-hill struggle; getting beyond cost to value seems to be a leap too far for many. I wonder if it is a psychological thing driven by fear of change but also a fear of assessing value.

How do you assess the value of a service; well, arguably, it is quite is simple…it is worth whatever someone is willing to pay for it. And with the increase prevalence of service providers vying with internal IT departments; it should be relatively simple. They’ve pretty much set the base-line.

And then there are the things that the internal IT department just should be able to do better; they should be able to assess Business need better than external. They should know the Business and be listening to the ‘water cooler’ conversations.

They should become experts in what their company does; understand the frustrations and come up with ways of doing things better.

Yet there is often a fear of presenting the Business with innovative and better services. I think it is a fear of going to the Business and presenting a costed solution; there is a fear of asking for money. And there is certainly a fear of Finance but present the costs to the Business users first and get them to come to the table with you.

So we offer the same old services and wonder why the Business are going elsewhere to do the innovative stuff and while they are at it; they start procuring the services we used to provide. Quite frankly, many Corporate IT departments are in a death spiral; trying to hang-on to things that they could let go.

Don’t think I can’t ask the Business for this much money to provide this new service…think, what if the Business want this service and ask someone else? At least you are going to be bidding on your own terms and not being forced into a competitive bid against an external service provide; when it comes down to it, the external provider almost certainly employees a better sales-team than you.

By proposing new services yourself or perhaps even taking existing ‘products’ and turning them into a service; you are choosing the battle-ground yourselves…you can find the high ground and fight from a position of power.

5 Minutes

One of the frustrations when dealing with vendors is actually getting real availability figures for their kit; you will get generalisation,s like it is designed to be 99.999% available or perhaps 99.9999% available. But what do those figures really mean to you and how significant are they?

Well, 99.999% available equates to a bit over 5 minutes of downtime and 99.9999% equates to a bit over 30 seconds downtime over a year. And in the scheme of things, that sounds pretty good.

However, these are design criteria and aims; what are the real world figures? Vendors, you will find are very coy about this; in fact, every presentation I have had with regards to availability are under very strict NDA and sometimes not even notes are allowed to be taken. Presentations are never allowed to be taken away.

Yet, there’s a funny thing….I’ve never known a presentation where the design criteria are not met or even significantly exceeded. So why are the vendors so coy about their figures? I have never been entirely sure; it may be that their ‘mid-range’ arrays display very similar real world availability figures to their more ‘Enterprise’ arrays…or it might be that once you have real world availability figures, you might start ask some harder questions.

Sample size; raw availability figures are not especially useful if you don’t know the sample size. Availability figures are almost always quoted as an average and unless you’ve got a real bad design; more arrays can skew figures.

Sample characteristics; I’ve known vendors when backed into a corner to provide figures do some really sneaky things; for example, they may provide figures for a specific model and software release. This is often done to hide a bad release for example. You should always try to ask for the figures for the entire life of a product; this will allow you to judge the quality of the code. If possible as for a breakdown on a month-by-month basis annotated with the code release schedule.

There are many tricks that vendors try to pull to hide causes of downtime and non-availability but instead of focusing on the availability figures; as a customer, it is sometimes better to ask different specific questions.

What is the longest outage that you have suffered on one of your arrays? What was the root cause? How much data loss was sustained? Did the customer have to invoke disaster recovery or any recovery procedures? What is the average length of outage on an array that has gone down?

Do not believe a vendor when they tell you that they don’t have these figures and information closely and easily to hand. They do and if they don’t; they are pretty negligent about their QC and analytics. Surely they don’t just use all their Big Data capability to crunch marketing stats? Scrub that, they probably do.

Another nasty thing that vendors are in the habit of doing is forcing customers to not disclose to other customers that they have had issues and what they were. And of course we all comply and never discuss such things.

So 5 minutes…it’s about long enough to ask some awkward questions.

Patience is a Virtue?

Or is patience just an acceptance of latency and friction? A criticism oft made of today’s generation is that they expect everything now and this is a bad thing but is it really?

If a bottle of fine wine could mature in an instant and be good as a ’61; would this be a bad thing? If you could produce a Michelin quality meal in a microwave, would it be a bad thing?

Yes, today we do have to accept that such things take time but is it really a virtue? Is there anything wrong with aspiring to do things quicker whilst maintaining quality?

We should not just accept that latency and friction in process is inevitable; we should work to try to remove them from the way that we work.

For example, change management is considered to be a necessary ITIL process but does it have to be the lengthy bureaucratic process that it is? If your infrastructure is dynamic, surely your change process should be dynamic too? If you are installing a new server, should you have to raise a change

1) to rack and stack
2) to configure the network
3) to install the operating system
4) to present the storage
5) to add the new server to the monitoring solution etc, etc

Each of these being an individual change being raised by separate teams. Or should you be able to do this all programmatically? Now obviously in a traditional data-centre, some of these require physical work but once the server has been physically commissioned, there is nothing there which should not be able to be done programmatically and pretty much automatically.

And so it goes for many of the traditional IT processes; they simply introduce friction and latency to reduce the risk of the IT department smacking into a wall. This is often deeply resented by the Business who simply want to get their services up and running, it is also resented by the people who are following the processes and then it is thrown away in an emergency (which happens more often than you would possibly expect ;-) ).

This is not a rant against ITIL, it was tool for a more sedate time but in a time when Patience is no longer really a virtue..do we need a better way. Or perhaps something like an IT Infrastructure API?

Don’t throw away the rule-book but replace it with something better.

p.s Patience was actually my grand-mother; she had her vices but we loved her very much.

Flash Changed My Life

All the noise about all flash arrays and acquisitions set me thinking a bit about SSDs and flash; how it has changed things for me.

To be honest, the flash discussions haven’t yet really impinged on my reality in my day-to-day job, we do have the odd discussion about moving metadata onto flash but we don’t need it quite yet; most of the stuff we do is sequential large I/O and spinning rust is mostly adequate. Streaming rust i.e tape is actually adequate for a great proportion of our workload. But we keep a watching-eye on the market and where the various manufacturers are going with flash.

But flash has made a big difference to the way that I use my personal machines and if I was going to deploy flash in a way that would make the largest material difference to my user-base, I would probably put it in their desktops.

Firstly, I now turn my desktop off; I never used to unless I really had to but waiting for it to boot or even awake from sleep was at times painful. And sleep had a habit of not sleeping or flipping out on a restart; turning the damn thing off is much better. This has had the consequence that I now have my desktops on an eco-plug which turns off all the peripherals as well; good for the planet and good for my bills.

Secondly, the fact that the SSD is smaller means that I keep less crap on it and am a bit more sensible about what I install. Much of my data is now stored on the home NAS environment which means I am reducing the number of copies I hold; I find myself storing less data. There is another contributing factor; fast Internet access means that I tend to keep less stuff backed-up and can stream a lot from the Cloud.

Although the SSD is smaller and probably needs a little more disciplined house-keeping; running a full virus check which I do on occasion is a damn sight quicker and there are no more lengthy defrags to worry about.

Thirdly, applications load a lot faster; although my desktop has lots of ‘chuff’ and can cope with lots of applications open, I am more disciplined about not keeping applications open because their loading times are that much shorter. This helps keeping my system running snappily, as does shutting down nightly I guess.

I often find on my non-SSD work laptop that I have stupid numbers of documents open; some have been open for days and even weeks. This never happens on my desktop.

So all-in-all; I think if you really want bang-for-buck and to put smiles on many of your users’ faces; the first thing you’ll do is flash-enable the stuff that they do everyday.

 

 

No Pain, No Gain?

I always reserve my right to change my mind and I am almost at the stage that I have changed my mind on blocks/stacks or whatever you want to call them? And for non-technical and non-TCO related reasons.

I think in general componentised and commodity-based stacks make huge sense; whether you are building out private or a public infrastructure; a building block approach is the only really scalable and sustainable approach. And I wrote internal design documents detailing this approach eight or nine years ago; I know I’m not the only one and we didn’t call it cloud…we called it governance and sensible.

But where I have changed my opinions is on the pre-integrated vendor stacks; I think that they are an expensive way of achieving a standardised approach to deploying infrastructure and I have not changed from this opinion.

However I think that this cost may well be the important catalyst for change; if you can convince a CFO/CEO/CIO/CTO etc that this cost is actually an investment but to see a return on the investment that you need to re-organise and change the culture of IT, it might well worth be paying.

If you can convince them that without the cultural change, they will fail….you might have done us all a favour. If it doesn’t hurt, it probably won’t work. If it is too easy to write things off when it’s tough…it’ll be too easy to fall back into the rut.

So EMC, VCE, Cisco, IBM, NetApp, HP etc….make it eye-wateringly expensive but very compelling please. Of course, once we’ve made the hard yards, we reserve the right to go and do the infrastructure right and cheap as well.

Archicultural….

It seems the more that I consider the architectural and technical challenges and changes to the Corporate IT world, the more I come back to the cultural issues which exist within many IT departments and the more I find myself feeling strongly that this is where the work really needs to be done.

Unfortunately it is pretty hard to buy a culture from a vendor, even though I suspect if Chuck could work out exactly how to do so; we’d have a product from EMC called V-CLT (or is that VMware?); so building a culture is going to be have to be an internal thing and that means it is going to be tough.

Too often the route into IT Management means either promoting excellent techies into management or sometimes promoting people into positions where they can do no more harm as opposed to moving people into positions which suits them and their personalities. I am sure that we can all think of examples of both; this is especially true in end-user organisations as the career paths are less varied than that of the vendor organisation. Vendor organisations have sales, marketing and other avenues for progression; they also have the traditional IT paths as well.

But all IT organisations are suffering from cultures which neither scale or are sustainable in the long term. There needs to be a long term shift which ensure that training and development are in more than just technical skills; there needs to be a move away from a hero culture that sees staff at all levels of an organisation regularly halving their hourly rates by working longer than their contracted hours, not taking leave and forgetting that you ‘Work to Live’.

Careers need to be thought of more than the fastest route to the top and when people find their natural level; this does not mean that they do not stop being valuable members of an organisation. Work on developing people horizontally (and you with the dirty mind can stop sniggering); I think that there is something relatively unhealthy when you find managers who have worked their way up through a team and only worked in one team.  Horizontal moves have immense value; I have learnt such a lot in the past couple of years running a test team as well as a storage team.

Horizontal moves will help to break down some of the siloed mentality; even if you do not believe in DevOps, moving people between these two disciplines even on secondment must have value.

If you have a graduate scheme in place, the natural roles that most graduates gravitate to are in development; make sure that they have a placement in an Operations/Infrastructure team. They will learn so much.

And if you work in management; you are doing a pretty hard job, make it easier on yourself by standing on the shoulders of giants and actually study the art of management and leadership. Most get to management by being good at something; being good at that something does not mean you know anything about management.

Thinking Architecturally

If you start an architecture with a shopping list of technologies that must be used; that architecture will be compromised. However this does not mean that you start working without an appreciation of the possible, obviously you need to be aware of limitations such as constants such as the speed of light and other real constraints.

But currently I see a trend from many, both vendors and users, trying to fix round-hole problems with square-shaped blocks. Not enough time is spent on the problem definition and truly understanding the problem; your existing tools may not be sufficient and although it may feel that it is more expensive to implement something new, at times it might be cheaper in the long-term to implement something right.

Also be aware of falling into the trap of implementing a feature just because you’ve made the mistake of purchasing something that does not fit your problem definition. If you’ve been sold something that you can’t use effectively, you have a couple of option; suck it up and learn from experience or shout and holler at your vendor/partner for selling you something which is merely shelf-ware. In my experience, the latter is often ultimately pointless and simply results in the vendor promising you some other product which you put on a shelf and not use. Use the experience to move away from architecting to utilise a feature and architecting to solve a problem.

This does not mean that you simply purchase a new system/technology for every problem; governance has a role but I would suggest that governance should be applied after the initial high-level-architecture. I like to think of it like more more traditional bricks and mortar architecture; the architect relies on a whole bunch of technical people to fulfil their vision and bring it to reality. At times these technical people will tell the architect that the architect is a complete moron; sometimes the architect will agree and sometimes the architect will work with the technical teams to come up with something innovative and new.

But in general the architect does not start their design with a specific make of brick in mind. Neither should an IT architect.

The Steve Ratio

I’ve seen a few blogs recently about management and especially man management in IT but before I begin, I’ll explain the title! In my career, I have had four managers called Steve; two have been good, one indifferent and one who words cannot describe, not when I know that are some ladies who read this. And actually, the ratio of 2:1:1 is pretty much representative of the managers I have had so far.

So what makes a good manager? Well for me, the most important quality of the two good Steves was that they allowed me to make decisions that I was comfortable with and when I made a mistake, they would sit down with me and work through the mistake and then let me fix it. Let’s break that down

1) They trusted me….good managers trust!

2) They let me make mistakes….good managers do not second guess and do not blame!

3) They gave me guidance but not the answers! Good managers show the way but don’t carry you!

4) They trusted me….good managers don’t let mistakes destroy trust!

Now with my current numbers of direct reports, around thirty (I loose count); I have no choice but to follow those principles and do any more would rapidly hasten my descent into insanity but I owe those two Steves a great debt.

For me, man management is the great under-rated skill in IT; they aren’t always the most senior people and sometimes they aren’t the greatest techies but they ensure your organisation works and that you can do your job.

It is ironic that so many people say that the road to Cloud requires a cultural change; perhaps this should also include a change in attitude to man management. Perhaps we can change the Steve Ratio and make it 3:1:0; no arse-holes and a majority of good managers.

Reflections on a Professional Future

Yay, go me…I’ve survived another year on this rock; although I’ve noticed my first gray hair which is little concerning. My birthday and other anniversaries around this time of year always lead me feeling a little reflective and thoughtful; this time, I am thinking about our profession and it’s future. I’ve wanted to work in IT pretty much as far back as I can remember and apart from a long-held ambition to be Indiana Jones and be an archaeologist uncovering lost cities and mysterious temples; I still can’t really think of any other profession that I’d be in.

Yet I am concerned about this profession and the direction it is taking; no I am not talking about the technical direction it is taking but more what it is doing to it’s people. This may be simply a factor to do with the economy and that we are also in the throes of a ‘transformation’ but this could be something more concerning.

I am seeing increasing numbers of people surfing the edge of burn-out; both in end-users and vendors. 60-80 hour weeks and more becoming the norm; not the exception and demands on people’s time increasing all the time. The mobile revolution leaving people on call all the time; the liberating impact of mobility being lost as the contactability at all times becomes a ball-and-chain.

Yet all the research suggests that most people are generally most effective when working something between 35-45 hours a week; we pride ourselves on being a smart bunch in IT, yet we seem to ignore both common sense and scientific research.

And if we factor in the high degrees of introversion in our profession, we are tending to force people to be always on when they need to be given time and solitude to recharge. Meeting after meeting, interaction after interaction, chewing away at the spirit of some of our most talented people.

The best people love this profession and they are an odd bunch; how many surgeons take patients home to operate on? How many dentists build home surgeries to try out the latest techniques? How many teachers? How many accountants? Lawyers? Very, very few! But amongst our fellows, it’s relatively common…I would argue that even those who do manage to stick to their official working hours, normally put in another day’s work in keeping themselves updated.

Yet as a profession, we pride ourselves on enabling our users to work smarter and better; our output makes people lives better and richer. Perhaps we should think about how we do the same for ourselves?

After all

Young man, as you perambulate down the pathway of life toward an unavoidable bald head bordered with gray hairs it would be well to bear in mind that the cemeteries are full of men this world could not get along without, and note the fact that things move along after each funeral procession at about the same gait they went before. It makes no difference how important you may be, don’t get the idea under your hat that this world can’t get along without you —Abilene Reporter, 1909.

 

Empower the Cloud

As various organisations continue their journey to the Cloud, I wonder how many will be truly successful and how many will simply Cloud-wash their existing IT. As vendors continue to wash their products and make them all fluffy and cloud-like, so I suspect will many end-user organisations. And in that, they will miss many of the benefits.

But what will guarantee a successful Cloud deployment? Well, if you start with infrastructure, you will probably seriously reduce your chances of a successful deployment. You have to start with the organisational culture, this is something that many pundits agree with but there is not a lot of real discussion about what culture needs to be in place.

I think fundamentally, Cloud is about empowerment; an empowered work-force will get the most out of Cloud and without an empowered organisation, Cloud will fail and dissipate on the old hierarchies.

Cloud empowers users to make decisions and more importantly make mistakes; fail early, fail often but fail cheaply. A huge investment does not need to be made in putting together an infrastructure to support an initiative; it can be quickly brought-up and then ripped-back down again without blame and significant cost. You can run innovative projects ‘lean’.

Cloud empowers IT to put in place standards which are not seen as restrictive as they become almost invisible to the users. A change in hardware should not necessitate a huge round of regression testing; hopefully we will get to the stage where a change in platform full-stop will not mean a massive amount of rework and regression.

Cloud should empower your IT teams to work cross-discipline and cross boundaries but if your teams are already siloed and not able to talk to each other, then ask yourself why? Are you so focussed on internal hierarchies and control that you are preventing this? You need to empower your teams to have peer-level conversations and decision-making ability.

Often when people talk about the tools required for Empowerment, they talk about ensuring that people have the right information to make good decisions; in many organisations, information is still hung onto; knowledge is power and many like to hoard power.

If your organisation already has a culture of knowledge sharing, then moving your knowledge infrastructure to Cloud is a logical step but if you have a culture of controlled ignorance, then the journey is going to be hard.