The jury is probably still out on the real value of ‘Big Data’ and what it will mean to our lives; whether it is a power for good or ill or even if it is a power for anything is probably still up for debate. But there is one thing which is probably true, ‘Big Data’ will change data-processing for the better.
At present, you will find that the prevailing wisdom is that if you have Data to store, you should store it in a relational database but the ‘new’ data processing techniques which ‘Big Data’ brings to the party changes this or at least seriously questions this wisdom.
I know many applications that currently store their data into relational databases that could possibly benefit from a change of focus; these are often log-oriented applications which are only using one or two tables to store their Data and often the indexes to enable fast processing are larger than the data stored.
So even if you have no ‘Big Data’, you may find that you have more candidates than you realise for ‘Big Data’ processing techniques….and I suspect this is what really scares our friends at Oracle. For too long now, serious Data processing required serious relational databases and that road took us into the realms of Oracle; increasing costs and infrastructure complexity.
The problem is that re-writes show little immediate business value and the investment will take two or three years to pay-off; it is this that your RDMS account manager is counting on. Yet as soon as you start to factor in maintenance, upgrade and recurring costs; this should be an economic no-brainer for the IT Manager with foresight.
Interesting commentary on big data. To what extent do you think technologies such as compression (to minimize capacity) and the use of SSD flash for accelerating index look ups help? There are some storage solutions that exist today which can offer compression (at 2x in some cases) without sacrificing database IO transaction performance.
Jeff,
they certainly help legacy but it still doesn’t fix the fact that many databases are probably architected using technology which is not entirely fit for purpose. Please note, I’m not saying rip-out all your RDMS and replace with HADOOP type technologies..you choose the best tool for the job and don’t try to ram round pegs into square holes.
While I understand your points, I don’t think Oracle is scared. In fact it offers Bigdata appliance and is trying to make a dent in the market. They have built tools which migrate from structured data to unstructured data support.
http://www.oracle.com/us/products/database/big-data-appliance/overview/index.html
When you say they? Do you mean we?
And Oracle not scared? They are certainly concerned, Cloud and Big Data has caught them on the hop. They aren’t about to die any time soon but anything which impacts Larry’s bank-balance concerns him. For a long time, Oracle only really had IBM to worry about impacting their core money-maker…now there’s lots of irritating little companies nibbling.