At a SNW briefing session today, X-IO (Xiotech) talked a lot of sense about Big Data; in fact it was almost the most sense that I have heard spoken about Big Data in a long time. The fact is that most Big Data isn’t really that big and the data-sets are not huge; there are exceptions but most big data-sets that many companies will use can be measured in a few terabytes and not the tens or hundreds of terabytes that the big storage vendors want to talk about.
Sentiment data which can derived from social networking, these are not necessarily big data sets. A tweet for example is 140 characters, so 140 bytes…a terabyte is 1 099 511 627 776 bytes; we can store a lot of tweets in a terabyte and within that data, there is a lot of information that can be extracted.
In fact, there are probably some Big Answers in that not so Big Data but we need to get rid of the noise; in order to do this, we need to be able to process this data differently and directly. The most important thing that the storage can do is to vanish and become invisible; allow data processing to be carried out in the most natural way and not require various work-arounds which hide the deficiencies of the storage.
If your storage vendor spends all their time talking about the bigness of data; then perhaps they might be the wrong vendor.