Posts

Aspera with Object Storage

Many of us are using a cloud storage solution such Dropbox, OneDrive, Google Drive and so on. But are does solutions good enough for your business? Are your management unsecure if it is secure enough or want to keep better control because of legal reason or any reason. There is many solutions out there that can fit for your, and if you build the solution on an Object Storage base, you will then see even more possibility to secure your data both from a backup and restore perspective and also to use the better technology in the right place. During this week have I got time to play with Aspera and Spectrum Scale together and see how it works. And to summarizing my conclusion with only one sentence.  “Wow, LOL, it’s so much fun and great combinations I can do”. To just start with is to be able to send data much faster than regular TCP, in our lab do we got a 100Mbps Internet access and we normally get 2 MB/s when we upload to United States and I think because of ...

Software Define Storage for Dockers

Image
I was reading a little bit about Dockers and how Dockers works and why it is so popular. The most interesting is stories I found about Dockers, was how it sharing information between the containers and how one container almost could have almost 0 byte of information accept metadata. When you deploy that container it will automatic pull in data from all other containers where the actual data are located. Almost like BitTorrent but internally in the Dockers application. My thoughts did then direct start thinking of how that will impact the storage system. Will it generate a lot of IOPS? How should I scale my storage system if it does? Can I use cheap NL-SAS storage for this? Can I use Network Attach Storage, NAS? Can I use any Scale-Out solution? A lot of questions comes up and as more I read about Dockers, I can’t find any information about this from HP, EMC, IBM, Hitachi or any other storage vendor. No one explains how to solve this. Why? Is this not an issue? ...

Flash vs AllFlash

Image
During 2015 have probably both you and I been well informed about Flash storage, and that will solve all your problems. Especially if you have been listen to EMC, IBM, HP and a bunch of smaller unique vendors. But not only them have talking a lot of Flash Storage, if you also look at ESG, Gartner, Radar and many more analytic companies. Just searching on the web on "Why Flash Storage" do we get over +55 million answers. What is the different between Flash vs AllFlash that a few are calling it. There is a YouTube video that explains very simple Flash vs Disk and what flash really does for you. I'm trying to copycat that video but via text and pictures instead. On top of this do you also have, what is the different between Flash vs Flash, let's say what is the different between IBM FlashSystem vs Violin? Or what is the different between EMC ExtremIO, HP 3Par AllFlash array and IBM Storwize All-Flash? I'm not going in to the different between does ...

Upgrade GPFS 3.4 to Spectrum Scale 4.1.1

It's funny because it's true…. That´s how I want to start with today's story… And today will I be more technical then normally. Let's start from beginning before I complain too much and showing how easy it is. I have now heard twice where end-users of GPFS heard it isn't possible to upgrade from GPFS 3.4 filesystem to Spectrum Scale 4.x without to shut down the entire cluster. This is not true and I will now show it for you how easy it is and how it's works.  I want with today's topic explain how easy it is to upgrade a GPFS 3.4 filesystem to Spectrum Scale 4.1.1 without to shut down the cluster. This how my test system looks like and see the time stamp when I start with my upgrade. [root@gpfs1 gpfs]# date Tue Jul 28 14:52:21 CEST 2015 [root@gpfs1 src]# mmlscluster GPFS cluster information ========================   GPFS cluster name:         gpfstest.cristie.se   GPFS cluster ...

More flexible Hadoop Cluster

Image
I had an  interesting   scenario  where the customer had a Hadoop Cluster based on HortonWorks. They had multi PB large HDFS cluster that have done a lot of great work for them, but the amount of data they pushed in daily in to the cluster and how much hardware they needed to buy to fit all data, was not cost effective for them. They run daily batch jobs and not real-time analytic, that mean they only need a sort of amount of CPU and memory capacity to be able to process all the new data that comes in. But they keep the old data to be able to get much better analytic result. When you read a standard Hadoop design are you recommended to run local disk on a server and then run multiple copies of each file to be able to secure and get a better performance then what a single SATA disk can produce. But this recommendation is because Hadoop was designed to run on cheap server hardware to play with. It's not until now when Hadoop get accepted and runs in to large ente...

Automated file distrobitionen

Image
I helped a company that had problem with their backup situation, and didn't have a great total cost of ownership (TCO) of their file archive system that contain all original pictures they take daily. They grow with 6 TB of new data weekly and had 400TB of total data and together with the old backup software that still run Gran father/father/son backup method (Monthly Full and Daily Incremental backups). The full backup took more than 24 hours and the daily took more than 8 hours to before. When they got hold of the new picture, they could then work with the picture almost direct, but manually publish the new picture on the website. Backup was a problem, and the disk cost was always growing with no idea how to slow down that cost. What we did was implementing a new Software Define Storage filesystem to the customer and connect the web servers to a part of the filesystem. The filesystem is also  policy based driven, and could then automatic backup the new files in...

We don't need backup anymore... LOL

The world turning in to an interesting place and the idea of having a backup is sometime over shadowed by the protection of the application itself. I have often heard customers saying that they don’t need backup anymore. Even larger companies like banks have said that backup is unnecessary because their applications have their own ways of protecting their data. About a month ago I received a call from a customer saying their Exchange consultants are telling the backup-team that there is no need to backup their new Exchange 2013 environment. During the migration from the old environment the customer decided to upgrade their Anti-virus/Spam software for their mails aswell. Guess what happened next? BAM!!! The Anti-Virus software went in to each mailbox and deleted all attachments in all emails, because it thought it was a virus. This was a bug in the Anti-Virus software and shouldn’t have happened but it did. The customer was lucky, they used IBM Spectrum Protect to backup all m...