Here is a new article about DataCrunch, following DataCrunch Introduction. We will focus on DataCrunch design and technical implementation.

This article details how we perform analytics computations on all our scientific datasets in native format (Netcdf, Hdf, ...), using terabytes of data and millions files, with only 30 lines of Python ...

Overview

Datacrunch is a framework based on Hadoop which aims to make easy and fast to extract and compute meaningful informations from earth observation datasets (ie. satellite, insitu ocean buoys/drifters, models...).

DataCrunch is a Cersat homemade framework, result of our experiments using BigData technologies, taking benefit from Map/Reduce ...