DataTorrent: Data Ingestion in Real Time

Today, the majority of data available in organizations is “unstructured”, and must be organized in a way that makes it suitable for information mining and subsequent analysis, as the significance of predictions made by sorting and analyzing big data has gained momentum. Hadoop is the core platform for structuring big data that enables distributed parallel processing of large volumes of data across servers resulting in the extraction of potential value from all this data. However, organizations face challenges in the form of getting data in and out of Hadoop, and no existing tool handles all the requirements demanded for Hadoop ingestion. Santa Clara, CA based DataTorrent is providing real-time big data analytics solution that has high performing, fault tolerant unified architecture for both data in motion and data at rest—through DataTorrent Real-time Streaming (RTS). The company has also introduced an enterprise-grade unified platform for both stream and batch processing on Hadoop—DataTorrent dtIngest. “dtIngest simplifies the collection, aggregation and movement of large amounts of data to and from Hadoop for a more efficient data processing pipeline,” says Phu Hoang, Co-founder and CEO, DataTorrent.

DataTorrent’s dtIngest is built on Apache Apex—an open source enterprise grade unified stream and batch processing platform, making the job of configuring and running Hadoop data ingestion and data distribution pipelines a point-and-click process. The Apache 2.0 open-source platform is completely fault tolerant and can ‘resume’ file ingest on failure. The solution is simple to use and manage, making it easier to configure, save, and launch multiple data ingestion and distribution pipelines. Its centralized management provides visibility, monitoring and summary logs. With secure and efficient data movement dtIngest supports compression and encryption during ingestion, is certified with Kerberos-enabled secure Hadoop clusters, and runs in any Hadoop 2.0 cluster.

The company was established by Yahoo! veterans who had in-depth expertise in overseeing big data for driving edge applications and framework on a massive scale. Their mastery in building large-scale platforms, have helped to build an infinitely scalable and fully fault-tolerant technology that leverages the very latest developments in the Big Data ecosystem.