-
Technology
-
Industry
-
Solutions
-
- ASSET MANAGEMENT
- CUSTOMER EXPERIENCE MANAGEMENT
- CYBER SECURITY
- DATA CENTER
- DOCUMENT MANAGEMENT
- ELECTRONIC DATA INTERCHANGE
- ENTERPRISE DATA MANAGEMENT
- ENTERPRISE RESOURCE PLANNING
- ENTERPRISE RISK MANAGEMENT
- ENTERPRISE-GRADE WEB DATA SOLUTIONS
- FACILITY MANAGEMENT
- FIELD SERVICE
- IDENTITY AND ACCESS MANAGEMENT
- INFRASTRUCTURE
- IT SERVICE MANAGEMENT
- MANAGED IT SERVICES
- PAYMENT AND CARD
- PROJECT MANAGEMENT
- SOFTWARE TESTING
- STORAGE
- VIDEO SOLUTIONS
- WORKFLOW
-
-
Platforms
-
Functions
- Leadership Perspectives
- Innovation Insights
- Research
- Magazines
- News
- CXO Awards
Einat Orr, Ph.D, Co-Founder, and CEOThis realization had emphasized the difference between the teams that were delivering software applications and the teams who were developing and maintaining the data assets for the company’s data-intensive products. While for years, software engineers benefited from engineering best practices such as the agile development methodology and the ALM tooling that support this methodology (such as git, Jenkins, testing platforms, etc.), data engineers were not equipped with such basic and needed tools for their data, and hence were struggling with fixing quality issues and recovering from these issues. These hurdles brought an enormous cost of error for the entire company.
This is the reason Orr and Katz developed lakeFS - an open source tool, which transforms object storage buckets into git-like repositories. lakeFS is providing versioned data lake operations and uses them to bring a development workflow and methodology into the field of big data. It arms data engineers with simple yet powerful tooling that can increase their productivity and reduce their cost of error.
“The basic need of engineers in general, and data engineers in particular, is to be able to develop and test things freely without worrying that their changes will break things in production. Data engineers need to safely & confidently develop and test the pipelines they are building with the entire production data”, says the co-founder and CEO of lakeFS. “lakeFS has created a simple way to develop and test in isolation without needing to copy the data lake multiple times. This is done without any compromise to the performance of the data lake, as lakeFS can manage exabytes of data and allows applications accessing the data to benefit from Git-like operations”.
lakeFS can manage billions of objects and provide highly scalable and high performance Data Version Control for data lakes. The open source lakeFS project supports AWS S3, Azure Blob Storage, and Google Cloud Storage (GCS) as its underlying storage service, together with on-prem object storages with s3 interface such as min.io, Chef, Vast, Weka and Dell EMC S3. It is API compatible with S3 and integrates seamlessly with popular data frameworks such as Spark, Hive, dbt, Trino, and many others.
Through its versioning engine, lakeFS enables built-in operations inspired by git, helping organizations to apply efficient lifecycle management practices to their data engineering. The first one - lakeFS’ Branch action, is a cost-effective metadata operation that provides businesses an isolated development and testing environment with a snapshot of the data lake repository - without copying any data. This drastically reduces the storage cost for organizations and enhances the efficiency of data engineers as they can develop freely and safely against production data in isolation.
Also, lakeFS provides the revert action (atomic and immediate, with no manual effort) whenever an error is found in production data. This enables the sought for “undo” functionality that is missing in the way data lakes are designed today.
Through its versioning engine, lakeFS enables built-in operations similar to Git, helping organizations apply efficient software engineering best practices to their data operations
In the process of introducing advanced practices in the data management arena, lakeFS solves crucial problems related to the transient nature of data. As data is in a state of constant flux, lakeFS opens the floodgates of data version control tools to incorporate efficiency and agility in data management. Adopting lakeFS will speed up companies’ development and deployment cycles, reduce the chance of incorrect data making it into production, and make recovery less painful if it does.
Open Source lakeFS data version control levels up to 1.0
Thursday, October 26, 2023
Treeverse, creators of the open-source lakeFS data version control system, today announced the release of lakeFS 1.0. This major update brings production-level stability, security and performance to the data lake version control software.
The lakeFS project got its start back in 2020 and has been steadily improving in the years since, providing an open source technology to help organizations with version control for object storage based data, stored in data lakes.
Treeverse, the lead company behind the technology, raised $23 million back in 2021 to build out the concept that delivers capabilities that are similar to the open source Git version control system, to data lakes. In 2022, the technology got a cloud service with Treeverse launching the lakeFS cloud offering providing a managed cloud service data version control. The lakeFS approach has found a receptive audience according to Treeverse, with large enterprises including Lockheed Martin, Volvo and Arm among the technology’s users.
The lakeFS 1.0 technology is now also able to integrate with other data lake technologies, including Databricks as well as the open source technology Apache Iceberg that is increasingly being widely adopted by cloud data vendors, including Cloudera and Snowflake among others.
“We have a large base of installations and really a product that reflects what people need for data version control over a data lake,” Einat Orr, Co-founder and CEO at Treeverse, told VentureBeat in an exclusive interview.
What lakeFS data version control bring to the data lake market
Data version control allows users to track changes to data over time, similar to how version control systems like Git track changes to code.
With the open source Git version control system, that is at the heart of GitHub and much of modern application development, there is the concept of being able to have different versions of code and different branches. It’s a wildly popular approach to development that lakeFS has extended to the world of data stored in data lakes.
The idea of versioning in data lake deployments has a lot of nuance, as multiple vendors and technologies have varying degrees of versioning capabilities. Orr noted that while other technologies including Databricks and Apache Iceberg may allow creating versions of tables or schemas, that is different than a full data version control system.
Orr explained that lakeFS provides a full version control experience across an organization’s entire data lake, not just specific tables or schemas. This allows versioning entire data pipelines and workflows together. The lakeFS technology stores metadata about each version and changes that are important for reproducibility and integration.
Treeverse is not necessarily positioning lakeFS as a competitor to technologies like Databricks or Apache Iceberg but rather as a complementary technology that provides additional benefits to users. Orr also noted that lakeFS integrates with data orchestration tools including Apache Airflow, Prefect and Dagster, bringing the power of data version control to the data pipeline workflow.
The intersection of lakeFS and AI
There are a number of different data analytics and AI use cases for the lakeFS technology.
Looking at AI and machine learning (ML), Orr said that one interesting use case is that data scientists can use lakeFS to version data locally for model development and testing purposes, through a new lakeFS local capability.
Orr explained that data scientists and AI/ML model developers will often deal with a lot of data. That said, she noted that for testing and development, developers will sometimes be doing the research on their own local systems, which is what the new lakeFS capabilities help to enable.
Looking forward, Orr said that her company is in the early stages of figuring out how to integrate and enable data version control capability for vector database technologies.
“Our vision is to be the version control tool that is running over all your data sources, and providing you the ability to version control your data pipelines, no matter where the data is,” she said.
Company
lakeFS
Management
Einat Orr, Ph.D, Co-Founder, and CEO
Description
lakeFS transforms object store buckets into git-like managed repositories, to enable similar development workflows for code and data and saving organizations money and engineering efforts.
