How Much Do You Value Data Engineers

[relemedf5w023]

According to Ciaran Dines, chief product officer at data pipeline specialist Matillion, the AI ​​revolution is affecting every area of ​​life, including big data management. “The role of the data engineer has evolved dramatically over the last decade,” he says. “ 2024 will be the year that tech companies make their lives easier. There will be tools that will be integrated into existing platforms, allowing generative AI to be added to existing data pipelines, with the ability to deploy these models in-house so users can interact with them live, just like they already do with ChatGPT. There will also be a huge increase in demand next year for data engineers to reskill themselves in how to engineer suggestions, fine-tune AI models, and dramatically improve their performance. Their lives are about to get a lot more interesting.”

? You’ll value them even more in 2024, according to Jeff Hollan, chief product officer at Snowflake. “There’s been a lot of talk about AI replacing the data engineer role,” he says. “That’s not the case, and in fact, their data science expertise will be more important than ever — just in new and different forms. To keep up with the spain mobile database landscape, data engineers will need to understand how generative AI adds value. The data pipelines they build and manage will likely be the first place organizations can connect to large language models (LLMs) to extract value. Data engineers will be the ones who understand how to use the model and plug it into the data pipeline to automate the process of extracting value. They will also need to oversee and understand the AI.”

You may feel like your data is out of control when it’s managed by a third party in the cloud. Peter Shafton, CTO of Ngrok, predicts that 2024 will be the year you can take back control of your data. “Data governance will shift significantly towards greater availability and control,” he says. “While the last decade saw the rapid growth of cloud-based data solutions, the pendulum has now swung towards more self-managed data. The reasons for this shift are twofold: privacy and cost efficiency. The constant threat of data breaches and the need for tighter access controls make companies wary of relying solely on external cloud platforms. Additionally, the unpredictability of data storage and processing costs in the cloud is driving organizations to look for more predictable and cost-effective solutions. This trend is also fueled by the proliferation of accessible and easy-to-use data management tools, often based on open source solutions from tech giants like Uber, Netflix, and Airbnb.”