What’s Subsequent for Knowledge Engineering in 2023? 13 Predictions


What’s subsequent for the way forward for knowledge engineering? Annually, we chat with certainly one of our business’s pioneering leaders about their predictions for the fashionable knowledge stack – and share a couple of of our personal.

Just a few weeks in the past, I had the chance to talk with famed enterprise capitalist, prolific blogger, and buddy Tomasz Tunguz about his prime 9 knowledge engineering predictions for 2023. It appeared like a lot enjoyable that I made a decision to seize my crystal ball and add a couple of recommendations to the combination.

Earlier than we start, nevertheless, it is vital to know what precisely we imply by fashionable knowledge stack:

  • It is cloud-based
  • It is modular and customizable
  • It is best-of-breed first (selecting one of the best instrument for a particular job, versus an all-in-one resolution)
  • It is metadata-driven
  • It runs on SQL (at the very least for now)

With these primary ideas in thoughts, let’s dive into Tomasz’s predictions for the way forward for the fashionable knowledge stack.

Professional-tip: be sure you take a look at his discuss from IMPACT: The Knowledge Observability Summit.

Prediction #1: Cloud Manages 75% Of All Knowledge Workloads by 2024 (Tomasz)

Picture courtesy of Tomasz Tunguz.

This was Tomasz’s first prediction, and primarily based on an analyst report earlier this 12 months displaying the expansion of cloud versus on-premises RDBMS income.

In 2017, cloud was about 20% of on-prem, and thru the course of the final 5 years, the cloud has mainly achieved equality by way of income. Should you mission three or 4 years, given the expansion fee we’re seeing right here, about 75% of all these workloads will likely be migrating to the cloud.

The opposite statement he had was that on-prem spend has mainly been flat all through that interval. That provides a variety of credence to the thought you may take a look at Snowflake’s revenues as a proxy for what’s occurring within the bigger knowledge ecosystem.

Snowflake went from 100 million in income to about 1.2 billion in 4 years, which underscores the terrific demand there may be for cloud knowledge warehouses.

Prediction #2: Knowledge Engineering Groups Will Spend 30% Extra Time On FinOps / Knowledge Cloud Value Optimization (Barr)

Through FinOps Basis

My first prediction is a corollary to Tomasz’s prophecy on the speedy progress of knowledge cloud spend. As extra knowledge workloads transfer to the cloud, I foresee that knowledge will change into a bigger portion of an organization’s spend and draw extra scrutiny from finance.

It is no secret that the macro financial setting is beginning to transition from a interval of speedy progress and income acquisition to a extra restrained give attention to optimizing operations and profitability. We’re seeing extra monetary officers play growing roles in offers with knowledge groups and it stands to purpose this partnership will even embrace recurring prices as properly.

Knowledge groups will nonetheless have to primarily add worth to the enterprise by appearing as a pressure multiplier on the effectivity of different groups and by growing income by way of knowledge monetization, however price optimization will change into an more and more vital third avenue.

That is an space the place greatest practices are nonetheless very nascent as knowledge engineering groups have centered on pace and agility to fulfill the extraordinary calls for positioned on them. Most of their time is spent writing new queries or piping in additional knowledge vs. optimizing heavy/deteriorating queries or deprecating unused tables.

Knowledge cloud price optimization can also be in one of the best curiosity of the info warehouse and lakehouse distributors. Sure, in fact they need consumption to extend, however waste creates churn. They might quite encourage elevated consumption from superior use circumstances like knowledge functions that create buyer worth and due to this fact elevated retention. They don’t seem to be on this for the short-term.

That is why you might be seeing price of possession change into a much bigger a part of the dialogue, because it was in my dialog at a current convention session with Databricks CEO Ali Ghodsi. You’re additionally seeing the entire different main players-BigQuery, RedShift, Snowflake-highlight greatest practices and options round optimization.

This improve in time spent will doubtless come each from further headcount, which will likely be extra immediately tied to ROI and extra simply justified as hires come beneath elevated scrutiny (a survey from the FinOps basis forecasts a mean progress of 5 to 7 FinOps devoted workers). Time allocation will even doubtless shift inside present members of the info crew as they undertake extra processes and applied sciences to change into environment friendly in different areas like knowledge reliability.

Prediction #3: Knowledge Workloads Section By Use (Tomasz)

Picture courtesy of Tomasz Tunguz.

Tomasz’ second prediction centered on knowledge groups emphasizing utilizing the suitable instrument for the suitable job, or maybe the specialised instrument for the specialised job.

The RBMS market has grown from about 36 billion to about 80 billion from 2017 to 2021, and most of these workloads have been centralized in cloud knowledge warehouses. However now we’re beginning to see segmentation.

Completely different workloads are going to want completely different sorts of databases. The best way Tomasz sees it, at this time every part is working in a cloud knowledge warehouse, however within the subsequent few years there will likely be a bunch of workloads which are pushed into in-memory databases, notably for smaller knowledge units. Consider, the overwhelming majority of cloud knowledge workloads are most likely lower than 100 gigabytes in dimension and one thing you would do on a specific machine in reminiscence for greater efficiency.

Tomasz additionally predicts notably giant enterprises who’ve completely different wants for his or her knowledge workloads could begin to take jobs that do not require low latency or the manipulation of great volumes of knowledge and truly transfer them to cloud knowledge lakehouses.

Prediction #4: Extra Specialization Throughout the Knowledge Workforce (Barr)

Search quantity for knowledge roles over time. Picture courtesy of ahrefs.

I agree with Tomasz’s prediction on the specialization of knowledge workloads, however I do not assume it is solely the info warehouse that is going to section by use. I feel we’re going to begin seeing extra specialised roles throughout knowledge groups as properly.

At the moment, knowledge crew roles are segmented primarily by knowledge processing stage:

  • Knowledge engineers pipe the info in,
  • Analytical engineers clear it up, and
  • Knowledge analysts/scientists visualize and glean insights from it.

These roles aren’t going wherever, however I feel there will likely be further segmentation by enterprise worth or goal:

  • Knowledge reliability engineers will guarantee knowledge high quality
  • Knowledge product managers will increase adoption and monetization
  • DataOps engineers will give attention to governance and effectivity
  • Knowledge architects will give attention to eradicating silos and longer-term investments

This is able to mirror our sister subject of software program engineering the place the title of software program engineer began to separate into subfields like DevOps engineer or website reliability engineer. It is a pure evolution as professions begin to mature and change into extra advanced.

Prediction #5: Metrics Layers Unify Knowledge Architectures (Tomasz)

Tomasz’s subsequent prediction handled the ascendance of the metrics layer, also called the semantics layer. This made an enormous splash at dbt’s Coalesce the final two years and it may begin reworking the way in which knowledge pipelines and knowledge operations look.

Picture courtesy of Tomasz Tunguz.

Immediately, the traditional knowledge pipeline has an ETL layer that is taking knowledge from completely different programs, and placing it right into a cloud knowledge warehouse. You have acquired a metrics layer within the center that defines metrics like income as soon as after which it is used downstream in BI for constant reporting and all the firm can use it. That is the principle worth proposition of that metrics mannequin. This expertise and thought has existed for many years, but it surely’s actually come to the fore fairly not too long ago.

Picture courtesy of Tomasz Tunguz.

As Tomasz suggests, now corporations require a machine studying stack, which seems similar to the traditional BI stack, but it surely’s really constructed a variety of its personal infrastructure individually. You continue to have the ETL that will get put right into a cloud knowledge warehouse, however now you have acquired a characteristic retailer, which is a database of the metrics that knowledge scientists use as a way to prepare machine studying fashions and finally serve them.

Nonetheless, should you take a look at these two architectures, they’re really fairly comparable. And it is not arduous to see how the metrics layer and the characteristic retailer may come collectively and align these two knowledge pipelines as a result of each of them are defining metrics which are used downstream.

In the end, Tomasz argues, the logical conclusion is that a variety of the machine studying work at this time ought to transfer into the cloud knowledge warehouse, or the database of alternative, as a result of these platforms are accustomed to serving very giant question volumes with very excessive availability.

Prediction #6: Knowledge Will get Meshier, However Central Knowledge Platforms Stay (Barr)

Picture courtesy of Monte Carlo.

I agree with Tomasz. The metrics layer is promising indeed- knowledge groups want a shared understanding and single supply of fact particularly as they transfer towards extra decentralized, distributed buildings, which is the center of my subsequent prediction.

Predicting knowledge groups will proceed to transition towards a knowledge mesh as initially outlined by Zhamak Dehgani just isn’t essentially daring. Knowledge mesh has been one of many hottest ideas amongst knowledge groups for a number of years now.

Nevertheless, I’ve seen extra knowledge groups making a pitstop on their journey that mixes area embedded groups and a middle of excellence or platform crew. For a lot of groups this organizing precept provides them one of the best of each worlds: the agility and alignment of decentralized groups and the constant requirements of centralized groups.

I feel some groups will proceed on their knowledge mesh journey and a few will make this pitstop a everlasting vacation spot. They may undertake knowledge mesh rules reminiscent of area-first architectures, self-service, and treating knowledge like a product-but they’ll retain a strong central platform and knowledge engineering SWAT crew.

Prediction #7: Notebooks Win 20% of Excel Customers With Knowledge Apps (Tomasz)

Picture courtesy of Tomasz Tunguz.

Tomasz’s subsequent prediction derived from his dialog with a handful of knowledge leaders from FORTUNE 500 corporations a couple of years in the past.

He requested them, “There are a billion customers of Excel on the earth, a few of that are inside your organization. What fraction of these Excel customers write Python at this time and what is going to that proportion be in 5 years?”

The reply was 5% of people that use Excel at this time write Python, however in 5 years, it’s going to be 50%. That is a fairly elementary change and it implies they are going to be 250 million folks searching for a subsequent technology knowledge evaluation instrument that does one thing like Excel, however in a superior method.

That instrument may very well be the Jupyter pocket book. It is acquired all the benefits of code: it is reproducible, you may verify it in GitHub, and it is very easy to share. It may change into the dominant mechanism for changing Excel for these extra subtle customers and use circumstances reminiscent of knowledge apps.

A knowledge engineer can take a pocket book, write a bunch of code even throughout completely different languages, pull in several knowledge sources, merge them collectively, construct an utility, after which publish this utility to their finish customers.

That is a extremely spectacular and vital development. As an alternative of passing round an Excel spreadsheet, Tomasz suggests, folks can construct an utility that appears and appears like an actual SaaS utility, however custom-made to their customers.

Prediction #8: Most machine studying fashions (>51%) will efficiently make it to manufacturing (Barr)

Within the spirit of Tomasz’s pocket book prediction, I imagine we’ll see the common group efficiently deploy extra machine studying fashions into manufacturing.

Should you attended any tech conferences in 2022, you may assume we’re all dwelling in ML nirvana; in any case, the profitable tasks are sometimes impactful and enjoyable to focus on. However that obscures the truth that most ML tasks fail earlier than they ever see the sunshine of day.

In October 2020, Gartner reported that solely 53% of ML tasks make it from prototype to production-and that is at organizations with some stage of AI expertise. For corporations nonetheless working to develop a data-driven tradition, that quantity is probably going far greater, with some failure-rate estimates hovering to 80% or extra.

There are a variety of challenges, together with

  • Misalignment between enterprise wants and machine studying targets,
  • Machine studying coaching that does not generalize,
  • Testing and validation points, and
  • Deployment and serving hurdles.

The explanation why I feel the tide begins to show for ML engineering groups is the mix of elevated give attention to knowledge high quality and the financial strain to make ML extra usable (of which extra approachable interfaces like notebooks or knowledge apps like Steamlit play an enormous half).

Prediction #8: “Cloud-Prem” Turns into The Norm (Tomasz)

Tomasz’s subsequent prediction addressed the closing chasm between completely different knowledge infrastructures and shoppers just like his metrics layer prediction.

The previous structure for knowledge motion was a company that may have, within the case of the picture above, three completely different items of software program. The CRM for gross sales, a CDP for advertising, after which the finance database. The information inside these databases doubtless overlap.

What you’d see within the previous structure (nonetheless very prevalent at this time) is you are taking all that knowledge, you pump it into the info warehouse, and then you definitely pump it again out to counterpoint different merchandise like a buyer success product.

The following technology of structure goes to be a learn and write cloud knowledge warehouse the place the gross sales database, the advertising database, the finance database, and the shopper success data, they’re all saved on a cloud knowledge warehouse with a bi-directional sync throughout them

There are a few completely different benefits to this structure. The primary is it is really a go to market benefit. If an enormous cloud knowledge warehouse accommodates knowledge from an enormous financial institution, they’ve gone by way of the data safety course of as a way to get the approval to control that data, the SaaS functions constructed on prime of that cloud knowledge warehouse solely have to get permissions to that data-you not have to undergo the data safety course of, which makes your gross sales cycles considerably quicker.

The opposite principal profit as a software program supplier, Tomasz suggests, is that you are going to have the ability to use and be a part of data throughout these knowledge units. That is doubtless an inexorable development that is most likely going to proceed for at the very least the following 10 to fifteen years.

Prediction #9: Knowledge contracts transfer to early stage adoption (Barr)

An instance of a knowledge contract structure. Picture courtesy of Andrew Jones.

Anybody who follows knowledge discussions on LinkedIn is aware of that knowledge contracts have been among the many most mentioned matters of the 12 months. There is a purpose why: they handle one of many largest knowledge high quality points knowledge groups face.

Sudden schema modifications account for a big portion of knowledge high quality points. Most of the time, they’re the results of an unwitting software program engineer who has pushed an replace to a service not figuring out they’re creating havoc within the knowledge programs downstream (maybe as a result of they do not have visibility into knowledge lineage).

Nevertheless it is vital to notice that given all the web chatter, knowledge contracts are nonetheless very a lot of their infancy. The pioneers of this process-people like Chad Sanderson and Andrew Jones-have proven the way it can transfer from idea to apply, however they’re additionally very straight ahead that it is nonetheless a piece in progress at their respective organizations.

I predict the vitality and significance of this subject will speed up its implementation from pioneers to early stage adopters in 2023. This can set the stage for what will likely be an inflection level in 2024 the place it begins to cross the chasm right into a mainstream greatest apply or begins to fade away.


Let us know what you consider our predictions. Something we missed?

Tomasz often shares his observations on his weblog and on LinkedIn – be sure you comply with him to remain knowledgeable!

The submit What’s Subsequent for Knowledge Engineering in 2023? 13 Predictions appeared first on Datafloq.


Please enter your comment!
Please enter your name here