Working as a Data Engineer Sucks
Analyzing hundreds of threads from r/dataengineering
Working as a data engineer sucks!
At least if you look at the data I pulled from the subreddit r/dataengineering. The data tells a story about mundane tasks, stress, blurred lines between roles, and lack of recognition.
But it’s not all despair and nightfall in data land, some engineers seem to thrive in their roles like a byte in a data lake.
👉 If you don’t care about my writing and analysis (rude!) here is a link to the data.
Who, me?
Full disclaimer: I work as a data engineer 9 to 5 at a Swedish company fighting food waste. Outside of work, I’m a full-time dad, but I also run a tiny SaaS that focuses on data extraction.
I’m curious about what people think about data engineering since my own experience has been, well, not always great. Especially in the beginning, when I was drowning in alerts and forced to model dirty data from SharePoint of all places.
I also wanted to do some product dogfooding, so I figured, why not use my own service AutoSERP to extract comments about working as a data engineer from or modern day Plato's Academy, Reddit.
The Data
I extracted data using AutoSERP.dev from around 100 threads created between 2024-01 and 2025-02 about working as a data engineer. I wanted fresh data because the field is relatively new and probably changing fast. Each thread was analyzed in its entirety and divided into “opinions”. These opinions were then classified into a main category (pro or con) and a subcategory related to the actual opinion.
👉In case you missed it, here is the data.
The Good
Benefits & Compensation
“My salary really started taking off as I started taking on more DE and DS tasks.”
Many data engineers highlight compensation and benefits as major advantages, often referring to high salaries and the ability to land senior DE roles.
Career Growth & Promotions
“Data engineering was a good in-between that I could settle into.”
“Many analysts become DEs later, and this is usually win-win.”
There seems to be a clear path for many: start as a data analyst, transition into data engineering, and then move on to data science.
It’s also interesting how many enjoy the ambiguity of the role, which makes it easy to transition into other software roles that aren’t strictly data-related.
Job Role & Responsibilities
“You will be talking to or about business much more frequently”
“More direct connection with clients and building things people actually use.”
Aside from comments that the job can be fun and chill (if systems are operational), people mention the relationship with other departments and the satisfaction of creating tangible value.
The Bad
Organizational & Process Issues
By far the biggest issue seems related to internal company workings. I doubt this is unique to data engineering, but the “why” can be quite different.
Role Ambiguity
A lot of data engineers find the same flexibility praised by others to be a downside, and it's easy to see why. Ambiguity often leads to excess responsibility, ad hoc tasks, and unmet expectations.
“Data Engineer is not well-defined and can mean wildly different things depending on the company, which also means complexity varies a lot.”
“Former DE, current SE. My DE role was hands-on with consistent scope creep. I had to know data, networking, security, cloud infrastructure, microservices, etc.”
Work-Life Balance & Stress
“Today I woke up by myself at 6, with gastritis, thinking about the freaking slides I have to deliver while the rest of the work piles up.”
“There will be some weeks where I feel like I’m running around with my head cut off and am drowning in work.”
I think much of the stress comes down to role ambiguity, the business-critical nature of the job, and dependence on third-party data sources, which creates exponential instability.
Mundane Tasks & Job Appreciation
I found these categories interesting because there’s such a clear pattern in the comments. Data can be fun and challenging, but it can also be extremely dull and thankless.
“It feels monotonous, as I am just building pipelines for dashboards.”
“Risk of being pigeonholed or undervalued, with companies sometimes treating data engineers as mere SQL operators or dashboard builders.”
My Own Thoughts
I believe I’ve experienced most of these issues at least once as a data engineer. But as I mentioned, my role has evolved over time. One important aspect is my company’s willingness to invest in the data team.
That investment in new technology and becoming a real product-driven team with tickets and a product owner really helped shield us from the worst. The ambiguity problem went away when we hired real data analysts who could work on the presentation side and field continuous questions about the data from different departments.
The emerging nature of the role creates ambiguity that can blur lines of responsibility, risking a “catch-all” scenario for anything related to data.
I would not recommend being a one-person show when it comes to data engineering, especially if it’s a business-critical function. The emerging nature of the role creates ambiguity that can blur lines of responsibility, risking a “catch-all” scenario for anything related to data.
The technology is constantly evolving, making the cognitive load of learning new tools a given. A third of all venture capitalist money went into AI companies last year, and we know there’s no AI without data. So the need for the role to mature is vital if we don’t want to see a quick rise in burnout.
Found you on Reddit, well done , enjoyed this post 🙏