Longtime data engineer - web apps in vanilla HTML/CSS/JS with Node - and some fun with ncurses in C++. Copilot is helping me out a bit but I’m having fun!
Hey, you're like my evil twin! Longtime web dev here who always wanted to learn more data engineering.
HTML/CSS/JS is fun and creative to some degree, but kinda abstract. Data engineers get to work with real-world phenomena! What prompted the interest in web stuff?
I never did proper "data engineering" per se, but we did have casual run-ins with geodatasets and GIS, and used them to build a home solar calculator that takes a ZIP code and estimates that area's available yearly sunlight, utility rates, electricity consumption, etc. and combine it into an estimated size for a home solar system. It was like a primitive version of Google Sunroof (https://sunroof.withgoogle.com/), which is much, much better.
I too love the intersection of data and visualization/usability, turning obscure spreadsheets into useful public interfaces :) The geospatial world is just one part of that. It used to be cool, but I guess nowadays it'd probably be more about turning data into LLM-digestible training sets so that they can more directly analyze questions and answer them in plain language.
Do you think traditional DE is still worth learning, with Skynet on the horizon? What's a good way to get started?
Wow - that sunroof thing is cool, I bet that was a fun area to work on.
There’s a lot to be said for data engineering when done at scale - systems design, devops, cloud engineering etc.
I’m not sure that’s going yet…
Building the individual units (reliable ETL pipelines) might be on the chopping block in the near future - I’m not convinced there’s 100 competing ways to build these.
Still - best way to get started is probably to get stuck in with running a data pipeline using a locally deployed Airflow instance. Read some data from some api and write it to a local database deployment (postgres?).
May I ask the same of web application development?
> that sunroof thing is cool, I bet that was a fun area to work on
Yeah, it was (and is) really cool! To be clear, I didn't work on that, just something similar but much more primitive and with much simpler datasets. Google did a really good job there.
> locally deployed Airflow instance
I'm pretty comfortable with basic ETL stuff, but never used Airflow. Will have to look into that, thanks!
> May I ask the same of web application development?
I can give you my opinion, but that's all it is. I'm not a FAANGer and primarily work with small businesses and nonprofits, so my perspective may be biased and incomplete. I should also note that I'm also kinda an AI-optimist, meaning I have a much more positive view of both its competence and its threat than many people. But ultimately, I'm a nobody, just a rando on the internet, so take all this with a big grain of salt :)
In the time I've been doing web stuff (20-30 years, depending), I've seen the industry move towards higher and higher levels of abstraction. In the old days (the 90s and early 2000s) it was a lot of hacked-together HTML + backend logic, then it gradually moved towards the clientside (frontend) with things like ActiveX, Flash, and eventually CSS and JS.
JS eventually won out and you can build really amazing apps almost entirely in the frontend now (like Figma, Photopea, OpenSolar, Felt, and other frontend-heavy, UI-driven things like those).
But those are what I'd consider proper "apps". There are also may websites that are just, well, sites and not what I'd consider apps, things like your average news or blog site, or maybe basic ecommerce stuff. That's where I see the most abstractions/consolidations into a few big frameworks, like Shopify or Wordpress + WooCommerce, or Wix/SquareSpace/Weebly for simpler sites, or headless CMSes + Jamstacks for more complex sites (disclaimer: I currently work for a headless CMS company). I think these sorts of sites are the most at risk of automation, which really has been happening for decades already. What used to take weeks of setup and then tons of ongoing maintenance (both on the frontend and backend) is basically just a one-click deploy these days with little need for actual coding anymore. And even complex, bespoke UIs are increasingly being AI-driven... the company that makes Next.js (a big frontend framework) is also using AI to try to replace their own customers (us frontend devs), lol: https://v0.dev/
I think there will always be a few humans needed in the loop, but probably fewer and fewer over time, and usually at the "architect" level, where you design overall systems and write some minor glue code to tie it altogether, but don't need to dive too deeply into the nitty-gritty anymore. I'm especially scared for junior coders, because even today, Copilot and ChatGPT are already way better than most of them (and often, better than myself too). One moderately experienced coder with a few AIs can easily replace what used to take a team, and do so with much less overhead (no need for Agile crap, three layers of mid-management, everlasting meetings, etc.). I think it's going to get a lot leaner, which means fewer openings, more productivity per remaining dev, but fewer dev openings overall.
But that's only if we assume that current web devs stay the course and don't upskill/adopt more and more AI practices. Just like fewer and fewer of us work in PHP or ASP or Ruby on Rails these days, I would assume that many would sidestep and just incorporate more AI and GPT into their workflows. Maybe there's some opportunities there for lean + mean startups who can do a lot more with fewer employees than before. But still, AI/ML is a fundamentally different enough skill set (like actual CS + math + modeling stuff, not just gluing together UI code) that a lot of us are going to flunk out and join the breadlines.
Even at this early stage of AI, I'm fairly confident that my career as a small-biz frontend dev is a dead end, with maybe 4-5 years left if I'm really lucky. It's not just because of AI doing a better job, necessarily, but the simple hype around AI means a lot of the money that used to be in Web is now pivoting towards AI. The bubble's burst, and sure, there'll always be a few dev jobs here and there, just like there are still newspaper or graphic design jobs here and there, but I'm fairly certain its heyday is over. Just my 2¢ =/
That said, I don't think it ever hurts to learn HTML + CSS at least. Those are relatively simple, declarative markup languages that's really more similar to Markdown than programming. JS (and especially React, etc.) is where it gets tricky, but even that ecosystem is finally somewhat mature/stable, such that it's a pretty easy time to get started, having missed the craziness of the late 2010s and early 2020s where it was going through very rapid iterations. Today it seems to have stabilized around React + Next.js as the go-to framework (by popularity), and the documentation and examples have gotten better. I wouldn't quit your day job to go learn any of this stuff -- like I don't think it's a good time to dive headfirst into becoming a web dev, with it becoming more and more yesterday's game instead of the future -- but as a side project? Sure. At the very least, it enables better UI design and visualizations, and that's always fun! Even with AI as the companion, there's a lot more creativity (and human psychology) there than gluing together pipelines and APIs (of course, that's probably just my bias as a frontend person creeping through).
Alternatively, it's also possible to get really good at some particular niche in the stack (like Canvas graphics or WebAssembly or WebGL) and find a super-specialist position at a bigger company. That would probably be a safer bet than my generalist background.
I dunno... sorry, I don't mean to be a downer, it's just that tech has always been fast-moving and cutthroat, and it's probably going to become even more ruthless in the future. Once AI starts self-iterating, it'll become exponential and I don't think humans will be able to keep up. Right now it seems like the biggest barrier is simply our hardware manufacturing capacity, but that's going to ramp the hell up soon. But anyway... we can only play the cards we're dealt, as we're dealt them. No need to fret about things out of my control, and Skynet is very much out of my control, lol.
As an old man (nearly 40), I have limited time and neuroplasticity left. It's harder for me to just pivot to the new shiny every few years. But if you're younger, I don't think it's ever wrong to explore and try new things and see what sticks (and what pays)!
Sorry, that's just the bigger-picture overview/rant. If you have any questions about more specific technologies/stacks, I'd be happy to share thoughts on those too.
Thanks for all the web dev advice and tech callouts. Some real neat examples there which I knew nothing about.
Regarding niches, I think as a data engineer, becoming fluent in an interactive front-end data-viz library might be the slam dunk that I'm probably looking for.
Having looked at D3, I am quite impressed! Do you have any experience or knowledge of this lib?
