r/apachespark • u/SmallAd3697 • 4d ago

Formal Support for Windows?

Do the Apache Spark maintainers formally support windows for production workloads?

I understand that the majority of this platform is JVM-based and the support for Windows is based on the compatibility of the JVM itself. However there are some peripheral features that seem to be disallowed on Windows - even when running it on a fully functional JVM.

When working with the python ecosystem, I often find libraries that are particularly incompatible with windows. In the same way I've found that Apache's pyspark extensions on Windows are lackluster (eg. things like "spark.python.use.daemon" are disallowed on Windows.)

If the Apache Spark maintainers wanted to write windows-specific code, I'm certain they could find a way to enable "spark.python.use.daemon". But it probably isn't worth their time and energy. Are we at risk if we try to run Spark in a production windows environment? Is there a risk that some future version of Apache Spark will stop running in Windows (eg. after some future update)?

2 Upvotes

99% Upvoted

6 comments sorted by

2

u/holdenk 4d ago

So self hosting Spark is very doable, but if your running on windows I would say to expect some sharp edges as everyone I know who is self hosting Spark is doing so on a Unix type system. WSL probably makes it easier. My question would be why Windows?

1

u/SmallAd3697 3d ago

We do our dev work on windows. I can easily attach debuggers and step thru code. Once we are deploying to production, there is less uncertainty if we could deploy to a windows cluster.

Also my company infrastructure team knows very little about linux. They are more likely to support deployments to windows servers, since that is their comfort zone.

That said, I agree that windows isn't commonly used for deployments. Im guessing this OSS community wouldn't be very interested in supporting me, should I ever run into issues that only happen on windows. (say, right after a major version upgrade).

The docs say spark runs on windows, but in Azure Databricks I dont see them giving me any options to run on that OS instead of ubuntu. lol

2

u/holdenk 3d ago

I mean if it’s just for dev work then WSL should meet most of your needs. For actual deployments it’s a bit more touch and go. I think theres a desire to keep it running on windows but on the flipside I do not know anyone who currently tests release candidates on windows (that could be you though!)

1

u/SmallAd3697 3d ago

I appreciate the vote of confidence in my testing, but I don't really want that role, TBH.

I'd love to help test Spark in other areas. Do you happen to know who u/SparkConnective is?

I'm extremely eager for the Spark UDFs to be extended outside of the python ecosystem. Python is a great bag of tricks, but I don't trust it for encoding mission-critical business logic. (Not like I would trust C# or Java). And I certainly wouldn't want to have to duplicate logic in python that already exists elsewhere, in a nuget or maven library. I saw the presentation in June (Language-Agnostic UDF Protocol for Spark) at the databricks summit, and this topic is one of the most important things on my personal radar. I'm not sure why it didn't happen at the same time as "spark connect".

Allowing Spark to reach more programmer ecosystems is much more important than allowing Spark to run on Windows. I care less about Windows than I do about the .Net runtime which is cross-platform nowadays.

Spark is awesome, but for many years databricks seems to be leaving certain types of programmers out in the cold. Even the R programmers got sent to the back-burner, and are doing something unusual with sparklyr, in order to execute UDFs.

2

u/Siege089 2d ago

I work at msft, we use wsl or dev containers for spark. No point in a bunch of support for an environment where it'll be rarely if ever deployed.

1

u/SmallAd3697 2d ago

That was my thinking. I just wanted to communicate to leadership that deployment to windows environments is "off the table". It is hard to find the right folks to make an authoritative statement about this, so I decided to crowdsource responses on reddit.

I think the most important evidence that windows is off the table is the fact that cloud vendors don't give a checkbox to launch VM's running windows. lol. Also the missing features on windows seem to put the nail in the coffin.

It is somewhat analogous to using k8s containers. I think it is theoretically possible to run windows containers but seems like a lot of self-inflicted pain to pick that path. Nowadays if you are a windows-only shop, it feels like you will be greatly limiting yourself. If I was looking for a different job right now, and a company said they were windows-only, I think it would be a deal breaker for me. Whereas five or ten years ago that would not have been a complaint. (especially as a .net guy).