{"id":85005,"date":"2019-08-19T14:15:04","date_gmt":"2019-08-19T14:15:04","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=85005"},"modified":"2026-04-14T16:38:09","modified_gmt":"2026-04-14T16:38:09","slug":"apache-spark-for-net-developers","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/development\/dotnet-development\/apache-spark-for-net-developers\/","title":{"rendered":"Apache Spark for .NET Developers: Setup and First Spark Program"},"content":{"rendered":"

Executive Summary<\/h2>\n

This article describes how to set up and write programs using .NET for Apache Spark (Microsoft.Spark) – the .NET driver that allowed C# and F# developers to use Apache Spark for distributed data processing. Note: this project was archived in 2024 and the instructions below reflect its original state. Apache Spark itself remains actively maintained and widely used – this article provides conceptual background on how .NET interfaced with Spark’s JVM-based execution model, how Spark’s DataFrame API works, and what kinds of big data processing Spark is suited for. That conceptual knowledge transfers to any Spark environment.<\/strong><\/p>\n

Apache Spark is a fast, scalable data processing engine for big data analytics. In some cases, it can be 100x faster than Hadoop. Ease of use is one of the primary benefits, and Spark lets you write queries in Java, Scala, Python, R, SQL, and now .NET. The execution engine doesn\u2019t care which language you write in, so you can use a mixture of languages or SQL to query data sets.<\/p>\n

The goal of .NET for Apache Spark is to make Spark accessible from C# or F#. You can bring Spark functionality into your apps using the skills you already have.<\/p>\n

The .NET implementation provides a full set of API\u2019s that mirror the actual Spark API so that, excluding a few areas still under development, the complete set of Spark functionality is available from .NET.<\/p>\n

Setting up Apache Spark on Windows<\/h2>\n

The .NET implementation still uses the Java VM, and so it isn\u2019t a separate implementation of Spark that replaces Spark but sits on top of the Java runtime and interacts with it. You still need to have Java installed.<\/p>\n

Spark is written in Scala and runs on a Java virtual machine so it can run on any platform including Windows. However, Windows does not have production support. The current version of Java that it supports is 1.8 (version 8).<\/p>\n

Oracle has recently changed the way that they support their JDK in that you need to pay a license fee to run it in production. Oracle also released a version called OpenJDK that doesn\u2019t have a license fee to pay when running in production. Spark can only run on Java 8 today and to run in a development environment doesn\u2019t cost anything so you can use the Oracle JRE 8 for this article, if you will be using Spark in production then it is something you should investigate.<\/p>\n

.NET for Apache Spark was released in April 2019 and is available as a download<\/a> on NuGet, or you can build and run the source<\/a> from GitHub.<\/p>\n

Install a Java 8 Runtime<\/h3>\n

You can download the JRE<\/a> from the Oracle site. You will need to create a free Oracle account to download.<\/p>\n

I would strongly suggest getting the 64 bit JRE because the 32-bit version is going to be very limited for Spark. The specific download is jre-8u212-windows-x64.exe,<\/em> although this will change when there are any more releases.<\/p>\n

Install Java, my installation of Java was in C:\\Program Files\\Java\\jre1.8.0_212<\/em> but take note of where your version is because you will need it later.<\/p>\n

Download and Extract a Version of Spark<\/h3>\n

You can download Spark here<\/a>. There are currently two versions of Spark that you can download, 2.3 or 2.4. The current .NET implementation supports both versions, but you do need to know which version you will be using. I would suggest downloading 2.4 at this point. The README for .NET spark shows which versions of Spark<\/a> are supported, currently any 2.3.* version is supported or and of 2.4.0, 2.4.1, 2.4.3 but note that 2.4.2 is not supported so stay clear of that version.<\/p>\n

At the time of this writing, the version of Spark supported by the current Microsoft.Spark is this version<\/a>.<\/p>\n

Once you have chosen the Spark version, you can select the package type, unless you want to compile Spark from source or use your own Hadoop implementation, then select the Pre-built for Apache Hadoop 2.7 and later<\/em> and then download the tgz. <\/em>Today, that is spark-2.4.3-bin-hadoop2.7.tgz<\/em>.<\/p>\n

Once it has downloaded, use 7-zip to extract the folder to a known location, c:\\spark-2.4.3-bin-hadoop2.7,<\/em> for example. Again, take note of where you extracted the Spark folder. My Spark folder looks like:<\/p>\n

<\/p>\n

If you have something that looks like this, then you should be in good shape.<\/p>\n

Download the Hadoop winutils.exe.<\/h3>\n

The last step is to download winutils<\/em>, which is a helper for Hadoop on windows. You can download it from GitHub<\/a>.<\/p>\n

When you have downloaded winutils.exe<\/em>, you need to put it in a folder called bin<\/em> inside another folder. I use c:\\Hadoop\\bin<\/em>, but as long as winutils.exe<\/em> is in a folder called bin<\/em>, you can put it anywhere.<\/p>\n

Configure Environment Variables<\/h3>\n

The final step in configuring Spark is to create some environment variables. I have a script I run from a cmd prompt when I want to use them but can also set system environment variables if you wish. My script looks like this:<\/p>\n

SET SPARK_HOME=c:\\spark-2.4.1-bin-hadoop2.7\nSET HADOOP_HOME=c:\\Hadoop\nSET JAVA_HOME=C:\\Program Files\\Java\\jre1.8.0_212\nSET PATH=%SPARK_HOME%\\bin;%HADOOP_HOME%\\bin;%JAVA_HOME%\\bin;%PATH%<\/pre>\n
What this script does is set SPARK_HOME<\/code> to the location of the extracted Spark directory, set JAVA_HOME<\/code> to the location of the JRE installation, set HADOOP_HOME<\/code> to the name of the folder that contains the bin directory that winutils.exe<\/em> is put in. Once the environment variables have been set, I add the bin folder from each to the PATH<\/code> environment variable.<\/p>\n
Testing Apache Spark on Windows<\/h2>\n
To check everything is set up correctly, check that the JRE is available and the correct version:<\/p>\n
In a command window, run Java -version<\/code> then spark-shell<\/code>. If you have set up all the environment variables correctly you should see the Spark-shell start. The Spark-shell is a repl that lets you run scala commands to use Spark. Using the repl is a great way to experiment with data as you can read, examine, and process files:<\/p>\n
<\/p>\n
When you are ready to continue, exit Spark-shell by typing :q<\/code>. You use the spark-shell to check that Spark is working. To run a job later, you use something called spark-submit<\/code>.<\/p>\n
If you can start the Spark-shell, get a prompt and the cool Spark logo, then you should be ready to write a .NET application to use Spark.<\/p>\n
Note, you may see a warning that says<\/p>\n
NativeCodeLoader: Unable to load native-Hadoop library for your platform\u2026 using builtin-java classes where available<\/em><\/p>\n
It is safe to ignore this; it means that you don’t have Hadoop running on your system. If this is a Windows machine, then that is highly likely.<\/p>\n
The .NET Driver<\/h2>\n
The .NET driver is made up of two parts, and the first part is a Java JAR file which is loaded by Spark and then runs the .NET application. The second part of the .NET driver runs in the process and acts as a proxy between the .NET code and .NET Java classes (from the JAR file) which then translate the requests into Java requests in the Java VM which hosts Spark.<\/p>\n
The .NET driver is added to a .NET program using NuGet and ships both the .NET library as well as two Java jars. One jar is for Spark 2.3 and one for Spark 2.4, and you do need to use the correct one on your installed version of Scala.<\/p>\n
There was a breaking change to version 0.4 of the .NET driver, so when you use the driver, if you are using version 0.4 or higher then you need to use the package name org.apache.spark.deploy.dotnet<\/em> and if you are on version 0.3 or less you should use org.apache.spark.deploy<\/em>, note the extra dotnet<\/em> at the end.<\/p>\n
Your First Apache Spark Program<\/h2>\n
The .NET driver is compiled as .NET standard so you can use either the Windows .NET runtime or .NET core to create a Spark program. In this example, you will create a new .NET runtime (4.6) console application:<\/p>\n
<\/p>\n
You will then add the .NET Spark driver from NuGet:<\/p>\n
<\/p>\n
Select Microsoft.Spark<\/em>. There was also an older implementation from Microsoft called Microsoft.SparkCLR<\/em> but that has been superseded, so make sure you use the correct one. For this example, use Spark version 2.4.1 and the 0.2.0 NuGet package \u2013 these have been tested and work together.<\/p>\n
When you add the NuGet package to the project, you should see in the packages folder the two Java jar\u2019s which you will need later:<\/p>\n
<\/p>\n
Execute Your First Program<\/h2>\n
For the first program, you will download a CSV from the UK government<\/a> website which has all of the prices for houses sold in the last year. If the file URL has changed, then you can get to it from here<\/a> after and searching \u201ccurrent month as CSV file\u201d.<\/p>\n
The program will read this file, sum the total cost of houses sold this month, and then display the results:<\/p>\n
using System;\nusing System.Linq;\nusing Microsoft.Spark.Sql;\nnamespace HousePrices\n{\n    class Program\n    {\n        static void Main(string[] args)\n        {\n            var Spark = SparkSession\n                           .Builder()\n                           .GetOrCreate();\n            var dataFrame = Spark.Read().Csv(args[0]);\n            dataFrame.PrintSchema();\n            dataFrame.Show();\n            var sumDataFrame = dataFrame.Select(Functions.Sum(dataFrame.Col(\"_c1\")));\n            var sum = sumDataFrame.Collect().FirstOrDefault().GetAs<Double>(0);\n            Console.WriteLine($\"SUM: {sum}\");\n        }\n    }\n}<\/pre>\n
The first thing to do is to either use the sample project<\/a> and build the project or create your own project and build it so you get an executable that you can call from Spark.<\/p>\n
First, take a look at this code:<\/p>\n
var Spark = SparkSession\n      .Builder()\n      .GetOrCreate();<\/pre>\n
Here you create the Spark session. The Spark session enables communication back with the .NET java code and through to Spark.<\/p>\n
Next review:<\/p>\n
            var dataFrame = Spark.Read().Csv(args[0]);\n            dataFrame.PrintSchema();\n            dataFrame.Show();<\/pre>\n
Here the Spark session created above reads from a CSV file. Pass in the path to the CSV on the command line (args[0]<\/code>). (I realise that you should validate if it exists.) Once the file has been read, the code will print out the schema and show the first 20 records.<\/p>\n
Finally look at this code::<\/p>\n
       var sumDataFrame = dataFrame.Select(Functions.Sum(dataFrame.Col(\"_c1\")));\n       var sum = sumDataFrame.Collect().FirstOrDefault().GetAs<Double>(0);\n       Console.WriteLine($\"SUM: {sum}\");<\/pre>\n
This will use the Sum<\/code> function against the _c1<\/code> column (the price column), it will then select it into a new DataFrame (sumDataFrame)<\/code> and then it iterates through the rows of the DataFrame<\/code>. It selects the first row and then retrieves the value of the 0\u2019th column and prints out the results.<\/p>\n
To run this, instead of just pushing F5 in Visual Studio, you need to first run Spark and tell it to load the .NET driver and pass onto the .NET driver the name of the program to execute.<\/p>\n
You will need these details to run the .NET app:<\/p>\n\n\n\n\n\n\n\n\n\n\n
Type<\/td>\nName<\/td>\nValue<\/td>\n<\/tr>\n
Environment Variable<\/td>\nJAVA_HOME<\/td>\nPath to JRE install such as C:\\Program Files\\Java\\jre1.8.0_212<\/em><\/td>\n<\/tr>\n
Environment Variable<\/td>\nHADOOP_HOME<\/td>\nPath to the folder that contains a bin<\/em> folder with winutils.exe inside such as c:\\Hadoop<\/em><\/td>\n<\/tr>\n
Environment Variable<\/td>\nSPARK_HOME<\/td>\nThe folder you extracted the contents of the downloaded spark (note that the file downloaded is a tar then gzipped file, so you need to un-gzip then un-tar the file)<\/td>\n<\/tr>\n
The driver package name<\/td>\n<\/td>\nFor 0.3 and less the driver package is org.apache.spark.deploy<\/em> and for 0.4 and greater it is org.apache.spark.deploy.dotnet<\/em><\/td>\n<\/tr>\n
The full path to the built .net executable<\/td>\n<\/td>\nI created my project in c:\\git\\simpletalk\\dotnet\\HousePrices<\/em> so my full path is c:\\git\\simpletalk\\dotet-spark\\HousePrices\\HousePrices\\bin\\Debug\\HousePrices.exe<\/em><\/td>\n<\/tr>\n
The full path to the jars that are included in the Microsoft.Spark NuGet package<\/td>\n<\/td>\nBecause I created my solution in c:\\git\\simpletalk\\dotnet,<\/em> my path is C:\\git\\simpletalk\\dotet-spark\\HousePrices\\packages\\Microsoft.Spark.0.2.0\\jars\\Microsoft-spark-2.4.x-0.2.0.jar<\/em>
\n(Note it is the full path including the name of the jar, not the path to where the jars are located) If your NuGet package is version 0.0.3 or something else then the name of the jar will be more like: packages\\Microsoft.Spark.0.3.0\\jars\\Microsoft-spark-2.4.x-0.3.0.jar<\/em> \u2013 every change to the NuGet package will cause this version to change.<\/td>\n<\/tr>\n
The full path to the downloaded house prices csv<\/td>\n<\/td>\nIn my example it is c:\\users\\ed\\Downloads\\pp-monthly-update-new-version.csv<\/em><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
In a command prompt that has these environment variables set, run the next command. (If you still have the spark-shell session open in your command prompt, close it using :q<\/code>).<\/p>\n
spark-submit --class org.apache.spark.deploy.DotnetRunner --master local \"C:\\git\\simpletalk\\dotet-spark\\HousePrices\\packages\\Microsoft.Spark.0.2.0\\jars\\Microsoft-spark-2.4.x-0.2.0.jar\" \"c:\\git\\simpletalk\\dotet-spark\\HousePrices\\HousePrices\\bin\\Debug\\HousePrices.exe\" \"c:\\users\\ed\\Downloads\\pp-monthly-update-new-version.csv\"<\/pre>\n
If your executable isn\u2019t called HousePrices.exe<\/em>, then replace that with the name of your program. When you build in Visual Studio, the output window should show the full path to your built executable. If you aren\u2019t called \u201ced\u201d then change the path to the CSV file, and if you decided to use Spark 2.3 rather than Spark 2.4, then change the version of the jar.<\/p>\n
The Scala code looks in the current working directory and any child directories underneath it to find HousePrices.exe<\/em>. To see how it does that, you can look at the function<\/a> resolveDotnetExecutable<\/code>. You can change the directory in your command prompt to your Visual Studio output directory and run it from there or be more specific in your command line.<\/p>\n
Note also that the version of the jar increases with each version of Spark, and because the version is part of the filename, I used 0.3.0 for this article, but new versions are released quite regularly:<\/p>\n
Spark-submit \u2013class org.apache.spark.deploy.DotnetRunner --master local PathToMicrosoftSparkJar PathToYourProgram.exe PathToYourCsvFile.CSV<\/pre>\n
If you run the command line successfully you should see:<\/p>\n
<\/p>\n
The interesting parts are the schema from dataFrame.PrintSchema()<\/code>:<\/p>\n
<\/p>\n
The first twenty rows from dataFrame.Show()<\/code>:<\/p>\n
<\/p>\n
Finally, the results of the Sum<\/code>:<\/p>\n
<\/p>\n
You may get a lot of Java IO exceptions such as:<\/p>\n
<\/p>\n
To stop these, in your Spark<\/em> folder there is a conf<\/em> directory. In the conf<\/em> directory, you will have a log4j.properties<\/em> file add these lines to the end of the file:<\/p>\n
log4j.logger.org.apache.spark.util.ShutdownHookManager=OFF\nlog4j.logger.org.apache.spark.SparkEnv=ERROR<\/pre>\n
If you don\u2019t have a log4j.properties<\/em> you should have a log4j.properties.template<\/em>, copy it to log4j.properties<\/em>.<\/p>\n
A Larger Example<\/h2>\n
The first example was very basic, and the file doesn\u2019t contain column header, so they are set to _c0<\/code>, _c1<\/code> etc. which isn\u2019t ideal. Also, the output from PrintSchema<\/code> shows that every column is a string.<\/p>\n
The first this to do is to get Spark to infer the schema from the csv file, which you do by adding the option inferSchema<\/code> when reading the csv. Change the line (line 15 in my program):<\/p>\n
            var dataFrame = Spark.Read().Csv(args[0]);<\/pre>\n
into:<\/p>\n
            var dataFrame = Spark.Read().Option(\"inferSchema\", true).Csv(args[0]);<\/pre>\n
Build your .net application and re-run the spark-submit command line which now causes PrintSchema()<\/code> to show the actual data types:<\/p>\n
<\/p>\n
Because you now know the data types, it goes on to break the GetAs<Double>(0)<\/code> with an Unhandled Exception: System.InvalidCastExceptionL Specified cast is not valid<\/em> so you also need to change the GetAs<double><\/code> to GetAs<long><\/code>, from:<\/p>\n
            var sum = sumDataFrame.Collect().FirstOrDefault().GetAs<Double>(0);<\/pre>\n
into:<\/p>\n
            var sum = sumDataFrame.Collect().FirstOrDefault().GetAs<long>(0);<\/pre>\n
You can test that the program now completes by building in Visual Studio and re-running the spark-submit command line.<\/p>\n
It would be good to have the correct column headers rather than _c0<\/code>, to do this, read the data frame and then re-read the data frame passing in the headers \u2013 this doesn\u2019t cause the data to be re-read or re-processed, so it is efficient. If you use this program which reads the data frame, prints the schema and then converts the data frame to a data frame with headers and re-prints the schema, you should see the original _c* column names and the corrected column names:<\/p>\n
using System;\nusing System.Linq;\nusing Microsoft.Spark.Sql;\nnamespace HousePrices\n{\n    class Program\n    {\n        static void Main(string[] args)\n        {\n            var Spark = SparkSession\n                           .Builder()\n                           .GetOrCreate();\n            var dataFrame = Spark.Read().Option(\"inferSchema\", true).Csv(args[0]);\n            dataFrame.PrintSchema();\n            dataFrame.Show();\n            dataFrame = dataFrame.ToDF(\"file_guid\", \"price\", \"date_str\", \"post_code\", \"property_type\", \"old_new\", \"duration\", \"paon\", \"saon\", \"street\", \"locality\", \"town\", \"district\", \"county\", \"ppd_Category_type\", \"record_type\");\n            dataFrame.PrintSchema();\n        }\n    }\n}<\/pre>\n
Build your .net application and then re-run your spark-submit command line and you should see the correct column names:<\/p>\n
Going further, you can use the column names to filter the data. KENSINGTON AND CHELSEA<\/em> is a beautiful part of London, see how much houses in that area cost to buy:<\/p>\n
using System;\nusing Microsoft.Spark.Sql;\nnamespace HousePrices\n{\n    class Program\n    {\n        static void Main(string[] args)\n        {\n            var Spark = SparkSession\n                           .Builder()\n                           .GetOrCreate();\n            var dataFrame = Spark.Read().Option(\"inferSchema\", true).Csv(args[0]);\n            \n            dataFrame = dataFrame.ToDF(\"file_guid\", \"price\", \"date_str\", \"post_code\", \"property_type\", \"old_new\", \"duration\", \"paon\", \"saon\", \"street\", \"locality\", \"town\", \"district\", \"county\", \"ppd_Category_type\", \"record_type\");\n            \n            dataFrame = dataFrame.Where(\"district = 'KENSINGTON AND CHELSEA'\");\n            Console.WriteLine($\"There are {dataFrame.Count()} properties in KENSINGTON AND CHELSEA\");\n            dataFrame.Show();\n            \n        }\n    }\n}<\/pre>\n
Build the .net application and run the spark-submit command line and you should see something like:<\/p>\n
<\/p>\n
In case you are struggling with the amount of output, you can hide the Info messages by going back to the log4j.properties<\/em> file located in the extracted spark directory and the conf folder inside that. Change the line:<\/p>\n
log4j.rootCategory=INFO, console<\/pre>\n
into:<\/p>\n
log4j.rootCategory=WARN, console<\/pre>\n
You will see warnings and output but not all the info messages. I would say it is generally better to leave the info messages on, so you get used to what is normal and learn some of the terminology that spark uses.<\/p>\n
This new program runs quickly, but Spark is great for processing large files. It\u2019s time to do something a little bit more complicated. First, download the entire history of the price paid data<\/a>. Download the Single File<\/em> or the complete Price Paid Transaction Data as a CSV file<\/em>, currently here<\/a>.<\/p>\n
You can then change the program, so instead of just filtering, it filters and then groups by year and gets a count of how many properties sold per year and the average selling price. One of the features of Spark is that you can use the methods found in Scala, Python, R, or .NET or you can write SQL.<\/p>\n
The date must be an actual date, but even with the inferSchema<\/code> option set to true, it\u2019s still a string rather than an exact date. To correct this, add an extra column to the data set which is the date cast to an actual date:<\/p>\n
            dataFrame = dataFrame.WithColumn(\"date\", dataFrame.Col(\"date_str\").Cast(\"date\"));<\/pre>\n
If you build this and then run the spark-submit command line, you should see the extra column:<\/p>\n
using System;\nusing Microsoft.Spark.Sql;\nnamespace HousePrices\n{\n    class Program\n    {\n        static void Main(string[] args)\n        {\n            var Spark = SparkSession\n                           .Builder()\n                           .GetOrCreate();\n            var dataFrame = Spark.Read().Option(\"inferSchema\", true).Csv(args[0]);\n            dataFrame = dataFrame.ToDF(\"file_guid\", \"price\", \"date_str\", \"post_code\", \"property_type\", \"old_new\", \"duration\", \"paon\", \"saon\", \"street\", \"locality\", \"town\", \"district\", \"county\", \"ppd_Category_type\", \"record_type\");\n            dataFrame = dataFrame.WithColumn(\"date\", dataFrame.Col(\"date_str\").Cast(\"date\"));\n            dataFrame.Show();\n        }\n    }\n}<\/pre>\n
To query the data using SQL syntax rather than just using .Net methods as shown up to until now, you can save the DataFrame<\/code> as a view. This makes it available to be queried:<\/p>\n
dataFrame.CreateTempView(\"ppd\");<\/pre>\n
You can then query the view from SQL:<\/p>\n
            Spark.Sql(\"select year(date), avg(price), count(*) from ppd group by year(date)\").OrderBy(Functions.Year(dataFrame.Col(\"date\")).Desc()).Show(100);<\/pre>\n
This runs the SQL query<\/p>\n
select year(date), avg(price), count(*) from ppd group by year(date)<\/pre>\n
It then orders the results by date descending and shows the last 100 years (the data only goes back to 1995 so you won\u2019t see 100 years of data).<\/p>\n
using System;\nusing Microsoft.Spark.Sql;\nnamespace HousePrices\n{\n    class Program\n    {\n        static void Main(string[] args)\n        {\n            var Spark = SparkSession\n                           .Builder()\n                           .GetOrCreate();\n            var dataFrame = Spark.Read().Option(\"inferSchema\", true).Csv(args[0]);\n            dataFrame = dataFrame.ToDF(\"file_guid\", \"price\", \"date_str\", \"post_code\", \"property_type\", \"old_new\", \"duration\", \"paon\", \"saon\", \"street\", \"locality\", \"town\", \"district\", \"county\", \"ppd_Category_type\", \"record_type\");\n            dataFrame = dataFrame.WithColumn(\"date\", dataFrame.Col(\"date_str\").Cast(\"date\"));\n            dataFrame.CreateTempView(\"ppd\");\n            \n            var result = Spark.Sql(\"select year(date), avg(price), count(*) from ppd group by year(date)\").OrderBy(Functions.Year(dataFrame.Col(\"date\")).Desc());\n            result.Show(100);\n        }\n    }\n}<\/pre>\n
You can then run this against the full dataset:<\/p>\n
spark-submit --class org.apache.spark.deploy.DotnetRunner --master local[8] Microsoft-spark-2.4.x-0.2.0.jar HousePrices.exe c:\\users\\ed\\Downloads\\pp-complete.csv<\/pre>\n
You can also change how many cores the processing takes. Instead of --master local<\/code> which uses one single core by itself, use --master local[8]<\/code> or whatever number of cores you have on a machine. If you have lots of cores, use them.<\/p>\n
When I ran this on my laptop with eight cores, it took 1 minute 45 seconds to complete, and the average house price in that area is about 2.5 million pounds:<\/p>\n
<\/p>\n
Conclusion<\/h2>\n
Using .NET for Apache Spark brings the full power of Spark to .NET developers who are more comfortable writing C# or F# than Scala, Python, R or Java. It also doesn\u2019t matter whether you are running Linux or Windows for your development.<\/p>\n
Source Code<\/h2>\n
I have included a working copy of the final version of the application<\/a> on GitHub. In the git repo, there are .NET Framework and core versions of the solution. If you use the .NET core version, then executing the program is the same except instead of HousePrices.exe<\/em>, you need to have dotnet HousePrices-core.dll<\/em> before the path to the CSV file:<\/p>\n
spark-submit --class org.apache.spark.deploy.DotnetRunner --master local PathTo\\Microsoft-spark-2.4.x-0.2.0.jar dotnet PathTo\\HousePrices-Core.dll c:\\users\\ed\\Downloads\\pp-monthly-update-new-version.csv<\/pre>\n
References<\/h2>\n
https:\/\/spark.apache.org\/<\/a><\/p>\n
https:\/\/github.com\/dotnet\/spark<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Note: The .NET for Apache Spark project was archived in 2024. This article covers the original setup and usage of Microsoft.Spark for .NET developers – a useful reference for understanding .NET\/Spark integration concepts.…<\/p>\n","protected":false},"author":59464,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[143538],"tags":[],"coauthors":[11314],"class_list":["post-85005","post","type-post","status-publish","format-standard","hentry","category-dotnet-development"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/85005","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/59464"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=85005"}],"version-history":[{"count":10,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/85005\/revisions"}],"predecessor-version":[{"id":109670,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/85005\/revisions\/109670"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=85005"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=85005"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=85005"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=85005"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}