PySpark is a powerful language for data manipulation and it\u2019s full of tricks. Let\u2019s discover some of them.<\/p>\n
If you are creating a pysspark dataframe, but one of the columns contains only null values (None), how could you control the type of the column?<\/p>\n
There is an interesting expression you can build using lit().cast()<\/em> to control the type of a column with null values.<\/p>\n The following statement will add a new column or fix the type of an existing column containing only Null values.<\/p>\n <\/p>\n <\/p>\n PySpark works very well with Big Data and blocks of data. Each operation is made to apply over an entire Dataframe.<\/p>\n In this way, updating a single row\/column brings a syntax not much traditional. This is an example:<\/p>\n <\/p>\n <\/p>\n <\/p>\n Let’s analyse the statement step-by-step:<\/p>\n In this way, the when<\/em><\/strong> clause specifies a filter, selecting one single row to receive a new value in the column we are updating. All the remaining rows will continue with the original value.<\/p>\n This example updates a single row. However, this syntax can be used as a SQL update\/where, updating multiple rows in different ways.<\/p>\n When you need to compare if a row was changed or not, one good method is to create a hash for the row and compare the hash.<\/p>\n Using just a few statements you can create a hash for the rows of a dataframe. If these rows are kept immutable, the hash can also be used as surrogate key.<\/p>\n The code below shows an example of how to create a hash for all the rows of a dataframe:<\/p>\n <\/p>\n \nOne of the most interesting parts of the code is the unpacking operator, ‘*’. Used with the df.columns collection, it unpack the collection, converting it into its individual elements. In this way, the concat_ws<\/em> receives all the columns as a parameter.<\/p>\nfrom<\/span> pyspark.sql.functions<\/span> import<\/span> lit\r\n\r\ndf=<\/span>df.<\/span>withColumn('myNewColumn'<\/span>,lit(None<\/span>).<\/span>cast('timestamp'<\/span>))\r\n<\/pre>\n<\/div>\n\u00a0<\/h2>\n
Updating a single row\/column in a Dataframe<\/h2>\n
from<\/span> pyspark.sql.functions<\/span> import<\/span> lit, col, when\r\n\r\nDF =<\/span> DF.<\/span>withColumn('fieldToChange'<\/span>, when(\r\n col(\"tableKey\"<\/span>) ==<\/span> 'uniqueKey'<\/span>,\r\n lit('literalValue'<\/span>)\r\n ).<\/span>otherwise(col(\"fieldToChange\"<\/span>)))\r\n<\/pre>\n<\/div>\n\n
Creating a Hash for Dataframe rows<\/h2>\n
df =<\/span> spark.<\/span>createDataFrame(\r\n [(1<\/span>,\"2\"<\/span>,5<\/span>,1<\/span>),(3<\/span>,\"4\"<\/span>,7<\/span>,8<\/span>)],\r\n (\"col1\"<\/span>,\"col2\"<\/span>,\"col3\"<\/span>,\"col4\"<\/span>)\r\n )\r\n\r\ndf.<\/span>withColumn(\"row_sha2\"<\/span>, \r\n sha2(concat_ws(\"||\"<\/span>, *<\/span>df.<\/span>columns), 256<\/span>))\r\n\r\ndisplay(df)\r\n<\/pre>\n<\/div>\n