To illustrate pipelining as simply as possible, I’ll introduce a simple pipeline that takes a sequence of inputs through three successive functions, mapping a sequence of inputs to a sequence of outputs. <\/p>\n
![\"2290-img14.jpg\"](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/2290-img14.jpg\")
<\/p>\n
Figure 1:<\/p>\n
Each pipeline function, called a filter, performs a transformation on the inputs fed to it and passes on the result. The operations performed as shown are deliberately simple and nonsensical: It is purely to illustrate the process. It is the pipeline itself that deserves attention here, rather than the particular transformations. Nonetheless, here to start us off are the simple implementations of each operation so you can follow along. <\/p>\n
# double the input\r\nFunction f1($x) { $x * 2 }\r\n\r\n# concatenate the nth letter to the input where n is half the input value\r\nFunction f2($x) { \"$x\" + [char]([byte][char] \"A\" + $x\/2 - 1) }\r\n\r\n# reverse the 2-character string\r\nFunction f3($x) { $x[1..0] -join '' }\r\n<\/pre>\nAs written, these are standard PowerShell functions that are not <\/em> , by themselves <\/em> suitable for pipelining. If you try 1, 2, 3 | f1 | f2 | f3 <\/code> you will get an incorrect result. <\/p>\nRight and Wrong <\/h2>\n
Instead of writing this …. <\/p>\n
1, 2, 3 | f1 | f2 | f3<\/pre>\n… you will need to execute this: <\/p>\n
1, 2, 3 | %{ f1 $_ } | %{ f2 $_ } | %{ f3 $_ } <\/pre>\n(meaning: <\/p>\n
\n- %
<\/code> <\/strong>– for each object that arrives as input <\/li>\n-
{ <\/code>– start of process block <\/li>\n-
f1 <\/code> <\/strong>– execute the function F1 … <\/li>\n-
$_ <\/code> <\/strong>– ….on the object passed down the pipeline <\/li>\n-
} <\/code>– end of process block) <\/li>\n<\/ul>\n…or you could, as we will soon do in this article, create filters to do it all more neatly <\/p>\n
1, 2, 3 | f1Runner | f2Runner | f3Runner <\/pre>\n… where the given filters have yet to be defined. <\/p>\n
In each of the examples that follow, you will see implementations of the functions that act as filters mentioned above ( f1Runner <\/code>, f2Runner <\/code>, and f3Runner <\/code>). Within each scenario, all of the functions are essentially identical except for the statement that produces a calculation. After the calculation, the value is output twice <\/em>: once with Write-Host<\/code> , just to see what is going on, and once with Write-Output<\/code> to allow it to feed the next function in the pipeline. To be clear, if you do not care about viewing intermediate results you can delete the Write-Host<\/code> statements and the pipeline will work just the same. Also, note that the f3Runner does not use the Write-Host<\/code> call because it is at the end of the pipe and you will get the results on the console from just the Write-Output<\/code> call itself. <\/p>\nIn order to make it simpler to see what is going on, we will add another filter, showInputs <\/code>, that merely displays the values at the start of the pipeline <\/p>\nScenario #1 – Generate all pipelineable output before emitting any <\/h2>\n
The special variable $input <\/code>is available within a function as a provider of pipeline data. So we just loop through that pipeline data, calculating a result, and then writing it out. This is a very common coding pattern seen in many questions on StackOverflow. It is perfectly reasonable for some languages, but generally it should be avoided in PowerShell. Even if data is coming in nicely through the pipeline to the first function, the pipeline dries up completely, because each function is processing the entire pipeline input until the pipeline is empty, and only then sending its results onward en masse to the next pipeline participant. <\/p>\nfunction showInputs\r\n{\r\n $result = @()\r\n foreach ($value in $input) { $result += $value }\r\n Write-Host $result\r\n Write-Output $result\r\n}\r\nfunction f1Runner\r\n{\r\n $result = @()\r\n foreach ($value in $input) { $result += f1 $value }\r\n Write-Host $result\r\n Write-Output $result\r\n}\r\nfunction f2Runner\r\n{\r\n $result = @()\r\n foreach ($value in $input) { $result += f2 $value }\r\n Write-Host $result\r\n Write-Output $result\r\n}\r\nfunction f3Runner\r\n{\r\n $result = @()\r\n foreach ($value in $input) { $result += f3 $value }\r\n #Write-Host $result\r\n Write-Output $result\r\n}} <\/pre>\nHere is what happens when you execute. showInputs displays all of its inputs (1 2 3). (It is an artifact of Write-Host<\/code> that it combines all of the values in to a single, space-separated string.) Then f1Runner<\/code> displays all of its calculations (2 4 6). Then f2Runner<\/code> does all of its work, yielding (2A 4B 6C). Finally,f3Runner<\/code> lets its results stream out the end of the pipeline, so you see that there are still 3 values being processed (because they are emitted one-per-line). <\/p>\n\t\r\n\tPS> 1,2,3 | showInputs | f1Runner | f2Runner | f3Runner\r\n\t1 2 3\r\n\t2 4 6\r\n\t2A 4B 6C\r\n\tA2\r\n\tB4\r\n\tC6 \r\n<\/pre>\nNote that there is often more than one way to do the same thing in PowerShell: for example, the loop in f3Runner<\/code> could be replaced with this line (and the others similarly) and it would produce exactly the same result-namely, no pipelining! Here the special $_ <\/code>variable indicates the current item in the loop. <\/p>\n<\/p>\n$input | ForEach-Object { $result += f3 $_ }\r\n<\/pre>\nScenario #2 – Collect all pipeline input before processing any <\/h2>\n
The way to fix the previous example is to emit each value as soon as it is calculated. The point to catch from the previous example is that there’s no need for you to explicitly aggregate the results; PowerShell will implicitly do that with the pipeline itself. This set of functions seemingly does just that, emitting one value at a time: <\/p>\n
Function showInputs\r\n{\r\n $input | ForEach-Object {\r\n $result = $_\r\n Write-Host $result\r\n Write-Output $result\r\n }\r\n}\r\nFunction f1Runner\r\n{\r\n $input | ForEach-Object {\r\n $result = f1 $_\r\n Write-Host $result\r\n Write-Output $result\r\n }\r\n}\r\nFunction f2Runner\r\n{\r\n $input | ForEach-Object {\r\n $result = f2 $_\r\n Write-Host $result\r\n Write-Output $result\r\n }\r\n}\r\nFunction f3Runner\r\n{\r\n $input | ForEach-Object {\r\n $result = f3 $_\r\n # Write-Host $result\r\n Write-Output $result\r\n }\r\n}<\/pre>\nWhen you execute the test line, however, it reveals essentially the same result as before (though there are now newlines between the intermediate results since they are being emitted individually): the output is not being pipelined! <\/p>\n
\tPS> 1,2,3 | showInputs | f1Runner | f2Runner | f3Runner\r\n\t1\r\n\t2\r\n\t3\r\n\t2\r\n\t4\r\n\t6\r\n\t2A\r\n\t4B\r\n\t6C\r\n\tA2\r\n\tB4\r\n\tC6 \r\n<\/pre>\nWhile these functions do not <\/em> suffer the deficiency of the previous example (generating all the output before emitting anything), the converse is actually the problem here: these functions are waiting for all the input before calculating anything! <\/p>\nAs you learn more about advanced PowerShell functions, you’ll discover that you actually have to put code in one of three blocks: the begin, process, or end blocks. The begin block runs before <\/em> any pipeline input is accepted; the process block runs once for each pipeline input; and the end block runs after <\/em> all pipeline input has been processed. If you do not explicitly specify a block, all your code is implicitly in the end block. And that explains why this scenario did no pipelining: each function waits until the pipeline empties and the function has collected all of its inputs, then <\/em> it runs the end block code, emitting all its output in one chunk to the next pipeline participant. Thus, again we have no pipelining benefit. <\/p>\nScenario #3 – Process each input when received and emit its output promptly <\/h2>\n
Scenario #2 moved a bit closer to real pipelining, and you might surmise from the discussion above that the final piece of the problem can be resolved by moving from the end block to the process block. And that would be quite correct; the set of functions below show how. Notice there is no loop here because the process block runs once for each input; in other words, there is a loop but it is handled by PowerShell itself. Within a process block, use the special $_<\/code> variable to access the current pipeline item. <\/p>\nFunction showInputs\r\n{\r\n process\r\n {\r\n $result = $_\r\n Write-Host $result\r\n Write-Output $result\r\n }\r\n}\r\nFunction f1Runner\r\n{\r\n process\r\n {\r\n $result = f1 $_\r\n Write-Host $result\r\n Write-Output $result\r\n }\r\n}\r\nFunction f2Runner\r\n{\r\n process\r\n {\r\n $result = f2 $_\r\n Write-Host $result\r\n Write-Output $result\r\n }\r\n}\r\nFunction f3Runner\r\n{\r\n process\r\n {\r\n $result = f3 $_\r\n # Write-Host $result\r\n Write-Output $result\r\n }\r\n}<\/pre>\nAnd indeed, with this approach, the output is properly pipelined, corresponding to Figure 3 above: <\/p>\n
\tPS> 1,2,3 | showInputs | f1Runner | f2Runner | f3Runner\r\n\t1\r\n\t2\r\n\t2A\r\n\tA2\r\n\t2\r\n\t4\r\n\t4B\r\n\tB4\r\n\t3\r\n\t6\r\n\t6C\r\n\tC6 \r\n<\/pre>\nBecause the process block is so central to pipelining, there is a special syntax to reduce the amount of code you have to write. As noted above using the Function keyword with no explicit begin, process, or end block executes the code in the context of the end block. Similarly, if you use the Filter keyword with no explicit block then the code executes in the process block. Thus, this more concise code produces exactly the same result: <\/p>\n
Filter showInputs {\r\n $result = $_\r\n Write-Host $result\r\n Write-Output $result\r\n}\r\nFilter f1Runner {\r\n $result = f1 $_\r\n Write-Host $result\r\n Write-Output $result\r\n}\r\nFilter f2Runner {\r\n $result = f2 $_\r\n Write-Host $result\r\n Write-Output $result\r\n}\r\nFilter f3Runner {\r\n $result = f3 $_\r\n # Write-Host $result\r\n Write-Output $result\r\n}\r\n<\/pre>\nSummary <\/h2>\n
As you have seen, it is very easy to think you are pipelining when, in fact, you are not. You could, instead, be waiting for all input to be received, or you might be processing all received inputs before sending any output. If, however, you are aware of the underlying concepts of the pipeline, it is straightforward to get true pipelining behavior in PowerShell. <\/p>\n
\n ![\"2290-img2B.jpg\"](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/2290-img2B.jpg\")
<\/p>\n
Figure 2:<\/p>\n<\/div>\n
Compare conceptually how the data moves without <\/em> <\/strong> pipelining (scenarios 1 and 2)… <\/p>\n<\/p>\n
\n ![\"2290-img3B.jpg\"](\"https:\/\/www.red-gate.com\/simple-talk\/wp-content\/uploads\/imported\/2290-img3B.jpg\")
<\/p>\n
Figure 3:<\/p>\n<\/div>\n
and with <\/em> <\/strong> proper pipelining: <\/p>\n<\/p>\nFor many situations-and particularly for time-consuming operations-you want to use proper pipelining so your end-user starts getting results as soon as possible. On the other hand, some operations actually require you to collect up the input and perform some operation on them together. Sorts are a classic example of something that can’t be done on an object-by-object basis: there is, of course a Cmdlet that does it for you but if there wasn’t … <\/p>\n
5, 6, 7, 2, 3, 0, 9 |% -begin { $v = @() } { $v += $_ } -end { [array]::Sort($v); $v } <\/pre>\nI would be remiss, however, if I did not mention that the filter solution is not optimal in all situations. If you only <\/em> need data to come through the pipeline, it is sufficient. But PowerShell provides the capability to accept input from either the pipeline or as direct arguments. That is, you often want to be able to execute either of these and get the same output: <\/p>\n\tPS> 1,2,3 | showInputs\r\n\tPS> showInputs 1,2,3\r\n<\/pre>\nAlas, the filters above will not <\/em> accept direct arguments. Here is a template for showInput that does (the other 3 filters can be adapted just as before, by changing the name and the calculation line). It is a bit more complicated, but that’s the trade-off for being more versatile. <\/p>\nFilter showInputs(\r\n[Parameter(ValueFromPipeline = $True)]\r\n[array]$item)\r\n{\r\n $item | ForEach-Object {\r\n $result = $_\r\n Write-Host $result\r\n Write-Output $result\r\n }\r\n}<\/pre>\n