Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
94 commits
Select commit Hold shift + click to select a range
c4146f4
Shush the linter
mhagger Dec 17, 2023
de3225f
pipeline_test.go: remove unnecessary tmpdirs and simplify test setup
mhagger Apr 8, 2026
9fd5b2c
Add some benchmarks of moving a bunch of data through a pipeline
mhagger Dec 16, 2023
ca78f76
Simplify the `NopCloser`s
mhagger Dec 15, 2023
2f59602
Stage: change the interface to make stdin/stdout handling more flexible
mhagger Dec 15, 2023
cc9cd67
Add some tests that `Pipeline.Start()` picks the right stdin/stdout
mhagger Dec 17, 2023
6c6dfe5
Port MemoryLimitWithObserver to new Stage interface
znull Apr 8, 2026
cd47557
Restore panic handler for Function stages
znull Apr 8, 2026
48dffbd
Fix memoryWatchStage.Wait() to always call stopWatching()
znull Apr 8, 2026
92ad57c
Fix lint errors
znull Apr 9, 2026
6a0abf3
Restore identity-copy behavior for empty pipelines
znull Apr 8, 2026
9606368
Add tests pinning command stdout fd-pass fast path
znull May 28, 2026
3a1f8d8
Pool buffers for the non-*os.File command stdout copy path
znull May 28, 2026
46f19e4
Forward panic handler through wrapper stages
znull May 28, 2026
121ae94
Avoid leaking pooled-stdout goroutine when cmd.Start() fails
znull May 29, 2026
bf5820b
Bump module path to v2 for breaking Stage interface change
znull May 29, 2026
6102173
remove IOPreferenceNil
znull May 29, 2026
754784b
properly handle nil input/output
znull May 29, 2026
9dca23e
allow earlier GC of lateClosers
znull May 29, 2026
9ee30b1
export Unwrap{Reader,Writer}; unwrap goStage stdin fully
znull May 30, 2026
75797b9
rewrite cleanup/unwind in forward order
znull May 29, 2026
512c345
Thread panic handler through Start; drop StagePanicHandlerAware
znull May 29, 2026
4278c36
Recover panics escaping the memory-watch goroutine
znull May 30, 2026
9b53532
update README for v2 bump
znull May 31, 2026
f893837
Replace the MemoryLimit constructor trio with MemoryWatch + options
znull May 31, 2026
f05b5c6
memoryWatcher: new helper type
mhagger Jun 2, 2026
2cc87f4
memoryWatcher: add some methods
mhagger Jun 2, 2026
ebdc8f5
memoryWatcher.watch(): simplify loop termination
mhagger Jun 2, 2026
4ab48c3
memoryWatcher.update(): new method
mhagger Jun 2, 2026
8a96683
memoryWatcher.stage: new field
mhagger Jun 2, 2026
d49ce65
memoryWatchConfig: subsume type into `memoryWatcher`
mhagger Jun 2, 2026
0e8133e
memoryWatchStage.monitor(): only recover if there's a panic handler
mhagger Jun 2, 2026
939002c
memoryWatcher.watch(): defer Ticker.Stop()
znull Jun 2, 2026
099bd87
Test memory-watch panic handling via a booby-trapped GetRSSAnon
znull Jun 2, 2026
9967cc9
Test no-handler panic propagation without a subprocess
znull Jun 2, 2026
e20065d
TestMemoryLimitKillsEvenIfEventHandlerPanics: use constructor
mhagger Jun 3, 2026
624322b
memoryWatcher: inline type into `memoryWatchStage`
mhagger Jun 3, 2026
f3d400e
memoryWatchStage: rename all method receivers to `m`
mhagger Jun 3, 2026
073178c
memoryWatchStage: reorder method definitions more thematically
mhagger Jun 3, 2026
418316c
Merge pull request #54 from github/split/memorywatch-api
znull Jun 3, 2026
c398b7b
Decouple close-responsibility from stdin/stdout dynamic type
znull May 31, 2026
df1841a
Consolidate Env + StartOptions into a single StageOptions
znull Jun 1, 2026
2e71004
Split Stage Start streams from closers
znull Jun 5, 2026
0fed40d
Replace stage IO preferences with requirements
znull Jun 5, 2026
df7ed3e
Allow stages to declare that stdin or stdout must be absent.
znull Jun 5, 2026
24e9427
Let Function stages declare stream-presence requirements.
znull Jun 5, 2026
4675a07
Use bools for Stage stream ownership
znull Jun 5, 2026
7c0898b
Merge pull request #53 from github/split/close-decouple
znull Jun 8, 2026
be3cf9d
Remove MemoryWatch from go-pipe
znull Jun 5, 2026
5af021e
Add stage-scoped environment wrapper
znull Jun 5, 2026
ffcc348
Fix stage env option slice aliasing
znull Jun 8, 2026
f89e773
env_stage.go: Preserve command hooks through stage env wrapper
znull Jun 8, 2026
1ae6d3b
lint fix: copyloopvar
znull Jun 8, 2026
145c720
Normalize env stage tests
znull Jun 8, 2026
5538f2a
Merge pull request #56 from github/znull/remove-memorywatch
znull Jun 8, 2026
a42eaea
we don't need both processProvider and processKiller
znull Jun 8, 2026
fa89a40
convert tests to testify
znull Jun 8, 2026
bcfadc2
PR feedback: comment update
znull Jun 8, 2026
75b65f5
don't allocate more space than needed in stage slice
znull Jun 8, 2026
0a7d303
make checkStreamRequirement* helpers into methods
znull Jun 8, 2026
e02a46a
only loop once while populating stage slice
znull Jun 8, 2026
cf0cb9d
pacify linter
znull Jun 8, 2026
dfdc413
partially restore WithStagePanicHandler comment
znull Jun 8, 2026
a327b2f
Make commandStage close stdout streams if pooled stdout setup fails
znull Jun 12, 2026
f040a53
Close stdout streams when startup fails before the final stage starts
znull Jun 12, 2026
08d5b4b
Introduce stream types
znull Jun 12, 2026
c6d12cb
windows could have these, we don't know until we try
znull Jun 11, 2026
1d7a969
test Go producer behavior after downstream early exit
znull Jun 10, 2026
33a4e28
document producer pipe errors in v2 migration
znull Jun 10, 2026
ac585a8
fully document stream ownership (close responsibility)
znull Jun 12, 2026
a65fafb
Document early-close producer caveats
znull Jun 12, 2026
433e104
Align start-failure cleanup with stream ownership
znull Jun 12, 2026
642eb3e
InputStream, OutputStream: move types to a separate file
mhagger Jun 13, 2026
67ca961
InputStream, OutputStream: change `Close()` methods to pass errors th…
mhagger Jun 13, 2026
e3a8223
Stage: update docstring to reflect use of streams
mhagger Jun 13, 2026
aa3db90
InputStream, OutputStream: remove the `Closer()` methods
mhagger Jun 13, 2026
63d8e51
commandStage.Start(): rename some local variables
mhagger Jun 13, 2026
6ef82b5
InputStream, OutputStream: make these into pointer types
mhagger Jun 13, 2026
29edba3
InputStream, OutputStream: make `Close()` idempotent
mhagger Jun 13, 2026
8466274
InputStream, OutputStream: improve docstrings
mhagger Jun 13, 2026
b30a165
Pipeline: use `InputStream` and `OutputStream`
mhagger Jun 13, 2026
07cbbcf
Stage: improve docstring
mhagger Jun 14, 2026
e772ff0
StreamRequirement: move to separate file
mhagger Jun 13, 2026
fbe470e
StreamRequirement: encode also whether the stream has to be nil
mhagger Jun 13, 2026
cb3bf1d
StreamRequirement.Validate(): make method public
mhagger Jun 13, 2026
8e22ddc
stageJoiner: new helper type for creating pipes between stages
mhagger Jun 12, 2026
42a4910
Stage: do some more work on the docstring
mhagger Jun 15, 2026
f2d53ab
stageJoiner: cache each stage's Requirements()
znull Jun 15, 2026
7072f01
Pipeline.Start: validate stream requirements before creating pipes
znull Jun 15, 2026
a274f74
Merge pull request #58 from github/moar-streams
mhagger Jun 15, 2026
a86cf5f
Merge pull request #59 from github/stage-joiner
mhagger Jun 15, 2026
f8adc48
Merge pull request #57 from github/znull/v2-early-close-regression-tests
znull Jun 15, 2026
9dacf4b
fix comment typo
znull Jun 15, 2026
3d273c2
use correct code example in README
znull Jun 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 27 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,34 @@
# go-pipe [![GoDoc](https://pkg.go.dev/badge/github.com/github/docs)](https://pkg.go.dev/github.com/github/go-pipe)
# go-pipe [![GoDoc](https://pkg.go.dev/badge/github.com/github/go-pipe/v2)](https://pkg.go.dev/github.com/github/go-pipe/v2)
A package used to easily build command pipelines in your Go applications

# Important
We have not thoroughly tested this package on OSs other than Linux, especially Windows. At this time, using this package on Windows based systems is considered experimental and will be supported only on a best effort basis.

# Migrating to v2

It's normal for pipelines to stop before all input has been consumed[^1]. If an earlier stage continues writing after that happens, the write side of the pipe can fail with `EPIPE`, `SIGPIPE`, or `io.ErrClosedPipe`.

In go-pipe v1 it was possible to get away without handling this case, because a command stage's stdin was connected in a way that often (but not necessarily!) drained the write side and hid the error from the previous stage feeding it. That was an implementation detail, not a guarantee. In go-pipe v2, producer stages are more likely to be connected directly to a command's stdin, and thus see the error themselves.

Fortunately, this is easily handled by wrapping the stage with `pipe.IgnoreError(stage, pipe.IsPipeError)`. If the producer only writes output and is otherwise stateless, that's the only thing needed.

If the producer also updates state, metrics, cursors, or has other side effects, in a way that depends on how much of the output was produced, then in addition to using `pipe.IgnoreError`, you must also ensure producer-owned state is brought to a consistent point before returning the error.

For example, if a stateful producer function must process its entire input for correctness regardless of whether it was read by the consumer, it should use a pattern like:

```go
var writeErr error
for _, item := range items {
updateState(item)
if writeErr == nil {
_, writeErr = fmt.Fprintln(stdout, item)
}
}
return writeErr
```

# Links

* [Docs](https://pkg.go.dev/github.com/github/go-pipe)
* [Docs](https://pkg.go.dev/github.com/github/go-pipe/v2)

[^1]: In `cat foo | head | grep -q`, for example, either `head` or `grep` could exit before its input is fully consumed.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module github.com/github/go-pipe
module github.com/github/go-pipe/v2

go 1.24.0

Expand Down
303 changes: 0 additions & 303 deletions internal/ptree/ptree.go

This file was deleted.

21 changes: 0 additions & 21 deletions internal/ptree/ptree_linux.go

This file was deleted.

Loading
Loading