Since everyone is sharing shells written in Rust, I've become quite fond of Nushell: https://www.nushell.sh/
I'd love to see more shell exploring things beyond POSIX. Text based stdin/stdout will always have its place, but having ways to express, serialize, and pass along data in more structured ways is quite nice.
+1 on nushell, it is incredible. Having all data typed and structured is an insane superpower.
Common worries I hear from people that were non-issues in practice:
- Not POSIX compatible: Nushell doesn't aim to replace POSIX, there's no problem with dropping back to bash to execute some shell snippets
- You need to re-learn everything: I'm not a huge fan of how the commands are organized, but I still didn't find it that difficult. nushell's prompt/readline comes with an inline help menu with fuzzy search. Hit CTRL+O to edit a command/pipeline in your IDE/editor of choice with LSP backed intellisense, type-checking and in-editor command docs/help. The syntax is very simple and intuitive.
- Just use python: Sure, but python comes with a lot of disadvantages. It's slow and uses dynamic typing. Static typing in nushell catch typos in pipelines & scripts before they execute. It also makes in-shell and IDE LSP tab-completions very accurate. Large files process quickly though it will still consume more memory if you aren't able to process all the data in a streaming fasion. It's like having jq but with autocomplete and it works on all command output & shell variables. Though if you really like python, check out Oil/OSH/YSH: https://oils.pub/
- All Unix commands output text, structured data is useless in a shell: `detect columns` (https://www.nushell.sh/commands/docs/detect_columns.html) - now it's structured. Or use `from <format>` if the command outputs CSV, JSON, INI, YAML, etc... Or don't, cause GNU tools work fine in nushell if you keep everything in text format
And there are other crazy features too.
- Write nushell plugins in your language of choice, plugins can work with structured data
- Plugins can run in the background and maintain state, nushell can automatically start a plugin when it is first used and stop the plugin when it is idle
- e.g. a plugin can open a SQL connection and use it across multiple commands. There's a built-in plugin for opening in-mem/on-disk SQLite databases
- Data can carry metadata, e.g. binary data can carry its mime type, strings often carry metadata about which line and file the string was read from.
- Ongoing work on DAP suppprt to allow debugging scripts from your IDE
- Create your own hooks to customize how different types of data are displayed. Display structured data in table/tree form, display binary data in hex, etc...
- Collect related commands/variables into modules. Load a module knowing that you can easily unload the whole module later, module contents don't pollute global state. Variable declarations, env vars and loaded modules are scoped to the current code block and disappear after the closing bracket, lowering the odds of a name collision.
- Native support of Polars dataframes to work with even moar data
- Complex parllelism: Message-passing/actor architecture background jobs. Turn-key parallelism: transform every element of a list in parallel - `par-each` (https://www.nushell.sh/commands/docs/par-each.html)
The biggest downside of nushell is that it hasn't hit 1.0 yet so commands occasionally get renamed. Expect that you may occasionally need to tweak a script to get it working again. Definitely a pain point.
Nushell is fun. When I tried it years ago, it was only half-baked.
But a lot of the structured data transformation use cases I encounter, I find myself tackling in DuckDB. It's a little harder for the simplest things, but it pulls ahead quickly. Or at least it does if you need to remember SQL anyway...
Ah, it underwent a complete rewrite somewhat recently, I heard it was a significant improvement.
But I must agree that dedicated data analysis tools like DuckDB and jq are more powerful, intuitive, and performant. I guess what makes nushell appealing is how the data is already in nushell. It's where I stash any inputs I plan to use, any output commands produce and also any datasets I'm currently working on.
The true value of nushell is it's role as a data exchange that preserves typing+structure and in providing tools so ingesting structured data is easy and parsing unstructured data is not daunting. I'm less pushing for nushell specifically and more hoping that it encourages more people ro think about some larger questions:
- It's time to question the role of "UTF-8 text" as "the basic fundamental unit of data in the POSIX ecosystem"
- Typed/structured data brings significant value and is not harder to work with
- How can we improve data interchange between tools/apps without causing breakage? There's been quite a bit of thinking on if CLI tools can negotiate with each other to switch to communicating using higher-level data formats. Optimally it should work over common transports like SSH too. Unfortunately, I haven't seen any proposals that don't also introduce new problems. The nushell authors are looking into this as well.
- How can we evolve terminals from the simple, reliable text renderers (that they never were) into simple, reliable renderers of general structured data?
I tried really hard to get into Nushell but gave up after a month or two. Muscle memory with backgrounding was my big issue -- I tend to edit in helix and then background for a while and then foreground, and I seem to recall this totally freezing or crashing nushell somehow. Tried to learn some kind of recommended alternate workflow with pqueue (? I think) but just couldn't get there.
Yeah prior to background job support, pausing the foreground process with CTRL-Z was a major footgun—nushell didn't detect when the foreground process was paused, nor did it offer any way to unpause it. The terminal was rendered useless until you manually resumed/killed the paused process via another terminal or GUI.
Powershell feels like it was designed by people who expect the underlying app to natively structure its data before handing it off to the shell. Typical Microsoft, acting like they own the place and expecting others to conform.
Nushell on the other hand takes the (IMO more pragmatic) stance that the underlying app will most likely be writing strings to stdout and it's the shell's job to make it easy to discern the structure in those strings.
Perhaps a powershell wizard can show me that I'm wrong about this, but my feeling is that the powershell equivalent to this nushell is going to either call some external program (in addition to docker) or be quite messy:
$ docker ps | detect columns | where NAME == "foo"
I think the Powershell cmdlet "ConvertFrom-Csv" does functionally the same thing as "detect columns" in nushell. Though in this case it's probably better to make docker output structured data using "--format '{{json .}}'" and parse that instead of the human-readable representation.
Re: just asking for structured output and parsing it accordingly--you're absolutely right. Do that if you can.
But merely concatenating processes which emit and consume data of known shape is something that the OS can do without an interactive shell. What you really need a shell for is those situations where a bit of duct tape and string is needed to keep the bits flowing... when "--format '{{json .}}'" doesn't work.
Re: ConvertFrom-Csv, touche, but I'm going to guess that it's not prepared to handle cases where the delimiter also appears in the output (spaces in this case).
$ docker ps | detect columns | select CREATED STATUS | get 1 # ConvertFrom-Csv equivalent
╭─────────┬───────╮
│ CREATED │ weeks │
│ STATUS │ 9 │
╰─────────┴───────╯
Nushell's answer here is --guess, which counts characters to determine how far from the left the column starts.
$ docker ps | detect columns --guess | select CREATED STATUS | get 1
╭─────────┬─────────────╮
│ CREATED │ 2 weeks ago │
│ STATUS │ Up 9 days │
╰─────────┴─────────────╯
I'm not trying to bash pwsh, It's just that they're just trying to be different kinds of thing.
It’s available in other OSs now, but my exposure to it has mainly been windows server and endpoint environments.
YMMV, I’m mainly speaking from my own experience/history with powershell. For a long time it was the only way I knew of to manage various aspects of low level windows OS settings.
Gotcha. Speaking as someone who isn’t a fan of windows after about windows 2000, powershell is quite powerful, if not a touch verbose. Worth reading up on if you have a need to use it on any kind of cadence.
I wish nushell and pwsh got together. Structured data shells are great, scraping is brittle, but they’re both trying to do the same thing and people should be able to compile native cmdlet type things using whatever language they like. Both projects, although great, have failed independently to create a popular structured data shell. Together they could actually do it.
I'd love to see more shell exploring things beyond POSIX. Text based stdin/stdout will always have its place, but having ways to express, serialize, and pass along data in more structured ways is quite nice.