Blog

nushell – A Shell Using Structured Data 2024-11-06Permalink

For years, I've used bash and sh rather interchangeably, but recently I've been using nushell as my default shell. In this post, I'll discuss some of the reasons why nushell is slowly changing how I think about the shell and the role it plays in scripting and data processing.

What is a shell?

Shells provide a REPL-like command-line interface, enabling users to interact with the operating system. Users type in commands, and the shell interprets these commands and executes them. Not so long ago, before modern graphical user interfaces, shells were the primary means of interacting with a computer and literally formed the outer "shell" of the operating system.

More specifically, the shell allows users to:

... Yeah, that's basically it. In a nutshell2, shells are just programs to help users execute processes and manage the data flowing between them.

The Role of a Shell

In my opinion, the goal of a shell should be to stay out of the way and let users do their work as efficiently as possible. Elaborating on this concept a bit, I think that a shell should do a few things:

Let's see how nushell has taken a different approach to scripting and stream processing.

Pipelines with Structured Data

The primary feature of nushell that sets it apart from bash is that data in nushell is structured; whereas, pipelines in bash are solely a stream of bytes. Quoting the nushell website:

Nu pipelines use structured data so you can safely select, filter, and sort the same way every time. Stop parsing strings and start solving problems.

Using "structured data everywhere" is a powerful concept, and it's refreshing to see it applied to a shell. Surely, human-readable strings have an important role to play, but I'd strongly argue that data should remain structured for as long as possible, ideally until the very moment they are displayed to a user.

To make this ecosystem work, nu has re-implemented many core commands (i.e. ls and ps) to return structured data and provides an extensive list of new commands. Here's an example of the redesigned ls command piping structured data to where and sort-by commands:

Screenshot showing using the ls command

This is pretty neat for browsing the file system, but Nu is really trying to be a data processing agent. It can parse JSON, for example. I cannot even express how amazingly useful this one feature is! Nu completely replaces jq.

nushell parsing JSON

There is also a built-in http command for sending HTTP requests, and it automatically parses HTTP response bodies. Quoting the nushell website again:

Nu speaks JSON, YAML, SQLite, Excel, and more out of the box. It's easy to bring data into a Nu pipeline whether it's in a file, a database, or a web API:

Screenshot showing fetch with a web API

How about looking at the HTTP headers:

nushell showing HTTP response headers

Here's a final example of nushell piping some Markdown to Typora:

nushell Piping ls to Markdown

The to md command will convert the structured data into Markdown and then pipe the byte stream to Typora:

Typora with list of files

As you can see from these few examples, Nushell is selling itself as a general-purpose data processing ecosystem while also acting as your general-purpose shell. Even the help commands return structured data:

help commands

Extending Nushell

One of the first things I wanted to try in nushell was a replacement for grep --recursive, but such a command was not implemented yet. Naturally, I set out to implement this for myself:

# Search terms in the specified files and/or folders based on the glob pattern provided.
def "find in" [
	glob: glob, # the glob expression
	...rest: any # terms to search
]: nothing -> table<path: string, line: int, data: string> {
	glob --no-dir $glob
		| par-each {|e|
			open $e | lines | enumerate | rename line data |
			find --columns [data] ...$rest |
			each {|match| {path: ($e | path relative-to $env.PWD), ...$match}}
	} | flatten
}

If you type the above into nushell, it will create a new "find in" command. This command can then be found by typing help commands , and you can also read more about the command by issuing the --help command-line option. Here's a screenshot with all of that in action:

find in command

Nushell also supports compliled plugins to extend Nu's functionality, as well.

Clipboard Integration

Since Nushell is so good at parsing data, I have found it incredibly useful to connect it to the clipboard. First, you will need a process to copy/paste to/from the clipboard from the command line. For Linux Mint, that process is xclip, but there are various options depending on your operating system. I can simply create aliases to xclip in Nushell like so:

alias clipc = xclip -selection clipboard
alias clipv = xclip -selection clipboard -o

Then, I can copy some JSON into the clipboard and run beautiful pipelines like this:

image-20241113114824990

Conclusion

Nushell is a continuation of a paradigm shift for shells and scripting languages. It promises better data processing capabilities while maintaining the bread and butter of a shell, which is executing external processes. Admittedly, more work on Nu is needed to fully replace bash such as support for background tasks; nevertheless, I will continue to use Nu as my default shell for the foreseeable future.

Resouces

Blog Post Index

  1. Frankly, bash doesn't really allow users to browse the file system. That job is for ls and friends,which are separate processes. ↩︎

  2. pun intended. I'm not sorry. ↩︎