Rest Roadmap

This post serves as a preamble for the development of the rest.nvim rewrite with which it’s expected to achieve an extremely notable improvement in speed, reliability, extensibility and stability of the plugin.

This is a spiritual successor of an old .norg document I had with specs. None of these changes are final, everything is subject to change.

Current state

First of all, we have to talk about the current state of the project. Let’s see what’s wrong, how can we address these issues with a rewrite, and what will happen to the current pull requests.

What is wrong with the current codebase?

1. rest.nvim was my first plugin for Neovim — which has already been around for 3 years, how quickly time flies. Because of that, it’s not as good code as what I would currently write, for example. In the same way, the code has been evolving without many changes, which is why we may have a couple of limitations or complications when wanting to change something internally, such as the HTTP parsing system.
2. We are currently using a cURL wrapper created in the plenary.nvim library. This wrapper worked well when I started working on rest.nvim, but currently it is too basic for the needs of the project and working on improving it is not worth that much of the effort.
3. The HTTP/1.1 specification used in current codebase (RFC 2616) by design decision has been obsolete for a long time and limits us.
4. The view of the results can be improved a lot, but this requires a bit of redesign.

How can we address these issues with a rewrite?

Given the previous point, I think we can get an idea of ​​why a rewrite is needed, right? However, let’s briefly explain some solutions that we will expand on later.

1. Having a tree-sitter parser that already provides us with the information from the HTTP document we can optimize the obtaining of the requests and pass them to cURL more quickly, instead of using the current method that requires doing the parsing ourselves. Additionally, this would help us clean up the code a bit and reduce the parsing and execution process.
2. Using a library with native cURL bindings using the Lua library Lua-cURL we will be able to have a much complete and native integration with cURL. To install this library we will use Luarocks. This will be explained in more depth later.
3. Change the HTTP/1.1 semantic specifications used to more current ones. We could use RFC 7231, which was published in 2014, or its successor, RFC 9112, which was published a little over a year ago.
4. The Neovim ecosystem has grown a lot in recent times, breaking down some limitations such as image rendering in buffers, and rest.nvim must adapt to new changes that bring improvement.

What will happen to the current pull requests?

Most likely only Pull Requests with bug fixes will be accepted, and the current version of rest.nvim will enter a maintenance phase (also known as maintenance mode) while the new version is being worked on.

But why this? Most of the Pull Requests that are still open are new features, but they conflict with the current changes in the main branch and we cannot merge them and in some of them the authors have not given signs of life.

However, those considered necessary or useful for the new version will be added manually and the original author will be included in the commits as Co-authored-by so that they continue to have their merit in participating in the project.

v2.0 - Thunder Rest

Version 2.0 of rest.nvim, which I have named Thunder Rest, will not be completely ready until approximately the first two months of 2024 at the latest. This ETA may seem long, and will probably be reduced if no issues or time constraints are encountered.

The Thunder Rest release will follow the full versioning of semver. This means, first it will enter an alpha phase, then it will go to beta when it is more stable, going through release candidates and finally reaching a final version which is completely stable and usable.

Note: we will refer to v2.0 as Thunder Rest throughout the specifications.

Specifications

This part of the post will be a bit long, so grab some coffee and get ready to see everything that is planned for the next release!

Installation

Installing rest.nvim is possible and preferably through Luarocks and rocks.nvim. This is because we will be using some dependencies that are there, such as Lua-cURL. In addition, Luarocks makes our work easier by having the ability to manage dependencies for us.

Installing rest.nvim will also still be possible through Git and plugin manager like lazy.nvim, but you will have to install the dependencies manually!

Notes:

1. If you are using lazy.nvim, you will not be able to use rocks.nvim to install rest.nvim due to the hijacks that lazy.nvim makes to the Neovim runtimepath by design.
2. If for some reason you still use packer.nvim as your plugin manager, you can install the dependencies directly from your use() declaration.
3. Instructions for manually installing dependencies without using rocks.nvim will be available in the README.

Documentation first and everywhere

Documentation is the most essential part of the software, especially if it is shared with someone else, as this is the pillar of understanding it.

This is why Thunder Rest will have extensive documentation inside and outside the code to ensure a positive coverage of understanding of how the project works and why it works the way it works.

If you currently look at the v1.0 code you will see little documentation in some parts that should have it, which can make it difficult to quickly understand the code. This is mostly due to the first point in the section explaining what is wrong with the current codebase, and it is something that must be changed. Therefore, Thunder Rest will have the documentation as a first-class citizen.

Goodbye setup, thanks for everything!

This release will say goodbye to the way rest.nvim was initialized, but why? There are a number of good reasons behind it, and all of them are explained in more detail in this blog post. This post basically proposes to separate the setup function into two parts, config and load respectively, in order to decouple the configuration from the plugin initialization, which is something we should have always done when writing plugins.

To avoid repeating what the post expresses here, it is better to take a look at the relevant parts of the post and then return here. It is short, precise and well explained!

In this way, configuring and loading rest.nvim would look like this:

-- Configure rest.nvim
--
-- It is probably unnecessary, since there will be sensitive defaults.
require('rest-nvim').config({
   -- Maybe there is a better way to catalog this?
   behavior = {
      -- ...
      encode_url = true,
      highlight = {
         enabled = true,
         timeout = 150,
      },
      -- ...
   },
   keybinds = {
      -- Hey, what is this? O.o
   },
   -- ...
})

-- Load rest.nvim
--
-- No parameters required here, this loads the user keybinds, set up autocommands and uses sane
-- defaults for configurations if the config function was not called at any time.
require('rest-nvim').load()

Keybindings

The way rest.nvim keybindings will change completely, to something more familiar and that works out-of-the-box. We will try to maintain the philosophy that keybindings do not get in your way or cause conflicts.

The way they currently work is not obtrusive, however sometimes people haven’t figured out how to use them because they work differently than a normal keybind. From :h using-<Plug>:

Both <SID> and <Plug> are used to avoid that mappings of typed keys interfere
with mappings that are only to be used from other mappings.  Note the
difference between using <SID> and <Plug>:

<Plug>	is visible outside of the script.  It is used for mappings which the
	user might want to map a key sequence to.  <Plug> is a special code
	that a typed key will never produce.
	To make it very unlikely that other plugins use the same sequence of
	characters, use this structure: <Plug> scriptname mapname
	In our example the scriptname is "Typecorr" and the mapname is "Add".
	We add a semicolon as the terminator.  This results in
	'<Plug>TypecorrAdd;'.  Only the first character of scriptname and
	mapname is uppercase, so that we can see where mapname starts.

To improve the quality of life of the user, Thunder Rest will create a configuration table called keybinds in which the user can define their keybindings in a practical and convenient way. This would be an example of how keybinds would be created now:

-- The values ​​used here will be the default ones.
require('rest-nvim').config({
   -- See `:h vim.keymap.set()`
   keybinds = {
      -- Keys | Action | Description
      "<localleader>r", "<Plug>(RestRun)", "Run request under the cursor",
      "<localleader>r", "<Plug>(RestLast)", "Re-run the last executed request",
      "<localleader>r", "<Plug>(RestPreview)", "Preview the cURL command for the request under the cursor",
   }
})

Notes:

1. Please, if you don’t know what localleader is or how to use it, take a look at the documentation at :h <LocalLeader>
2. By default, rest.nvim will set <localleader> to , if vim.g.maplocalleader has not been defined.
3. keybinds table will automatically set rest.nvim keybinds only for .http files and will also set noremap.

Configuration

The configuration options will most likely remain the same, although some values ​​may be adjusted to make them more sane or opinionated.

An example of this would be the recently added stay_in_current_window_after_split option in #257. This option causes the focus of the buffer not to be changed to the HTTP request results buffer and although it’s very useful, it defaults to false so the default functionality is not changed. Since it’s considered a useful option, it will be enabled by default, as part of the necessary “breaking changes” to the Thunder Rest release.

Rest command

Thunder Rest will expose a Neovim command called :Rest. This command is a simple yet powerful wrapper around the HTTP client that rest.nvim is currently using under the hood (e.g. cURL or HURL).

This command works with subcommands — aka actions. Some of them work with additional and optional flags like *nix terminal utilities (e.g. wc -l). You can complete these actions by pressing <TAB> and these actions are the following:

• run
  • --last - Re-run last executed request.
  • --cursor - Run the request under the cursor.
  • --document - Run all requests found in the current HTTP document.
• last (alias to run --last to retain backwards compatibility with the old keybinds layout)
• preview

Rest functions

As :Rest command is a wrapper around the current HTTP client, it does require some functions to be exposed by ech third-party client init.lua module (which we are going to explain later). This is a public API, and can be used in place of the <Plug> mappings if you want some more extensibility.

exec

Execute or preview one or several HTTP requests depending on given scope and return request(s) results in a table that will be used to render results in a buffer.

Arguments

• scope: string
  • Defines the request execution scope. Can be: last, cursor or document.
• preview: boolean
  • Whether execute the request or just preview the command that is going to be ran. Default is false.

Returns

• table
  • Request results (HTTP client output)

Lua-cURL and Luarocks

To use cURL in the way the project currently requires it and have a better experience for both rest.nvim developers and users, Thunder Rest will abandon the use of the cURL wrapper made in plenary.nvim and will embrace native bindings to libcURL for Lua 5.1.

rest.nvim will also embrace Luarocks as a first-class citizen, due to a few reasons of which a couple are listed below:

• Ease of installation, true versioning management.
• Automatic management of dependencies and build steps.

If you want to dig a little deeper into the benefits of this decision and the problems it fixes for us as plugin developers and also for you as a user, please take a look at the following blog posts.

• Automatic install of neovim plugin dependencies
• Automatic install of neovim plugin dependencies (2)
• Publish your Neovim plugins to LuaRocks

Third-party clients

Thunder Rest is going to provide an API to interact with third-party add-ons, written by the rest-nvim community or any person. Something similar to nvim-cmp completion sources.

You can checkout #144 for more context about the initial discussion for the implementation of third-party clients integrations.

This API is going to be built-in in the rest.nvim source code. As an example, let’s suppose we are adding support for official postman CLI:

We are going to add rest-postman as a rest.nvim dependency, where rest-postman add-on has the following structure:

lua/
└── rest-nvim/          # Root rest-nvim module directory
   └── add-ons/         # Add-ons directory so we can require("rest-nvim.add-ons.foo")
      └── postman/      # Postman CLI add-on module directory
         ├── init.lua   # Postman CLI add-on module init file
         └── utils.lua  # Postman CLI add-on module utilities file

We can see that it does have a standard structure (rest-nvim/add-ons/client-name), this is essential as rest.nvim will look for an add-ons module named like the third-party client specified in the user config function.

An add-on can internally contain any files additionally to the mandatory init.lua file, as we can see in the example.

Add-ons will also work with data extracted from HTTP documents by the rest.nvim core parser, that means add-ons are going to directly work with tree-sitter data. That way, add-ons maintainers will only need to care about integrating the request data (request method and URL, headers, body, etc) in the third-party client arguments when running it.

An user will be able to choose what third-party client he wants to use by installing add-ons and then choosing the add-on on his rest.nvim config function like this:

require("rest-nvim").config({
   client = "postman",  -- default is "curl"
})

This way we keep things simple for end users and rest.nvim codebase, everything should work automatically!

Third-party clients init file

As we already know, an init.lua is a mandatory file for rest.nvim add-ons as it does handle the Lua module initialization and how it behaves. However, do we already know what should that init.lua file contain and return to rest.nvim when requiring it? Well, this is what we are going to learn now.

As you may know, rest.nvim does expose commands to Neovim (e.g. :Rest run) that triggers different actions like: running request under the cursor, preview a request, etc. All third-party clients integrations should keep an standard with the built-in cURL integration. That means, add-ons must return several functions like run and preview as these functions are called during :Rest command executions.

Note: At the time of writing this section, integrations should return the functions mentioned in the Rest functions section.

Hello, Tree-Sitter!

Thunder Rest will contain a parser module. This module is going to handle all the HTTP documents parsing logic. That means, rest-nvim.parser task is extracting the information of the current request under the cursor or all requests in the HTTP document, as requested by the end user. The extracted information is the following:

• Request method and URL (e.g. POST http://localhost:8080/api/v1/user/create)
• Headers
• Body

Note that not everything can be directly handled by tree-sitter. For example, document variables (@url = http://localhost:8080) and their expansion, this is why Thunder Rest also helps tree-sitter to parse the HTTP document using utility functions. This means that both work together in order to get work done.

Extracted information is returned by rest-nvim.parser as a structured Lua table, here is an example:

HTTP document:

@API_PATH = api/v1

POST http://localhost:8080/{{API_PATH}}/users/create
Content-Type: application/json

{
   "email": "example@mail.xyz",
   "username": "NTBBloodbath",
   "password": "3ncryptEd_p@s2w0rd"
}

rest-nvim.parser output:

{
   request = {
      method = "POST",
      url = "http://localhost:8080/api/v1/users/create",
   },
   headers = {
      ["Content-Type"] = "application/json",
   },
   body = {
      email = "example@mail.xyz",
      username = "NTBBloodbath",
      password = "3ncryptEd_p@s2w0rd",
   }
}

Please note how rest-nvim.parser automatically expanded environment variables and returns only what the HTTP clients needs in order to make the HTTP request.

Also, in case of parsing whole file with multiple requests, rest-nvim.parser will return a table with nested tables (array of tables) where parsed_document[1] is the first request and so on. Each nested table has exactly the same structure as the example output table above.

Parser

Parsing HTTP documents is something like rest.nvim’s heart and brain — it does nothing without parsing them. This is where we implement what the Hello, Tree-Sitter! section does.

Right so we would usually think something like “we need to make a parser here”, and that is true, however, we already got a core Tree-Sitter parser for our HTTP documents. Sadly this is not enough for us, but why?

This is due to what rest.nvim is capable of, this means, rest.nvim can handle any kind of variable and expand it (document-scoped, environment, etc). This is something tree-sitter cannot directly do so we need to make what I like th call a layer on top of that tree-sitter parser.

This layer is responsible for doing the dirty work that the tree-sitter parser cannot do by itself due to its nature. What is this dirty work, just reading and expanding environment variables in the document requests? Not really, it does much more than just that.

Variables

As we may know, our tree-sitter parser should be capable of understanding what a variable is and its scope so we can help our parser to expand it later so we can actually use the value it holds.

These variables can have different scopes that not only depends on the parser and we must also scan some external files (e.g. .env) in order to find their values!

So we can say that variables rules are the following:

1. Document-scoped variables
   1. Variables that are declared directly in our HTTP document (@API_PATH = api/v1).
2. Environment variables
   1. System-wide or shell session variables (e.g. $HOME).
   2. Variables coming from an environment file (e.g. .env).
3. Dynamic variables
   1. Special rest.nvim variables like uuid that gets evaluated on execution.

Contributing

rest.nvim

We highly encourage you to work with the following coding style in order to keep codebase consistency and also the following contribution workflow in order to be more organized.

Coding style

There is a .editorconfig file in the root of rest.nvim repository, everything should be automatically set if you are using at least Neovim >= 0.9, as EditorConfig is builtin from that version. rest.nvim uses the following style:

• 2 spaces indentation
• spaces over tabs
• double quotes over single quotes
• 120 characters as the maximum line length
• always use parentheses on function calls

There is also a stylua configuration file in case you want to use stylua formatter after you are done with your changes to the codebase.

Workflow

This is not mandatory, however, it does bring organization and a better readability to your codebase changes!

• A commit should do only one thing.
  • e.g. A commit that fixes a bug should only fix that bug.
• A new feature/fix, a new branch.
  • Every new feature or bug fix should be self-contained.

Important: verification and signing of commits will likely be required for future contributions.

Third-party HTTP clients integration

This point has not yet been completely decided, so a separate post will most likely be made when the time comes.

Special thanks

I want to say a huge thank you to all the people who have been donating lately to support rest.nvim, this wouldn’t have been possible if it weren’t for you. Stay tuned for changes! 💜

If you think I’ve missed something, please let me know through one of my contact forms at About me.