Neovim under the hood - Understanding the language server client

  Tuesday, October 27, 2020 » Neovim

This is a short introduction to the language server-client architecture in Neovim and how to customize the diagnostics display.

Language server protocol introduction

Starting with 0.5, Neovim will contain a built-in client for the language server protocol (LSP).

The Language Server Protocol (LSP) was invented to decouple semantic knowledge of a programming language from the editor. The knowledge about programming languages is moved out of the editor into separate language server implementations. All editors who speak this protocol can communicate with any language server to provide functionalities like code completion to the user. The advantage is that language server implementations can then be reused across many editors.

The editors only need to implement the communication with the servers and expose the functionality to the users, rather than implementing specific functionality for each supported programming language.

The language server protocol is based on a request and response model. Either the client sends a request to the server and receives a response. Or, the server pushes a request to the client and gets a response. The latter is sometimes referred to as reverse request.

Neovim provides a few primitives to work with a language server. There are higher level functions with which users usually interact with. And at the core there is a client that handles the communication.

Client

To communicate with a server, Neovim provides a client object. There is a one-to-one relationship between one client and one language server process. The client can send requests to the language server and receive and process responses. More about how responses are handled will follow in the Callbacks section.

To create a client, the vim.lsp.start_client() method is used.

To request data from the server, the client exposes a request method, but that’s not how users typically use the functionality provided by a language server. Neovim provides higher-level functions instead.

Buffers

In Neovim, if you edit a text document, you edit it within a buffer. A buffer is the Neovim primitive for working with text. To switch between files, you switch between buffers. To edit a new file, you load a new buffer or load new text into an existing buffer. Buffers are everywhere!

For a language server client to do something useful, it needs to be attached to a buffer.

A client can be attached to a buffer using vim.lsp.buf_attach_client(). This creates a connection between a buffer and a client. Once attached to a buffer, the client starts sending notifications to the server for certain events. For example, if you save the buffer it emits a notification to the language server, telling it that you saved the document. If you write text, the client will send the server a request to tell it that you made some changes.

Neovim exposes multiple user interaction functions on the buffer level. For example, to get all the references for a given method, a user would invoke vim.lsp.buf.references(). This causes all clients attached to the current buffer to request the references for the symbol under the cursor from the running language server. Once received, they’re by default loaded into the quickfix list.

Under the hood, vim.lsp.buf.references() uses vim.lsp.buf_request() to retrieve all attached clients and then uses client.request to send a request for the references.

One of the parameters of the client.request function is the method name of the resource you want to use. The language server protocol defines many resources. The resource which can be used to retrieve references is called textDocument/references.

Callbacks

Update: callbacks has been renamed to handlers. Mentally replace callbacks with handlers in the text below. s/callbacks/handlers.

The responses the language server sends back after it receives a request are handled via different callback implementations. In Neovim, there exists a hierarchy of callbacks. The client.request method takes a callback argument. If it is present, this callback will be called once the response is received.

If it isn’t present, there will be a lookup for a callback implementation in the initial client’s config table. The callback will be looked up under the name of the method that was requested. If a user requested textDocument/references, then there will be a lookup for a callback named textDocument/references.

That there are callbacks within the client’s configuration means that each client can have its custom callbacks.

If the config of the client doesn’t have a callback definition, there is another and final fallback: Using the callbacks defined within vim.lsp.callbacks.

With a standard setup, this last lookup is usually the one that ends up being used because the typical setup doesn’t provide callbacks in the configuration, and the vim.lsp.buf. method implementations don’t specify a callback directly when using the buf_request or client.request functions.

This design allows great flexibility for customizations. But you also have to be careful: if you modify some callbacks in the wrong way, it may break other functionality.

Customizing diagnostics

Update: A diagnostic module was introduced so this content here is outdated, but the overall gist of how the handlers can be overridden is still accurate.

A client does not explicitly request diagnostics, but rather, the server sends them based on other events. For example, if you write text, the client sends a didChange notification, and based on that, the server might send new diagnostics to the client.

Neovim has a default implementation to handle diagnostics within vim.lsp.callbacks. The method name for diagnostics is textDocument/publishDiagnostics. You can see the data structure that the server sends to the client in the Diagnostics specification

To customize the functionality, you’ve two options:

  1. You add a custom implementation in the callbacks argument of the config table used in vim.lsp.start_client.

  2. You override the vim.lsp.callbacks implementation.

The second variant could look like this:

vim.lsp.callbacks['textDocument/publishDiagnostics'] = function(err, method, result)
end

This would deactivate any form of diagnostics reporting because it would start ignoring any servers’ diagnostics.

But that would also break other functionality that depends on diagnostics. For example, code actions rely on diagnostics because when code actions are requested from the server, the client needs to refer the diagnostics for which it would like to have code actions.

Let us have a look at the current implementation within Neovim. I added some comments to explain things and removed some of the real implementation comments.

vim.lsp.callbacks['textDocument/publishDiagnostics'] = function(err, method, result)
  -- if there is no result, then do nothing
  if not result then return end

  local uri = result.uri
  -- The response contains the URI which identifies the file for which the diagnostics are for
  -- Internally most functions work with buffer numbers instead to identify the "files"
  local bufnr = vim.uri_to_bufnr(uri)
  if not bufnr then
    err_message("LSP.publishDiagnostics: Couldn't find buffer for ", uri)
    return
  end

  -- The severity is optional, if it is not present the current implementation defaults to Error
  -- This ensures other functions can assume the severity is always present
  for _, diagnostic in ipairs(result.diagnostics) do
    if diagnostic.severity == nil then
      diagnostic.severity = protocol.DiagnosticSeverity.Error
    end
  end

  -- util here is really vim.lsp.util, there is a `local util = require 'vim.lsp.util'` at the top
  -- to a) avoid repeated lookup of the module, and b) avoids typing or reading some boilerplate
  util.buf_clear_diagnostics(bufnr)

  -- This stores the diagnostics in a table, so that other functions can use them.
  -- The earlier mentioned code actions command needs them.
  -- But users can also display them in various ways, or can write functions to
  -- load them into the quickfix list or similar
  util.buf_diagnostics_save_positions(bufnr, result.diagnostics)

  -- If the buffer is not loaded, then stop doing anything here
  -- The actions below are more expensive and not useful if the buffer is not active
  if not api.nvim_buf_is_loaded(bufnr) then
    return
  end
  -- This underlines text that the diagnostics refer to
  util.buf_diagnostics_underline(bufnr, result.diagnostics)

  -- This adds virtual text to display the diagnostic messages inline
  util.buf_diagnostics_virtual_text(bufnr, result.diagnostics)

  -- This displays signs in the lines where there are diagnostics
  util.buf_diagnostics_signs(bufnr, result.diagnostics)

  -- Produces a autocmd that users can subscribe to
  vim.api.nvim_command("doautocmd User LspDiagnosticsChanged")
end

You could customize this in any number of ways. For example, if you don’t like the virtual text, you’d remove the util.buf_diagnostics_virtual_text method call. If you don’t like the signs, you can remove the util.buf_diagnostics_signs method call. If you wanted to have some extra logic, you add whatever you need.