Tests
=====
Tests are broadly divided into *unit tests* ([test/unit](https://github.com/neovim/neovim/tree/master/test/unit/)),
*functional tests* ([test/functional](https://github.com/neovim/neovim/tree/master/test/functional/)),
and *old tests* ([src/nvim/testdir/](https://github.com/neovim/neovim/tree/master/src/nvim/testdir/)).
- _Unit_ testing is achieved by compiling the tests as a shared library which is
loaded and called by [LuaJit FFI](http://luajit.org/ext_ffi.html).
- _Functional_ tests are driven by RPC, so they do not require LuaJit (as
opposed to Lua).
You can learn the [key concepts of Lua in 15 minutes](http://learnxinyminutes.com/docs/lua/).
Use any existing test as a template to start writing new tests.
Tests are run by `/cmake/RunTests.cmake` file, using `busted` (a Lua test-runner).
For some failures, `.nvimlog` (or `$NVIM_LOG_FILE`) may provide insight.
Depending on the presence of binaries (e.g., `xclip`) some tests will be
ignored. You must compile with libintl to prevent `E319: The command is not
available in this version` errors.
---
- [Running tests](#running-tests)
- [Writing tests](#writing-tests)
- [Lint](#lint)
- [Configuration](#configuration)
---
Layout
======
- `/test/benchmark` : benchmarks
- `/test/functional` : functional tests
- `/test/unit` : unit tests
- `/test/config` : contains `*.in` files which are transformed into `*.lua`
files using `configure_file` CMake command: this is for acessing CMake
variables in lua tests.
- `/test/includes` : include-files for use by luajit `ffi.cdef` C definitions
parser: normally used to make macros not accessible via this mechanism
accessible the other way.
- `/test/*/preload.lua` : modules preloaded by busted `--helper` option
- `/test/**/helpers.lua` : common utility functions for test code
- `/test/*/**/*_spec.lua` : actual tests. Files that do not end with
`_spec.lua` are libraries like `/test/**/helpers.lua`, except that they have
some common topic.
- `/src/nvim/testdir` : old tests (from Vim)
Running tests
=============
Executing Tests
---------------
To run all tests (except "old" tests):
make test
To run only _unit_ tests:
make unittest
To run only _functional_ tests:
make functionaltest
Legacy tests
------------
To run all legacy Vim tests:
make oldtest
To run a *single* legacy test file you can use either:
make oldtest TEST_FILE=test_syntax.vim
or:
make src/nvim/testdir/test_syntax.vim
- Specify only the test file name, not the full path.
Debugging tests
---------------
- You can set `$GDB` to [run tests under gdbserver](https://github.com/neovim/neovim/pull/1527).
And if `$VALGRIND` is set it will pass `--vgdb=yes` to valgrind instead of
starting gdbserver directly.
- Hanging tests often happen due to unexpected `:h press-enter` prompts. The
default screen width is 50 columns. Commands that try to print lines longer
than 50 columns in the command-line, e.g. `:edit very...long...path`, will
trigger the prompt. In this case, a shorter path or `:silent edit` should be
used.
- If you can't figure out what is going on, try to visualize the screen. Put
this at the beginning of your test:
```lua
local Screen = require('test.functional.ui.screen')
local screen = Screen.new()
screen:attach()
```
Afterwards, put `screen:snapshot_util()` at any position in your test. See the
comment at the top of `test/functional/ui/screen.lua` for more.
Filtering Tests
---------------
### Filter by name
Another filter method is by setting a pattern of test name to `TEST_FILTER`.
``` lua
it('foo api',function()
...
end)
it('bar api',function()
...
end)
```
To run only test with filter name:
TEST_FILTER='foo.*api' make functionaltest
### Filter by file
To run a *specific* unit test:
TEST_FILE=test/unit/foo.lua make unittest
To run a *specific* functional test:
TEST_FILE=test/functional/foo.lua make functionaltest
To *repeat* a test:
BUSTED_ARGS="--repeat=100 --no-keep-going" TEST_FILE=test/functional/foo_spec.lua make functionaltest
### Filter by tag
Tests can be "tagged" by adding `#` before a token in the test description.
``` lua
it('#foo bar baz', function()
...
end)
it('#foo another test', function()
...
end)
```
To run only the tagged tests:
TEST_TAG=foo make functionaltest
**NOTE:**
* `TEST_FILE` is not a pattern string like `TEST_TAG` or `TEST_FILTER`. The
given value to `TEST_FILE` must be a path to an existing file.
* Both `TEST_TAG` and `TEST_FILTER` filter tests by the string descriptions
found in `it()` and `describe()`.
Writing tests
=============
Guidelines
----------
- Luajit needs to know about type and constant declarations used in function
prototypes. The
[helpers.lua](https://github.com/neovim/neovim/blob/master/test/unit/helpers.lua)
file automatically parses `types.h`, so types used in the tested functions
could be moved to it to avoid having to rewrite the declarations in the test
files.
- `#define` constants must be rewritten `const` or `enum` so they can be
"visible" to the tests.
- Use **pending()** to skip tests
([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
Do not silently skip the test with `if-else`. If a functional test depends on
some external factor (e.g. the existence of `md5sum` on `$PATH`), *and* you
can't mock or fake the dependency, then skip the test via `pending()` if the
external factor is missing. This ensures that the *total* test-count
(success + fail + error + pending) is the same in all environments.
- *Note:* `pending()` is ignored if it is missing an argument, unless it is
[contained in an `it()` block](https://github.com/neovim/neovim/blob/d21690a66e7eb5ebef18046c7a79ef898966d786/test/functional/ex_cmds/grep_spec.lua#L11).
Provide empty function argument if the `pending()` call is outside of `it()`
([example](https://github.com/neovim/neovim/commit/5c1dc0fbe7388528875aff9d7b5055ad718014de#diff-bf80b24c724b0004e8418102f68b0679R18)).
- Really long `source([=[...]=])` blocks may break Vim's Lua syntax
highlighting. Try `:syntax sync fromstart` to fix it.
Where tests go
--------------
Tests in `/test/unit` and `/test/functional` are divided into groups
by the semantic component they are testing.
- _Unit tests_
([test/unit](https://github.com/neovim/neovim/tree/master/test/unit)) should
match 1-to-1 with the structure of `src/nvim/`, because they are testing
functions directly. E.g. unit-tests for `src/nvim/undo.c` should live in
`test/unit/undo_spec.lua`.
- _Functional tests_
([test/functional](https://github.com/neovim/neovim/tree/master/test/functional))
are higher-level (plugins and user input) than unit tests; they are organized
by concept.
- Try to find an existing `test/functional/*/*_spec.lua` group that makes
sense, before creating a new one.
Lint
====
`make lint` (and `make lualint`) runs [luacheck](https://github.com/mpeterv/luacheck)
on the test code.
If a luacheck warning must be ignored, specify the warning code. Example:
-- luacheck: ignore 621
http://luacheck.readthedocs.io/en/stable/warnings.html
Ignore the smallest applicable scope (e.g. inside a function, not at the top of
the file).
Configuration
=============
Test behaviour is affected by environment variables. Currently supported
(Functional, Unit, Benchmarks) (when Defined; when set to _1_; when defined,
treated as Integer; when defined, treated as String; when defined, treated as
Number; !must be defined to function properly):
- `BUSTED_ARGS` (F) (U): arguments forwarded to `busted`.
- `GDB` (F) (D): makes nvim instances to be run under `gdbserver`. It will be
accessible on `localhost:7777`: use `gdb build/bin/nvim`, type `target remote
:7777` inside.
- `GDBSERVER_PORT` (F) (I): overrides port used for `GDB`.
- `VALGRIND`