Rust from a Gopher - Lesson 11 - Testing

Hello and welcome to the seventh post in my series about learning Rust. In case you want to hit it from the start, here’s a link to the first one! This entire series covers my journey from being a completely land-locked Gopher to becoming (hopefully) a hardened Rustacean, able to skitter the hazardous seabed of any application safely.


This chapter covers a topic that engineers will have strong thoughts and feelings about, myself included. In college, I recall learning precisely zero about testing. This perhaps, because I studied Computer Science rather than Software Engineering? Alas, when I entered into my first real dev job it was a slightly jarring experience, figuring out how write tests for my code. In that nubile period, I can make no mistake in saying that test writing was the most tedious and painstaking task I can recall… (Alongside waiting 2 minutes and thirty seconds for a our Java backend to compile)

Mercifully, I’ve encountered many great engineers throughout my career whom took time to educate me, and I’ve grown to love testing. It once got to the point where testing was the sole driver for all code I wrote, via TDD, but now has come back around to something less intense. I find unit tests less helpful and instead, lean on acceptance/integration testing to capture most things.

Ultimately how and if you write tests will depend on a few key factors;

1. Development Style

Are you practicing Test-Driven Developement, or are you writing in a more conventional or exploratory way? In simpler terms - are you writing your tests before you code, as you code, or after you code?

2. Existing Tests

Does the project you’re working on contain existing test infrastructure? Can you answer yes to these questions?

  • are unit/integration tests widespread in the project already?
  • do existing tests contain scaffolding that enables you to easily add more tests? (i.e. mocks/stubs of major dependencies)
  • is the test suite run automatically via continuous integration pipelines? If so, is the pipeline passing a programmatic requirement for all pull requests?

The more yeses from these questions, the more likely change requests will come with at least a couple new test cases, no matter the development style of a contributor.

3. Barrier to Extension

This is simply how hard it would be to add tests had there not been existing infrastructure in place. This could be for a brand-new project or in an older project that was missing a particular type of test (i.e. no integration tests).

My experience with Java has always been bad in this respect. The testing environment was so fragmented that you had to use different frameworks depending on what style test you were wanting, and you also had to integrate said frameworks with your build framework too, just to make them run. Go was a huge breath of fresh air for me here as it had testing built in as a first class citizen from day one.


With all these factors in mind, it makes sense to view testing in Rust purely from the third point, as this is where a language has the most potential for enablement.

11. Writing Automated Tests

Not to start on the wrong foot but straight off the bat this chapter managed to annoy me thoroughly by leading with this excerpt:

In his 1972 essay “The Humble Programmer,” Edsger W. Dijkstra said that “Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence.” That doesn’t mean we shouldn’t try to test as much as we can!

I scratched my head and thought, “this doesn’t sound right”. So off I went to read The Humble Programmer essay (which is fantastic). It became immediately obvious that Dijkstra had anything but an anti-testing sentiment. My guess is that the authors of the Rust Book may have just made a simple translation error. Take for example, the very next line of the essay. Dijkstra writes:

The only effective way to raise the confidence level of a program significantly is to give a convincing proof of its correctness.

No, Dijkstra isn’t talking about a formal proof. He’s talking about Test Driven Development in 1972. He continues:

If one first asks oneself what the structure of a convincing proof would be and, having found this, then constructs a program satisfying this proof’s requirements, then these correctness concerns turn out to be a very effective heuristic guidance.

One can only imagine the “Program testing” Dijkstra referred to in the original quote, could simply mean manual testing in today’s tongue. The Book missed a great opportunity to use Dijkstra to promote testing and instead we got only half of it. Oh well.


Setup

I got a feeling that this chapter had a different author than previous ones, as the examples were numerous and verbose. I managed to skim over most of them as in essence the concepts were quite basic. Basic though, is a great quality in a testing framework.

The Basics

In no relevant order, let me talk about features of Rust testing that I loved.

First - tests are simple to add, just a [#test] annotation for a function and you’re off to the races. In Go you have to do two things - create a file that ends in _test.go and then write your test functions as

func TestXxx(t *testing.T)

… Now let me tell you that I’ve reviewed unit tests in Go which had me scratch my head and ask myself “how the f*** did these tests pass??". It turned out that after another 10 minutes of faffing about and running the tests locally, did I find the test had never been run! Along with many others in the package for the past 6 months! Why? Because someone forgot to capitalize the first letter of the test function names, meaning their testMyProgramWorks “tests” weren’t exported, meaning Go didn’t classify them as real tests. Add some copy-pasta and there we were…

Having a much simpler API for specifying tests is a big win in my book. Thank you, Rust.

Seamless Integration

Next on my testing crushlist is Intellij+Rust-plugin testing integration. I can just hit shift+F10 and the IDE will run the tests for my crate and spit out a summary with links to failures and stuff. For one reason or another, I never run my Go tests from Goland (Intellij for Go). I use the command line with filters to run tests in specific packages. I don’t even know if I’ve ever even tried to run Go tests in Goland!?

After getting excited about this, I decided to try the IDE’s debugger with an integration test. Again, it just worked! I’ve used a debugger once or twice in Go, but you have to use Delve, then find some tutorials for setting that up with your IDE… In practice, I’ve never felt it was worth my time to use a debugger in Go. I’m not sure this would change with Rust - but debuggers are definitely nice to have when you’re a newbie and want to get a stronger feel for what’s happening in your code. Very cool to have it in the IDE with zero effort.

Almost, Seamless

As I continued my tool-driven gorge, I hovered over a polished button that had inlaid in it, “Run ‘Hello World’ with Coverage”. Oooooh, ahhhh, I ogled.

*click*

BZZZZZRRP!!!!

Oh no.

Code coverage is available only with nightly toolchain.

Well excuuuuuse me for being so basic that I only use the default rustup install… To reinforce this further, the Book slaps me with this one too,

Benchmark tests are, as of this writing, only available in nightly Rust.

I am a big fan of Go benchmarks. Possibly the most underrated aspect of the Go testing ecosystem. It’s basically the standard for figuring out how alloc-y a package is and gives you hard numbers in that respect. Maybe benchmark tests aren’t that good in Rust, given you have much stricter control over your allocs?

So for now, I’ll add nightly-rust to my list of things to understand, post-Book.

Ok enough tool talk, The Tests!

The Rust testing tool chain is simple and comprehensive. The assert! style macros are a godsend. Every test I write in Go starts with me importing the testify library, just to get access to assert. Rust testing is close to what Go offers, but looks to have a clear edge, out of the box.

On the opposite end of the stick, I’m not a huge fan of coercing us to put unit tests within the module being tested. Splitting up test code from production code makes for easier reading. Thankfully, this post showed me how to split your tests into a sibling module, so with some slight hacks it appears my woes are surmountable.

A point of confusion came about when the Book ended a section saying we can write tests to return Result<T,E>’s. Apparently this is an acceptable way to write tests over assert! macros… The Book doesn’t make a suggestion about which is more idiomatic, so until I’m convinced otherwise, I’ll stick with asserts.

Delight me more

Rust continued to delight me by doing simple things that Go doesn’t, for example having sane help printouts for the test command. Just trying to figure anything out by running go help test (and yes, in that specific arg order) is an exercise in futility. Any time I have to figure out how to disable test caching in Go, it’s a visit to Uncle Google.

Simiplicity continues in Rust with sane ways to filter running tests, and a simple mechanism to arbitrarily partition your tests too, using the #[ignore] annotation.

Splitting Hairs

There is no logical division in Go to encourage splitting unit tests from integration tests. You have to make such a divide yourself. I love that Rust has thought deeper about this and there are patterns to follow for each case.

There’s debate within the testing community about whether or not private functions should be tested directly, and other languages make it difficult or impossible to test private functions.

I haven’t heard this debate before, I’m guessing it would go something along the lines of “test the contract, if that works then you’re good”. Whilst I wouldn’t buy it completely, I do find substantially more value in integration and acceptance testing than unit testing every function.

Teardown

Both Rust and Go have testing as first class citizens, but Rust wins for ease of use and completeness in tooling. Go requires third party libraries to be pleasant, and even then there are nicks and edge cases which can be frustrating.

Benchmarking is Go is a nice feature and even though we didn’t go into the Rust benchmarking support here, I get the feeling Rust probably hits it out the park. (Am I becoming a fanboy already?)

Overall I’m walking away from this chapter feeling excited for my future of Rust tests. Really!

comments powered by Disqus