This is why end to end tests are important: I've noticed that an unreleased
development version of our SDK could brick devices, because a configuration
change was made, and this was not discovered because module tests do not
consider how it's integrated.
3 replies
Replying to @coderbyheart
So I started to work on a test for @bifravst:
https://github.com/bifravst/firmware/pull/180
This is a full end-to-end test that runs on real hardware which boots a device
and actually performs a firmware update over the air once the device has
connected.
Doing that I discovered yet another issue:
Replying to @coderbyheart
An underlying module's configuration was causing the download to fail, because
there was not enough memory configured to store the longer hostname of the test
environments firmware server.
This took two additional days to figure out.
Replying to @coderbyheart
Just because we suddenly have multiple firmware tests per day, compared to maybe
one manual smoke test per month or so, we already were able to mitigate
potentially catastrophic bugs.
That's why I love #endtoendtesting, because it tests what really happens.