That's the fix:
https://github.com/coderbyheart/adventofcode/commit/b9c57830331f8669da34bbed83c6fb1ed457cdee

Instead of iterating over all remaining positions all the time, only put the
last visited into the queue. Eventually the algorithm will visit all points
anyway. Runtime is now 4 seconds.