API BREAK: libxdot: use size_t instead of int for op sizes/counts

API BREAK: libxdot: use size_t instead of int for op sizes/counts

This allows exceeding operation counts of 2³¹ - 1, which is not particularly
likely. But its more important effect is to make arithmetic operations less
error prone and make it impossible by-construction to arrive at a negative
operation size or count.

Doing this as a single sweeping change seemed less error prone and clearer,
despite how large the resulting diff is.

For lines that would have been changed anyway, this commit also fixes white
space and tightens variable scoping where possible.