Fix team scoreboards: The sequel

The previous merge request !2587 (merged) was a disaster, it just broke it further in never displaying any player no matter what team was thx to my poor testing.

This merge request aims to fix all making it work properly and also again, aware of custom team colors.