Ally ships killed mine (Alliance bug/oversight?)

I recently atked the planet of an opponent my ally had a fleet on (as he had unallied and was waiting for the 24 hours). When I hit it I didn’t take the planet as I expected cause my allys fleet had fought mine too.

So when your allied but your allys fleet is at your enemys planet (who he’s allied to) you can still fight.

It’s purposeful if a bit obscure. The help files cover it: Neptune's Pride: Triton. Explore, Expand, Exploit, Exterminate!

More than 2 Players

In combats that involve more than two players, the players are arranged into teams. The Defenders are all players in a formal alliance with the star owner. The Attackers are all other players.
[…]

Your ally defended because the formal alliance wasn’t finished yet. He was still allied to the star he was orbiting so helped defend.