The Serbian opened the scoring from 12 yards, only for Palace to fall behind to goals from Granit Xhaka and Pierre-Emerick Aubameyang.
However, Xhaka would soon turn from hero to villain as he conceded an 83rd-minute penalty.
Wilfried Zaha drove into the box in trademark fashion, with his pace, power and trickery too much for Xhaka, who could only bring the Ivorian down.
Milivojevic stepped up once again and whilst Bernd Leno guessed correctly, there was nothing to go do to keep the ball from finding the net away to his right.
The strike ended Arsenal's 11-game winning run in all competitions, with Palace finally finding the net at home in the Premier League after a wait of six hours, 57 minutes.