North and south in the sky are staightforward. North is in the same general direction as the Earth’s North Pole. Thus, in Orion, Betelgeuse and Bellatrix are north of Saiph and Rigel. And of course Polaris is north of all other visible stars, and this is true even if it is lower in the sky. For the planets, the North Pole is the pole on the northern side of its orbital plane.
Pictures of star fields and nebulae are normally presented with north at the top. The planets are usually presented similarly, but many older pictures, especially of the Moon, were shown with south at the top in the belief that most observers viewed the Moon by looking directly through a refracting telescope, which inverts the image. However almost nobody does that today; we use a star diagonal, which is a mirror, so we see a mirror reversal but in which direction depends on the orientation of the eyepiece.
East and west are more complicated and can be confusing. For stars, the east is to the left of an observer in the northern hemispere looking south, and of course west is to the right. Thus, in Orion, Betelgeuse and Saiph are east of Bellatrix and Rigel. This convention also applies to double stars although the components may swap over from time to time.
At one time, this same convention applied to the Sun and Moon. Hence Mare Orientale (the Eastern Sea) was on the eastern limb of the Moon. However this meant that the Moon rotated east to west whereas the Earth rotates west to east, but they rotate in the same direction (anticlockwise as seen from above the North Pole). In preparation for the Apollo Program, this anomoly was corrected by reversing east and west on the Moon so that now Mare Orinentale in on the western limb. (Actually this is not as odd as it sounds because the mare is on the far side of the Moon.) This convention is now extended to all the objects in the solar system with east and west defined such that all bodies rotate west to east. This is consistent with the derivation of the names; east comes from the Greek eos meaning the dawn, and west comes from the latin vesper meaning the evening.
This convention causes some confusion for Venus and Uranus which rotate retrograde, that is right to left as seen from the Earth. This is indicated in the planets’ defining data by quoting the axial tilt as greater than 90°. It is about 177° for Venus and 98° for Uranus. This does not mean that their north poles are south of their orbital planes, mearly that they are in retrograde rotation. Thus an anomaly will arrise for any people visiting the surface of Venus. Standing on the surface looking north will put east on the left and west on the right. Fortunately it is likely to be a long time before astronauts have to contend with this situation.