Adobe Photoshop is a Raster Based Software. Raster Images are images that are made out of pixels. So if you zoom in, you will see the pixels (that all together make an image).
If you want to make this circle without the jagged edges (pixels) when you zoom in, you need to use a Vector Based Software such as Adobe Illustrator. Vector images are images that are made up of points, lines, curves etc that are based on Mathematical Equations, and not Pixels.
TL;DR - If you want a circle without the jagged edges, you need to use a vector based software such as Adobe Illustrator.