The main difference is that when you use a factory, it is invoked each and every time you request that type. It's basically a set of code that at runtime, will run to determine how to create your object.
Consider something like so :
//Here I can do work to think about which configuration object I actually want to return
While your example uses singletons, so it's a little less relevant, it is very handy when returning transient/scoped objects as you can use things at runtime to decide how to new things up.