어머나 (O-mon-ah) vs. 엄마야 (um-mah-yah)?

There really is no difference.  Just your preference, and how you say things.

BTW, both are an expression used to express surprise. 

...

How about 엄마 (um-mah) vs. 어머니 (o-mon-ni)? 

Well 엄마 is informal and 어머니 is formal... and they both mean mom/mother (in Heirs if you watched it Kim Tan differentiates his moms my 엄마 and 어머니).

But if you notice, 어머나 has the beginning of 어머니 in it.  And 어마야 has 엄마 in it.

So 어머나 and 엄마야 translate to mean along the lines of... "MOMMY!!!" 

Korean people are such wimps.

Just Kidding. 

It really does mean MOMMY!!! and we are so not wimps.

But more or less, o-mon-ah and um-mah-yah are a more... polite and mature version of MOMMY!!!