Listed in the 1860 census as being 9 years old and living with his uncle and brother, William. This entry is pure speculation, not backed by any firm evidence other than the census data.
1860 Georgia census the district Calhoun Co. GA 4th District
http://homepages.rootsweb.com/~genea/Dga1860.html
Name Age Sex Occup Born
Hayes, Henry, 57 m Farmer VA (brother of Martha Hayes - Emanuel's wife)
" , Sarah T., 49 f GA
Daverson, Jasaphene 12 f GA in school
Dodd, William C. 29 m School Teacher TN
" , Eliza A. 22 f GA
Moomaugh, William 16 m GA in school
" , James 9 m GA in school
Pon, Julia 17 f GA
Glass, Emma 18 f Instructist GA
Holloway, David 54 m Carpenter NC