William's father was murdered in 1855 when he just a year old. William was raised by his grandmother, Elizabeth Tumlinson Taylor. His brother, Joseph, was born after his dad was killed during the Taylor-Sutton feud and was raised by his uncle John Milam Taylor.