James was born on Veteran's Day, 1919, but other than that, little is known about his childhood.
James attended the University of Michigan, where he excelled in most of his courses, except German which he did fairly poor in. Afterwards, he attended West Point where he graduated in 1942 and joined the United States Army Corps soon afterwards.
Many of the missions during the war involved Patterson having to sabotage enemy weapons, notably stealing a secret stealth fighter (Medal of Honor: Frontline) and destroying a possible nuclear warhead and the facility it was being stored inside (Medal of Honor). Along the way he meet Manon Batuste, a member of the French Resistance whom he developed romantic feelings with. He was also assigned to terminate high ranking Nazi official Haupsturmführer Rudolf Ulbricht von Sturmgeist.
After the War ended in 1945, Patterson joined the Central Intelligence Agency during the course of the Cold War. Patterson proposed to Batiste, but whether she accepted is unknown. Patterson did however go on to have at least one child, who in turn when on to have a child of their own, whom was named in honor of his grandfather and served in the War on Terrorism.