Contrary to the impression given by his public persona, Phillips came from a background of poverty. He was born in Tottenham, then in Middlesex, England, the son of Cecelia Margaret (née Newlove) and Frederick Samuel Phillips, who worked at Glover and Main, manufacturers of cookers in Edmonton; the "filthy, sulphurous" air of the factory gave Frederick a weak heart and edema, leading to his death at the age of 44. In 1931, the family moved to Chingford, then in Essex, where Phillips attended Larkswood Primary School.
It was his mother who decided that Phillips should be sent to the Italia Conti Academy to receive elocution lessons to lose his natural cockney accent. At that time a strong regional accent was a major impediment to an aspiring actor. It proved to be an astute move and by the age of 14 Phillips was the family's main breadwinner, saving his mother from penury.