Tannenbaum was born in Pennsylvania, and moved to the Lower East Side of Manhattan at the age of three; his family later relocated to Brookyn. He quit school at the age of 17 to work as a stock boy in the garment district. After a stint as a salesman, he began work at the Loch Sheldrake Country Club, which was owned by his father. When Tannenbaum was 25, he met a guest at the club by the name of Jacob "Gurrah" Shapiro, a close associate of infamous Jewish gangster Lepke Buchalter. Shapiro felt that Tannenbaum had what it took to become a mob hitman, and introduced him to the underworld.
Tannenbaum progressed rapidly through the ranks of organized criminal violence. He began as an enforcer and strikebreaker being paid $50 a week; his salary was raised to $75 and then later to $100. When he became a full-fledged contract killer in Murder, Inc., he was paid $125 a week for his services.