Sharma was born on 13 January 1938, in Jammu in the princely state of Jammu and Kashmir in British India. His father Uma Dutt Sharma was a vocalist and a tabla player. His father started teaching him vocals and tabla, when he was just five. His father saw an opportunity to introduce him to the santoor, a hammered dulcimer, which was a folk instrument that traced its origins to ancient Persia, but was played in Kashmir. He saw the styles that integrated Sufi notes with traditional Kashmiri folk music and had his son play the instrument that was then new to Indian classical music. Sharma started learning santoor at the age of thirteen and gave his first public performance in Mumbai in 1955.

Sharma in 1988

Starting his career playing the santoor with his father, Sharma is credited with introducing the santoor as a popular Indian classical music instrument. He recorded his first solo album in 1960.[1] Sharma took the santoor as an Indian classical musical instrument playing at various music venues. He collaborated with Indian tabla player Zakir Hussain and with flautist Hariprasad Chaurasia on many of his performances as well as on his albums.In 1967, he teamed up with Chaurasia and guitarist Brij Bhushan Kabra to produce a concept album, Call of the Valley (1967), which turned out to be one of Indian classical music's greatest hits.

Sharma composed the background music for one of the scenes in V. Shantaram's film Jhanak Jhanak Payal Baje (1955) where Gopi Krishna performed a Kathak dance piece. Further, he composed music for many Hindi films in collaboration with Chaurasia, starting with Silsila They came to be known as the Shiv–Hari music duo.Some of the movies they composed music for were musical hits, such as Faasle (1985), Chandni (1989), Lamhe (1991), and Darr (1993).Sharma also played tabla including in the popular song "Mo Se Chhal Kiye Jaaye" sung by Lata Mangeshkar in the 1965 film Guide, on the insistence of music director S. D. Burman.However, Sharma's focus remained on classic music rather than film songs; he said, "Classical music is not for entertainment. It is to take you on a meditative journey, ye toh mehsoos karne ki cheez hai (This has to be experienced).

Sharma was awarded the Sangeet Natak Akadeemi Award in 1986, the Padma Shri, India's fourth highest civilian award in 1991, and the Padma Bhushan, India's third highest civilian award in 2001.

Sharma married Manorama and had one son.His son Rahul, who started learning at the age of 13, is also a santoor player, and they performed together since 1996. In a 1999 interview, Sharma stated that he chose Rahul as his shishya, because he thought he had the "gift of God"

Sharma died on 10 May 2022 from a cardiac arrest. He was 84 years old. He had kidney failure for the last few months and went through regular dialysis. He received a state funeral at Pawan Hans Juhu aerodrome, Mumbai, on 11 May 2022.