Knowledge gaps remain regarding the cardiorespiratory impacts of ambient volatile organic compounds (VOCs) for the general population. This study identified contributing sources to ambient VOCs and estimated the short-term effects of VOC apportioned sources on daily emergency hospital admissions for cardiorespiratory diseases in Hong Kong from 2011 to 2014. We estimated VOC source contributions using fourteen organic chemicals by positive matrix factorization. Then, we examined the associations between the short-term exposure to VOC apportioned sources and emergency hospital admissions for cause-specific cardiorespiratory diseases using generalized additive models with polynomial distributed lag models while controlling for meteorological and co-pollutant confounders. We identified six VOC sources: gasoline emissions, liquefied petroleum gas (LPG) usage, aged VOCs, architectural paints, household products, and biogenic emissions. We found that increased emergency hospital admissions for chronic obstructive pulmonary disease were positively linked to ambient VOCs from gasoline emissions (excess risk (ER%): 2.1%; 95% CI: 0.9% to 3.4%), architectural paints (ER%: 1.5%; 95% CI: 0.2% to 2.9%), and household products (ER%: 1.5%; 95% CI: 0.2% to 2.8%), but negatively associated with biogenic VOCs (ER%: −6.6%; 95% CI: −10.4% to −2.5%). Increased congestive heart failure admissions were positively related to VOCs from architectural paints and household products in cold seasons. This study suggested that source-specific VOCs might trigger the exacerbation of cardiorespiratory diseases.