Squamous cell carcinoma of the head and neck (SCCHN) is common worldwide and related to several risk factors including smoking, alcohol consumption, poor dentition and human papillomavirus (HPV) infection. Different etiological factors may influence the tumor microenvironment and play a role in dictating response to therapeutics. Here, we sought to investigate whether an early-stage SCCHN-specific prognostic matrisome-derived gene signature could be identified for HPV-negative SCCHN patients (n = 168), by applying a bioinformatics pipeline to the publicly available SCCHN-TCGA dataset. We identified six matrisome-derived genes with high association with prognostic outcomes in SCCHN. A six-gene risk score, the SCCHN TMI (SCCHN-tumor matrisome index: composed of MASP1, EGFL6, SFRP5, SPP1, MMP8 and P4HA1) was constructed and used to stratify patients into risk groups. Using machine learning-based deconvolution methods, we found that the risk groups were characterized by a differing abundance of infiltrating immune cells. This work highlights the key role of immune infiltration cells in the overall survival of patients affected by HPV-negative SCCHN. The identified SCCHN TMI represents a genomic tool that could potentially aid patient stratification and selection for therapy in these patients.