You might also want to look into PBR (physically-based rendering).

This acknowledges that rather than simply a "color", it's better to consider a "material" which has additional properties. For instance, how shiny it is, which determines the matte-vs-specular balance. Matte is essentially a direction-independent color, while specular is a directional reflection and thus dependent on the angle of the surface from the viewer, and the angle of the light sources from that surface. Other relevant properties are emission (light generated by the object, not reflected--how else will you understand the color of a lit light bulb?), how transparent the object is, the index of refraction, and so on.

One thing to note is that PBR is used to consistently assign these properties in computer graphics, so under similar lighting conditions you should get a similar appearance. But it is very rare for PBR to take spectral effects into account, and normally it just does calculations assuming we just want a red, green, and blue channel in the end. So this is all orthogonal to what other people are saying about integrating spectral values and such. They are two different aspects about how to talk about color.

In principle we really want all of it combined: we want to talk about how the material reacts to light at each wavelength, and what the intensity of the output would be for every wavelength. This is enormously computationally intensive and tends not to be done. But if you really wanted a reliable description of what "color" an object is, this is how you'd have to do it. And rather than a color, you'd really be describing a material, with very many properties and all of those properties having not a single number as a value, but instead a complex function depending on wavelength.

Or you can skip all that and go straight to calculating quantum mechanics. That will be the most accurate, but of course it's the "real" process without any simplification. It's...incredibly hard.