Is it compliant and secure to allow our U.S.-based MSP to install a remote management and monitoring tool on servers and workstations that are part of a NIST 800 SP-171 environment?

Regardless of the compliant framework, I suggest that you conduct a threat modeling on the in scope systems to identify the risks and come up with your remediation recommendations.  BTW you can't prove something is secure.  You can only prove that something is insecure by testing it. 

I agree with the post by our colleague, Kristen Skinner. There is no simple answer.

“Secure” and “compliant” are two different concepts with various definitions depending upon the situation.  Compliance is usually binary, think in or out of compliance, while security is on a continuum.  One way to get to an answer in your scenario is to understand “why”. The managed service provider seeks to install a remote access tool on systems required to follow Federal security standards. “NIST SP 800-171 is a NIST Special Publication that provides recommended requirements for protecting the confidentiality of controlled unclassified information (CUI)” Nist.gov

Why they want to install remote monitoring could be to provide you with 24/7/365 monitoring, to allow offshoring to the cheapest country, to integrate telemetry from your environment more closely into their security operations center, to avoid paying a resource to physically visit your location or maybe something else.

My view is that the MSP’s change must not decrease your security posture, you’re protecting controlled information.  It is straightforward to control individuals entering your facilities, but more challenging to continually determine who has access to the remote access tool being used by the MSP. It is common practice to remotely administer systems but you also accept the potential liability formed by creating a new attack surface.

I would tell the vendor no and require that they convince you the risk is minimal, with documentation appropriate to the use case. Your company is still on the hook if there’s a “glitch” but you will have documentation to substantively demonstrate that you performed commercially reasonable due diligence on the process. I think you’d agree “they said it was secure” is not a winning phrase when explaining a snafu to a government procurement manager.

There isn't a clear yes/no answer to a question like this.  Here are some things I'd use as a starting point for evaluating this question.

Tools like this usually fall under components that map to various controls in the AU and CM family of NIST controls, possibly others as well.  If you review the SP800-171 CUI overlay spreadsheet NIST publishes, data from many of those controls fall into the CUI category, so it will depend on your organization's agreements and access restrictions applied to the MSP.  If the tool also allows for any sort of remote access/control, then you'll also want to review the AC controls in the overlay since many of those fall into the FED (Federal Agency responsibility) category, especially if the operations fall under AC-14.  I would start with an evaluation of the data and capabilities the tool implements and map them to the appropriate controls, to evaluate classifications of access, control, and data involved.

Determining if it is secure involves evaluating the data and access the tool provides, the agreements and controls in place between you and your MSP, and the technology stack in use (encryption of data in transit, at rest, access controls, and so on), which becomes part of the system security evaluation.  That data then feeds into your risk assessment and authorization process, where the final determination can be documented.