Hello all!
I am thrilled to present this video Q&A session about the CMMC Assessment processes.
This CMMC Q&A is not technical
Unlike many topics on this website, this discussion is relevant to ALL people interested in CMMC. Business owners, IT professionals, cybersecurity professionals, CMMC assessors, CMMC assessment organizations, etc.
We purposefully try to use CMMC level 1 examples where possible, so this discussion doesn’t get very technical.
Participants:
Jeff Dalton is the CMMC Accreditation Body Board of Directors and Chairman of the Accreditation and Credentialing Committee.
Amira Armond is the owner of Kieri Solutions LLC, a CMMC C3PAO candidate, and is the chief editor for CMMCaudit.org.
CMMC Q&A about Assessments discussed in Part 1:
- What is a “gap analysis”? What is a “pre-assessment”?
- Are RPOs authorized to perform “gap analysis”? *apparently this is a trick question*
- Are C3PAOs authorized to perform consulting? *another trick question*
- If a C3PAO plans to perform an official assessment, can they perform a gap analysis for that client?
- Why use a Registered Practitioner for gap analysis instead of a C3PAO?
- What happens if you get bad advice from a consultant that leads to a failed assessment?
- When should you pull in a full C3PAO to do a pre-assessment?
- What is a readiness review?
- Is there an opportunity for companies to fix problems found during an official assessment?
- If an official assessment looks like it will fail dramatically, should it be stopped to save money?
- If your documentation describes all your systems, will the assessor stay with the CMMC-specific scope?
Show notes for the interview will be added below the video. Make sure to scroll down for additional information and links.
I think this is extremely valuable content for the entire DIB. Thanks to Jeff Dalton and the CMMC-AB for their efforts. Please share this with other companies and sign up for our newsletter. This series will hopefully save contractors a lot of angst about the assessment process. – Amira
CMMC Questions and Answers (paraphrased)
These notes from the interview are paraphrased and may have been interpreted wrong (all responsibility for correctness of these notes is held by Amira). For the best source, watch the video! Please comment or email us if you see anything wrong.
Introduction
- Jeff Dalton is the CMMC Accreditation Body Board of Directors and Chairman of the Accreditation and Credentialing Committee.
- Amira Armond is the owner of Kieri Solutions LLC, a CMMC C3PAO candidate, and is the chief editor for CMMCaudit.org.
RPOs, C3PAOs, and Gap Analysis or “Pre-assessments”
Amira: Regarding “readiness reviews” or “pre-assessments”. We have Registered Practitioner Organizations (RPOs) and Certified Third Party Assessment Organizations (C3PAOs) in the ecosystem. To my understanding, C3PAOs are authorized to perform pre-assessments as well as official assessments. And RPOs are authorized to perform “Gap assessments” or “Readiness reviews”. What is the difference between those two services?
Jeff: Language is always an issue when you roll out something like this. We tried to be careful when we rolled this out in training.
Jeff: We have assessments, which are “certified events”. Assessments are unique because only a C3PAO and a Certified or Provisional Assessor are authorized to perform certification assessments. These are events that can result in achieving a Maturity Level certification for CMMC. These are also the only thing that is “authorized” by the CMMC-AB.
Jeff: A C3PAO is allowed to perform services as long as they abide by the Code of Professional Conduct (CoPC), but the only thing they are “authorized” to do is an assessment for certification.
Jeff: Likewise, an RPO is allowed to perform services as long as they abide by the CoPC, but there are no services by an RPO which are “authorized” by the CMMC-AB. A Registered Practitioner is free to perform a Gap Analysis or Pre-assessment as well as give advice and perform consulting to help a client prepare for the CMMC. This is a free market.
Jeff: If a C3PAO will ever perform an assessment for certification on a client, they cannot give advice or perform consulting for that client (this would fall into the category of “assessing your own work”). A C3PAO can perform a “gap analysis” for these clients, but they can’t give advice or consult. They can only identify problems.
Assessment Readiness Reviews
Jeff: Readiness Review is a term used to describe a preparatory event for the official assessment. It checks non-CMMC readiness items such as whether there is a date set for the assessment, whether all parties are available, whether the plan has been created and signed off by all parties, etc. The goal of the Readiness Review is to make sure that the client (the Organization Seeking Certification) and the C3PAO are ready to perform the assessment.
Still unclear about “authorized”
Amira: So a C3PAO is authorized to perform consulting as long as they won’t perform an assessment for that client later?
Jeff: Let me clarify, because I’m not comfortable with your language. The only thing that the CMMC-AB “authorizes” is the performance of certified assessments. The AB does not authorize anyone to perform consulting (advice, guidance to help a company prepare for CMMC).
Amira: So the AB does not take responsibility for any companies in the ecosystem to perform consulting?
Jeff: If any accreditation body certifies an individual, they are basically making a statement that they have properly trained the individual and they are monitoring them. They have some responsibility for oversight. We aren’t using that word for consultants (RPs, RPOs).
So should a defense contractor call an RPO, a C3PAO, both?
Amira: So when a company is preparing for CMMC, should they work with an RPO for their Gap Analysis, or should they get one from a C3PAO for a higher level of assurance?
Jeff: I think there is value for a company reaching out to an RPO. You (Amira) made a post about a month ago that consultants will have very deep technical knowledge about implementations, that a dedicated assessor might not have. Working with an RPO that knows your company and will work for you long-term can be very advantageous. There is nothing wrong with getting a gap analysis from a C3PAO, but you won’t have access to that consulting and guidance for implementation, unless they won’t be the same C3PAO you use for the certification assessment.
Consultants giving bad CMMC advice
Amira: What happens if your consultant doesn’t understand something about the CMMC, and maybe they give you bad advice? Is there an official way to complain about the RPO? Is that even something that people should do?
Jeff: There are something like 182 practices in the model and a lot of them are very complex. Could a single consultant miss something? Well, yeah. This is more of a question about whether people make mistakes. It seems obvious to me that with something this complex, one person couldn’t possibly know everything about every practice.
Jeff: But there is a CoPC. So if you feel that a Registered Practitioner has violated the CoPC in some way, for example, by misrepresenting themselves as being more knowledgeable or skilled than they are, it is something that can be reported to the CMMC-AB.
Jeff: But overall, the question of whether a consultant will always give you the right answer for every single practice… I’m not sure sometimes if there is a right answer. There are many right answers. Again, we aren’t regulating this area. But there is a CoPC which will help guide reactions to any instances of fraud or misrepresentation.
Gap Analysis – if a C3PAO intends to assess you later, what can they do?
Amira: OK, so I think at this point it is safe to use the word “Gap Analysis” for an unofficial assessment against CMMC maturity levels to see which practices you would have trouble with.
Amira: When a C3PAO performs a gap analysis for a client they plan to officially assess (for certification), is it correct that a C3PAO would not be able to give advice for how to solve problems? They could say that you have problems, but no guidance past that?
Jeff: That is correct. C3PAOs will need to be very careful about this, because part of the quality review process is to evaluate whether the C3PAO violated the CoPC in any way. It is very easy to give advice accidentally. There will need to be a very carefully constructed contract.
Gap Analysis – how important are these? Should they be the last step before an assessment?
Amira: When should a company seek out a gap analysis? At the very beginning, when they first start working on CMMC compliance? A month before they do their official assessment for certification?
Jeff: It is a little hard to say, because we don’t have experience with CMMC yet. But this same question is addressed in other similar industries. It is fairly common in other industries to have multiple gap analysis performed during the preparation process. For example, companies could get a very brief analysis to identify large issues at the very beginning, then schedule another analysis later for more detailed review.
Amira: When a C3PAO is preparing to perform an assessment for certification, should they insist on performing a gap analysis ahead of time to ensure that the client won’t “fail spectacularly”?
Jeff: Our regulatory control only covers official assessments. But I definitely think that companies should utilize gap analysis. Any company that goes in for an official assessment without an external gap analysis is probably in for a very big surprise. Anyone who performs a self-assessment should understand that these are generally not as deep or as broad as an outside analysis.
Certified Assessors performing consulting
Jeff: Another thing we haven’t discussed is that there isn’t anything stopping a company from hiring a certified assessor to perform gap analysis / consulting. It is commonly done in the ISO and CMMI world.
Amira: Yes, I definitely expect that companies (especially better funded ones) will seek out the highest level of experience and talent that they can find, which would probably be a certified assessor at the highest level they can find.
Jeff: If this is anything like other industries, a gap analysis will be less costly than an official assessment, because a gap analysis will have less requirements and less preparation required.
Should companies cancel gap analysis if there are big problems found?
Amira: (I’m channeling the DIB companies for this question) If you are in the middle of an analysis and you find out that there is a big problem, such as missing antivirus on all your computers, should you cancel the analysis to fix the problem, or continue on?
Jeff: There are pluses and minuses to this, but in general, if you stop each time you find a problem, you might never get through a gap analysis. It is good to get an overall view of exactly what your state of maturity is. I like to use a “heat map” in this scenario, where I have all the practices on one sheet, and use colors to show areas of problems and of high performance. The one caveat is that there can be interdependencies between practices. A weakness in one practice may cause other practices to fail, and fixing one thing can fix multiple practices.
Jeff: It is good to get an overall picture, otherwise you may end up playing “whack-a-mole” when you are trying to remediate. So I recommend completing the entire thing, and also the same thing for certification assessments.
Certification assessments – 90 day window to correct deficiencies
Amira: You anticipated my next question about what to do if an official assessment will fail.
Jeff: So there is a mechanism in the methodology for correction or remediation. There is a “fourth phase” which I haven’t mentioned yet. The remediation phase is a 90 day period of time where the client can make corrections, then the assessor comes back and verifies them.
Jeff: There are Organizational Conflict of Interest (OCI) problems with stopping assessments in the middle. There is a phenomenon called “assessor shopping” which we are trying to prevent, where clients contact potential assessors and ask whether they consider certain implementations to be acceptable or not, before hiring them.
Amira: I can tell you that I haven’t heard anything about this 90 day remediation window from anywhere else. Is this official?
Jeff: This is something that we’ve trained all of the provisional assessors on, it was used during pilot mock assessments, it is in version 2 of the Method document. This is different from a POA&M. You can’t get the certification until you’ve met all objectives. But you get some time to do it.
Jeff: There are some parameters around this. It must be performed by the same assessment team. It must be within a 90 day window. The request for a remediation window must be submitted within 7 days. The CMMC-AB needs to be notified. But there is a 90 day window to make corrections and have the assessor come back and perform a delta against those practices.
Amira: When you say a delta, that means you are checking just the differences. You don’t do a full assessment again. For example, the assessor would only ask “Do you have antivirus now?”
Jeff: It still counts as a mini assessment. It still needs to go through all three phases of an assessment. But it would only be for those specific practices, so it would take much less time than a full assessment.
Amira: That is very reassuring! Thank you for that clarification!
Conflict of interest by assessors
Amira: You mentioned that there are Conflict Of Interest (COI) concerns with stopping assessments if problems are found. One of the requirements for ISO 17020 (that C3PAOs need to meet), is that nothing should influence an assessor to either pass or fail an assessment. One of the things I considered is that I don’t want to encourage an assessor to pass a company by stopping assessments if there is a failure (because this would reduce the number of hours worked and payments to the assessor).
Jeff: COI is complicated. One reason why the DoD wants C3PAOs to use ISO 17020 is because it addresses COI. But COI exists everywhere. There is no such thing as absence of conflict. There is management of conflict. I didn’t mean to imply that stopping an assessment is a COI. But it could be. So we need to manage these.
Jeff: If an assessor calls the C3PAO and says they need to cancel an assessment, the C3PAO should review to ensure it doesn’t introduce a COI.
Should companies minimize their System Security Plan to only address the assessment scope?
Amira: For CMMC Level 2 and above, there is an expectation that a company will have a System Security Plan.
Jeff: Sigh
Amira: I might already be mis-stating something.
Jeff: Yeah.
Amira: In my understanding, a well-done system security plan will describe all systems, and it won’t necessarily be limited to CMMC topics. It might talk about plans to improve security in the future, not necessarily limited to the requirements of the CMMC. There is a concern (I have a concern) that an assessor might read a system security plan, see this additional information, and decide they need to verify security on topics that should be out of scope.
Amira: Should a company create a system security plan just for the assessment, which is limited to the exact scope that will be assessed?
Jeff: I think it is a mistake for companies to create a system security plan that is CMMC only. A company should have a plan and policies and training that promote strong cybersecurity performance. The CMMC is a way of evaluating whether you have high-performing cyber systems, process, and people.
Amira: When an assessor is looking at a system security plan, they aren’t assessing the system security plan, they are assessing the characteristics of the system security plan. Does it address the right things? Does it have needed attributes? Is it signed off by the right person?
Amira: In the assessment industry, we call this phenomenon “overinterpretation”, when an assessor sees an unrelated thing and decides to pursue it.
Amira: CMMC assessments are not assessments of the system security plan, but rather of the framework, of the overall model. I recommend focusing your system security plan, and all of your policies, on being the best that you can.
Jeff: In my opinion.
Amira: Yes, absolutely. Always that caveat. And the DoD has the final word on everything.
Jeff: The DoD sets policy.
Wrap up
Amira: Thank you for this, I have lots of additional questions about assessments. The next topic is putting together an assessment plan.
Jeff: There are a lot of interesting nuances. Our goal is to make a fair, unbiased, consistent process which can scale to 300,000 companies. That is our goal. When it comes to processes, I am very much a goal person. What is the goal, how do we design our processes to accomplish it? There is a lot to this topic, I think Organizations Seeking Certification (OSCs) (defense contractors) will be interested in it, so I am happy to keep doing these.
Next articles:
CMMC-AB Jeff Dalton – the CMMC Assessment Process – Part 2
CMMC Scope – are you ready for an assessment?
CMMC Compliance FAQs – Organizations seeking certification
Thanks for the read, all!
Please share this with Defense Industrial Base contractors who are interested in getting CMMC certified. This series of interviews should be very helpful in understanding the assessment process.
Send me a connection on LinkedIn, sign up for our newsletter, and share this site with others! Please comment if you know more, or submit an article with an in-depth analysis on a topic.
Thanks again to the CMMC-AB for authorizing this interview and helping the ecosystem stay on track.
-Amira
V. Amira Armond (CISSP, CISA, PMP, MBA) is a computer systems architect, cyber-security consultant, and owner of Kieri Solutions LLC.
Kieri Solutions LLC is in progress to become a CMMC assessment organization and has several Registered Practitioners and Certified Assessor candidates on staff. Amira is also the chief editor for cmmcaudit.org, a public resource for news and informational articles about the Cybersecurity Maturity Model Certification.