Conditional Access What If Tool: Test Policies Before They Test You

The Conditional Access What If tool is the closest thing Microsoft Entra ID has to a flight simulator. You get to crash the plane as often as you like, and nobody rings the helpdesk.

Conditional Access is brilliant right up until one policy, saved at 4:55pm on a Friday, quietly blocks every admin in the tenant. A single overly broad “All users” rule can do more damage in ten seconds than a year of phishing emails.

This guide covers how to test Conditional Access policies safely: report-only mode first, then the What If tool, then policy impact and the sign-in logs to prove it. I also go through reading the results, troubleshooting odd matches and a PowerShell version for the people who like a script.

It also corrects some break glass advice from the first version of this post, which has aged about as well as a Windows 98 screensaver.

Never miss a guide: subscribe here and check out my YouTube channel.

What is the Conditional Access What If tool?

Conditional Access is the bouncer for Microsoft Entra ID. Every sign-in is checked against your policies: who the user is, which app they want, where they are, what device they are on and how risky the sign-in looks.

The Conditional Access What If tool lets you ask the bouncer a hypothetical question. You describe a sign-in, it evaluates your enabled and report-only policies against it, and it lists which policies would apply and why. No real sign-in happens and no policy is changed.

Under the bonnet it uses the What If Evaluation API in Microsoft Graph, which follows the same logic as real sign-in evaluation. The older experience was more generous with its matches, so results you remember from a few years ago may not line up with what you see now.

You can simulate four types of identity:

• Users, including guest or external users.
• Workload identities, meaning single tenant service principals registered in your tenant.
• Agent identities, which are still in preview.

And three types of target resource: cloud apps, user actions (such as registering security information) and authentication context. That last one is handy if you protect sensitive SharePoint sites or PIM activations with an authentication context.

Why test Conditional Access policies before switching them on?

Several Conditional Access policies can apply to the same sign-in, and every one of them has to be satisfied. If one policy requires MFA and another requires a compliant device, the user needs both. If any applicable policy blocks, the user is blocked.

So a new policy rarely fails on its own. It fails in combination with three other policies somebody created in 2022 and named “Test 2 FINAL”.

Example scenarios, to imagine landing on a Monday morning:

• A policy meant for staff blocks external examiners or guest auditors.
• A “Require compliant device” policy stops teachers using shared classroom PCs that are not enrolled yet.
• An “All users” block policy catches your break glass accounts, which were your only way back in.

Report-only mode and the What If tool catch all three before Dave from accounts starts forwarding error screenshots to the head teacher.

Prerequisites and licensing

What you need	Why
Microsoft Entra ID P1 (included in Microsoft 365 Business Premium, E3 and E5)	Conditional Access, report-only mode and the What If tool
Microsoft Entra ID P2	Sign-in risk and user risk conditions, which come from Microsoft Entra ID Protection
Workload Identities Premium	Creating or editing policies scoped to service principals
Microsoft Purview Insider Risk Management	Insider risk conditions in your policies
A Log Analytics workspace receiving sign-in logs	The Conditional Access insights and reporting workbook
A test user (not an admin) in a pilot group	Something to test against that isn’t the CEO

Roles follow least privilege, so you don’t need Global Administrator to poke around:

Task	Least privileged role or permission
Read Conditional Access policies and configuration	Security Reader
Create or edit policies, including report-only	Conditional Access Administrator
View policy impact	Security Reader
Use the insights and reporting workbook	Security Reader plus access to the Log Analytics workspace
Run What If evaluations from Microsoft Graph	Policy.Read.ConditionalAccess

How to use the Conditional Access What If tool in 7 steps

Step 1: Name the policy so future you understands it

Pick a naming convention before you create anything. Something like CA001-Admins-RequirePhishingResistantMFA tells you the number, the scope and the intent at a glance.

When a What If result lists twelve policies, names like “New policy” and “Copy of New policy” are about as helpful as a Windows 98 error code. Keep a short register of what each policy is for and who owns it.

Step 2: Build the policy in report-only mode

1. Sign in to the Microsoft Entra admin centre as at least a Conditional Access Administrator.
2. Go to Entra ID > Conditional Access > Policies and select New policy.
3. Under Users, include your pilot group. If the policy will block or restrict sign-in once enforced, exclude your emergency access group.
4. Under Target resources, choose the apps the policy should protect.
5. Under Grant, pick the control, for example Require multifactor authentication or Require authentication strength.
6. Set Enable policy to Report-only and select Create.

Report-only policies are evaluated at sign-in and the results are logged, but grant and session controls are never enforced. Nobody gets an MFA prompt and nobody gets blocked.

Step 3: Open the Conditional Access What If tool

Go to Entra ID > Conditional Access > Policies and select What If in the toolbar at the top of the Policies page. It sits next to New policy, New policy from template and Upload policy file.

Older guides, including the first version of this one, send you to Protection > Conditional Access. That menu has moved, so if your muscle memory still goes there, your muscle memory needs a Windows Update.

Step 4: Choose the identity and target resource

Under Identity, set Select identity type to Users, Guest or external users, Workload identities or Agent identities (Preview). Then pick the exact user, guest, service principal or agent you want to test.

Under Target resource, choose Cloud apps, User actions or Authentication context. Pick the actual app the user will open, for example Office 365 Exchange Online, rather than whatever sounds closest.

Step 5: Set the sign-in conditions

Identity, target resource, device platform and client app are required. Everything else is optional and treated as none if you leave it blank.

Condition	What to enter	When it matters
Device platform	Windows, macOS, iOS, Android or Linux	Policies scoped to platforms
Client app	Browser, or mobile apps and desktop clients	Legacy authentication blocks and browser-only policies
Authentication flow	Device code flow or authentication transfer	Policies that block those flows
Insider risk	The risk level you want to simulate	Purview Insider Risk Management policies
Sign-in risk and user risk	Low, medium or high	Risk-based policies (P2)
IP address and country	A real public IP and its country	Named location policies
Filter for devices	Device properties your filters use	Policies with a device filter

Fill in every condition your policies actually use. If a policy targets a named location and you leave IP address blank, the evaluation can’t match it and the policy shows as not applying. Technically correct, practically useless.

Step 6: Run the Conditional Access What If tool and read the results

Select What If at the bottom of the form. The report shows:

• An indicator showing whether classic policies still exist in your tenant.
• Policies that will apply, with the grant and session controls the sign-in must satisfy.
• Policies that won’t apply, each with a reason.

The reason is the first condition that wasn’t satisfied, not every condition. A policy might miss on user, location and platform, but you’ll only see the first of those.

Has filter flags policies that use app filters based on custom security attributes. Check those by hand, because a simulated app may not carry the attributes you think it does.

Step 7: Prove it with report-only results, then switch it on

What If tells you what should happen. Report-only tells you what actually happened to real sign-ins, so give it a few days of normal traffic, longer if half your users only appear on a Monday. Check the results in three places:

• Sign-in logs: Entra ID > Monitoring & health > Sign-in logs. Open a sign-in and check the Conditional Access and Report-only tabs.
• Policy impact: open the policy and review its impact over the last 24 hours, 7 days or month, with a sample of sign-ins to click through.
• Insights and reporting workbook: Entra ID > Conditional Access > Insights and reporting. It needs sign-in logs streamed to Log Analytics through the SignInLogs diagnostic setting.

Result	What it means
Report-only: Success	Conditions matched and the controls were already satisfied, for example MFA was already in the token
Report-only: Failure	Conditions matched but a non-interactive control failed, such as a block or a non-compliant device
Report-only: User action required	The user would have been prompted, for example for MFA or terms of use
Report-only: Not applied	The policy conditions didn’t match, for example the user was excluded

When the numbers look right, open the policy, move Enable policy from Report-only to On and save. Then watch the sign-in logs for an hour with a cup of tea in hand.

Run Conditional Access What If checks with PowerShell and KQL

The portal is fine for one-off checks. For a monthly break glass review, the Microsoft Graph PowerShell SDK calls the same evaluation API with Test-MgIdentityConditionalAccess.

# Microsoft Graph PowerShell SDK
Import-Module Microsoft.Graph.Identity.SignIns
Connect-MgGraph -Scopes "Policy.Read.ConditionalAccess","User.Read.All"

$user = Get-MgUser -UserId "test.user@contoso.com"

$params = @{
    signInIdentity = @{
        "@odata.type" = "#microsoft.graph.userSignIn"
        userId        = $user.Id
    }
    signInContext = @{
        "@odata.type"       = "#microsoft.graph.applicationContext"
        includeApplications = @("00000003-0000-0ff1-ce00-000000000000")
    }
    signInConditions = @{
        devicePlatform = "windows"
        clientAppType  = "browser"
        country        = "GB"
        ipAddress      = "203.0.113.10"
    }
    appliedPoliciesOnly = $false
}

Test-MgIdentityConditionalAccess -BodyParameter $params |
    Select-Object DisplayName, State, PolicyApplies, AnalysisReasons |
    Format-Table -AutoSize

The app ID in that example is SharePoint Online. Use exact app IDs here, because app groups such as Office 365 or Microsoft Admin Portals don’t produce a match through the API. Set appliedPoliciesOnly to $true if you only want the policies that apply.

To summarise report-only outcomes across real sign-ins, run this in the Log Analytics workspace that receives your sign-in logs:

SigninLogs
| where TimeGenerated > ago(7d)
| mv-expand CAPolicy = ConditionalAccessPolicies
| extend PolicyName = tostring(CAPolicy.displayName), Result = tostring(CAPolicy.result)
| where Result startswith "reportOnly"
| summarize SignIns = count() by PolicyName, Result
| order by PolicyName asc, SignIns desc

A policy with lots of reportOnlyFailure results is the one to read twice before you switch it on.

Troubleshooting Conditional Access What If tool results

Unexpected matches

A policy applies to users, apps or locations you never intended. Usual causes are “All users” or “All resources” left over from early testing, nested groups inside a pilot group, or a named location that no longer matches the office IP range.

• Replace “All users” with a named pilot group while you test.
• Check group membership and nesting, and prefer static pilot groups for first deployments.
• Re-check named locations against your current public IP ranges.

Overlapping policies

Two or more policies hit the same sign-in with different controls. Conditional Access has no priority order: every applicable policy must be satisfied, so the user gets the combined requirements of all of them.

• Layer policies deliberately: an organisation-wide baseline, admin protection, then app-specific rules.
• Retire legacy catch-all policies once their replacements are enforced.
• Run What If for your key personas to confirm the combined effect.

Gaps in coverage

A policy you rely on doesn’t apply to some users, devices or apps. Often a new app was never added to scope, guests were excluded by mistake, or a “Require compliant device” policy went live before devices were enrolled in Intune.

• Target All resources with explicit exclusions rather than a hand-picked list that quietly goes stale.
• Give guests and external users their own policy.
• Roll out Intune enrolment and compliance policies before you enforce compliant devices.

Results that don’t match real sign-ins

What If says a policy won’t apply, yet users are still blocked. Check service dependencies first: What If doesn’t test them, so a Teams test ignores policies on Office 365 Exchange Online even though Teams depends on it.

Then check you supplied every condition, and remember that policies set to Off are never evaluated. For a real failed sign-in, the Entra Sign-in Diagnostic usually explains it faster than a third cup of coffee.

Break glass accounts and the What If tool

Even with careful testing, things go wrong, which is why you keep at least two cloud-only emergency access accounts with Global Administrator. The first version of this guide told you to disable MFA on them and exclude them from everything. Please don’t.

Mandatory MFA now applies to the Azure portal, the Microsoft Entra admin centre, the Intune admin centre and the Microsoft 365 admin centre, with no opt-out for break glass accounts. Current emergency access guidance looks like this:

• Protect each account with a passkey (FIDO2) or certificate-based authentication.
• Exclude them, through a dedicated group, only from Conditional Access policies that block or restrict sign-in. Report-only policies need no exclusion.
• Alert on every sign-in and test both accounts at least every 90 days.
• Run the Conditional Access What If tool against both accounts after every policy change.

The full setup, including the alert query, is in my break glass account guide.

Conditional Access What If tool gotchas

Service dependencies. What If doesn’t evaluate them. Test the dependent service as well, for example Exchange Online alongside Teams.

Off means invisible. Only enabled and report-only policies are evaluated, so a policy set to Off never appears in the results, however badly it is configured.

User actions. Report-only mode doesn’t cover the User actions scope, such as registering security information. Test those with What If and a small pilot group instead.

Service principals in groups. A policy assigned to a group that contains a service principal isn’t enforced for that service principal. Assign workload identities to the policy directly.

Rollback. If a policy still goes wrong, Microsoft Entra Backup and Recovery can help you restore it, as long as somebody can still sign in. That somebody is your break glass account.

Should you use the What If tool for every Conditional Access change?

Yes. It takes a couple of minutes per persona. Report-only mode shows you real traffic, and What If answers “what about this one user, on this device, from this country”. Together they turn Conditional Access from a dare into a change you can defend in a meeting.

If you only do three things today: put new policies in report-only first, run What If for your four personas, and fix your break glass accounts. Then switch the policy on before 4pm, not at 4:55 on a Friday.

Want to go further? My Conditional Access session controls guide and phishing-resistant MFA rollout are good next steps, and both are worth a What If test of their own.

Conditional Access What If tool FAQs

Further reading

• Troubleshoot Conditional Access using the What If tool (Microsoft Learn)
• Analyze Conditional Access policy impact (Microsoft Learn)
• Conditional Access insights and reporting (Microsoft Learn)
• conditionalAccessRoot: evaluate (Microsoft Learn)
• Manage emergency access accounts in Microsoft Entra ID (Microsoft Learn)
• Break glass account setup
• SC-300 exam study guide

Found this useful? Buy me a coffee, and if you handle subject access requests, have a look at Redact Ninja for redacting PDFs without sending files to a server.