Introduction
This blog will walk you through how to use the Defender for Office 365 Recommended Configuration Analyzer Report (ORCA). ORCA is a PowerShell tool developed by Cam Murray that can audit your existing settings against Microsoft best practices to identify misconfigurations, and if remediated, will reduce your risk exposure to attacks targeting email vectors.
ORCA focusses primarily on exchange online protection (EOP) configurations which can impact Defender for Office 365 settings, formerly known as Office 365 Advanced Threat Protection (O365 ATP). These include overall EOP configuration, hyperlink protection known as ‘SafeLinks’, malicious attachment protection known as ‘SafeAttachments’ and protection against impersonation known as ‘AntiSpoofing’.
Whilst you can run ORCA on tenants without Defender for Office 365 you will only get a subset of results and recommendations. To receive the full benefit, Defender for Office 365 licensing is required which is usually provided via the M365 E5 licensing as displayed below.
Getting started with ORCA
Now you know what ORCA is, it is time to walk through the prerequisites for getting started. Firstly, you are going to need a Microsoft 365 tenant to test this on as you should always seek permission before testing anything in a live environment.
Once you have permission you will need either the Global Reader Azure Active Directory (AAD) role or View-Only Organization Management, a built-in Exchange Online role, to run ORCA. Access to the report is provided through PowerShell or using the Azure Cloud Shell via the Azure Portal.
Option 1 - PowerShell
If you’re using PowerShell to do this remotely, you will need to run an administrative PowerShell and install a couple of modules. These are the Exchange Online management shell and ORCA. The latest version of both can be pulled from the PowerShell gallery.
ORCA can be obtained from Cam’s GitHub repository or again via the PowerShell gallery using the below cmdlet.
Once installed, running the tool is straight forward using the ‘Get-ORCAReport’ cmdlet. The default output is a nicely formatted HTML report but there are a variety of output formats such as CSV, JSON or CosmosDB. Output formats can be specified using the ‘-Output’ switch and selecting an output format such as CSV.
You can see the cmdlet to run ORCA below.
Once run, the ORCA report and the Exchange Online management shell is launched and you are prompted to authenticate to Exchange Online. Enter your credentials for the tenant you are auditing, and the report will start to run.
ORCA will then generate a HTML report (or format of your choosing) that can be viewed in your browser or applicable application.
Option 2 - Azure CloudShell
The second option for running ORCA is to use the Cloud Shell built into Azure. The Cloud Shell is an interactive browser accessible shell for managing resources in the cloud (more info can be found in the Microsoft Azure documentation online). To access the Cloud Shell, select the command line icon to the far right of the main Azure search bar at the top of the screen.
Once selected, the Cloud Shell will start to load. If you have never accessed the Cloud Shell it will ask you to setup a storage account where the Cloud Shell data can persist between sessions.
After the command has run you should see a folder called ORCA has been created in the current directory.
If you navigate to the ‘ORCA’ directory using the ‘cd’ command you can run the ‘ls’ command to display the report that has been generated.
Unlike before, it is not possible to launch the report directly from the Cloud Shell and you may see an error message after the report has been created. To overcome this, we need to download the file to our local machine. The first step is to use the ‘pwd’ command to print the working directory. This gives us the path we need to download the report.
To download the report, we can select the file transfer menu option that is displayed in the below figure. Then choose the download option.
After selecting the download option, we enter the path of the file that we discovered using the ‘pwd’ command, add the relevant filename for the report, select download, and the file is saved to your local machine.
After selecting the download option, we enter the path of the file that we discovered using the ‘pwd’ command, add the relevant filename for the report, select download, and the file is saved to your local machine.
Once the Cloud Shell has loaded, we can install the ORCA module in the same manner as before via PowerShell.
At the time of writing there was no need to install additional modules for the Exchange Online shell. You can initiate the report by running the below command.
After the command has run you should see a folder called ORCA has been created in the current directory.
If you navigate to the ‘ORCA’ directory using the ‘cd’ command you can run the ‘ls’ command to display the report that has been generated.
Unlike before, it is not possible to launch the report directly from the Cloud Shell and you may see an error message after the report has been created. To overcome this, we need to download the file to our local machine. The first step is to use the ‘pwd’ command to print the working directory. This gives us the path we need to download the report.
To download the report, we can select the file transfer menu option that is displayed in the below figure. Then choose the download option.
After selecting the download option, we enter the path of the file that we discovered using the ‘pwd’ command, add the relevant filename for the report, select download, and the file is saved to your local machine.
After selecting the download option, we enter the path of the file that we discovered using the ‘pwd’ command, add the relevant filename for the report, select download, and the file is saved to your local machine.
The report
The first section of the report provides a high-level summary of the current security state. The report I generated displays results for the following security configuration features:
- Advanced Threat Protection Policies
- Anti-Spam Policies
- Connectors
- DKIM
- Malware Filter Policy
- Tenant Settings
- Transport Rules
- Zero Hour Autopurge
The below figure provides an example of the type of findings you can expect to see in the report
The overview is followed by a breakdown of each subsection, containing detailed information regarding the current configuration of Exchange Online Protection (EOP) and the Defender for Office 365 Specific security settings.
The report marks each configuration item with colour coded statuses: green (passed), yellow (non-compliant) and grey (informational). These provide a visual representation of compliance against best practices and clearly highlight the areas requiring attention to improve your email security posture. Guidance is provided via links to articles on how to configure each setting securely in the related Defender for Office 365 security feature.
The figure below provides an idea of the settings and outputs you can expect to see generated by the report.
Blogs