Monitoring Copilot Usage

Lesson: Monitoring Microsoft 365 Copilot Usage

Introduction: Why Monitoring Matters

In the modern digital workplace, the introduction of generative AI tools like Microsoft 365 Copilot represents a significant shift in how employees interact with data, create content, and manage workflows. As an administrator, your responsibility goes far beyond simply assigning licenses. You are the steward of an environment where AI processes sensitive organizational data. If you cannot see how Copilot is being used, you cannot effectively secure it, justify its return on investment, or ensure that it is actually helping your users reach their productivity goals.

Monitoring Copilot usage is essentially about gaining visibility into the "AI heartbeat" of your organization. It allows you to understand which applications (Word, Excel, PowerPoint, Teams, Outlook) are seeing the most activity, identify potential training gaps where users might be struggling to prompt the AI effectively, and ensure that security policies are functioning as intended. Without robust monitoring, you are flying blind, potentially leaving your organization vulnerable to data over-exposure or missing out on opportunities to optimize software spend.

This lesson focuses on the administrative tools, data sources, and methodologies required to track, analyze, and report on Copilot usage within a Microsoft 365 tenant. We will move from the high-level dashboard views down to the granular data export techniques, providing you with a complete toolkit for managing the AI lifecycle in your organization.

The Microsoft 365 Admin Center: Your Starting Point

The primary destination for monitoring Copilot activity is the Microsoft 365 Admin Center. This is where you find the native reports that provide a high-level view of adoption and usage patterns. These reports are designed to be accessible, providing at-a-glance metrics that are useful for stakeholders who need to understand the impact of AI investments without needing to dive into raw server logs.

Accessing Usage Reports

To begin, navigate to the Reports section in the left-hand navigation pane of the Microsoft 365 Admin Center. From there, select Usage. Within the Usage dashboard, you will find a specific report labeled Microsoft 365 Copilot.

This report provides several key performance indicators (KPIs) that are essential for tracking adoption:

• Active Users: The number of unique users who have interacted with Copilot in a given time period.
• Copilot Interactions by App: A breakdown of how many prompts or actions were executed within specific applications like Teams or Outlook.
• Trend Analysis: A visual representation of adoption over the last 30, 90, or 180 days, allowing you to see if usage is growing or stagnating.

Callout: Adoption vs. Engagement It is important to distinguish between "Adoption" and "Engagement." Adoption, in this context, is often measured by the number of assigned licenses that are being used at least once. Engagement, however, refers to the frequency and depth of use. A user might "adopt" Copilot by opening it once, but they are only truly "engaged" if they are using it daily to draft emails, summarize meetings, or analyze spreadsheets. Focus your reporting on engagement metrics to truly understand the value of the tool.

Interpreting the Data

When you look at these reports, don't just look for high numbers. Look for anomalies. For example, if you see a sudden drop in Copilot usage in Teams, it might indicate a technical issue, a change in user policy, or simply a lack of awareness regarding new features. Conversely, if you see a massive spike, it might suggest that a team has discovered a new, highly efficient way to use the tool, which you can then document and share as a "best practice" across the rest of the company.

Advanced Monitoring with Microsoft Graph API

While the Admin Center is excellent for quick checks, it has limitations regarding customization and long-term data retention. For organizations that need to integrate Copilot usage data into their own business intelligence platforms (like Power BI or a custom SQL data warehouse), the Microsoft Graph API is the superior choice.

Understanding the Reporting API

The Microsoft Graph reporting API allows you to programmatically pull usage data. This is particularly useful if you want to correlate Copilot usage with other employee performance metrics or if you need to build custom dashboards for department heads that are not available in the standard Admin Center.

To access this data, you will need to register an application in Microsoft Entra ID and grant it the appropriate permissions. The key permission you are looking for is Reports.Read.All. Once authenticated, you can query endpoints such as:

GET /reports/microsoft365CopilotUsageUserCounts(period='D30')

This request returns a JSON object containing the daily counts of active Copilot users over the last 30 days.

Practical Implementation: Fetching Data with PowerShell

PowerShell is the most common tool for administrators to interact with the Graph API. Below is a simplified example of how you might fetch the daily Copilot usage report:

# This script assumes you have already connected to Microsoft Graph
# with the appropriate administrative permissions.

$period = "D30"
$uri = "https://graph.microsoft.com/v1.0/reports/microsoft365CopilotUsageUserCounts(period='$period')"

# Execute the request
$response = Invoke-MgGraphRequest -Method GET -Uri $uri

# The response is usually a CSV format stream from the API
$usageData = $response | ConvertFrom-Csv

# Display the data to the console
$usageData | Select-Object ReportRefreshDate, CopilotActiveUsers, AppDisplayName

Note: The data returned by the Graph API is often provided as a CSV stream. Always ensure your scripts include error handling for scenarios where the API might return a 404 or 429 (Too Many Requests) status code.

Monitoring Security and Compliance (The Audit Log)

Monitoring is not just about productivity; it is also about security. Because Copilot has the ability to access data across your Microsoft 365 environment (emails, files, chats), it is vital to monitor what Copilot is accessing. This is where the Unified Audit Log (UAL) comes into play.

Tracking Copilot Queries

The Unified Audit Log records activities related to Copilot. You can search for specific events to understand if users are querying sensitive data or if there are unusual patterns of data retrieval. To search the audit logs, go to the Microsoft Purview compliance portal and navigate to Audit.

When searching, you can filter by:

• Activity: Look for "Copilot" related activities.
• User: Focus on specific departments or high-risk users.
• Date Range: Narrow down to a specific incident window.

Identifying Potential Data Over-Exposure

A common pitfall is assuming that Copilot will only show users what they are already allowed to see. While this is true technically (Copilot respects existing permissions), users often forget that they have access to older files or shared folders that they haven't touched in years. If your audit logs show a user querying a high volume of legacy documents, it might be a sign that your organization needs to revisit its data governance and permissions strategy.

Warning: Privacy Considerations While the Audit Log is powerful, be mindful of local privacy laws and internal policies regarding employee monitoring. Ensure that your monitoring practices are transparent and that you are not collecting more data than is necessary for security and operational purposes. Always consult with your legal or HR department before implementing aggressive logging strategies.

Creating Custom Dashboards with Power BI

Once you have mastered pulling data from the Admin Center and the Graph API, the next logical step is visualization. Power BI is the industry standard for this task. By connecting Power BI to your Graph API data, you can create a centralized dashboard that tracks Copilot adoption in real-time.

Steps to Build a Copilot Dashboard

1. Data Acquisition: Use the Power BI "Web" connector to point to the Microsoft Graph API endpoints.
2. Transformation: Use Power Query to clean the data. You will often need to convert the CSV stream into a proper table format.
3. Modeling: Create relationships between your user data (from Entra ID) and your usage data (from the Graph API). This allows you to slice usage by department, location, or job role.
4. Visualization: Create charts that show:
   • Adoption Rate: Percentage of licensed users who are active.
   • Prompt Volume: Total prompts per week.
   • App Distribution: Which app is the primary entry point for AI tasks.

This dashboard becomes a powerful tool for management. Instead of just saying "people are using it," you can show that "the Marketing department has seen a 20% increase in content creation efficiency since implementing Copilot, with 80% of the team using it daily."

Best Practices for Copilot Administration

Monitoring is a process, not a one-time task. To be effective, you must establish a rhythm. Here are the industry-recommended best practices for maintaining a healthy Copilot environment:

• Establish a Baseline: Before you launch Copilot fully, measure your standard data retrieval patterns. This gives you a baseline to compare against once the AI is active.
• Use Role-Based Access Control (RBAC): Do not give every administrator access to the full audit logs. Limit access to only those individuals who need it for security investigations.
• Automate Reporting: Don't manually pull reports every Monday morning. Use Power Automate or Azure Functions to pull the data, store it in a central repository, and send an automated summary email to stakeholders.
• Review Permissions Regularly: Copilot is only as secure as your underlying file permissions. If your SharePoint sites are "open to everyone," Copilot will be able to surface that data to everyone. Use the audit logs to find sites that are being accessed frequently by unexpected users.
• Provide Feedback Loops: If your monitoring shows low usage, don't just blame the users. Reach out to them. Use surveys or focus groups to understand why they aren't using the tool. Perhaps they don't know how to prompt effectively, or the tool isn't integrating well with their specific workflows.

Common Pitfalls and How to Avoid Them

Even with the best tools, administrators often fall into traps that can lead to inaccurate data or security gaps. Here are the most frequent mistakes:

1. Ignoring "Dark Data"

Many organizations have vast amounts of "dark data"—old files, abandoned projects, and unorganized SharePoint sites. Because Copilot indexes this, it can suddenly make this old data discoverable to users who might not even know it existed.

• How to avoid: Perform a data audit before rolling out Copilot. Use Microsoft Purview to label sensitive data and ensure that access permissions are current.

2. Focusing Only on Technical Metrics

It is easy to get lost in the numbers—how many prompts, how many active users, how many errors. However, these numbers don't tell the whole story.

• How to avoid: Always pair your technical metrics with qualitative feedback. If usage is high but user satisfaction is low, there is a problem with the training or the deployment strategy.

3. Misinterpreting API Limits

The Microsoft Graph API has throttling limits. If you try to pull data for 50,000 users every hour, your requests will be blocked.

• How to avoid: Implement a "back-off" strategy in your code. If you receive a 429 error, your script should wait for a specified period before retrying.

4. Failing to Communicate Monitoring

Users often feel uncomfortable when they know they are being monitored.

• How to avoid: Be transparent. Let your organization know that you are monitoring usage to ensure the tool is working correctly and to provide better support. Frame it as "improving the experience" rather than "watching the employees."

Comparison: Admin Center vs. Graph API

Frequently Asked Questions (FAQ)

Q: Does Copilot usage data include the content of the prompts? A: No. Microsoft does not store the content of your prompts in the usage reports. The reports focus on metadata—who used it, when they used it, and where they used it.

Q: Can I see which documents a user accessed via Copilot? A: You can see the files that were retrieved or referenced by Copilot in the audit logs. However, this requires advanced auditing capabilities and is intended for security and compliance investigations, not for general productivity reporting.

Q: How long does it take for usage data to appear in the reports? A: Generally, data is available within 24 to 48 hours. It is not real-time, so do not expect to see a prompt made five minutes ago in your morning report.

Q: Is there a cost associated with using the Graph API for reporting? A: Accessing the Graph API itself is included in your M365 subscription, but you may incur costs if you are storing the data in Azure (e.g., Azure SQL or Log Analytics) or if you are using Power BI Pro/Premium licenses to view the reports.

Key Takeaways

1. Visibility is Mandatory: You cannot manage what you cannot see. Monitoring is the foundation of a successful and secure Copilot deployment.
2. Layered Approach: Use the Microsoft 365 Admin Center for quick, high-level snapshots, and the Graph API for deep, custom, and long-term analysis.
3. Security First: Always monitor the audit logs for unusual patterns of data access. Copilot is a tool that surfaces existing data, so ensure your permission structure is sound before and during usage.
4. Data-Driven Decisions: Use the data you gather to inform your training strategy. If you see that users are struggling with Excel, create a targeted training session for that specific app.
5. Automation is Essential: Don't rely on manual processes. Automate your data collection and reporting to save time and reduce the likelihood of human error.
6. Context Matters: Numbers alone are insufficient. Always supplement your technical usage reports with qualitative feedback from the users themselves.
7. Compliance is Non-Negotiable: Ensure that your monitoring practices align with your organizational privacy policies and local regulations. Transparency with your end-users is the best policy.

By following these strategies, you will transition from being a passive license administrator to an active, data-driven AI operations manager. Monitoring is the bridge between simply providing a tool and ensuring that the tool provides genuine, measurable value to your organization. Take the time to set up your reporting pipeline correctly today, and you will save countless hours of troubleshooting and justification in the months to come.