Category: Monitoring and Diagnostics

this photo by Evgeni Tcherkasski on Unsplash
Azure

The power of Azure Lighthouse

One of the biggest challenges facing partners offering support around Azure is how best to permission access across multiple customer tenancies. Specifically how to assign the correct level of permissions to users that require them and how to monitor the users once access is enabled.

Another problem that goes hand in hand with this is how to centrally manage and monitor large volumes of resources, spread across various customers, whilst using a single shared set of tools and dashboards.

Until Azure Lighthouse, access for support would often be configured in one of the following ways:

  • Dedicated accounts within the customer tenancy (Azure AD) for partners to use for support
  • Partner accounts added as guests to every customer tenancy using Azure B2B

Contacting customers every time the partner takes on a new member of the support team. Having permissions assigned to individual guest accounts rather than security groups. Support staff switching constantly between directories. Are just some of the problems that highlight that neither option is suited to offering support at scale.

CSP partner accounts with enough permissions can use Administer-on-Behalf-Of (AOBO) to offer support. AOBO offers access to customer subscriptions at an administrator level. This method of access is generally a concern to both customers and partners. Would you really want a backup admin having full access to all services?

Azure Lighthouse uses delegated resource management to project resources from customer Azure environments, into a single partner tenancy. Delegated partner accounts can then view resources from all customer tenancies within a single portal for centralised management.

At a high level, security groups are created in the partner’s tenant, based on support roles. Users are then assigned to those security groups. In the customer’s tenancy, the required amount of permissions can then be assigned to these security groups using RBAC roles. Delegate access for these groups can then be assigned at the subscription or resource group level.

Benefits include:

  • Resources spread across multiple customers can be managed from a single portal
  • Single instances of native Azure services can be used, for example, Service Health, Azure Advisor, Azure Monitor, Security Center (A full list of enhanced services can be found here)
  • Governance can be applied centrally across all of your tenants for consistency
  • Share dashboards can be populated with information from multi-customers
  • Azure security across the entire customer base is easily visible from one location
  • More opportunity for unified automation and reporting using scripting and APIs

Ultimately, this should enable easier management at scale across your entire customer base. No more switching directories!

Delegating access or “onboarding” can be done in one of two way.

  1. Deploying an ARM template into the client’s subscription to enable delegation management as defined within the template
  2. Customer offer that has been previously published to the Marketplace.

NOTE: It is not currently possible to do this via the portal, Azure CLI or cloud shell.

Using an ARM template to Onboard

Microsoft has made available various sample templates that can be found in this Github Repository. In the following example, I’m going to be onboarding a customer at the subscription level. I’ve downloaded the template that best fits this scenario and now need to modify it to represent the offer.

Each template comes with a parameters file which needs to be populated with the following offer information:

  • mspOfferName: Name describing the offer
  • mspOfferDescription: Description of the offering
  • managedByTenantId: Partner tenant ID
  • Authorizations:  Description of each entity (user/group/service principal) from the partner tenant being granted access to the customer deployment

Having gathered the information above, I can populate the parameters.json file.

Now that I have a template for the offer, I need to deploy the template in the customer’s tenant. Although I’m running it at the subscription level, if the customer has multiple subscriptions, this would need to be repeated for each.

I’m going to use PowerShell to kick off the deployment but it could as easily be done with Azure CLI.

Once the deployment has run, I can go ahead and confirm the delegated link has been properly established.

Logged in as the customer I can browse to the Service providers blade, click on Delegations and then select the respective offering to view what roles have been assigned to the partner.

The image below shows that the Tier 1 Support and Tier 2 Support groups have been successfully assigned with the roles outlined in the parameters.json file passed to the deployment.

Having seen how this appears to the customer, let’s now switch and take a look from the service provider perspective.

Once signed in to the Azure portal as the supporting partner, I browse to the My customers blade. From here I can see the customers who I have delegated access to manage, the role assignments and subscriptions or resources groups that the partner has permissions over.

If I switch to the Virtual machines blade, I can now view virtual machines provisioned across all of my customers from the single window. This shared experience is similar across all Azure services that are now integrated with Azure Lighthouse, for example, Azure Advisor, Security Center, etc.

Removing delegation

If support is no longer required by the customer, the offering or delegation can simply be removed. Browse to the Service provider blade within the portal, select the offering or delegation and then click on the Delete button.

As you would expect, it’s also possible to remove the delegation using PowerShell or Azure CLI as outlined in this Microsoft article. https://docs.microsoft.com/en-us/azure/lighthouse/how-to/onboard-customer#remove-access-to-a-delegation

Recommendations

When planning to integrate Azure Lighthouse for the first time, there are a couple of security recommendations and general best practices that are worth keeping in mind.

  • Multi-Factor Authentication should be enforced
  • Use the least privilege principle when assigning permissions
  • Consider using Privileged Identity Management (PIM)
  • Where possible try not to assign access to entire subscriptions unless required
  • Use Azure Policy to enforce standards across all of your customers
  • Include a user with the Managed Services Registration Assignment Delete Role
  • Any user who needs to view the My customer’s page in the Azure portal has the Reader role (or another built-in role which includes Reader access).

For a more in-depth look into the Azure Lighthouse service and what it offers, take a look at Microsoft docs: https://docs.microsoft.com/en-us/azure/lighthouse/overview

Azure

Azure: Using Service Health to alert about planned maintenance

Microsoft periodically perform updates to improve the reliability, performance, and security of their global cloud platform. Most of the updates that Microsoft apply have no impact on the hosted virtual machines and go unnoticed. However, there are some instances where the running virtual machines are affected, the most recent example of this being the Spectre and Meltdown CPU vulnerability.

When an update like this is applied, depending if the maintenance requires a rebooted or not, it can affect running virtual machines in one of two ways:

  • In-place migration – During an in-place migration the affected Azure virtual machine is paused (typically for 30 seconds or less) to preserve memory in RAM while the host environment is updated. Once the upgrade is complete the virtual machine is resumed and the system clocks synchronised
  • Maintenance requiring a reboot – During reboot maintenance, the virtual machine is moved to a node that has already been patched and then powered back on.

Under normal circumstances, when a virtual machine needs rebooting, you are notified in advance and given the option to start the maintenance at a preferred time during the initial self-service window. If the self-service window is missed, the scheduled maintenance window begins and it is no longer possible to manage the process.

NOTE: Datacentre in paired regions will not have maintenance performed at the same time; therefore, workloads balanced across paired regions will be unaffected.

Azure Service Health


Azure Service Health (in public preview at the time of writing) can be used to view problems with Azure services that may impact any of your cloud services. Service Health monitors three types of health event:

  • Service issues – Azure services that are currently experiencing problems
  • Planned maintenance – Any known future maintenance that may affect the availability of your services
  • Health advisories – Changes in services, for example, deprecated features or exceeded quota usage.

Using the Service Heath service, it is not only possible to view in one location any service problems but also setup Health alerts which are a vast improvement on the basic email notification. Service alerts can be configured to:

  • Send Email/SMS/Push/Voice notification
  • Send Webhooks to third party app i.e. ServiceNow or Azure Logic App
  • Send an IT Service Management Ticket
  • Trigger an Automation Runbook.

Creating planned maintenance alerts using Azure Service Health

To demonstrate the alerting capabilities of the Health Service, the rest of this article steps through the process involved in configuring a basic email alert to notify of any planned maintenance events.

1. Login into the Azure portal and select Service Health.
2. Select Health alerts followed by + Create service health alert from the top of the window on the right.


3. In the Edit Alert blade, give the alert a Name, Description, check the subscription is correct and choose a resource group.


4. The next step is to work through the Criteria section choosing which services, regions and types of event alerts should be monitored. For the purpose of this article all services and regions have been checked but only planned maintenance events.


5. Select or create an Action group. (An Action group is a group of actions to be taken, should an event be logged.)


6. Configure the actions to be taken. We are only configuring an email alert, so we first name the action, then chose Email/SMS/Push/Voice from the drop down list.


7. Enter the name and email address of the person who will be sent the notification. Then finally click OK twice to add the activity log alert.


It should now be possible to see the newly created alert, listed in the Health alerts blade.


Should action groups need to be edited or new actions added, either click on Edit this action group from the lower half of the Health alerts blade or browse to the Monitor service and select the Action groups tab.

For more details checkout the Microsoft document site https://docs.microsoft.com/en-us/azure/service-health/service-health-overview