Resolved: Certain document list operations failing on a few solutions

Update: 08-05-2023 15:15

We haven't registered any new errors on any solutions. We consider the issue resolved. If Microsoft provide us with feedback regarding the root cause we will update the status accordingly.

Update: 04-05-2023 08:45

Since yesterday 21:32 UTC+2 we haven't registered any new errors on any solutions. We will keep monitoring the situation.

Update: 03-05-2023 14:00

We are still monitoring the situation. It's still only affecting a small subset of solutions with a very limited impact. We have notified relevant partners directly about customer solutions we have identified being affected by the issue. 

Microsoft is engaged in a dialog with us and we are awaiting their feedback.

Update: 02-05-2023 16:30

We have today noticed certain document list operations performed through SharePoint CSOM failing on a subset of solutions.

We have raised an issue at Microsoft Support regarding this and are awaiting response from them.

We will monitor the situation and update the status accordingly.

Resolved: Microsoft is currently experiencing an issue with Azure Redis Cache - Connectivity Issues that may impact WorkPoint 365 backend

Update: 16-03-2023 17:30

From Microsoft:

We mitigated this by stopping the deployment which was causing these issues, and can confirm that this has mitigated impact for customers. We are still in the process of rolling out the hotfix as a long term fix.

Update: 16-03-2023 14:30

From Microsoft:

We have identified the potential root cause to be a bug which was introduced during a recent deployment and is causing the unexpected failovers to occur. We are working to pause the deployment to avoid further disruption. There is no current workaround for this issue, however, once failover process is completed for all nodes of a cache resource, cache health should return to normal. We are also working on a hotfix that will be deployed in the coming days to resolve the underlying issue.

Update: 16-03-2023 14:00

From Microsoft:

We have identified the potential root cause to be an issue with a recent deployment. We are in the early stages of developing a hotfix to mitigate this, in the meantime, we are looking into pausing the deployment to avoid further disruption. The next update will be provided in 2 hours, or as events warrant.

Update: 16-03-2023 12:00

From Microsoft:

We have identified the cause of this to be an issue with a recent deployment. We are in the early stages of developing a hotfix to mitigate this issue. 

Update: 16-03-2023 11:00

From Microsoft:
Customers using Azure Redis Cache may experience some service degradation such as unexpected failovers, timeouts and intermittent connectivity issues.

Resolved: Microsoft is experiencing an issue with Azure Key Vault and Azure Storage

Update: 06-03-2023 19:35

Microsoft reports that this incident has been mitigated.

Update: 06-03-2023 18:55

From Microsoft health status:

We have applied configuration changes to bring the service back to a healthy state, and we are currently monitoring our telemetry to ensure full-service functionality is restored. Some customers may already be experiencing signs of recovery as part of our mitigation processes.

Update: 06-03-2023 14:50

From Microsoft health status:

Current Status: We have identified that a spike in incoming traffic has caused components of our service to become unable to process incoming requests. This impact is limited to a single stamp in this region. We are currently in the process of performing recovery actions to alleviate customer impact.

Update: 06-03-2023 14:00

From Microsoft health status:
An emerging issue is under investigation. We are currently investigating an issue with Azure Storage in West Europe. Azure services dependent on Azure Storage may experience intermittent failures and degraded performance. We are currently investigating, and further updates will be provided in 60 minutes, or as events warrant. 

Update: 06-03-2023 13:30

Microsoft is experiencing an issue with Azure Key Vault. 

This affects customers running WorkPoint 365 version 4 system processes and custom endpoints.

Resolved: WorkPoint 365 v.4 - Processes randomly fails to load

Update: 21-02-2023 13:00

The root cause has been identified and resolved. 

Update: 21-02-2023 10:00

Processes have randomly and infrequently started not to load correctly in the browser displaying an error message indicating an issue with an invalid hexadecimal character. We are currently working on finding the root cause and resolving the issue.

Resolved: WorkPoint 365 Web API not responding correctly

Update: 08.02.2023 09:00

The Web API is responding correctly again.

Update: 08.02.2023 07:45

We are currently experiencing an issue with our WorkPoint 365 Web API. We are working on a resolution.

Resolved: Microsoft is currently having issues with Microsoft 365 services worldwide.

Update: 13:13, 25-01-2023

Current status from Microsoft : We're monitoring telemetry which continues to show that the service is stable, and the majority of users are able to access the service successfully. 
Microsoft 365 Service health status (office365.com)

Update: 09:22, 25-01-2023

You can follow the status about the Microsoft 365 service issue at:
Microsoft 365 Service health status (office365.com)

Resolved: WorkPoint 365 App issue

Update: 21:15, 09-01-2023

The root cause for the issue has been identified and the issue has been resolved.

Affected customers will be informed about the incident directly.

Update: 18:30, 09-01-2023

WorkPoint solutions running on App14, App15 and App17 are currently unable to communicate with the WorkPoint 365 backend and process requests. We are currently working on resolving the issue.

Resolved: Unable to view license information

UPDATE: 20:00, 01/009-2022

The issue is solved with the 3.17.1 hotfix

UPDATE: 12:00, 25/08-2022

We currently have an issue showing all users under license information.

Until we have fixed the issue, please contact support in case you a list of user.

Thank you

Resolved: Microsoft SharePoint Search not working

UPDATE: 10:10, 22/06-2022

Microsoft has confirmed that the service outtake affecting various Office 365 services has been restored.

The root cause was according to Microsoft a infrastructure power outage. Please refer to MO394389 in Office 365 admin health history for details.

UPDATE: 12:30, 21/06-2022

Microsoft is having an ongoing issue with SharePoint Online Search service. This also impacts areas of WorkPoint 365 and WorkPoint Express, where SharePoint Search is utilized. 

We suggest to follow the status of the issue in your tenant on the Microsoft 365 Health status page located at:
https://portal.office.com/adminportal/home?#/servicehealth/

Solved: WorkPoint Express datastore connection issue

UPDATE: 13:50, 09/06-2022

The issue has been resolved.

UPDATE: 13:30, 09/06-2022

We are currently experiencing an issue with the WorkPoint Express datastore.

Solved: WorkPoint Express document search issue

UPDATE: 08:00, 17/03/2022

The issue was resolved in WorkPoint Express version 6.2.1 released on March 17th.

UPDATE: 11:00, 10/03/2022

Because of a recent Microsoft search change, it is currently not possible to search for documents through WorkPoint Express.

As a workaround until we get it fixed, you can right-click on the entity and select "Go To Search".

The issue will be solved in the next version 6.2.1 to be released on March 17th.

SOLVED: WorkPoint 365 Web API Performance issue

UPDATE: 12:30, 23/11/2021

Our WorkPoint 365 Web API is responding correctly again and all services are working as expected.
We are still in the process of analyzing the root cause.

UPDATE: 11:30, 23/11/2021

We are currently experiencing an performance issue with our WorkPoint 365 Web API.

The impact can be timeouts and slow responsiveness in features depending on the API.

SOLVED: WorkPoint Express datastore performance issue

UPDATE: 12:00, 30/08-2021

The issue was due to a bottleneck on the backed. We've managed to resolve the issue and operation is now back to normal.

UPDATE: 11:00, 30/08-2021

We're currently experiencing high load on the WorkPoint Express datastore.

The impact for Express users can be timeouts and slow responsiveness, when loading Express.

We're investigating the issue.

SOLVED: Site provisioning failing 

UPDATE: 8:00, 02/08-2021

The issues was fixed in the announced hotfix release. Please see the Hotfix page for more information.

UPDATE: 9:00, 30/07-2021

Subject to successful internal testing, we plan to release a hotfix tonight.

UPDATE: 13:00, 29/07-2021

When using inheritance site provisioning is currently failing. The impact is that when all buffer sites are used no new site can be created.

The error is "This field type does not support validation formulas".

We're investigating the issue and will provide an update when we have more information.

SOLVED: Task Overview webpart issue

UPDATE: 16:00, 05/07-2021

We have a fix ready for this issue which will be deployed tonight.

It is a requirement to update ModernUI to apply the fix.

UPDATE: 15:00, 02/07-2021

We've identified an issue with the Task Overview webpart.

It is being investigated and we'll provide an update when we have more information.

SOLVED: Document/Item security rule issue

UPDATE: 19:00, 24/06-2021

The issue is now fixed. Release Note

UPDATE: 14:00, 24/06-2021

We've completed the code change to resolve this issue. 

Subject to successful internal testing, we plan to release a hotfix tonight.

UPDATE: 13:00, 23/06-2021

We've identified an issue with security replication for Document/Item security rules causing it to fail.

If this type of rule isn't used, this message can be disregarded.

We'll provide an update as soon as we have a fix ready.

SOLVED: Issue introduced with WorkPoint 365 with version 3.10.2

UPDATE: 19:00, 24/06-2021

The issue is now fixed. Release Note

It is a requirement to update ModernUI to apply this fix.

UPDATE: 14:00, 24/06-2021

We've completed the code change to resolve this issue. 

Subject to successful internal testing, we plan to release a hotfix tonight. 

UPDATE:  12:30, 23/06-2021

We're still investigating issues with ModernUI, specifically Entity Details, so we recommend not updating ModernUI untill this is fully resolved. 

UPDATE:  14:30, 18/06-2021

SOLVED: WorkPoint 365 UI elements not loading issue

UPDATE:  11:00, 17/06-2021

The issue is now solved.

This was caused by the scripts used from the WorkPoint CDN being blocked by the browser.

We haven't been able to identify what change suddenly caused this, except that it wasn't a change in the WorkPoint infrastructure.

To mitigate this we've set up some CORS rules (Cross Origin Resource Sharing), to make sure that our scripts are not blocked.

UPDATE:  09:30, 17/06-2021

Some customers are currently experiencing an issue where WorkPoint UI elements are not loading.

We're investigating the issue and will provide an update as soon as possible.

SOLVED: WorkPoint 365 Web API issue

UPDATE: 11:15, 26/05-2021

Our WorkPoint 365 Web API is responding correctly again and all services are working as expected.

SOLVED: Microsoft Azure AD issues

UPDATE: 11:30, 14/12-2020

Microsoft have now solved the issue.

See below statement.

Azure Active Directory - Authentication errors - Mitigated (Tracking ID PS0T-790)

Summary of impact: Between 08:00 and 09:20 UTC on 14 Dec 2020, a subset of customers using Azure Active Directory may have experienced high latency and/or sign in failures while authenticating through Azure Active Directory. Users who had a valid authentication token prior to the impact window would not have been impacted. However, if users signed out and attempted to re-authenticate to the service during the impact window, users may have experienced impact

Preliminary root cause: We determined that a single data partition experienced a backend failure.

Mitigation: We performed a change to the service configuration to mitigate the issue. 

UPDATE: 10:00, 14/12-2020

Microsoft is currently having issues with Azure AD authentication.

This is possibly affecting all areas of WorkPoint where we need to authenticate users for some customers, but not all.

We'll monitor the situation closely and provide an update as soon as we know more.

SOLVED: Stage Model issues

UPDATE: 11:00, 04/12-2020

The hotfix ver. 3.8.1.1 is now deployed.

For a full list of issues solved in this hotfix please take a look at our hotfix release notes

UPDATE: 09:30, 03/12-2020

This applies to all solutions using Stage Change triggers in Action Management

We've identified an issue where Stage Change triggers are not triggered after upgrade to 3.8.1.

A hotfix for this is currently being developed.

Meanwhile we strongly recommend to postpone all upgrades to 3.8.1.

We'll provide an update when the hotfix is ready for deployment.

SOLVED: Classic stage webpart issue

UPDATE: 11:00, 04/12-2020

A fix for this issue was included in the hotfix ver. 3.8.1.1.

It is not necessary to wait for the 3.8.2 version.

For a full list of issues solved in this hotfix please take a look at our hotfix release notes

UPDATE: 16:00, 02/12-2020

This only applies to Classic UI solutions upgraded to 3.8.1.

The Stage webpart does not render the stage model correctly, as only the current stage is shown.

It is still possible to change stage using the Change Stage button from the My Tools menu.

We'll include a fix for this issue in the 3.8.2 version, that will be released on December 19th.

We recommend customers to wait for the 3.8.2 release before upgrading from 3.7.5 versions or earlier.

SOLVED: Sites are not created

UPDATE: 16:15, 13/11-2020

The issue has now been resolved.

The issue was limited to site creation through the WorkPoint API (manual site creation and integration). Automatic site creation was not affected by this issue.

UPDATE: 13:45, 13/11-2020

We're currently having issues with site creation. We're currently investigating the issue and will provide and update as soon as we have more information.

SOLVED: Issue with Azure AD authentication

UPDATE: 11:45, 3/11-2020

Today, WorkPoint identified a brief issue with Microsoft's Azure AD authentication which impacted all WorkPoint customers and users. After reporting the issue, it has now been solved.

The issue which was identified today at around 10:30 caused users to experience limited access to functionalities in the WorkPoint 365 interface and that some available functions would not be working. In addition, some users could not access their WorkPoint solution via WorkPoint Express.

Microsoft has now solved the issue and customers should no longer be impacted by this.

SOLVED: WorkPoint Express cannot startup

UPDATE: 10:30, 28/09-2020

We have tuned the database and the issue is now solved.

During our analysis we can see that this issue started around 8.10 and operation was back to normal at 10.15

UPDATE: 10:00, 28/09-2020

SOLVED: Expiry of the WorkPoint365.com certificate

UPDATE: 09:31, 07/09-2020

SOLVED: An issue with WorkPoint's Web API caused WorkPoint Express to fail

UPDATE: 16:04, 24/06-2020

The issue with WorkPoint's Web API has been solved by Microsoft and users should no longer experience issues with WorkPoint Express.

The issue was due to maintenance work by Microsoft but has now been fully solved. Both WorkPoint and Microsoft continues to monitor the WorkPoint Web API to secure that no further issues should arise. 

SOLVED: Many customers are currently experiencing 503 errors

UPDATE: 16/04-2020 20.50

The capacity on SharePoint is now back to normal levels and all customers may now resume full regular use of their WorkPoint solutions.

UPDATE: 3/4-2020 10.30

We've received reports from customers that they receive 503 - Server Unavailable errors.

This is SharePoint Online responding that the service isn't available. This can’t be fixed in WorkPoint, therefore we encourage all customers to report this directly to Microsoft from your own Office365 tenant.

WorkPoint has also contacted Microsoft to get the service restored. We'll provide an update as soon as we have new information to share.

UPDATE: 27/03-2020 15.00

With the latest corrections from Microsoft, we are now happy to inform You, that all customers can now resume using WorkPoint Express and our WorkPoint Office add-ins. We no longer expect our customers to experience the SharePoint servers being unavailable, thus users should only exceptionally experience related error messages.

Some customers may still occasional reduced system performance, especially with regards to creating site collections, due to the generally reduced capacity on SharePoint.

The following blog is now available with some additional information about these temporary changes: Awareness of temporary adjustments in SharePoint Online 

We also advise you to take a look at SharePoint Incident SP207374 in your Microsoft 365 Admin Center. 

UPDATE: 25/03-2020 16.00

We've been monitoring all customer solutions during the day and have only seen very few throttling issues (503 errors), however we are currently maintaining our previously announced recommendations to minimize traffic against SharePoint.

UPDATE: 25/03-2020 08.00

Microsoft have found some unexpected throttling (503 errors) and have addressed the issue during the night. We're monitoring all customer solutions to confirm if the change is working as expected.

UPDATE: 25/03-2020 22.00

The SharePoint Engineering Team at Microsoft is investigating possible solutions

UPDATE: 23/03-2020 22.30

Our Microsoft case has now reached the SharePoint Product Group.

Tomorrow we expect a public announcement from Microsoft regarding the issue.

WorkPoint will also provide some guidelines/actions to minimize traffic against SharePoint.

UPDATE: 23/03-2020 12.00

Microsoft is still having issues with their service.

Unfortunately we haven't been provided with a time frame for a solution.

It is primarily the APIs to SharePoint that is affected, so as a workaround customers can use the browser to access WorkPoint/SharePoint.

UPDATE: 20/03-2020 12.00

We're working with Microsoft and some customers to collect trace logs.

The issue has also been escalated within Microsoft.

SOLVED: Unexpected error in wizards

Updated 11:01, March 13th, 2020

The previously identified issue which resulted in unexpected errors when trying to launch wizards from WorkPoint 365 has now been fully solved.

SOLVED: An issue has been identified which prevents users from creating new site collections.

Updated 09:26, February 26th, 2020

The previously identified issue which prevented WorkPoint users from creating site collections has now been fully solved.