AppZen Status - Expense Reports - Workflow Stuck in PEV due to Concur API Error

Expense Reports - Workflow Stuck in PEV due to Concur API Error

Incident Report for AppZen

Postmortem

Hi,

We sincerely apologize for the issue with report workflow stuck due to Concur API error. We are glad to inform you that we have RCA in place by Concur along with the fix to avoid reoccurrence of this issue. Below is the detailed RCA.

‌

Impact as Reported:

Customers depending on AppZen services observed “Report stuck in PEV”. AppZen integration for Expense Report processing has been interrupted causing numerous customers with Expense reports to be stuck in queue. Impact to customers occurred at different periods during the event duration. Given the nature of the product service impact this was classified as high priority requiring immediate mitigation.

Root Cause:
The event was caused by a bulk creation of system users created on behalf of the customers that occurred on September 21st. These system users are necessary to allow AppZen to make calls to Concur on behalf of the customer for their integration. Specifically, some of these system users created in bulk did not replicate across the Concur Platform completely. This partial data for the system users is the root cause for the API calls to fail with Access Denied error.

Corrective Measures:
- We have mitigated the system user replication issue by forcing a replication to required sub-services. This completed and was verified September 29th and was reverified from logs on September 30th.
- We have reviewed our process to create users in bulk finding a deficiency in the handling/replication process of system users.
- We have established corrective actions that will address this issue from occurring in the future. Short term mitigation is available immediately. Full remediation to be addressed with improvement in our user creation/management automation.

‌

Regards,

AppZen Support Team

Posted Oct 28, 2021 - 15:56 PDT

Resolved

Hi,

We are happy to inform you that the Concur API Issue has been resolved.

Apologies for the inconvenience caused.

Regards,
AppZen Support Team

Posted Sep 29, 2021 - 14:07 PDT

Update

Hi,

We are happy to inform you that the Concur API Issue has been resolved.

Apologies for the inconvenience caused.

Regards,
AppZen Support Team

Posted Sep 29, 2021 - 14:06 PDT

Update

Hi,

We are happy to inform you that the Concur API Issue has been resolved.

It will take some time to reflect all the reports in AppZen.

Regards,
AppZen Support Team

Posted Sep 29, 2021 - 12:10 PDT

Update

Hello,

The fix is in progress as we are in touch with Concur team in order to get this resolved.
We will do the best we can to keep you updated on the progress.

Thank you for your patience and cooperation.

Regards,
AppZen Support Team

Posted Sep 29, 2021 - 07:05 PDT

Update

Hello,

The implementation and fix are in progress in collaboration with Concur team.
We will make sure to give updates on the progress of the restoration.

Thank you for your patience and cooperation.

Regards,
AppZen Support Team

Posted Sep 29, 2021 - 03:35 PDT

Update

Hi,

We are in touch with concur team in order to get the fix implemented as early as possible.
We will keep you updated on the progress of the restoration.

Regards,
AppZen Support Team

Posted Sep 29, 2021 - 00:25 PDT

Update

Hi,

We are happy to inform you that we have identified the fix for Concur API Issue and currently the fix is being implemented with the help of Concur.

The fix will be deployed soon in order to get this resolved. We will keep you updated on the progress of the restoration.

Regards,
AppZen Support Team

Posted Sep 28, 2021 - 16:06 PDT

Update

Hi,

We are still actively working with Concur on this issue and we will be sure to provide timely updates on the progress.

Apologies for the inconvenience caused.

Regards,
AppZen Support Team

Posted Sep 28, 2021 - 13:26 PDT

Update

Hi,

We have raised and high-priority incident with Concur Support for the Concur API Issue. This specific issue is causing delays for ingestion as well as reports stuck in Pending External Validation.

We are working closely with the Concur team to come to a resolution and this is our top priority. We will be sure to provide continuous updates on the progress of the restoration.

Apologies for the inconvenience caused.

Regards,
AppZen Support Team

Posted Sep 28, 2021 - 10:10 PDT

Update

Hi,

We are continuing to observe the intermittent delays in workflow status in Expense Audit after actioning a report due to a delay in response from Concur API. We are continuously following up with Conur on further updates on a resolution.

We sincerely apologize for the inconvenience caused due to this.

Regards,
AppZen Support Team

Posted Sep 28, 2021 - 05:55 PDT

Update

Hi,

We have noticed intermittent delays in the workflow status - Ingesting report as well as reports stuck in Workflow due to a delay in the response from the Concur API issue for some applicable customers. We are continuing to follow up with Concur for further updates.

We sincerely apologize for the inconvenience caused due to this.

Regards,
AppZen Support Team

Posted Sep 27, 2021 - 18:32 PDT

Update

We are continuing to monitor for any further issues and updates.

Posted Sep 27, 2021 - 16:26 PDT

Update

We are continuing to monitor for any further issues and updates.

Posted Sep 27, 2021 - 13:42 PDT

Update

We are continuing to monitor for any further issues.

Posted Sep 27, 2021 - 07:57 PDT

Monitoring

We have noticed intermittent delays in the workflow status updating in Concur after actioning a report due to a delay in the response from the Concur API for some applicable customers.

We have opened a case with Concur to investigate further. According to the update on Concur Status page, there is an issue with their Data Center and currently Concur is working on fixing the issue.

Posted Sep 27, 2021 - 05:37 PDT

This incident affected: Expense Audit (Expense Audit).