tag:www.aemstatus.net,2005:/historyEdge Delivery Services for Adobe Experience Manager Status - Incident History2024-03-28T18:40:32ZEdge Delivery Services for Adobe Experience Managertag:www.aemstatus.net,2005:Incident/203185762024-03-20T07:00:00Z2024-03-21T11:07:49ZDelayed GitHub Actions Notifications<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Resolved</strong> - Triggering GitHub actions events for resource published, or unpublished is being delayed significantly. All actions will be triggered eventually, just with some time delay.</p>tag:www.aemstatus.net,2005:Incident/203185372024-03-18T04:00:00Z2024-03-21T11:02:02ZSlow Index Updates<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Resolved</strong> - Index updates are processed with significant delays.</p>tag:www.aemstatus.net,2005:Incident/200343642024-02-21T02:22:43Z2024-03-04T19:58:44ZPublishing Issues Observed<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>02:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>02:02</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting site publishing for AEM customers. Page delivery is not affected, but updates to your sites may be delayed or prevented. The issue is under active investigation.</p>tag:www.aemstatus.net,2005:Incident/198860112024-02-01T19:36:20Z2024-02-02T21:31:01ZPage Delivery Issues Observed<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>19:36</var> UTC</small><br><strong>Resolved</strong> - All pending form submissions have been processed as of 19:33 UTC on Thursday, February 1.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>18:14</var> UTC</small><br><strong>Identified</strong> - We experienced a service disruption beginning at 15:16 UTC on Thursday, February 1 that caused a delay in form submissions.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>16:19</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting page delivery for AEM customers. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>tag:www.aemstatus.net,2005:Incident/197666652024-01-19T15:54:11Z2024-01-26T16:03:21ZPage Delivery Issues Observed<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>15:54</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>15:47</var> UTC</small><br><strong>Update</strong> - Root cause seems to be https://www.cloudflarestatus.com/incidents/t36xcbnxfnng</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>15:43</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>15:38</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. One of our CDN provider is experiencing performance issues, we are switching away from Cloudflare to mitigate.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>15:38</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>15:29</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting page delivery for AEM customers. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>tag:www.aemstatus.net,2005:Incident/197170412024-01-15T23:07:22Z2024-01-26T16:03:36ZPage Delivery Issues Observed<p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>23:07</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>15</var>, <var data-var='time'>22:34</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting page delivery for AEM customers. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>tag:www.aemstatus.net,2005:Incident/194481192023-12-15T15:09:48Z2023-12-15T15:11:49ZFastly Image Optimizer issue<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>15:09</var> UTC</small><br><strong>Resolved</strong> - We discovered that the errors were caused by an issue with a specific IO region in Fastly. The impact of the incident lasted for approximately 22 minutes, from 11:42 to 12:04 UTC. During that period we saw an increase of the media delivery error rate from 0% to 4.8%. About 50% of the errored requests were initiated by Bots. Customer impact has been negligible.<br /><br />We will continue to closely monitor error rates of the the infrastructure we are using and take action if necessary.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>12:12</var> UTC</small><br><strong>Investigating</strong> - We are observing an increased error rate on media delivery.</p>tag:www.aemstatus.net,2005:Incident/194256812023-12-12T04:00:00Z2023-12-15T12:18:16ZCloudflare Workers Issue<p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Resolved</strong> - A small number of customers running on the Cloudflare stack were affected by the Cloudflare Workers Issue https://www.cloudflarestatus.com/incidents/m6y3t8mxy1c7 that caused slow load times and elevated error rates on hlx.page. This lead to a minor disruption of authoring and preview, but no impact to delivery.</p>tag:www.aemstatus.net,2005:Incident/193747462023-12-08T12:27:35Z2024-02-02T10:38:16ZSlack Bot is unable to upload images or videos<p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>12:27</var> UTC</small><br><strong>Resolved</strong> - The Slack bot now provides correct instructions on uploading files through Sharepoint, Google Drive and Sidekick. Direct uploads from Slack are no longer possible, due to changes in Slack's API and the deprecation of the bot skill.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>09:57</var> UTC</small><br><strong>Investigating</strong> - The FranklinBot Slack bot is currently unable to upload images or videos to content bus. We are investigating the issue. As a workaround, upload your content to Sharepoint or Google Drive first, then use the Sidekick to preview and publish the file.</p>tag:www.aemstatus.net,2005:Incident/189827582023-10-30T20:20:00Z2023-12-15T12:18:16ZDelivery Increased Error Rate<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>20:20</var> UTC</small><br><strong>Resolved</strong> - Between 19:53 UTC and 20:20 UTC we observed an increased error rate of about 2% of origin requests due to an incident in of one of our cloud providers https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc</p>tag:www.aemstatus.net,2005:Incident/188424602023-10-18T09:18:00Z2023-12-15T12:18:16Z[Publishing Issues Observed] Sidekick Library<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>09:18</var> UTC</small><br><strong>Resolved</strong> - We are observing issues that are affecting site publishing for AEM customers. Page delivery is not affected, but updates to your sites may be delayed or prevented. The issue is under active investigation.<br /><br />A limited set of customers with certain configurations may have seen Sidekick Libraries be impacted by an unrelated change. We are actively working to determine the scope of the impact.<br /><br />The issue has been resolved in the meantime.</p>tag:www.aemstatus.net,2005:Incident/186333092023-09-28T19:39:02Z2023-12-15T12:18:16ZOngoing DNS Disruption for hlx.page and hlx.live<p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>19:39</var> UTC</small><br><strong>Resolved</strong> - We have received no more messages from users about DNS resolution errors and consider this closed. As always, a post-mortem will follow in a few days after we had a chance to regroup.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>17:55</var> UTC</small><br><strong>Monitoring</strong> - The move to new DNS infrastructure has been completed. We are now monitoring the situation worldwide and will resolve once we have certainty that all users can access our services as before.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>16:52</var> UTC</small><br><strong>Update</strong> - We are seeing DNS service recovering globally, although some locations and ISPs still fail to resolve. This is due to a switch in DNS providers at the registrar, which can require some time for caches to expire.</p><p><small>Sep <var data-var='date'>28</var>, <var data-var='time'>15:36</var> UTC</small><br><strong>Identified</strong> - The DNS servers that are providing services for hlx.page and hlx.live are still experiencing issues. This can lead to slow or failing DNS resolution, with varying impact, depending on network provider. We are working with the Adobe DNS team to provide a resolution.</p>tag:www.aemstatus.net,2005:Incident/186111702023-09-26T00:00:00Z2023-12-15T12:18:16ZDNS Disruption for hlx.page and hlx.live<p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>00:00</var> UTC</small><br><strong>Resolved</strong> - Adobe has been experiencing issues with the Domain Name System (DNS) provider that serves hlx.page and hlx.live that caused these domains to be unresolvable from certain networks. Changing the DNS provider to 1.1.1.1 (Cloudflare) or 8.8.8.8 (Google) provided an effective workaround.</p>tag:www.aemstatus.net,2005:Incident/175625252023-06-13T20:53:21Z2023-12-15T12:18:17ZPage Delivery And Publishing Issues Observed<p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>20:53</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>13</var>, <var data-var='time'>18:54</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that is affecting our delivery services for AEM customers. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>tag:www.aemstatus.net,2005:Incident/175047312023-06-07T19:12:33Z2023-12-15T12:18:17ZPublishing Issues Observed<p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>19:12</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 7</var>, <var data-var='time'>18:56</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting site publishing for Project Helix customers. Page delivery is not affected, but updates to your sites may be delayed or prevented. The issue is under active investigation.</p>tag:www.aemstatus.net,2005:Incident/162480662023-02-23T11:44:48Z2023-12-15T12:18:17ZPublishing Issues Observed<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>11:44</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>11:27</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting site publishing for Project Helix customers. Page delivery is not affected, but updates to your sites may be delayed or prevented. The issue is under active investigation.</p>tag:www.aemstatus.net,2005:Incident/160743972023-02-09T10:18:49Z2023-12-15T12:18:17ZGitHub Synchronization Issues Observed<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>10:11</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting site publishing for Project Helix customers. Page delivery is not affected, but updates to your sites may be delayed or prevented. The issue is under active investigation.</p>tag:www.aemstatus.net,2005:Incident/159301102023-01-25T10:50:38Z2023-12-15T12:18:17ZPublishing Issues Observed<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>10:50</var> UTC</small><br><strong>Resolved</strong> - The Azure Systems have recovered.<br /><br />From 7:14AM to 9:29AM UTC we saw 25 related errors of failed publish actions. The impact was therefore only minor.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>09:41</var> UTC</small><br><strong>Investigating</strong> - Azure is having problems at the moment which is affecting authoring and publishing of Franklin content: https://status.azure.com/en-us/status<br /><br />The issue is under active investigation.</p>tag:www.aemstatus.net,2005:Incident/159245402023-01-24T18:21:00Z2023-12-15T12:18:17ZIncreased error rate with the Franklin Admin Service.<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>18:21</var> UTC</small><br><strong>Resolved</strong> - Franklin experienced a temporary increased error rate due to a problem with one of our cloud providers. The outage effected user of the Franklin Admin Service and the Sidekick.<br /><br />This incident has been resolved.</p>tag:www.aemstatus.net,2005:Incident/158871852023-01-19T23:16:55Z2023-12-15T12:18:17ZPage Delivery Issues Observed<p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>23:16</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>23:07</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting page delivery for Project Helix customers. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>tag:www.aemstatus.net,2005:Incident/156963612022-12-22T22:20:12Z2023-12-15T12:18:17ZPublishing Issues Observed<p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>22:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>11:09</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting site publishing for Project Helix customers. Page delivery is not affected, but updates to your sites may be delayed or prevented. The issue is under active investigation.</p>tag:www.aemstatus.net,2005:Incident/136731512022-11-25T14:20:33Z2023-12-15T12:18:17ZAuthoring issues observed<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>14:20</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>13:48</var> UTC</small><br><strong>Monitoring</strong> - The issue has been identified and a fix has been applied. Authors should no longer experience any errors, but we will continue to monitor the situation.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>13:11</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting page authoring for Project Helix customers in SharePoint. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>tag:www.aemstatus.net,2005:Incident/127182392022-11-04T17:18:41Z2023-12-15T12:18:17ZPage Delivery Issues Observed<p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>17:18</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 4</var>, <var data-var='time'>16:34</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting page delivery for Project Helix customers. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>tag:www.aemstatus.net,2005:Incident/104363632022-06-29T04:00:00Z2023-12-15T12:18:17ZIncreased error rate on media delivery<p><small>Jun <var data-var='date'>29</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Resolved</strong> - From 01:03 UTC to 04:04 UTC we saw increased error rates for media requests.</p>tag:www.aemstatus.net,2005:Incident/103560192022-06-20T04:45:50Z2023-12-15T12:18:17ZForm Processing Issues Observed<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>04:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>00:45</var> UTC</small><br><strong>Investigating</strong> - We are observing issues that are affecting form processing for Project Helix customers. The issue is under active investigation and we are working with full effort to reach a speedy resolution.</p>